Download Extending Player/Stage/Gazebo towards Cognitive

Extending Player/Stage/Gazebo towards
Cognitive Robots Acting in Ubiquitous Sensor-equipped Environments
Radu Bogdan Rusu, Alexis Maldonado, Michael Beetz
Brian Gerkey
Intelligent Autonomous Systems, Technische Universität München
{rusu, maldonad, beetz}@cs.tum.edu
Artificial Intelligence Center SRI, International
[email protected]
Abstract— Standardized middleware for autonomous robot
control has proven itself to enable faster deployment of robots,
to make robot control code more interchangeable, and experiments easier to replicate. Unfortunately, the support provided
by current middleware is in most cases limited to what current
robots do: navigation. However, as we tackle more ambitious
service robot applications, more comprehensive middleware
support is needed. We increasingly need the middleware to
support ubiquitous sensing infrastructures, robot manipulation
tasks, and cognitive capabilities. In this paper we describe
and discuss current extensions of the Player/Stage/Gazebo
(P/S/G) middleware, one of the most widespread used robot
middlewares, of which we are active developers, that satisfy
these requirements.
I. I NTRODUCTION
Up to a decade ago the robotics community was focusing
mainly on robots acting autonomously in real and unmodified
environments. The goal was to enable the robots to do, like
humans and animals do, all sensing, deliberation, and action
selection on board. With the establishment of new research
fields including ubiquitous computing, alternative paths to
competent robotic agency have become very plausible and
even more promising.
In ubiquitous computing, sensors, effectors, and computing devices are distributed and embedded invisibly into
the objects of everyday life. They connect automatically
to each others, exchange information, and pass commands
to connected components. In sensor-equipped environments
we have cupboards that “know” what’s inside of them
because objects are tagged with RFID tags and the boards
equipped with RFID tag readers. Or, we have people who are
instrumented with inertial measurement units at their joints
providing sensor data about the orientations of the limbs.
If we think about the future of service robotics it seems
likely that service robots will be competent and versatile
agents in sensor- and effector-equipped operating environments rather than autonomous and insular entities.
In ubiquitous robotics, a typical setting is the following
one. A service robot establishes a connection to the ubiquitous computing, sensing, and actuation infrastructure and
makes itself a part of it. Having established the connection,
the robot then perceives what is inside a cupboard in the same
way as it perceives what is in its hand: by simply retrieving
the respective sensor data and interpreting it — although it
is not physically connected to the sensor.
To enable and support such seamless integration we propose a common middleware infrastructure for autonomous
robots and ubiquitous computing environments [1], [2].
Our running example will be a mobile kitchen robot (see
Figure 1) acquiring the skills for setting a table through
imitation learning, where a sensor-equipped kitchen observes
people setting the table. The robot learns activity models
from these observations, and uses the acquired action models
as resources in order to learn high performance action
routines. This is an interesting and challenging problem for
cognitive robotics because it involves complex manipulation tasks, the acquisition and use of 3D object maps, the
learning of complex action models and high-performance
action routines, and the integration of an ubiquitous sensing
infrastructure into robotic control — aspects that are beyond
the scope of current autonomous robot control systems.
Fig. 1.
Cognitive Household Robotic Assistant
This paper gives (1) an overview of recent additions to
the Player/Stage/Gazebo software library that include drivers
and interfaces of an ubiquitous sensing infrastructure, (2) a
report on an extended and comprehensive experiment using
this infrastructure for recording kitchen activities of people
and learning action models, (3) a sketch of ongoing work to
integrate higher level support for complex manipulation and
model acquisition tasks, and (4) our plans on developing and
integrating cognitive mechanisms into P/S/G. Taken together,
these components give an overview of the current state and
the future plans of a key development thread of P/S/G.
In more detail, the main contributions of this paper are:
1) Extensions for interfacing ubiquitous sensing infrastructure include a coherently designed library of in-
terfaces to a variety of sensing and computing devices
with incompatible native software interfaces, additional
logging and synchronization mechanisms, and mechanisms for querying multiple sensors and combining
the resulting data.
2) Extensions for the integration of robotic manipulation.
The upcoming need for more sophisticated manipulation capabilities constitutes another interesting challenge for robotic middleware. Robotic manipulation in
a kitchen scenario requires reach planning in sophisticated 3D maps that are generated on the fly during
a reach motion. It also requires the recognition and
sometimes even the identification of objects, measuring
their form accurately in order to propose adequate grip
positions.
3) Extensions for enabling cognitive processing in P/S/G
will enable Player to realize self-calibrating sensorequipped environments, to effectively support the simplification of sensing tasks by exploiting contextual information, and to realize higher-level perceptual tasks
such as activity recognition as it is needed to realize
cognitive robots in human environments.
In the remainder of the paper we proceed as follows.
Section II provides an overview of our system. Section
III describes our architectural and cognitive layer. Section
IV presents our Player/Stage/Gazebo extensions in greater
detail. Section V describes our application scenario and some
experimental results. Section VI presents related work and
finally, in section VII we draw the conclusions.
II. A PPLICATION S CENARIO AND OVERVIEW
Let us now (Section II-A) look at a specific application
scenario that we will use to motivate our P/S/G extensions
and derive requirements for these extensions (Section II-B).
A. Kitchen Scenario
As the example application scenario that we consider for
the extension of P/S/G, we take an autonomous mobile robot
equipped with two arms with grippers acting in a sensorequipped kitchen environment.
The robot is a RWI B21 robot (see figures 1, 2, 5) equipped
with a stereo CCD system and laser rangefinders as its
primary sensors. It also has two Amtec Powercube arms with
simple grippers. The robot’s task will be to set the kitchen
table by getting plates and glasses out of the cupboard and
putting them on the table.
The sensor-equipped kitchen environment consists of
RFID tag readers placed in the cupboards for sensing the
identities of the objects placed there. The cupboards also
have contact sensors that sense whether the cupboard is
open or closed. A variety of wireless sensor nodes equipped
with accelerometers and/or ball motion sensors are placed on
objects or other items in the environment. Several small, nonintrusive laser range sensors are placed in the environment
to track the motions of the people acting there.
The kitchen table is equipped with a suite of capacity
sensors that essentially report the capacitance of different
areas on the table, when an object is placed there. In
addition, seven CCD cameras are mounted such that they
cover the whole environment. Finally, machines and tools in
the kitchen are also equipped with sensors.
Fig. 2.
A general overview of the AwareKitchen architecture
Small ubiquitous devices offer the possibility to instrument
people acting in the environment with additional sensors. In
our case, we have built a glove equipped with a RFID tag
reader (see Figure 7) that enables us to identify the objects
that are manipulated by the person who wears it. In addition,
the person is equipped with tiny inertial measurement units
(XSens MTx) who provide us with detailed information
about the person’s limb motions, and allow us to reconstruct
them (see Figure 3).
It is characteristic for sensor-equipped environments to
have sensors which provide redundant information, thus
evidencing that some state variables are provided by multiple
sensors. Another aspect is that the sensors only provide
partial information about the state we are interested in. For
example, the RFID tag reader in the glove senses that an
object with a certain id is close to the glove but it cannot
tell us whether the object is in the hand, the state we might
be interested in.
Fig. 3.
Images from the AwareKitchen experiment
An important property of the sensing infrastructure of
the kitchen is that the sensors are wirelessly connected to
other small ubiquitous devices (like Gumstix) and personal
computers that perform state estimation and data-mining
tasks. This way, activity data can be collected and the activity
models are acquired in a distributed manner.
As our application task we consider the following scenario
that we want to support with our extensions to the P/S/G
middleware. Several people are setting the table for different
numbers of guests. The sensor-equipped kitchen observes the
setting and learns abstract models for the setting activity.
These abstract models are then transferred to the service
robot which uses the models to imitate the coarse table
setting activity. It then uses the learned activity model as
the starting point for behavior optimization.
B. Requirements for a Middleware in Ubiquitous Robotics
In order to support future application scenarios as the one
described in the previous section, a middleware infrastructure
should successfully address the following requirements.
It has to address interoperability protocols, making communication between heterogeneous sensors and actuators
possible, and then leave room for fusion, processing and
reasoning extensions.
As an example, it should be possible to easily combine
an off-the-shelf navigation system (motors, encoders and
controller) with a range finder sensor (sonar, laser, etc)
to build a mobile platform with a minimum programming
effort, and use the same platform to develop localization and
mapping algorithms, thus making the mobile platform aware
of its location.
From an architectural point of view, the system must be
as flexible and as powerful as possible. For infrastructure to
become widely adopted by the community, it must impose
few, if any, constraints on how the system can be used.
Specifically, we require independence with respect to programming language, control paradigm, computing platform,
sensor/actuator hardware, and location within a network.
In other words, a researcher should be able to: write a
control program in any programming language, structure
the program in the best way for the application at hand,
run the program on any computer (especially low-power
embedded systems), make no changes to the program after
integrating new hardware, and remotely access the program
over a network. Though not a strict requirement, we also
aim to maximize the modularity of our architecture, so that
researchers can pick and choose the specific components that
they find useful, without using the entire system.
We also require a realistic, sensor-based simulation. Simulation is a key capability for ubiquitous computing infrastructure. The main benefits to the user of using a simulation over
real hardware are convenience and cost: simulated devices
are usually easy to use, their batteries do not run out,
and they are much cheaper than real devices. In addition,
simulation allows the user to explore system configurations
and scales that are not physically realizable because the
necessary hardware is not available. The simulation must
present the user with the same interface as the real devices, so
that moving an experiment between simulation and hardware
is seamless, requiring no changes to the code. Though they
may seem lofty, these goals are in fact achievable, as we
explain below.
As we envision technical cognitive robotic systems who
will become more and more decentralized, using ubiquitous
computing devices that communicate and exchange information between each other, one of our goals is to combine research results from both the robotics and ubiquitous
computing research communities, thus bringing them closer
together.
III. P/S/G A RCHITECTURE
The Player/Stage/Gazebo (P/S/G) project produces tools
for rapid development of robot control code. In the P/S/G
software suite, Player provides a simple and flexible interface
for communicating with, and controlling different physically
distributed sensors and actuators. Stage and Gazebo are 2D
and 3D simulators for multiple robots that include physical
simulations.
Fig. 4.
A general, brief overview of the Player/Stage/Gazebo architecture
Player provides a simple and flexible interface for robot
control by realizing powerful classes of interface abstractions
for interacting with robot hardware, in particular sensors
and effectors. These abstractions enable the programmer to
use devices with similar functionality identically from the
code point of view, thus increasing the transferability of the
code. In addition, an enhanced client/server model featuring
auto-discovery mechanisms as well as permitting servers and
clients to communicate between them in a heterogeneous
network, enables programmers to code their clients from a
large variety of programming languages. Due to the standards
that P/S/G imposes on its architectural and transport layer,
several client libraries for programming languages such as:
C, C++, Java, Lisp, Python, Ada, Tcl, Ruby, Octave, Matlab,
and Scheme, are already available [3].
An important aspect of the P/S/G middleware is that it is
developed as a Free Software infrastructure for robots and
embedded sensor systems that improves robotics research
practice and accelerates development by handling common
tasks and provides a standard development platform. By
collaborating on a common system, we share the engineering burden and create a means for objectively evaluating
published work. If you and I use a common development
platform, then you can send me your code and I can replicate
your experiments in my lab.
The core of the project is Player itself, which functions as
the OS for a sensor-actuator system, providing an abstraction
layer that decouples the user’s program from the details of
specific hardware (see Figure 5).
Player specifies a set of interfaces, each of which defines
the syntax and semantics for the allowable interactions with
a particular class of sensor or actuator. Common interfaces
include laser and ptz, which respectively provide access to
scanning laser range-finders and pan-tilt-zoom cameras. A
hardware-specific driver does the work of directly controlling
a device and mapping its capabilities onto the corresponding
interface. Just as a program that uses an OS’s standard mouse
interface will work with any mouse, a program that uses
Player’s standard laser interface will work with any laser
—be it a SICK or a Hokuyo range-finder.
Fig. 5.
processing, and (3) Reasoning algorithms.
One can think of the Player driver system as a graph
(see Figure 6), where nodes represent the drivers which
interact via well-defined interfaces (edges). Because it is
embodied, this graph is grounded in the robot’s (physical
or simulated) devices. That is, certain leaf drivers of the
graph are connected to sensors and actuators. Internal drivers
that implement algorithms (e.g., localization) are connected
only to other drivers. Because the interfaces are welldefined, drivers are separable in that one can be written
without knowledge of the internal workings of another. If
my algorithm requires data from a laser range-finder, then
the driver that implements my algorithm will take input
over a laser edge from another driver; I don’t care how
that data gets produced, just that it is standard laser data.
A wide variety of control systems can be constructed by
appropriately configuring and connecting drivers. The control
system is also accessible from the outside; an external
program (e.g., a client) can connect to any driver, via the
same standard interfaces. Thus the system is, to a limited
extent, reconfigurable at run-time.
An example of the Player ↔ client architecture
The 2D simulator Stage and the 3D simulator Gazebo (see
Figure 4) also use a set of drivers to map their simulated
devices onto the standard Player interfaces. Thus the interface that is presented to the user remains unchanged from
simulation to hardware, and back. For example, a program
that drives a simulated laser-equipped robot in Gazebo will
also drive a real laser-equipped robot, with no changes to the
control code. Programs are often developed and debugged
first in simulation, then transitioned to hardware for deployment.
In addition to providing access to (physical or simulated)
hardware, Player drivers can implement sophisticated algorithms that use other drivers as sources and sinks for data.
For example, the lasercspace driver reads range data from
a laser device and convolves that data with the shape of a
robot’s body to produce the configuration-space boundary.
That is, it shortens the range of each laser scan such that
the resultant scan delimits the obstacle-free portion of the
robot’s configuration space [4]. The lasercspace driver’s
output conforms to the laser interface, which makes it easy
to use. Other examples of algorithm drivers include adaptive
Monte Carlo localization [5]; laser-stabilized odometry [6];
and Vector Field Histogram navigation [7]. By incorporating
well-understood algorithms into our infrastructure, we eliminate the need for users to individually re-implement them.
Our architectural layer consists of (1) Low-level connections to the hardware platforms, (2) Sensor fusion and
Fig. 6.
Graph of Player drivers and their appropriate interfaces
IV. E XTENDING P/S/G
The following sections describe the extensions that we are
currently implementing into P/S/G as part of our ongoing
research. Some of them are already available in the CVS
repository on SourceForge, and the rest will be made available soon, together with the release of Player 3.0.
A. Ubiquitous Sensing Infrastructure
Extending P/S/G towards ubiquitous sensing includes a
development effort to provide drivers for sensors such as heterogeneous Wireless Sensor Networks, RFID technologies,
Inertial Measurement Units, and many more.
A series of new hardware platforms are going to be
supported with the upcoming release of Player 2.1. We have
added support for:
• Wireless Sensor Networks - with a wide variety of
different sensor nodes, ranging from the RCores and
Particles from TeCO/Particle Computers to the Mica2
and Mica2Dots from Crossbow or the Spine;
• RFID technologies - several readers such as the Inside
M/R300, the Skyetek M1 and the Skyetek M1-mini are
now supported;
• Inertial Measurement Units - supporting the XSens
MT9 as well as the XSens MTx, which provide driftfree 3D orientation and kinematic data.
Besides these, we implemented a number of virtual drivers
which take care of sensor calibration, data fusion, synchronization and logging. Furthermore, automatic feature
extraction plugins are now available (see [1] for more details)
and drivers for feature boosting as well as learning SVM
models are underway.
Fig. 7.
3D mapping, trajectory planning in 3D space, and finally the
safe execution of such trajectories.
1) 3D Mapping: In order to be able to perform robotic
manipulation in a complex environment, the system needs
a fine-grained 3D polygonal map, updated preferably as
often as possible. In the extended P/S/G such maps can be
acquired using a probabilistic algorithm. The algorithm gets
an unorganized 3D point cloud as input, and provides a 3D
polygonal description as output. Model acquisition based on
planar surfaces is particularly adequate for mapping indoor
environments. Figure 8 depicts pre-processed laser range data
(left) and a plane-based representation (right) of the kitchen
from Figures 9 and 2.
Wireless Sensor Networks and RFID technologies in Player
Besides driver development, the integration of ubiquitous
sensing and computing infrastructure yields interesting challenges, like the incorporation of new sensing infrastructure
during operation and the active querying of sensor networks
for specific information, e.g. like it is done in TinyDB [8].
B. Supporting robotic manipulation
Most middleware for robots provide little support for manipulation tasks. As we go for cognitive/service robots that
are more sophisticated and have to solve more complicated
tasks, manipulation is essential. Therefore a key direction in
extending P/S/G is to provide the adequate infrastructure.
Our robot platform (see section II-A) has been modified
with two Amtec Powercube arms with six degrees of freedom
each. They are very recent state-of-the-art manipulators and
support a variety of commands that make control very
flexible. Each joint has its own integrated controller that
supports position, velocity, and current commands. There
is also support for synchronized commands to all joints
of one manipulator, and synchronized sampling of position
data. In order to make use of these advanced capabilities,
we have created a P/S/G middleware infrastructure for the
manipulators. This middleware components provide tools
for acquiring and using 3D obstacle maps, and efficient
manipulator control. We discuss these issues by looking at
Fig. 8. Preliminary results obtained with the polymap driver (left: corrected
3D cloud point, right: polygonal interpretation. Each plane is visualized with
a different color)
In order to separate the 3D space into planes and later
on, polygons, we use a random consensus based algorithm,
which makes use of the distribution of points in space, and
clusters the points into inliers and outliers. Once a planar
region has been found, using a least-squares technique, the
inliers are projected onto it, and a Delaunay triangulation followed by an outline shape detection algorithm is performed.
One usage example is demonstrated in Figure 6. As shown
there, the laserptzcloud driver takes a laser 2D scan from
a laser interface and the pan-tilt angles of a PTZ unit
via a ptz interface, computes the corrected 3D coordinates
of each point in space, and outputs a 3d point cloud via
the pointcloud3d interface. The polymap driver uses the
resulted 3d point cloud, and, via a few configuration requests,
computes the best N planes that contain X (given) inliers, fits
the inliers to the planes, extracts the appropriate clusters and
tries to find polygonal outlines (see Figure 8).
2) 3D Trajectory planning: For reaching and manipulation tasks with the robotic arms, it is important to find
the best way to reach a desired position without colliding
with obstacles, therefore a trajectory planner is needed. Our
proposal is to use the Motion Strategy Library (MSL) [9],
that includes a variety of motion planners, including an
available RTT planner[10]. The MSL needs a geometrical
model of the environment in order to operate, model which
is provided by the 3D mapping driver described above.
3) Execution of the planned trajectories: Once we have
a suitable trajectory for the end effector, the next step is
finding the correct angles for each joint, so that it reaches the
desired points in space with the desired orientations (known
as the inverse kinematics problem). We have developed a
Player driver that has a limb interface for inputting the
desired position and pose, and controls an actarray interface connected to a robotic manipulator. This driver reads
the description from the arm expressed with the DenavitHartenberg convention, thus it can be used for any robot
manipulator that has a driver in P/S/G.
In order to interface with our Powercube manipulators,
we have developed a P/S/G driver that implements an actarray interface (which we extended accordingly). It receives
commands from the driver described above and controls the
arms of the robot through the CAN interface. Since the
manipulators can move with high velocities and could easily
collide with the environments, the driver also implements
safety features like a watchdog, that will stop the arm if the
Player server stops responding for a fraction of a second.
One of the goals of our project is to create models from
actions using information from the arms. On this context
it is very important to have precise data regarding the
state of all the joints in the arm, and the data for all
joints should be sampled synchronized and with a constant
frame rate. To solve this problem, the Powercube driver
uses broadcast commands on the CAN bus to instruct all
modules to record their positions at the same time, and the
data is then recovered serially from all of them. To have a
very constant frame rate, the driver includes code to order
the kernel scheduler to treat the Player server as a (soft)
real time process, and it also takes advantage of the recent
improvements to POSIX timers in the Linux kernel, called
the High-Resolution-Timers infrastructure. In this way, the
control loop of the arm runs with deviations of less than
10uS in the cycle time on modern computers without having
to run the arm controller in kernel-space.
Learning a model of the manipulator’s dynamics is very
useful if one wants to improve the arm’s movements. The
whole arm can be conceptualized as a kinematic chain made
up of rigid sections and joints between them. The kinematic
response of each joint depends on the torque applied by the
motor of the joint, and the positions, velocities and accelerations of all joints in the arm. All these make the feature
space for learning the model very large. To successfully learn
a model for each joint, we apply an algorithm that works
well in high-dimensional spaces: Locally weighted projection
regression (LWPR) [11].
C. Extensions for Cognitive Processing
We also extend P/S/G to support cognitive processing. It
is certainly an issue whether or not cognitive capabilities
should be incorporated into a middleware tool such as P/S/G.
The pros are that many perceptual and sensing tasks can
be drastically simplified using contextual information. And
also that often the behavior or action performance can only
be optimized if the cognitive processing is tightly coupled
with the low-level perception/action loop. One con is that
the provision of cognitive mechanisms could be viewed as
a methodological commitment that we might not want to
make at the middleware layer. We believe that the advantages
outweigh the disadvantages.
We can take advantage of cognitive capabilities in the
middleware layer to ease the interpretation of sensor data
for safety and collision avoidance. For example, when the
robotic arm is moving in what it thinks is free space, and it
senses a small force applied on it, it should stop immediately,
because it has just crashed against something. But when the
arm is picking up an object, a sensed force is expected and it
is part of the normal manipulation action. The threshold for
the emergency stop of the arm based on sensed forces can
be set accordingly to the context and the intended action.
In the opposite direction, in order to facilitate research
in cognitive areas, the middleware has to support certain
features. One example is applying learning for the arm
manipulator, which requires a very tight integration of the
low-level and high level control processes. The sensor data
in particular must be synchronized and sampled uniformly
and the middleware must support this. This has been taken
into account when designing the Player driver for our robot
manipulators, as explained in section IV-B.
Another example of the benefits of cognitive processing
embedded in the middleware is the use of specialized perception routines instead of general ones if the necessary context
conditions of the specialized methods are met. Consider a
robot navigating through an indoor environment. By using
the 3D polygonal mapping driver, it can verify easily the
presence of an uniform ground plane.
The different parts of the middleware can be combined to
create powerful systems. For example, if we want to pick up
a cereal box with the robotic arm we can use the 3D Mapping
driver to obtain a geometric model of the object and find out
its position, while using an RFID reader to detect its identity.
Knowing it, the grasp planner can search for a detailed stored
model, and select the best grasping method.
An important lesson that we have learned is that the
cognitive processing layer should not be something isolated
on top of the middleware. Instead, embedding it into the
the middleware enables and eases the development of more
powerful and flexible systems.
D. Usage examples
The usage of the above described extensions is pretty
straightforward, and requires little programming skills. All
the parameters of a driver, may it be a low-level hardware
driver or a higher level virtual one, are set via the Player
configuration file, and can also be modified later via client
configuration requests. The following example depicts the
usage of our automatic acceleration feature extraction driver,
for nodes in a Wireless Sensor Network.
driver (
... ↓ ...


name
”accelfeatures”
driver (






provides [”features:0” ”wsn:1”]
name ”accelfeatures”






requires [”wsn:0”]
provides [”features:1” ”wsn:2”]


 window size 16
 requires [”wsn:1”]








)
queue size 10000
overlapping 50
feature list [”wavelet coeff” ”ica”]
wavelet params [”daubechies” 20]
... ↓ ...









window size 16
queue size 10000
overlapping 50
feature list [”energy” ”rms” 11
”skewness” ”magnitude” 15 18]
)
(3) All changes of state in the doors of the cupboards. (4) All
tags detected by the RFID readers. (5) Movies from all
cameras recorded at 15 frames per second. (6) Data from
various other sensors. The data was recorded in log-files
with timestamps for each event. These research efforts profit
from using P/S/G, because of the comprehensive and reliable
sensor infrastructure that enables us to perform replicable and
long-time experiments with little set-up time.
In our ongoing research we use this data for first studies
in learning action models for table setting tasks. We will also
use the data to investigate whether a subset of the sensors
used are sufficient to recognize the activities and explore the
effect of particular combinations of them on the reliability,
accuracy and coverage.
In this case, the accelfeatures driver will spawn two different threads, using the output of the first as the input of the
second. Therefore, the first thread will receive acceleration
data via the wsn:0 interface, calculate wavelet coefficients
using Daubechies 20 and perform Independent Component
Analysis (ICA) in parallel, and finally pack the resulted
values in the wsn:1 interface. The second thread will take
the results from wsn:1 as input, and will compute standard
features such as energy, RMS, magnitude and so on and will
provide the results to the user via the wsn:2 and features:1
interfaces. The calculated features will be used later on to
learn SVM (Support Vector Machines) models (see [1] for
more information), and build a library of motion blueprints.
V. E XPERIMENTAL R ESULTS
We tested the ubiquitous sensing infrastructure in an
experiment that lasted more than ten hours. The complete
sensor network was controlled by P/S/G the whole time.
Our experiment consisted in having a number of people
instrumented with tiny inertial measurement units as well
as with a RFID-enabled glove, setting up the table, while
the system observes and gathers sensor data.
The environment was instrumented with a large number
of sensors. Seven video cameras were positioned so that all
possible angles are covered, and three small, non-intrusive,
laser sensors have been carefully placed in the environment
for tracking people movements. Two long-range RFID readers were placed in the cupboards, and several kitchen objects
have been equipped with RFID tags. Additionally, four more
smaller RFID readers together with eight capacitive sensors
were placed under a kitchen table for detecting where and
what type of objects were on the table. All sensor data was
synchronized and logged using several computing devices in
the network.
On each cupboard door, we placed magnetic sensors in
order to find out if the door is open or closed. Light and
temperature sensors together with wireless sensor network
nodes have been scattered throughout the AwareKitchen in
strategic places.
We performed 10 table-setting activities. The collected
data includes: (1) 75 measurements per second from the
inertial units. (2) 10 scans per second from the laser scanners.
Fig. 9.
A Gazebo simulation of the AwareKitchen
The manipulation control system described in Section IVB has also been tested in extensive learning experiments,
where we learned predictive forward models for movement
routines when carrying objects of different weights. Those
experiments required reliable recording of data and the action
control in the low level perception/action loop, as described
in section IV-B.3. We also learn action models that were used
to optimize sequences of movement actions that produced
both substantial as well statistically significant performance
improvements. This would not have been possible without
the extended P/S/G infrastructure.
Using our polygonal mapping algorithms, we were able
to reconstruct environments such as the kitchen, resulting in
detailed and accurate 3D maps (see Figure 8). We currently
focus on extending these maps towards 3D object maps, eg.
maps in which we can identify objects and regions of interest,
which the robot assistant can use for high-level planning.
Most of our maps are created using the small laser rangefinders placed on the end effectors of the robot.
Before attempting to use our robot assistant in the real
world, we go through several stages of fine-grained simulation of our plans and controllers. To perform an accurate
3D simulation, we make use of the Gazebo simulator, under
which we created an environment similar to the real one
(see Figure 9). By introducing error models in our sensory
data and employing probabilistic techniques in the simulator,
we can achieve similar results in the real world, without
changing our programming infrastructure or algorithms.
VI. R ELATED W ORK
Related work can be categorized on several dimensions.
On the middleware side, there are several other software
packages positioned closely to Player/Stage/Gazebo. Due
to space constraints, a comparison between them will not
be made here. Instead, we will give a list of publications
that cover a lot of pros and cons of each project from an
architectural point of view [12], [13], [2].
A very important aspect for a robotic middleware is that
it has to keep up with the rapid advancements in the field,
and thus be as flexible as possible. During the last 8 years,
Player/Stage has been consistently and continuously evolving. According to [3], over 50 different research laboratories
and institutions all around the world are currently involved in
the active development of P/S/G, and even more are using it.
Overall, we believe that one of the key features of a project
of this size should be its ease of maintenance (achieved in
P/S/G’s case through simplicity and transparency of the developer API), because the pool of researchers and developers
changes rapidly [12].
Research projects in the ubiquitous community have created fully instrumented sensor-equipped environments, one
of the leading initiatives in this field being MIT’s PlaceLab
[14]. There are other efforts underway, but they do not
cover complete environments (entire apartments), but rather
more limited scenarios. In comparison to the PlaceLab, we
comprise a more comprehensive sensor suite in our AwareKitchen, mostly because we are interested in transferring skills
from humans to robots through developmental learning, thus
we need more complete action models.
With respect to other related work regarding our P/S/G
extensions, we have referenced the appropriate articles in
our technical section.
At the best of our knowledge, the integration of such
advanced techniques in such a comprehensive way has not
been performed in other comparable initiatives.
VII. C ONCLUSION AND F UTURE WORK
In this paper we have described current extensions of
the P/S/G middleware for autonomous robot control. The
extensions cover three areas of robot control: (1) extensions
to integrate ubiquitous sensing infrastructure, (2) extensions
needed for supporting robotic manipulation, and (3) extensions allowing for cognitive processing. Taken together they
enable P/S/G to effectively support the development and
deployment of leading-edge service robots that can serve
as robot assistants with comprehensive manipulation capabilities acting in ubiquitous sensor-equipped environments.
We have also argued that cognitive capabilities should not
be realized strictly on top of the middleware layer but rather
they should be tightly integrated into it. Among other things,
the tight integration allows for much more powerful learning
mechanisms and for the use of contextual information to
simplify low-level sensor data processing. While substantial
parts of the extensions are still under investigation and
development and will become parts of future P/S/G versions,
the kernel providing much of the basic functionality has been
shown to reliably work in an extended experiment.
VIII. ACKNOWLEDGMENTS
This work has been jointly conducted by the Intelligent
Autonomous Systems group at the Technische Universität
München and the Artificial Intelligence Center at SRI, International. This work is supported in part by the National
Science Foundation under grant IIS-0638794.
We would like to thank Matthias Kranz for his numerous
contributions to the AwareKitchen project, and also our
students Lorenz Mösenlechner and Nico Blodow for their
excellent project work.
Special thanks go to the entire P/S/G community for
their continuous effort in developing and maintaining free
software tools for robot and sensor applications.
R EFERENCES
[1] M. Kranz, R. B. Rusu, A. Maldonado, M. Beetz, and A. Schmidt, “A
Player/Stage System for Context-Aware Intelligent Environments,” in
Proceedings of UbiSys’06, System Support for Ubiquitous Computing
Workshop, at the 8th Annual Conference on Ubiquitous Computing
(Ubicomp 2006), September 17-21 2006.
[2] B. Gerkey, R. T. Vaughan, and A. Howard, “The Player/Stage Project:
Tools for Multi-Robot and Distributed Sensor Systems,” in In Proceedings of the 11th International Conference on Advanced Robotics
(ICAR 2003), pages 317-323, June 2003.
[3] “Player/Stage/Gazebo: Free Software tools for robot and sensor applications,” http://playerstage.sourceforge.net.
[4] J.-C. Latombe, Robot Motion Planning. Norwell, Massachusetts:
Kluwer Academic Publishers, 1991.
[5] D. Fox, “KLD-sampling: Adaptive particle filters,” in Proc. of Advances in Neural Information Processing Systems (NIPS). MIT Press,
2001.
[6] F. Lu and E. Milios, “Robot pose estimation in unknown environments
by matching 2D range scans,” J. of Intelligent and Robotic Systems,
vol. 18, no. 3, pp. 249–275, May 1997.
[7] I. Ulrich and J. Borenstein, “VFH+: Reliable obstacle for fast mobile
robots,” in Proc. of the IEEE Intl. Conf. on Robotics and Automation
(ICRA), May 1998, pp. 1572–1577.
[8] S. R. Madden, J. M. Hellerstein, and W. Hong, “TinyDB: In-network
query processing in TinyOS,” http://telegraph.cs.berkeley.edu/tinydb,
September 2003.
[9] S. M. LaValle et al, “The Motion Strategy Library Main Page
(Online),” http://msl.cs.uiuc.edu/msl/.
[10] S. M. LaValle, Planning Algorithms. Cambridge, U.K.: Cambridge
University Press, 2006, also available at http://planning.cs.uiuc.edu/.
[11] S. Vijayakumar, A. D’Souza, and S. Schaal, “Incremental online
learning in high dimensions.” Neural Computation, vol. 17, no. 12,
pp. 2602–2634, 2005.
[12] T. H. Collett, B. A. MacDonald, and B. P. Gerkey, “Player 2.0: Toward
a Practical Robot Programming Framework,” in Proceedings of the
Australasian Conference on Robotics and Automation (ACRA 2005),
December 2005.
[13] R. T. Vaughan, B. Gerkey, and A. Howard, “On Device Abstractions For Portable, Reusable Robot Code,” in In Proceedings of
the IEEE/RSJ International Conference on Intelligent Robot Systems
(IROS 2003), pages 2121-2427, October 2003.
[14] S. S. Intille, K. Larson, J. S. Beaudin, J. Nawyn, E. M. Tapia, and
P. Kaushik, “A living laboratory for the design and evaluation of
ubiquitous computing interfaces,” 2005.