Download From Imitation Learning to Innovation in Designing - Neuron

Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
From Imitation Learning to Innovation
in Designing Behaviors for Humanoid
Robots
Mária VIRČÍKOVÁ, Peter SINČÁK
Dept. of Cybernetics and Artificial Intelligence, Faculty of Electrical Engineering and
Informatics, Technical University of Košice, Slovak Republic
{maria.vircikova, peter.sincak}@tuke.sk
Abstract — The proposed research deals with a usage of artificial intelligent techniques in
humanoid robotics. The objective is to design an interactive robotic behavior. The focus is on social
robotics and how to improve the interaction between human and robot. In the first part user
interactively cooperates with the System and this is the way how the robot acquires knowledge. Our
approach is based on real manipulation where the robot freely interacts with humans and objects,
and learns during continuous interaction with the environment.
In the second part of the system we use the Interactive Evolutionary Computation – a technique
which optimizes systems based on subjective human evaluation. So far, the algorithm is applied to a
system of evolution of pleasant motions or dances. It adapts to a user´s preferences and we think
adaptability can be considered a requisite feature of intelligence. A user can build a robot behavior
without any programming knowledge and then the evolutionary algorithm helps user to discover
new possibilities of motions.
Keywords — cognitive architecture, evolutionary computation, human—robot interaction,
interactive evolution, learning from demonstration, robot NAO
I.
INTRODUCTION
“Art begins in imitation
and ends in innovation.”
Mason Cooley (1927—2002)
The focus of robotic research continues to shift from industrial environments, in which
robots perform a repetitive task in a very controlled environment, to mobile service robots
operating in a wide variety of environments, often in human—habited ones. New applications
for robots make grow many problems which must be solved in order to increase their
autonomy. These problems are as said in [3], but are not limited to, navigation, localization,
behavior generation and human—machine interaction.
According to Brooks [1], the implicit dream of Artificial Intelligence (AI) has been to build
human level intelligence. Building a humanoid is the challenge par excellence for AI and
robotic workers. AI tries to make programs or computers do things that, when done by people,
are described as having indicated intelligence. The goal of AI has been characterized as both
the construction of useful intelligent systems and the understanding of human intelligence.
Since AI’s earliest days there have been thoughts of building truly intelligent autonomous
robots. In academic research circles, work in robotics has influenced work in Al and vice versa
[2]. Recent generations of humanoid robots (HRs) increasingly resemble human in shape and
capacities. HRs are robots that are at least loosely based on the appearance of the human body.
We work with humanoid robot Nao developed by Aldebaran Robotics. The robot (fig. 1) has a
human—like appearance and various sensors for interacting with humans.
In community of AI, by humanoid we mean not only a robot that has the physical appearance
of a human, but also has a fairly advanced artificial intelligence, allowing it to reasonably
approximate human behavior and interactions. New trends in research consider the robots
ability to interact with people safely and in natural way. The current leading topics in HRs are
so—called service and social robotics — where the target is getting robots closer to people and
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
338
Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
their social needs. There are many attempts to create robots capable of expressing emotions or
communicating in a natural language.
This paper starts by reviewing the basic features of the learning by imitation paradigm and
considers a new way how to map the teacher skills to a robotic agent during the interaction –
teaching process. We suggest an idea of learning behaviors one robot from another. The paper
continues with a discussion of the emergence of creativity – innovative behaviors – by
employing the Interactive Evolution in the development of the behaviors (which are in our case
motor skills of the robot).
II. LEARNING FROM DEMONSTRATION
A. Robots learning by imitating human
An enduring challenge in social robotics has been to improve the interaction between human
users and robots – it is still unnatural, slow and unsatisfactory for the human interlocutor.
Robots have different abilities, knowledge, and expectations than humans. They need to react
appropriately to human expectations and behavior. With this respect, scientific advances have
been made to date for applications in entertainment and service robotics that largely depend on
intuitive interaction.
Dautenhahn and Billard [8] called the type of systems which we are trying to develop social
robots: Embodied agents that are part of a heterogeneous group: a society of robots or humans.
They are able to recognize each other and engage in social interactions, they possess histories –
perceive and interpret the world in terms of their own experience – and they explicitly
communicate with and learn from each other.
It is a huge research field including gesture and natural language communication, perceiving
emotions and expressing their own artificial emotions, establishing social relationships,
exhibition of different personalities and characters, recognition of interaction partner and many
others. In this paper we present our work in learning motion tasks from observation.
Fig. 1 Humanoid robot Nao comes with 25 degrees of freedom for great mobility. The inertial sensor and
closed loop control provide great stability while moving and enables positioning within space. Its state-of-theart onboard actuators give Nao extreme precision in its movements. It supports multiple programming
environments and programming languages like URBI Script, C++, Python, .Net Available on Linux. It can be
programmed and controlled using Linux, Windows or Mac OS. The most used simulators for Nao are
Microsoft Robotics Studio, Gostai Urbi Studio, Pyro, Choreographe and Webots. The position in angles of the
degrees of freedom – servos or motors of the robot will represent the phenotype of the genetic algorithm The
robot is running on a Linux platform and can be programmed using a proprietary SDK called NaoQi. It is
based on a client-server architecture, where NaoQi itself acts as a server. The modules plug into NaoQi either
as a library or as a broker, with the latter communicating over IP with NaoQi. Running a module as a broker
thus has some IP-related overhead, but also allows the module to be run on a remote machine, which is very
useful for debugging. NaoQi itself comes with modules which we use.
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
339
Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
Fig. 2 Article [21] describes how Kinect works: It generates a three-dimensional (moving) image of the objects in its
field of view, and recognizes (moving) human beings among those objects.
Authors [9] formulated a statement: Imitation takes place when an agent learns a behavior
from observing the execution of that behavior by a teacher. They also enumerated the
advantaged a learning mechanism should give: adaptation, efficient communication,
compatibility with other learning mechanisms, efficient learning and thus, learning by imitation
seems to be one of the major components of general intelligent behavior. Imitation
encompasses a set of different competences such as recognizing other people's actions,
recognizing the goal of a particular action and the objects and/or subjects involved. This project
attempts to implement a similar set of abilities in a humanoid robot following a biologically
motivated perspective.
Learning tasks from demonstrations offers a convenient mechanism to customize and train a
robot by transferring task related knowledge from a user to a robot. This avoids the time—
consuming and complex process of manual programming. The way in which the user interacts
with a robot during a demonstration plays a vital role in terms of how effectively and accurately
the user is able to provide a demonstration. Teaching through demonstrations is a social
activity, one that requires bidirectional communication between a teacher and a student.
Approach "Learning these tasks from demonstrations according to [10] is a convenient
mechanism to customize and train a robot by transferring task related knowledge from a user
to a robot. This avoids the time—consuming and complex process of manual programming.
The way in which the user interacts with a robot during a demonstration plays a vital role in
terms of how effectively and accurately the user is able to provide a demonstration. Teaching
through demonstrations is a social activity, one that requires bidirectional communication
between a teacher and a student. The key question was how to map a teacher action to an
agent action. These agent actions in our experiments are the motor capabilities of the robots.
We mimic some user movements using Microsoft Kinect sensor device, a horizontal bar
connected to a small base with a motorized pivot and is designed to be positioned lengthwise
above or below the video display. The device features an ”RGB camera, depth sensor
and multi—array microphone running proprietary software” [11], which provide full—body
3D motion capture, facial recognition and voice recognition capabilities. Figure 3 illustrates
how Kinect sensor tracks a user. As described in [21],”Kinect relies on depth mapping to
determine an object’s location in a field. Much like sonar, the unit measures the time it takes for
light to bounce of the object and return to the source. It creates a 3D visualization of the object
this way. This method offers greater reliability than older techniques, which relied on color and
texture differences to determine the location and nature of an object. In that model, the results
can be thrown off by low contrast situations, like wearing a shirt the same color as the wall
behind you. Depth mapping also doesn’t rely on external light sources to measure anything.
Kinect will ostensibly work in a pitch black room, because it is generating its own “light
pings.” Once the device has data to process, the processor crunches the data to make it work. It
differentiates between faces, limbs, and other environmental aspects of the field, as well as
recognizes people and remembers who they are when they step back in the field.”
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
340
Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
Fig. 3 Skeleton tracking of users using Kinect sensor.
B. Robots learning from each other
At the present time we are trying to model different robot´s personalities via its interaction with
several human users and this way we personalize robot´s behaviors according to the interaction
of the specific human user. This way various models for multiple robot approach are
developed. Robots are interacting between each other and are learning new motion tasks in the
community of robots.
When robots will start to learn from each other they will be able to adapt much faster to new
environments and to perform even much complicated tasks. This approach means also that all
robots within the group could acquire tremendous quantities of useful knowledge and they
would not have to reinvent their abilities and skills from beginning again.
III. DISCOVERING NEW BEHAVIORS ACCORDING TO USER´S PREFERNCE VIA
INTERACTIVE EVOLUTION
A. Genetic Algorithm: The wisdom is in the population
Jorge Luis Borges in his The Library of Babel writes about a library which consists of
indeterminate and infinite number of books. Most of them contain nonsense and rational
relation is almost miraculous exception, but somewhere there are books that contain all the
knowledge of the universe.
A traveler in such a Library wanting to find something rational has a very difficult task, as
the most of the books are just a cluster of incomprehensible letters. How we could help this
traveler to find a “masterpiece” in the endless space of options”. We need a tool that would fly
in this space effectively and thanks to it we would find something inspiring. One of such a tool
could be an evolutionary algorithm.
An evolutionary algorithm is an optimization algorithm which uses some mechanisms
inspired by biological evolution coming out from Darwinian Theory. Evolutionary computation
is a biologically inspired general computational concept and includes Genetic Algorithms (GA),
Genetic Programming and Evolutionary Strategies. GA provides a very efficient search method
working on population, and has been applied to many problems of optimization and
classification.
General GA process is as follows:
1) Initialize the population of chromosomes.
2) Calculate the fitness for each individual in the population using fitness function.
3) Reproduce individuals to form a new population according to each individual’s fitness.
4) Perform crossover and mutation on the population.
Crossover selects genes from parent chromosomes and creates a new offspring.
After a crossover is performed, mutation takes place. This is to prevent falling all
solutions in population into a local optimum of solved problem. Mutation changes
randomly the new offspring.
5) Go to step (2) until some condition is satisfied.
The EC is a population-based searching algorithm and outputs multiple candidates, each
called an individual, as system outputs. The space of all feasible solutions (it means objects
among those the desired solution is) is called search space (also state space). Each point in the
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
341
Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
search space represents one feasible solution. Each feasible solution can be "marked" by its
value or fitness for the problem.
Fig. 4. The process of Genetic Algorithm and Interactive Genetic Algorithm where the fitness function
is replaced by the subjective user evaluation [5].
A fitness function is a particular type of objective function that prescribes the optimality of a
solution (that is, a chromosome) in a genetic algorithm so that that particular chromosome may
be ranked against all the other chromosomes. Optimal chromosomes, or at least chromosomes
which are more optimal, are allowed to breed and mix their datasets by any of several
techniques, producing a new generation that will (hopefully) be even better. We could say that
the fitness function is at the heart of an EC application because it determines which solutions
within a population are better at solving the concrete problem.
In reality, there are two types of target systems for system optimization: systems whose
optimization performances are numerically — or at least quantitatively — defined as evaluation
functions and systems whose optimization indexes are difficult to specify. How would you
mathematically express your feelings looking at some object of art? There are many
applications when the fitness function cannot be explicitly defined – not only in the art domain.
The second question is whether the evolutionary algorithms could be verily designated as
innovative.
B. Interactive Evolution: a Source of Inspiration
Evolution´s use of competition, variation, reproduction and selection provides an inspiration
for a method creating artificial system [15]. Fogel [12] has written that design variations
acquired by such systems are passed on to their offspring. We examine an interactive genetic
algorithm (IGA) that serves as a creativity tool for users.
Interactive Evolutionary Computation, introduced by Takagi [5], is an optimization method
that adopts EC among system optimization based on subjective human evaluation. It is simply
an EC technique whose fitness function is replaced by a human user or better said; it uses
human’s response as a fitness value. The user acts as critic of the robotic behaviors. We can say
that the IEC is a technology that embeds human preference, intuition, emotion,
psychological aspects, or a more general term, KANSEI, in the target system.
The goal of our design of pleasant movements is to create some ’nice motion performance’.
However, there is no general standard of ’beauty of motions’, and it is almost impossible to
organize fitness function. IGA might be a solution for this: it can reflect personal preference
because it percepts fitness directly from user instead of computing using some function. Users
evaluate robotic motions and give them marks according to their preferences. The process starts
from motion performances learned before (Section II: Learning from Demonstration). After
user evaluates the first generation of motions of the robots, the algorithm produces the next
generation. We use the tournament selection method: the winner of each tournament is the one
with the highest fitness value and this one is selected for crossover.
Goldberg [22] formulated the “innovation intuition” within these two relations:
1) Selection + Mutation = Continual Improvement
2) Selection + Recombination = Innovation
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
342
Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
In GA, the crossover & the mutation are sources of variation. Much research has been done
where IGAs are used to assist users in the development of creative ideas. (Evaluating
creativity is of course still mostly subjective.) Under creativity we mean producing something
new or unexpected and valuable. As reported by [20] a well known inhibitor of creative
thinking is design fixation, where the suggestion of a particular solution makes it difficult to
imagine other good solutions. Unlike many other computational search algorithms, EC methods
work with populations of “fairly good” solutions. Therefore, there is less danger that creativity
will be harmed by design fixation on one “best” solution.
IV. DESIGN OF ROBOTIC PLEASANT MOTIONS
Our goal was to discover new frontiers of humanoid behaviors for aesthetic pleasure,
entertainment or communication. In the field of developing robotic dances, numerous works are
presented in [1]. Researchers at the University of Tokyo have developed the learning —from —
observation training method, which enables a robot to acquire knowledge of what to do and
how to do it from observing human demonstrations. At the Kyoto University applied a method
called intermodality mapping to generate robot motion from various sounds and also to
generate sounds from motions using the back—propagation through—time algorithm. Other
approach from Tokyo University is using Chaos to trade synchronization and autonomy in a
dancing robot. Dancing Robot Partner built at the Tohoku University is well—known in the
domain of robotics. Their robot acts as a female partner and realizes ballroom dances in
coordination with a male human dancer by estimating his intention.
Although some of the mentioned systems are examples where interaction between human
and robot is the key factor of their success, they are exceptions in the world of robotic dance.
The majority of dance systems of robots are only pre—programmed motions and after a while
user find them boring.
The proposed system in this work interacts with human and the motion is evolving in
accordance with his evaluation of the seen behaviors learned in the first part of the
project. The system is a good aid for inexperienced observers to create their own “robotic
dance”. The result is that every person can adapt, in our case, the dance choreography, in
accordance to his own expectations and preferences.
V. CONCLUSION
In the first part of the project we engage people in face —to —face interaction with a
humanoid robot Nao. A naive human teacher can develop new motion tasks via interaction
process. In the second part we presented a multi—robot system capable of evolution of motions
using an interactive evolutionary computation. We believe that the interactive evolutionary
computation will make interactive robots more personal and will help user to creatively
discover robot´s behaviors during the interaction process that he does not imagine before.
Thus, there is a need to extend the search space in our model with an explicit expansion of
the robot behaviors solution space by adding more variables.
Our future dream is for Nao to acquire its own mental model of people. Currently, he does
not reason about the emotional state of others. Thus, we want to extract the information about
the own preferences of human during his evaluation of the behavior of the robot in the
interactive evolutionary algorithm and make this process autonomous.
According to Brooks and Breazeal [2]:
“The ability to recognize, understand, and reason about another’s emotional state is an
important ability for having a theory of mind about other people, which is considered by many
to be a requisite of adult—level social intelligence.”
ACKNOWLEDGMENT
This work is the result of the project implementation: Development of the Center of
Information and Communication Technologies for Knowledge Systems (ITMS project code:
26220120030) supported by the Research & Development Operational Program funded by the
ERDF.
REFERENCES
[1]
[2]
[3]
Aucouturier, J. J.—Ikeuchi, K. et. al.: Cheek To Chip: Dancing Robots and AI’s Future. In Intelligent Systems,
Ieee, 23(2): Pp. 74-84, 2009.
Brooks, R.A.—Breazeal, C. et. al.: The Cog Project: Building A Humanoid Robot. In: Computation For
Metaphors, Analogy and Agents. C. Nehaniv (Ed), Lecture Notes In Artificial Intelligence 1562. New York,
Springer,Pp. 52-87, 1999.
Garcia, E.—Jimenez, M., A. et. al.: The Evolution Of Robotics Research, From Industrial Robotics To Field And
Service Robotics. In: Ieee Robotics And Automation Magazine, March 2007.
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
343
Electrical Engineering and Informatics II
Proceeding of the Faculty of Electrical Engineering and Informatics of the Technical University of Košice
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
Gouaillier, D.—Hugel, V. et. al.: The Nao Humanoid: A Combination Of Performance And Affordability. In: Ieee
Transactions On Robotics, 2008.
Takagi, H.: Interactive Evolutionary Computation. Cooperation Of Computational Intelligence And Human
Kansei. In: Proceedings Of The Ieee, Vol. 89, Issue 9, Pp. 1275 - 1296, Issn: 0018-9219, September 2001.
Vircikova, M.: Artificial Intelligence In Humanoid Systems, Diploma Thesis, 2010.
Fong, T—Nourbakhsh, I.—Dautenhahn, K.: A Survey of Socially Interactive Robots: Concepts, Design and
Applications, technical report No. Cmu-Ri-Tr-02-29, Carnegie Mellon University 2002.
Dautenhahn, K—Billard, A.: Bringing up Robots or The Psychology of Socially Intelligent Robots: from Theory
to Implementation. In: Agents ´99, Proceedings of the third annual conference on Autonomous Agents, ISBN 158113-066-X, ACM New York, 1999.
Berthouze, L.—Bakker, P.—Kuniyoshi, Y.: Learning of Oculo-Motor Control: A prelude to Robotic Imitation.
In: Proceedings of the 1996 Lee/RSJ International Conference on Intelligent Robots and Systems, 1996.
Koenig, N.—Takayama, L.—Mataric, M:: 2010 Special Issue: Communication and Knowledge Sharing in
Human-Robot Interaction and Learning from Demonstration. In: Journal Neural Networks, Vol. 23, Issue 9,
Elsevier Science ltd. Oxford, 2010.
Totilo: Natal Recognizes 31 Body Parts, Uses Tenth Of Xbox 360, Computing Resources, Kotaku, Gawker
Media Retrieved November 25, 2010.
Fogel, D.: Evolutionary computation: Toward a new philosophy of machine intelligence, 2nd edition. IEEE Press,
2000.
Calinon, S.—Billard, A.: Incremental learning of gestures by imitation in a humanoid robot, in: Proceedings of
the 2nd ACM/IEEE International Conference on Human_Robot Interactions, HRI'07, 2007.
Ogino, M.—Toichi, H.—Yoshikawa, Y.—Asada, M.: Interaction rule learning with a human partner based on
an imitation faculty with a simple visuomotor mapping, in: The Social Mechanisms of Robot Programming by
Demonstration, Robotics and Autonomous Systems 54 (5), 2006.
Goldin, D. — Keil, D.: Interaction, Evolution and Intelligence, IEEE Press, 2001.
Ishiguro, T.—Ono, M. Imai—T. Kanda.: Development of an interactive humanoid robot “Robovie”— An
interdisciplinary approach, in Robotics Research, R. A. Jarvis and A. Zelinsky, Eds. New York: SpringerVerlag, 2003.
Banerjee, A.—Quiroz, J.C—Louis, S. J: A Model of Creative Design Using Collaborative Interactive Genetic
Algorithms, pages 397–416. Springer, 2008.
DiPaola, S.—Gabora, L.: Incorporating characteristics of human creativity into an evolutionary art algorithm.
In (D. Thierens, Ed.). Proceedings of the Genetic and Evolutionary Computing Conference (GECCO). July 7 —
11, 2007, University College London, England, pp. 2442-2449, 2007.
Boden, M.A.: What is creativity?, in Boden (ed.), Dimensions of creativity, The MIT Press, Cambridge,
Massachusetts, USA, 1994.
Buelow, P.: Advantages Of Evolutionary Computation Used For Exploration In The Creative Design Proces. In:
Journal of Integrated Design and Process Science, IOS Press, ISSN 1875-8959, Vol. 11, No 3, 2007, online date
2009.
Carbody, T.: Wired, Gadget Lab: How Motion Detection works in Xbox Kinect. November 3, 2010.
http://www.wired.com/gadgetlab/2010/11/tonights-release-xbox-kinect-how-does-it-work/all/
Goldberg, D.: The Design of Innovation: Lessons from Genetic algorithms for the Real World. In Technological
Forecasting And Social Change, Vol. 64 (2000), Pp. 7-12.
ISBN 978-80-553-0611-7 © 2011 FEI TUKE
344