Download Mirror Proposal 8-01 - USC - University of Southern California

Arbib: Mirror Neurons and Action Recognition
August 2001
Page 1
Mirror Neurons and the Neural Basis for Action Recognition:
A Computational Model
Principal Investigator: Michael A. Arbib
Project Summary
The monkey brain's mirror system for grasping matches the neural command for an action with the neural code for the
recognition of the same action executed by another. Analysis of this system has major implications for the understanding of
imitation and language in humans, but the focus of the present proposal is a new theory of the brain mechanisms underlying
action recognition in monkeys. We will explore this by the development of a detailed neural model of the mirror system and
related brain regions, with special emphasis on development and learning. The model will explain a large body of recent neurophysiological and neuroanatomical data from monkeys and lead to new experiments.
The modeling environment will include a primatoid hand-arm avatar for generating actions (to provide output in studies of
learning to grasp, and input stimuli for studies of action recognition); preprocessing routines for visual input; and tools for
modeling adaptive networks of biologically plausible neurons responsive to the constraints of neurophysiological and neuroanatomical data. Within this environment, we will provide and validate models for a range of specific aims: those related to
the development of grasping and the mirror system, paying particular attention to the hypothesis that visual feedback on hand
configuration during grasping underlies the mirror mechanism; those analyzing the function and plasticity of the mirror system
in guiding and recognizing single actions, with new hypotheses on population coding in the mirror system; and finally those
which extend recognition to compound actions in context, showing how this extension may bridge the gap from action recognition to understanding.
Our modeling will be enriched by collaboration with experimentalists working on monkey neurophysiology and neuroanatomy at the University of Parma. To facilitate the exchange and comparison of experimental data and modeling results, we
will further develop two of our neuroinformatics tools, NeuroBench for analysis of neurophysiological data, and the NeuroHomology Database for analysis of neuroanatomical data.
1. Introduction
1.1
The Interest of the Mirror System
As Vilayanur Ramachandran (2000) has commented:
[I'm] fascinated by the rostral part of the ventral premotor area. Giacomo Rizzolatti at the University of Parma has elegantly explored the properties of neurons in this part of the brain  the so-called "mirror" neurons (which) will fire when a test monkey
performs a single, highly specific action with its hand … [and] in response to [similar actions performed by others]. … With
knowledge of these neurons, you have the basis for understanding a host of very enigmatic aspects of the human mind: imitation
learning, intentionality, "mind reading," … Rizzolatti and Michael Arbib … suggest that mirror neurons may also be involved in
miming lip and tongue movements, … [and] present the crucial missing link between vision and language.
More specifically, mirror neurons in area F5 of the monkey, one of the motor areas forming the frontal agranular cortex,
become active both when the monkey makes a particular action (like grasping an object or holding it) and when it observes
another individual (monkey or human) making a similar action. The vast majority of F5 mirror neurons shows a marked similarity between the action effective when observed and the action effective when executed. This congruence is sometimes extremely strict, but broadly congruent neurons are of particular interest because they generalize the goal of the observed action
across many instances of it. Typically, mirror neurons do not respond to the sight of a hand mimicking an action in the absence of a target object nor do they respond to the observation of an object alone, even when it is of interest to the monkey
(Gallese et al. 1996; Rizzolatti et al.1996).
Ramachandran's account typifies the high level of current interest in the mirror system. Indeed, PET experiments in humans
showed that both observation and execution of hand actions activate left Broca's area (Rizzolatti et al., 1996), while fMRI
(Iacoboni et al. 1999) and MEG (Nishitani & Hari 2000) recordings showed the involvement of right Broca’s area in imitation. Broca's area is homologous to area F5 of monkey premotor cortex (Rizzolatti and Arbib 1998). Analysis of the human
mirror system  and especially its role in imitation and language  is a vital topic for our future research. However, the research proposed here will focus on neural mechanisms for the mirror system and action recognition in the monkey, developing
a detailed neural model based on monkey neurophysiology and related data. We will use the model to investigate a complete
system from sensing to acting by coupling it to a primatoid “avatar”, an extensible biomechanical simulation of the arm and
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 2
hand of a primate. The avatar will have components adjustable to match the dimensions of monkey and human since we must
analyze, among other things, how a monkey can recognize actions whether those of a monkey or a human.
We next outline the specific aims of the proposed research, the collaborations which will provide data to ground and test
our models, and the responsibilities of the personnel for whom funding is requested. We then review key data from neuroanatomy and neurophysiology relevant to the mirror system in monkey and provide a sample of our own prior modeling of the
monkey nervous system. The proposal concludes with a detailed Research Plan for modeling and validation for each of the
specific aims in turn: those related to the development of grasping and the mirror system; those analyzing the function and
plasticity of the mirror system in guiding and recognizing single actions; and finally those which extend recognition to compound actions in context, showing how this extension may bridge the gap from action recognition to understanding.
1.2
Specific Aims
The present proposal focuses on modeling the neural mechanisms of action recognition in monkeys, leaving modeling related to brain imaging studies of action recognition and imitation in humans to other proposals. This work builds upon our longstanding and ongoing collaboration with Rizzolatti (Jeannerod et al., 1995; Grafton et al., 1996, 1997; Arbib and Rizzolatti,
1997; Rizzolatti and Arbib, 1998) to study the mirror neuron system in monkeys and humans. Our collaboration has been
funded in part by the Human Frontier Science Program (HFSP), which also funded recent collaboration with the UCLA group
of Woods and Iacoboni (Modeling: Arbib et al., 2000; fMRI: Iacoboni et al., 1999). Since HFSP funding will expire in 2001,
it is timely to seek NSF funding to extend this research effort. We will develop the MNS2 model for the monkey mirror system (Figure 1; see the section "Grasping and the Mirror System in Monkey" for a review of relevant data, including a description of each of the brain regions referred to here) to elucidate how visual recognition of actions rests on neural structures
common to all primates, including neurons in STS coding for the tracking of gaze direction and body motions, neurons in the
parietal lobe which code hand shapes for different types of grasps; areas involved in movement composition (we focus on
SMA and basal ganglia); and the mirror neuron cells which respond to both observation and execution of actions. Our five
Specific Aims are grouped under 2 broad headings, "Development of the Mirror System", and "Recognition of Novel and
Compound Actions and their Context".
Figure 1. Overview of the
proposed MNS2 mirror neuron model with brain regions
assigned. Current modeling
has assessed overall functionality to predict physiological
responses of the F5 mirror
neurons. The proposed modeling will incorporate and
explain a wide range of detailed anatomical and physiological data, and will offer
new insights into the development and functionality of
the mirror system.
Development of the Mirror System:
In this proposal, we seek to explain the circuitry that makes mirror neuron activity possible, and then explore a number of
hypotheses on the role of mirror neurons in the brain mechanisms for a range of adaptive behaviors related to action recognition. We are concerned with three basic types of F5 neurons: motor neurons, canonical neurons, and mirror neurons. Nonvisual motor neurons comprise about 80% of F5 neurons. The other two classes comprise visuomotor neurons. They have
motor properties indistinguishable form those of motor neurons, but, in addition, they respond to specific type of visual stimuli – canonical neurons to object observation; mirror neurons to action observation. All F5 neurons are motor neurons. Some of
them receive connection from parietal region AIP, other from parietal region PF, others have no connections with the parietal
lobe. A major goal of our modeling will be to show how, during development, motor neurons with parietal connections from
AIP and PF will acquire visual properties and become respectively canonical or mirror neurons:
Development of Grasp Specificity in F5 Motor and Canonical Neurons: This project explores the development of the neurons which form the action generation circuitry which is an inseparable partner of the action recognition circuitry of the mirror
system. We will demonstrate how somatosensory feedback can play a crucial role in defining the population of F5 motor neurons, and how input from the parietal region AIP shapes up the F5 canonical subpopulation and is shaped up in turn, as the
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 3
developing. F5 canonical neurons select via re-afferent connections visual neurons describing a variety of surfaces. Only those
selected become AIP neurons that code affordances.
Visual Feedback for Grasping: A Possible Precursor of the Mirror Property: We offer a new hypothesis for the generalization from the visual description of action made by the acting individual to that made by other individuals. We propose to
demonstrate how neurons which develop to provide feedback for self-generated goal-directed grasping movements, using the
association between F5 motor activity and the visual stimuli resulting from this activity, will extract "hand configuration" data
concerning the relation of the moving hand to an object that will readily generalize to the movements of others' hands. The
model will involve a self-organization process which exploits bidirectional connectivity across F5, PF and STS areas to illuminate the developing role of STS and PF in the functioning of the mirror neuron system
Recognition of Novel and Compound Actions and their Context:
The modeling of development defined above emphasizes how the infant monkey builds a basic motor repertoire of reachand-grasp actions and how these come to be integrated with a set of visual processes that, we hypothesize, develops first to
provide feedback for the monkey's own actions and then serves to recognize hand-object relations in other monkeys which
signal similar actions. Here, each grasp is seen as related to the affordance of a target object. Our task in the present section is
to propose models which place the mirror system in a broader perspective. As we have noted, our analysis of the monkey mirror system is to be seen as grounding efforts to understand the brain mechanisms underlying imitation in humans. For imitation, the key issue is the recognition of the structure of novel actions, extending the prior repertoire of actions. For the monkey
studies proposed here, we focus on three related issues: (i) How does a variant of a known action come to be recognized? (ii)
How can a novel action be recognized a compound of (variants of) known actions. (iii) How are actions "understood"? We
argue that understanding will in general involve more than recognition of the action (movement + goal) in isolation, but will
also involve recognition of the context in which the action occurs and expectations as to its consequences.
The Pliers Experiment: Extending the Visual Vocabulary: The mirror neurons of the monkey will not fire when a monkey
first sees the experimenter grasp a raisin with a pair of pliers, but certain mirror neurons will come to fire after the monkey has
been repeatedly exposed to this stimulus. We will model how the system may recognize such novel stimuli, thus extending its
"input vocabulary" beyond a prior set of hand configurations. This will lead us to adopt a more subtle approach to the visual
input than used in our earlier work, and will involve modeling processes of learning which "work back in time" so that recognition that a raisin is being picked up can draw attention to the way in which earlier movement of the novel gripper can be
predictive of the grasp. We will also explore the hypothesis that it is a set of mirror neurons that provides a nuanced representation of an action, rather than the broadly tuned response of a single neuron. This modeling of population coding will be
strongly linked with the increasing push of the Parma group to use multi-electrode recording.
The present proposal contains three kinds of model: those strongly driven by available data; those which will be developed
in tandem with empirical efforts now planned by our collaborators in Parma; and those which offer conceptual analyses which
will set the stage for design of experiments several years in the future. The last two Specific Aims belong more to the third
class, but will nonetheless be grounded in empirical data.
Recognition of Compounds of Known Movements: We will extend our analysis of population coding of single actions to
model the learning and recognition of compounds. We will first extend our earlier work on the interaction of basal ganglia
(BG) and supplementary motor area (SMA) on the generation of sequences of movements to analyze their linkage with the
mirror system for recognition of sequences of movements; we will then generalize this approach to handle the recognition of
novel actions that are formed as a temporally coordinated superposition, rather than a sequence, of known actions.
From Action Recognition to Understanding: Context and Expectation: Finally, we will analyze "understanding" within the
framework created by all the earlier work. Our key idea here is that understanding is not simply the recognition of an action in
isolation, but must involve some notion of "meaning", e.g., the context in which the action is appropriate or the expectations
that such a behavior evokes. This relates to preliminary findings from Parma which will serve as the basis for further collaboration.
Precision
grasp
Power
grasp
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 4
Figure 2. Left: Data from the Rizzolatti group take the form of records of a cell's firing across a number of trials; a histogram
summing such records provides the view of the "expected" behavior that is used to challenge our modeling. In this example, the
experimenter grasps a piece of food with his hand, then moves it toward the monkey who, at the end of the trial, grasps it. The
neuron discharges during observation of the experimenter's grasp, ceases to fire when the food is given to the monkey and discharges again when the monkey grasps it. The rasters are aligned with the moment when the food is first grasped (vertical line).
Each small vertical line in the rasters corresponds to a spike. Right: An example of the output from 2 mirror neurons simulated
in the MNS1 model described below. The curves are to approximate or predict the histogram data for physiologically identifiable neurons. This simulation demonstrates resolution by the mirror neurons between a power and precision grasp when the observed avatar makes a precision grasp, grasping a long thin object by its ends. In the initial portion of the trajectory, the hand
preshape is mistaken for a power grasp, but the simulated mirror system corrects this misclassification as more of the trajectory
is taken into account. This is an example of a novel prediction from our modeling which is readily amenable to testing in the
Rizzolatti laboratory.
1.3
Validation: Models, Collaborations, and Datasets
The models proposed here will yield predictions and explanations addressing data from neurophysiological correlates of
monkey action recognition and will be linked to the work of the Rizzolatti group. Throughout the research, empirical predictions and tests will accompany our modeling. In many cases, new experiments will support our modeling; in other cases, the
data will lead us to revise the models, leading on to further predictions as we extend the scope of the models. The present section summarizes the datasets to be gathered by our collaborators (in addition we will, of course, continue to address many data
culled from the literature) and also outlines a collaborative effort on biomechanical simulation. Support letters are appended
to this proposal.
Neurophysiology and Behavior
Giacomo Rizzolatti of Parma has focused on the neurophysiology of the premotor cortex of monkeys during visually guided
reaching (Figure 2 Left); this has been broadened to include parietal inputs to the mirror system. In addition to collaboration
on a variety of papers (with the proposed modeling of the ontogeny of the grasping and mirror system a current target for future joint papers), our modeling will build upon extant data to make predictions for new experiments (Figure 2 Right). An
important issue (discussed further below) is that the data published in papers do not always yield sufficient detail for our
modeling. We have thus developed new software, NeuroBench, for viewing and analysis of neurophysiological records. The
Parma group recently released recordings from 37 mirror neurons to us and these have been entered into NeuroBench. We
find these preliminary efforts promising, and thus propose to develop NeuroBench into an advanced knowledge discovery
tool and incorporate many more data into the database. We will supply interface routines to enable the modelers (at USC) to
transfer the results of a simulation run directly to NeuroBench with annotations, date, and author information, with similar
tools to enable neurophysiologists to transfer their recordings with annotations to NeuroBench. NeuroBench will not only
serve as a database and visualization tool, but will also perform comparative and data mining computations. Since the Parma
group is now planning multi-electrode experiments, we propose to develop routines for temporal analysis including tools to
compare simulation-generated data with multi-electrode data. The result will provide the Parma neurophysiologists with new
tools for data analysis, as well as stimulating more detailed modeling by our group. The results of data analysis and modeling
will feed back into design of new experiments, thus advancing the theory-experiment cycle.
Neuroanatomy
Also in Parma, Massimo Matelli and Giuseppe Luppino have focused on neuroanatomy of the monkey, with new findings
on the relation between premotor cortex, parietal cortex and other regions. These data have constrained our models as we
begin to make functional sense of the connections they have revealed. They will provide further data on connections of the
monkey brain to constrain our modeling in the coming years, as well as engaging in further analysis of homologies between
the human and monkey brains relevant to the broader research program of which the present effort is part. The analysis of
anatomical data will benefit from our work on the NeuroHomology Database (NHDB; Bota, 2001; Bota & Arbib, 2001), an
online knowledge management system that handles information about brain structures, neuroanatomical connections and similarities between brain structures. Currently, NHDB comes in two online versions, NHDB-I and NHDB-II (Bota, 2001a,b). It
contains modules for Brain Structures, Connections (users can search connection information and also evaluate the confidence
level for a given neuroanatomical tract between two brain structures) and Homologies (our homology inference engine uses
eight criteria to evaluate the degree of homology between two brain structures). To date, the Homology module of NHDB-I
contains more than 50 similarities between different brain structures from rodents, macaques and humans. Clearly, the focus
on NHDB for the present proposal will be on systematizing macaque data from Matelli and Luppino with data culled from the
literature. The analysis of macaque-human homologies will be a target for other research.
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 5
Simulation of Biomechanics
Figure 3: The reach/grasp simulator we have already developed takes the representation of the
shape and position of a (virtual) 3D object and the initial position of the (virtual) arm and hand
and computes a trajectory that ends with simulated grasping of the object. The system, programmed in Java, simulates a 19 degree of freedom (DOF) arm and hand that can be controlled
through a user interface or automatically (e.g. for learning and testing). The 19 DOFs are 3 at
the shoulder, 1 for elbow flexion/extension, 3 for wrist rotation, 2 for each finger joint with 2
more for the thumb. The system provides routines to perform realistic grasps. In "inverse model" mode, it solves the inverse kinematics problem by simulated gradient descent with noise
added to the gradient
Finally, we mention a collaboration focused on simulation methodology rather than empirical data. Stefan Schaal (Computer Science, USC, and Computational Learning, Kawato Dynamic Brain Project) will share his expertise on developing learning algorithms for both robot control and human motor control (e.g., Schaal 1997). He will make available his simulation
software package for use in extending our present reach-and-grasp simulator (Figure 3) to include estimates of contact forces
including slip. We expect this collaboration to also yield fruitful interactions with his own NSF-funded research on imitation
and rehabilitation for human patients with motor disorders. Schaal's Simulation-Laboratory (SL) software package
(http://www-slab.usc.edu/publications/schaal-TRSL) allows rapid development of arbitrary rigid body dynamics systems with
complete control and data visualization tools. For studies of learning to grasp and development of the mirror system, we will
interface the arm-hand model with our neural models of brain regions. The simulator will also simulate contact force with
external objects that will be essential for our work on learning to grasp, where we want to use the slippage of the object in the
hand as a negative reinforcement learning signal to the learning system. Future development of the simulator will also allow
us to include muscle-based motor control, which we envision to be interesting for our project at an advanced stage. The simulation software is available to us at no expense. The dimensions of the arm and hand will be adjustable to match the dimensions of monkey and human to explore the monkey's recognition of human as well as monkey actions.
Integration of Databases and Modeling Tools
In addition to the above, we will link NeuroBench and NHDB with our Brain Models on the Web database (BMW; Arbib
and Bischoff-Grethe, 2001). BMW currently contains a set of models developed using our NSL Neural Simulation Language,
and will be extended to include biomechanical simulations of the kind just described. An important feature of NSL 3.0 is the
development of a Schematic Capture System (SCS) to allow hierarchical viewing of model structures (Arbib et al., 2001). We
will extend SCS to provide a hierarchical view of biological data as well, enabling the user to compare the structures of the
modules and connections in our models with the neurobiological information from NHDB and NeuroBench. We can thus ensure strong integration of empirical data into the design and testing of our models.
1.4
Personnel
The USC investigators will work closely with our collaborators on the analysis of relevant data to both constrain and validate our modeling. Funding is requested only for the modeling effort, plus travel funds to allow members of the USC team to
visit the Parma group and vice versa. Michael Arbib has long experience in computational neuroscience, especially in modeling neural mechanisms underlying visually guided behavior in close collaboration with experimentalists. He will continue his
work on detailed neural modeling of the monkey nervous system while managing the collaboration with the Parma group. To
a first approximation, the responsibilities that the 2 RAs will share with Arbib are to be divided as follows:
RA 1 (currently Erhan Oztop) will be responsible for developing NeuroBench as a repository of neurophysiological data
from Parma and elsewhere; for developing the primatoid hand-arm avatar; for the Development of the Mirror System component of the modeling; and for analyzing data and developing predictions related to population coding in the mirror system.
RA2 (Salvador Marmol) will be responsible for developing NHDB as a repository of neuroanatomical data from Parma and
elsewhere; for linking BMW, NeuroBench and NHDB to facilitate the use of empirical data in the grounding and testing of
models; and for the Compound Actions and their Context component of the modeling.
Arbib: Mirror Neurons and Action Recognition
2
August 2001
Page 6
Background
Figure 4. Monkey brain areas relevant to the
proposed modeling: The brain regions involved
in the mirror neuron system constitute a large
portion of the brain  parietal cortex, premotor
and motor cortex, the superior temporal sulcus
and prefrontal cortex. In the macaque brain, posterior parietal cortex and the cortex of caudal superior temporal sulcus (STS) have been subdivided into numerous areas mainly involved in spatial
analysis of the visual environment and in the control of spatially oriented behavior (Maioli et al.
1998). F1: motor cortex; F3: SMA-proper; F4:
proximal ventral premotor area; F5: distal ventral
premotor area; F6: pre-SMA; LIP: lateral intraparietal sulcus; MIP & VIP: medial and ventral intraparietal sulcus; AIP: anterior intraparietal sulcus; 7b: parietal area PF; 7a: parietal area PG.
2.1
Grasping and the Mirror System in Monkey
The rostral (F5) and caudal (F4) inferior premotor cortex of monkey (Figure 4) are involved in reaching and grasping
movements, while F1 is primary motor cortex. Neurons in F4 appear to be primarily involved in the control of proximal
movements (Gentilucci et al., 1988), whereas the neurons of F5 are involved in distal control (Rizzolatti, et al., 1988). Neurons located in area F5 of the monkey discharge during specific goal-directed hand and/or mouth movements. Our interest
centers on a subclass of F5 neurons that become active not only during the monkey's own movements, but also when the monkey observes the experimenter or another monkey make an action similar to the one that, when actively performed, triggers
the neuron. Neurons endowed with these properties are referred to as mirror neurons (di Pellegrino et al. 1992; Gallese et al.,
1996). In monkeys trained to grasp objects requiring different types of grip, about half the neurons in the anterior intraparietal
sulcus (AIP) related to hand movements fired almost exclusively during one type of grip, with precision grip being the most
represented grip type (Murata et al., 1997; Sakata & Kusunoki, 1992; Taira et al., 1990). In addition, this region has very significant recurrent cortico-cortical projections with area F5 (Matelli 1994, Sakata, 1997a). Some AIP cells demonstrate specificity toward the size of the object to be grasped, while showing a certain degree of independence from the type of object;
other cells demonstrate independence from the size of the object. The fact that there are cells that are both size-selective and
size-independent indicates that within a population of cells that code for a particular object, a sub-population of these cells are
responsible for capturing size of the object. Finally, a small number of cells shows modulation based upon the object's position and/or orientation in space.
Based on cytoarchitectonic and connectional criteria, Brodmann’s area 7 (inferior parietal lobule) includes areas 7a, 7b/PF
and 7ip. Brodmann's area 7 reaches its highest development in primates (Cavada and Goldman-Rakic, 1989). Damage to this
area can cause impairments in spatial perception, neglect of sensory stimuli contralateral to damage, and defects in visually
guided reaching and oculomotor control (Stein 1991). Complementing AIP, PF receives an important input from the cortex of
the superior temporal sulcus (STS) and sends its output to F5. Many neurons in PF respond to the observation of biological
actions and some of them show mirror properties. This area appears to represent the link between STS, where biological actions are heavily represented, and the F5 mirror system, where an observed biological action is matched to the internal motor
representation of the same action. As noted earlier, all F5 neurons are motor neurons. Some of them receive connections from
AIP, others from parietal region PF, while some have no parietal input. Motor neurons with parietal connections from AIP
and PF will acquire visual properties and become canonical or mirror neurons, respectively.
Perrett et al. (1990a,b) report neurons in the superior temporal sulcus (STS) responsive to goal directed hand motion; while
Barnes and Pandya (1992) argue that neurons of multimodal areas of the STS could be concerned with analyzing the position
of the organism relative to the environment. We posit that STS codes, in addition to biological motion, some physical aspect
of it such as velocity and direction. While there is evidence for direction coding, there are no studies that we are aware of
where the effect of stimulus speed was analyzed in STS. Note however that MT and MST project to STS, thus suggesting that
some of the STS neurons may code some parametric aspects of hand movements. We assume also that STS can code the distance between hand and object. There is evidence that a specific set of neurons discharge selectively during the observation of
the hand approaching and coming in contact with the object. This suggests that some distance aspects may be encoded in STS
neurons. Again no firm evidence on this point is available at the moment.
We briefly introduce key data on other regions of the brain relevant to the design and validation of the proposed model:
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 7
Sakata et al. (1997a,b) discovered binocular visual neurons in the lateral bank of the caudal intraparietal sulcus (cIPS) and
the neighboring area V3a. These neurons respond preferentially to an object in a particular orientation in space. They also
found neurons selective for a particular axis and orientation of surfaces. Intraparietal regions MIP/LIP/VIP encode the space
around the animal (Colby and Goldberg 1999). Although it is hard to assign a single type of encoding for these regions, a
rough separation is that VIP is concerned with ultra-near space (less than 5cm from the face), MIP with stimuli within reaching distance while LIP represents the space that we explore best with our eyes. Area PG of parietal cortex plays an important
role in representing space (e.g., Stein 1991; Siegel and Read 1997; Andersen et al. 1999). The recent findings from Parma on
mirror-like neurons in area PF of parietal cortex (Fogassi et al., 1998) and the connection of this area with the mirror neuron
region of area F5 (Matelli 1994) indicates an intimate relation between PF and F5 mirror neurons.
The supplementary motor area (SMA) is formed by two cytoarchitectonically distinct areas, SMA-proper (F3) and preSMA (F6) (Luppino et al. 1991). F3 contains a complete somatotopic representation of body movements (Luppino et al.
1991). Most F3 neurons discharge in association with active movements whereas area F6 requires more complex movements
to be activated (Matelli et al., 1993) in the human. F3 is connected strongly with F1 and F4. The F5 connection to F6 is rather
modest (Luppino et.al.,1993). F6 on the other hand has strong connections to F5 and a strong input originates from prefrontal
cortex (area 46). F6, unlike F3, is not connected to F1. The data suggest that F6 is responsible for higher-level motor functions, whereas F3 is more closely related to execution (Luppino et al., 1993).
In non-human primates, the ventrolateral prefrontal cortex (PFC) has a role in response inhibition, whereas the dorsolateral
PFC is implicated in spatial processes, working memory and sequencing of behavior (Pandya and Yeterian 1998; Barone and
Joseph, 1989; Jenkins et al. 1994). In monkey the dorsolateral prefrontal convexity above the principal sulcus is strongly connected with area PG, whereas the ventrolateral convexity is mainly connected with PF and 7ip (Cavada and Goldman-Rakic,
1989). In addition, both the dorsal and ventral prefrontal areas receive input from STS and SMA. PFC projects to pre-SMA.
SMA-proper contains neurons representing an overall task sequence and the subsequences within the task (Tanji and Shima
1994; Tanji et al., 1996). Since pre-SMA is responsible for preparing a movement sequence, particularly if visually guided,
PFC may be a source for the sequences that pre-SMA “knows.” We see pre-SMA and the basal ganglia as together crucial for
action-sequence parsing.
2.2
Prior Modeling of the Monkey Nervous System
Our group has a long history of models for visually directed grasping and grasp recognition (e.g., Arbib 1981; Iberall and
Arbib, 1990; Hoff and Arbib, 1992, 1993; and Jeannerod et al., 1995). Here, after a review of our modeling methodology, we
present a brief summary of recent models of direct relevance to this proposal: the FARS model of monkey grasping, our first
Mirror Neuron System model, and our models of the role of basal ganglia and SMA in control of compound behaviors.
Modeling Methodology
We seek where possible to provide a detailed structural description of the biological neural works of brain regions relevant
to our research. However, in some cases we "schematize" a brain region, such as the way cIPS can recognize surface orientation from visual input. This frees resources for detailed neural modeling of those regions most central to action recognition.
This does not imply that we think that neural details of regions such as cIPS are irrelevant. Rather, their analysis is a topic
whose scope requires separate funding. In any case, most of the regions here are modeled as neural networks, whether "connectionist" or "neurophysiological". Connectionist modeling replaces the schema by an adaptive artificial neural network,
using "training" rather than explicit programming either to achieve context for other more detailed analyses or to ensure that
the overall schema network meets behavioral criteria. To the extent that "training" yields output neurons whose response is
similar to that of neurophysiologically observed neurons, the network may yield novel predictions concerning the behavior of
real neurons in new conditions exercised in the modeling (see Figure 2).
Indeed, our main concern is with models which are grounded in, and can be tested by, neurophysiological data. Here, a survey of the neurophysiological and neuroanatomical data on a brain region calls attention to its basic cell types. As modelers
we then create one array of cells for each such type. The data tell us what the activity of the cells should be in a variety of
situations, but in many cases experimenters do not know the action of, e.g., their synapses in any quantitative detail. Thus, the
modeler has to make a number of hypotheses about weights, time constants and so on, to get the model to run. The modeler
may even have to postulate cell types that experimenters have not yet looked for, and show by simulation that the resulting
network will indeed perform in the observed way when known experiments are simulated, in which case it must match the
external behavior and, for those populations that were based on cell populations with measured physiological responses, it
must match those responses at some level of detail. What raises the ante is that the modeler's hypotheses suggest new experiments on neural dynamics and connectivity, and the model can be used to simulate experiments that have never been conducted with real nervous systems.
While much work on artificial neural networks focuses on networks of simple discrete-time neurons whose connections
obey various learning rules, most work in brain theory now uses continuous-time models that represent either the variation in
average firing rate of each neuron or the time course of membrane potentials. The study of a variety of connected "compartments" of membrane in dendrite, soma, and axon can help us understand the detailed properties of individual neurons (see,
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 8
e.g., Bower and Beeman, 1998; Hines & Carnevale, 1997; Hodgkin and Huxley, 1952), but much can still be learned about
the large-scale properties of interconnected regions of the brain by modeling each region as a set of subpopulations ("arrays")
using the leaky integrator model for each neuron. In this case the internal state of the neuron is described by a single variable,
the membrane potential m(t) at the spike initiation zone. The time evolution of m(t) is given by the differential equation
dm(t)
= - m(t) + i wi Xi(t) + h, with resting level h, time constant , Xi(t) the firing rate at the ith input, and wi the corredt
sponding synaptic weight. This model does not compute spikes on an individual basis – firing when the membrane potential
reaches threshold – but rather defines the firing rate as a continuously varying measure of the cell's activity, approximated by
a simple, sigmoid function of the membrane potential, M(t) = (m(t)). Such models can then be tested by careful comparison
of histograms of firing of a cell across repeated trials against the average firing rate computed by the model for cells of this
type (Figure 2). This is the level for much of the modeling proposed here, and we have developed a range of tools for simulating networks of such neurons in our Neural Simulation Language NSL (Arbib et al., 2001).
The FARS Model
In our approach to reaching and grasping, we first modeled the AIP-F5canonical circuitry (the FARS model described in
this section) and then proceeded therefrom to develop MNS1, our first mirror neuron system model, described in the next
section. The FARS model (Fagg and Arbib 1998) analyzed the data of Sakata and Rizzolatti to show how F5 and AIP may act
as part of a visuo-motor transformation circuit. The FARS model is based on the physiological properties of the circuits for
transforming object intrinsic properties into grasping movements. The main components of this circuit are the AIP neurons
and the F5 canonical and motor neurons. Gibson (1966) observed that the pattern of optic flow, or the movement of features
across the retina from moment to moment, contained valuable information that could be used to guide navigation through the
environment without prior recognition of objects. We thus adopted Gibson's term affordances to mean parameters for motor
interaction that are signaled by sensory cues without invocation of high-level object recognition processes. The FARS model
developed two hypotheses: (1) The anterior intra-parietal area AIP serves a dual role of computing a set of affordances appropriate for the object being attended and then maintaining an active memory of the single affordance as the corresponding
grasp is executed. (2) Premotor area F5 integrates many constraints to decide on the single grasp that is to be executed. The
model gave the first causal account of the firing patterns observed in AIP and F5, made novel predictions for response to situations involving multiple affordances, and led to Synthetic PET predictions (Arbib et al., 1995; Arbib et al., 2002) and related
PET studies of humans (Grafton et al., 1998).
An important feature of the work leading to the development of the FARS model was that it involved analysis of an extensive set of unpublished records of F5 and AIP cell activity made available to us by Rizzolatti and Sakata. Whereas published
work showed very few records and emphasized only the correlation of cell activity with a specific type of grasp, our data
analysis revealed that neurons also showed a variety of temporal relationships with different phases of the monkey's behavior.
This led us both to a population model of F5 that provided both the first exploration of this rich data set, and to a convincing
explanation of the data. This collaboration with experimentalists to integrate extensive data analysis with the development of
new models is paradigmatic of the effort proposed here. Having offered this paradigm, we trust to the reader's powers of generalization to infer related details for each Specific Aim below.
MNS1: The First Mirror Neuron System Model
The first Mirror Neuron System model (MNS1; Oztop and Arbib to appear) analyzes the role of F5 and related brain regions in the recognition of single actions. MNS1 analyzes pathways directed toward F5 mirror neurons which match arm-hand
trajectories to the affordances and location of a potential target object. The crucial notion here was the extraction of the hand
state which relates the hand to an object. In our MNS1 simulations the feature vector that serves as the hand state input is given by trajectories F(t) with 7 components  d(t): distance to target at time t; v(t): tangential velocity of the wrist ; a(t): aperture
of the virtual fingers involved in grasping at time t; o 1(t): angle between the object axis and the (index finger tip – thumb tip)
vector; o2(t): angle between the object axis and the (index finger knuckle – thumb tip) vector; and o3(t), o4(t): angles defining
how close the thumb is to the hand as measured relative to the side of the hand and the inner surface of the palm. The choice
of the ok's was based on our earlier work on opposition spaces (Iberall et al., 1986).
MNS1 combines three "grand schemas"  Visual Analysis of Hand State, Reach and Grasp, and the Core Mirror Circuit 
for each of which we presented a detailed implementation. With this implementation we showed how the mirror system may
learn to recognize actions already in the repertoire of the F5 canonical neurons, and then studied the performance of the model under conditions of spatial perturbation, altered kinematics, and grasp object mismatch. Despite the use of artificial neural
networks, our MNS1 model yielded solid empirical predictions for the temporal firing patterns of mirror neurons, showing
that neurophysiologists should analyze them in a more quantitative way with respect to observed actions. For just one example
from many (Figure 2 Right), we created a scene where the observed action consisted of grasping an unusually wide object
with a precision pinch. MNS1 first recognized the action (in its early stages) as a power grasp, but as the action progressed the
modeled mirror neurons representing precision pinch became active while the power grasp activity started to decline. Thus we
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 9
specifically predict that in such a grasp observation experiment performed with monkeys, power-grasp related cells will fire
actively but only initially, while precision-pinch cells will start firing as the power grasp activity declines. This particular prediction, like the others already developed for MNS1, is clearly testable using experimental setups such as those in the Rizzolatti lab. Our new modeling will yield further predictions.
Modeling Sequential Behavior
Patients with diseases of the BG, particularly Huntington’s disease and Parkinson’s disease, do not have significant motor
control difficulties when visual input is available (Bronstein and Kennard, 1985), but do have problems with specific forms of
internally driven sequences of movements (Curra et al. 1997). This implies that the basal ganglia (BG) may be involved in
assisting cortical planning centers in some fashion as well as providing sequencing information for cross-modal movements,
e.g., simultaneous arm reach, hand grasp, head movement and eye movement. Dominey and Arbib (1992) explicated the interaction between cerebral cortex and basal ganglia in saccade control. They showed how thalamocortical loops could implement target memory, while remapping of retinotopic targets during saccades was conducted by circuitry intrinsic to the region
LIP of posterior parietal cortex. We (Bischoff, 1998; Bischoff-Grethe and Arbib, in preparation) built on this effort to model
the interactions between BG, prefrontal cortex (PFC) and supplementary motor area (pre-SMA and SMA-proper) in the control of sequences of skeletomotor actions. We modeled PFC as responsible for working memory and as a source for the sequences that pre-SMA “knows.” Pre-SMA projects sequential information both to SMA-proper and to the basal ganglia’s
indirect pathway. SMA-proper is involved in the internal generation of sequences and repetitive movements. SMA-proper
neurons contain information on the overall sequence to be performed; they keep track of which movement is next and which
movement is currently being performed; they project the current movement to be performed to MC and to the direct pathway
of the basal ganglia; and they project the preparations of the next movement of the sequence to another population of premovement neurons within motor cortex (F1) and to the indirect (inhibitory) pathway of basal ganglia. The motor cortex, then,
carries out the motor command and handles the fine tuning of the movement (e.g., target position, amplitude, force) partly
based upon information provided by the basal ganglia. The motor cortex projects the motor parameters to both the brainstem
and the basal ganglia’s direct pathway. We postulate that the basal ganglia’s two pathways perform two different roles: the
indirect pathway inhibits upcoming motor commands from being performed while the current movement is in progress, while
the direct pathway projects the next sensory state back to cortex. This informs SMA-proper and F1 of the expected next state
and allows these regions to switch to the next movement of the sequence.
3
Research Plan
The five modeling efforts which together define MNS2, our new mirror neuron system model, will be described under two
headings, "Development of the Mirror System" and "Recognition of Novel and Compound Actions and their Context".
3.1
Development of the Mirror System
In Section 4.2, we will model the "mature" mirror system, making contact with a growing body of data on the role of areas
like STS and PF in providing necessary input concerning hand movements and their relation to objects. However, there is
immense importance in analyzing how a system develops in addition to exploring and understanding its adult structure. This
topic was the focus of discussion when Arbib and Oztop visited Parma in May 2001, and the present section offers a plan of
research based on this phase of our collaborative effort with Parma. To illuminate the development of so apparently complicated a system will be no small achievement. Indeed, the idea that complex cognitive functions, like action understanding and
imitation learning, can be based on simple motor schemas and a system that recognizes the actions they generate offers exciting new ideas for the research on human imitation that will complement the research proposed here.
We will model three interwoven stages for mirror neuron development: a) the formation of F5 motor neurons on the basis
of random grasping and the haptic feedback generated by successful grips; b) the formation of F5 canonical neurons on the
basis of random grasping and visual input via AIP concerning object properties; and c) the formation of F5 mirror neurons on
the basis of self-generated goal-directed grasping movements using the association between F5 motor activity and the visual
stimuli from STS and PF concerning hand movements in relation to the grasped object. This work will involve collaboration
with the Schaal group on the biomechanical model of the arm-hand system and its interactions with objects.
Arbib: Mirror Neurons and Action Recognition
August 2001
Object Identity
(IT)
SII
?
Haptic
feedback
FARS
Object Info
(cIPS)
AIP
F5canonical
F1
F5mirror
Hand Conf.
(STS?)
PF
F2
F4
Hand
Page 10
Figure 5. The diagonal hatching
marks the regions involved in the
grasp learning task. F5canonical 
F1  Hand  SII + AIP  F5 is the
loop that we model in this study. The
horizontal-lined rectangle marks the
main regions involved in the full
visuo-motor transformation for grasping. These were previously "hardwired" in the FARS model; the challenge now is to show how the circuitry could "self-organize" via the grasp
learning circuit, with which it overlaps. F4 supplies the egocentric position of the object. The vertical-lined
rectangle marks the main regions
involved in mirror neuron functioning.
Where research to date has assumed that the mirror system arose to facilitate social understanding, it has not enquired into
the possible biological roots of this function. MNS1 showed how the observation of self-action may serve as the learning
stimulus for shaping the mirror system but did not address the issue of why the brain might contain such learning hardware in
the first place. We here offer a dramatically new hypothesis: that it was the need for precise visual feedback for delicate hand
actions that led to the appearance of mechanisms for extracting "hand configuration", and that it is this that was readily exapted to form the bias for recognition of hand movements made by others.
Development of Grasp Specificity in F5 Motor and Canonical Neurons
In modeling to date (and in the experiments), the tendency was to pair mirror neurons with an externally defined set of
grasps. In the proposed modeling, we would hope to show how a basic set of grasps emerges from a repertoire of basic
movements (e.g. reaching and enclosing) through learning. To model the development of the F5 motor neurons and canonical
neurons, we will demonstrate how somatosensory feedback can play a crucial role in defining the population of F5 motor neurons, and how AIP input shapes up the F5 canonical subpopulation and is shaped up in turn, as the developing. F5 canonical
neurons select visual neurons describing a variety of surfaces via re-afferent connections. Only those selected become AIP
neurons that code affordances. The model will take visual input from a non-neural schema for cIPS which will encode surface
orientation in a fashion which captures the data of Sakata et al. (1997a,b), as well as somatosensory information computed
using the proposed extension of the hand-arm avatar to determine contact forces and slip when the hand encounters an object.
The output will initially associate a random pattern of grasping with F5 activity but will, through learning, create a repertoire
of grasp actions (precision grip, power grip etc.) that are appropriate for the objects to which the model is exposed. In particular, we expect to see the emergence of a population code in F5 for grasping actions. Our predictions on grasp population coding may lead to experiments that will complement the reach-related findings (e.g., Georgopoulos et al., 1988 for data;
Lukashin et al., 1996 for modeling). The modeling will be constrained by available data on the development of reaching and
grasping; the performance of the "adult" neurons of the model will be tested against data from Parma.
Based on the assumption that development of reaching and grasping involve similar processes in human and monkey, we
will use studies of human infants to set behavioral goals for our grasp-learning model. Human infants are able to reach an
object by around 12 weeks of age, which precedes by 3 to 4 weeks the time when the infant starts to grasp objects (Clifton et
al. 1993). The fractionated control of finger movements is not possible at this stage of reflex grasping and early voluntary
grasping since this requires the cortico-motoneuronal system, which has not been developed by this stage (Lantz et al, 1996).
Therefore, it is unlikely that the premotor specialisation for the different types of grasp (e.g., precision grasp, side grasp) has
been formed at this age. During these 3 to 4 weeks, the motor primitives for grasping are developed, but are not properly triggered by visual stimuli. However, when the infant's hand touches the object, grasping will often triggered by somatosensory
stimuli. This is due either to the innate reflex grasp or to the joy of grasping the object. The reflex grasp stays with the infant
until six months of age and it takes 4 more weeks to stabilize the grasp (Clifton et al., 1993).
The first components of our MNS2 model are aimed at capturing the discovery of grasp configurations (i.e. the premotor
specializations) starting from a reach capable (model) stage. The learning procedure which yields the basic population of F5
mirror neurons will adjust the connectivity of the circuits within and between F5 and F1 based on somatosensory inputs, so to
encode different grasp actions, assuming that the mechanism for producing a reach to a given target in peripersonal space
already exists. Through learning, the reaches directed to objects will be shaped into grasp actions via the enclosure (palmar
reflex) triggered by the touch of the object to the hand. Then the haptic feedback from the fingers will be used to determine a
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 11
successful grasp. The model postulates that this information will be available in the somatosensory cortex after a grasp attempt. This area encodes information about slip and force feedback during prehension of objects (Salimi et al., 1999a,b) and
is involved in sensory monitoring of hand-object interactions (Debowy et al., 2001). In the model, somatosensory cortex will
supply the training or reinforcing signal generated by our expanded hand simulator's estimate of contact force and slippage to
adjust the grasp planning circuit (F5-F1) connection strengths.
A perfect, adult-like grasp requires considerable visual analysis of the object (affordance extraction). We postulate that the
visual information to the grasp learning circuit is very limited at the early phases of grasping (4-9 months) as suggested by
Rosenbaum (1991): “what is most likely is that infants younger than nine months have not yet learned to pre-program grip
size on the basis of visual information, just as infants younger than nine months have not yet learned to pre-program hand
orientation based on vision.” Thus, augmenting the development of F5 motor neurons in general will be the development of
F5 canonical neurons on the basis of AIP's recoding of the surface orientation data provided by cIPS. We will attempt to understand how the reciprocal connections between AIP and F5 canonical neurons enable each to shape the other so that, as the
F5 canonical neurons develop, so too do AIP neurons become better adapted to convert cIPS input into affordance-encoding
output. At the early stages of learning, the affordance representation covers only egocentric object location (this information
must be present because the reach task can be performed). Based on currently available information about affordances, F5
specifies to F1, and the motor plant executes, a specific grasp action. If it turns out that the plan was successful, that is if somatosensory cortex signals a success, then the connections (cIPS  AIP, AIP  F5, F5  AIP) contributing to that decision
are enhanced. If it was a failure then the connections contributing to the decision are decreased. Thus the learning we use can
be called supervised anti-Hebbian or simple reinforcement learning. A minor problem is that the grasp plan for a given input
need not be unique. For example, given an apple to grasp we can approach the target from any. In some sense then the circuit
learns the conditional probabilities of grasp parameters given the affordance input (and other contextual biasing) using reinforcement learning.
Visual Feedback for Grasping: A Possible Precursor of the Mirror Property
We will explore here the novel hypothesis that the F5 mirror neurons develop by selecting, via re-afferent connections, patterns of visual input describing those relations of hand shapes and motions to objects that prove effective in visual guidance of
a successful grasp. Such a hypothesis will let us tap data on error correction to refine our developing model of action recognition. The validation here is computational: if the hypothesis is correct, we will be able to show that such a hand control system
indeed exhibits most of the properties needed for a mirror system for grasping. For a reaching task, the simplest visual feedback is some form of signal of the distance between object and hand. This may suffice for grabbing bananas, but for peeling a
banana, feedback on the shape of the hand relative to the banana, as well as force feedback become crucial. We predict that
the parameters needed for such visual feedback for grasp will look very much like those we specified explicitly for our MNS1
hand state.
We will model superior temporal sulcus (STS) and PF as providing crucial inputs for the premotor mirror system. The result will be a neural representation of the observed scene to be processed by other regions. The modeling will address available data, but also feed into the design of new experiments. For example, Jellema et al. (2000) found cells in anterior STS
(STSa) which respond selectively to the sight of reaching but only when the agent performing the action is seen to be attending to the target position of the reaching. One cell population responds selectively to faces, eye gaze, and body posture, and
they argue that subsets of these cells code for the direction of attention of others. A second cell population is selectively responsive to limb movement in certain directions. The responses of a subset of cells sensitive to limb movement are modulated
by the direction of attention (indicated by head and body posture of the agent performing the action). The Parma group plans
to record from monkey STS with multiple electrodes to understand better its organization (somatotopy, motor properties,
movement vs. action).1 The recent findings from Parma on mirror-like neurons in area PF of parietal cortex (Fogassi et al.,
1998) and the connection of this area with the mirror neuron region of area F5 (Matelli 1994) indicates an intimate relation
between PF and F5 mirror neurons. We propose that PF mirror neurons provide crucial input for F5 mirror neurons. The similarity of STS and PF responses to active hands, combined with the connectivity pattern of superior temporal sulcus and area 7
makes the STS-PF circuit a plausible approximation to the primate hand shape-motion recognition circuit. Fogassi is studying
the mouth field and will continue his PF studies.
1 The Parma group will also study imitation learning in humans, with the idea that the efferent copy of a movement activates STS (as
shown by preliminary data with our HFSP colleague Iacoboni at UCLA), and that STS (and/or PF) compares the observed action with the
efferent copy of the action, thus allowing the matching necessary to learn new actions. We will explore similar ideas in our analysis of the
monkey data as a basis for work on human imitation under other funding.
Arbib: Mirror Neurons and Action Recognition
Learn by Imitation
"Social Learning"
view
of action
action
description
STS and PF
expectation
ENN
Mirror neurons
Non-Mirror Neurons
August 2001
Page 12
Learn by Doing Figure 6. The paired forward-inverse model
Try to Grasp view of the F5 grasp control circuit of Arbib
view of object
and Rizzolatti (1997). The vertical path at
Object
right is the execution system from "view of
object" via AIP and F5 to "grasp of object" 
AIP
it provides mechanisms for grasping a seen
object. The loop on the left of the figure proMP: Action
Motor Program
vides mechanisms for imitating observed gesaction
tures in such a way as to create expectations
recognition
which enable the visual feedback loop to serve
both for "social learning" (i.e., learning an
F5
action through imitation of the actions of others) and also for (delayed) error correction
command
during, e.g., reaching towards a target. It comcorollary
bines the observation matching system (indischarge
MPG
verse model) from "view of gesture" via gesture description (STS) and gesture recognition
(PF) to a representation of the "command" for
grasp of object
such a gesture, and the expectation system
(direct model) from an F5 command via the
expectation neural network ENN to MP, the
motor program for generating a given gesture.
The latter path may mediate a comparison
between "expected gesture" and "observed
gesture" for the monkey’s self-generated
movements.
Arbib and Rizzolatti (1997) gave an informal view of how the mirror system might be viewed as pairing a forward and inverse model (Figure 6). This approach was inspired by the work of Jordan and Rumelhart (1992) who proposed a "distal
learning" scheme wherein the forward model (a neural circuit predicting the outcome of a given motor command) is trained
first, and then the inverse model (a neural circuit computing the command required to achieve a desired outcome) is trained
using the error that is propagated backward through the forward model. Leaving aside Arbib and Rizzolatti's interest in imitation for the present proposal, we now propose formal neural modeling to focus on the role of error correction in the development of the mirror system. This effort will take into account recent findings (Wolpert and Kawato, 1998; Haruno et al. 2001)
suggesting the utility of multiple paired forward and inverse models in motor control  it may be easier to combine controllers
specialized for specific subtasks to meet the needs of a wide variety of tasks rather than to have a single super-complex controller which can handle every task directly.
Our earlier model of the mirror neuron system, MNS1 actually modeled the forward model of Figure 6 where the output of
visual processing trained the mirror neurons. For MNS1 we did not model the low level motor control and learning of the
inverse model but used the Grasp Simulator to perform the desired grasp (the lower row of the figure). Now we propose to
implement full learning in the forward model and inverse model with the sole goal of accomplishing grasp control. Then what
we predict is that output units of the forward model will be armed with mirror property while the output units of the inverse
model will attain the F5 canonical property. The muscimol study of Fogassi et al. (2001) strongly supports our proposal. The
inactivation of mirror neurons does not abolish grasping but only slows down the actions. However, the inactivation of canonical neurons heavily degrades the grasping performance in terms of preshaping and orienting the hand. As can be seen in Figure 6, in the proposed model the inverse model generates motor programs, thus the destruction of this structure will abolish
the motor output of the model while the destruction of the forward model will not have an effect in the short term. In the long
term, assuming learning still continues, it will disrupt the inverse model by channeling wrong error signals. Just as we propose
to analyze plasticity of the connections cIPS  AIP, AIP  F5, and F5  AIP in understanding the tuning of AIP to more
accurately encode affordances for F5 canonical neurons, so too will we use the present study to better understand the tuning of
STS and PF in providing input to the F5 mirror neurons. Implementation of the model will enable us to make more detailed
predictions on the relation between F5 canonical and mirror neurons.
3.2
Recognition of Novel and Compound Actions and their Context
The modeling of development defined above emphasizes how the infant monkey builds a basic motor repertoire of reachand-grasp actions and how the infant comes to recognize hand-object relations in other monkeys which signal similar actions.
In the present section we propose models for the recognition of novel actions, presenting hypotheses for: (i) How a variant of
a known action comes to be recognized. (ii) How a novel action may be recognized as a compound of (variants of) known
actions. (iii) How actions are "understood". We argue that this will in general involve more than recognition of the action
(movement + goal) in isolation, but will also involve recognition of the context in which the action occurs and expectations as
to the consequences of that action.
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 13
The Pliers Experiment: Extending the Visual Vocabulary
When a monkey first watched the experimenter grasping a raisin with a pair of pliers, no mirror neurons discharged, but after several demonstrations, some of the previously silent neurons started to fire when the pliers were approaching the raisin
(G. Rizzolatti, personal communication). We will analyze this as part of a more general study of how an action comes to be
recognized when it is a variant of a known action. Experimental studies to date have emphasized the actions (both observed
and executed) that are encoded by a single mirror neuron. However, many neurons are active for each type of grasp, and we
thus argue that it is more appropriate to see a set of neurons providing a nuanced representation of a grasp rather than focusing on the very broadly tuned response of a single neuron. A key aspect of this modeling will thus be a focus on population
coding which will be strongly linked with the increasing push of the Parma group to use multi-electrode recording. We expect
to show that population coding is an emergent property from our modeling of development and learning in the mirror system
 that once one has a variety of grasps, cells will be more or less correlated with distinctive subsets of mirror neurons. We
further predict that learning a variation on a movement can be done more efficiently by building on the population code for
the original movement and others that have already been learned than by training dedicated "grandmother" cells, and we will
model this phenomenon to provide explicit predictions. Building upon the pliers study, we will explore the hypothesis that the
mirror system can (a) recognize an action as similar to a known action while (b) delivering a crude analysis of the difference
between the observed motion and this approximation as the means for recognizing a new class of actions. Such capabilities
should be facilitated by population coding in the mirror neuron system. The activity of the population will either indicate a
confident recognition of an action or a representation of how different the action is from the ones in the action repertoire (i.e.,
departure from the projection onto the basis vectors formed by recognizable actions). The representation of this difference
could lead to extension of the basis to accommodate novel features of the new action. In either case, we have to study what is
a good representation, computationally and biologically, to give the system the flexibility that we observe in monkey mirror
neurons.
We have already noted in our discussion of the FARS model that our analysis of unpublished data "revealed that neurons …
showed a variety of temporal relationships with different phases of the monkey's behavior." Our modeling will thus make predictions to be tested by timing studies in answer to questions such as: How soon after a movement has begun will a mirror
neuron become active? Such temporal relations will be tested using proposed new features of NeuroBench to analyze Parma
multi-electrode data for temporal patterns and for population coding across neurons.
Grammont (Parma) has trained monkeys to grasp with tools. One of his tools (“escargot device”) grasps an object when the
monkey opens the hand. The issue is whether the action is coded by F5 neurons in abstract terms (grasp) or in terms of
movements when action and movements are in opposite directions. He will also examine the mirror properties of the toolusing monkey after tool learning. To model this, we must show how the brain develops an extended hand configuration representation which includes such extensions as a hand holding a tool. In particular, we must update visual input processing in our
models. Although we had completed encouraging studies of the recognition of hand state from photographs of human hands
(with colored patches to aid segmentation), our MNS1 work used a quasi-neural encoding of the positions of arms and hands
relative to objects based on the hand state from the reach-grasp simulator as input. But this will not work for modeling the
pliers experiment. Thus a subgoal of this work will be to develop a more generic vision system to recognize a hand holding a
tool, and then combine this with affordance data to recognize extensions of hand configuration. It would be very useful to
have a vision system that can work on real video data and extract the information required for the circuits we model, but modeling the actual biological circuitry (e.g., that of inferotemporal cortex) is in itself a major research project beyond the scope
of the present proposal.. Nevertheless there are successful vision studies from which we can benefit and which can be adapted
for useful integration with our models to obtain structurally complete systems. We plan to accomplish such a task using Arbib's collaboration with Dr. E.J. Holden and Professor Robyn Owens of the University of Western Australia who have worked
on vision algorithms for the recognition of hand, head and lip movements (Holden et al., 1996, 2000; Holden and Owens,
2000, 2001).
Recognition of Compounds of Known Movements
Our prior modeling of "compound actions" (see "Modeling Sequential Behavior") has focused on sequences of known actions, whether saccades or arm movements. Indeed, Byrne (in press) suggests that a novel action can be imitated (and so, a
fortiori, recognized) by dissecting it into a string of simpler sequential parts that are already in the observer’s repertoire. Rizzolatti et al. (to appear) comment that Byrne's analysis "opens new empirical possibilities and may clarify why only humans
and some species of primates appear to be able to imitate in the proper sense." However, Arbib (2000) has already observed
that imitation is far more subtle than Byrne suggests. Figure 7 offers a stripped-down example of an alternative view of the
recognition of novel actions. This open up even more "new empirical possibilities."
Arbib: Mirror Neurons and Action Recognition
=
=
+
+
August 2001
Page 14
Figure 7. This diagram makes the point that the recognition of an action need not reduce to recognition of a sequence of familiar actions but may instead involve increasing success in approximation as details are attended to. In
this case, the movement at top is first recognized as a linear movement from start to finish, then as a movement with
a "bump" in the middle, and it is only after this second
approximation has been learned that the loop can be
properly attended to and recognized. The action is then
recognized as a temporally coordinated superposition of
movements, rather than a sequence of known actions.
We will extend our analysis of population coding of single actions to model learning and recognition of compounds. We
will first extend our earlier work on the interaction of basal ganglia (BG) and supplementary motor area (SMA) working
memory and sequences of movement (e.g., Dominey and Arbib, 1992; Bischoff-Grethe and Arbib, in preparation) to develop
hypotheses on how BG and SMA interact with the mirror system so that temporal sequences ("the Byrne case") can be extracted, stored and learned. We will then generalize this approach to handle the recognition of novel actions that are formed as
temporally coordinated superpositions, rather than sequences, of known actions. This modeling effort will provide a conceptual analysis that sets the stage for design of experiments several years in the future. "To see which is the best (or possible)
solution in terms of modeling can help a lot in terms of future experiments" (Rizzolatti, personal communication). However,
the model will contain components that will be richly constrained by available neurophysiological data (e.g., Tanji and Shima
1994; Tanji et al., 1996). We also note that our USC colleague Aude Billard has developed a learning mechanism for the
recognition of variants and compounds of known movements in robots and humans (Billard, 2000a, b; Billard and Hayes,
1999; Billard and Mataric, 2000) so that comparison of our work on the monkey mirror system with her work on mechanisms
of human imitation will enrich the research proposed here.
From Action Recognition to Understanding: Context and Expectation
In one Parma study of PF, 61 cells were responsive when the monkey observed biological actions, and 2/3 of these were also active during the monkey's own actions. However, about a quarter of these “PF mirror neurons” do not match observed
actions to congruent executed actions. For example, a cell active for observation of downward motion of the hand when
grasping an object may also be active during execution of grasping by mouth. At first this may seem counter to the notion of a
mirror neuron but for us it sets the stage for a deeper analysis. It has often been said that mirror neurons are involved in "understanding" of actions, but for the present effort we stress that understanding will in general involve more than the recognition of an action in isolation, and may also involve some notion of "meaning", e.g., the context in which the action is appropriate and the expectations that such a behavior evokes. This relates to our earlier work on Schema-Based Learning (Corbacho, 1997). Thus where the previous Aim focused on the recognition of compound actions, the present Aim emphasizes the
recognition of context and expectations, where, for example, recognition of one action may be seen as a preliminary for either
doing something or predicting what the observed primate will do next (e.g., bringing food to the mouth to eat). The context
and expectations set the stage for action recognition, action recognition modifies the context and expectations, and so on. This
will let us explore the notion that mirror neurons can act as the basis for "understanding" if a given action can be placed in the
context of its observed (in self and/or others) consequences.
The modeling here will build on recent work (Suri, Bargas and Arbib, 2001; Suri and Schultz 1999, 2001) to extend the
Temporal Difference (TD) Model (Sutton and Barto 1998; Doya 1996, 1999) of reinforcement learning. It will also build on
our earlier efforts relating the basal ganglia to various types of learning (Dominey, Arbib and Joseph, 1995). The TD Model
reproduces reward-predictive aspects of dopaminergic activity (Apicella et al., 1992, Montague et al. 1996, Schultz et al.
1993, 1997) but, in contrast to the proposed modeling, cannot reproduce predictive neural activity discriminating between
events. Such neural activity was reported in several studies (Hikosaka et al. 1989, Duhamel 1992, Kermadi and Joseph 1995,
Watanabe 1996) relevant to the work proposed here. We also note the finding of Mauritz and Wise (1986) of neuronal activity in rhesus premotor cortex anticipating predictable environmental events, and of Romo and Schultz (1992) of the relation
between basal ganglia and supplementary motor area in the internal generation of movements. Suri et al. (2001) showed that
the capability for planning is improved by influences of dopamine on the durations of membrane potential fluctuations and by
manipulations that prolong the reaction time of the model, suggesting that responses of dopamine neurons to conditioned
stimuli contribute to sensorimotor reward learning, that novelty responses of dopamine neurons stimulate exploration, and that
transient dopamine membrane effects are important for planning – all factors of relevance in explicating the role of basal ganglia in recognition of novel compound actions. In Model Predictive Control (Garcia et al., 1989), predictions for future sequences of control actions are computed and then the actions are selected which optimize the outcome. Showing how neural
activity might reflect variables of such control algorithms will help to bridge the gap between these brain processes and wellknown control algorithms.
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 15
References
Andersen R.A., Shenoy K.V., Snyder L.H., Bradley D.C., Crowell J.A., 1999, The contributions of vestibular signals to the
representations of space in the posterior parietal cortex. Ann NY Academy of Science, 871: 282-292
Apicella, P., Scarnati, E., Ljungberg, T. and Schultz, W. , 1992, Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J. Neurophysiol. 68:945-960.
Arbib MA, Bischoff A, Fagg A, Grafton S (1995). Synthetic PET: Analyzing Large-Scale Properties of Neural Networks.
Human Brain Mapping 2: 225-33.
Arbib, M. A., 1981, Perceptual Structures and Distributed Motor Control, in Handbook of Physiology, Section 2: The Nervous System, Vol. II, Motor Control, Part 1 (V. B. Brooks, Ed. ), American Physiological Society , pp. 1449-1480.
Arbib, M., and Rizzolatti, G., 1997, Neural expectations: a possible evolutionary path from manual skills to language, Communication and Cognition, 29, 393-424.
Arbib, M.A., 2000, The Mirror System, Imitation, and the Evolution of Language, in Imitation in Animals and Artifacts,
(Chrystopher Nehaniv and Kerstin Dautenhahn, Editors), The MIT Press (in press).
Arbib, M.A., Alexander, A., and Weitzenfeld, W., 2001, NSL Neural Simulation Language, in Computing the Brain: A Guide
to Neuroinformatics, (M. A. Arbib, and J.S. Grethe, Eds.), San Diego: Academic Press, pp.71-90.
Arbib, M.A., and Bischoff-Grethe, A., 2001, Summary Databases and Model Repositories, in Computing the Brain: A Guide
to Neuroinformatics, (M. A. Arbib, and J.S. Grethe, Eds.), San Diego: Academic Press, pp.287-296.
Arbib, M.A., Billard, A., Iacoboni, M., and Oztop, E., 2000, Synthetic Brain Imaging: Grasping, Mirror Neurons and Imitation, Neural Networks, 13: 975-997.
Arbib, M.A., Fagg, A.H., and Grafton, S.T., 2002, Synthetic PET Imaging for Grasping: From Primate Neurophysiology to
Human Behavior, in Explorative analysis and data modelling in functional neuroimaging, (F. Sommer and A. Wichert,
Eds.), Cambridge MA: The MIT Press (to appear).
Barnes C.L., Pandya D.N., 1992, Efferent Cortical Connections Of Multimodal Cortex of the Superior Temporal Sulcus In
The Rhesus-Monkey. Journal of Comparative Neurology, 318: (2) 222-244
Barone, P. and Joseph, J.-P., 1989, Prefrontal cortex and spatial sequencing in macaque monkey, Exp Br Res 78:447-464.
Barto, A. G., S. Richard and C. W. Anderson (1983). Neuronlike adaptive elements that can solve difficult learning control
problems. IEEE Trans. on Systems, Man, and Cybernetics SMC-13: 834-846.
Billard, A, 2000a, Imitation: a means to enhance learning of a synthetic proto-language in an autonomous robot. In C.
Nehaniv and K. Dautenhahn, editors, Imitation in Animals and Artifacts. MIT Press, to appear.
Billard, A., 2000b, Learning motor skills by imitation: a biologically inspired robotic model. Cybernetics & Systems Journal,
special issue on Imitation in animals and artifacts, to appear.
Billard, A., and Hayes, G., 1999, Drama, a connectionist architecture for control and learning in autonomous robots. Adaptive
Behavior, 7:35-64.
Billard, A., and Mataric, M., 2000, Learning human arm movements by imitation: Evaluation of a biologically-inspired connectionist architecture. In Proceedings, First IEEE-RAS International Conference on Humanoid Robotics (Humanoids2000), MIT, Cambridge, MA, Sep 7-8, 2000.
Bischoff, A., 1998, Modeling the Basal Ganglia in the Control of Arm Movements, Ph.D. Thesis (May 1998), Department of
Computer Science, University of Southern California.
Bischoff-Grethe, A., and M.A. Arbib, Sequential movements: A computational model of the roles of the basal ganglia and the
supplementary motor area, (in preparation).
Bota, M., 2001, Neural homologies: principles, databases and models, University of Southern California, Ph.D. Thesis in
Neuroscience (to appear).
Bota, M., 2001a, The NeuroHomology Database Version I, NHDB-I,
http://bsl9.usc.edu/scripts/webmerger.exe?/database/homologies-main.html.
Bota, M., 2001a, The NeuroHomology Database Version I, NHDB-II,
http://java.usc.edu/neurohomologies/apb/webdriver?MIval=homologies-main.
Bota, M., and Arbib, M.A., 2001, The Neurohomology Database, in Computing the Brain: A Guide to Neuroinformatics,
(M.A. Arbib, and J.S. Grethe, Eds.), San Diego: Academic Press, pp.337-351.
Bower, J.M., and Beeman, D., 1998, The Book of GENESIS: Exploring Realistic Neural Models with the GEneral NEural SImulation Systems, 2nd edition, TELOS/Springer-Verlag.
Bronstein, A.M., and C. Kennard, Predictive ocular motor control in Parkinson's disease, Brain. 108:925-940 (1985).
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 16
Byrne, R.W. (in press). Imitation in action. Advances in the study of behaviour.
Cavada C., Goldman-Rakic P.S., 1989, Posterior parietal cortex in Rhesus monkey: II. Evidence for segregated corticocortical networks linking sensory and limbic areas with the frontal lobe. Journal of Comparative Neurology 287:422-445
Clifton, R.K., W.M. Darwin, D.H. Ashmead, M.G. Clarkson, 1993, Is Visually Guided Reaching in Early Infancy a Myth?
Child Development, 64: 1099-1110.
Colby, C. L., Goldberg, M. E., 1999. Space and Attention in Parietal Cortex. Annu. Rev. Neurosci. 22:319-49.
Corbacho, F., 1997, Schema Based Learning: Towards a Theory of Organization for Adaptive Autonomous Agents, Ph.D.
thesis, Computer Science, University of Southern California, August 1997.
Curra, A., A. Berardelli, R. Agostino, N. Modugno, C.C. Puorger, N. Accornero, and M. Manfredi, Performance of sequential
arm movements with and without advance knowledge of motor pathways in Parkinson's disease, Mov Disord. 12:646-654
(1997).
Debowy DJ, Ghosh S, Ro JY, Gardner EP, 2001, Comparison of neuronal firing rates in somatosensory and posterior parietal
cortex during prehension. Exp Brain Res 137: 269-291
di Pellegrino, G., L. Fadiga, L. Fogassi, V. Gallese, and G. Rizzolatti, 1992. Understanding motor events: a neurophysiological study. Experimental Brain Research, 91:176-180.
Dominey, P. F., and Arbib, M. A., 1992, A Cortico-Subcortical Model for Generation of Spatially Accurate Sequential Saccades, Cerebral Cortex, 2:153-175.
Dominey, P.F., Arbib, M.A., and Joseph, J.-P., 1995, A Model of Corticostriatal Plasticity for Learning Associations and Sequences, J. Cog. Neurosci., 7:311-336.
Doya, K., 1996, Temporal difference learning in continuous time and space, in Advances in Neural Information Processing
Systems (Touretzky, D. S., Mozer, M. C., and Hasselmo, M. E., editors), The MIT Press, 8:1073-1079.
Doya, K., 1999, Reinforcement learning in continuous time and space. Neural Computation, 12:243-269.
Duhamel JR, Colby CL, Goldberg ME, 1992, The updating of the representation of visual space in parietal cortex by intended
eye movements. Science 255:90-92.
Fagg, A. H., and Arbib, M. A., 1998, Modeling Parietal-Premotor Interactions in Primate Control of Grasping, Neural Networks, 11:1277-1303.
Fogassi L., Gallese V, Buccino G, Craighero L, Fadiga L, Rizzolatti G., 2001, Cortical mechanism for the visual guidance of
hand grasping movements in the monkey. Vol. 124, No. 3: 571-586
Fogassi L., Gallese V., Fadiga L., Rizzolatti G., 1998, Neurons responding to the sight of goal-directed hand/arm actions in
the parietal area PF of the macaque monkey. 28th Annual Meeting of Society for Neuroscience.
Gallese V., Fadiga L., Fogassi L., Rizzolatti G., 1996, Action recognition in the premotor cortex. Brain, 119: 592-609
Garcia CE, Prett DM and Morari M., 1989, Model Predictive Control: Theory and Practice-a survey. Automatica, 25:335348.
Gentilucci M., Fogassi L., Luppino G., Matelli M., Camarda R., Rizzolatti G., 1988, Functional Organization of Inferior Area
6 in the Macaque Monkey I. Somatotopy and the Control of Proximal Movements. Experimental Brain Research, 71: 475490.
Georgopoulos, A. P., Kettner, R. E., and Schwartz, A. B., 1988, Primate motor cortex and free arm movements to visual targets in three-dimensional space, II: Coding of the direction of movement by a neuronal population, J. Neurosci.,
8:2928-2937.
Gibson, J. J., 1966, The Senses Considered as Perceptual Systems, Allen and Unwin.
Grafton, S. T., Fagg, A. H., & Arbib, M. A., 1998, Dorsal Premotor Cortex and Conditional Movement Selection: A PET
Functional Mapping Study. Journal of Neurophysiology 79:1092-1097
Grafton, S.T., Arbib, M.A., Fadiga, L., and Rizzolatti, G., 1996, Localization of grasp representations in humans by PET: 2.
Observation compared with imagination. Experiment Brain Res. 112:103-111.
Grafton, S.T., Fadiga, L., Arbib, M.A., and Rizzolatti, G., 1997, Premotor Cortex Activation during Observation and Naming
of Familiar Tools, NeuroImage, 6:231-236.
Haruno M, Wolpert DM, Kawato M, 2001, Multiple paired forward-inverse models for sensorimotor learning and control.
Neural Computation, in press
Hikosaka O, Sakamoto M, Usui S, 1989, Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J Neurophysiol 4(4)814-832.
Hines, M.L. & Carnevale N.T., 1997, The NEURON simulation environment, Neural Computation, 9:1179-1209.
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 17
Hodgkin, A.L., and A.F. Huxley, 1952, A quantitative description of membrane current and its application to conduction and
excitation in nerve, J. Physiol. Lond., 117:500 – 544.
Hoff, B., and Arbib, M. A., 1992, A model of the effects of speed, accuracy, and perturbation on visually guided reaching, in
Control of Arm Movement in Space: Neurophysiological and Computational Approaches, (R. Caminiti, P. B. Johnson,
and Y. Burnod, Eds. ), Experimental Brain Research Series 22, pp. 285-306.
Hoff, B., and Arbib, M. A., 1993, Simulation of Interaction of Hand Transport and Preshape During Visually Guided Reaching to Perturbed Targets, J . Motor Behav. 25: 175-192.
Holden, E. J., Roy, G.G., Owens, R., 1996, Hand movement classification using an adaptive fuzzy expert system International
Journal of Expert Systems V9(4): 465-480.
Holden, E.J. and Owens, R., 2000, Visual Speech Recognition using Cepstral Images Proceedings of the IASTED International Conference on Signal and Image Processing, pp.331-336.
Holden, E.J. and Owens, R., 2001, Visual Sign Language Recognition Multi-Image Search and Analysis (Klette, R., Huang
T., and Gimel'farb, G., Eds), Lecture Notes in Computer Science, Springer (to appear).
Holden, E.J., Loy, G., and Owens, R., 2000, Accommodating for 3D head movement in visual lipreading. Proceedings of the
IASTED International Conference on Signal and Image Processing, pp.166-171.
Iacoboni M, Woods RP, Brass M, Bekkering H, Mazziotta JC, Rizzolatti G., 1999, Cortical mechanisms of human imitation.
Science 286:2526-8.
Iberall T., Arbib M.A., 1990, Schemas for the Control of Hand Movements: An Essay on Cortical Localization. In Goodale
M.A., editor. Vision and action: the control of grasping. Norwood, NJ: Ablex, 163-180.
Iberall, T., Bingham, G., and Arbib, M.A., 1986, Opposition Space as a Structuring Concept for the Analysis of Skilled Hand
Movements, Experimental Brain Res. Series, 15:158-173.
Jeannerod, M. , Arbib, M. A. , Rizzolatti, G. , and Sakata, H. , 1995, Grasping objects: the cortical mechanisms of visuomotor
transformation, Trends in Neurosciences, 18:314-320.
Jellema T, Baker CI, Wicker B, Perrett DI, 2000, Neural representation for the perception of the intentionality of actions.
Brain Cogn 44:280-302
Jenkins, I.H., D.J. Brooks, P.D. Nixon, R.S.J. Frackowiak, and R.E. Passingham, Motor sequence learning: A study with positron emission tomography, J Neurosci. 14:3775-3790 (1994).
Jordan MI, Rumelhart DE, 1992, Forward models: supervised learning with distal teacher. Cognitive Science 16: 307-354
Kermadi I, Joseph JP, 1995, Activity in the caudate nucleus of monkey during spatial sequencing. J Neurophysiol 74: 911933.
Lantz, C., K. Melen, H. Forssberg, 1996, Early infant grasping involves radial fingers, Developmental Medicine and Child
Neurology, 38: 668-674.
Lukashin, A. V., Amirikian, B. R., and Georgopoulos, A. P., 1996, Neural computations underlying the exertion of force: a
model, Biol. Cybern., 74:469-478.
Luppino G., Matelli M., Camarda R.M., Gallese V., Rizzolatti G., 1991, Multiple representations of body movements in mesial area 6 and the adjacent cingulate cortex: an intracortical microstimulation study in the macaque monkey. Journal of
Comparative Neurology. 311:463-482
Luppino G., Matelli M., Camarda R.M., Rizzolatti G., 1993, Corticocortical connections of area F3 (SMA-proper) and area
F6 (pre-SMA) in the macaque monkey. Journal of Comparative Neurology. 338(1):114-140.
Maioli M.G., Squatrito S., Samolsky-Dekel B.G. and Sanseverino E.R.(1998) Corticocortical connections between frontal
periarcuate regions and visual areas of the superior temporal sulcus and the adjoining inferior parietal lobule in the macaque monkey. Brain research, 798: 118-125.
Matelli M., Camarda R., Glickstein M., Rizzolatti G., 1985, Afferent and efferent projections of the inferior area 6 in the macaque monkey. Journal of Comparative Neurology, 251: 281-298
Matelli M., Luppino G., Murata A., Sakata H., 1994, Independent anatomical circuits for reaching and grasping linking the
inferior parietal sulcus and inferior area 6 in macaque monkey. Soc. Neurosci. Abstr. 20:404.4
Mauritz KH, Wise SP, 1986, Premotor cortex of the rhesus monkey: neuronal activity in anticipation of predictable environmental events. Exp Brain Res 61 (2): 229-244.
Montague, P.R., Dayan, P. and Sejnowski, T.J., 1996, A Framework for mesencephalic dopamine systems based on predictive
hebbian learning. J Neurosci 16(5):1936-1947.
Murata A., Fadiga L., Fogassi L., Gallese V., Raos V., Rizzolatti G., 1997, Object representation in the ventral premotor cortex (Area F5) of the monkey. Journal of Neurophysiology, 78: 2226-2230
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 18
Nishitani, N., and Hari R, 2000, Temporal dynamics of cortical representation for action. Proceedings of the National Academy of Sciences of the United States of America. 97(2: 913-918.
Oztop, E., and Arbib, M.A., to appear, Schema Design and Implementation of the Grasp-Related Mirror Neuron System, Biological Cybernetics.
Pandya, D.N., Yeterian, E.H., 1998, Comparison of prefrontal architecture and connections. In The Prefrontal Cortex, Executive and Cognitive Functions, Eds. Roberts, A.C., Robbins, T.W., Weiskrantz, L. Oxford University Press.
Perrett D.I., Harries M.H., Benson P.J., Chitty A.J. and Mistlin, A.J., 1990a, Retrieval of structure from rigid and biological
motion: an analysis of the visual responses of neurons in the macaque temporal cortex. In Blake A. and Troscianko
T.(eds.) AI and the eye. John Wiley and Sons Ltd., 181-199,
Perrett D.I., Mistlin A.J., Harries M.H., Chitty A.J., 1990b, Understanding the visual appearance and consequence of hand
actions. In Goodale M.A., editor. Vision and action: the control of grasping. Norwood, NJ: Ablex, 163-180.
Petrides M. and Pandya D. N. (1984) Projections of the frontal cortex from the posterior parietal region in the rhesus monkey.
Journal of Comparative Neurology, 228: 105-116.
Ramachandran, V., http://www.feedmag.com/brain/parts/ramachandran.html, in the internet magazine Feed.
Rizzolatti G., Camarda R., Fogassi L., Gentilucci M., Luppino G., Matelli M., 1988, Functional Organization of Inferior Area
6 in the Macaque Monkey II. Area F5 and the Control of Distal Movements. Experimental Brain Research, 71: 491-507.
Rizzolatti G., Luppino G., Matelli M., 1998, The organization of the cortical motor system: new concepts. Electroencephalography and clinical Neurophysiology 106: 283-296
Rizzolatti, G., and Arbib, M.A., 1998, Language Within Our Grasp, Trends in Neuroscience, 21(5):188-194.
Rizzolatti, G., Craighero, L., and Fadiga, L., to appear, The mirror system in humans.
Rizzolatti, G., L. Fadiga, V. Gallese, and L. Fogassi. Premotor cortex and the recognition of motor actions. Cognitive Brain
Research, 3:131-141, 1996.
Romo, R. and Schultz, W., 1992, Role of primate basal ganglia and frontal cortex in the internal generation of movements. iii.
neuronal activity in the supplementary motor area. Exp.Brain Res. 91:396-407.
Rosenbaum, D. A., 1991, Human motor control. Academic Press.
Sakata H, Taira M, Murata A, Mine S., 1995, Neural mechanisms of visual guidance of hand action in the parietal cortex of
the monkey. Cereb Cortex 5(5): 429-38.
Sakata H., Taira M., Kusunoki M., Murata A., Tanaka Y., 1997a, The parietal association cortex in depth perception and visual control of action. Trends in Neuroscience, 20: 350-357
Sakata H., Taira M., Murata A., Gallese V., Tanaka Y., Shikata E., Kusunoki M., 1997b, Parietal Visual Neurons Coding
Three-Dimensional Characteristics of Objects and Their relation to Hand Action, In Parietal Lobe Contributions to Orientation in 3D Space. Their P. and Karnath H.O. (Eds.). Springer-Verlag, Heidelberg.
Salimi I, Brochier T, Smith AM, 1999a, Neuronal activity in somatosensory cortex of monkeys using a precision grip. I. Receptive Fields and Discharge Patterns. J Neurophysiol 81:825-834
Salimi I, Brochier T, Smith AM, 1999b, Neuronal activity in somatosensory cortex of monkeys using a precision grip. III
Responses to altered friction perturbations. J Neurophysiol 81:845-857
Schaal, S., Learning from demonstration, 1997, M.C. Mozer and M. Jordan and T. Petsche (Eds), Advances in Neural Information Processing Systems, 9, 1040-1046, Cambridge, MA: MIT Press.
Schultz W, Dayan P, Montague PR, 1997, A neural substrate of prediction and reward. Science 275:1593-1599.
Schultz, W., Apicella, P. and Ljungberg, T., 1993, Responses of monkey dopamine neurons to reward and conditioned stimuli
during successive steps of learning a delayed response task. J.Neurosci. 13: 900-913.
Schultz, W., R. Romo, T. Ljungberg, J. Mirenowicz, J. R. Hollerman and A. Dickinson (1995). Reward-related signals carried by dopamine neurons. In J. R. Houk et al. (Eds.), Models of information processing in the basal ganglia. Cambridge,
MA, The MIT Press. 233-248.
Siegel R.M., Read H.L., 1997, Analysis of optic flow in the monkey parietal area PG. Cereb Cortex, 7: (4) 327-346
Stein J.F., 1991, Space and the parietal association areas. In Paillard J. Editor. Brain and Space, Oxford University Press,
chapter 11.
Suri, R.E., Bargas, J., and Arbib, M.A., 2001, Modeling Functions of Striatal Dopamine Modulation in Learning and Planning, Neuroscience, 103:65-85.
Suri RE, Schultz W. 1999, A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91:871-90.
Arbib: Mirror Neurons and Action Recognition
August 2001
Page 19
Suri RE, Schultz W. 2001, Temporal difference model reproduces anticipatory neural activity, Neural Comput. 13:841-62.
Sutton RS and Barto AG, 1998, Reinforcement learning, an introduction. MIT Press/Bradford Books, Cambridge, MA,
Sutton, R. (1988). Learning to predict by the methods of temporal differences. Machine Learning 3: 9-44.
Sutton, R.S., Barto, A.G. 1990, Time derivative models of Pavlovian reinforcement. In: Learning and computational neuroscience: Foundations of adaptive networks (eds Gabriel M. and Moore. J.) MIT Press, Cambridge: pp.539-602.
Taira M., Mine S., Georgopoulos A. P., Murata A., Sakata H., 1990, Parietal Cortex Neurons of the Monkey Related to the
Visual Guidance of Hand Movement. Experimental Brain Research, 83: 29-36.
Tanji, J., and Shima, K., 1994, Role for supplementary motor area cells in planning several movements ahead, Nature.
371:413-416.
Tanji, J., Shima, K., Mushiake, H., 1996, Multiple cortical motor areas and temporal sequencing of movements. Cognitive
Brain Research, 5: 117-122
Watanabe M, 1996, Reward expectancy in primate prefrontal neurons. Nature 382(6592):629-632.
Wolpert DM, Kawato M, 1998, Multiple paired forward and inverse models for motor control. Neural Networks 11:13171329.