Download No Slide Title

LANGUAGE ORIGINS SOCIETY
UNIVERSITY OF NIJMEGEN
4-5 JULY 2003
LANGUAGE AND SPEECH AS MOTOR
ACTIVITIES
Robin Allott
http://www.percepp.demon.co.uk
WORDS AS GESTURES
ABSTRACT
For the last half-century study of the functioning of language has taken two separate forms:
an abstract purely linguistic approach - and the application of the increasingly powerful
techniques of neuroscience to the functioning of language in the brain ( PET, fMRI, MEG and
ERP ).
The language capacity is a product of the evolution of the human brain and not a purely
conventional and cultural phenomenon (the Saussurean concept).
The motor theory of language is that there is a direct relation between the functioning of
speech and motor control generally, with language depending on pre-existing motor primitives
together with the operation of motor equivalence.
Motor primitives are elementary action-units instantiated in the CNS which make possible the
construction of complex action-sequences.
Motor equivalence means that the same motor program can be executed by different
muscle/joint assemblies.
MOTOR THEORY OF LANGUAGE
The Motor Theory is a theory of the origin and functioning of language. The theory is that
the structures of language (phonological, lexical and syntactic) were derived from and
modelled on the pre-existing complex neural systems which had evolved for the control of
body movement. Motor control at the neural level requires pre-set elementary units of
action which can be integrated into more extended patterns of bodily action -- neural
motor programs. Speech is essentially a motor activity (a stream of articulatory gestures).
Language made use of the elementary pre-set units of motor action to produce equivalent
phonological units (phonemic categories). The neural programs for individual words were
constructed from the elementary units in the same way as motor programs for bodily
action. The syntactic processes and structures of language were modelled on the motor
‘syntax’.
Hockett, C.F. 1987. Refurbishing our foundations: Elementary
linguistics from an advanced point of view.
“Although listeners obviously cannot have kinesthetic feedback
from someone else's articulation, they interpret what they hear by
implicit motor- matching. ... actual movements of the organs of
speech become unnecessary; the appropriate pattern of impulses
within the central nervous system is enough.”
The paper seeks to integrate the motor theory of language into neurological
research on motor control, particularly recent research on motor programs,
motor imagery, motor primitives and motor equivalence.
The presentation draws on recent research in neuroscience by Rizzolatti
and his colleagues on mirror neurons, by Graziano on cortical action
programs, by Mussa-Ivaldi and Decety on motor imagery, and by a number
of other researchers on motor primitives and motor equivalence.
Also older material from Lotze, William James, Lashley, and Bernstein
as well as books by Berthoz and Jeannerod.
This talk is about the relation between the movements we make in speaking
and the movements we make generally: of the limbs, the head, the body, of
the face.
How do you move your arm? How do you move your tongue?
The outline of an explanation is beginning to emerge
MOTOR CONTROL
A great deal of research is attempting to discover where in the brain
particular aspects of motor control are dealt with
And ultimately to discover the network connections which are
operative in motor control.
This not the subject of this paper. The research in any case is not
sufficiently advanced and is often controversial
The aim is to look in broader terms at the progress made in thinking about
the hierarchy, stages and phases in motor control which need to be
established before the location or identification of the network connections
becomes possible (if it ever will be.)
The problem in motor control is that of the multiple degrees of freedom,
the multiplicity of muscles involved in each movement, the complex
relation between posture and movement, together with the need to
integrate motor action with visual perception, to allow for dynamic factors,
etc.
Ways in which the problem has been tackled:
“If mirror neurons form the basis for a “vocabulary” of actions, then, it
is of great importance to understand how the words of this vocabulary
may be combined with each other by the brain to span a repertoire of
purposeful behaviors how these action goals may be translated into
movements so that their concurrent activation lead to meaningful
results.”[Mussa-Ivaldi 1999]
Theoretical and experimental studies converge on the concept that
complex control problems may be solved by a combination of
independent modules.
From a neurophysiological perspective, these modules organize
specific synergies of muscles. From a mechanical perspective, these
modules generate force fields upon the controlled limbs. [MussaIvaldi]
The present view:
We make any movement through the combined operation of:
A motor image - a motor idea
Conversion of the motor image into a motor program
Integration of the motor program with the body image and the
environment image
Assembly of pre-existing motor routines into the motor program
Disinhibition of the motor program
Execution of the motor program using motor primitives of muscle/joint
organisation
[The motor program may form part of higher level action plans]
E
N
V
I
R
O
N
M
E
N
T
I
M
A
G
E
MOTOR CONTROL
MOTOR IMAGE
MOTOR PROGRAM
MOTOR PROGRAM
EXECUTE !
DISINHIBIT!
MOTOR PRIMITIVES
MOTOR ROUTINES
B
O
D
Y
I
M
A
G
E
The typical subject matter of recent research has been concerned with
movements of the hand, arm and foot - or of animal limbs: how the
muscles and joints are organised for movement and flexing of the
arm.
What is involved?
MOTOR IMAGERY
A validation and extension of the ideas of William James and
Hermann Lotze is now provided through brain-scanning techniques.
Current research in motor imagery is focused on similarities between
actual and imagined movements on a central and a peripheral level of
the nervous system.
The motor system not only executes actions but also internally
represents them in terms of ‘motor ideas’.
PET and fMRI scanning demonstrates that bodily action is preceded
by a mental picturing of the proposed action. A perceptuallyorganised pattern is transduced into a motor program and executed
by the changes in posture, changes in limb positions which
constitute human action.
This gives a neurological realisation of what in evolutionary terms
must have been an intimate intertwining of perception and action not
as separate functions but as part of single system for effective
behaviour in any creature's environment.
DECETY
Primary motor cortex activation during actual (right)
and imagined (left) gesture with the right hand - fMRI
MIRROR NEURONS
”Mirror" neurons discharge both when the monkey makes a particular action and when it
observes another individual (monkey or human) making a similar action.
Transcranial magnetic stimulation and positron emission tomography (PET) experiments suggest
that a mirror system for gesture recognition also exists in humans and includes Broca's area.
Rizzolatti et al. 1999]
Mirror neurons by linking brain systems for perception and action make
possible the imitation of gesture and other bodily action.
MIRROR NEURONS
VISUOMOTOR NEURONS
AUDIOVISUAL NEURONS
MULTIFUNCTIONAL NEURONS
VISUOMOTOR NEURON
E
Y
E
MOTOR
PROGRAM
PERCEIVED
EXECUTE
ACTION
P
R
E
M
O
T
O
R
MOTOR PROGRAMS
The concept of the motor program has evolved from earlier
understanding of motor action in terms of ‘schemas’ (Henry Head)
and motor ‘patterns’.
The motor programs, patterns or schemas operate at different
levels, or in parallel. At the highest level the motor program may be
better described as an ‘action program’; at the lowest level there
are motor sub-routines, motor elements.
Motor elements are combined in chains and in combination contingent
on the interaction of feedback and central motor programs. [Marsden]
Brain motor commands are patterned in terms of movements rather than
in terms of muscles. [Desmedt 1985]
Experimentally, the importance of preprogramming in the control of
movement has been well established. [Hollerbach 1985]
Innate and acquired motor programs
Many aspects of motor behaviour, and particularly expressive motor
behaviour, are found in new-born infants; the neural connections to support
the behaviour must have been established before birth. This requires that the
elementary motor subprograms, motor units, from which the more complex
movements are constructed, must also be pre-wired.
A range of hardwired motor programmes is available within the
immature nervous system [Marsden et al. 1985]
That this must be so is shown very clearly in the case of many animals.
MOTOR PRIMITIVES
“Our data provide strong, objective support for the conjecture made in the
past by many other motor control researchers that a repertoire of motor
primitives constitute fundamental building blocks of complex motions.
There is compelling evidence that early post-stroke recovered motions are
composed of isolated segments. In 20 [patients recovering from a single
cerebral vascular accident (stroke), we identified the apparent
submovements that composed a continuous arm motion.”[Krebs et al. 1999]
Low-level primitives
The neural circuits in the spinal cord [of the frog] are organised into a
number of distinct functional modules. The simultaneous stimulation
of two sites [in the spinal cord] leads to the vectorial summation of the
endpoints generated by each site separately. This linear behavior is
quite remarkable.
“We regard these force fields as computational primitives that are used by
the CNS for generating a rich grammar of motor behaviors - motor
primitives in analogy with language primitives used to generate unlimited
sentences out of a finite vocabulary of words.” [Mussa-Ivaldi & Bizzi 2000]
Higher-level primitives
““If mirror neurons form the basis for a “vocabulary” of actions, then, it
is of great importance to understand how the words of this vocabulary
may be combined with each other by the brain to span a repertoire of
purposeful behaviors how these action goals may be translated into
movements so that their concurrent activation lead to meaningful
results.”[Mussa-Ivaldi 1999]
PENFIELD’S HOMUNCULUS
2002 The Cortical Control of Movement Revisited.
Michael S.A.Graziano, Charlotte S.R. Taylor, Tirin Moore, and Dylan F. Cooke, Department of Psychology,
Princeton. Neuron, 36, 349–362, October 24, 2002.
Stimulation of each cortical site in the right hemisphere evoked a
different final posture of the left hand and arm.
Regardless of the starting position, stimulation
caused the hand to move toward a specific final
position.
"One possibility is that the mechanisms for speech were
built on a preexisting mechanism for motor control"
(Graziano et al.355)
MOTOR EQUIVALENCE
That the same movement can be executed by different effectors was
called ‘motor equivalence’ by Lashley (1930); Bernstein described the
concept of motor equivalence to achieve specific kinematic
goals.[1967]
The motor image or motor idea floats free of the particular limb or
particular muscle/joint assembly usually employed to execute the motor
program.
The same pen-stroke can be realised by an infinite number of joint
rotation patterns.
“I can write the letter A with my hand, with my foot, or even with my mouth;
I could even make an A by walking on the beach.” [Berthoz 1997]
Subjects wrote their signature with their dominant index finger and
ipsilateral big toe. fMRI showed that movement parameters for this
movement are stored in secondary sensorimotor cortices of the dominant
hand.
These areas can be accessed by the foot and are therefore functionally
independent from the primary representation of the effector. [Rijntjies et al.
1999]
SIGNATURE
SIGNATURE
IMAGE
MOTOR PROGRAM
MOTOR CORTEX
MOTOR EQUIVALENCE
MUSCLES OF HAND
AND ARM
MUSCLES OF BODY
WALKING ON BEACH
MUSCLES OF LEG
AND FOOT
The present results demonstrate the existence of motor equivalence in a
combined upper and lower extremity task. [Marteniuk et al. 2000]
Dynamic cortical activity in the human brain reveals motor
equivalence. [Kelso et al. 1998]
LANGUAGE AND SPEECH AS MOTOR
ACTIVITIES
Language is also a mode of action. Language and speech depend
basically upon complex serial neuronal events (action programs).
A strict relation exists between language and motor control. [Gentilucci]
The results show the same type of alterations of the temporal
organisation of speech as those characteristic for rapid alternating
limb movements.
They support the view that the speech and skeletomuscular
systems share common neural control modes despite fundamental
biomechanical differences. [Volkmann et al. 1992]
Comparing findings on the motor organisation of speech with the
organization of voluntary movements about the elbow, we have found
that the kinematic patterns for movements of the tongue dorsum were
similar to those of voluntary flexion-extension movements about the
elbow. [Ostry and Cooke 1987]
The task dynamic model we are using for speech was exactly the model
used for controlling arm movements, with the articulators
of the vocal tract simply substituted for those of the arm.
Such gestures not only can characterize the movements of the speech
articulators but also can act as phonological primitives. [Browman and
Goldstein 1991]
The motor control of the tongue is a section of the total system of motor
control for all bodily movement (including arm and facial movement)
So what is the nature of the tongue and other articulatory components?
The tongue manipulates food, dilates the airway during inspiration,
and shapes the sounds of speech. While performing these functions
the tongue morphs through many complex shapes.
Tongue and other articulatory elements
TONGUE MUSCLES AND NEURAL CONTROL
The tongue: a large collection of muscles covered by mucous
membrane.
Extrinsic muscles move the tongue about in the oral cavity; intrinsic
muscles change the shape of the tongue. 4 extrinsic muscles:
genioglossus pulls the tongue forward or protrudes it; hyoglossus pulls
the tongue down and posterior or retracts and depresses; styloglossus
retracts and elevates the tongue.
Intrinsic muscles: longitudinis linguae shapes the tongue for speech and
mastication; transversus linguae compresses the sides of the tongue;
verticalis linguae with fibres running suproinferiorly.
Tongue movements can be explained in terms of a small number
of independent muscle groups, each corresponding to an
elementary or ‘primitive’ movement, making use of a relatively
small number of invariant muscle synergies. [Sanguineti 1997]
In dogs, the intrinsic muscles are composed of many
neuromuscular compartments. The superior and inferior
longitudinal muscles each had an average of 40 distinct muscle
fascicles, each supplied by a nerve branch. Each of the transverse
and vertical muscles is composed of over 140 separate muscles
sheets and every sheet is innervated by a separate terminal
nerve.[Mu & Sanders 1999]
[The human tongue is not likely to be less complicated in its
muscles and neural control than the tongue of the dog.]
If motor primitives are necessary to avoid the degrees of freedom
combinatorial explosion for movements of the arm or leg, it seems
inevitable that they are even more acutely required for the overwhelming
complexity of the the tongue musculature and the unlimited possibilities
of movement and change of shape the tongue can assume in speech.
MOTOR EQUIVALENCE IN RELATION TO SPEECH
A as in HAT
SPEECHSOUND IMAGE
MOTOR PROGRAM
MOTOR CORTEX
MOTOR EQUIVALENCE
ARTICULATORY
MUSCLES
ARM MUSCLES
EQUIVALENCE
OF
SPEECH AND MOTOR
ELEMENTS
Motor equivalence is demonstrated most remarkably in the relation
between speech and gesture.
Motor equivalence can operate from speech to gesture or from gesture
to speech.
It also seems likely that it can operate between other modalities and
speech - or more precisely between motor programs for other
modalities and motor programs for speech.
Speech and gesture arise as interacting elements of a single system.
[McNeill 1987]
Every articulatory program can be redirected (through motor
equivalence) to produce an equivalent movement of the hand and
arm.
Every gesture structured by a perceived object or action can be
redirected to produce an equivalent articulatory action.
SPEECH
VISION
MOTOR EQUIVALENCE
HEARING
GESTURE
IMITATION
Motor equivalence can function between speech and perception and
between perception and motor action because perception is also a motor
activity. There are motor programs and motor primitives for vision.
These can be transferred by motor equivalence to other muscle/joint
assemblies e.g. imitating the shape of a scanned object or a perceived
movement by a gesture.
Vision is a highly motoric activity >>>>>
NOTON AND STARK
Eye Movements and Visual Perception 1971
EYE MOVEMENTS of subject viewing photograph of a bust of Queen Nefertiti [from
Yarbus]
Imitation is of central importance in the relation between gesture
and speech, that is between bodily gesture and articulatory
gesture.
The capacity to transfer visual patterning from what is seen to
corresponding bodily action is innate.
Newly-born infants can imitate what they see >>>>
MELTZOFF
Imitation based on the neonate's capacity to represent visually and proprioceptively
perceived information in a form common to both modalities. Observations in six
newborns- one only 60 minutes old - suggest that the ability to use intermodal
equivalences is innate [Meltzoff and Moore 1977]
From the motor processes associated with perception can be
constructed articulatory gestures - word-programs related to what is
perceived
Making specific the relation
between what has been said about
motor control generally and speech
and language
MOTOR CONTROL AND THE MOTOR THEORY OF LANGUAGE
Mirror neurons, motor primitives and motor imagery research fit
closely with the motor theory of language origin and function.
The motor theory is that there is a direct relation between the
functioning of speech and motor control generally, with language
depending on pre-existing motor primitives coupled with the operation
of motor equivalence.
The motor control of the tongue is a section of the total system of
motor control for all bodily movement. Tongue movements can be
explained in terms of a small number of independent muscle
groups, each corresponding to an elementary or ‘primitive’
movement
Motor equivalence is demonstrated most remarkably in the relation
between speech and gesture. Motor equivalence can operate from
speech to gesture or from gesture to speech and from vision to
speech
Arm movements - associated gesture - result from transferring the
motor image of the speech-sound primitives or the motor
programs for words to the arm by motor equivalence, in the same
way as the motor program for writing one’s signature can be
transferred by motor equivalence to writing one’s signature with
one’s foot.
Individual speech sounds are MOTOR PRIMITIVES
Words formed from primitive speech-sound elements are MOTOR
PROGRAMS
Movements of the arm are the MOTOR EQUIVALENTS of the speech
primitives
Before we produce a sentence there is a MOTOR IMAGE of the sentence
A sentence is a high-level motor program or ACTION PLAN
E
N
V
I
R
O
N
M
E
N
T
I
M
A
G
E
HE HIT ME
MOTOR IMAGE
EVENT IMAGE
HE
SENTENCE
MOTORHIT
PROGRAM
PROGRAM
ME
SPEECH
MOTOR CONTROL
UTTERANCE!
DISINHIBIT!
MOTOR PRIMITIVES
SPEECHSOUND
EHIMT
PRIMITIVES
ME
MOTOR
E HHIT
IROUTINES
MT
WORDHE
SHAPES
A
R
T
I
C
U
L
A
T
O
R
Y
M
U
S
C
L
E
S
B
O
D
Y
I
M
A
G
E
Words and equivalent gestures