Download Gesture-Speech Mismatches: Considerations for How

Gesture-Speech Mismatches: Considerations for How Language Is Encoded & Decoded
Natasha Gibson-Livits, B. A., Monica Gordon-Pershey Ed.D., CCC-SLP
ASHA, San Diego, November 19, 2011
Abstract
Do gestures enhance the meaning conveyed by words, or do gestures supply unique
meaning? This literature review explores this question by focusing on gesturespeech mismatches. Mismatches occur when gesture and speech convey different
messages, suggesting that gesture supplies unique information. Gesture reveals
cognitive processes at work during verbal encoding and decoding.
1
Introduction
Ninety percent of human gesture is found in the context of spoken language (McNeill, 2005).
Because gesture seems to be a communicative phenomenon, researchers have posed many
questions, including Why do people gesture when they speak? Does gesture help speakers
organize their thoughts and express themselves better? Do gesture and spoken language arise
from the associated cognitive-linguistics processes, or from different processes? How do
listeners comprehend gestures?
While the study of gesture is fairly well developed in disciplines such as neuroscience,
psychology, anthropology, and linguistics, it is largely overlooked in communication sciences
and disorders. There is a lack of research focusing specifically on gesture as it relates to the
etiologies, diagnosis, and treatment of language impairments. SLPs need a better theoretical
understanding of the role gesture plays in encoding and decoding information in order to
apply this knowledge to clinical diagnosis and treatment.
The goal of this session is to provide an understanding of how gesture is encoded by a speaker
and decoded by a listener. We will discuss the major theories that describe how gesture
facilitates the production and comprehension of spoken language. This theoretical
understanding may suggest how SLPs can observe a client’s competence in gestural encoding
and decoding and use this information diagnostically and therapeutically.
2
Gestural Expression
Researchers have asked whether gestures enhance the meaning conveyed by words or whether
gestures supply their own meaning. Several linguistic studies of gesture indicate that gesture
and spoken language are derived from the associated cognitive processes (Kita, 2000).
Production of verbal and gestural messages requires organized thought. One reason that
people may gesture as they speak is that gesture may help speakers organize their thoughts.
Some investigators suspect that gesture produces an image-activating affect which helps
speakers visualize their thoughts for better clarification (Kita, 2000). Even so, this premise
does not reveal whether gestures and speech co-occur as one act or whether they are
independent actions that a speaker must closely coordinate in order for the visual and verbal
messages to be related in meaning.
In investigating how gesture and speech inter-relate, researchers are primarily concerned with
whether gestures enhance the meaning conveyed by words or whether gestures supply their
own meaning. It is generally thought that a speaker gestures to enhance a verbal message.
Gesture may “match” the message that is spoken, in that gesture may convey a similar or
supplemental meaning. When gesture and speech match, it is hard for investigators to
determine whether the gesture is part of the cognitive and linguistic activity needed to
produce the spoken message or whether the gesture is its own cognitive and linguistic act.
3
Gestural Expression
To attempt to separate gesture and speech, researchers have looked at instances where gesture
and speech do not match. Gesture-speech “mismatches” occur when gesture and speech
convey two different messages, suggesting that gesture supplies distinct information that
encodes meaning separately or differently from speech (Goldin-Meadow, 2003; McNeil,
2005). Gesture-speech mismatches are more observable phenomena and lend themselves to
research study. People gesture when they speak in either conscious or unconscious effort to
supplement or to contradict verbal meaning.
Mismatches indicate that there are periods of mental transition where the speaker has
difficulty retrieving the lexicon (words) that matches the meaning conveyed by gesture.
However, the explanation for gesture-speech mismatches is not that simple. Although
theorists concur that speakers gesture during periods of verbal difficulty, studies show that
speakers gesture just as frequently when there is no difficulty speaking (Kita 2000; Mayberry
& Jaques, 2000; McNeil, 2000; 2005).
Kita (2000) proposes that gesture helps organize the cognitive-linguistic processes necessary
for speech. Analytic thinking (needed for cognition, language, and speech) and spatio-motoric
thinking (needed for gesture) work together to organize information into a series of
“packages” that are produced. The image-boosting properties of gesture operate together with
analytic thinking until a match is achieved.
4
Gestural Expression
Gesture-speech mismatch reveals the transition phase when spatio-motoric thinking
(used to produce gesture) is organizing information differently than analytic
thinking (used to produce speech). Because analytic thinking is also responsible for
self-monitoring, the speaker detects the inadequacy of the verbal message being
spoken. This triggers the spatio-motoric thinking needed to continue searching for a
more suitable representational gestural image. Kita (2000) observes that this
process would explain how, during a gesture-speech mismatch, gesture is
reorganized and modified to match speech. According to Kita, speech is seldom
modified to match gesture.
5
Gestural Comprehension
Gesture plays two roles in comprehension. First, it affects how listeners decode information,
and second, it affects memory recall. For example, someone might say, “I carried the bag all
day,” while leaning over to indicate how heavy the bag was and how tired she was carrying it.
The gesture is conveying a lot more information than the verbal message alone, and the
gestural input affects how the listener perceives and comprehends the message. Later, to recall
the message, the listener is likely to remember what the gesture conveyed as well as the
verbal message.
Goldin-Meadow (2003; 2004) proposes that gesture helps listeners process incoming
messages in working memory. Working memory receives support primarily when an action
conveyed by gesture matches the action stated in the verbal message. When speech and
gesture match they reinforce the same message, making the message easier to remember.
However, when the action evoked by gesture conflicts with the action stated in the verbal
message, there are two conflicting messages. Wagner, Nusbaum, and Goldin-Meadow (2004)
contend that gesture-speech mismatches increase the burden on working memory, making it
harder for a listener to recall the verbal message.
6
Implications
Studies of gesture-speech mismatches have important implications for diagnosing
and treating persons with communication disorders. In regards to expressive
communication, do speakers with expressive language impairments produce
gesture-speech mismatches? If so, how, when, and why? Do atypical gestures
reveal a language impairment? Regarding receptive communication, how is the
comprehension of gesture disturbed by deficits in receptive language, memory,
analytic thinking, and spatio-motoric thinking?
7
TYPES OF SPONTANEOUS GESTURES: DEFINITIONS
Iconic: By the form of the gesture, iconic
gestures depict some feature or event
being described. These gestures specify
the manner in which an action is carried
out. Iconic gestures can be pantomimes
that show who a speaker imagines
himself to be.
Metaphoric: Metaphoric gestures
illustrate a concept that has no physical
form. These gestures are used to illustrate
the meaning of an abstract concept.
Deictic: Deictic gestures are used to
indicate physical space between the
speaker and listener. These gestures are
used to indicate points of reference and
spatial proximity.
Beat: Beat gestures are sudden, baton
like movements which are accompanied
by speech repairs. Beats can also be used
for prosodic emphasis.
8
WHY DO PEOPLE GESTURE WHEN THEY SPEAK: THEORIES ON
GESTURE PRODUCTION
CAN THEORY EXPLAIN WHY GESTURE AND SPEECH SOMETIMES
MISMATCH?
Independent Systems
(Butterworth & Beattie, 1978)
Gesture functions as a back up or auxiliary system for the temporary absence or failure of speech (speech breakdown). Mismatch
occurs when speech breaks down and there is only gesture.
Image Activation/Lexical
(Krauss et al., 2000)
Representational gestures help to maintain an image mentally while a lexical connection is made with the image. Lexical delay
causes mismatch of gesture and speech.
Growth Point
(McNeill, 1992)
Speech and gesture combine to create one meaning. Gestures are found to co-occur with a speaker's intonation and pitch so that
gesture is synchronous with speech. During periods of speech interruption, gestures are found to halt and slow until normal speech
fluency resumes. Mismatches arise from loss of speech fluency.
Information Packaging
(Kita, 2000)
Analytic thinking (being speech) and spatio-motoric thinking (being gesture) work together towards the goal of organizing
information into a series of packages that can be verbalized. The self monitoring aspect of analytic thinking can detect if there is an
inadequacy of logic, which triggers spatio-motoric thinking (gesture) to continue searching for a more suitable representation of
the verbal message. This causes the mismatch.
9
DO GESTURES HELP ORGANIZE THOUGHTS?
WHAT GESTURE-SPEECH MISMATCH REVEALS
Gesture-Speech Mismatch: Occurs when the action conveyed
in gesture does not match the action in speech (GoldinMeadow, 2003.)
This is a mismatch because the gesture conveys the action of
rolling while the action in the verbal message is to go down a
set of stairs.
“I went down the stairs”
Gesture-Speech Match: Occurs when the action conveyed in
gesture matches the action in speech (Goldin-Meadow, 2003.)
This is a match because the hands are held up to indicate a
stopping motion which matches the action in speech.
“I came to a complete stop.”
Studies of mismatched gestures show that
gesture is a supplier of information that has
meaning separate from the speech it
accompanies. Gesture tells us how an action
was carried out, showing speed, spatial
proximity and other points of reference that
the verbal message does not convey.
Mismatches give us a window into the
cognitive process of how we decode
information to produce verbal expression.
The Information Packaging Hypothesis best
explains how mismatches arise by attributing
mismatches to a thought that is not fully
developed -- the gesture cannot appropriately
integrate with the thought. The self monitoring
aspect of language compels us to modify our
gestures and choice of words until a
satisfactory match between the two modalities
is achieved.
10
GESTURE DEVELOPS IN TANDEM WITH LANGUAGE MILESTONES
Symbolic gestures first appear
with one word stage
9 mos.
12-18 mos
Deictic gestures emerge with
intentional communication.
(pointing, grabbing, eye gaze)
18-24 mos
Symbolic gesture continues to
develop with two word
combinations
24-36 mos
Gesture+speech
combinatinos first appear: ex:
point at cup+"Mommy"
36-42 mos
Beat gestures start to
emerge after 3 years when
major linguistic milestones
have been reached.
11
IMPLICATIONS FOR DIAGNOSIS AND TREATMENT
• Gesture may have an image activating affect which helps a speaker visualize thoughts for better expression.
• Gesture-speech mismatches indicate that there are periods of mental transition where the speaker has
difficulty retrieving the lexicon (words) that matches the meaning conveyed by gesture. Gesture-speech
mismatches provide a window into the cognitive processes at work during language production. These
cognitive processes are not as easily observed when gesture and speech match. The Information Packaging
Hypothesis is presently undisputed in addressing how mismatches arise and best explains how gesture cooccurs with verbalization. This hypothesis proposes that mismatches result when a speaker holds in mind two
ideas that have not yet been integrated into a unified thought.
• Several studies demonstrate the affect of gesture-speech mismatches and matches on processing and memory.
Gesture use may mitigate a learner’s confusion and help learners clarify information. Studies of school
children show there is a need for better awareness of gesture’s role in the learning process.
• Developmentally, gestures co-occur with major linguistic milestones. Some gestures, such as pointing, often
precede the use of words. Interestingly, children who use gestures in the early verbal period naturally produce
mismatched gesture and speech before vocabulary has a chance to catch up. This may imply that gesturing
can be seen as a precursor to later vocabulary development. Gesture activities may be explored as a technique
to stimulate lexical development. Perhaps, by implication, the potential for a plateau in language
development might be anticipated by paying attention to lags in gestural development. Relatedly, it is worth
consideration whether children with suspected autism spectrum disorders exhibit any developmental
differences in intentional gesturing.
• Lastly, neurobiological studies of gesture provide evidence that the same neural areas that process language
also process symbolic gestures. This knowledge may advance treatment of disorders that affect cognitivelinguistic skills, learning, and memory, such as aphasia, dementias, language-based learning disabilities, and
autism spectrum disorders.
12
References
(Partial Listing)
Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge,
MA: The Belknap Press of Harvard University.
Goldin-Meadow, S. (2004). Gesture’s role in the learning process. Theory into Practice,
43(4), 314-321.
Kita, S. (2000). How representational gestures help speaking. Cambridge, UK:
Cambridge University Press.
Mayberry, R., & Jaques, J. (2000). Gesture during stuttered speech: Insights into the
nature of gesture-speech integration. In D. McNeill (Ed.), Language and gesture
(pp. 199-214). Cambridge, UK: Cambridge University Press.
McNeill, D., & Duncan, S. D. (2000). Growth points in thinking for speaking. In D.
McNeill (Ed.), Language and gesture (pp. 141-161). Cambridge, UK: Cambridge
University Press.
McNeill, D. (2005). Gesture and thought. Chicago, IL: University of Chicago Press.
Wagner, S.M., Nusbaum, H., & Goldin-Meadow, S. (2004) Probing the mental
representation of gesture: Is hand waving spatial? Journal of Memory and
Language, 50, 395-407.
13