Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Gesture-Speech Mismatches: Considerations for How Language Is Encoded & Decoded Natasha Gibson-Livits, B. A., Monica Gordon-Pershey Ed.D., CCC-SLP ASHA, San Diego, November 19, 2011 Abstract Do gestures enhance the meaning conveyed by words, or do gestures supply unique meaning? This literature review explores this question by focusing on gesturespeech mismatches. Mismatches occur when gesture and speech convey different messages, suggesting that gesture supplies unique information. Gesture reveals cognitive processes at work during verbal encoding and decoding. 1 Introduction Ninety percent of human gesture is found in the context of spoken language (McNeill, 2005). Because gesture seems to be a communicative phenomenon, researchers have posed many questions, including Why do people gesture when they speak? Does gesture help speakers organize their thoughts and express themselves better? Do gesture and spoken language arise from the associated cognitive-linguistics processes, or from different processes? How do listeners comprehend gestures? While the study of gesture is fairly well developed in disciplines such as neuroscience, psychology, anthropology, and linguistics, it is largely overlooked in communication sciences and disorders. There is a lack of research focusing specifically on gesture as it relates to the etiologies, diagnosis, and treatment of language impairments. SLPs need a better theoretical understanding of the role gesture plays in encoding and decoding information in order to apply this knowledge to clinical diagnosis and treatment. The goal of this session is to provide an understanding of how gesture is encoded by a speaker and decoded by a listener. We will discuss the major theories that describe how gesture facilitates the production and comprehension of spoken language. This theoretical understanding may suggest how SLPs can observe a client’s competence in gestural encoding and decoding and use this information diagnostically and therapeutically. 2 Gestural Expression Researchers have asked whether gestures enhance the meaning conveyed by words or whether gestures supply their own meaning. Several linguistic studies of gesture indicate that gesture and spoken language are derived from the associated cognitive processes (Kita, 2000). Production of verbal and gestural messages requires organized thought. One reason that people may gesture as they speak is that gesture may help speakers organize their thoughts. Some investigators suspect that gesture produces an image-activating affect which helps speakers visualize their thoughts for better clarification (Kita, 2000). Even so, this premise does not reveal whether gestures and speech co-occur as one act or whether they are independent actions that a speaker must closely coordinate in order for the visual and verbal messages to be related in meaning. In investigating how gesture and speech inter-relate, researchers are primarily concerned with whether gestures enhance the meaning conveyed by words or whether gestures supply their own meaning. It is generally thought that a speaker gestures to enhance a verbal message. Gesture may “match” the message that is spoken, in that gesture may convey a similar or supplemental meaning. When gesture and speech match, it is hard for investigators to determine whether the gesture is part of the cognitive and linguistic activity needed to produce the spoken message or whether the gesture is its own cognitive and linguistic act. 3 Gestural Expression To attempt to separate gesture and speech, researchers have looked at instances where gesture and speech do not match. Gesture-speech “mismatches” occur when gesture and speech convey two different messages, suggesting that gesture supplies distinct information that encodes meaning separately or differently from speech (Goldin-Meadow, 2003; McNeil, 2005). Gesture-speech mismatches are more observable phenomena and lend themselves to research study. People gesture when they speak in either conscious or unconscious effort to supplement or to contradict verbal meaning. Mismatches indicate that there are periods of mental transition where the speaker has difficulty retrieving the lexicon (words) that matches the meaning conveyed by gesture. However, the explanation for gesture-speech mismatches is not that simple. Although theorists concur that speakers gesture during periods of verbal difficulty, studies show that speakers gesture just as frequently when there is no difficulty speaking (Kita 2000; Mayberry & Jaques, 2000; McNeil, 2000; 2005). Kita (2000) proposes that gesture helps organize the cognitive-linguistic processes necessary for speech. Analytic thinking (needed for cognition, language, and speech) and spatio-motoric thinking (needed for gesture) work together to organize information into a series of “packages” that are produced. The image-boosting properties of gesture operate together with analytic thinking until a match is achieved. 4 Gestural Expression Gesture-speech mismatch reveals the transition phase when spatio-motoric thinking (used to produce gesture) is organizing information differently than analytic thinking (used to produce speech). Because analytic thinking is also responsible for self-monitoring, the speaker detects the inadequacy of the verbal message being spoken. This triggers the spatio-motoric thinking needed to continue searching for a more suitable representational gestural image. Kita (2000) observes that this process would explain how, during a gesture-speech mismatch, gesture is reorganized and modified to match speech. According to Kita, speech is seldom modified to match gesture. 5 Gestural Comprehension Gesture plays two roles in comprehension. First, it affects how listeners decode information, and second, it affects memory recall. For example, someone might say, “I carried the bag all day,” while leaning over to indicate how heavy the bag was and how tired she was carrying it. The gesture is conveying a lot more information than the verbal message alone, and the gestural input affects how the listener perceives and comprehends the message. Later, to recall the message, the listener is likely to remember what the gesture conveyed as well as the verbal message. Goldin-Meadow (2003; 2004) proposes that gesture helps listeners process incoming messages in working memory. Working memory receives support primarily when an action conveyed by gesture matches the action stated in the verbal message. When speech and gesture match they reinforce the same message, making the message easier to remember. However, when the action evoked by gesture conflicts with the action stated in the verbal message, there are two conflicting messages. Wagner, Nusbaum, and Goldin-Meadow (2004) contend that gesture-speech mismatches increase the burden on working memory, making it harder for a listener to recall the verbal message. 6 Implications Studies of gesture-speech mismatches have important implications for diagnosing and treating persons with communication disorders. In regards to expressive communication, do speakers with expressive language impairments produce gesture-speech mismatches? If so, how, when, and why? Do atypical gestures reveal a language impairment? Regarding receptive communication, how is the comprehension of gesture disturbed by deficits in receptive language, memory, analytic thinking, and spatio-motoric thinking? 7 TYPES OF SPONTANEOUS GESTURES: DEFINITIONS Iconic: By the form of the gesture, iconic gestures depict some feature or event being described. These gestures specify the manner in which an action is carried out. Iconic gestures can be pantomimes that show who a speaker imagines himself to be. Metaphoric: Metaphoric gestures illustrate a concept that has no physical form. These gestures are used to illustrate the meaning of an abstract concept. Deictic: Deictic gestures are used to indicate physical space between the speaker and listener. These gestures are used to indicate points of reference and spatial proximity. Beat: Beat gestures are sudden, baton like movements which are accompanied by speech repairs. Beats can also be used for prosodic emphasis. 8 WHY DO PEOPLE GESTURE WHEN THEY SPEAK: THEORIES ON GESTURE PRODUCTION CAN THEORY EXPLAIN WHY GESTURE AND SPEECH SOMETIMES MISMATCH? Independent Systems (Butterworth & Beattie, 1978) Gesture functions as a back up or auxiliary system for the temporary absence or failure of speech (speech breakdown). Mismatch occurs when speech breaks down and there is only gesture. Image Activation/Lexical (Krauss et al., 2000) Representational gestures help to maintain an image mentally while a lexical connection is made with the image. Lexical delay causes mismatch of gesture and speech. Growth Point (McNeill, 1992) Speech and gesture combine to create one meaning. Gestures are found to co-occur with a speaker's intonation and pitch so that gesture is synchronous with speech. During periods of speech interruption, gestures are found to halt and slow until normal speech fluency resumes. Mismatches arise from loss of speech fluency. Information Packaging (Kita, 2000) Analytic thinking (being speech) and spatio-motoric thinking (being gesture) work together towards the goal of organizing information into a series of packages that can be verbalized. The self monitoring aspect of analytic thinking can detect if there is an inadequacy of logic, which triggers spatio-motoric thinking (gesture) to continue searching for a more suitable representation of the verbal message. This causes the mismatch. 9 DO GESTURES HELP ORGANIZE THOUGHTS? WHAT GESTURE-SPEECH MISMATCH REVEALS Gesture-Speech Mismatch: Occurs when the action conveyed in gesture does not match the action in speech (GoldinMeadow, 2003.) This is a mismatch because the gesture conveys the action of rolling while the action in the verbal message is to go down a set of stairs. “I went down the stairs” Gesture-Speech Match: Occurs when the action conveyed in gesture matches the action in speech (Goldin-Meadow, 2003.) This is a match because the hands are held up to indicate a stopping motion which matches the action in speech. “I came to a complete stop.” Studies of mismatched gestures show that gesture is a supplier of information that has meaning separate from the speech it accompanies. Gesture tells us how an action was carried out, showing speed, spatial proximity and other points of reference that the verbal message does not convey. Mismatches give us a window into the cognitive process of how we decode information to produce verbal expression. The Information Packaging Hypothesis best explains how mismatches arise by attributing mismatches to a thought that is not fully developed -- the gesture cannot appropriately integrate with the thought. The self monitoring aspect of language compels us to modify our gestures and choice of words until a satisfactory match between the two modalities is achieved. 10 GESTURE DEVELOPS IN TANDEM WITH LANGUAGE MILESTONES Symbolic gestures first appear with one word stage 9 mos. 12-18 mos Deictic gestures emerge with intentional communication. (pointing, grabbing, eye gaze) 18-24 mos Symbolic gesture continues to develop with two word combinations 24-36 mos Gesture+speech combinatinos first appear: ex: point at cup+"Mommy" 36-42 mos Beat gestures start to emerge after 3 years when major linguistic milestones have been reached. 11 IMPLICATIONS FOR DIAGNOSIS AND TREATMENT • Gesture may have an image activating affect which helps a speaker visualize thoughts for better expression. • Gesture-speech mismatches indicate that there are periods of mental transition where the speaker has difficulty retrieving the lexicon (words) that matches the meaning conveyed by gesture. Gesture-speech mismatches provide a window into the cognitive processes at work during language production. These cognitive processes are not as easily observed when gesture and speech match. The Information Packaging Hypothesis is presently undisputed in addressing how mismatches arise and best explains how gesture cooccurs with verbalization. This hypothesis proposes that mismatches result when a speaker holds in mind two ideas that have not yet been integrated into a unified thought. • Several studies demonstrate the affect of gesture-speech mismatches and matches on processing and memory. Gesture use may mitigate a learner’s confusion and help learners clarify information. Studies of school children show there is a need for better awareness of gesture’s role in the learning process. • Developmentally, gestures co-occur with major linguistic milestones. Some gestures, such as pointing, often precede the use of words. Interestingly, children who use gestures in the early verbal period naturally produce mismatched gesture and speech before vocabulary has a chance to catch up. This may imply that gesturing can be seen as a precursor to later vocabulary development. Gesture activities may be explored as a technique to stimulate lexical development. Perhaps, by implication, the potential for a plateau in language development might be anticipated by paying attention to lags in gestural development. Relatedly, it is worth consideration whether children with suspected autism spectrum disorders exhibit any developmental differences in intentional gesturing. • Lastly, neurobiological studies of gesture provide evidence that the same neural areas that process language also process symbolic gestures. This knowledge may advance treatment of disorders that affect cognitivelinguistic skills, learning, and memory, such as aphasia, dementias, language-based learning disabilities, and autism spectrum disorders. 12 References (Partial Listing) Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge, MA: The Belknap Press of Harvard University. Goldin-Meadow, S. (2004). Gesture’s role in the learning process. Theory into Practice, 43(4), 314-321. Kita, S. (2000). How representational gestures help speaking. Cambridge, UK: Cambridge University Press. Mayberry, R., & Jaques, J. (2000). Gesture during stuttered speech: Insights into the nature of gesture-speech integration. In D. McNeill (Ed.), Language and gesture (pp. 199-214). Cambridge, UK: Cambridge University Press. McNeill, D., & Duncan, S. D. (2000). Growth points in thinking for speaking. In D. McNeill (Ed.), Language and gesture (pp. 141-161). Cambridge, UK: Cambridge University Press. McNeill, D. (2005). Gesture and thought. Chicago, IL: University of Chicago Press. Wagner, S.M., Nusbaum, H., & Goldin-Meadow, S. (2004) Probing the mental representation of gesture: Is hand waving spatial? Journal of Memory and Language, 50, 395-407. 13