Download presentation

16th International Summer School in Cognitive Science, New Bulgarian University, Sofia, Bulgaria, 2009
The Embodied Cognition Literature
…cont’d
Michael J. Spivey
Department of Cognitive Science
University of California, Merced
Sensorimotor Influences in Cognition
• Visual Properties Influence Concepts
• Motor Properties Influence Vision
• Language Influences Visual Properties
• Language and Motor Influence Each Other
• Your Entire Body Participates in Thought
Sensorimotor Influences in Cognition
QuickTime™ and a
decompressor
are needed to see this picture.
Scheerer & Lyons (1957)
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory
Sensorimotor Influences in Cognition
• Visual Properties Influence Concepts
• Motor Properties Influence Vision
• Language Influences Visual Properties
• Language and Motor Influence Each Other
• Your Entire Body Participates in Thought
Barsalou, Solomon, & Wu (1999)
List the features
of a watermelon
Large
Green
Sweet
List the features
of half a watermelon
Red
Sweet
Seeds
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory
Sensorimotor Influences in Cognition
• Visual Properties Influence Concepts
• Motor Properties Influence Vision
• Language Influences Visual Properties
• Language and Motor Influence Each Other
• Your Entire Body Participates in Thought
Stimulus-Response Compatibility
Tucker & Ellis (1998)
Stimulus-Response Compatibility
Tucker & Ellis (1998)
Stimulus-Response Compatibility
Tucker & Ellis (1998)
Stimulus-Response Compatibility
Tucker & Ellis (1998)
The Importance of Being
Somatosensory Input
Bosbach, Cole, Prinz, Knoblich (2005)
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory
Sensorimotor Influences in Cognition
• Visual Properties Influence Concepts
• Motor Properties Influence Vision
• Language Influences Visual Properties
• Language and Motor Influence Each Other
• Your Entire Body Participates in Thought
Mental Simulations
Stanfield & Zwaan (2001)
John put the pencil in the cup.
John put the pencil in the drawer.
QuickTime™ and a
decompressor
are needed to see this picture.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
Mental Simulations
Zwaan, Stanfield & Yaxley (2001)
The ranger saw the eagle in the sky.
The ranger saw the eagle in the nest.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
QuickTime™ and a
decompressor
are needed to see this picture.
Mental Simulations
Zwaan, Madden, Yaxley, & Aveyard (2004)
The pitcher hurled the softball at you.
You hurled the softball at the pitcher.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
Mental Simulations
Meteyard, Bahrami, & Vigliocco (2007)
Downward:
“bomb, drop, fall…”
QuickTime™ and a
Video decompressor
are needed to see this picture.
Control:
“cook, sew, write…”
Upward:
“climb, rise, fly…”
QuickTime™ and a
Video decompressor
are needed to see this picture.
Mental Simulations
Meteyard, Bahrami, & Vigliocco (2007)
Downward:
“bomb, drop, fall…”
QuickTime™ and a
Video decompressor
are needed to see this picture.
Control:
“cook, sew, write…”
Upward:
“climb, rise, fly…”
QuickTime™ and a
Video decompressor
are needed to see this picture.
Mental Simulations
Meteyard, Bahrami, & Vigliocco (2007)
Downward:
“bomb, drop, fall…”
QuickTime™ and a
Video decompressor
are needed to see this picture.
Control:
“cook, sew, write…”
Upward:
“climb, rise, fly…”
QuickTime™ and a
Video decompressor
are needed to see this picture.
Verb Image Schemas
• An intimate relationship between language and space
– Language is embodied and metaphorical (Gibbs, 2006;
Lakoff, 1999)
– Spatial registration hypothesis (Chatterjee, 2001;
Coslett, 1999)
– Language is not describable without reference to
cognitive processes such as attentional mechanisms
(Langacker, 1991; Talmy, 1983)
– Perceptual symbol systems: cognitive representations
are schematics, or simulations of perceptual
experiences (Barsalou, 1999)
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Offline Data
Richardson, Spivey, Barsalou, & McRae (2003)
• Forced Choice Norming Study
Verb Image Schemas
Richardson, Spivey, Barsalou, & McRae (2003)
• Free Drawing Norming Study
argued with
respected
Verb Image Schemas
• Did subjects agree with each other?
– 70% agreement in forced choice; Std errors about 10˚ in freeform.
• Did the tasks tap the same representation?
– Axis/aspect angle correlation, R = 0.71
• Did they agree with linguistic intuitions?
– Significant effect of expected axis in both studies
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Online Data
• Experiment 1 - Visual discrimination
– Hypothesis: comprehending a verb will activate a form
of spatial representation that will be extended along
either a horizontal or vertical axis, as shown by
norming data.
– Non-compatible mental imagery will interfere with the
detection or discrimination of overlapping visual
stimuli (Perky, 1910; Craver & Arterberry, 2001)
– Participants presented with a spoken sentence
containing a horizontal or vertical verb, and identified a
rapidly flashed visual stimulus.
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Online Data
• Performance on the visual task was impaired when the
visual stimulus appeared along the axis of the verb’s image
schema
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Online Data
• Experiment 2 - Memory
• Hypothesis: the spatial structure associated with a verb will influence
the encoding of visual stimuli
• Imagery improves memory (Paivio, 1969), and visual stimuli are
remembered better when they are presented in the same spatial
locations at presentation and test (Santa, 1977; Zimmer 1998)
• Participants heard a sentence and saw pictures of the agent and patient
of the sentence, presented centrally. At a later test stage, they were
shown pairs of pictures and asked if they had been previously been
presented together. At test, the pictures were presented in a horizontal
or vertical alignment.
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Online Data
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Online Data
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas - Online Data
• At test, picture pairs were recognized faster if they were
presented in the same orientation as the associated verb’s
image schema
Richardson, Spivey, Barsalou, & McRae (2003)
Verb Image Schemas
• In two offline tasks, naïve participants categorized verbs as having a
horizontal or vertical image schema
• There was a good degree of consistency between tasks and between
participants
• The offline tasks provided norms for experiments involving online
verb comprehension
• In both a visual discrimination and a memory task, reaction times
showed an interaction between the horizontal/vertical nature of the
verb’s image schema, and the horizontal/vertical position of the visual
stimuli
• Such spatial effects of verb comprehension provide evidence for the
perceptual-motor character of linguistic representations (see also
Bergen, Lindsay, Matlock & Narayanan, 2007).
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory
Iconicity Affects Semantic Relatedness Judgments
(Zwaan & Yaxley, 2003)
attic
basement
Iconicity Affects Semantic Relatedness Judgments
(Zwaan & Yaxley, 2003)
basement
attic
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory
Sensorimotor Influences in Cognition
• Visual Properties Influence Concepts
• Motor Properties Influence Vision
• Language Influences Visual Properties
• Language and Motor Influence Each Other
• Your Entire Body Participates in Thought
(Spivey, Tanenhaus, Eberhard, & Sedivy, 2002)
Ambiguous: “Put the apple on the towel in the box”
Unambiguous: “Put the apple that’s on the towel in the box”
Motor Constraints During Language
(Chambers, Tanenhaus, & Magnuson, 2004)
Pour the egg in the bowl onto the flour.
Pour the egg that’s in the bowl onto the flour.
Compatible
Incompatible
Motor Constraints During Language
(Chambers, Tanenhaus, & Magnuson, 2004)
Motor Facilitation During Language
The Action-Sentence Compatibility Effect
(Glenberg & Kaschak, 2002)
QuickTime™ and a
decompressor
are needed to see this picture.
Motor Facilitation During Language
The Action-Sentence Compatibility Effect
(Glenberg & Kaschak, 2002)
Close the drawer.
Open the drawer.
QuickTime™ and a
d eco mpres sor
are nee ded to s ee this picture.
Motor Facilitation During Language
The Action-Sentence Compatibility Effect
(Glenberg & Kaschak, 2002)
Andy delivered the pizza to you.
You delivered the pizza to Andy.
QuickTime™ and a
d eco mpres sor
are nee ded to s ee this picture.
Motor Facilitation During Language
The Action-Sentence Compatibility Effect
(Glenberg & Kaschak, 2002)
Liz told you the story.
You told Liz the story.
QuickTime™ and a
d eco mpres sor
are nee ded to s ee this picture.
Motor Facilitation During Language
The Action-Sentence Compatibility Effect (Glenberg & Kaschak, 2002)
QuickTime™ and a
decompressor
are needed to see this picture.
(For variation in the temporal dynamics,
see Kaschak & Borregine, 2008)
Motor Activation During Language
(Pulvermüller, 1999)
Motor Activation During Language
(Pulvermüller, 1999)
Language Networks
Motor Activation During Language
(Pulvermüller, 1999)
Language Networks Associated with Visual Objects
Motor Activation During Language
(Pulvermüller, 1999)
Language Networks Associated with Actions
Motor Activation During Language
(Martin, Wiggs, Ungerleider, & Haxby, 1996)
Motor Activation During Language
(Pulvermüller, Härle, & Hummel, 2001)
QuickTime™ and a
decompressor
are needed to see thi s picture.
Motor Activation During Language
(Pulvermüller, Härle, & Hummel, 2001)
Motor Activation During Language
(Pulvermüller, Härle, & Hummel, 2001)
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory
Sensorimotor Influences in Cognition
• Visual Properties Influence Concepts
• Motor Properties Influence Vision
• Language Influences Visual Properties
• Language and Motor Influence Each Other
• Your Entire Body Participates in Thought
Your Entire Body Participates in Thought
• Subliminal Priming of Old Age (Bargh et al., 1996)
• Push/Pull response to pos/neg words (Chen &Bargh)
• Associations with Approach or Avoid Gestures
(Cacioppo and colleagues)
• Orofacial muscles influence judgments (and IAT)
(Cacioppo and colleagues)
• A Fleeting Moment of Zen and movement/memory
Sensorimotor Influences in Cognition
OroFacial
Skeletal
motor Oculo
motor
Haptic
Concepts
Visual
Auditory
Linguistic
Memory