Download ling411-17-Perception - OWL-Space

Ling 411 – 17
Language and Perception
Perception: Starting view
 Perception is a bottom-up process
• From primary perceptual area upwards
 E.g. primary auditory, for auditory perception
 Takes place in a single perceptual area
• E.g. auditory cortex for auditory perception
 Works by processing input to the sense organ
• E.g. auditory input for auditory perception
 Each of these points is wrong!
The McGurk Effect
http://www.youtube.com/watch?v=aFPtc8BVdJk




Acoustic syllable [ba] presented to subjects
with visual presentation of articulatory gestures for [ga]
Subjects typically heard [da] or [ga]
“Evidence has accumulated that visual speech modifies
activity in the auditory cortex, even in the primary auditory
cortex.”
Mikko Sams (2006)
How does it work?
(1) Visual input
(2) Top-down processing
Perception – Refining a simple-minded view
1.
It is not confined to a single perceptual modality
• The McGurk effect
 Auditory perception affected by visual input
• Conceptual structure affects auditory perception
 The influence of context on speech perception
Perception – Refining a simple-minded view
Not confined to a single perceptual modality
• The McGurk effect
 Visual input affects auditory perception
• Conceptual structure affects auditory perception
2. Not just bottom-up
• Top-down processing fills in unsensed details
3. Not even confined to posterior cortex
• Can also use motor neurons
 Experiment: left hand or right hand?
 Mirror neurons
1.
Top-down processing in perception
Node for CUP in
conceptual area
for drinking
vessels
Conceptual and perceptual
information
CUP
T
MADE OF GLASS
SHORT
CERAMIC
HAS HANDLE
Properties
Visual properties
are in occipital
and lower
temporal areas
Bidirectional processing and inference
These connections
are bidirectional
CUP
T
MADE OF GLASS
SHORT
CERAMIC
HANDLE
Pertinent neuroanatomical findings: Bidirectional
Processing
 An established fact of neuroanatomy:
• A connection from point A to point B in
the cortex is generally accompanied by
a connection from point B to point A
 Separate fibers (axons): (1) A to B, (2) B to
A
 In short, cortico-cortical connections are
generally bidirectional
Bidirectional processing: reciprocal links
excitatory
inhibitory
Bidirectional processing and inference
Thought process:
CUP
T
SHORT
HANDLE
1. The cardinal concept
node is activated by a
subset of its property
nodes
2. Feed-backward
processing activates
other property nodes
Consequence:
We “apprehend”
properties that are not
actually present in the
sensory input
Bidirectional processing and inference
These connections
are bidirectional
CUP
T
MADE OF GLASS
SHORT
CERAMIC
HANDLE
Separate fibers for the
two directions; shown as
one line in the notation
Cortical Structure and Inference:
Perceiving things that are not in the input
Category
T
A
Consequence:
If A and B, then
E and F
B
C
Properties
F
D
E
Examples
 Looks like a duck
• Probably quacks
 Ceramic, cup-shaped, handle
• Probably holds coffee (without
breaking)
 Dark clouds, thunder
• It’s going to rain
 ATM
• Probably has money
Perception depends mainly on cortical structures
already present before sensory input
“Perception is hallucination
constrained by sensory data”
Shepherd
A terminological problem
 We need to distinguish
• Perception narrowly conceived
 The basic process of recognition
 Single perceptual modality
 Bottom-up processing
 No motor involvement
• Perception broadly conceived
 Two different terms needed
• Recognition (a.k.a. ‘microperception’)
 Bottom-up process in a single perceptual modality
• Perception (the broad conception) (a.k.a.
‘macroperception’)
“Micro-perception” and “macro-perception”
 Microperception and macroperception
 Microperception
• A.k.a. recognition
• The local process of integrating features
• Performed in one perceptual modality
• Bottom-up
 Macroperception
• The overall process of perception
• Uses multiple modalities
• Uses top-down processing
Perception – Refining a simple-minded view
Not just bottom-up
• Top-down processing fills in unsensed details
2. Not confined to a single perceptual modality
• The McGurk effect
 Visual input affects auditory perception
• Conceptual structure affects auditory perception
3. Not even confined to posterior cortex
• Can also use motor neurons
 Experiment: left hand or right hand?
 Mirror neurons
1.
Left hand or right hand?
Left hand or right hand?
Left hand or right hand?
Left hand or right hand?
Left or right hand?




Imaging experiment
Subjects were shown pictures of one hand
Asked to identify: left or right
Functional imaging showed increased CBF in
hand area of motor cortex
Peter Fox, ca. 2000
Motor structures in perception






The left-hand vs. right-hand experiment
‘Mirror neurons’ in motor cortex
Articulation as aid to phonological perception
Articulation in reading
Motor activity in listening to music
Watching an athletic event
Mirror Neurons
 NY Times: “One mystery remains: What
makes them so smart?” (Jan. 10, 2006)
 Answer: They are not smart in themselves
• Their apparent smartness is a result of
their position: at top of a hierarchy
• Compare:
 The general of an army
 The head of a business
 Similarly, high-level conceptual nodes
• The “grandmother node”
Mirror Neurons
 What makes mirror neurons appear to be special?
• Ans.: They receive input from visual perception
• The superior longitudinal fasciculus
 Connects visual perception to motor areas
 How can a motor neuron receive perceptual input?
• Motor neurons are supposed to operate top-down
• Answer: bidirectional processing
 They also receive perceptual information
• Bottom-up processing
Superior Longitudinal Fasciculus
From O. D. Creutzfeldt, Cortex Cerebri (1995)
Are some neurons “smarter” than others?
 Claim: A grandmother node would have to
be very smart
• Identifies very complex object
• Even in many varieties
 Alternative: the head of a hierarchy
• It is the hierarchy as a whole that has
those ‘smarts’
• Similarly, mirror neurons
 They get visual input since they are
connected to visual areas
• Superior longitudinal fasciculus
Implications of hierarchical organization
 Nodes at a high level in a hierarchy may give
the appearance of being very “smart”
 This appearance is a consequence of their
position — at top of hierarchy
 As the top node in a hierarchy, a node has the
processing power of the whole hierarchy
• Grandmother nodes
• Mirror neurons
• Compare:
 The general of an army
 The head of a business organization
Multi-Modal Perception
Perception is not just bottom-up
• Top-down processing fills in unsensed details
2. It is not confined to a single perceptual modality
• The McGurk effect
 Visual input affects auditory perception
• Conceptual structure affects auditory perception
3. It is not even confined to posterior cortex
• Can also use motor neurons
 Motor activation in speech perception
 Mirror neurons
1.
Perceptual structures in motor production


Perceptual structure is used in two ways
1. Planning (e.g. visualizing while painting)
2. Monitoring
Examples
• Phonological recognition in speech production
 Cf. Wernicke’s aphasia
• Painting
• Musical production
• Baseball, soccer, tennis, etc.
The Influence of language on non-linguistic perception
 As we have seen, non-speech input
affects phonological perception
 It is also the case that language
affects non-linguistic perception
• E.g., visual perception
Language and (Visual) Perception
Phonological
Production
Phonological
Perception
Object
Categories
Language
N.B.: These connections are bidirectional
Vision
Recent experiments of Kay et al.
 Experiments at UC Berkeley
• Color perception: do differences in color naming
across languages influence color perception?
 Main finding:
• Lateralized influence of language on perception
• Response time faster for between-category
discrimination – especially for RVF presentation
• A left hemisphere (RVF) phenomenon
green
blue
Perception: A simple-minded view, revisited
 Perception is a bottom-up process
• From primary perceptual area upwards
 E.g. primary auditory, for auditory perception
 Takes place in a single perceptual area
• E.g. auditory cortex for auditory perception
 Works by processing input to the sense organ
• E.g. auditory input for auditory perception
 Each of these points in wrong
Another hypothesis of Whorf
 Grammatical categories of a language influence
the thinking of people who speak the language
 Can we explain this too in terms of brain
structure?
Mechanisms of operation
Entrenchment
• Strengthening of connections
through repeated activation
 An automatic brain process
 Important in learning
2. Reverberation of activation
3. Priming
4. Language as a major means of learning
conceptual and perceptual distinctions
1.
Entrenchment and thinking: a mechanism
 Connections become stronger with use
• (entrenchment)
 Grammatical categories make speakers
constantly heed selected phenomena
 Connections for phenomena which
speakers must constantly heed..
• Will be repeatedly traversed
• Therefore will get progressively stronger
Example: Grammatical gender
 Does talking about inanimate objects as if they
were masculine or feminine actually lead
people to think of inanimate objects as having a
gender?
 Could the grammatical genders assigned to
objects by a language influence people’s mental
representation of objects?
Boroditsky (2003)
Plausibility of the possibility
 Children learning to speak a language
with grammatical gender may suppose
that gender indicates a meaningful
distinction between types of objects
 Other grammatical distinctions do reflect
actual perceptual differences:
singular:plural
Children learning a language with gender
 “For all they know, the grammatical
genders assigned by their language
are the true universal genders of
objects.”
Boroditsky et al, 2003
Experiment: Gender and Associations
(Boroditsky et al. 2002)
 Subjects: speakers of Spanish or German
• All were fluent also in English
• English used as language of experiment
 Task: Write down the 1st 3 adjectives that
come to mind to describe each object
• All the (24) objects have opposite gender
in German and Spanish
 Raters of adjectives: Native English speakers
Examples:
 Key (masc in German, fem in Spanish)
• Adjectives used by German speakers:
 Hard, heavy, jagged, metal, serrated, useful
• Adjectives used by Spanish speakers:
 Golden, intricate, little, lovely, shiny, tiny
 Bridge (fem in German, masc in spanish)
• Adjectives used by German speakers:
 Beautiful, elegant, fragile, peaceful, pretty
• Adjectives used by Spanish speakers:
 Big, dangerous, long, strong, sturdy,
towering
Results of the Experiment
(Boroditsky et al. 2002)
 Raters of adjectives were native English
speakers
 Result: Adjectives were rated as masculine or
feminine in agreement with the gender in
subject’s native language
end