Download JAY McCLELLAND

Representation, Development and
Disintegration of Conceptual Knowledge:
A Parallel-Distributed Processing
Approach
James L. McClelland
Department of Psychology and
Center for Mind, Brain, and Computation
Stanford University
Parallel Distributed Processing
Approach to Semantic Cognition
• Representation is a pattern
of activation distributed
over neurons within and
across brain areas.
• Bidirectional propagation
of activation underlies the
ability to bring these
representations to mind
from given inputs.
• The knowledge underlying
propagation of activation is
in the connections.
language
A Principle of Learning and
Representation
• Learning and representation are sensitive to
coherent covariation of properties across
experiences.
What is Coherent Covariation?
• The tendency of properties of objects to cooccur in clusters.
• e.g.
– Has wings
– Can fly
– Is light
• Or
– Has roots
– Has rigid cell walls
– Can grow tall
Development and Degeneration
• Sensitivity to coherent covariation in an
appropriately structured Parallel Distributed
Processing system underlies the development
of conceptual knowledge.
• Gradual degradation of the representations
constructed through this developmental
process underlies the pattern of semantic
disintegration seen in semantic dementia.
Some Phenomena in Development
• Progressive differentiation of concepts
• Overgeneralization
• Illusory correlations
The Rumelhart Model
The Training Data:
All propositions true of
items at the bottom level
of the tree, e.g.:
Robin can {grow, move, fly}
Target output for ‘robin can’ input
Forward Propagation of Activation
aj
wij
neti=Sajwij
ai
wki
Back Propagation of Error (d)
aj
wij
di ~
Sdkwki
ai
wki
Error-correcting learning:
At the output layer:
At the prior layer:
…
dk ~ (tk-ak)
Dwki = edkai
Dwij = edjaj
Early
Later
Later
Still
E
x
p
e
r
i
e
n
c
e
What Drives
Progressive
Differentiation?
•
Waves of differentiation reflect
coherent covariation of properties
across items.
•
Patterns of coherent covariation are
reflected in the principal
components of the property
covariance matrix.
•
Figure shows attribute loadings on
the first three principal components:
– 1. Plants vs. animals
– 2. Birds vs. fish
– 3. Trees vs. flowers
•
•
Same color = features covary in
component
Diff color = anti-covarying
features
Coherence
Training
Patterns
Items
Properties
Coherent
Incoherent
is can has is can has …
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
No labels are provided
Each item and each property occurs with equal frequency
Effect of Coherence on Representation
Overgeneralization of Frequent
Names to Similar Objects
“goat”
“tree”
“dog”
Illusory Correlations
• Rochel Gelman found that children think that
all animals have feet.
– Even animals that look like small furry balls
and don’t seem to have any feet at all.
• A tendency to over-generalize properties
typical of a superordinate category at an
intermediate point in development is
characteristic of the PDP network.
A typical property that
a particular object lacks
e.g., pine has leaves
An infrequent,
atypical property
Sensitivity to Coherence
Requires Convergence
A
A
A
Another key property of the model
• Sensitivity to coherent covariation can be
domain- and property-type specific, and such
sensitivity is acquired as differentiation occurs.
• Obviates the need for initial domain-specific
biases to account for domain-specific patterns
of generalization and inference.
Differential Importance
(Marcario, 1991)
• 3-4 yr old children see a puppet
and are told he likes to eat, or
play with, a certain object (e.g.,
top object at right)
– Children then must choose
another one that will “be the
same kind of thing to eat” or
that will be “the same kind of
thing to play with”.
– In the first case they tend to
choose the object with the
same color.
– In the second case they will
tend to choose the object
with the same shape.
Adjustments to
Training
Environment
•
•
•
Among the plants:
– All trees are large
– All flowers are small
– Either can be bright or
dull
Among the animals:
– All birds are bright
– All fish are dull
– Either can be small or
large
In other words:
– Size covaries with
properties that
differentiate different
types of plants
– Brightness covaries
with properties that
differentiate different
types of animals
Testing Feature Importance
• After partial learning, model is shown eight test objects:
– Four “Animals”:
• All have skin
• All combinations of bright/dull and large/small
– Four “Plants”:
• All have roots
• All combinations of bright/dull and large/small
• Representations are generated by using
back-propagation to representation.
• Representations are then compared to see which
animals are treated as most similar, and which plants
are treated as most similar.
Similarities of Obtained
Representations
Size is relevant
for Plants
Brightness is relevant
for Animals
Development and Degeneration
• Sensitivity to coherent covariation in an
appropriately structured Parallel Distributed
Processing system underlies the development
of conceptual knowledge.
• Gradual degradation of the representations
constructed through this developmental
process underlies the pattern of semantic
disintegration seen in semantic dementia.
Disintegration of Conceptual
Knowledge in Semantic Dementia
• Progressive loss of specific knowledge of
concepts, including their names, with
preservation of general information
• Overgeneralization of frequent names
• Illusory correlations
Picture naming
and drawing in
Sem. Demantia
Grounding the Model in What we Know
About The Organization of Semantic
Knowledge in The Brain
• There is now evidence for
specialized areas subserving
many different kinds of
semantic information.
• Semantic dementia results
from progressive bilateral
disintegration of the
anterior temporal cortex.
• Rapid acquisition of new
knowledge depends on
medial temporal lobes,
leaving long-term semantic
knowledge intact.
language
Proposed Architecture for the
Organization of Semantic Memory
name
action
Temporal
pole
motion
color
valance
form
Medial Temporal Lobe
Rogers et al (2005)
model of semantic
dementia
temporal
pole
name
function
assoc
vision
• Gradually learns through
exposure to input patterns
derived from norming
studies.
• Representations in the
temporal pole are
acquired through the
course of learning.
• After learning, the
network can activate each
other type of information
from name or visual input.
• Representations undergo
progressive differentiation
as learning progresses.
• Damage to units within
the temporal pole leads to
the pattern of deficits seen
in semantic dementia.
Errors in Naming for As a Function of Severity
Patient Data
Simulation Results
omissions
within categ.
superord.
Severity of Dementia
Fraction of Neurons Destroyed
Simulation of Delayed Copying
temporal
pole
name
function
assoc
vision
• Visual input is
presented, then
removed.
• After several time
steps, pattern is
compared to the
pattern that was
presented initially.
• Omissions and
intrusions are
scored for typicality
IF’s ‘camel’
DC’s ‘swan’
Simulation results
Omissions by feature type
Intrusions by feature type
Development and Degeneration
• Sensitivity to coherent covariation in an
appropriately structured Parallel Distributed
Processing system underlies the development
of conceptual knowledge.
• Gradual degradation of the representations
constructed through this developmental
process underlies the pattern of semantic
disintegration seen in semantic dementia.
Sensitivity to Coherence
Requires Convergence
A
A
A