Download PPT

The ICSI/Berkeley
Neural Theory of Language Project
 Graduate Students
•
Principal investigators




Jerome Feldman (UCB,ICSI)
George Lakoff (UCB Ling)
Srini Narayanan (UCB,ICSI)
Lokendra Shastri (now India)
• Affiliated faculty




Chuck Fillmore (ICSI)
Eve Sweetser (UCB Ling)
Rich Ivry (UCB Psych)
Lisa Aziz-Zadeh (USC)









Leon Barrett (CS)
*Johno Bryant (CS)
*Nancy Chang (CS)
Ellen Dodge (Ling)
Michael Ellsworth (Ling)
Joshua Marker (Ling)
*Eva Mok (CS)
Shweta Narayan (Ling)
*Steve Sinha (CS)
 Alumni









Terry Regier (UCB Ling)
David Bailey (Google)
Andreas Stolcke (ICSI, SRI)
Dan Jurafsky (Stanford Ling)
Olya Gurevich (Powerset)
Benjamin Bergen (U. Hawaii Ling)
Carter Wendelken (UCB)
Srini Narayanan (ICSI, UCB)
Gloria Yang (UTD)
Unified Cognitive Science
Neurobiology
Psychology
Computer Science
Linguistics
Philosophy
Social Sciences
Experience
Take all the Findings and Constraints Seriously
Constrained Best Fit in Nature
inanimate
physics
chemistry
biology
animate
lowest energy
state
molecular fit
vision
fitness, MEU
Neuroeconomics
threats, friends
language
errors, NTL
society, politics
framing,
compromise
Brains ~ Computers
•
•
•
•
•
•
•
•
1000 operations/sec
100,000,000,000 units
10,000 connections/
graded, stochastic
embodied
fault tolerant
evolves
learns
•
•
•
•
•
•
•
•
1,000,000,000 ops/sec
1-100 processors
~ 4 connections
binary, deterministic
abstract, disembodied
crashes frequently
explicitly designed
is programmed
Fast Brain ~ Slow Neurons
Mental Connections are Active
Neural Connections
There is No Erasing in the Brain
Constraints on Connectionist
Models
100 Step Rule
Human reaction times ~ 100 milliseconds
Neural signaling time ~ 1 millisecond
Simple messages between neurons
Long connections are rare
No new connections during learning
Developmentally plausible
Connectionist Models in Cognitive Science
Structured
PDP
Hybrid
Neural
Conceptual
Existence
Fast Mapping
Data Fitting
Skill Learning
Not discussed in meeting
Triangle nodes and
McCullough-Pitts Neurons?
A
B
C
A
B
C
Representing concepts using
triangle nodes
Functionalism
In fact, the belief that neurophysiology is even relevant to the
functioning of the mind is just a hypothesis. Who knows if we’re
looking at the right aspects of the brain at all. Maybe there are
other aspects of the brain that nobody has even dreamt of
looking at yet. That’s often happened in the history of science.
When people say that the mental is just the neurophysiological
at a higher level, they’re being radically unscientific. We know a
lot about the mental from a scientific point of view. We have
explanatory theories that account for a lot of things. The belief
that neurophysiology is implicated in these things could be true,
but we have very little evidence for it. So, it’s just a kind of hope;
look around and you see neurons: maybe they’re implicated.
Noam Chomsky 1993, p.85
Embodiment
Of all of these fields, the learning of
languages would be the most impressive,
since it is the most human of these
activities. This field, however, seems to
depend rather too much on the sense
organs and locomotion to be feasible.
Alan Turing (Intelligent Machines,1948)
Continuity Principle of the American Pragmatists
The ICSI/Berkeley
Neural Theory of Language Project
ECG
Learning early
constructions
(Chang, Mok)
Ideas from Cognitive Linguistics
• Embodied Semantics (Lakoff, Johnson, Sweetser, Talmy
• Radial categories
(Rosch 1973, 1978; Lakoff 1985)
– mother: birth / adoptive / surrogate / genetic, …
• Profiling (Langacker 1989, 1991; cf. Fillmore XX)
– hypotenuse, buy/sell (Commercial Event frame)
• Metaphor and metonymy
(Lakoff & Johnson 1980, …)
– ARGUMENT IS WAR, MORE IS UP
– The ham sandwich wants his check.
• Mental spaces (Fauconnier 1994)
– The girl with blue eyes in the painting really has green eyes.
• Conceptual blending (Fauconnier & Turner 2002, inter alia)
– workaholic, information highway, fake guns
– “Does the name Pavlov ring a bell?” (from a talk on ‘dognition’!)
Simulation-based language
understanding
“Harry walked to the cafe.”
Utterance
Constructions
Analysis Process
General
Knowledge
Belief State
Schema
walk
Trajector
Harry
Cafe
Goal
cafe
Simulation
Specification
Simulation
Psycholinguistic evidence
• Embodied language impairs action/perception
– Sentences with visual components to their meaning
can interfere with performance of visual tasks
(Richardson et al. 2003)
– Sentences describing motion can interfere with
performance of incompatible motor actions
(Glenberg and Kashak 2002)
– Sentences describing incompatible visual imagery
impedes decision task (Zwaan et al. 2002)
• Simulation effects from fictive motion sentences
– Fictive motion sentences describing paths that require
longer time, span a greater distance, or involve
more obstacles impede decision task (Matlock 2000, Matlock
et al. 2003)
Neural evidence: Mirror neurons
• Gallese et al. (1996) found “mirror” neurons
in the monkey motor cortex, activated when
– an action was carried out
– the same action (or a similar one) was seen.
• Mirror neuron circuits found in humans (Porro
et al. 1996)
• Mirror neurons activated when someone:
– imagines an action being carried out (Wheeler et al.
2000)
– watches an action being carried out (with or
without object) (Buccino et al. 2000)
Active representations
• Many inferences about actions derive from what we
know about executing them
• Representation based on stochastic Petri nets
captures dynamic, parameterized nature of actions
• Used for acting, recognition, planning, and language
walker at goal
energy
walker=Harry
goal=home
Walking:
bound to a specific walker with a
direction or goal
consumes resources (e.g., energy)
may have termination condition
(e.g., walker at goal)
ongoing, iterative action
Learning Verb Meanings
David Bailey
A model of children learning their first verbs.
Assumes parent labels child’s actions.
Child knows parameters of action, associates with word
Program learns well enough to:
1) Label novel actions correctly
2) Obey commands using new words (simulation)
System works across languages
Mechanisms are neurally plausible.
System Overview
Learning Two Senses of PUSH
Model merging based on Bayesian MDL
NTL Manifesto
• Basic Concepts are Grounded in Experience
– Sensory, Motor, Emotional, Social,
• Abstract and Technical Concepts map by
Metaphor to more Basic Concepts
• Neural Computation models all levels
Simulation based Language
Understanding
Utterance
Discourse & Situational
Context
Constructions
Analyzer:
incremental,
competition-based,
psycholinguistically
plausible
Semantic Specification:
image schemas, frames,
action schemas
Simulation
“Harry walked into the cafe.”
Pragmatics
Semantics
Syntax
Morphology
Phonology
Phonetics
“Harry walked into the cafe.”
Pragmatics
Semantics
Syntax
Morphology
Phonology
Phonetics
U
T
T
E
R
A
N
C
E
Embodied Construction Grammar
• Embodied representations
– active perceptual and motor schemas
(image schemas, x-schemas, frames, etc.)
– situational and discourse context
• Construction Grammar
– Linguistic units relate form and
meaning/function.
– Both constituency and (lexical) dependencies
allowed.
• Constraint-based
– based on feature unification (as in LFG, HPSG)
– Diverse factors can flexibly interact.
Embodiment and Grammar Learning
Paradigm problem for Nature vs. Nurture
The poverty of the stimulus
Embodiment and Grammar Learning
Paradigm problem for Nature vs. Nurture
The poverty of the stimulus
The opulence of the substrate
Intricate interplay of genetic and environmental,
including social, factors.
Embodied Construction Grammar
ECG
(Formalizing Cognitive Linguisitcs)
1. Linguistic Analysis
2. Computational Implementation
a. Test Grammars
b. Applied Projects – Question Answering
3. Map to Connectionist Models, Brain
4. Models of Grammar Acquisition