Download Language - Ohio University

Cognitive Neuroscience
and Embodied Intelligence
Language
Based on book Cognition, Brain and Consciousness ed. Bernard J. Baars
courses taught by Prof. Randall O'Reilly, University of Colorado, and
Prof. Włodzisław Duch, Uniwersytet Mikołaja Kopernika
and http://wikipedia.org/
http://grey.colorado.edu/CompCogNeuro/index.php/CECN_CU_Boulder_OReilly
http://grey.colorado.edu/CompCogNeuro/index.php/Main_Page
Janusz A. Starzyk
EE141
1
Modeling speech
Models account for:
distributed lexicon, orthography, phonology, semantics.
The same learning mechanisms in the brain, but different
inputs/outputs.
Levels of processing: phonemes/syllables, letters, words, ideas,
phrases, sentences, situations, stories.
Distributed representations, great possibilities of combining many
representations
Semantic representations of word co-occurrence.
Semantic representations on the level of sentence shapes.
Phonological neighborhood density of words = the number of words
that sound similar to a given word, so creating similar activations in
the brain.
Semantic neighborhood density of words = the number of words with
2
a similar meaning (widened activation subnetwork).
EE141
Modeling speech
Learning based on processing temporal sequences
Word sequences must produce meaning representations
Language is the result of unpacking distributed meaning
representations in the brain and communicating them to other
people through communication channels, with the expectation
that their corresponding representations will be created in the
brain of the receiver
Learning to read
dyslexia
Sign recognition, mapping orthography onto phonology (not trivial for
English) and intonations (important in Chinese)
Regularities and exceptions
creating too-regular past tenses.
3
EE141
Biological foundations
Controlling the vocal apparatus is responsible for the correct pronunciation
of syllables. Mainly responsible for this control is Broca's area in the frontal
cortex; for speech analysis, Wernicke's area in the superior temporal lobe.
Broca's: surface representation, Wernicke's: deep representation.
4
EE141
Organization of phonemes
Neuron uczy się prawdopodob.
warunkowego, korelacji
pomiędzy pożądaną
aktywnością a sygnałami
wejściowymi; optymalna wartość
0.7 osiągana jest szybko tylko
przy małej stałej uczenia 0.005
Consonants: 3 dimensions.
Coding: 7 positions for location (loc),
5 for manner (Mnr), 2 for voicing
(Vce).
EE141
Phonemes: consonants and
vowels, IPA alphabet.
4 dimensions characterizing
vowels, tongue positions.
Coding: 7 positions for
front/back, 6 for up/down, 4
for the rest (shape of the
lips and length).
5
Questions
We will try, with the help of computer simulations, to find and verify with
the help of models, the answers to several questions:
 What
processes are involved in the reading process and why do they
sometimes let us down (dyslexia)?
 How
do we read known words: cat, yacht, and how do we read
invented words, eg. nust, deciding on some pronunciation?
 Why
do children say "I goed” instead of "I went”?
 Where
 How
does the meaning of words come from?
to go from words to sentences?
6
EE141
Distributed lexicon and dyslexia
Phonological level of dyslexia:
nonexistent words don't activate
deeper areas (Wernicke's).
Deep level: phonological and semantic
errors (cat – cot, cat - dog),
mistakes in sign recognition.
Surface dyslexia: new words don't create a problem, but there is a lack of
access to the semantic level + difficulties in reading exception words +
mistakes in recognition.
A model of reading and dyslexia has two paths from orthography to
phonology: direct (by mapping) and indirect, via semantics.
Uncommon and difficult words are pronounced through the indirect
pathway.
EE141
7
Words to read
40 words, 20 concrete and 20 abstract; activations in the model show
phonological-semantic similarities.
8
EE141
Model
Project dyslex.proj.gz
The network was trained because
it requires 250 epochs.
Training: random selection of one
of the 3 layers (orthography,
phonology, semantics) as input
and the other 2 layers as outputs,
mapping one aspect onto the two
others.
kWTA = 25% for hidden layers
View TestLog, StepTest
TestLog shows the word, distance, most similar, error sm_nm
EE141
9
Simulating dyslexia
Dyslexia: depending on the
degree of damage and the
pathway damaged, we get
different forms of dyslexia:
phonological, deep and surface.
lesion_path = Semantics
Turns off the whole layer.
Errors: Trial2_TestLog
need, loan, flow, past => coat
Hire and coat are the most
frequently mistaken.
Phonological distance from 0 to 1
measured by cos(S1,S2) =
S1*S2/|S1||S2|
10
EE141
Effect of lesions on dyslexia
Errors of hidden neurons
OS_Hid orthographic-semantic
SP_Hid semantic-phonetic
Error types for semantic pathway lesions.
11
EE141
Complete lesions
Errors resulting from semantic and direct pathway lesions.
12
EE141
Direct pathway lesions
Errors resulting from direct pathway lesions, with both no lesions in the
semantic pathway (Full Semanitcs) and with a complete semantic
13
pathway
lesion
(No
Semantics).
EE141
Reading
Reading models: mapping orthography onto phonology.
Two issues:
 can one system learn to pronounce regular words and
simultaneously deal with exception words?
 simulating pronunciation of nonexistent words requires the discovery
of subtle regularities of pronuncation.
Mint, hint, flint => "i" is the same, but in pint it's different...
Regularities are often modified, depending on the context, they have
groupings (neighborhoods), and exceptions are on the extremes of
these modifications.
Regularities and exceptions form a continuum.
14
EE141
Reading: distributed lexical model
Representations are not localized in one region.
Interactions lead to an interesting division of labor.
EE141
15
Reading as object recognition
Exchange between dependent and
independent is similar in object
recognition
We need constants but we also need
feature connections
16
EE141
Reading model
Model ss.proj.gz Chapt. 10.4.2
7 blocks of 3*9 = 189 inputs,
5*84 = 420 in orthography, and
600 hidden, 7 blocks of
2*10 = 140 phonological
elements.
Input: words up to 7 letters,
completion, eg.
best = bbbestt
17
EE141
Regularities
Tests of regularity:
Głuszko
Pseudo-homophony
phyce => Choyce
Network relaxation times
as a function of
frequency and
consistency of words.
Human reading speeds
shows similar
dependencies.
18
EE141
Past tenses
Union of semantics and phonology
Project pt.proj.gz
Regularization tendencies change
with maturation and learning new
words.
Initial training first on irregular
words and then on regular ones,
this is controversial but gives a Ushaped curve. Models gradually
changing the learning environment
but without Hebbian learning don't
work.
19
EE141
Leabra model
Network: semantic input, hidden layer +
phonological.
Data: 389 verbs, including 90 irregular in the
past tense, 4 possible regular endings:
-ed, -en, -s, -ing, total 1945 examples.
Cooperation + competition +
Hebbian learning gives a
network, in which mapping
regular and irregular verbs
reaches a dynamic equilibrium.
Priming after several exposures
changes the behavior of the
network.
EE141
20
Leabra model
Cooperation + competition + Hebbian
learning gives a network, in which mapping
regular and irregular verbs reaches a
dynamic equilibrium.
Priming after several exposures changes the
behavior of the network.
21
EE141
Word meaning
Idea semantics is the result of
activations distributed across
many areas.
Simplest model: Strong Hebbian
correlations between words, like
correlations between elements of
images or phonemes creating
syllables.
LSA- Latent Semantic Analysis,
type of PCA, which can be
realized by Hebbian learning.
22
EE141
Words in the brain
Psycholinguistic experiments about speech show that in the brain we
have discrete phonological representations, and not acoustic ones.
Acoustic signal => phonemes => words => semantic concepts.
Semantic activations follow 90 ms after phonological activations (N200
ERPs).
F. Pulvermuller (2003) The Neuroscience of Language. On Brain
Circuits of Words and Serial Order. Cambridge University Press.
Action networks – observations, findings of ERP and fMRI
tests.
EE141
23
Words: simple model
Goals:
 making the simplest model of creative thinking;
 creating interesting new names, conveying product features;
 understanding new words, which aren't in the dictionary.
A model inspired by overlapping brain processes which happen during
invention of new words. Given is a set of key words, which activate the
auditory cortex.
Phonemes are resonances, orderly activation of phonemes activates known
words and new combinations equally; context + inhibition in the winnertakes-all process leaves one word.
Creativity
= imagination (fluctuations) + filtering (competition)
Imagination: many temporary resonances arise in parallel, activating
representations of words and non-words, depending on the connection
strength of oscillators.
Filtering: associations, emotions, phonological/semantic density.
24
EE141
Associations - revision
Why does priming neutral for simple associations and nonsensical words
worsen results for creative people?
Weak creativity = weak associations (connections) between oscillators;
adding noise (nonsensical words) strengthens already overlapping
oscillations, enabling mutual activations; for a strongly connected neural
network and simple associations, it leads to confusion, when it activates
many states.
For difficult associations, adding noise in weakly creative people won't help
because of a lack of connections, priming words cause only chaos.
For orthographically similar priming words with close associations, this
activates the representation of the second word, always increasing the
chance of resonance and shortening latency.
25
EE141
Quiz
Project sem.proj.gz, description 10.6.2
An already trained network responds to questions...
26
EE141
Sentence meaning
Traditional approach:
grammatical breakdown of sentences.
Alternative approach: distributed
representations, gestalt of the sentence.
Like in recognition of 3D objects, there is
no central representation.
Small world: sentences containing people's names, active and passive
actions, objects and places.
People: busdriver, teacher, schoolgirl, pitcher.
Actions: eat, drink, stir, spread, kiss, give, hit, throw, drive, rise.
Objects: Spot (the dog), steak, soup, ice cream, crackers, jelly, iced tea,
kool aid, spoon, knife, anger, rose, bat (animal), bat (baseball), ball, ball
(party), bus, pitcher, fur
Places: kitchen, living room, shed, park.
27
EE141
Network and project
Project sg.proj.gz, description 10.7.2
The input presents words, localized
representations, in the Encode layer
are created distributed
representations, integrated in time in
the Gestalt and Gestalt_Context
layers, questions are connected with
roles (agent, patient, instrument ...),
the network decodes representations,
completing them (Filler)
Eg. bat (animal) and bat (baseball) requires differentiation.
28
EE141
Tests
Role assignment, word ambiguity, concept instantiation, role elaboration,
conflict resolution.
Small world: sentences containing people's names, active and passive
actions, objects and places.
29
EE141
Verb similarity
Unambiguous verbs,
after training of the
network, have these
cosine similarities of
internal activations.
30
EE141
Noun similarity
Unambiguous nouns,
after training of the
network, have these
cosine similarities of
internal activations.
31
EE141
Shapes of sentences
Similarities of sample
sentences:
sc_ = schoolgirl
bu_ = busdriver
te_ = teacher
pi_ = pitcher
_at = ate
_dr = drank
_st = stirred
_so = soup
_st = steak
_ic = ice tea
_ko = Kool-Aid
32
EE141
Questions/answers concerning language
What processes are involved in the reading process and why do they
sometimes betray us (dyslexia)? Distributed lexical representations,
interactions between sign recognition, level of spelling (orthography),
phonology and semantics.
How do we read known words: cat, yacht, and how do we read
invented words, eg. nust? Thanks to contextually activated
representations, giving a continuum between regular forms and
exceptions.
Why do children say "I goed” instead of "I went”? Because of the
dynamic equilibrium between mapping regular forms and exceptions.
Where does the meaning of words come from? Statistics of cooccurrence, interactions with representations of sensory data.
How to go from words to sentences? This is enabled by the "Sentence
Gestalt" (a theory in psychology).
33
EE141