Download Affective Models - Cognitive Systems Lab

Cognitive Modeling – Human and Machine Learning
Human and Machine Learning
Cognitive Modeling
Dominic Heger,
Felix Putze,
Tanja Schultz
20.6.2013
1/48
Cognitive Modeling – Human and Machine Learning
Outline
• Administrative Stuff
• Human Learning
• Behaviorist Learning Theory
 Classical and operant conditioning
• Cognitive Learning Theory
 Jean Piaget and Lev Vygotsky
• Connectionism / Learning in the brain
• Machine Learning
• Artificial Neural Networks
• Imitation Learning /
Learning by Demonstration
• Social Interactive Learning
• Learning in ACT-R
• Explicit
• Implicit
2/48
Cognitive Modeling – Human and Machine Learning
Oral Exams
• If you would like to take an oral exam in Cognitive Modeling
• Please send an email to Mrs Scherer ([email protected])
• The email should include:
 You would like to have an exam in KM and within which module
 Your study program (BA Info, MA InWi, etc.)
 Your preferred day for the exam
• Exam days are:
• 26.07.2013
• 16.10.2013
• 17.10.2013
• First come first serve
• You have to sign up for the exam online!
3/48
Cognitive Modeling – Human and Machine Learning
Lecture Evaluation
4/48
Innate Behavior
• Innate behavior: Abilities not acquired after birth
Cognitive Modeling – Human and Machine Learning
• Does not need to be learned or taught
• Reflexes and Automatisms
• Automatic unconscious reactions
(may not require brain activity -> reflex arc)
• Reflex: caused by stimulus
• E.g. Startle reflex, coughing; breathing, heart beat, …
• Instincts
• Key stimulus causes (a complex) Fixed Action Pattern
• E.g. Want attention -> cry, Hungry -> eat, …
• Basic motor skills
• E.g. Grasping, chewing, …
• Learning and recognition
• → Humans learn nearly all of their abilities after birth
5/48
Learning
Cognitive Modeling – Human and Machine Learning
• Learning is the core concept behind
most human abilities
• Definitions
• Lefrancois1: Behavioral changes as a result of experiences
• Tom Mitchell: Learning is improving performance
at some task through experience
• Learning theory and behavioral theory often used
synonymously
• Learning generates potential behavioral changes
• Human and Machine Learning: To build systems motivated by
Human abilities has long-established tradition in research
1Lefrancois: Psychologie des
6/48
Lernens, Springer Heidelberg, 4. Ed., p. 20
Why human learning ideas for machines?
• Why transfer ideas from human learning to machine learning?
Cognitive Modeling – Human and Machine Learning
• Human abilities and machines abilities differ
• Applications of Machine Learning
• Where algorithms are too difficult to develop
 Algorithms describing natural processes can be extremely complex
 E.g. Speech recognition, walking
• Adaptive Systems
 Continuously changing requirements
 E.g. Spam Filter
 Natural machine interaction
 E.g. Intuitive interfaces, Human-like behavior
• Synthetic approach to science
 Model human behavior using computers/robots
 If successful: These are potentially good principles and methods
 → Understanding of human behavior and mind
7/48
Cognitive Modeling – Human and Machine Learning
Important Factors of Human Learning
• What are important factors influencing Human Learning?
• Perception
• Remember: Lecture on Perception and Attention
• Memory
• Remember: Lecture on Knowledge Representation and memory
models
• Limited throughput
• Temporal characteristics: short term, long term, ...
• Types of information: declarative, procedural, language, …
• Motivation
•
•
•
•
Emotions, Attitudes and Self concept
Drives and needs (e.g. Maslow’s pyramid)
Arousal (e.g. Yerkes-Dodson) and probability of success (e.g. Brehm)
Cognitive dissonance (Festinger)
8/48
Overview on Learning Theory
Cognitive Modeling – Human and Machine Learning
• Traditional schools of learning theory
• Behaviorism
 Psychology is science of behavior. Sources of behavior are external
(observable environment) and not internal (mind)
 Variables: Stimulus and Response
 Methods: Animal experiments, transfer results to humans
• Cognitivism
 Behavior is result of information processing with aspects of
thinking, feeling, intentions, desires, expectations, appraisals,…
 Variables: Cognitive (intellectual) processes
 Methods: Mainly investigation of humans, empiric studies, …
• Today, learning is mainly seen as cognitive process, however
behaviorist ideas and principles are still important and valid
and still vitally researched
9/48
Machine Learning
Cognitive Modeling – Human and Machine Learning
• Machine Learning is often classified into three broad types
1.
Supervised Learning:
 Learning with a teacher
 Training instances are labeled
2.
Unsupervised Learning:
 Learning with no help
 Training instances are not labeled
3.
Reinforcement Learning:
 Learning with limited feedback
 Actions are rewarded or penalized
10/48
Cognitive Modeling – Human and Machine Learning
Neural Networks
• Idea: ML inspired by Human learning should use similar
entities to those in the brain
• Model complex network of simple basic elements (neurons)
Property
Biological NN
Artificial NN
#Computational
units
>6*10^10 neurons
~3*10^9 transistors in 10 core Xeon
CPU (2011)
#Connections
~10 to 10^4 inputs
per neuron
Usually < a few thousands of inputs
per neuron
Speed
~10^-3 s per operation
~10^-9 s per operation
Energy
10^-16 joules per
operation per sec
10^-6 joules per operation per sec
Evolution
Millions of years
Early works in 1940s
• Biological -> Artificial Neural Nets = Radical simplification
11/48
Cognitive Modeling – Human and Machine Learning
Biological Neurons
•
Structure of
Biological Neurons:
1.
2.
3.
4.
Dendrites receive signals from connected neurons (through synapses)
The cells body (soma) sums the incoming signals (spatially and temporally)
Sufficient input is received (i.e. a threshold is exceeded) → neuron fires
That action potential is transmitted along the axon to other neurons, or to
structures outside the nervous systems (e.g., muscles)
Received input is not sufficient (i.e. the threshold is not exceeded)
→ inputs quickly decay and no action potential is generated.
Timing is important – input signals must arrive together, strong inputs will
generate more action potentials per unit time
5.
6.
12/48
Cognitive Modeling – Human and Machine Learning
Neuron Action Potential
13/48
McCulloch-Pitts Neurons
Cognitive Modeling – Human and Machine Learning
• Simplified computational model of biological neurons (1943)
• Set of synapses bring in activation from other neurons
• Processing unit sums up weighted inputs, and
then applies non-linear activation function (aka transfer/threshold
function)
• Output (spike or no spike) is transmitted to other neurons
14/48
Spiking Neurons models
Cognitive Modeling – Human and Machine Learning
• Biological neurons are more complex
•
•
•
•
Non-binary inputs and outputs,
Non-linear summation,
Timing characteristics,
Intensity of postsynaptic potentials,
instead of binary activation
• Stochasticity,…
• There are 20 different neuron
spiking behaviors
(Izhikevich, 2004)
• Several models try to model
biologically plausible behavior
15/48
Cognitive Modeling – Human and Machine Learning
Hodgkin-Huxley model
• Hodgkin and Huxley received Nobel Prize in Physiology or
Medicine 1963 for description of mechanisms underlying the
initiation and propagation of action potentials
• Squid giant axons: ~0.5-1mm diameter
• Computational model for
ionic mechanisms (Na, K)
of action potentials
• The Hodgkin-Huxley model is usually not used in ANNs
because it is computationally expensive
(~1200 FLOPS per neuron)
16/48
Hodgkin-Huxley model
Cognitive Modeling – Human and Machine Learning
• Coupled non-linear differential equations tor temporal
evolution of the membrane potential
• Total membrane current is sum of K, Na, leakage, and external
currents
• 10s of parameters, but all biophysiological meaningful
• Can produce (all of the) 20 different behaviors of neuron spiking
17/48
Cognitive Modeling – Human and Machine Learning
Learning in the brain
• Synaptic plasticity: Change of connection strength (synapse)
between neurons
• Repeated excitation of neurons increases cell connectivity
strength (synaptic plasticity)
• Hebb’s law (1949): “When an axon of cell A is near enough to
excite cell B and repeatedly or persistently takes part in firing
it, some growth process or metabolic change takes place in
one or both cells such that A's efficiency, as one of the cells
firing B, is increased”
• “Neurons that fire together wire together”
• Experimental evidence
• Long Term Potentiation (LTP): Enhancement of synaptic transmission
for hours and longer
• LTP assumed to be one of the major cellular processes underlying
learning
18/48
Hebbian Learning
Cognitive Modeling – Human and Machine Learning
• Hebbian Learning in Artificial Neural Networks (ANNs)
• Biological learning model used
e.g. in Self Organizing Maps and Hopfield Nets
• Increase weights if conditioned and
unconditioned stimulus occur simultaneously
• Adjustment of the weights
 ∆wij=η ˑ ei‘ˑ ej
 η: learning rate
 ei’: postsynaptic potential at time t+1
 ej: presynaptic potential at time t
i
• Note: Other common methods for
training ANNs have no biological correspondence
(e.g. error backpropagation)
j
wij
Source mainly: Handbuch der Künstlichen Intelligenz, 4th ed. , Oldenbourg, p. 79f, 94f
19/48
Cognitive Modeling – Human and Machine Learning
Classical Conditioning
• Ivan Pavlov (1904 Nobel prize for Physiology or Medicine)
• Classical Behavioristic experiment (Stimulus->Response):
Measure physiological properties of the dog (salivation)
• Unconditioned Stimulus (US)
• Stimulus causing response (UR) without
previous learning
• Conditioned Stimulus (CS)
• Previously neutral stimulus (NS) that
causes response (CR) because of
multiple occurrences in combination
with US
• Example:
• Before training: Bell -> no specific reaction; Food (US) -> salivation (UR)
• Training: Food (US) + bell (NS) -> salivation (UR)
• Result: Bell (CS) -> salivation (CR)
20/48
Hebbian Learning and Classical Conditioning
Cognitive Modeling – Human and Machine Learning
• Before conditioning
• Only US evokes response (R)
• CS does not evoke response
• Example: Dog sees food causes salivation
i
wij
j
• Repeatedly US + CS causing salivation
• Increases of wij
• ∆wij=η ˑ ei‘ˑ ej
• Example: Simultaneously ring bell and
serve food
i
wij
j
• CS can evoke response
• CS evokes response (R)
• Example: Ringing bell causes salivation
i
21/48
wij
j
ANN Architectures
Cognitive Modeling – Human and Machine Learning
• Common ANN architectures are:
• Single-Layer Feed-forward NNs
 One input layer and one output layer of processing units.
 No feed-back connections
 E.g. simple Perceptron
• Multi-Layer Feed-forward NNs
 One or more hidden layers of processing units
between one input and one output layers
 No feed-back connections
 E.g. Multi-Layer Perceptron
• Recurrent NNs
 Any network with at least one feed-back connection.
 It may, or may not, have hidden units.
 E.g. Hopfield nets
22/48
Hopfield Nets
Cognitive Modeling – Human and Machine Learning
• Recurrent artificial neural networks
• Each neuron is connected to each other neuron
• Symmetric weights wij = wji , no self-loops wii=0
• Discrete case: Neurons take binary value -1 or 1
(can be seen as on and off states)
• All neurons can be both input and output
• Output of a neurons is input vector in the next step
s   wij  si
s'j
sj (t+1)=sgn(s’j(t))
Neuronj
'
j
i
wNj
w1j
w2j
s1
23/48
s2
…
sN
Cognitive Modeling – Human and Machine Learning
Hopfield Nets as Associative Memory
• Human ability to retrieve information associated with an
(incomplete) cue
• Hopfield Nets as content-addressable associative memory
•
•
•
•
Several different activation patterns can be learned in a network
Produces for any input pattern a similar stored pattern
Autoassociative memory: pattern completion of noisy or partial data
Can reliably store up to
0.183*#neurons different examples
• Asynchronous Network recall
1.
2.
3.
4.
•
Set pattern as input to the neurons
Pick a neuron randomly
Update its state
Goto 2 until state does not change
Synchronous recall is also possible
24/48
Cognitive Modeling – Human and Machine Learning
Hopfield Nets as Associative Memory
25/48
Learning and Stability in Hopfield Nets
Cognitive Modeling – Human and Machine Learning
• Stability issues
• Input activation causes process of neuron activation updates
• Recurrency may cause divergence of activations
or periodic cycles of activation
• -> find weights wij so that activity converges into stable state for
certain input patterns
• Hopfield Nets utilize Hebb’s learning rule (training)

P
1
ss
• wij 
P is the number of training examples and i ≠ j
N
p 1 1 i j
• Preserves mutual excitation and
mutual inhibition of connected neurons
26/48
Cognitive Modeling – Human and Machine Learning
Energy state of Hopfield Net
• Thermodynamical model
 Each state of the net (activity of all neurons at certain point of
time) is associated with an energy value
 Prove stability of the system1
 E is a Lyapunov function, i.e. locally positive-definite function
 Energy always decreases when neuron states change
 If state is a local minimum in the energy function it is a stable
state for the network
 From any initial state of the network, the network recall algorithm
converges into a state, which is local minimum of E
 Local minima in energy function E correspond to states that are
‘near’ training examples

1See
www.linux-related.de/studium/knn/Hopfield_Konvergenz.pdf
for a coherent proof of convergence
27/48
Cognitive Modeling – Human and Machine Learning
B. F. Skinner
• B. F. Skinner (1904-1990)
• Radical Behaviorism
• Experiments “Skinner Box”
• Different stimuli
• Reward and punish actions of the rat
28/48
Operant Conditioning
• Operant Conditioning
Cognitive Modeling – Human and Machine Learning
• Behavioral changes due to positive or negative consequences of behavior
• Learning is based on reinforcement processes
• Reinforcement/punishment: in/decrease of probability of occurrence
•
Applied
Removed/avoided
Rewarding event
Positive reinforcement
-> strengthen behavior
Negative punishment
-> weakening behavior
Aversive event
Positive punishment
-> weakening behavior
Negative reinforcement
-> strengthen behavior
• Examples
•
•
•
•
Positive reinforcement: Good grade in the exam after learning hard
Negative reinforcement: Getting up earlier to avoid heavy traffic
Positive punishment: Detention after continuously talking in class
Negative punishment: No computer gaming after truancy
29/48
Cognitive Modeling – Human and Machine Learning
Reminder: Reinforcement Learning in ML
• Reinforcement Learning can be seen as a ML form of Operant
Conditioning (relationship between neurotransmitters and
rewards)
•
•
•
•
State space
S
Action space
A
Reward function
r: S x A → R
Goal: Find optimal policy π: S → A
 Policy that select the “best” action for each state
• → Markov Decision Process(MDP)
• Q-Learning
30/48
Cognitive Modeling – Human and Machine Learning
Biofeedback
• Application of Operant Conditioning
• Feedback loop between human and his/her biosignals (brain,
heart, muscle, breathing,…)
• Support humans in learning to control certain biophysiological activities
• Examples: Control heart rate, control certain brain activity
patterns associated with concentration, therapy of epilepsy,
headaches, attention-deficit disorder, …
31/48
Evolutionary Algorithms
Cognitive Modeling – Human and Machine Learning
• Evolution is a form of species learning / development
• Darvin: Improve due to survival of the fittest
• Lamarckism: Learned skills may be inherited
• Simulated evolution performed by genetic algorithms
• Population: Collection of hypotheses
• Hypotheses can also be algorithms (genetic programming),
neural networks, etc.
• Iterative computation of successor hypotheses (generations)
1.
2.
3.
4.
5.
Selection of fittest hypotheses as seed for next generation
Application of operations (breeding) on individual hypotheses,
e.g. random mutation, cross over, etc.
Evaluation of each individual hypothesis by fitness measure
Replace least-fit population with new individuals
Until termination condition is met
32/48