Download Weng_Schultz2004

Neural Coding of Basic Reward Terms of Animal
Learning Theory, Game Theory, Microeconomics and
Behavioral Ecology
Wolfram Schultz
Current Opinion in Neurobiology 2004, 14: 139-147
Qing Yu Christina Weng
October 10, 2008
Purpose
To summarize current knowledge of the basic components of reward
information extracted from environmental stimuli and processed by
reward mechanisms in the brain.
To explore neural correlates for the role of rewards in learning,
behavior, and decision-making.
Background
Animal Learning Theory:
Functions of rewards
1) Positive reinforcement
2) Induce consummatory behavior
3) Induce positive emotions
Learning requires:
1) a cause-and-affect association between a conditioned stimulus or
movement and the reinforcer.
2) Contingency
Types of reward-directed learning:
1) Pavlovian conditioning
2) Operant conditioning
Background
Motivational Value
Uncertainty: maximum at probability p = 0.5
assessed as entropy, variance,….
Negative Contrast Effect
Behavioral Theories and Terms
Learning Theory —attempts to describe how organisms react to rewards and
acquire new reactions to stimuli.
Microeconomics —assess value of rewards for decision-making
Game Theory —used in behavioral ecology; attempts to model behavior in
strategic situations.
Individuals will try to maximize utility of outcome and maintain stability
based on behavior of other players 
Nash Equilibrium— “efficient conditioned behavior” where players reach
optimal returns.
Reward Value = magnitude x probability
The Role of Expectation
Reward-predicting stimuli:
-Through learning, subjects are able to collect the award before it is
administered.
Reward expectation period:
-Some neurons maintain elevated activity
Expectation and Response
Increased neuronal activity
Increased preference
Increased expectation
Increased expected reward value 
elevated task-related activation of parietal neurons
Schematic forms of Reward Coding
(a) Receipt of award  change in neuronal activity (simplest form)
(b) Stimulus  neuronal response (reward prediction)
(c) Behavioral reaction toward stimulus  elevated neuronal activity  reward
(expectation)
Behavior-related Activation in Dopamine Neurons
The Effect of Probability
Dopamine neurons:
Increased probability of reward increased response to reward-predicting
stimulus
Parietal neurons:
Increased expected reward value  increased task-related activation
Track recently experienced rewards
Neuronal Response with Respect to Food and Liquid Rewards
Dopamine Neurons
Detect reward;
expectation
response
Discriminate among
stimuli on basis of
predicted rewards
Orbitofrontal cortex
Striatum/nucleus accumbens
Amygdala
Stronger task-related
activation in rewarded rather
than unrewarded trials.
Parietal cortex
Decision-making;
behavioral
organization
Dorsolateral prefrontal cortex
Mnemonic and movement
preparatory periods of
delayed response tasks
Preference Coding for Food Rewards in the Orbitofrontal Neuron
Neuronal Response with Respect to Food and Liquid Rewards
Posterior orbitofrontal cortex: physical characteristics of rewards
(ie. Glucose concentration, fat texture)
Parietal neurons: activity reflects subjective utility for the individual
animal rather than the expected value.
Experiment Results and Conclusions
Difference in activation increases with difference in value between the
unrewarded and rewarded trials.
A neural correlate exists for reward-based behavior
Utility of rewarding outcomes is coded following microeconomics
Reward Prediction Error
Dopamine
neurons
Overestimated
reward
Expected
reward
Underestimated
reward
Depressed
Unchanged
Activated
Neuronal Activation as a result of Unexpected and Failed Rewards
Unexpected
Reward
Orbitofrontal neurons
Absent
reward due to
organism
error
Dorsolateral
prefrontal
cortex
Anterior
cingulate
Striatal neurons
Posterior
cingulate
Frontal eye
fields
Neural-coding for goal-direction in the caudate nucleus neuron
Experiment Results and Conclusions
Reward information influences neural activations related to arm and
eye movements.
These influences reflect the reward and action-reward relationship
represented in goal-directed mechanisms.
The Underlying Mechanisms of Decision-Making
Game Theory:
Magnitude of reward
Probability
Utility
Race Model:
Threshold of neural activity  behavioral choice
Constantly incoming information  approach decision
Conclusions and Future Directions
Reward coding can be explained in terms of animal learning theory,
microeconomics, and game theory
Multiple mechanisms exist in different brain structures to extract reward
information from stimuli
Lingering Questions:
What are the brain processes that evaluate reward values to make
priority decisions?
What is the effect of delay of reward on reward value?
How is time coded?