Download Classical conditioning of instrumental conditioning?

CLASSICAL CONDITIONING OF
INSTRUMENTAL CONDITIONING?
WHOA…THEY ARE SUPPOSED TO BE SEPARATE!
THORNDIKE’S ROLE IN CC-OC CONNECTION
• Thorndike wrote that instrumental conditioning occurs in context of specific
environmental stimuli
• Cues involved that tell you “contingency in effect”
• Really a THREE term contingency: S,R and O
•
•
•
Stimulus
Response
Outcome
• S: RConsequence
• Stimulus can be an S+
• Stimulus can be an S• Can predict a reinforcer OR a punisher
S-R ASSOCIATION AND LAW OF EFFECT
• Contextual stimuli (S) predict the instrumental response contingency (R or RO).
• Is an association between the S and the R
• Is an association between the R and the O
• According to THORNDIKE: Role of the reinforcer is to “stamp in” or “strengthen”
the S-R connections
• NOT the RO contingency
• Motivation for engaging in RO contingency are the setting contextual stimuli (S)
• Data did NOT support this model
•
•
Association IS formed between the R and O
However, contextual cues are VERY important in operant conditioning
EXPECTANCY OF REWARD
• Reward expectancy can be a motivator for the operant response!
• Cues for this expectancy are classically conditioned
• Expectancy cues can be highly varied and complex
•
•
•
•
The situation
The individual(s)
Signal learning
Cues paired with RO contingency because they predict the contingency
• Clark Hull and Kenneth Spence:
• Behavior is result of interaction between organism and its environment.
• Environment provides the stimuli; The organism responds (all of which is observable)
• Hull’s theory = drive reduction theory
• Basic premise of their theory
• Instrumental response increases during conditioning because
• Presence of S evokes instrumental response through CC
• Instrumental response is emitted because of operant contingency
TWO PROCESS THEORY OF OPERANT CONDITIONING
• Assumes 2 distinct types of learning:
• Pavlovian/CC
• Instrumental/Operant
• Related together in special way:
•
•
•
Presence of stimuli (S) come to predict R-O relationship
R-O relationship is also strengthened
Thus S-O become connected
• Rescorla and Solomon assume S-O association activates emotional state
• Motivates the operant behavior
• This emotional state assumed to be positive or negative depending on the consequating
stimulus
TEST OF 2-PROCESS THEORY
• Pavlovian Instrumental Transfer experiment:
• Can counterbalance phase 1 and phase 2
• Critical transfer phase it the transfer test: Will the rat lever press to tone alone?
WHAT DO THE DATA SAY?
• Large research area
• Organisms respond more in presence of S if S is positive (appetitive)
• Organisms respond less in presence of S if S is negative (aversive)
• BUT: still respond in absence of the S
• Response interactions occur in Pavlovian Instrumental Transfer
• Evidence suggests do develop emotional responses to the S
• Also develop sign tracking behavior (now sign tracking begins to make sense!)
KRANK, ET. AL., 2008
• Rat study!
• 2 response levers, 1 on either side of water bottle
• Trained to press either response lever for drop of sweetened water, then ethanol
• CONC VI 20 sec VI 20 sec schedule
• 8 Pavlovian sessions:
• No response levers
• CS (light) of 10 sec on either left or right side (above lever hole)  0.2 ml of ethanol
• Unpaired group: CS and ethanol separated by 10 sec.
• Paired group: CS just before ethanol: GOT sign tracking
• Replaced levers and added Pavlovian Transfer Test
• CS light periodically presented while rats responding for ethanol; alternated over both levers
KRANK, ET. AL., 2008
• What happened?
• Prats pressed each response lever about 2x/min prior
•
•
to CS
Unpaired group: did not change much when CS
added
Paired group: significant increase in lever pressing
during CS presentation IF presented to same side as
training trials
• Why is this important?
• Shows that CS facilitated lever pressing
• Demonstrates importance of classical conditioning in
•
operant conditioning contingencies
Fairly specific as well- not just any CS, but the
correct-side CS
CONDITIONAL EMOTIONAL STATES? REWARD
SPECIFIC EXPECTANCIES?
• 2-process theory assumes classical conditioning mediates instrumental
conditioning through conditioning of positive or negative emotions
• BUT: also develop specific reward expectancies
• These reward specific expectancies can undermine the emotional conditioning
• Kruse, et. al., 1983:
•
•
Food pellets vs. sugar solution
•
Suggested rats formed specific expectancies about what the reward would be.
CS+ for food pellets elicited more instrumental responding when pellets were the reward than
when sugar and vice versa
R-O AND S(R-O) RELATIONSHIPS IMPORTANT
• HIERARCHICAL S(R-O) RELATIONS
• In addition to the simple associations of 2 elements (i.e., S-R, S-O, R-O), can develop hierarchical
•
associations
• the (S) signals the relationship between a response and its outcome S -> (R -> O)
the (S) becomes an occasion setter that signals when a specific response will be followed by a
specific reinforcer
• (S) can be a context or a specific cue
• EXPERIMENTAL APPROACH
• One (S) tone signals (R) lever push – (O) food
• Another (S) light signals (R) pull string – (O) sucrose
• Then switch the (S) – (R-O) combinations
• Animals confused by the switch- responding decreases
• Suggests that reward expectancies are formed
• Considerable support for S(R-O) relationships
WANTING VS NEEDING
• WANTING IS THE MOTIVATIONAL CONSEQUENCES OF REWARDS
• Is an underlying incentive salience
• Motivation to get the reward
• Wanting and Needs usually go together
• Needing the food that you Want
• WANTING CAN BE SEPARATED FROM NEEDS
• Drug addiction, gambling and overeating: wants or physiological needs
• Impulse control disorders where wanting has much more intensity
• Self-control study with children
• Get one marsh mellow now or two later
• Incentive salience is the psychological process underlying temptation
• Produces “surges of motivation to obtain and consume the reward”
WANTING VS NEEDING
•
INCENTIVE SALIENCE INTENSITY MODULATED BY
•
•
Needs: really are hungry
•
If needs are paired with emotional arousal and stress: Now stress
cues “need seeking” behavior
•
Needs can become “wants” when paired with emotional arousal
Individual personality differences important
• Different people have different levels of emotional arousal
and stress
•
Individual differences in types of pairing history.
WANTING VS NEEDING
• INCENTIVE SALIENCE OF SPECIFIC REWARDS CHANGES WITH EXPERIENCE
• Foods such as chocolate
• Drugs such as cocaine
• Reward system becomes sensitized
• High levels of responding to reward cues
• The sight of food, drugs or other incentives
• SENSITIZED INCENTIVE SALIENCE PRODUCES IMPULSIVE BEHAVIOR
• Difficult to exercise self-control
• External conditions such as stress reduce self-control
• HIGH LEVELS OF RELAPSE IN DRUG ADDICTS
• Not well explained by pleasure “liking” to get high
• Not well explained by withdrawal “avoiding” the discomfort
• Mostly cues with incentive salience that produce excessive wanting
WHAT IS ADDICTION?
• Physiologic dependence and withdrawal avoidance do not explain addiction
• Neurobiology of addiction attempts to explain the mechanisms by which drug
seeking behaviors are consolidated into compulsive use:
•
•
-long persistence of relapse risk
-drug-associated cues control behavior
ALTHOUGH ADDICTIVE DRUGS ARE
PHARMACOLOGICALLY DIVERSE…
• Stimulants (act as
a serotonin-norepinephrine-dopamine reuptake inhibitors)
• Cocaine, amphetamines, MDMA
• Opioids (agonist action)
• Heroin, morphine, oxycodone, fentanyl
• GABAergic agonists/modulators
• Alcohol, benzodiazepines, barbiturates
• Cannabis (binds cannabinoid receptors)
…THEY ALL LEAD TO A COMMON PATHWAY
• All addictive drugs pharmacologically
release dopamine in the nucleus
accumbens
• Why?
• Addiction is essentially a learned process
• Predictive cues along with operant response
• On of best examples of associative learning with instrumental conditioning.
REINFORCEMENT AND DOPAMINE?
• Olds and Milner:
• Brain stimulation = lots of behavior
• Animals would work until death to gain access to this brain stimulation
• Thought had discovered pleasure center:
•
•
Nucleus accumbens
Mesolimbic pathway
• Dopamine (DA) was neurotransmitter involved in these areas
THE DOPAMINE SYSTEM
REINFORCEMENT AND DA
•
EBS = electrical brain stimulation
•
•
•
•
releasing LOTS of dopamine (DA)
Results in lots of locomotion or exploratory behavior
Salamone and Schultz’ modern work has shown this release
DA release modulates “appetitive behavior”
•
•
•
Occurs in modes or modules related to terminal event
E.g., food modes, sex modes
Which mode depends on context of environment
• Search, capture, prepare, consume
• DA not affect consummatory behavior
DA IS REGULATED IN TWO WAYS:
• DA is released in pulses (phasic) and has an overall tone in synapse (tonic)
• Result is feedback system
• DA release in response to stimuli in environment
• Motivated or energize appropriate behavior
• Feedback system follows Rescorla Wagner model
DA REGULATION
•
DA also has a constant level or overall tone in the synapse:
•
•
•
•
•
More released when “surprised above what expected
Less when “surprised” below what expected
When get what expected- behavior is “well learned” and appears to become habit (not sure
how this works yet)
Think Rescorla Wagner model
Thus, fluctuations in DA as learn, and then serve as feedback regarding state of
environment
HOW RELATE TO CHOICE?
•
The value of the choice should affect the size of the DA burst
•
•
Experience can affect DA release
•
•
•
•
•
Small rewards = less DA release than larger rewards
Use to large rewards; less responsive to smaller rewards
Becomes a problem with gambling, other addicitons
Used to large bursts of DA via your gambling, drug, etc.
More typical rewards such as a paycheck, family, friends, etc., no longer elicit a response or
as large a response as the “addictive” responses
Question: can we reshape/retrain the brain?
THE DOPAMINE REWARD PATHWAY
HOW DOPAMINE LEADS TO BEHAVIOR CHANGE
• Dopamine required for natural stimuli (food, opportunity for mating, etc) to be
rewarding and drive behavior
• Natural rewards
and addictive drugs both cause dopamine release in the
Nucleus Acumbens
• Addictive drugs mimic effects of natural rewards and thus shape behavior
THE DOPAMINE REWARD PATHWAY
HOW DOPAMINE LEADS TO BEHAVIOR CHANGE
• Survival demands that organisms find and obtain needed resources (food,
shelter) and opportunity for mating despite risks -survival relevant goals
• These goals have natural “rewards” (eating, safety, sex)
• Behaviors with rewarding goals persist to a conclusion and increase over time as they are
positively reinforcing
• Dopamine is a “feedback” system”:
• If it is a rewarding behavior….then do it again!
• If it is not a rewarding behavior, don’t do it.
THE DOPAMINE REWARD PATHWAY
HOW DOPAMINE LEADS TO BEHAVIOR CHANGE
• Internal states (hunger) increase value of goal-related cues and increase
pleasure of consumption
• As internal states increase, seeking behavior for a resolution to that increases
• Thus: likelihood that complex behavioral sequence (hunting) will be brought to successful
conclusion
THE DOPAMINE REWARD PATHWAY
HOW DOPAMINE LEADS TO BEHAVIOR CHANGE
• Behavioral sequences involved in obtaining reward (steps required to hunt)
become overlearned/automatized
• Automatized behavioral repertoires can be activated by cues which are
predictive of reward
PREDICTION ERROR HYPOTHESIS:
SCHULTZ, 1999; 2005
• Exposure to an unexpected reward causes transient firing of dopamine
neurons which signals brain to learn a cue.
• Once cue is learned, burst of firing occurs at cue, not at reward.
•
• If the reward does not arrive, dopamine firing will decrease below baseline
levels  serves as an error signal about reward predictions
• If reward comes at unexpected time, dopamine firing will increase  positive
predictive error signal: “better than expected!”
• Remember Rescorla-Wagner!
DOPAMINE GATING HYPOTHESIS:
BERRINGER
•
Because drugs cause dopamine release (due to pharmacological actions), dopamine
firing upon use does not decay over time  brain repeatedly gets positive
predictive error signal: “better than expected!”
•
Drug cues become ubiquitous (drug cues difficult to extinguish)
•
Cues that predict drug availability take on enormous incentive salience (consolidates
drug seeking behavior)
•
Drug cues will become powerfully over-weighted compared to other choices
(contributes to loss of control over drug use)
CLINICAL IMPLICATIONS
• Addictive behaviors are a important and normal part of human
behavior
• Addictive drugs pharmacologically modify functioning of reward
circuits to overvalue drug rewards and reduce the comparative value
of other rewards
• Intention to stop use is not enough to stably quit substance use.
SO: TREATMENT OF ADDICTION
• First: detox…..get drugs out of system
• Next: work on breaking
• The operant response of seeking
• The cues that predict the drug
• Easy-peasy, correct?
IS RELAPSE HIGHER FOR DRUG ADDICTION?
• Yes, it is up there and comparable to other “behavioral” diseases
• How are addiction, hypertension and asthma behavioral disorders?
WHAT DOES WORK?
• Understanding the importance of environmental setting cues and conditions
• Drug paraphernalia itself
• Environmental cues: location, time of day, activities etc.
• People: Family, friends, etc.
• Stressors: what stressors are related to drug addiction
•
Two potential types of alchoholism:
•
•
•
Excitement seeking
Risk/aversion avoidance
Treatment of “triggers” will be very different
WHAT DOES WORK?
• Must rebuild new cues linked to positive behavior
•
•
•
•
New rules for living
New friends
New social skills
New skills for dealing with stressors, etc.
• Sober Living House programs are very effective
• Allow slow transition back into life
• Allows for learning to fluency the new skills and cues
IS THIS JUST TRUE FOR ADDICTION?
•
What about other “habits”?
•
•
•
Takes 2-3 weeks to change a habit
That is about how long it takes to change neural circuits
Any long term behavior change must involve change of cues as well as change in
behavior
•
•
•
Think of dieting, etc.
Transitioning of treatment programs: special schools placement back to regular schools
Others?