Download Dynamic Decision Problems: Hybrid Use of Decision Trees

Dynamic Decision Problems:
Hybrid Use of Decision Trees &
System Dynamics Models
Nathaniel Osgood
(with some slides courtesy of Karen Yee)
CMPT 858
March 24, 2011
1
Local Context
• Saskatchewan suffered the highest
incidence of WNV in Canada in 2003 and
2007
• Saskatoon Health Region (SHR) reported
6.5% and 25% of the provincial cases in
2003 and 2007, respectively
2
Mosquito species Culex tarsalis primarily responsible for spreading
WNV in Saskatchewan
Source: Penn State University
3
Adult
Source:
www.azstarnet.com/metro/295104
Terrestrial
Pupa
Aquatic
Egg Raft
Mosquito
Lifecycle
Source:
http://www.flickr.com/photos/lor
dv/207198441/
Source:
www.comosquitocontrol.com/
Mosquito_Biology.html
Larvae
Source: unknown
4
Mosquito Environmental
Dependencies
For increasing their numbers:
• Temperature (Average number of night
temperatures above 15ºC; heat accumulated days)
• Habitat availability
• Rainfall
5
Human Dependencies
For using protective measures:
• Perceived risk
• Knowledge of WNV
• Temperature
6
Health Managers’ Dilemma
Need to make decisions now taking into account
uncertainties regarding:
1. Mosquito population
•
•
abundance
WNV prevalence
2. Environmental conditions
•
•
current
forecasted
3. Human behaviour
7
Different Levels of Challenge in
Dynamic Decision-Making
• Type A: Making complex dynamic choices given some
expected/typical course of important factors outside
our control
– Here, the focus is centred on building models that help us
understand the complex impact of our choices given this
‘expected course’
– Tough
• Type B: Making complex dynamic choices when we
can’t anticipate the course of the important factors
outside our control
– Focus on both dynamic model and adaptive planning given
uncertainty
– Tougher
8
Implications
• Type A: When important exogenous conditions are
known, we often seek to identify & stick to an
‘optimal’ pre-set plan
– Don’t have to worry much about unfolding external
conditions – they are known or unimportant
 Type B: Rather than putting “all our eggs in one basket”, it
is typically best to avoid a pre-set plan, and instead to
adaptively make choices over time
– What we will do over time will depend on what is
observed
9
Decision making under dynamic
uncertainty: Adaptive Planning
• Type A: When important exogenous conditions are
known, we often seek to identify & stick to an
The
presentation
focuses
on
this
type
‘optimal’ pre-set plan
of
“dynamic
decision”
problems
– Don’t have to worry much about unfolding external
conditions – they are known or unimportant
 Type B: Rather than putting “all our eggs in one basket”,
we typically seek to avoid a pre-set plan, and instead to
adaptively make choices over time
– What we will do over time will depend on what is
observed
10
Adaptive Decision Problems: Relevant
Questions
How do we make decisions now, when the choice of
the best decision depends so much on what plays
out (unfolds) in things beyond our control?
– Temperature trends
– Precipitation
– Prevalence of infection in migratory bird populations
Should we make our decisions now despite these
uncertainties? Or should we wait to see how things
are trending before making decisions?
11
Characteristics of “Adaptive Decision”
Problems
• Can’t count on one particular future trajectory
unfolding for things outside our control
• Choosing decisions now requires considering the
different possibilities of what might unfold in the
future
• We must make decisions over time, as we observe
things unfold
– The later we wait, the more information we’ll have
• It may be advantageous to decide to “wait and see”
as to how things play out until a later decision point
12
In these Conditions…

What decision we make at a particular point in time will
depend on

Our current situation




What we’ve observed as happening to this point (what we’ve “learned” –
e.g. recent levels of growth)
State (as given by stocks & derived quantities)
Possible future eventualities, in light of what we’ve already seen
(e.g. future levels of growth, given recent growth)
Our possible decision points in the future
• Here, we are balancing two desires:
• To “seize the moment” and act early
• To “wait and see” what happens, and decide on the basis of this
13
A Hybrid System Architecture to
Address these “Tougher” Problems
Simulation
Model
Events & Decisions
Consequences
Decision Tree
14
Introduction to Decision Trees
• We will use decision trees both for
– Diagrammatically illustrating decision making
w/uncertainty
– Quantitative reasoning
• Represent
– Flow of time
– Decisions
– Uncertainties (via events)
– Consequences (deterministic or stochastic)
Decision Tree Nodes
• Decision (choice) Node
• Chance (event) Node
• Terminal (consequence) node
Time
Identifying the Optimal Decision Rule
• To select decision rules, we perform a
“rollback” of the tree (dynamic programming)
– For terminal nodes, pass up value
– For event nodes, pass up expected value of
children
– For decision nodes, select whichever child offers
highest value and pass up that value for this node
Example Tree
Feature Decision Making
Best Option
Extended Example
Extended Rule
Decision Rules
• Decision trees can be used to identify
“optimal” decision rules
– Remember: Optimality is in light of (simplified)
assumptions!
• A decision rule specify what we should do
given any possible eventuality
Decision Tree To Structure Policy Space
Event node
Time
High later growth
Expand capacity
PL
Low later growth
High early growth
1-PL
High later growth
PE
No expansion
Decision node
PL
Low later growth
Expand capacity
1-PL
High later growth
Expand capacity
PL
Low later growth
Low early growth
1-PL
1-PE
High later growth
No expansion
PL
Low later growth
Initial Decision
1-PL
High later growth
Expand capacity
PL
Low later growth
High early growth
1-PL
PE
High later growth
No expansion
PL
Low later growth
No expansion
1-PL
High later growth
Expand capacity
PL
Low later growth
Low early growth
1-PL
High later growth
1-PE
No Expansion
PL
Low later growth
1-PL
Consequence1
Consequence2
Consequence3
Consequence4
Consequence5
Consequence6
Consequence7
Consequence8
Consequence9
Consequence10
Scenario
Consequence11
Consequence12
Consequence13
Consequence14
Consequence15
Consequence16
Terminal
Node
(Consequence
for Scenario)
Terminology
• A static decision rule pursues the same
predetermined decisions (actions) regardless of
eventualities
• An adaptive decision rule varies its decisions
(actions) based on which events have occurred
• Observation: Static decision rules are rarely
optimal
A Static Decision Rule
High later growth
Expand capacity
PL
Low later growth
High early growth
1-PL
PE
High later growth
No expansion
PL
Low later growth
Expand capacity
1-PL
High later growth
Expand capacity
PL
Low later growth
Low early growth
1-PL
High later growth
1-PE
No expansion
PL
Low later growth
Initial Decision
1-PL
High later growth
Expand capacity
PL
Low later growth
High early growth
1-PL
High later growth
PE
No expansion
PL
Low later growth
No expansion
1-PL
High later growth
Expand capacity
PL
Low later growth
Low early growth
1-PL
1-PE
High later growth
No Expansion
PL
Low later growth
1-PL
Consequence1
Consequence2
Consequence3
Consequence4
Consequence5
Consequence6
Consequence7
Consequence8
Consequence9
Consequence10
Consequence11
Consequence12
Consequence13
Consequence14
Consequence15
Consequence16
Observation:
The consequences
observed at a
particular terminal
node are a function of
the associated
scenario (particular
sequence of
decisions and events
on the path leading
to that terminal node)
– and are the same
regardless as to which
decision rule that gives
rise to this sequence
An Adaptive Decision Rule
High later growth
Expand capacity
PL
Low later growth
High early growth
1-PL
PE
High later growth
No expansion
PL
Low later growth
Expand capacity
1-PL
High later growth
Expand capacity
PL
Low later growth
Low early growth
1-PL
High later growth
1-PE
No expansion
PL
Low later growth
Initial Decision
1-PL
High later growth
Expand capacity
PL
Low later growth
High early growth
1-PL
High later growth
PE
No expansion
PL
Low later growth
No expansion
1-PL
High later growth
Expand capacity
PL
Low later growth
Low early growth
1-PL
1-PE
High later growth
No Expansion
PL
Low later growth
1-PL
Consequence1
Consequence2
Consequence3
Consequence4
Consequence5
Consequence6
Consequence7
Consequence8
Consequence9
Consequence10
Consequence11
Consequence12
Consequence13
Consequence14
Consequence15
Consequence16
Observation:
The consequences
observed at a
particular terminal
node are a function of
the associated
scenario (particular
sequence of
decisions and events
on the path leading
to that terminal node)
– and are the same
regardless as to which
decision rule that gives
rise to this sequence
Analysis Using Decision Trees
• Decision trees are a powerful analysis tool
• Addition of symbolic components to decision
trees greatly expand power
• Example analytic techniques
– Strategy selection
– One-way and multi-way sensitivity analyses
– Value of information
Decision Tree w/Variables
Risk Preference
• People are not indifferent to uncertainty
– Lack of indifference from uncertainty arises from
uneven preferences for different outcomes
– E.g. someone may
• dislike losing $x far more than gaining $x
• value gaining $x far more than they disvalue losing $x.
• Individuals differ in comfort with uncertainty
based on circumstances and preferences
• Risk averse individuals will pay “risk
premiums” to avoid uncertainty
Risk Preference
(Decision Tree Preview)
Categories of Risk Attitudes
• Risk attitude is a general way of classifying risk
preferences
• Classifications
– Risk averse fear loss and seek sureness
– Risk neutral are indifferent to uncertainty
– Risk lovers hope to “win big” and don’t mind
losing as much
• Risk attitudes change over
– Time
– Circumstance
Preference Function
• Formally expresses a particular party’s degree
of preference for (satisfaction with) different
outcomes ($, time, level of conflict, quality…)
• Can be systematically derived
• Used to identify best decision when have
uncertainty with respect to consequences
– Choice with the highest mean preference is the
best strategy for that particular party
Challenge:
Identify these Preference Functions
• (On the Board)
Risk Attitude in Preference Fns
Identifying Preference Functions
• Simple procedure to identify utility value
associated with multiple outcomes
• Interpolation between these data points
defines the preference function
Notion of a Risk Premium
• A risk premium is the amount paid by a (risk
averse) individual to avoid risk
• Risk premiums are very common – what are
some examples?
– Insurance premiums
– Higher fees paid by owner to reputable
contractors
– Higher charges by contractor for risky work
– Lower returns from less risky investments
– Money paid to ensure flexibility as guard against
risk
• Consider a risk averse individual with
preference fn f faced with an investment c
that provides
Certainty
Equivalent
Example
– 50% chance of earning $20000
– 50% chance of earning $0
• Average money from investment =
– .5*$20,000+.5*$0=$10000
• Average satisfaction with the investment=
.50
Mean satisfaction with
investment
.25
– .5*f($20,000)+.5*f($0)=.25
Certainty equivalent
of investment
• This individual would be willing to trade for
a sure investment yielding satisfaction>.25
instead
Mean value
Of investment
• We call this the certainty equivalent to the
investment
– Therefore this person should be willing to trade
this investment for a sure amt of money>$5000
$5000
– Can get .25 satisfaction for a sure f-1(.25)=$5000
Example Cont’d (Risk Premium)
• The risk averse individual would be willing to
trade the uncertain investment c for any
certain return which is > $5000
• Equivalently, the risk averse individual would
be willing to pay another party an amount r
up to $5000 =$10000-$5000 for other less risk
averse party to guarantee $10,000
– Assuming the other party is not risk averse, that
party wins because gain r on average
– The risk averse individual wins b/c more satisfied
Certainty Equivalent
• More generally, consider situation in which have
– Uncertainty with respect to consequence c
– Non-linear preference function f
• Note: E[X] is the mean (expected value) operator
• The mean outcome of uncertain investment c is E[c]
– In example, this was .5*$20,000+.5*$0=$10,000
• The mean satisfaction with the investment is E[f(c)]
– In example, this was .5*f($20,000)+.5*f($0)=.25
• We call f-1(E[f(c)]) the certainty equivalent of c
– Size of sure return that would give the same satisfaction as
c
– In example, was f-1(.25)=f-1(.5*20,000+.5*0)=$5,000
Risk Attitude Redux
• The shapes of the preference functions means
can classify risk attitude by comparing the
certainty equivalent and expected value
– For risk loving individuals, f-1(E[f(c)])>E[c]
– For risk neutral individuals, f-1(E[f(c)])=E[c]
– For risk averse individuals, f-1(E[f(c)])<E[c]
Motivations for a Risk Premium
• Consider
– Risk averse individual A for whom f-1(E[f(c)])<E[c]
– Less risk averse party B
• A can lessen the effects of risk by paying a risk
premium r of up to E[c]-f-1(E[f(c)]) to B in
return for a guarantee of E[c] income
– The risk premium shifts the risk to B
– The net investment gain for A is E[c]-r, but A is
more satisfied because E[c] – r > f-1(E[f(c)])
– B gets average monetary gain of r
Multiple Attribute Decisions
• Frequently we care about multiple attributes
– Cost
– Time
– Quality
– Relationship with owner
• Terminal nodes on decision trees can capture
these factors – but still need to make different
attributes comparable
WNV Hybrid Approach
SD Model
Decision Tree
InterventionSelected
Effects to be Factored in: Season/Temperature
Dependence of Mosquito biting preferences
(presumably through baby bird depletion). Late-season
mosquitoes heading into Burrows to hibernate (this being
more driven by length of day)
<Average lifespan of
a human>
Vaccinated Humans
Deaths of
Vaccinated
Humans
<Mean Time to
Waning Immunity>
Rate of Loss Immunity
of Vaccinal Humans
Loss Immunity of
Vaccinated Humans
Mean Time to
Waning Immunity
Vaccine Rate
Loss of Immunity of Recovered and
WNV Immune Patients
Vaccination
Mean Time to Recover for
Asymtomatically Infected
Patients
Recruitment rate of
susceptible humans
Humans
Susceptible Humans
Entrants of
Humans
Per Bite Probability of
transmission from infected
mosquito to human
<Average lifespan of
a human>
Recovered and WNV Immune
Patients
Asymptomactically Infected Humans
Recovery of
Asymptomatically Infected
Patients
Asymptomatic
Infection
Deaths of
Susecptible Humans
Force of Infection
for Humans
<Fraction of Exposed
Humans that remain
asymptomatic>
Newly Infected
Pre-symptomatic Human
Cases
Deaths from
Asymptomactically
Infected Humans
Deaths of Recovered and
WNV Immune Patients
Average lifespan
of a human
<Average lifespan of
a human>
<Number of bitings of
susceptible humans by infected
mosquitoes per day>
Mean Time Until WNV
Incubates in Humans
Mean Time to
Recover for
PostHospital WNF
Patients
Number of Human Cases per
Day Completing WNV
Incubation
Non-Hospitalized WNF Patients Under
Recovery
Exposed Humans
Recovery of
WNF Patients
Progression to N on-Hospitalized WNF
<Number of Human Cases
per Day Completing WNV
Incubation>
Deaths of Exposed
Humans
Hospitalized for
WNF
<Fraction of Exposed
Humans with WNF that
are hospitalized>
<Fraction of Exposed Humans
with NonHospitalized WNF>
Deaths of WNF
Patients under
Recovery
<Average lifespan of
a human>
<Average lifespan of
a human>
Hospitalized WNV Patients
Discharge of WNF
Cases from Hospital
<Average lifespan of
a human>
Deaths due to
WNV
Mean Time in Hospital
for WN F Patients
Deaths of Hospitalized
WNV Patients
WNV-induced death
rate for humans
<Progression to
Non-Hospitalized
WNF>
<Hospitalized for
WNF>
CumulativeWNVSymptomaticCases
New Symptomatic
Cases
NegativeOfCumulativeWNVSymptomaticCases
User Interface
44
The Hybrid Approach: Critical Points
1.
2.
3.
4.
5.
Is a framework geared toward an ongoing process of
observation & decision making
Captures uncertainties as time progresses
Simulates a broad range of possibilities (e.g. for temperature)
and not just a single scenario
Allows for staging of decisions over different time points –
including decisions to “wait & see” (exploiting future options)
Could be used for diverse planning challenges (e.g. H1N1 given
uncertainty regarding public reaction, vaccine availability)
45
Responsibilities in the Hybrid
Approach
Simulation Model
Decision Tree
• Calculates dynamic consequences of • Represents over time possible
sequences of
a sequence over time of
• Uncertainties (event nodes)
• Events
• Decisions (decision nodes)
• Choices
• Takes care of deterministic simulation• Consequences (outcomes – e.g.
Cost, quality of life, etc.)
given events & decisions
• Takes care of encapsulating
• Capturing all uncertainties
• “policy space” – where policies
are made over time
46
Example: WNV Hybrid Approach
Simulation Model
• Mosquito lifecycle (includes temperature
effects)
• Bird lifecycle
• Transmission between mosquitos & bird
• Human infection & disease progression
• Future: costs & resource use (via resource
intensity weights, length
of stay), quality of life
Decision Tree
• Decision options over time (source
reduction, larvaciding, vaccination,
wait & see)
• Uncertainties (temperature)
• Consequences (all WNV cases, severe
neurological cases, costs, etc.)
47
Larvacide Reduction Death
Rate of Larval Female
Mosquitoes
<InterventionSelected>
Source Reduction Death
Rate of Larval Female
Mosquitoes
Number of adult blood
meals per day for Given
Temperature
Death Rate for Larval
Female Mosquitoes
Natural Death Rate of
Larval Female
Mosquitoes
Number of adult
blood meals per
day
Mean Time as Egg
Death for Larval
Female Mosquitoes
Mosquitoes
Larvacide Reduction Death
Rate of Immature Female
Mosqutioes
Source Reduction Death
Rate of Immature Female
Mosquitoes
Eggs Hatching to
Mosquito Larvae
Natural Death Rate of
Immature Female
Mosquitoes
Mean Time to Larval
Maturation for Given
Temperature
<CurrentTemperatureIn
Centigrade>
Source Reduction Death
Rate of Pupal Female
Mosquitoes
Death rate for Pupal
Female Mosquitoes
Larvacide Reduction Death
Rate of Pupal Female
Mosquitoes
Mean Time to
Pupal
Maturation
Maturation of
Pupae
Birth Coefficient
<Total Adult Female
Mosquitoes>
Virus Incubation
Rate
Pathogen Transmission
From Infected Bird to
Susceptible Mosquitoes
Death of Susceptible
Adult Female
Mosquitoes
Endogeneously
Calculated Infectious
Mosquitoes
Disease Incubation
Death of Exposed
Female Adult
Mosquitoes
<Disease Transmission to
Susceptible Mosquitoes Through
Contact with Infectious Juvenile
Birds (JB to M)>
Death rate for Adult
Mosquitoes
Virus Incubation
Threshold
Temperature
Extrinsic
Incubation Period
<Disease Transmission to
Susceptible Mosquitoes Through
Contact with Infectious Adult
Birds (AB to M)>
Exposed Adult
Female Mosquitoes
Susceptible Adult
Female Mosquitoes
<InterventionSelected>
CurrentTemperatureInCentigrade
Egg Laying by Adult
Female Mosquitoes
Density
Egg to Mosquito
Larva Ratio
Pupae Female
Mosquitoes
Death for Pupal
Female Mosquitoes
Egg Laid per
Blood Meal
Egg Laying Rate for Adult
Female Mosquitoes
Eggs
Female Eggs Hatching
to Mosquito Larvae
Maturation of
Larvae
Mean Time to
Larval Maturation
Natural Death Rate of
Pupal Female
Mosquitoes
West Nile
Virus SD
Model
Larval Female
Mosquitoes
Death of Infectious
Mosquitoes
<Death rate for Adult
Mosquitoes>
<Death rate for Adult
Mosquitoes>
Natural death rate for
Adult Mosquitoes
Adulticiding
Death Rate
InterventionSelected
Effects to be Factored in: Season/Temperature
Dependence of Mosquito biting preferences
(presumably through baby bird depletion). Late-season
mosquitoes heading into Burrows to hibernate (this being
more driven by length of day)
<Average lifespan of
a human>
Vaccinated Humans
Deaths of
Vaccinated
Humans
<Mean Time to
Waning Immunity>
Rate of Loss Immunity
of Vaccinal Humans
Loss Immunity of
Vaccinated Humans
Mean Time to
Waning Immunity
Vaccine Rate
Loss of Immunity of Recovered and
WNV Immune Patients
Vaccination
Mean Time to Recover for
Asymtomatically Infected
Patients
Recruitment rate of
susceptible humans
Humans
Susceptible Humans
Entrants of
Humans
Per Bite Probability of
transmission from infected
mosquito to human
<Average lifespan of
a human>
Recovered and WNV Immune
Patients
Asymptomactically Infected Humans
Recovery of
Asymptomatically Infected
Patients
Asymptomatic
Infection
Deaths of
Susecptible Humans
Force of Infection
for Humans
<Fraction of Exposed
Humans that remain
asymptomatic>
Newly Infected
Pre-symptomatic Human
Cases
Deaths from
Asymptomactically
Infected Humans
Deaths of Recovered and
WNV Immune Patients
Average lifespan
of a human
<Average lifespan of
a human>
<Number of bitings of
susceptible humans by infected
mosquitoes per day>
Mean Time to
Recover for
PostHospital WNF
Patients
Number of Human Cases per
Day Completing WNV
Incubation
Non-Hospitalized WNF Patients Under
Recovery
Mean Time Until WNV
Incubates in Humans
Exposed Humans
Recovery of
WNF Patients
Progression to
Non-Hospitalized WNF
<Average lifespan of
a human>
Deaths of
Exposed
Humans Hospitalized for WNF
& Neurological
Symptoms
<Fraction of Exposed
Humans with Neurological
Symptoms that are
hospitalized>
<Fraction of Exposed
Humans with WNF that
are hospitalized>
Hospitalized WNV Patients
back
WNV-induced death
rate for humans
<Average lifespan of
a human>
Discharge of WNF
Cases from Hospital
<Average lifespan of
a human>
Deaths due to
WNV
Deaths of WNF
Patients under
Recovery
<Fraction of Exposed Humans
with NonHospitalized WNF>
Mean Time in Hospital
for WNF Patients
Deaths of Hospitalized
WNV Patients
48
Decision Tree
Current temp = 15oC
Adulticiding = 3
…. same tree structure as the “do
nothing” branch
Current temp = 20oC
Do Nothing = 0
Current temp = 30oC
Larvaciding = 2
…. same tree structure as the “do nothing”
branch
Current temp = 15oC
Larvaciding = 2
Current temp = 20oC
Source Reduction = 1
…. same as below
Current temp = 20oC
Current temp = 30oC
Do Nothing = 0
Current temp = 30oC
Current temp = 20oC
Current temp = 20oC
Current temp = 20oC
Larvaciding = 2
Do Nothing = 0
Current temp = 20oC
Do Nothing = 0
Do Nothing = 0
Current temp = 30oC
Current temp = 30oC
Current temp = 20oC
Larvaciding = 2
Current temp = 30oC
Current temp = 30oC
Current temp = 20oC
Larvaciding = 2
Current temp = 30oC
Current temp = 20oC
Current temp = 20oC
Adulticiding = 3
Current temp = 30oC
Larvaciding = 2
Current temp = 20oC
Larvaciding = 2
Current temp = 30oC
Current temp = 30oC
Current temp = 20oC
Time (weeks)
Adulticiding = 3
Week 1
back
Week 2
49