Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Dynamic Decision Problems: Hybrid Use of Decision Trees & System Dynamics Models Nathaniel Osgood (with some slides courtesy of Karen Yee) CMPT 858 March 24, 2011 1 Local Context • Saskatchewan suffered the highest incidence of WNV in Canada in 2003 and 2007 • Saskatoon Health Region (SHR) reported 6.5% and 25% of the provincial cases in 2003 and 2007, respectively 2 Mosquito species Culex tarsalis primarily responsible for spreading WNV in Saskatchewan Source: Penn State University 3 Adult Source: www.azstarnet.com/metro/295104 Terrestrial Pupa Aquatic Egg Raft Mosquito Lifecycle Source: http://www.flickr.com/photos/lor dv/207198441/ Source: www.comosquitocontrol.com/ Mosquito_Biology.html Larvae Source: unknown 4 Mosquito Environmental Dependencies For increasing their numbers: • Temperature (Average number of night temperatures above 15ºC; heat accumulated days) • Habitat availability • Rainfall 5 Human Dependencies For using protective measures: • Perceived risk • Knowledge of WNV • Temperature 6 Health Managers’ Dilemma Need to make decisions now taking into account uncertainties regarding: 1. Mosquito population • • abundance WNV prevalence 2. Environmental conditions • • current forecasted 3. Human behaviour 7 Different Levels of Challenge in Dynamic Decision-Making • Type A: Making complex dynamic choices given some expected/typical course of important factors outside our control – Here, the focus is centred on building models that help us understand the complex impact of our choices given this ‘expected course’ – Tough • Type B: Making complex dynamic choices when we can’t anticipate the course of the important factors outside our control – Focus on both dynamic model and adaptive planning given uncertainty – Tougher 8 Implications • Type A: When important exogenous conditions are known, we often seek to identify & stick to an ‘optimal’ pre-set plan – Don’t have to worry much about unfolding external conditions – they are known or unimportant Type B: Rather than putting “all our eggs in one basket”, it is typically best to avoid a pre-set plan, and instead to adaptively make choices over time – What we will do over time will depend on what is observed 9 Decision making under dynamic uncertainty: Adaptive Planning • Type A: When important exogenous conditions are known, we often seek to identify & stick to an The presentation focuses on this type ‘optimal’ pre-set plan of “dynamic decision” problems – Don’t have to worry much about unfolding external conditions – they are known or unimportant Type B: Rather than putting “all our eggs in one basket”, we typically seek to avoid a pre-set plan, and instead to adaptively make choices over time – What we will do over time will depend on what is observed 10 Adaptive Decision Problems: Relevant Questions How do we make decisions now, when the choice of the best decision depends so much on what plays out (unfolds) in things beyond our control? – Temperature trends – Precipitation – Prevalence of infection in migratory bird populations Should we make our decisions now despite these uncertainties? Or should we wait to see how things are trending before making decisions? 11 Characteristics of “Adaptive Decision” Problems • Can’t count on one particular future trajectory unfolding for things outside our control • Choosing decisions now requires considering the different possibilities of what might unfold in the future • We must make decisions over time, as we observe things unfold – The later we wait, the more information we’ll have • It may be advantageous to decide to “wait and see” as to how things play out until a later decision point 12 In these Conditions… What decision we make at a particular point in time will depend on Our current situation What we’ve observed as happening to this point (what we’ve “learned” – e.g. recent levels of growth) State (as given by stocks & derived quantities) Possible future eventualities, in light of what we’ve already seen (e.g. future levels of growth, given recent growth) Our possible decision points in the future • Here, we are balancing two desires: • To “seize the moment” and act early • To “wait and see” what happens, and decide on the basis of this 13 A Hybrid System Architecture to Address these “Tougher” Problems Simulation Model Events & Decisions Consequences Decision Tree 14 Introduction to Decision Trees • We will use decision trees both for – Diagrammatically illustrating decision making w/uncertainty – Quantitative reasoning • Represent – Flow of time – Decisions – Uncertainties (via events) – Consequences (deterministic or stochastic) Decision Tree Nodes • Decision (choice) Node • Chance (event) Node • Terminal (consequence) node Time Identifying the Optimal Decision Rule • To select decision rules, we perform a “rollback” of the tree (dynamic programming) – For terminal nodes, pass up value – For event nodes, pass up expected value of children – For decision nodes, select whichever child offers highest value and pass up that value for this node Example Tree Feature Decision Making Best Option Extended Example Extended Rule Decision Rules • Decision trees can be used to identify “optimal” decision rules – Remember: Optimality is in light of (simplified) assumptions! • A decision rule specify what we should do given any possible eventuality Decision Tree To Structure Policy Space Event node Time High later growth Expand capacity PL Low later growth High early growth 1-PL High later growth PE No expansion Decision node PL Low later growth Expand capacity 1-PL High later growth Expand capacity PL Low later growth Low early growth 1-PL 1-PE High later growth No expansion PL Low later growth Initial Decision 1-PL High later growth Expand capacity PL Low later growth High early growth 1-PL PE High later growth No expansion PL Low later growth No expansion 1-PL High later growth Expand capacity PL Low later growth Low early growth 1-PL High later growth 1-PE No Expansion PL Low later growth 1-PL Consequence1 Consequence2 Consequence3 Consequence4 Consequence5 Consequence6 Consequence7 Consequence8 Consequence9 Consequence10 Scenario Consequence11 Consequence12 Consequence13 Consequence14 Consequence15 Consequence16 Terminal Node (Consequence for Scenario) Terminology • A static decision rule pursues the same predetermined decisions (actions) regardless of eventualities • An adaptive decision rule varies its decisions (actions) based on which events have occurred • Observation: Static decision rules are rarely optimal A Static Decision Rule High later growth Expand capacity PL Low later growth High early growth 1-PL PE High later growth No expansion PL Low later growth Expand capacity 1-PL High later growth Expand capacity PL Low later growth Low early growth 1-PL High later growth 1-PE No expansion PL Low later growth Initial Decision 1-PL High later growth Expand capacity PL Low later growth High early growth 1-PL High later growth PE No expansion PL Low later growth No expansion 1-PL High later growth Expand capacity PL Low later growth Low early growth 1-PL 1-PE High later growth No Expansion PL Low later growth 1-PL Consequence1 Consequence2 Consequence3 Consequence4 Consequence5 Consequence6 Consequence7 Consequence8 Consequence9 Consequence10 Consequence11 Consequence12 Consequence13 Consequence14 Consequence15 Consequence16 Observation: The consequences observed at a particular terminal node are a function of the associated scenario (particular sequence of decisions and events on the path leading to that terminal node) – and are the same regardless as to which decision rule that gives rise to this sequence An Adaptive Decision Rule High later growth Expand capacity PL Low later growth High early growth 1-PL PE High later growth No expansion PL Low later growth Expand capacity 1-PL High later growth Expand capacity PL Low later growth Low early growth 1-PL High later growth 1-PE No expansion PL Low later growth Initial Decision 1-PL High later growth Expand capacity PL Low later growth High early growth 1-PL High later growth PE No expansion PL Low later growth No expansion 1-PL High later growth Expand capacity PL Low later growth Low early growth 1-PL 1-PE High later growth No Expansion PL Low later growth 1-PL Consequence1 Consequence2 Consequence3 Consequence4 Consequence5 Consequence6 Consequence7 Consequence8 Consequence9 Consequence10 Consequence11 Consequence12 Consequence13 Consequence14 Consequence15 Consequence16 Observation: The consequences observed at a particular terminal node are a function of the associated scenario (particular sequence of decisions and events on the path leading to that terminal node) – and are the same regardless as to which decision rule that gives rise to this sequence Analysis Using Decision Trees • Decision trees are a powerful analysis tool • Addition of symbolic components to decision trees greatly expand power • Example analytic techniques – Strategy selection – One-way and multi-way sensitivity analyses – Value of information Decision Tree w/Variables Risk Preference • People are not indifferent to uncertainty – Lack of indifference from uncertainty arises from uneven preferences for different outcomes – E.g. someone may • dislike losing $x far more than gaining $x • value gaining $x far more than they disvalue losing $x. • Individuals differ in comfort with uncertainty based on circumstances and preferences • Risk averse individuals will pay “risk premiums” to avoid uncertainty Risk Preference (Decision Tree Preview) Categories of Risk Attitudes • Risk attitude is a general way of classifying risk preferences • Classifications – Risk averse fear loss and seek sureness – Risk neutral are indifferent to uncertainty – Risk lovers hope to “win big” and don’t mind losing as much • Risk attitudes change over – Time – Circumstance Preference Function • Formally expresses a particular party’s degree of preference for (satisfaction with) different outcomes ($, time, level of conflict, quality…) • Can be systematically derived • Used to identify best decision when have uncertainty with respect to consequences – Choice with the highest mean preference is the best strategy for that particular party Challenge: Identify these Preference Functions • (On the Board) Risk Attitude in Preference Fns Identifying Preference Functions • Simple procedure to identify utility value associated with multiple outcomes • Interpolation between these data points defines the preference function Notion of a Risk Premium • A risk premium is the amount paid by a (risk averse) individual to avoid risk • Risk premiums are very common – what are some examples? – Insurance premiums – Higher fees paid by owner to reputable contractors – Higher charges by contractor for risky work – Lower returns from less risky investments – Money paid to ensure flexibility as guard against risk • Consider a risk averse individual with preference fn f faced with an investment c that provides Certainty Equivalent Example – 50% chance of earning $20000 – 50% chance of earning $0 • Average money from investment = – .5*$20,000+.5*$0=$10000 • Average satisfaction with the investment= .50 Mean satisfaction with investment .25 – .5*f($20,000)+.5*f($0)=.25 Certainty equivalent of investment • This individual would be willing to trade for a sure investment yielding satisfaction>.25 instead Mean value Of investment • We call this the certainty equivalent to the investment – Therefore this person should be willing to trade this investment for a sure amt of money>$5000 $5000 – Can get .25 satisfaction for a sure f-1(.25)=$5000 Example Cont’d (Risk Premium) • The risk averse individual would be willing to trade the uncertain investment c for any certain return which is > $5000 • Equivalently, the risk averse individual would be willing to pay another party an amount r up to $5000 =$10000-$5000 for other less risk averse party to guarantee $10,000 – Assuming the other party is not risk averse, that party wins because gain r on average – The risk averse individual wins b/c more satisfied Certainty Equivalent • More generally, consider situation in which have – Uncertainty with respect to consequence c – Non-linear preference function f • Note: E[X] is the mean (expected value) operator • The mean outcome of uncertain investment c is E[c] – In example, this was .5*$20,000+.5*$0=$10,000 • The mean satisfaction with the investment is E[f(c)] – In example, this was .5*f($20,000)+.5*f($0)=.25 • We call f-1(E[f(c)]) the certainty equivalent of c – Size of sure return that would give the same satisfaction as c – In example, was f-1(.25)=f-1(.5*20,000+.5*0)=$5,000 Risk Attitude Redux • The shapes of the preference functions means can classify risk attitude by comparing the certainty equivalent and expected value – For risk loving individuals, f-1(E[f(c)])>E[c] – For risk neutral individuals, f-1(E[f(c)])=E[c] – For risk averse individuals, f-1(E[f(c)])<E[c] Motivations for a Risk Premium • Consider – Risk averse individual A for whom f-1(E[f(c)])<E[c] – Less risk averse party B • A can lessen the effects of risk by paying a risk premium r of up to E[c]-f-1(E[f(c)]) to B in return for a guarantee of E[c] income – The risk premium shifts the risk to B – The net investment gain for A is E[c]-r, but A is more satisfied because E[c] – r > f-1(E[f(c)]) – B gets average monetary gain of r Multiple Attribute Decisions • Frequently we care about multiple attributes – Cost – Time – Quality – Relationship with owner • Terminal nodes on decision trees can capture these factors – but still need to make different attributes comparable WNV Hybrid Approach SD Model Decision Tree InterventionSelected Effects to be Factored in: Season/Temperature Dependence of Mosquito biting preferences (presumably through baby bird depletion). Late-season mosquitoes heading into Burrows to hibernate (this being more driven by length of day) <Average lifespan of a human> Vaccinated Humans Deaths of Vaccinated Humans <Mean Time to Waning Immunity> Rate of Loss Immunity of Vaccinal Humans Loss Immunity of Vaccinated Humans Mean Time to Waning Immunity Vaccine Rate Loss of Immunity of Recovered and WNV Immune Patients Vaccination Mean Time to Recover for Asymtomatically Infected Patients Recruitment rate of susceptible humans Humans Susceptible Humans Entrants of Humans Per Bite Probability of transmission from infected mosquito to human <Average lifespan of a human> Recovered and WNV Immune Patients Asymptomactically Infected Humans Recovery of Asymptomatically Infected Patients Asymptomatic Infection Deaths of Susecptible Humans Force of Infection for Humans <Fraction of Exposed Humans that remain asymptomatic> Newly Infected Pre-symptomatic Human Cases Deaths from Asymptomactically Infected Humans Deaths of Recovered and WNV Immune Patients Average lifespan of a human <Average lifespan of a human> <Number of bitings of susceptible humans by infected mosquitoes per day> Mean Time Until WNV Incubates in Humans Mean Time to Recover for PostHospital WNF Patients Number of Human Cases per Day Completing WNV Incubation Non-Hospitalized WNF Patients Under Recovery Exposed Humans Recovery of WNF Patients Progression to N on-Hospitalized WNF <Number of Human Cases per Day Completing WNV Incubation> Deaths of Exposed Humans Hospitalized for WNF <Fraction of Exposed Humans with WNF that are hospitalized> <Fraction of Exposed Humans with NonHospitalized WNF> Deaths of WNF Patients under Recovery <Average lifespan of a human> <Average lifespan of a human> Hospitalized WNV Patients Discharge of WNF Cases from Hospital <Average lifespan of a human> Deaths due to WNV Mean Time in Hospital for WN F Patients Deaths of Hospitalized WNV Patients WNV-induced death rate for humans <Progression to Non-Hospitalized WNF> <Hospitalized for WNF> CumulativeWNVSymptomaticCases New Symptomatic Cases NegativeOfCumulativeWNVSymptomaticCases User Interface 44 The Hybrid Approach: Critical Points 1. 2. 3. 4. 5. Is a framework geared toward an ongoing process of observation & decision making Captures uncertainties as time progresses Simulates a broad range of possibilities (e.g. for temperature) and not just a single scenario Allows for staging of decisions over different time points – including decisions to “wait & see” (exploiting future options) Could be used for diverse planning challenges (e.g. H1N1 given uncertainty regarding public reaction, vaccine availability) 45 Responsibilities in the Hybrid Approach Simulation Model Decision Tree • Calculates dynamic consequences of • Represents over time possible sequences of a sequence over time of • Uncertainties (event nodes) • Events • Decisions (decision nodes) • Choices • Takes care of deterministic simulation• Consequences (outcomes – e.g. Cost, quality of life, etc.) given events & decisions • Takes care of encapsulating • Capturing all uncertainties • “policy space” – where policies are made over time 46 Example: WNV Hybrid Approach Simulation Model • Mosquito lifecycle (includes temperature effects) • Bird lifecycle • Transmission between mosquitos & bird • Human infection & disease progression • Future: costs & resource use (via resource intensity weights, length of stay), quality of life Decision Tree • Decision options over time (source reduction, larvaciding, vaccination, wait & see) • Uncertainties (temperature) • Consequences (all WNV cases, severe neurological cases, costs, etc.) 47 Larvacide Reduction Death Rate of Larval Female Mosquitoes <InterventionSelected> Source Reduction Death Rate of Larval Female Mosquitoes Number of adult blood meals per day for Given Temperature Death Rate for Larval Female Mosquitoes Natural Death Rate of Larval Female Mosquitoes Number of adult blood meals per day Mean Time as Egg Death for Larval Female Mosquitoes Mosquitoes Larvacide Reduction Death Rate of Immature Female Mosqutioes Source Reduction Death Rate of Immature Female Mosquitoes Eggs Hatching to Mosquito Larvae Natural Death Rate of Immature Female Mosquitoes Mean Time to Larval Maturation for Given Temperature <CurrentTemperatureIn Centigrade> Source Reduction Death Rate of Pupal Female Mosquitoes Death rate for Pupal Female Mosquitoes Larvacide Reduction Death Rate of Pupal Female Mosquitoes Mean Time to Pupal Maturation Maturation of Pupae Birth Coefficient <Total Adult Female Mosquitoes> Virus Incubation Rate Pathogen Transmission From Infected Bird to Susceptible Mosquitoes Death of Susceptible Adult Female Mosquitoes Endogeneously Calculated Infectious Mosquitoes Disease Incubation Death of Exposed Female Adult Mosquitoes <Disease Transmission to Susceptible Mosquitoes Through Contact with Infectious Juvenile Birds (JB to M)> Death rate for Adult Mosquitoes Virus Incubation Threshold Temperature Extrinsic Incubation Period <Disease Transmission to Susceptible Mosquitoes Through Contact with Infectious Adult Birds (AB to M)> Exposed Adult Female Mosquitoes Susceptible Adult Female Mosquitoes <InterventionSelected> CurrentTemperatureInCentigrade Egg Laying by Adult Female Mosquitoes Density Egg to Mosquito Larva Ratio Pupae Female Mosquitoes Death for Pupal Female Mosquitoes Egg Laid per Blood Meal Egg Laying Rate for Adult Female Mosquitoes Eggs Female Eggs Hatching to Mosquito Larvae Maturation of Larvae Mean Time to Larval Maturation Natural Death Rate of Pupal Female Mosquitoes West Nile Virus SD Model Larval Female Mosquitoes Death of Infectious Mosquitoes <Death rate for Adult Mosquitoes> <Death rate for Adult Mosquitoes> Natural death rate for Adult Mosquitoes Adulticiding Death Rate InterventionSelected Effects to be Factored in: Season/Temperature Dependence of Mosquito biting preferences (presumably through baby bird depletion). Late-season mosquitoes heading into Burrows to hibernate (this being more driven by length of day) <Average lifespan of a human> Vaccinated Humans Deaths of Vaccinated Humans <Mean Time to Waning Immunity> Rate of Loss Immunity of Vaccinal Humans Loss Immunity of Vaccinated Humans Mean Time to Waning Immunity Vaccine Rate Loss of Immunity of Recovered and WNV Immune Patients Vaccination Mean Time to Recover for Asymtomatically Infected Patients Recruitment rate of susceptible humans Humans Susceptible Humans Entrants of Humans Per Bite Probability of transmission from infected mosquito to human <Average lifespan of a human> Recovered and WNV Immune Patients Asymptomactically Infected Humans Recovery of Asymptomatically Infected Patients Asymptomatic Infection Deaths of Susecptible Humans Force of Infection for Humans <Fraction of Exposed Humans that remain asymptomatic> Newly Infected Pre-symptomatic Human Cases Deaths from Asymptomactically Infected Humans Deaths of Recovered and WNV Immune Patients Average lifespan of a human <Average lifespan of a human> <Number of bitings of susceptible humans by infected mosquitoes per day> Mean Time to Recover for PostHospital WNF Patients Number of Human Cases per Day Completing WNV Incubation Non-Hospitalized WNF Patients Under Recovery Mean Time Until WNV Incubates in Humans Exposed Humans Recovery of WNF Patients Progression to Non-Hospitalized WNF <Average lifespan of a human> Deaths of Exposed Humans Hospitalized for WNF & Neurological Symptoms <Fraction of Exposed Humans with Neurological Symptoms that are hospitalized> <Fraction of Exposed Humans with WNF that are hospitalized> Hospitalized WNV Patients back WNV-induced death rate for humans <Average lifespan of a human> Discharge of WNF Cases from Hospital <Average lifespan of a human> Deaths due to WNV Deaths of WNF Patients under Recovery <Fraction of Exposed Humans with NonHospitalized WNF> Mean Time in Hospital for WNF Patients Deaths of Hospitalized WNV Patients 48 Decision Tree Current temp = 15oC Adulticiding = 3 …. same tree structure as the “do nothing” branch Current temp = 20oC Do Nothing = 0 Current temp = 30oC Larvaciding = 2 …. same tree structure as the “do nothing” branch Current temp = 15oC Larvaciding = 2 Current temp = 20oC Source Reduction = 1 …. same as below Current temp = 20oC Current temp = 30oC Do Nothing = 0 Current temp = 30oC Current temp = 20oC Current temp = 20oC Current temp = 20oC Larvaciding = 2 Do Nothing = 0 Current temp = 20oC Do Nothing = 0 Do Nothing = 0 Current temp = 30oC Current temp = 30oC Current temp = 20oC Larvaciding = 2 Current temp = 30oC Current temp = 30oC Current temp = 20oC Larvaciding = 2 Current temp = 30oC Current temp = 20oC Current temp = 20oC Adulticiding = 3 Current temp = 30oC Larvaciding = 2 Current temp = 20oC Larvaciding = 2 Current temp = 30oC Current temp = 30oC Current temp = 20oC Time (weeks) Adulticiding = 3 Week 1 back Week 2 49