Download Meeting the Future in Managing Chronic Disorders

Dynamic Treatment Regimes
S.A. Murphy
CASBS
November 2, 2007
1
Outline
–
–
–
–
Three apparently dissimilar problems
Myopic decision making
Constructing strategies
Challenges
• Unknown, unobserved causes
• Small, expensive data sets
– Discussion
2
Three Apparently Dissimilar
Problems
– Artificial Intelligence: Autonomous Helicopter
Flight
– Management of Substance Abuse/Mental
Illness
– Management of a Welfare Program
3
Artificial Intelligence
• Autonomous Helicopter Flight
– Observations: characteristics of the helicopter (position,
orientation, velocity, angular velocity, ….),
characteristics of the environment (wind speed, wind
angle, turbulence….)
– Actions/treatments: cyclic pitch (causes
forward/backward and sideways acceleration), tilt angle
of main rotor blades (direction), tail rotor pitch control
(turning)
– Rewards: Closeness of helicopter’s flight path to the
desired path; avoidance of crashes(!)
4
Andrew Ng’s Helicopter: http://ai.stanford.edu/~ang/
5
The Management of Substance
Abuse/Mental Illness
• Treating Patients with Opioid Dependence
(heroin)
– Observations: individual characteristics (withdrawal
symptoms, craving, attendance at counseling sessions,
results of urine tests….), characteristics of the
environment (housing, employment.…)
– Actions/treatments: methadone dose, amount of
weekly group counseling sessions, daily dosing time of
methadone, individual counseling sessions, methadone
taper
– Rewards: minimize opioid use and maximize
health/functionality, minimize cost
6
http://www.nida.nih.gov/perspectives/vol1no1.html 7
Management of a Welfare Program
• “Jobs First” Program in Connecticut
– Observations: individual characteristics (assets, income,
age, health, employment), characteristics of the
environment (domestic violence, incapacitated family
member, # children, living arrangements…)
– Actions/treatments: child care, job search skills
training, amount of cash benefit, medical assistance,
education
– Rewards: maximize employment/independence.
8
9
The Common Thread: Multi-Stage
Decision Making
• Observation, action, observation, action,
observation, action,…………………….
• A strategy tells us how to use the
observations to choose the actions.
• We’d like to develop strategies that
maximize the rewards.
10
Role of the Statistician
• What kinds of data are most useful for developing
strategies?
• How do we use limited and expensive data to
construct good strategies?
• How do we evaluate strategies using the limited
data?
(A strategy tells us how to use the observations to
choose the actions.)
11
Outline
–
–
–
–
Three apparently dissimilar problems
Myopic decision making
Constructing strategies
Challenges
• Unknown, unobserved causes
• Small, expensive data sets
– Discussion
12
Myopic Decision Making
• In myopic decision making, decision makers use strategies
that seek to maximize immediate rewards. Problems:
– Ignore longer term consequences of present actions.
– Ignore the range of feasible future actions/treatments
– Ignore the fact that immediate responses to present actions
may yield information that pinpoints best future actions
•
(A strategy tells us how to use the observations to choose
the actions.)
13
Autonomous Helicopter Flight
The helicopter has veered from flight plan.
•
Myopic action: Choose an acceleration and direction that will
ASAP bring us back to the flight plan.
•
The result: The myopic action results in the helicopter overshooting
the planned flight path and in drastic situations may lead to the
helicopter cycling out of control.
•
The mistake: We did not consider the range of actions we can take
following the initial action. The ability to slow down is
mechanically limited.
•
The message: Use an acceleration that will not return us as quickly
to the planned flight path but will take into account the ability of the
helicopter to slow down and reduce the overshoot.
14
Treatment of Psychosis
•
Myopic action: Offer patients a treatment that reduces
psychosis for as many people as possible.
•
The result: Some patients are not helped and/or experience
abnormal movements of the voluntary muscles (TDs). The
class of subsequent medications is greatly reduced.
•
The mistake: We should have taken into account the variety
of treatments available to those for whom the first treatment is
ineffective.
•
The message: Use an initial medication that may not have as
large a success rate but that will be less likely to cause TDs.
15
Treatment of Opioid Dependence
•
Myopic action: Choose an intensive multi-component
treatment (methadone + counseling + behavioral
contingencies) that immediately reduces opioid use for as
many people as possible.
•
The result: Behavioral contingencies are
burdensome/expensive to implement and many people may
not need the contingencies to improve.
•
The mistake: We should allow the patient to exhibit poor
adherence prior to implementing the behavioral
contingencies.
•
The message: Use an initial treatment that may not have as
large an immediate success rate but carefully monitor patient
adherence to ascertain if behavioral contingencies are
16
required.
Outline
–
–
–
–
Three apparently dissimilar problems
Myopic decision making
Constructing strategies
Challenges
• Unknown, unobserved causes
• Small, expensive data sets
– Discussion
17
Basic Idea for Constructing a Strategy:
Move Backwards Through Time.
Observations
Treatment
(Action)
Time 1
Observations
Time 2
(Pretend you are “All-Knowing”)
Treatment
(Action)
Time 2
Reward
Time 3
18
Outline
–
–
–
–
Three apparently dissimilar problems
Myopic decision making
Constructing strategies
Challenges
• Unknown, unobserved causes (e.g. how data might
mislead you)
• Small, expensive data sets
– Discussion
19
Artificial Intelligence
• Scientists who construct strategies in autonomous
helicopter flight use mechanistic theory (physical laws:
momentum=m*v, W=F*d*cos(θ)…) to model the
interrelationships between observations and how the
actions might impact .
– Scientists know many (most?) of the causes of the
observations and know how the observations relate to one
another.
• Scientists can quickly evaluate strategies for selecting the
actions (within a matter of months).
20
Comparatively Less Known
Mechanistic Models in
Behavioral/Social/Medical Sciences
• Scientists who want to use data on
individuals to construct strategies must
confront the fact that non-causal
“associations” occur due to the unknown
causes of the observations.
21
Conceptual Structure in the
Behavioral/Social/Medical Sciences
Unknown
Causes
Observations
Unknown
Causes
Treatment
Time 1
Observations
Time 2
Treatment
Time 2
Reward
Time 3
22
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Maturity/
Decision
to join "Adult"
Society
Unknown
Causes
+
+
Binge Drinking
Treatment
Time 1
Binge Drinking
Time 2
Counseling
Time 2
Functionality
Time 3
23
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
• Problem: Non-causal associations between
treatment (here counseling) and rewards are
likely.
• Solution: Construct strategies using data
sets in which randomization is used to
assign treatments to students. This breaks
the non-causal associations yet permits
causal associations.
24
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Maturity/
Decision
to join "Adult"
Society
Unknown
Causes
"+"
Observations
Treatment
Time 1
Binge Drinking
Time 2
Counseling
Time 2
Functionality
Time 3
25
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Maturity/
Decision
to join "Adult"
Society
Unknown
Causes
+
-
Binge Drinking
Yes
Counseling on
Health
Consequences
Yes/No
-
Binge Drinking
Yes/No
Time 2
Sanctions
+ counseling
Yes/No
Functionality
Time 3
26
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Unknown
Causes
High SAT
Scores
+
+
Observations
Student
is a superior
athlete
+
Student
admitted to
University
Treatment
Time 2
Grades
27
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
• The problem: Even when treatments are
randomized, non-causal associations occur in the
data.
• The solution: Statistical methods for constructing
strategies must be conducted in stages as opposed
to “all-at-once.” Statistical methods should
appropriately “average” over the non-causal
associations between treatment and reward.
28
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Maturity/
Decision
to join "Adult"
Society
Unknown
Causes
+
-
Binge Drinking
Yes
Counseling on
Health
Consequences
Yes/No
-
Binge Drinking
Yes/No
Time 2
Sanctions
+ counseling
Yes/No
Functionality
Time 3
29
Summary of Solutions To Causal
Problems
• Experiments should randomize treatments (e.g.
actions).
• Develop statistical methods that avoid being
influenced by non-causal associations yet help you
construct the strategy.
• Subjects in your data should be representative of
population of subjects.
30
Outline
–
–
–
–
Three apparently dissimilar problems
Myopic decision making
Constructing strategies
Challenges
• Unknown, unobserved causes
• Small, expensive data sets
– Discussion
31
Expensive Data on a Limited
Number of Individuals
• Scientists who want to use data on
individuals to construct treatment strategies
must provide measures of confidence and
also evaluations of alternative treatment
strategies.
• Above is challenging because methods for
constructing strategies are non-smooth.
32
Basic Idea for Constructing a Strategy:
Move Backwards Through Time.
Observations
Treatment
Time 1
Observations
Time 2
Treatment
Time 2
Reward
Time 3
33
Expensive, Limited Data on
Individuals
• In order to provide measures of confidence
and comparisons of strategies, the statistical
methods for constructing strategies must be
regularized.
• A number of theoreticians are working hard
on this open question.
34
Outline
–
–
–
–
Three apparently dissimilar problems
Myopic decision making
Constructing strategies
Challenges
• Unknown, unobserved causes
• Small, expensive data sets
– Experiments & Discussion
35
ExTENd
• Ongoing study at U. Pennsylvania (D.
Oslin)
• Goal is to learn how best to help alcohol
dependent individuals reduce alcohol
consumption.
36
Oslin ExTENd
Naltrexone
8 wks Response
Random
assignment:
Early Trigger for
Nonresponse
Random
assignment:
TDM + Naltrexone
CBI
Nonresponse
CBI +Naltrexone
Random
assignment:
8 wks Response
Naltrexone
Random
assignment:
TDM + Naltrexone
Late Trigger for
Nonresponse
Random
assignment:
Nonresponse
CBI
CBI +Naltrexone
37
Adaptive Treatment for ADHD
• Ongoing study at the State U. of NY at
Buffalo (B. Pelham)
• Goal is to learn how best to help children
with ADHD improve functioning at home
and school.
38
ADHD Study
A1. Continue, reassess monthly;
randomize if deteriorate
Yes
8 weeks
A. Begin low-intensity
behavior modification
A2. Add medication;
bemod remains stable but
medication dose may vary
AssessAdequate response?
No
Random
assignment:
Random
assignment:
A3. Increase intensity of bemod
with adaptive modifications based on impairment
B1. Continue, reassess monthly;
randomize if deteriorate
8 weeks
B. Begin low dose
medication
AssessAdequate response?
No
Random
assignment:
B2. Increase dose of medication
with monthly changes
as needed
B3. Add behavioral
treatment; medication dose
remains stable but intensity
of bemod may increase
with adaptive modifications
39
based on impairment
Studies under review
• H. Jones study of drug-addicted pregnant
women (goal is to reduce cocaine/heroin
use during pregnancy and thereby improve
neonatal outcomes)
• J. Sacks study of parolees with substance
abuse disorders (goal is reduce recidivism
and substance use)
40
Jones’ Study for Drug-Addicted
Pregnant Women
rRBT
2 wks Response
Random
assignment:
tRBT
Random
assignment:
tRBT
tRBT
Nonresponse
eRBT
Random
assignment:
2 wks Response
aRBT
Random
assignment:
rRBT
rRBT
Random
assignment:
Nonresponse
tRBT
rRBT
41
Sack’s Study of Adaptive
Transitional Case Management
4 wks Response
Standard TCM
Standard TCM
Nonresponse
Random
assignment:
Augmented TCM
Random
assignment:
Standard TCM
Standard Services
42
Discussion
• The best management of chronic disorders
(poverty, mental illness, other medical conditions)
requires multi-stage decision making.
• Avoid myopic decision making!
– Allow for longer term effects of the treatment
– When comparing treatment options take into account
the effect of future treatments
– Appreciate the value of observing patients outcomes
such as adherence
• Basic experimental designs and statistical methods
are available.
43
This seminar can be found at:
http://www.stat.lsa.umich.edu/~samurphy/
seminars/CASBS07.ppt
Email me with questions or if you would like a
copy:
[email protected]
44
Unknown, Unobserved Causes
Unknown
Causes
Observations
Unknown
Causes
Treatment
Time 1
Observations
Time 2
Treatment
Time 2
Reward
Time 3
45
Unknown, Unobserved Causes
Unknown
Causes
Maturity
of Student
+
-
Binge Drinking
Treatment
Time 1
Frequent Drinking
Binge Drinking
Time 2
Treatment
Time 2
Grade
Time 3
46
Unknown, Unobserved Causes
• Problem: We recruit students via flyers
posted in dormitories. Associations between
observations and rewards are highly likely
to be (due to the unknown causes) nonrepresentative.
• Solution: Sample a representative group of
college students.
47
STAR*D
• This trial is over and the data is being
analyzed (PI: J. Rush).
• One goal of the trial is construct good
treatment sequences for patients suffering
from treatment resistant depression.
www.star-d.org
48
49