Download Introduction to causal inference from observational data

Causal inference: emulating a
target trial when a randomized
trial is not available
Miguel Hernán
DEPARTMENTS OF EPIDEMIOLOGY AND
BIOSTATISTICS
The situation:
We need to make decisions NOW
 Treat with A or with B?
 Treat now or later?
 When to switch to C?
 A relevant randomized trial would, in principle,
answer each comparative effectiveness and safety
question
 Interference/scaling up issues aside
Hernán - Target trial
2
But we rarely have randomized trials
 expensive, untimely, unethical, impractical
 And deferring decisions is not an option
 no decision is a decision: “Keep status quo”
 Question:
 What do we do?
Hernán - Target trial
3
Answer: We analyze observational data
(pre-existing or collected for research)






Epidemiologic studies
Electronic medical records
Administrative claims databases
National registers
Disease registries
Other
Hernán - Target trial
4
We analyze observational data
 but only because we cannot conduct a randomized trial
 Observational studies are not our preferred choice
 For each observational study, we can imagine a
hypothetical randomized trial that we would prefer to
conduct
 If only it were possible
Hernán - Target trial
5
The Target Trial
 An analysis of observational data (e.g., large health
care database) can be viewed as an attempt to
emulate a hypothetical pragmatic randomized trial
 That would answer our question
 If the observational study succeeds at emulating
the target trial, both studies would yield identical
effect estimates
 except for random variability
Hernán - Target trial
6
Procedure to answer clinical/policy questions
 Step #1
 Describe the protocol of the target trial
 Step #2
 Option A: Conduct the target trial
 Option B
 Use observational data to explicitly emulate the target trial
 Apply appropriate causal inference analytics to estimate the
effects of interest
Hernán - Target trial
7
Key elements of
the protocol of the target trial
 Eligibility criteria
 Treatment strategies
 randomly assigned at start of follow-up





Randomized assignment
Start/End of follow-up
Outcomes
Causal contrast(s) of interest
Analysis plan
Hernán - Target trial
8
The observational study
needs to emulate
 Eligibility criteria
 Treatment strategies
 randomly assigned at start of follow-up





Randomized assignment
Start/End of follow-up
Outcomes
Causal contrast(s) of interest
Analysis plan
Hernán - Target trial
9
Example
 Suppose we use observational data
 a large health care claims database
 to emulate a target trial
 of hormone therapy and heart disease
 First we need to outline the protocol of the target
trial
Hernán - Target trial
10
Target trial: Hormone therapy and coronary heart disease
Protocol summary
Eligibility criteria
Postmenopausal women within 5 years of menopause between the years
2005 and 2010, and with no history of cancer and no use of hormone therapy
in the last 2 years.
Treatment strategies 1. Initiate estrogen plus progestin hormone therapy at baseline and remain
on it during the follow-up, unless deep vein thrombosis, pulmonary
embolism, myocardial infarction, or cancer are diagnosed
2. Refrain from taking hormone therapy during the follow-up
Assignment
procedures
Participants will be randomly assigned to either strategy at baseline, and will
be aware of the strategy they have been assigned to.
Follow-up period
Starts at randomization and ends at diagnosis of coronary heart disease,
death, loss to follow-up, or 5 years after baseline, whichever occurs earlier.
Outcome
Coronary heart disease diagnosed by a cardiologist
Causal contrasts
Intention-to-treat effect, per-protocol effect
Analysis plan
Intention-to-treat analysis, non-naïve per-protocol analysis (to be discussed)
Hernán - Target trial
11
Target trial emulation is not straightforward:
data experts only
 When observational data were not collected for
research purposes
 e.g., “coronary heart disease” may be recorded when a
woman was diagnosed with it, or when her physician
suspected it and ordered a diagnostic test
 Must consult with knowledgeable data users
 Time-varying clinical workflows, idiosyncratic coding
practices, software versions…
Hernán - Target trial
12
Besides expert knowledge of the data
 Validation studies to quantify data accuracy
 Internal consistency checks to detect problems
 Cross-datasets comparisons to better understand coding
differences
 Let’s say we have consulted with experts and done
the above before attempting to emulate the target
trial
Hernán - Target trial
13
Eligibility criteria
Emulation
 Apply same criteria as the target trial to women
who at baseline have been included in the database
for at least 2 years
 Potential problems
 Insufficient data to characterize individuals eligible for
the target trial
 Example: If target trial required baseline screening to
exclude prevalent cases, emulation may be hard if
database records the performance of a test (for billing
purposes) but not its findings
Hernán - Target trial
14
Treatment strategies
Emulation
 Eligible individuals assigned to the strategy consistent
with their baseline data
 Strategy 1: women who start estrogen plus progestin therapy
 Strategy 2: women who do not start hormone therapy
 Excluded: women who start a different hormone therapy
 Target trial is typically a pragmatic trial
 observational data cannot be used to emulate trials with tight
monitoring and enforcement of adherence to the study
protocol
 cannot emulate a placebo-controlled trial
 at most a trial with a “usual care” group
Hernán - Target trial
15
Assignment procedures
Emulation of blinding
 Generally impossible
 individuals in the dataset, and their health care workers,
are usually aware of the treatment they receive
 Observational data can only emulate target trials
without blind assignment
 standard for pragmatic trials
 not a limitation if the goal is comparing real-world
treatment strategies
Hernán - Target trial
16
Randomized assignment
Emulation of randomization
 Generally requires adjustment for all confounding
factors
 via matching, stratification or regression, standardization
or inverse probability (IP) weighting, g-estimation…
 If insufficient information on baseline confounders
or we fail to identify them, then successful
emulation of the target trial’s random assignment is
not possible
 Confounding bias
Hernán - Target trial
17
Start of follow-up
When is time zero (baseline)?
 In true trials
 the time of eligibility and randomization
 In emulated trials
 the time of eligibility and treatment assignment
 Failure to assign time zero correctly may lead to
misunderstandings
 e.g., immortal time bias, others to be discussed
Hernán - Target trial
18
Outcome
Emulation
 Use the database to identify women with a
diagnosis of coronary heart disease during the
follow-up
 Potential problem: observational data cannot be
generally used to emulate a target trial with
systematic and blind outcome ascertainment
 Except if outcome ascertainment cannot be affected by
treatment history, e.g., if the outcome is mortality
independently ascertained from a death registry
Hernán - Target trial
19
The observational analysis
needs to emulate
 Eligibility criteria
 Treatment strategies
 randomly assigned at start of follow-up





Randomized assignment
Start/End of follow-up
Outcomes
Causal contrast(s) of interest
Analysis plan
Hernán - Target trial
20
Causal contrast
Emulation
 Truly randomized trials can be used to estimate
different types of causal effects
 We first define these effects in randomized trials,
then their analogs in observational studies
Hernán - Target trial
21
Causal contrasts
1. Intention-to-treat effect
 The effect of being assigned to a strategy,
regardless of strategy received
 In our target trial, contrast of the outcome
distribution under
 Assignment to initiation of estrogen plus progestin therapy at
baseline vs.
 Assignment to no hormone therapy
 at baseline
Hernán - Target trial
22
Causal contrasts
1. Intention-to-treat effect
 In some trials, assignment to and initiation of the
treatment strategies occur simultaneously
 ITT effect is effect of initiation of the treatment strategies
 Effect magnitude depends on treatment decisions made
after baseline
 discontinuation or initiation of the treatments of interest, use
of concomitant therapies…
 Two trials with the same protocol in similar settings
may have different ITT effect estimates with neither of
them being biased
Hernán - Target trial
23
Causal contrasts
2. Per-protocol effect
 The effect of receiving the treatment strategies
specified in the study protocol
 In our target trial, contrast of the outcome
distribution under
 receiving estrogen plus progestin therapy continuously
(unless toxicity or contraindications arise) vs.
 receiving no hormone therapy
 between baseline and study end
Hernán - Target trial
24
Causal contrasts
2. Per-protocol effect
 Often the implicit target of inference
 When investigators say that the ITT effect estimate is
biased, they mean biased for the per-protocol effect
 Cannot be biased by treatment changes that are
consistent with the protocol
 If protocol lets physicians make their own treatment
decisions when toxicity or contraindications arise, then
treatment discontinuation because of toxicity is not a
deviation from protocol and should not be adjusted for
Hernán - Target trial
25
Causal contrasts
3. Other
 The effect of receiving strategies other than the
ones specified in the study protocol
 The per-protocol effect in an alternative trial
 In our target trial, contrast of the outcome
distribution under
 receiving statin and estrogen plus progestin therapy
continuously (unless toxicity or contraindications arise) vs.
 receiving neither statin nor hormone therapy
 between baseline and study end
Hernán - Target trial
26
Observational analog
Intention-to-treat effect
 Contrast of the outcome distribution under
 Initiation of estrogen plus progestin hormone therapy vs.
 No initiation of hormone therapy
 at time zero
 This contrast preserves the key feature of the
intention-to-treat analysis: groups are defined solely
by baseline information
 When using prescription data, analog of ITT effect in a target
trial
 When using dispensing data, analog of ITT effect in a target
trial in which everybody initiates their assigned strategy
Hernán - Target trial
27
Observational analog
Per-protocol effect
 Defined identically as that for the target trial
 Contrast of the outcome distribution under
 receiving estrogen plus progestin therapy continuously
(unless toxicity or contraindications arise) vs.
 receiving no hormone therapy
 between time zero and study end
Hernán - Target trial
28
The observational analysis
needs to emulate
 Eligibility criteria
 Treatment strategies
 randomly assigned at start of follow-up




Start/End of follow-up
Outcomes
Causal contrast(s) of interest
Analysis plan
Hernán - Target trial
29
Analysis plan
 Identical for true and emulated trials
 Except that no adjustment for baseline confounding is
expected in intention-to-treat analysis of true trials
 Both true and emulated trials require adjustment
for post-baseline confounding and selection bias
 Possibly using Robins’s g-methods
 Which means longitudinal data on treatment,
confounders, and outcomes are required
Hernán - Target trial
30
The target trial
will be a compromise
 between the ideal trial we would really like to conduct
and the trial we may reasonably emulate using the
available data
 The drafting of the protocol of the target trial is
typically an iterative process
 That requires detailed knowledge of the database
Hernán - Target trial
31
Examples of trial emulation
using Big Data
1. Classic cohort study


Nurses’ Health Study
Postmenopausal hormone therapy and coronary heart disease
2. Electronic medical records - THIN


Statins and coronary heart disease
Static strategies
 treat vs.no treat
3. Claims database - USRDS Medicare (not today)


Epoetin and mortality
Static and dynamic strategies
 intervention depends on response to previous intervention
Hernán - Target trial
32
EXAMPLE #1
Hormone therapy and heart disease
 Question
 What is the intention-to-treat effect of hormone therapy
on the risk of coronary heart disease in postmenopausal
women?
Hernán - Target trial
33
Answers
(shocking discrepancy)
 Observational studies
 >30% lower risk in current users compared with never
users
 e.g., HR 0.68 in Nurses Health Study
 Grodstein et al. J Women’s Health 2006
 Randomized trial
 >20% higher risk in initiators compared with noninitiators
 HR 1.24 in Women’s Health Initiative
 Manson et al. NEJM 2003
Hernán - Target trial
34
The WHI randomized trial
Manson et al, NEJM 2003
 Double-blind
 Placebo-controlled
 Large
 >16,000 U.S. women aged 50-79 yrs
 Randomly assigned to estrogen plus progestin therapy or
placebo
 Women followed approximately every year like in many
large observational studies
 No intervention after baseline
Hernán - Target trial
35
WHI: Effect estimates
Intention-to-treat hazard ratio (95% CI) of CHD
 Overall
 Years of
follow-up
1.23 (0.99, 1.53)
 0-2
 >2-5
 >5
1.51 (1.06, 2.14)
1.31 (0.93, 1.83)
0.67 (0.41, 1.09)
 <10
 10-20
 >20
0.89 (0.54, 1.44)
1.24 (0.86, 1.80)
1.65 (1.14, 2.40)
 Years since
menopause
Hernán - Target trial
36
Why did observational studies
get it “wrong”?
 Popular theory: residual confounding
 insufficient adjustment for lifestyle and
socioeconomic indicators
 Corollary: causal inference from observational data
is a hopeless undertaking
 An alternative theory: Observational and
randomized studies asked different
questions
Hernán - Target trial
37
Randomized trial estimated
the intention-to-treat effect
 What is the CHD risk in women who initiate
hormone therapy compared with women who do
not?
 Design and analysis:
 Women randomly assigned to initiation of hormone
therapy or placebo
 Analytic approach
 Compare risk between incident users and nonusers of
hormone therapy
Hernán - Target trial
38
Observational studies did not
estimate intention-to-treat effect
 What is the CHD risk in women who are currently
taking hormone therapy compared with women
who are not?
 Design and analysis:
 Women are asked about therapy use
 Analytic approach
 Compare risk between prevalent users and nonusers of
hormone therapy (current users vs. never users)
Hernán - Target trial
39
“Current vs. never users”
contrast is not clinically relevant
 Consider a woman wondering whether to start
hormone therapy
 The current vs. never contrast does not provide the
information she needs
 Consider a woman wondering whether to stop
hormone therapy
 The current vs. never contrast does not provide the
information she needs
Hernán - Target trial
40
What if we re-analyze the observational study…
 … to compare the risk in incident users vs.
nonusers?
 That is, what if we use the observational data to
answer same question as randomized trial?
 estimate the observational analog of the intention-totreat effect


Hernán et al. Biometrics 2005
Hernán et al. Epidemiology 2008
Hernán - Target trial
41
Effect estimates (ITT hazard ratios)
Randomized
Observational
Women’s Health Initiative
Nurses’ Health Study
 Overall
1.23 (0.99, 1.53)
1.05 (0.82, 1.34)
 Years of
follow-up
 0-2
 >2
1.51 (1.06, 2.14)
1.07 (0.81, 1.41)
1.43 (0.92, 2.23)
0.91 (0.72, 1.16)
 Years since
menopause
 <10
 10-20
 >20
0.89 (0.54, 1.44)
1.24 (0.86, 1.80)
1.65 (1.14, 2.40)
0.88 (0.63, 1.21)
1.13 (0.85, 1.49)
--
Hernán - Target trial
42
When same question is asked
 No shocking observational-randomized
discrepancies for ITT estimates
 though wide CIs in both studies
 What about the popular hypothesis? Any residual
confounding?
 Probably, but insufficient to explain the original
discrepancy
Hernán - Target trial
43
Aside:
Analysis can/should be extended in two ways
 Causal contrast
 Estimate per-protocol effect
 Rather than intention-to-treat effect only
 Effect measure
 Estimate survival (or cumulative risk) curves
 Rather than hazard ratios only
Hernán - Target trial
44
ITT effect is problematic
Hernán, Hernández-Díaz. Clinical Trials 2012
 Depends on adherence patterns
 Substantial non-adherence in both randomized trial and
observational study
 Inappropriate for safety outcomes
 Not patient-centered
 We also estimated per-protocol effect
 via IP weighting (more later)
 Again no randomized-observational discrepancies
 Toh et al. Ann Intern Med 2010
Hernán - Target trial
45
Hazard ratios are problematic
Hernán. Epidemiology 2010
 Overall hazard ratio is a weighted average of the
time-varying hazard ratios
 Effect size depends on follow-up length
 Hazard ratio may be vary over time because of
built-in selection bias
Hernán - Target trial
46
End of aside
Hernán - Target trial
47
Conclusion: CER based on
epidemiologic studies is possible
 If high-quality observational data on treatment,
outcome, and confounders are available
 e.g., the Nurses Health Study
 But most observational CER relies on large
databases (big, pre-existing data)
 Health claims, electronic medical records
 Can emulation of a target trial work in that
setting?
 See next case studies
Hernán - Target trial
48
EXAMPLE #2
Statins and coronary heart disease
 Question
 What is the effect of statin therapy on the risk of
coronary heart disease?
 Extreme example of confounding
Hernán - Target trial
49
Target trial: Statin therapy and coronary heart disease
Protocol summary
Eligibility criteria
Individuals aged 55–84 in the years 2000-2006 with no prior history of CHD,
stroke, peripheral vascular disease, heart failure, cancer, schizophrenia or
dementia, no symptoms of subclinical CHD, and no use of statin therapy in
the last 2 years.
Treatment strategies 1. Initiate statin therapy at baseline and remain on it during the follow-up,
unless contraindications arise
2. Refrain from taking statin therapy during the follow-up
Assignment
procedures
Participants will be randomly assigned to either strategy at baseline, and will
be aware of the strategy they have been assigned to.
Follow-up period
Starts at randomization and ends at diagnosis of coronary heart disease,
death, loss to follow-up, or January 2007, whichever occurs earlier.
Outcome
Coronary heart disease diagnosed by a cardiologist
Causal contrasts
Intention-to-treat effect, per-protocol effect
Analysis plan
Intention-to-treat analysis, non-naïve per-protocol analysis
Hernán - Target trial
50
Observational data
The Health Improvement Network
 THIN is a database of electronic medical records
 6.2 million individuals from 350 general practices in the
UK (2009)
 For each individual
 demographic and socioeconomic characteristics
 symptoms, signs and diagnoses, referrals, laboratory
test results
 some lifestyle information
 Vital status and cause of death data
Hernán - Target trial
51
Target trial emulation
 Use observational data from THIN to emulate the
components of the target trial
 Eligible individuals classified into
 Strategy 1 if they initiated statin treatment
 Strategy 2 if they did not initiate statin treatment
 during the baseline month
 Baseline month January 2000
 3178 individuals met all the eligibility criteria
 18 were initiators
 1 initiator developed CHD
Hernán - Target trial
52
Sequence of target trials
Emulation
 Emulate a target trial starting each calendar
month between January 2000 and November
2006
 83 target trials with a 1-month enrollment period
 For each trial
 Follow-up starts at the trial-specific baseline and
ends at diagnosis of CHD, death, lost to follow-up, or
January 2007
 Eligibility criteria applied at each baseline
Hernán - Target trial
53
Sequence of target trials
Emulation
 74 806 individuals eligible for at least 1 trial
 On average each eligible individual participated in the
emulation of 11 trials
 many non-initiators in the January 2000 trial still met all
eligibility criteria in February
 All 18 initiators in January 2000 were ineligible in February
2000 because they received treatment during the washout
period for the February 2000 trial, and so on
 Danaei et al. Statistical Methods in Medical Research
2013
Hernán - Target trial
54
CONSORT
flowchart of emulated trials
32.9 million potential
person-trials
(617,432 patients)
- 3.2 million prior statin use or <2
years on database
- 12.5 million major chronic diseases
- 16.4 million no recent data on
confounders
844,800 eligible
person-trials
(74,806 patients)
13,599 initiators
831,201 non-initiators
- 245 died
- 62 lost to follow-up
- 16,094 died
- 2,894 lost to follow-up
- 117 cases
- 13,175 alive and event free at the
end of follow-up
- 6,218 cases
- 805,995 alive and event-free at
the end of follow-up
Hernán - Target trial
55
Adherence to treatment
___ Non-initiators
Probability of continuing initial treatment
___ Initiators
Months of follow-up
Hernán - Target trial
56
Intention-to-treat effect
Emulation
 Compare CHD incidence in initiators vs. noinitiators at baseline of each emulated trial
 Regardless of their subsequent treatment
 Pool data across emulated trials to obtain a
more precise effect estimate
 Robust variance because of within-subject
correlation
Hernán - Target trial
57
Hazard ratio (95% CI) of CHD
THIN trials 2000-2006
Intention-to-treat analysis
Per-protocol analysis
635
74,806
6,335
844,800
488
74,806
4,849
844,800
Adjusted for age and sex
1.29 (1.06, 1.56)
1.54 (1.09, 2.18)
Adjusted for all covariates
0.89 (0.73, 1.09)
0.84 (0.54, 1.30)
Unique cases
Unique persons
Cases
Person-trials
Hernán - Target trial
58
What if we had compared prevalent (not
incident) users vs. nonusers?
 Current users
 HR: 1.42 (1.16, 1.73)
 Persistent (1 yr) current users
 HR: 1.05
 Persistent (2 yrs) current users
 HR: 0.77 (0.51, 1.18)
 We can get any result we want by changing the
definition of current user!
 Confounding-selection bias tradeoff
Hernán - Target trial
59
Mortality hazard ratio for statins in CHD
secondary prevention studies
 RCTs: 0.84 (0.77, 0.91)
 Observational studies
 Incident users:
0.77 (0.65,0.91)
 Prevalent-incident mix: 0.70 (0.64, 0.78)
 Prevalent users:
0.54 (0.45, 0.66)
 Danaei et al. Am J Epidemiol 2012
Hernán - Target trial
60
Other static comparisons:
same analytic approach
 Head-to-head comparisons:
 Example: Lipophilic statins (atorvastatin, simvastatin) vs
other statins
 Danaei et al. Diabetes Care 2013
 Joint interventions
 Example: statins plus antihypertensives vs standard care
 Danaei et al. J Clin Epidemiol 2016 (in press)
Hernán - Target trial
61
Advantages of the target trial approach (I)
 Provides ready access to the application of formal
counterfactual theory and concepts to Big Data
 without the need for technical jargon,
 Organizing principle for causal inference methods
 which implicitly rely on counterfactual reasoning
 e.g., new user design, active comparators, outcome
controls
Hernán - Target trial
62
Advantages of the target trial approach (I)
 Provides ready access to the application of formal
counterfactual theory and concepts to Big Data
 without the need for technical jargon,
 Organizing principle for causal inference methods
 which implicitly rely on counterfactual reasoning
 e.g., new user design, active comparators, outcome
controls
Hernán - Target trial
63
Advantages of the target trial approach (II)
 Facilitates the comparison of complex strategies
that are sustained over time and may depend on a
patient’s evolving characteristics
 Dynamic treatment strategies
 Not “treat vs no treat” but rather “when to treat, when
to switch, when to monitor” depending on time-varying
factors
Hernán - Target trial
64
Advantages of the target trial approach (III)
 Establishes a link between methods for the analysis
and reporting of randomized trials and Big Data
analytics
 Observational studies analyzed like randomized trials,
and vice versa
 Provides a scaffolding to organize discussions about
which data are required/missing
Hernán - Target trial
65
Advantages of the target trial approach (IV)
 Naturally leads to analytic approaches that prevent
apparent paradoxes and common biases




Selection bias related to prevalent users
Immortal time bias
Birth weight paradox, obesity paradox
Etc.
Hernán - Target trial
66
Advantages of the target trial approach (V)
 Facilitates a systematic methodologic evaluation of
observational studies
 which components of the target trial we weren’t able to mimic
approximately?
 which components of the target trial would be problematic
even if we were able to conduct a truly randomized trial?
 An approach adopted by the Cochrane Collaboration Risk
of Bias Tool for Nonrandomised Studies and the IOM
Report on the Safety of Approved Drugs
Hernán - Target trial
67
Advantages of the target trial approach (VI)
 Helps understand why estimates differ across
studies
 assess the sensitivity of estimates to different design
choices for the target trial
 focus research efforts on “sensitive” choices
Hernán - Target trial
68
Advantages of the target trial approach (last)
 If we can influence how data are recorded
 the target trial approach helps record them
 If we are using data as they exist
 the target trial approach guides the validation studies and the
development and evolution of the Data Model
 The target trial approach allows you to systematically
articulate the tradeoffs that you are willing to accept

regarding eligibility criteria, interventions confounders, outcomes
Hernán - Target trial
69
A common misinterpretation
 You are saying that observational studies are as
good as RCTs?
 “This is a cohort study that tries to turn itself into a
clinical trial. This involves a series of assumptions and
manoeuvres which lack credibility.”
 Anonymous JAMA reviewer, April 2014
 No, the point is not that observational studies can
turn themselves into randomized experiments
 They can’t
Hernán - Target trial
70
The point is that we can do better
 by using observational data to explicitly emulate
randomized trials
 The limitations of observational studies (e.g.,
confounding, mismeasurement) remain, but we do
not compound them with additional problems
Hernán - Target trial
71
Remember
 Observational studies are what we do when
we cannot conduct a randomized trial
 In the absence of practical and ethical constraints, sane
people will always prefer a randomized trial
 No alternative to observational studies
 So we better keep improving them
 because people will keep using (Big and Small) observational
data to guide their decisions
Hernán - Target trial
72