Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Causal inference: emulating a target trial when a randomized trial is not available Miguel Hernán DEPARTMENTS OF EPIDEMIOLOGY AND BIOSTATISTICS The situation: We need to make decisions NOW Treat with A or with B? Treat now or later? When to switch to C? A relevant randomized trial would, in principle, answer each comparative effectiveness and safety question Interference/scaling up issues aside Hernán - Target trial 2 But we rarely have randomized trials expensive, untimely, unethical, impractical And deferring decisions is not an option no decision is a decision: “Keep status quo” Question: What do we do? Hernán - Target trial 3 Answer: We analyze observational data (pre-existing or collected for research) Epidemiologic studies Electronic medical records Administrative claims databases National registers Disease registries Other Hernán - Target trial 4 We analyze observational data but only because we cannot conduct a randomized trial Observational studies are not our preferred choice For each observational study, we can imagine a hypothetical randomized trial that we would prefer to conduct If only it were possible Hernán - Target trial 5 The Target Trial An analysis of observational data (e.g., large health care database) can be viewed as an attempt to emulate a hypothetical pragmatic randomized trial That would answer our question If the observational study succeeds at emulating the target trial, both studies would yield identical effect estimates except for random variability Hernán - Target trial 6 Procedure to answer clinical/policy questions Step #1 Describe the protocol of the target trial Step #2 Option A: Conduct the target trial Option B Use observational data to explicitly emulate the target trial Apply appropriate causal inference analytics to estimate the effects of interest Hernán - Target trial 7 Key elements of the protocol of the target trial Eligibility criteria Treatment strategies randomly assigned at start of follow-up Randomized assignment Start/End of follow-up Outcomes Causal contrast(s) of interest Analysis plan Hernán - Target trial 8 The observational study needs to emulate Eligibility criteria Treatment strategies randomly assigned at start of follow-up Randomized assignment Start/End of follow-up Outcomes Causal contrast(s) of interest Analysis plan Hernán - Target trial 9 Example Suppose we use observational data a large health care claims database to emulate a target trial of hormone therapy and heart disease First we need to outline the protocol of the target trial Hernán - Target trial 10 Target trial: Hormone therapy and coronary heart disease Protocol summary Eligibility criteria Postmenopausal women within 5 years of menopause between the years 2005 and 2010, and with no history of cancer and no use of hormone therapy in the last 2 years. Treatment strategies 1. Initiate estrogen plus progestin hormone therapy at baseline and remain on it during the follow-up, unless deep vein thrombosis, pulmonary embolism, myocardial infarction, or cancer are diagnosed 2. Refrain from taking hormone therapy during the follow-up Assignment procedures Participants will be randomly assigned to either strategy at baseline, and will be aware of the strategy they have been assigned to. Follow-up period Starts at randomization and ends at diagnosis of coronary heart disease, death, loss to follow-up, or 5 years after baseline, whichever occurs earlier. Outcome Coronary heart disease diagnosed by a cardiologist Causal contrasts Intention-to-treat effect, per-protocol effect Analysis plan Intention-to-treat analysis, non-naïve per-protocol analysis (to be discussed) Hernán - Target trial 11 Target trial emulation is not straightforward: data experts only When observational data were not collected for research purposes e.g., “coronary heart disease” may be recorded when a woman was diagnosed with it, or when her physician suspected it and ordered a diagnostic test Must consult with knowledgeable data users Time-varying clinical workflows, idiosyncratic coding practices, software versions… Hernán - Target trial 12 Besides expert knowledge of the data Validation studies to quantify data accuracy Internal consistency checks to detect problems Cross-datasets comparisons to better understand coding differences Let’s say we have consulted with experts and done the above before attempting to emulate the target trial Hernán - Target trial 13 Eligibility criteria Emulation Apply same criteria as the target trial to women who at baseline have been included in the database for at least 2 years Potential problems Insufficient data to characterize individuals eligible for the target trial Example: If target trial required baseline screening to exclude prevalent cases, emulation may be hard if database records the performance of a test (for billing purposes) but not its findings Hernán - Target trial 14 Treatment strategies Emulation Eligible individuals assigned to the strategy consistent with their baseline data Strategy 1: women who start estrogen plus progestin therapy Strategy 2: women who do not start hormone therapy Excluded: women who start a different hormone therapy Target trial is typically a pragmatic trial observational data cannot be used to emulate trials with tight monitoring and enforcement of adherence to the study protocol cannot emulate a placebo-controlled trial at most a trial with a “usual care” group Hernán - Target trial 15 Assignment procedures Emulation of blinding Generally impossible individuals in the dataset, and their health care workers, are usually aware of the treatment they receive Observational data can only emulate target trials without blind assignment standard for pragmatic trials not a limitation if the goal is comparing real-world treatment strategies Hernán - Target trial 16 Randomized assignment Emulation of randomization Generally requires adjustment for all confounding factors via matching, stratification or regression, standardization or inverse probability (IP) weighting, g-estimation… If insufficient information on baseline confounders or we fail to identify them, then successful emulation of the target trial’s random assignment is not possible Confounding bias Hernán - Target trial 17 Start of follow-up When is time zero (baseline)? In true trials the time of eligibility and randomization In emulated trials the time of eligibility and treatment assignment Failure to assign time zero correctly may lead to misunderstandings e.g., immortal time bias, others to be discussed Hernán - Target trial 18 Outcome Emulation Use the database to identify women with a diagnosis of coronary heart disease during the follow-up Potential problem: observational data cannot be generally used to emulate a target trial with systematic and blind outcome ascertainment Except if outcome ascertainment cannot be affected by treatment history, e.g., if the outcome is mortality independently ascertained from a death registry Hernán - Target trial 19 The observational analysis needs to emulate Eligibility criteria Treatment strategies randomly assigned at start of follow-up Randomized assignment Start/End of follow-up Outcomes Causal contrast(s) of interest Analysis plan Hernán - Target trial 20 Causal contrast Emulation Truly randomized trials can be used to estimate different types of causal effects We first define these effects in randomized trials, then their analogs in observational studies Hernán - Target trial 21 Causal contrasts 1. Intention-to-treat effect The effect of being assigned to a strategy, regardless of strategy received In our target trial, contrast of the outcome distribution under Assignment to initiation of estrogen plus progestin therapy at baseline vs. Assignment to no hormone therapy at baseline Hernán - Target trial 22 Causal contrasts 1. Intention-to-treat effect In some trials, assignment to and initiation of the treatment strategies occur simultaneously ITT effect is effect of initiation of the treatment strategies Effect magnitude depends on treatment decisions made after baseline discontinuation or initiation of the treatments of interest, use of concomitant therapies… Two trials with the same protocol in similar settings may have different ITT effect estimates with neither of them being biased Hernán - Target trial 23 Causal contrasts 2. Per-protocol effect The effect of receiving the treatment strategies specified in the study protocol In our target trial, contrast of the outcome distribution under receiving estrogen plus progestin therapy continuously (unless toxicity or contraindications arise) vs. receiving no hormone therapy between baseline and study end Hernán - Target trial 24 Causal contrasts 2. Per-protocol effect Often the implicit target of inference When investigators say that the ITT effect estimate is biased, they mean biased for the per-protocol effect Cannot be biased by treatment changes that are consistent with the protocol If protocol lets physicians make their own treatment decisions when toxicity or contraindications arise, then treatment discontinuation because of toxicity is not a deviation from protocol and should not be adjusted for Hernán - Target trial 25 Causal contrasts 3. Other The effect of receiving strategies other than the ones specified in the study protocol The per-protocol effect in an alternative trial In our target trial, contrast of the outcome distribution under receiving statin and estrogen plus progestin therapy continuously (unless toxicity or contraindications arise) vs. receiving neither statin nor hormone therapy between baseline and study end Hernán - Target trial 26 Observational analog Intention-to-treat effect Contrast of the outcome distribution under Initiation of estrogen plus progestin hormone therapy vs. No initiation of hormone therapy at time zero This contrast preserves the key feature of the intention-to-treat analysis: groups are defined solely by baseline information When using prescription data, analog of ITT effect in a target trial When using dispensing data, analog of ITT effect in a target trial in which everybody initiates their assigned strategy Hernán - Target trial 27 Observational analog Per-protocol effect Defined identically as that for the target trial Contrast of the outcome distribution under receiving estrogen plus progestin therapy continuously (unless toxicity or contraindications arise) vs. receiving no hormone therapy between time zero and study end Hernán - Target trial 28 The observational analysis needs to emulate Eligibility criteria Treatment strategies randomly assigned at start of follow-up Start/End of follow-up Outcomes Causal contrast(s) of interest Analysis plan Hernán - Target trial 29 Analysis plan Identical for true and emulated trials Except that no adjustment for baseline confounding is expected in intention-to-treat analysis of true trials Both true and emulated trials require adjustment for post-baseline confounding and selection bias Possibly using Robins’s g-methods Which means longitudinal data on treatment, confounders, and outcomes are required Hernán - Target trial 30 The target trial will be a compromise between the ideal trial we would really like to conduct and the trial we may reasonably emulate using the available data The drafting of the protocol of the target trial is typically an iterative process That requires detailed knowledge of the database Hernán - Target trial 31 Examples of trial emulation using Big Data 1. Classic cohort study Nurses’ Health Study Postmenopausal hormone therapy and coronary heart disease 2. Electronic medical records - THIN Statins and coronary heart disease Static strategies treat vs.no treat 3. Claims database - USRDS Medicare (not today) Epoetin and mortality Static and dynamic strategies intervention depends on response to previous intervention Hernán - Target trial 32 EXAMPLE #1 Hormone therapy and heart disease Question What is the intention-to-treat effect of hormone therapy on the risk of coronary heart disease in postmenopausal women? Hernán - Target trial 33 Answers (shocking discrepancy) Observational studies >30% lower risk in current users compared with never users e.g., HR 0.68 in Nurses Health Study Grodstein et al. J Women’s Health 2006 Randomized trial >20% higher risk in initiators compared with noninitiators HR 1.24 in Women’s Health Initiative Manson et al. NEJM 2003 Hernán - Target trial 34 The WHI randomized trial Manson et al, NEJM 2003 Double-blind Placebo-controlled Large >16,000 U.S. women aged 50-79 yrs Randomly assigned to estrogen plus progestin therapy or placebo Women followed approximately every year like in many large observational studies No intervention after baseline Hernán - Target trial 35 WHI: Effect estimates Intention-to-treat hazard ratio (95% CI) of CHD Overall Years of follow-up 1.23 (0.99, 1.53) 0-2 >2-5 >5 1.51 (1.06, 2.14) 1.31 (0.93, 1.83) 0.67 (0.41, 1.09) <10 10-20 >20 0.89 (0.54, 1.44) 1.24 (0.86, 1.80) 1.65 (1.14, 2.40) Years since menopause Hernán - Target trial 36 Why did observational studies get it “wrong”? Popular theory: residual confounding insufficient adjustment for lifestyle and socioeconomic indicators Corollary: causal inference from observational data is a hopeless undertaking An alternative theory: Observational and randomized studies asked different questions Hernán - Target trial 37 Randomized trial estimated the intention-to-treat effect What is the CHD risk in women who initiate hormone therapy compared with women who do not? Design and analysis: Women randomly assigned to initiation of hormone therapy or placebo Analytic approach Compare risk between incident users and nonusers of hormone therapy Hernán - Target trial 38 Observational studies did not estimate intention-to-treat effect What is the CHD risk in women who are currently taking hormone therapy compared with women who are not? Design and analysis: Women are asked about therapy use Analytic approach Compare risk between prevalent users and nonusers of hormone therapy (current users vs. never users) Hernán - Target trial 39 “Current vs. never users” contrast is not clinically relevant Consider a woman wondering whether to start hormone therapy The current vs. never contrast does not provide the information she needs Consider a woman wondering whether to stop hormone therapy The current vs. never contrast does not provide the information she needs Hernán - Target trial 40 What if we re-analyze the observational study… … to compare the risk in incident users vs. nonusers? That is, what if we use the observational data to answer same question as randomized trial? estimate the observational analog of the intention-totreat effect Hernán et al. Biometrics 2005 Hernán et al. Epidemiology 2008 Hernán - Target trial 41 Effect estimates (ITT hazard ratios) Randomized Observational Women’s Health Initiative Nurses’ Health Study Overall 1.23 (0.99, 1.53) 1.05 (0.82, 1.34) Years of follow-up 0-2 >2 1.51 (1.06, 2.14) 1.07 (0.81, 1.41) 1.43 (0.92, 2.23) 0.91 (0.72, 1.16) Years since menopause <10 10-20 >20 0.89 (0.54, 1.44) 1.24 (0.86, 1.80) 1.65 (1.14, 2.40) 0.88 (0.63, 1.21) 1.13 (0.85, 1.49) -- Hernán - Target trial 42 When same question is asked No shocking observational-randomized discrepancies for ITT estimates though wide CIs in both studies What about the popular hypothesis? Any residual confounding? Probably, but insufficient to explain the original discrepancy Hernán - Target trial 43 Aside: Analysis can/should be extended in two ways Causal contrast Estimate per-protocol effect Rather than intention-to-treat effect only Effect measure Estimate survival (or cumulative risk) curves Rather than hazard ratios only Hernán - Target trial 44 ITT effect is problematic Hernán, Hernández-Díaz. Clinical Trials 2012 Depends on adherence patterns Substantial non-adherence in both randomized trial and observational study Inappropriate for safety outcomes Not patient-centered We also estimated per-protocol effect via IP weighting (more later) Again no randomized-observational discrepancies Toh et al. Ann Intern Med 2010 Hernán - Target trial 45 Hazard ratios are problematic Hernán. Epidemiology 2010 Overall hazard ratio is a weighted average of the time-varying hazard ratios Effect size depends on follow-up length Hazard ratio may be vary over time because of built-in selection bias Hernán - Target trial 46 End of aside Hernán - Target trial 47 Conclusion: CER based on epidemiologic studies is possible If high-quality observational data on treatment, outcome, and confounders are available e.g., the Nurses Health Study But most observational CER relies on large databases (big, pre-existing data) Health claims, electronic medical records Can emulation of a target trial work in that setting? See next case studies Hernán - Target trial 48 EXAMPLE #2 Statins and coronary heart disease Question What is the effect of statin therapy on the risk of coronary heart disease? Extreme example of confounding Hernán - Target trial 49 Target trial: Statin therapy and coronary heart disease Protocol summary Eligibility criteria Individuals aged 55–84 in the years 2000-2006 with no prior history of CHD, stroke, peripheral vascular disease, heart failure, cancer, schizophrenia or dementia, no symptoms of subclinical CHD, and no use of statin therapy in the last 2 years. Treatment strategies 1. Initiate statin therapy at baseline and remain on it during the follow-up, unless contraindications arise 2. Refrain from taking statin therapy during the follow-up Assignment procedures Participants will be randomly assigned to either strategy at baseline, and will be aware of the strategy they have been assigned to. Follow-up period Starts at randomization and ends at diagnosis of coronary heart disease, death, loss to follow-up, or January 2007, whichever occurs earlier. Outcome Coronary heart disease diagnosed by a cardiologist Causal contrasts Intention-to-treat effect, per-protocol effect Analysis plan Intention-to-treat analysis, non-naïve per-protocol analysis Hernán - Target trial 50 Observational data The Health Improvement Network THIN is a database of electronic medical records 6.2 million individuals from 350 general practices in the UK (2009) For each individual demographic and socioeconomic characteristics symptoms, signs and diagnoses, referrals, laboratory test results some lifestyle information Vital status and cause of death data Hernán - Target trial 51 Target trial emulation Use observational data from THIN to emulate the components of the target trial Eligible individuals classified into Strategy 1 if they initiated statin treatment Strategy 2 if they did not initiate statin treatment during the baseline month Baseline month January 2000 3178 individuals met all the eligibility criteria 18 were initiators 1 initiator developed CHD Hernán - Target trial 52 Sequence of target trials Emulation Emulate a target trial starting each calendar month between January 2000 and November 2006 83 target trials with a 1-month enrollment period For each trial Follow-up starts at the trial-specific baseline and ends at diagnosis of CHD, death, lost to follow-up, or January 2007 Eligibility criteria applied at each baseline Hernán - Target trial 53 Sequence of target trials Emulation 74 806 individuals eligible for at least 1 trial On average each eligible individual participated in the emulation of 11 trials many non-initiators in the January 2000 trial still met all eligibility criteria in February All 18 initiators in January 2000 were ineligible in February 2000 because they received treatment during the washout period for the February 2000 trial, and so on Danaei et al. Statistical Methods in Medical Research 2013 Hernán - Target trial 54 CONSORT flowchart of emulated trials 32.9 million potential person-trials (617,432 patients) - 3.2 million prior statin use or <2 years on database - 12.5 million major chronic diseases - 16.4 million no recent data on confounders 844,800 eligible person-trials (74,806 patients) 13,599 initiators 831,201 non-initiators - 245 died - 62 lost to follow-up - 16,094 died - 2,894 lost to follow-up - 117 cases - 13,175 alive and event free at the end of follow-up - 6,218 cases - 805,995 alive and event-free at the end of follow-up Hernán - Target trial 55 Adherence to treatment ___ Non-initiators Probability of continuing initial treatment ___ Initiators Months of follow-up Hernán - Target trial 56 Intention-to-treat effect Emulation Compare CHD incidence in initiators vs. noinitiators at baseline of each emulated trial Regardless of their subsequent treatment Pool data across emulated trials to obtain a more precise effect estimate Robust variance because of within-subject correlation Hernán - Target trial 57 Hazard ratio (95% CI) of CHD THIN trials 2000-2006 Intention-to-treat analysis Per-protocol analysis 635 74,806 6,335 844,800 488 74,806 4,849 844,800 Adjusted for age and sex 1.29 (1.06, 1.56) 1.54 (1.09, 2.18) Adjusted for all covariates 0.89 (0.73, 1.09) 0.84 (0.54, 1.30) Unique cases Unique persons Cases Person-trials Hernán - Target trial 58 What if we had compared prevalent (not incident) users vs. nonusers? Current users HR: 1.42 (1.16, 1.73) Persistent (1 yr) current users HR: 1.05 Persistent (2 yrs) current users HR: 0.77 (0.51, 1.18) We can get any result we want by changing the definition of current user! Confounding-selection bias tradeoff Hernán - Target trial 59 Mortality hazard ratio for statins in CHD secondary prevention studies RCTs: 0.84 (0.77, 0.91) Observational studies Incident users: 0.77 (0.65,0.91) Prevalent-incident mix: 0.70 (0.64, 0.78) Prevalent users: 0.54 (0.45, 0.66) Danaei et al. Am J Epidemiol 2012 Hernán - Target trial 60 Other static comparisons: same analytic approach Head-to-head comparisons: Example: Lipophilic statins (atorvastatin, simvastatin) vs other statins Danaei et al. Diabetes Care 2013 Joint interventions Example: statins plus antihypertensives vs standard care Danaei et al. J Clin Epidemiol 2016 (in press) Hernán - Target trial 61 Advantages of the target trial approach (I) Provides ready access to the application of formal counterfactual theory and concepts to Big Data without the need for technical jargon, Organizing principle for causal inference methods which implicitly rely on counterfactual reasoning e.g., new user design, active comparators, outcome controls Hernán - Target trial 62 Advantages of the target trial approach (I) Provides ready access to the application of formal counterfactual theory and concepts to Big Data without the need for technical jargon, Organizing principle for causal inference methods which implicitly rely on counterfactual reasoning e.g., new user design, active comparators, outcome controls Hernán - Target trial 63 Advantages of the target trial approach (II) Facilitates the comparison of complex strategies that are sustained over time and may depend on a patient’s evolving characteristics Dynamic treatment strategies Not “treat vs no treat” but rather “when to treat, when to switch, when to monitor” depending on time-varying factors Hernán - Target trial 64 Advantages of the target trial approach (III) Establishes a link between methods for the analysis and reporting of randomized trials and Big Data analytics Observational studies analyzed like randomized trials, and vice versa Provides a scaffolding to organize discussions about which data are required/missing Hernán - Target trial 65 Advantages of the target trial approach (IV) Naturally leads to analytic approaches that prevent apparent paradoxes and common biases Selection bias related to prevalent users Immortal time bias Birth weight paradox, obesity paradox Etc. Hernán - Target trial 66 Advantages of the target trial approach (V) Facilitates a systematic methodologic evaluation of observational studies which components of the target trial we weren’t able to mimic approximately? which components of the target trial would be problematic even if we were able to conduct a truly randomized trial? An approach adopted by the Cochrane Collaboration Risk of Bias Tool for Nonrandomised Studies and the IOM Report on the Safety of Approved Drugs Hernán - Target trial 67 Advantages of the target trial approach (VI) Helps understand why estimates differ across studies assess the sensitivity of estimates to different design choices for the target trial focus research efforts on “sensitive” choices Hernán - Target trial 68 Advantages of the target trial approach (last) If we can influence how data are recorded the target trial approach helps record them If we are using data as they exist the target trial approach guides the validation studies and the development and evolution of the Data Model The target trial approach allows you to systematically articulate the tradeoffs that you are willing to accept regarding eligibility criteria, interventions confounders, outcomes Hernán - Target trial 69 A common misinterpretation You are saying that observational studies are as good as RCTs? “This is a cohort study that tries to turn itself into a clinical trial. This involves a series of assumptions and manoeuvres which lack credibility.” Anonymous JAMA reviewer, April 2014 No, the point is not that observational studies can turn themselves into randomized experiments They can’t Hernán - Target trial 70 The point is that we can do better by using observational data to explicitly emulate randomized trials The limitations of observational studies (e.g., confounding, mismeasurement) remain, but we do not compound them with additional problems Hernán - Target trial 71 Remember Observational studies are what we do when we cannot conduct a randomized trial In the absence of practical and ethical constraints, sane people will always prefer a randomized trial No alternative to observational studies So we better keep improving them because people will keep using (Big and Small) observational data to guide their decisions Hernán - Target trial 72