Download Subgroup Analyses in Early Phase Clinical Trials

Subgroup Analyses in Early Phase
Clinical Trials
Björn Bornkamp
East User Group Meeting, March 16th, 2016
based on joint work with Marius Thomas, see
Thomas, M., Bornkamp, B. (2016) Comparing Approaches to
Treatment Effect Estimation for Subgroups in Early Phase
Clinical Trials, http://arxiv.org/abs/1603.03316
Outline
§  Subgroup Analysis
•  Overview & Challenges
§  Adjusted Treatment Effect Estimation: Methods
•  Model Averaging
•  Resampling Methods
•  Penalized Multivariate Regression
§  Simulation Study
2
Subgroup Analyses
A „hot topic“
§  EMA: Draft guideline on the investigation of subgroups in
confirmatory clinical trials (2014; finalization 2016?)
http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2014/02/WC500160523.pdf
§  Jan 2015: Precision medicine initiative in United States
§  FDA has a working group on subgroup analyses
§  Recent review articles
•  Ondra, T. et al (2016), J Biopharm Stat., 26, 99-119
•  Dmitrienko A. et al (2016), J Biopharm Stat., 26, 71-98
•  Ruberg, S. et al (2015), Stat Biopharm Res., 7, 214-229
•  Mayer, C. et al (2015), Stat Biopharm Res., 7, 272-282
•  Alosh, M. et al (2015), Stat Biopharm Res., 7, 286-303
•  Special issue on subgroup analysis: J Biopharm Stat (2014), 24
3
Out of scope for this presentation
§  Fully Confirmatory Setting
•  Pre-specification of subgroup
-  Multiple testing strategy in place
-  Adequate sample size for subgroup
•  Adaptive Enrichment designs
-  Primarily a multiple testing problem
4
In scope for today
§  Trial-level analyses
•  Routinely performed in early phase (e.g. proof-of-concept, Phase IIa
studies) but also on an exploratory basis in late stage trials
•  Sample size: Comparisons in candidate subgroups not „powered“
•  Pre-specification of subgroups?
-  Usually a list of subgroup defining covariates is pre-defined
•  How many covariates?
-  ~5-30
•  What type of covariates?
-  Clinical variables (baseline disease severity, renal function, ...),
demographic covariates, selected genetic covariates
-  Usually all variables have some clinical plausibility
5
Aims & consequences of subgroup analyses
§  Aim of these analyses
•  Identify a subgroup of patients with an increased treatment effect to
be confirmed in later trials
§  Potential consequences of this analysis on subsequent
development (if a subgroup of patients is identified)
•  Development in overall population, but „keep an eye“ on the
subgroup
•  Enrichment based on the identified subgroup
•  Development in the subgroup only
6
Subgroup analyses
Statistical model for normally distributed endpoints
Data model: ​𝑦↓𝑖 ~ 𝑁(​𝜇↓𝑖 , ​𝜎↑2 ), i=1, ..., n
​𝜇↓𝑖 =​𝛽↓0 +​𝛽↓1 ​𝑇↓𝑖 ​𝜇↓𝑖 − Mean response of patient i ​𝑇↓𝑖 − Treatment indicator
​𝛽↓0 − Placebo effect
​𝛽↓1 − Treatment effect
Subgroup analysis model
​𝜇↓𝑖 =​𝛽↓0 +​𝛽↓2 ​𝑠↓𝑖 +​(𝛽↓1 +​𝛽↓3 ​𝑠↓𝑖 )​𝑇↓𝑖 ​𝑠↓𝑖 − Subgroup indicator (1 in the subgroup, 0 in the complement)
​𝛽↓2 − Prognostic effect of subgroup (independent of treatment)
​𝛽↓3 − Predictive effect of the subgroup (change in the treatment effect)
7
Subgroup analyses
Standard approach
§  For all potential K subgroups (i.e. subgroup indicators)
•  fit the model and obtain treatment effect in subgroup: ​𝛽↓1↑(𝑘) +​
𝛽↓3↑(𝑘) and a p-value for testing ​𝛽↓3↑(𝑘) =0 („interaction test“)
•  Choose model/subgroup, based on
-  magnitude of ​𝛽↓1↑(𝑘) +​𝛽↓3↑(𝑘) for different subgroups k
-  on the p-value for the interaction test
§  Challenge with this approach
•  When searching for a group with an increased treatment effect, one
will find a subgroup by chance / overfitting
•  Treatment effect estimate in the subgroup will be upwards „biased“
•  The problem increases the more subgroups are investigated & if
subgroups of low plausibility are included
8
Continuous covariates & categorization
§  Subgroups defined in terms of continuous covariates are
often defined as a cut-off (for clinical convenience)
•  Modelling could still utilize continuous covariates (e.g. using splines)
§  In what follows
•  Focus on binarization approach (represents current practice better)
•  Focus in this presentation is on comparing different „adjusted
treatment effect estimates“
•  Adjustment methods also work in the continuous setting, e.g. using
spline modelling approach (results not shown)
9
Treatment Effect
Estimation: Methods
10
Model Averaging
11
Model averaging
Motivated from Bayesian ideas
§  Subgroup Selection can be viewed as Model Selection*
§  Each subgroup defines a different statistical model
§  Picking one model (e.g. with high or the highest
(standardized) treatment effect) ignores model uncertainty
•  This is like setting the posterior model probabilities to 1 for one
model and 0 for all others
•  Idea of model averaging: Use weighted inference based on the
posterior model probabilities for each model/subgroup
*Berger, J. et al. (2014) J Biopharm Stat, 24, 110-129
12
Model averaging
General idea
§  Idea of model averaging
•  Assume a candidate set M1,..., MK of statistical models is available
•  Assume we are interested in a quantity θ that can be derived under
each model
-  in our case the treatment effect in the subgroup
•  Final estimate of the parameter θ
-  Weighted average
∑𝑘↑▒​𝑤↓𝑘 ​​𝜃↓𝑘 -  Weights wk are posterior model probabilities, ​𝜃↓𝑘 is the estimate (e.g.
posterior mean) under the k-th model
13
Model averaging
§  Assume a set of K candidate subgroups is pre-specified
•  Candidate subgroups s(1), ..., s(K)
§  These correspond to K candidate models M1,..., MK
•  the k-th model is given by
yi ~ N(µμi(k),σ(k)2)
µμi(k) = β0(k) + ​β↓2↑(k) ​s↓i↑(k) +​(β↓1 +​β↓3 ​s↓i↑(k) )​T↓i with prior distributions for β0(k),δ1(k),δ2(k) and σ(k)2
and prior model probabilites for M1,..., MK
14
Model averaging
How to apply in this setting?
§  For subgroup s(k) under model Mk
•  The treatment effect in the subgroup is ​𝛽↓1↑(𝑘) +​𝛽↓3↑(𝑘) •  The treatment effect in the complement is ​𝛽↓1↑(𝑘) §  For subgroup s(k) under model Mk‘ (with k‘ ≠ k)
•  Can calculate the treatment effect for each patient in the subgroup
(either equal to ​𝛽↓1↑(𝑘′) +​𝛽↓3↑(𝑘′) or ​𝛽↓1↑(𝑘′) ).
-  Treatment effect for subgroup s(k) under model Mk‘
​𝛽↓1↑(𝑘′) +​𝑤↓​𝑘,𝑘↑′ ​𝛽↓3↑(𝑘′) where ​𝑤↓𝑘,​𝑘↑′ ∈[0,1] is the proportion of patients in subgroup k that are also
in subgroup k‘
15
Example output of analysis
Overall estimate
§  Extreme cases
•  Posterior probability 1 for selected subgroup/model: No shrinkage
•  Equal posterior probabilities for all possible subgroups/models
-  Shrinkage towards overall treatment effect
16
Model averaging
Why does it induce shrinkage?
§  Shrinkage
•  For each model the estimate is bounded by ​𝛽↓1↑(𝑘) and ​𝛽↓1↑(𝑘) +​
𝛽↓3↑(𝑘) •  For many subgroups ​𝛽↓3↑(𝑘) ≈0 à Shrinkage towards overall effect
as long as these models receive positive posterior probability
•  Main driver in shrinkage: Posterior model probabilities
-  If a subgroup has high posterior probability à Little or no shrinkage
-  If many subgroups are included à every single subgroup receives less
prior probability à more shrinkage
-  If many similar subgroups are included (large overlap) à less shrinkage
17
Model averaging: Implementation*
§  BUGS/JAGS
•  Can incorporate prior information on model parameters and
plausibility of different subgroups
§  ML estimation and BIC model weights
•  Cannot incorporate prior information on model parameters; only prior
weights on plausibility of different subgroups
•  Used in the upcoming simulations (with equal model weights)
§  NB: Can also include a model where the treatment effect
is the same across all groups
•  More shrinkage towards the overall effect (not considered further)
*Bornkamp. et al. (2016) Technical report
18
Resampling
19
Resampling
§  Idea*
•  Split data into training and test set
•  Perform subgroup identification on the training data
•  Compare the treatment effect in the selected subgroup in the training
and test data set
§  Can be implemented using a bootstrap approach
•  Use „out-of-bag“ samples as the test data-set
*Sun, L. et al (2005), Genet Epidemiol., 28, 352-367
20
Treatment effect estimate
§  For each bootstrap resample
•  Perform selection of a subgroup as on the original data
•  Calculate treatment effect estimate in selected subgroup based on
identification sample and out-of-bag sample (test sample)
§  „bias-corrected“ estimate of treatment effect
•  Bias estimation („rsbias“ in what follows)
-  Estimate mean difference between estimate in identification and out-of-bag
sample, this can be used as a bias correction
•  .632 bootstrap („rs632“ in what follows)
-  (1-0.632) * (original estimate on complete data)
+ 0.632 * (mean in estimate in test sample over bootstrap replications)
21
Penalized multivariate
regression: LASSO
22
Penalized multivariate regression: LASSO
§  Multivariate regression model including all covariates
​𝜇↓𝑖 =​𝛽↓0 +∑𝑘↑▒​𝛽↓2,𝑘 ​𝑥↓𝑖↑(𝑘) +(​𝛽↓1 +∑𝑘↑▒​𝛽↓3,𝑘 ​𝑥↓𝑖↑(𝑘) )​𝑇↓𝑖 §  This model can get unstable to fit (if n is small compared
to K) and clearly it would be overfitting data
23
Penalized multivariate regression: LASSO
§  Idea of LASSO* regression
•  Instead of minimizing the residual sum of squares, minimizes a
penalized residual sum of squares
∑𝑖↑▒​(​𝑦↓𝑖 −​𝜇↓𝑖 )↑2 +𝜆(∑𝑘↑▒|​𝛽↓2,𝑘 | +∑𝑘↑▒|​𝛽↓3,𝑘 | ) •  This will „shrink“ the coefficients ​𝛽↓2,𝑘 and ​𝛽↓3,𝑘 towards 0 (i.e.
subgroup estimates towards the overall treatment effect)
•  The estimated ​𝛽↓2,𝑘 and ​𝛽↓3,𝑘 might get exactly equal to 0
•  The parameter 𝜆 determines the amount of shrinkage (can be
chosen by cross-validation)
•  Here shrinkage not used for overall intercept ​𝛽↓0 and treatment
effect estimate ​𝛽↓1 *Tibshirani, R. (1996), J R Stat Soc Series B, 58, 267-288
24
Treatment effect estimate
§  How to obtain treatment effect estimate for a given
subgroup?
•  Predict treatment effect for every patient in the subgroup based on
the LASSO model
•  Average this over all patients in the subgroup
§  Shrinkage towards overall effect
•  Will be induced by the fact that many of the ​𝛽↓3,𝑘 will be estimated
to be 0 à the corresponding covariate for this subgroup will have no
effect
25
Comparison between adjustment methods
§  Resampling methods explicitly adjust for the specific
„selection mechanism“
•  Different selection mechanisms will lead to different amount of
shrinkage
•  But only provides inference for the selected subgroup
§  Model averaging, penalized multivariate regression
•  Shrinkage does not depend specific selection mechanism (more a
consequence of the modelling assumptions being made)
•  Provides inference for all subgroups investigated
§  Can a subgroup selection rule be 100% algorithmic?
26
Simulation Study
27
Data generation
​𝜇↓𝑖 =​𝑔(𝑥↓1,𝑖 )​𝑇↓𝑖 §  ​𝑥↓𝑗,𝑖 - Continuous covariate (j=1,...,K)
•  Covariates ​𝑥↓1,𝑖 , …, ​𝑥↓𝐾,𝑖 generated independently according to
N(0,1)
§  No covariate has a prognostic effect
§  Only covariate ​𝑥↓1,𝑖 has a predictive effect
•  g(𝑥) – Continuous function describing the treatment effect
•  Determines the treatment effect for each patient, depending on ​𝑥↓1,𝑖 -  Also other covariates/subgroups might seem to have an effect depending
on overlap/correlation with ​𝑥↓1 28
Simulation settings
29
Subgroup Identification
§  Each covariate defines 3 candidate subgroups
•  3 cut-points used (based on the 3 quartiles)
§  Use the p-value of the interaction test to select the group
§  Might represent current practice relatively well
30
31
Bias
For estimating the treatment effect in the selected subgroup
32
Mean squared error
For estimating the treatment effect in the selected subgroup
33
Coverage of 90% confidence intervals
For estimating the treatment effect in the selected subgroup
34
Conclusion
§  Variety of analysis methods are available for use to reduce
bias induced by a selection process
•  All methods (apart from „rsbias“) improve upon the naive approach
compared in the performance metrics studied
•  (Up to now) not one single uniformly best method available
35