Download EVALUATING THE CLINICAL UTILITY OF PREDICTION MODELS IN A HETEROGENEOUS

EVALUATING THE CLINICAL UTILITY OF PREDICTION MODELS IN A HETEROGENEOUS
MULTICENTER POPULATION USING DECISION-ANALYTIC MEASURES: THE RANDOMEFFECT WEIGHTED NET BENEFIT
Laure Wynants, MSc1, Dirk Timmerman, PhD, MD2, Sabine Van Huffel, PhD1 and Ben Van Calster,
PhD2, (1)KU Leuven, Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical
Systems, Signal Processing and Data Analytics, Leuven, Belgium, (2)KU Leuven Department of
Development and Regeneration, Leuven, Belgium
Purpose: To investigate methods to evaluate the clinical utility of a prediction model in
heterogeneous multicenter datasets based on decision-analytic measures.
Method: We focus on the Net Benefit (NB) statistic from Vickers and Elkin (Med Decis Making
2006). NB is defined as (TP-w*FP)/N, with TP the number of true positives, FP the number of false
positives, and w the ‘harm-to-benefit ratio’ of treating a false positive versus a true positive. This
ratio equals the odds of the risk threshold t used to classify patients as positive or negative. A
model’s NB can be compared to the default strategies of classifying all as positive (treat all) or
negative (treat none). We averaged center-specific NBs for specific values of t using random-effect
weights 1/(se2+τ2), with se2 the within-center and τ2 the between-center variance of NB. We also
calculated center-specific Relative Utilities, i.e. normalized differences between the model’s NB and
the best default strategy. These were also averaged using random-effect weights. We present a case
study in which a prediction model (LR2) for malignancy of ovarian tumors is evaluated in a dataset of
5914 women recruited at 13 oncology referral centers and 11 non-oncology centers. We computed
separate weighted averages of NB for oncology and non-oncology centers, thereby re-estimating τ2
in each sub-population.
Result: There was considerable heterogeneity in NB: 97.4% (at t=0.1, 95% CI 96.9% to 97.9%) of the
variance in NB was due to differences between centers. The NB corresponding to t=0.1 was 0.339
(95% prediction interval 0.063-0.616) for oncology centers and 0.111 (0.026-0.196) for non-oncology
centers (see figure). LR2 was always better than the best default strategy in the average nononcology center. However, in the average oncology center, the NB of LR2 was lower than the NB of
treating all patients when t ≤0.1. Risks of malignancy were underestimated in a number of oncology
centers. Refitting LR2 to resolve calibration issues improved the clinical utility.
Conclusion: We conclude that NB can be highly heterogeneous in multicenter studies. NB may
increase because of increased prevalence of malignancy, and decrease due to insufficient calibration
or reduced classification performance of the model in specific centers. This heterogeneity should be
recognized and explored using appropriate techniques.
Random-Effect Weighted Net Benefit Facilitates the Evaluation of Prediction Models
in a Heterogeneous Multicenter Population
Laure Wynants, MSc(1,2), Dirk Timmerman, PhD, MD(3,4), Sabine Van Huffel, PhD(1,2) and Ben Van Calster(3,5)
(1) KU
Leuven, Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Leuven, Belgium, (2) KU Leuven iMinds Medical IT Department, Leuven, Belgium, (3) KU Leuven Department of Development and Regeneration, Leuven, Belgium, (4) Department of Obstetrics and Gynecology,
University Hospitals Leuven, Leuven, Belgium, (5) Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.
Decision-analytic measures are used for the evaluation of clinical
prediction models
Collect
patient
data on an
event and
predictors
Develop a
mathematical model
to predict
the event
Evaluate
the model
in new
patients
The random-effect weighted NB is lower than the NB from the
pooled analysis
Refitting the LR2 model improves the calibration and the NB
NB model -max NB treat all , NB (treat none)
o Relative Utility=
outcome prevalence-max NB treat all , NB (treat none)
Use the
model in
clinical
practice
Update
the model
if needed
Extension: Random-effect weighted Relative Utility
o An alternative measure of clinical utility.
o Easier to compare models at low cut-offs.
o If RU<0, the model is “harmful”, i.e. it is better to use one of the default strategies.
o Model evaluation is traditionally done in terms of:
o More complex dependence on prevalence: negative association for t<prevalence, positive for
• Correctness of classification of events and non-events (C-index,
t≥prevalence.
sensitivity, specificity,…).
• Accuracy of predicted risks of experiencing the outcome (calibration).
o Decision-analytic measures quantify the clinical utility of the model, e.g. Net Benefit (NB).
o Net Benefit=
number of true positives-w×number of false positives
total number of observations
• w is the ‘harm-to-benefit ratio’ of treating a false positive versus a true
positive.
• w= t (1-t), where t is the risk threshold t used to classify patients as
positive (events) or negative (non-events).
• A model can be compared to the default strategies of classifying all as
o The NB was remarkably low in a number of oncological centers.
positive (treat all) or negative (treat none).
o The probability of cancer was underestimated in these centers.
• The higher the NB, the more clinical utility.
o There is considerable between-center heterogeneity (I2=97.4 at t=0.1, 95% CI 96.9 to 97.9).
o We refitted the LR2 model, re-estimating predictor effects, allowing for separate intercepts for
o A few large centers with high NBs have a high impact in the pooled analysis, which is
Random-effect weights can be used to combine center-specific
NBs in a multicenter dataset
o Multicenter studies: to enhance representativeness of data or reduce recruitment times.
o Models may have more clinical utility in one center than another.
moderated by the random-effect weighting.
The heterogeneity in NB can be partly explained by differences
between oncological and non-oncological centers
oncological and non-oncological centers, and a random center effect.
o This yielded more accurate predicted probabilities and hence better NBs.
NB is used to compare competing prediction models
o Random-effect weights 1/(se2+τ2) to combine NBs, with se2 the within-center and τ2 the
insufficient calibration or reduced classification performance of the model in specific centers.
o Hence, NB can be highly heterogeneous in multicenter studies.
Acknowledgements
between-center variance of NB.
This work was supported by a PhD fellowship from the Flanders’ Agency for Innovation by Science and
Technology (IWT Vlaanderen) to LW; a postdoctoral fellowship of the Research Foundation-Flanders
(FWO) to BVC; a fundamental clinical research fellowship of the Research Foundation-Flanders (FWO) to
DT.
Bubble size represents the number of observations per center
τ2>0
References
τ2=0
0
0,1
0,2
0,3
Weights
0,4
0,5
0,6
o We compared the NB computed from a multicenter dataset (pooled analysis) to the weighted
average of center-specific NBs using random-effect weighting.
o We computed se2 using the regular bootstrap.
o Proc Mixed (SAS) was used to compute the weighted NB.
A case study on the clinical utility of the LR2 model to classify
ovarian masses
A logistic regression model to compute the probability of malignancy of an ovarian mass
based on ultrasound and clinical characteristics.
o
o NB may increase because of increased prevalence of malignancy, and decrease due to
o This heterogeneity should be recognized and explored using appropriate techniques.
o Random-effect weighting assumes a distribution of NBs instead of one “true” NB.
o
Conclusion
Evaluated in an international dataset of 5.914 women recruited at 13 oncology referral centers
o NB increases with the prevalence of events.
and 11 non-oncology centers (regional or tertiary).
o Malignant tumors are more prevalent in oncology centers than in non-oncology centers.
o
LR2 and LR2-refitted perform well for all risk thresholds.
o
LR2-refitted and simple rules outperform the other models at relevant risk thresholds for
cancer detection (0.03≤t≤0.20).
Baker SG. Putting Risk Prediction in Perspective: Relative Utility Curves. Journal of the National Cancer
Institute. 2009; 101(22):1538-1542.
Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of
the Royal Statistical Society Series A, (Statistics in Society). 2009; 172(1):137-159.
Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):15391558.
Hilden J. Prevalence-free utility-respecting summary indices of diagnostic power do not exist. Stat Med.
2000; 19(4):431-440.
Kaijser J, Bourne T, Valentin L et al. Improving strategies for diagnosing ovarian cancer: a summary of the
International Ovarian Tumor Analysis (IOTA) studies. Ultrasound Obstet Gynecol. 2013; 41(1):9.
Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and
Updating. New York, NY: Springer US; 2009.
Timmerman D, Testa AC, Bourne T et al. Logistic Regression Model to Distinguish Between the Benign
and Malignant Adnexal Mass Before Surgery: A Multicenter Study by the International Ovarian Tumor
Analysis Group. Journal of Clinical Oncology. 2005; 23(34):8794-8801.
van Klaveren D, Steyerberg E, Perel P et al. Assessing discriminative ability of risk models in clustered
data. BMC Med Res Methodol. 2014; 14(1).
Vickers AJ, Elkin EB. Decision Curve Analysis: A Novel Method for Evaluating Prediction Models. Med
Decis Mak. 2006; 26(6):565-574.
Vickers A, Cronin A, Elkin E et al. Extensions to decision curve analysis, a novel method for evaluating
diagnostic tests, prediction models and molecular markers. BMC Medical Informatics and Decision
Making. 2008; 8(1).