A TWO LEVEL MONTE CARLO APPROACH TO CALCULATING EXPECTED VALUE OF PERFECT INFORMATION:- RESOLUTION OF THE UNCERTAINTY IN METHODS. Alan Brennan, , Jim Chilcott, Samer Kharroubi, Tony O’Hagan, University of Sheffield, England. [email protected] The Two level Algorithm 6) Simulate a data collection exercise for your parameter subset of interest by: sampling each parameter of interest once from its prior uncertain range (1st level) Cost of drug % admissions Days in Hospital Cost per Day % Responding Utility change if respond Duration of response (years) 9) calculate the best strategy given this new knowledge on the fixed parameter of interest i.e. strategy which gives the most net benefit over all the simulations in (8) % Side effects Change in utility if side effect Duration of side effect (years) 70% % Responding 80% 0.3000 Utility change0.3000 if respond 3.0 3.0 (years) Duration of response £ T0 1 2% 1.00 200 10% 0.1000 0.5 10% 0.0500 1.0 88% 0.3119 3.1 70% 0.2120 4.1 -18% 0.0998 1.0 25% -5% % Side effects 20% -0.10 0.00 Change in utility -0.10 if side effect 0.50 Duration of side 0.50 effect (years) - 10% 0.02 0.20 5% 0.02 0.20 31% -0.06 0.70 24% -0.14 0.65 - -7% -0.08 0.05 1,208 £ Total Cost 1,695 0.6175 Total QALY 0.7100 £ 487 A “UK school” version of this algorithm is identical except that for each simulated parameter value at step 11 we would measure the reduction in opportunity loss achieved, and at step 12 simply average the reductions in opportunity loss . -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1,937 £ 629 0.5948 -0.2446 3,256 -£2,570 £4,011 -£3,075 1.6 -£500 £5,267 £ -£1,000 1,558 £ -£1,500 -£2,000 Inc QALY £7,086 Overall EVPI = £1,351 • 1 level UK and US results are different, 2 level approaches are equivalent • a linear model is accurately analysed using 1 level ‘US’. EVPI for Parameters - 2 level methods 1 level EVPI for Parameters Approximations Expression can be seen to be equivalent to a 1 level simulation procedure for overall EVPI Mathematical Formulation: EVPI for Parameters i = the parameters of interest for partial EVPI -i = the other parameters (those not of interest, i.e. remaining uncertainty) E i max E i NB(d, ) | i d 8) EVPI (parameters) = 8) EVPI (parameters) = the overall opportunity loss (5) opportunity loss if parameters were average net benefit given perfect information on parameters of interest (7) Equation 5: Partial EVPI for parameters i (4) – (1) E i max E i NB(d, ) | i max E NB(d, ) Partial EVPI = known (7) – net benefit of the baseline decision (3) This is a 2 level simulation due to 2 expectations Rationale: •Takes account of remaining uncertainty having obtained the information on the parameter of interest Rationale: •Allows the parameter of interest to vary across its range of uncertainty. Implications of Further Mathematics Why it’s wrong: •The parameter of interest may not be discovered at its prior mean – it could be found at any point across the range of uncertainty. •Does not take account of remaining uncertainty having obtained information on the parameter of interest. All other parameters should be allowed to vary. d (4) (5) % Side effects T1 Duration of side effect (years) T1 Duration of response (years) T1 Change in utility if side effect T1 Cost per Day T1 % Responding T1 Utility change if respond T1 Cost of drug T1 % admissions T1 Days in Hospital T1 % Side effects T0 Duration of side effect (years) T0 Duration of response (years) T0 Change in utility if side effect T0 Cost per Day T0 % Side effects T1 % Responding T0 £Duration of side effect (years) T1 -£100 (3) d £100 £0 Duration of response (years) T1 d £200 £100 Change in utility if side effect T1 E max NB(d, ) max E NB(d, ) £300 £200 Cost per Day T1 Global EVPI = £400 £300 admissions T0 Equation 3: Overall or “Global” EVPI (2) - (1) Net benefit of full information minus the net benefit of current information. £500 £400 Cost of drug T0 (2) 1 level UK £500 2 level US 2 level UK Days in Hospital T0 (1) % Responding T1 d £700 1 level US £600 Utility change if respond T1 £800 £700 £600 d Equation 2: full information With full perfect information on parameters we always choose the optimal strategy. Expected net benefit of full information = E max NB(d, ) £900 £800 Cost of drug T1 Equation 1: no further information (the value of the baseline decision) Given current information chose decision giving maximum expected net benefit. Expected net benefit (no further info) = max E NB(d, ) Equation 4: Perfect Information on i Expected Net benefit, perfect info only on i = Why it’s wrong: -0.6 Illustrative Model Conclusions on EVPI for Parameter Methods: Mathematical Formulation: Overall EVPI = the parameters for the model (uncertain currently). d = set of possible decisions or strategies. NB(d, ) = the net benefit for decision d, and parameters % admissions T1 7) For each sample of parameters, work out the optimal strategy given that particular “possible future world”, record the net benefit -0.8 T1 is optimal on 54.5% of simulations Days in Hospital T1 7) For each sample of parameters, work out the optimal strategy given that particular “possible future world”, record the net benefit, and the opportunity loss. Opportunity loss = net benefit of optimal decision - net benefit of baseline decision -1 EVPI for parameters shows duration of T1 response is most important uncertainty % Side effects T0 6) Simulate a “perfectly accurate” data collection exercise for your parameter subset of interest by : •allow parameter set of interest to vary according to prior uncertainty •keep remaining unknown parameters constant at their prior mean value 1,956 £ QALY 2,388 Cost per Baseline decision = T1 Change in utility if side effect T0 6) Simulate a “perfectly accurate” data collection exercise for your parameter subset of interest by : •keeping parameter set of interest constant at their prior mean values •allow remaining unknown parameters to vary according to prior uncertainty £ £0 -1.2 Illustrative Model Results: Duration of side effect (years) T0 “US school” 1 level algorithm EVPI for parameters (e.g. Felli and Hazen) 3 Do steps 1 to 5 above and then …. 1,308 0.8394 £500 -1.4 Illustrative Model :2 treatments T1 versus T0, 10 parameters each on benefits, side effects and costs, all uncertainties characterised as normal distributions, decision rule = £10,000 per QALY % Responding T0 “UK school” 1 level algorithm EVPI for parameters (incl. Claxton and ourselves) 1, 2 Do steps 1 to 5 above and then …. Net Benefit of T1 versus T0 £ 5,405 £437.80 £ 4,967 Utility change if respond T0 i £ Duration of response (years) T0 NetBenefitOfOptimalStrategy NetBenefitOfBaseline Strategy £1,000 Inc Cost 12) The average value of gaining perfect information on the parameter of interest is then the average net benefit given information on the parameter (11) minus the average net benefit given no further information i.e. of the baseline decision (3) Opportunity loss = net benefit of optimal decision - net benefit of baseline decision 5) With full or “perfect” information (i.e no uncertainty in the values of each parameter) we would always choose the optimal strategy. The overall expected value of perfect information is therefore 10, 000 £ £2,000 0.0925 Net Benefit of T1 versus T0 admissions T0 4) For each sample set of parameters, work out the optimal strategy given that particular “possible future world”, record the net benefit, and the opportunity loss. Increment T1 (T1 over T0) 1,499 £ 499 11% 0% 7.05 2.05 589 £ - T0 1,000 £ 10% 5.01 589 £ 10% - Cost per QALY Cost per Day T0 3) Work out the baseline decision – i.e. strategy giving (on average over the 1,000 simulations) the highest net benefit. 11) Repeat steps 6 to 10, say 1,000 times, and calculate the average of the net benefits obtained in (10) Cost of drug T0 2) Simulate (say 1,000) sample sets of uncertain parameter values (monte carlo). Sampled Values T1 1 £ 2% 1.00 200 £ £1,500 Total QALY Days in Hospital T0 1) Characterise uncertain parameters with probability distributions e.g. normal(μ,σ2), beta(a,b), gamma (a,b), triangular(a,b,c) … etc – e 10) calculate the mean net benefit of this new best strategy 0) Set up a decision model comparing different strategies and set up a decision rule e.g. “select strategy if marginal cost per QALY is < $50,000”. This is equivalent to “chose strategy with highest net benefit” where, net benefit = $50,000 * QALY – Cost i 1 £ 8) simulate the other remaining unknown parameters 1,000 times allowing them to vary according to prior uncertainty (2nd level) d Uncertainty in Parameter Means Standard Deviations Increment T0 T1 (T1 over T0) Mean Model 1,000 500 Cost of£drug 1,500 £ 10% 8% -2% % admissions 5.20 0.90 Days in Hospital6.10 400 per £ Day 400 £ Cost £ Total Cost Agreed First Steps in the Algorithm EVPI PopulationToBenefit a Number of Patients in the UK 1,000 Threshold cost per QALY £10,000 Mean Model 7) set the parameter of interest fixed constant at its sampled value Illustrative Model b c Illustrative Model Number of Patients in the UK Threshold cost per QALY Utility change if respond T0 Purpose: Calculating EVPI in health economic decision model enables understanding of variables’ uncertainty and quantifies the value of obtaining further information on the uncertain variables. Recent authors suggest Monte Carlo simulation, but several different algorithms appear to be used (in particular for subsets of model parameters) and there is some confusion as to the most appropriate algorithm to be used. This paper resolves the issue, describes a generalised two level Monte Carlo simulation approach, and identifies when the approach is necessary. We set out a mathematical notation for overall EVPI, and EVPI for parameter(s), and use an illustrative model to show feasibility of the method and differences in results. . . Conclusions 1) A 2 level monte carlo method provides a feasible and accurate solution to EVPI for parameter(s) 2)The 2 level mathematics solves the uncertainty on UK/US approximations to EVPI for parameters. 3) Both are correct using the 2 level approach! 4) Jim Chilcott has applied 2 level monte carlo in a forthcoming NICE appraisal 4 5) Further work is needed on computational issues- how many samples? Non-linear approximations? - see Matt Stevenson, Jeremy Oakley posters d a) The 2 level ‘UK’ and ‘US’ expressions, are mathematically equivalent. b) Expressions for 1 level ‘US’ and ‘UK’ approximations can be derived – The 1-level ‘US’ approximation is accurate if net benefit is a linear function of the model parameter subsets i and -I, .irrespective of the shape of the uncertain probability distributions for parameters. 1 Claxton et al. Bayesian Value of Information Analysis: An application to a Policy Model of Alzheimer's Disease, International Journal of Technology Assessment in Health Care, Volume 17 Issue 1, 2001 2 Chilcott et al. Liquid-based cytology in cervical screening: a rapid and systematic review; Health Technology Assessment 2000; Vol. 4: No. 18 3 Felli, Hazen. Sensitivity Analysis and Expected Value of Perfect Information. Medical Dec Making 1998; 18:95-109 4 Chilcott et al. Kidney Preservation Systems: a rapid review. Health Technology Assessment 2002 (In Press) Acknowledgements: Co-authors: Jim, Samer and Tony (for help in thinking, working through maths and encouragement to get it done). I am a member of CHEBS: The Centre for Bayesian Statistics in Health Economics. Particular thanks to Karl Claxton and Tony Ades who helped finalise our thinking at a CHEBS “focus fortnight” event in April 2002.