* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Survey

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Document related concepts

Transcript

Risk scoring Allan Wardhaugh Why bother? • Comparison of performance between units • Used in RCT to adjust for case-mix Standardised Mortality Ratio • Measured mortality • Predicted mortality – risk adjustment tool • SMR = Measured/ Predicted – SMR > 1 – SMR < 1 performing poorly performing well Making a risk adjustment tool Regression statistics • Target variable – ‘dependent variable’ • Predictors – ‘independent variables’ • Regression statistics use association between • variables to predict one (DV) from another (IV). Simplest form y = b0 + b1(x) where y = predicted value, b0= regression constant, b1= regression coefficient • Multiple regression y = b0+b1(x1)+b2(x2)+…bn(xn) Regression statistics - logistic • Linear multiple regression – DV and IV quantitative • For non-quantitative DV (e.g. dead/alive), logistic regression is used – Relationship with IV may be non-linear • For each IV, odds are calculated for likelihood of having • DV Odds very assymetrical – very small number (0 – 1) if event unlikely – very large if event likely (>1 - ∞) • Rectified by using natural log off odds – called logit – makes it a linear function Regression statistics - logit • Logit = b0+b1(x1)+b2(x2)+…bn(xn) • Probability = odds/(1 + odds) • Logit = ln odds p = logit e /(1 + logit e ) PRISM • Pediatric Risk of Mortality Score • 14 physiological variables – Worst measurement in first 24 hours – Now on PRISM III – relies on scores in first 12 or 24 hours • Probability of PICU death = eR/1 + eR Where R = 0.207 PRISM – 0.005 age(mo) – 0.433 operative status – 4.782 PRISM – example 60 month old non-surgical patient PRISM Score Mortality Risk (%) 3 1.6 6 2.7 9 5 12 8.9 15 15.3 18 25.2 21 38.6 24 46.1 27 68.5 30 80.2 PRISM - disadvantages • Data collection cumbersome (14 variables over a 24 hour period) • May diagnose death rather than predict it (40% deaths occur in first 24 hours) • Score may not allow comparison between units – patients poorly managed in first 24 hours will develop high PRISM score, so disease severity will appear to be greater PRISM III PIM – Paediatric Index of Mortality – initial cohorts • 678 consecutive admissions PICU RCHM 1988 • 814 consecutive admissions RCHM 1990 • 1412 consecutive admissions 1994–5 RCHM PIM – identifying variables • Data collected for admission (for most) and first 24 hours • 34 Physiological Stability Index measurements • MAP, PIP, PEEP, and others • Worst value in first 24 hours for all PIM – derivation of model • All PRISM data collected plus additional information • Univariate analysis carried out on all factors to test for association with mortality (Chi squared dichotomous variables, Copas p by x plots continuous variables) • Factors not associated (p>0.1) excluded from further analysis • Logistic regression analysis used to derive preliminary model. PIM – testing the model • Learning and Test cohorts – 1994 – 96 5695 patients in 8 PICUs (Australia, Birmingham) • Enough patients in each unit to include 20 deaths. • Learning sample data analysed to calculate regression • • • coefficients Model then tested on test sample, and examined for goodness of fit. Regression coefficients re-estimated using all 8 units for final model. Risk of death assigned to 5 groups - <1%, 1–4%, 5– 14%, 15–29% and 30% PIM - results PIM – final equations • elogit/(1+elogit) • Logit = (2.357.pupils) +(1.826.specified diagnosis) +(–1.552.elective admission) +(1.342.mechanical ventilation) +(0.021.(SBP–120)) +(0.071.Baseex) +(0.415.(100.FiO2/PaO2)) –4.873 UHW PICU PIM 1.2 Relative risk 1 0.8 0.6 0.4 0.2 0 <1% 1-4% 5-14% 15-29% Probability of Death (by percentage risk) >30% PIM and PRISM compared • Variables used by PIM that are not used by PRISM are – presence of a specified diagnosis – use of mechanical ventilation – plasma base excess • Variables used by PRISM that are not used by PIM – diastolic blood pressure, heart rate – respiratory rate, pCO2 – the Glasgow Coma Score (three separate variables) – prothrombin time, serum bilirubin, serum potassium, serum calcium, blood glucose and plasma bicarbonate PRISM vs PIM • PRISM predicted 66% more deaths in this • • • • • sample Score altered by treatment in the first 24 hours May diagnose rather than predict death PRISM III data requires 96 measured variables License required Note that neither are adequate fro individual case prediction – apply to populations only PIM - recalibration • PICU outcomes change with time • Referral patterns change with time • Attitudes to withdrawing and limiting care may change with time PIM 2 • 14 PICUs – 8 Australia – 4 UK – 2 NZ • 20 787 patients 1997-1998 • Units randomly assigned to be learning sample or testing sample for new model PIM 2 • PIM applied to new population (all units) – Observed to expected deaths • Poorly performing variables altered to make • • prediction better Re-tested by forward and backward logistic regression to produce new model New model applied to learning sample – coeffciients adjusted and applied to testing sample Calibration findings • Specific diagnosis – Resp illness O:E 160:212 – Non-cardiac post-op O:E 48:82 • 293 coded diagnostic categories examined – In-hospital cardiac arrest associated with increaed risk of death – Asthma. Bronchiolitis, croup, obstructive sleep apnoea, DKA associated with reduced risk • New ‘high risk’ and ‘low risk’ categories introduced • Post – op subdivided into with or without CBP. • IQ <35 omitted (difficult to code reliably) SMR • Australia and New Zealand SMR 0.84 (0.76–0.92) • UK 0.89 (0.77–1.00). New coefficients United Kingdom Paediatric Intensive Care Outcome Study UK PICOS (phase I) PIM mortality ratio (observed/expected unit deaths) by unit. Generated using UK PICOS recalibration 2 1 .5 200 400 600 800 1000 Number of admissions Your unit Other units Control limits Mortality ratio calculated using the UK PICOS calibration of PIM in the UK. Upper and lower control limits represent a 99.9% confidence interval around a mortality ratio of 1 based on the UK PICOS overall mortality of 6.2%.. Phase I outcome • PRISM III 24 hour score re-calibrated for UK • Performance of PIM-2 and PRISM III very similar • PIM – 2 recommended as model of choice as data easier to collect • DoH/ WAG funded • Run from Universities of Sheffield, Leicester and • Leeds First annual report March 2003 – February 2004 PELOD • Death is relatively infrequent outcome (6%) in PICU – Sample sizes needed for trials need to be large to detect different outcomes • MODS more prevalent (11 – 27%) – Correlates well with risk of death – Good proxy outcome measure for risk of death PELOD • Prospective study – 7 PICUS France, Canada, Switzerland • 18months 11998 – 2000 • 1806 patients (<18yrs) probability of death=1/(1+exp [7·64–0·30PELOD score]) League tables • Governments like them • Journalists like them • Local politicians like them • Patients groups like them Do any of the above understand them? • 9 NICUs over 6 years • Crude and risk adjusted (CRIB score) mortality • Hospitals ranked in league tables each year according to W score – W= 100 (observed - expected deaths)/No of admissions. – Mortality lower than expected if W < 0 Results Conclusions • Hospitals varied annually in their league position • Confidence intervals for W scores overlapped for • all hospital every year except year 3 ‘Overall, hospital 1 did perform significantly better than expected but it is debatable whether this makes it a model hospital since its performance was inconsistent’. Summary • PIM/ PIM 2 data easy to collect • Useful in comparing unit performance • Interpret with care if number of deaths low • • • • (especially <20). Not for use as an individual prediction test Important to complete as accurately as possible PICANET randomly check to ensure data quality League tables are unreliable