Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Measures in Health Services Research: From Constructs to Care Jacob Kean, PhD CCC-SLP Research Health Scientist, Center for Health Information and Communication, Roudebush VA Medical Center Research Scientist, Regenstrief Institute Assistant Research Professor, Department of Physical Medicine and Rehabilitation, Indiana University School of Medicine 1 High-Level Motivation: • To improve the delivery, quality, cost, access to, and outcomes of care. Approaches: • Development and implementation of measures • Health information exchange Contexts: Actions Interpretations Models Constructs • Medical and psychiatric rehabilitation • Primary care 2 CONSTRUCTS: DELIRIUM/POSTTRAUMATIC AMNESIA 3 Construct: Delirium/PTA • Post-traumatic amnesia (PTA) – Duration – Measured retrospectively, and later prospectively – Index of injury severity and prognosis • Prospective measures operationalized the end point • Prospective measures used to log severity, but poor construct validity when used that way 4 PTA/PTCS = Delirium Stuss et al. (1999) definition of PTCS Lipowski’s (1987) definition of delirium • “A confusional state can be defined as a transient organic mental syndrome with acute onset characterized by a global impairment of cognitive functions with a concurrent disturbance of consciousness, attentional abnormalities, reduced or increased psychomotor activity, and a disrupted sleep-wake cycle.” • “Delirium is a transient organic mental syndrome of acute onset, characterized by global impairment of cognitive functions, a reduced level of consciousness, attentional abnormalities, increased or decreased psychomotor activity, and a disordered sleep-wake cycle.” Construct: Delirium/PTA • 3 Factor Model – Attention – Higher-level thinking – Sleep/wake cycle disturbance • Primary question: – Can we identify the endpoint of this period of impaired consciousness and track severity with fewer (but construct valid) items? 6 Construct: Delirium/PTA (Kean et al., 2010) • Sample – 57 with brain injury were recruited, 18 declined, 3 diagnosed with dementia • N = 36; Age range 19-91 • No history of brain injury, substance abuse • Measures – DSM-IV Diagnostic Criteria for Delirium – Delirium Rating Scale – Revised-98 – Delirium Diagnostic Tool – Provisional • Methods – 3x/week; raters blinded 7 Construct: Delirium/PTA • Diagnostic accuracy – ROC analysis at cutoff (≤6) resulted in AUC=0.994 (35/36) classified accurately as referenced against the DSM-IV gold standard • Duration – Correlation between DRS-R98 and DDT-Pro duration estimates r = 0.975 8 Construct: Delirium/PTA • Three construct-valid items can accurately – Identify the end of delirium/PTCS following TBI – Measure severity • Representative papers – Kean & Ryan (2008) – Kean et al. (2010) – Seel et al. (2010) 9 MODELS: ITEM RESPONSE THEORY 10 Models: Item Response Theory • Resistance to delirium construct in brain injury settings • Our simple, short measure (i.e., DDT-Pro) was limited – Ordinal-level (nonparametric statistics) • Spurious interactions, underestimation of effect sizes, and impact on gain scores. – Person measures are sample-dependent – Focus on group-level metrics (e.g., reliability, error) – Scores obtained from different sets of items (measuring the same construct) are not directly comparable – Few techniques for validating response patterns and systematic variation 11 Models: Item Response Theory • Build on existing PTA measure (Orientation Log) • Incorporate IRT model to: – Allow better understanding of items and construct – Achieve interval-level measurement – Differentiate ability 12 Methods (Kean et al., 2011) • 257 (321) ratings of 90 patients admitted for inpatient rehabilitation following TBI • 48.25 years (SD 18.87; range from 17 to 93), 75% were male • Twenty unique items from three scales were administered: O-Log, C-Log, DDT-Pro • Analyses were conducted with WINSTEPS version 3.6 using a partial credit model Hybrid Measure Items Person-Item Maps •Hybrid measure has improved targeting vs. O-Log •Separation of both measures suffer due to the poor targeting of more impaired persons Models: Item Response Theory • IRT model-driven approach – Improves measurement quality and precision – Construct-relevant items contribute to differentiation of person “ability” • Representative Papers – Kean et al. (2011a, 2011b) – Malec, Kean et al. (2012) – Kean, Malec et al. (2013) – McGuire, Kean et al. (2014) 16 INTERPRETATIONS: RESPONSIVENESS AND SENSITIVITY TO CHANGE 17 Patient-Reported Outcome Measurement Information System (PROMIS) • The PROMIS assessment system is based on a comprehensive (i.e., physical, mental, social) selfreported health framework composed of many domains • Domains are represented as unidimensional hierarchies of dozens of items, called “item banks” – Item banks include many items to represent fully the range of impairment in a given domain • Developed using IRT – Item banks that can be administered adaptively or assembled as static “short forms” 18 SCOPE Trial (Kroenke et al., 2014) • Enrolled 250 veterans with moderate to severe and persistent musculoskeletal pain – Mean age of 55.1 years (range, 28 to 65) – 83% were men – Duration of pain was 1 year or longer in 98% of participants • Tested a telemedicine/collaborative care intervention • 244 patients who completed both baseline and 3month assessments 19 Comparative Responsiveness • Measures – PROMIS Pain Interference Short Forms – Brief Pain Inventory – SF-36 Bodily Pain – Reference Standard – Patient-reported global change (Better, Same, Worse) • Standardized Response Means • Standardized Effect Sizes 20 Comparison of SRM Better Same Worse BPI Severity 0.71 0.13 -0.47 BPI Interference 0.94 0.38 0.03 BPI Total 0.93 0.31 -0.22 PEG 0.86 0.25 -0.14 SF-36 Bodily Pain 0.71 0.38 0.17 PROMIS-29 0.33 0.29 -0.11 PROMIS-57 0.37 0.30 -0.16 PROMIS Pain 6b 0.51 0.27 -0.02 21 Comparison of SES Pain scale Intervention change Control change SES BPI severity 0.74 (1.83) 0.11 (1.46) 0.38 BPI interference 1.33 (1.94) 0.61 (1.87) 0.37 BPI total 1.04 (1.70) 0.36 (1.45) 0.42 PEG 1.18 (2.07) 0.44 (1.89) 0.37 SF-36 pain 8.24 (16.13) 4.29 (15.76) 0.25 PROMIS®-29 1.81 (5.67) 0.81 (5.88) 0.17 PROMIS®-57 2.05 (5.54) 0.67 (5.81) 0.24 PROMIS® Pain 6b 2.48 (5.27) 0.94 (5.79) 0.28 22 Interpretations: Responsiveness and Sensitivity • PROMIS Pain Interference short forms were – Less sensitive to change – Less responsive to treatment than BPI – Surprising finding • Respondent fatigue • Placement of measure in interview • BPI familiarity effect from automated symptom monitoring 23 Projects Underway PCORI Kroenke – PI (2014-16) Role: Co-I ISDH Kean – PI (2014-16) NIH (NIAMS) Monahan – PI (2012-16) Role: Co-I NIH (NICHD) Malec – PI (2014-16) Role: Co-I VA (RR&D) Kean – PI (2012-17) NIDILRR Hammond – PI (2012-17) Role: Co-I 24 Brain Research in Aggression and Irritability Network (BRAIN): Building Evidence-Based Approaches to Managing Traumatic Brain Injury (Role: Co-I) • Aggression and Irritability Impact Measure (AIIM) • Prior work shows that expression may be less burdensome than impact of irritability/aggression – E.g., mild irritability likely interferes less with participation of a person with TBI who works in a selfdirected and self-paced job. 25 Measurement-Based Telehealth Care of Mild Traumatic Brain Injury (Role: PI) • VA Telehealth systems for mild TBI are limited • Assessment of symptoms + selfmanagement aligns with clinical practice guidelines • Developing measures of self-management 26 Responsiveness and Clinical Validity of PROMIS Pain and Depression Measures (Role: Co-I) Effectiveness and Patient Selection in Post-Hospital Brain Injury Rehabilitation (Role: Co-I) • Responsiveness and minimally important differences – PROMIS Pain and Depression measures – Mayo-Portland Adaptability Inventory-4 27 Incorporating PROMIS Symptom Measures into Primary Care Practice (Role: Co-I) • PCORI is interested in integrating PROMIS measures into clinical settings • Effectiveness trial of providing patient symptom scores to clinicians on symptom improvement, satisfaction with treatment 28 NEXT STEPS: ABERRANT RESPONSE, CSI, PRACTICE-BASED EVIDENCE 29 Aberrant Response • Persons and items are calibrated together in IRT analysis – Person fit and other metrics used in educational measurement to detect cheating and identify unique cases for remediation • Diagnosis and assessment in mild traumatic brain injury depends heavily on self-report • Symptom validity approaches are coarse 30 Comprehensive Severity Index (CSI®) (Horn et al.) • Severity defined as “physiologic complexity presented to medical personnel due to the extent and interactions of a patient’s diseases” • Disease-specific: 5,500 disease-specific groups; over 2,200 distinct criteria. ICD-9 codes trigger disease-specific patient signs, symptoms, and physical findings used to score disease-specific and overall severity levels 31 CSI In what ways can CSI change practice? How responsive are CSI measures? Can advanced models improve precision and overcome missing data? Extensive Development 32 SCORE! Trial (Cooper, Bowles et al.) • Fidelity measurement • RCT Cognitive rehabilitation – 10 hours week/6 weeks • Implementation (CFIR constructs) – Outer setting – Inner setting – Characteristics of individuals involved – Process 33 Components of Practice-Based Evidence Designs (Horn et al.) Standardize documentation for : Process Factors • Patient Education and Management Strategies • Interventions and surgeries • Medications Control for: Patient Factors • Psychosocial/demographic Factors • Co-occurring Conditions • Severity of Illness and Injury • Genetic information • Measured at Multiple Points in Time Measure: Primary Outcomes • • • • • • Clinical Health Status Functional Cost/LOS/Encounters Discharge Disposition Post-discharge Outcomes 34 Thank You! • Kurt Kroenke • Patrick Monahan • Linda Williams • Teresa Damush • Flora Hammond • Jim Malec • Chris Pretz • Al Kozlowski • • • • Brian Dixon Laura Myers Jessica Coffing Erica Evans 35