Download Utah Talk - University of Utah School of Medicine

Measures in Health Services
Research: From Constructs to Care
Jacob Kean, PhD CCC-SLP
Research Health Scientist, Center for Health Information and
Communication, Roudebush VA Medical Center
Research Scientist, Regenstrief Institute
Assistant Research Professor, Department of Physical Medicine and
Rehabilitation, Indiana University School of Medicine
1
High-Level
Motivation:
• To improve the delivery,
quality, cost, access to, and
outcomes of care.
Approaches:
• Development and
implementation of
measures
• Health information
exchange
Contexts:
Actions
Interpretations
Models
Constructs
• Medical and psychiatric
rehabilitation
• Primary care
2
CONSTRUCTS: DELIRIUM/POSTTRAUMATIC AMNESIA
3
Construct: Delirium/PTA
• Post-traumatic amnesia (PTA)
– Duration
– Measured retrospectively, and later prospectively
– Index of injury severity and prognosis
• Prospective measures operationalized the end
point
• Prospective measures used to log severity, but
poor construct validity when used that way
4
PTA/PTCS = Delirium
Stuss et al. (1999)
definition of PTCS
Lipowski’s (1987)
definition of delirium
• “A confusional state can be
defined as a transient organic
mental syndrome with acute
onset characterized by a global
impairment of cognitive
functions with a concurrent
disturbance of consciousness,
attentional abnormalities,
reduced or increased
psychomotor activity, and a
disrupted sleep-wake cycle.”
• “Delirium is a transient organic
mental syndrome of acute
onset, characterized by global
impairment of cognitive
functions, a reduced level of
consciousness, attentional
abnormalities, increased or
decreased psychomotor
activity, and a disordered
sleep-wake cycle.”
Construct: Delirium/PTA
• 3 Factor Model
– Attention
– Higher-level thinking
– Sleep/wake cycle disturbance
• Primary question:
– Can we identify the endpoint of this period of
impaired consciousness and track severity with
fewer (but construct valid) items?
6
Construct: Delirium/PTA (Kean et al., 2010)
• Sample
– 57 with brain injury were recruited, 18 declined, 3
diagnosed with dementia
• N = 36; Age range 19-91
• No history of brain injury, substance abuse
• Measures
– DSM-IV Diagnostic Criteria for Delirium
– Delirium Rating Scale – Revised-98
– Delirium Diagnostic Tool – Provisional
• Methods
– 3x/week; raters blinded
7
Construct: Delirium/PTA
• Diagnostic accuracy
– ROC analysis at cutoff (≤6) resulted in AUC=0.994
(35/36) classified accurately as referenced against
the DSM-IV gold standard
• Duration
– Correlation between DRS-R98 and DDT-Pro
duration estimates r = 0.975
8
Construct: Delirium/PTA
• Three construct-valid items can accurately
– Identify the end of delirium/PTCS following TBI
– Measure severity
• Representative papers
– Kean & Ryan (2008)
– Kean et al. (2010)
– Seel et al. (2010)
9
MODELS: ITEM RESPONSE THEORY
10
Models: Item Response Theory
• Resistance to delirium construct in brain injury settings
• Our simple, short measure (i.e., DDT-Pro) was limited
– Ordinal-level (nonparametric statistics)
• Spurious interactions, underestimation of effect sizes, and impact
on gain scores.
– Person measures are sample-dependent
– Focus on group-level metrics (e.g., reliability, error)
– Scores obtained from different sets of items (measuring
the same construct) are not directly comparable
– Few techniques for validating response patterns and
systematic variation
11
Models: Item Response Theory
• Build on existing PTA measure (Orientation
Log)
• Incorporate IRT model to:
– Allow better understanding of items and construct
– Achieve interval-level measurement
– Differentiate ability
12
Methods (Kean et al., 2011)
• 257 (321) ratings of 90 patients admitted for
inpatient rehabilitation following TBI
• 48.25 years (SD 18.87; range from 17 to 93),
75% were male
• Twenty unique items from three scales were
administered: O-Log, C-Log, DDT-Pro
• Analyses were conducted with WINSTEPS
version 3.6 using a partial credit model
Hybrid Measure Items
Person-Item Maps
•Hybrid measure has improved
targeting vs. O-Log
•Separation of both measures
suffer due to the poor targeting of
more impaired persons
Models: Item Response Theory
• IRT model-driven approach
– Improves measurement quality and precision
– Construct-relevant items contribute to
differentiation of person “ability”
• Representative Papers
– Kean et al. (2011a, 2011b)
– Malec, Kean et al. (2012)
– Kean, Malec et al. (2013)
– McGuire, Kean et al. (2014)
16
INTERPRETATIONS: RESPONSIVENESS
AND SENSITIVITY TO CHANGE
17
Patient-Reported Outcome Measurement
Information System (PROMIS)
• The PROMIS assessment system is based on a
comprehensive (i.e., physical, mental, social) selfreported health framework composed of many
domains
• Domains are represented as unidimensional
hierarchies of dozens of items, called “item banks”
– Item banks include many items to represent fully the
range of impairment in a given domain
• Developed using IRT
– Item banks that can be administered adaptively or
assembled as static “short forms”
18
SCOPE Trial (Kroenke et al., 2014)
• Enrolled 250 veterans with moderate to severe
and persistent musculoskeletal pain
– Mean age of 55.1 years (range, 28 to 65)
– 83% were men
– Duration of pain was 1 year or longer in 98% of
participants
• Tested a telemedicine/collaborative care
intervention
• 244 patients who completed both baseline and 3month assessments
19
Comparative Responsiveness
• Measures
– PROMIS Pain Interference Short Forms
– Brief Pain Inventory
– SF-36 Bodily Pain
– Reference Standard – Patient-reported global
change (Better, Same, Worse)
• Standardized Response Means
• Standardized Effect Sizes
20
Comparison of SRM
Better
Same
Worse
BPI Severity
0.71
0.13
-0.47
BPI Interference
0.94
0.38
0.03
BPI Total
0.93
0.31
-0.22
PEG
0.86
0.25
-0.14
SF-36 Bodily Pain
0.71
0.38
0.17
PROMIS-29
0.33
0.29
-0.11
PROMIS-57
0.37
0.30
-0.16
PROMIS Pain 6b
0.51
0.27
-0.02
21
Comparison of SES
Pain scale
Intervention change Control change
SES
BPI severity
0.74 (1.83)
0.11 (1.46)
0.38
BPI interference
1.33 (1.94)
0.61 (1.87)
0.37
BPI total
1.04 (1.70)
0.36 (1.45)
0.42
PEG
1.18 (2.07)
0.44 (1.89)
0.37
SF-36 pain
8.24 (16.13)
4.29 (15.76)
0.25
PROMIS®-29
1.81 (5.67)
0.81 (5.88)
0.17
PROMIS®-57
2.05 (5.54)
0.67 (5.81)
0.24
PROMIS® Pain 6b
2.48 (5.27)
0.94 (5.79)
0.28
22
Interpretations: Responsiveness and
Sensitivity
• PROMIS Pain Interference short forms were
– Less sensitive to change
– Less responsive to treatment than BPI
– Surprising finding
• Respondent fatigue
• Placement of measure in interview
• BPI familiarity effect from automated
symptom monitoring
23
Projects Underway
PCORI Kroenke – PI (2014-16)
Role: Co-I
ISDH Kean – PI (2014-16)
NIH (NIAMS) Monahan – PI (2012-16)
Role: Co-I
NIH (NICHD) Malec – PI (2014-16)
Role: Co-I
VA (RR&D) Kean – PI (2012-17)
NIDILRR Hammond – PI (2012-17)
Role: Co-I
24
Brain Research in Aggression and Irritability Network (BRAIN):
Building Evidence-Based Approaches to Managing Traumatic Brain
Injury (Role: Co-I)
• Aggression and Irritability
Impact Measure (AIIM)
• Prior work shows that
expression may be less
burdensome than impact of
irritability/aggression
– E.g., mild irritability likely
interferes less with
participation of a person with
TBI who works in a selfdirected and self-paced job.
25
Measurement-Based Telehealth Care of Mild Traumatic Brain
Injury (Role: PI)
• VA Telehealth systems
for mild TBI are limited
• Assessment of
symptoms + selfmanagement aligns
with clinical practice
guidelines
• Developing measures of
self-management
26
Responsiveness and Clinical Validity of PROMIS Pain and Depression
Measures (Role: Co-I)
Effectiveness and Patient Selection in Post-Hospital Brain Injury
Rehabilitation (Role: Co-I)
• Responsiveness and
minimally important
differences
– PROMIS Pain and
Depression measures
– Mayo-Portland
Adaptability Inventory-4
27
Incorporating PROMIS Symptom Measures into Primary Care
Practice (Role: Co-I)
• PCORI is interested in
integrating PROMIS
measures into clinical
settings
• Effectiveness trial of
providing patient
symptom scores to
clinicians on symptom
improvement,
satisfaction with
treatment
28
NEXT STEPS: ABERRANT RESPONSE,
CSI, PRACTICE-BASED EVIDENCE
29
Aberrant Response
• Persons and items are calibrated together in
IRT analysis
– Person fit and other metrics used in educational
measurement to detect cheating and identify
unique cases for remediation
• Diagnosis and assessment in mild traumatic
brain injury depends heavily on self-report
• Symptom validity approaches are coarse
30
Comprehensive Severity Index (CSI®)
(Horn et al.)
• Severity defined as “physiologic complexity
presented to medical personnel due to the
extent and interactions of a patient’s
diseases”
• Disease-specific: 5,500 disease-specific
groups; over 2,200 distinct criteria. ICD-9
codes trigger disease-specific patient signs,
symptoms, and physical findings used to score
disease-specific and overall severity levels
31
CSI
In what ways can CSI
change practice?
How responsive are
CSI measures?
Can advanced models
improve precision and
overcome missing data?
Extensive Development
32
SCORE! Trial (Cooper, Bowles et al.)
• Fidelity measurement
• RCT Cognitive rehabilitation
– 10 hours week/6 weeks
• Implementation (CFIR constructs)
– Outer setting
– Inner setting
– Characteristics of individuals involved
– Process
33
Components of Practice-Based
Evidence Designs (Horn et al.)
Standardize documentation for :
Process Factors
• Patient Education and
Management Strategies
• Interventions and surgeries
• Medications
Control for:
Patient Factors
• Psychosocial/demographic Factors
• Co-occurring Conditions
• Severity of Illness and Injury
• Genetic information
• Measured at Multiple Points in Time
Measure:
Primary Outcomes
•
•
•
•
•
•
Clinical
Health Status
Functional
Cost/LOS/Encounters
Discharge Disposition
Post-discharge Outcomes
34
Thank You!
• Kurt Kroenke
• Patrick
Monahan
• Linda Williams
• Teresa
Damush
• Flora
Hammond
• Jim Malec
• Chris Pretz
• Al Kozlowski
•
•
•
•
Brian Dixon
Laura Myers
Jessica Coffing
Erica Evans
35