Download Measurement 2.0 - A Collaborative Outcomes Resource Network

Introduction to
Outcomes Informed Care


The content for this course is offered by A Collaborative
Outcomes Resource Network (ACORN).
 https://www.psychoutcomes.org
ACORN is a non-profit organization promoting Outcomes
Informed Care in behavioral health care and related fields.
 TWiki site provides information, fosters collaboration, and
offers support for organizations launching and nurturing
outcomes-informed care initiatives.
 The organization is supported by health plans, managed
care companies, and employers seeking to improve
treatment outcomes in mental health care.
Course Content
1.
2.
3.
4.
5.
6.
Required and Recommended Readings
What is “Outcome Informed Care?”
Background Research
Measurement Concepts
Clinician’s Use of Feedback
Financial Implications
1. Required Readings
a. Outcomes Informed Care: An overview
b. Outcomes Measurement 2.0: Emerging
technologies for managing treatment
outcomes in behavioral healthcare
c. ACORN Criteria for Effectiveness (ACE)
Treatment Outcomes Evaluation Method
d. Outcomes Measurement: Concepts and
definitions
1. Recommended Reading
This is an up-to-date review
of what really makes a
difference in psychotherapy
outcomes
Duncan, Miller, Wampold & Hubble (Eds.),
Heart & Soul of Change (2nd ed.).
Washington, DC: American Psychological
Association
2. What is “Outcomes Informed
Care?”

Outcomes Informed Care is the philosophy
and method of providing treatment and care
that is informed by outcomes reported by
patients.
2. What is “Outcomes Informed
Care?”

Outcomes Informed Care is done by:



Collecting the patient’s clinical symptoms and
perception of therapeutic alliance throughout the
course of treatment using questionnaires.
Questionnaires are administered frequently so as
to actively inform the patient’s course of
treatment.
Clinicians are provided continuous feedback on
these outcomes to actively improve treatment.
3. Background Research
a.
b.
c.
d.
e.
The “Dodo Bird Effect”
Clinician effects
The relationship between the patient and clinician
Clinician’s use of feedback
Outcomes benchmarking
3. Background Research
a. The “Dodo Bird Effect”

The “Dodo Bird Effect”

Taken from the article by Rosenzweig S. (1936)
titled, “Some implicit common factors in diverse
methods of psychotherapy: At last the Dodo said,
‘Everybody has won and all must have prizes.’”
(Am J Orthopsychiatry 6:412-5)

Refers to the conjecture that all psychotherapies
intended to be effective are roughly equivalent in
their treatment outcomes due to factors that are
common to all psychotherapies
3. Background Research
a. The “Dodo Bird Effect”

Common factors

Three decades of meta-analytic studies have
served to reinforce Rosenzweig’s (1936)
observation (e.g., Wampold et al., 1997)


Lack of evidence for specific treatment effects bolster the
argument that almost all of the effects of psychotherapy
are due to factors common to all psychotherapies
No evidence that effects of treatments have
increased over the past three decades in
psychotherapy and pharmacotherapy research
3. Background Research
b. Clinician Effects

Meta-analyses of psychotherapy outcomes
reveal that the treatment effect due to the
clinician are larger than the techniques used
in the treatment


Recent analyses show that who prescribes the
medication also affects treatment outcomes in
pharmacotherapy outcomes
Other recent analyses suggest that the
psychotherapist even effects the effect of
medication in treatments that combine both
psychotherapy and pharmacotherapy!
3. Background Research
c. Relationship Between The Patient and Clinician
 Relationship building is an Evidence Based
Practice!
“Practitioners are encouraged to routinely monitor patients’
responses to the therapy relationship and ongoing treatment.
Such monitoring leads to increased opportunities to repair
alliance ruptures, improve the relationship, modify technical
strategies, and avoid premature termination.”
—John Norcross & Michael Lambert (2006) from “The Therapy Relationship”
in Norcross, Beutler, & Levant (Eds.), Evidence-Based Practices in Mental
Health, p. 218
3. Background Research
c. Relationship Between The Patient and Clinician

Practice implications for relationship building





Listen to client
Privilege the client’s experience
Request feedback on the therapy relationship
Avoid critical or pejorative comments
Ask what has been most helpful in this therapy
—John Norcross (2009) from “The Therapeutic Relationship” in Duncan,
Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2nd ed.), pp.
116-117
3. Background Research
d. Clinician’s Use of Feedback

Research indicates that routine use of
questionnaires provides clinicians ability to
identify “at risk” cases and prevent premature
termination


Statistical algorithms compare actual patient
improvement against “expected” improvement
based on large normative samples and provide
this information to clinicians
Strong evidence from controlled studies and
real world applications that patients benefit
3. Background Research
d. Clinician’s Use of Feedback

“Yes, it is time for clinicians to routinely
monitor treatment outcome”
“The use of outcomes management systems is ushering in a significant
change in how psychotherapy is conducted. This review underscores
the value of monitoring treatment response, applying statistical
algorithms for identifying problematic cases, providing timely feedback
to therapists (and clients), and providing therapists with problemsolving strategies. It is becoming clear that such procedures are well
substantiated, not just matters for debate or equivocation. When
implemented, these procedures enhance client outcome and improve
quality of care.”
—Michael Lambert (2009) from “Yes It Is Time for Clinicians to Routinely
Monitor Treatment Outcomes” in Duncan, Miller, Wampold & Hubble (Eds.),
Heart & Soul of Change (2nd ed.), p. 259.
3. Background Research
e. Outcomes Benchmarking

Benchmarking refers to the practice of
comparing one set of outcomes to a criterion
(the benchmark)


Commonly, comparison of effect sizes to
“benchmarks” obtained from meta-analyses of
clinical trials
An individual clinician’s average effect size could
also be compared against effect sizes of other
clinicians in a large normative sample
4. Measurement Concepts
a. Measurement 2.0
b. Questionnaire development
i.
ii.
iii.
iv.
Global distress factor
Therapeutic alliance
Reliability & validity
Item response theory
c. Effect size
d. Analysis
i.
ii.
iii.
Case mix adjustment
Inadequacies of analyses used in clinical trials
Hierarchical linear modeling
4. Measurement Concepts
a. Measurement 2.0
General differences:
Measurement 1.0



Reliance on copyrighted and
published questionnaires
Copyright holder may charge
fees for the use of
questionnaires
Copyright holder may place
conditions or restrictions on
the use of questionnaires
Measurement 2.0



Item banks and questionnaires
that use them belong to the
community of users
No fees for questionnaires
constructed from items in the
shared item bank
Each organization is
responsible for determining the
appropriate content (items)
and implementation of
questionnaires
4. Measurement Concepts
a. Measurement 2.0
Differences in questionnaire development:
Measurement 1.0






A pool of items are tested in various
samples
Item analysis used to select items for final
questionnaire
Questionnaire is usually validated through
factor analysis and studies correlating it with
other questionnaires measuring the same
construct
Questionnaire published in final and
unchangeable form
Manual for questionnaire published in paper
form
Many years may pass before a new version
is published
Measurement 2.0






A pool of items are tested in various
samples
Item analysis used to select items; multiple
versions of the questionnaires are created
by combining a set of items that reflect the
needs of the users
Questionnaire validated by factor analysis
and correlational studies
Questionnaires are constantly updated as
data accumulate and needs of users change
Online manual constantly updated as data
accumulate and needs of users change
New version is immediately available if the
other versions do not capture the unique
needs of the users
4. Measurement Concepts
b. Questionnaire Development
i.
What is “global distress?”


Most measures commonly used in mental health
research correlate strongly with a common factor,
commonly called the “global distress factor”
Global distress includes items measuring:




Symptoms of depression and anxiety
Attention and concentration problems
Family and interpersonal relations
Work place productivity and functionality
4. Measurement Concepts
b. Questionnaire Development
ii. Therapeutic alliance


Measuring the quality of the therapeutic
relationship between the patient and the clinician
during the session
Three Components:



Task: Behaviors and processes within the therapy
session that constitute the actual work of therapy
Bond: The positive interpersonal attachment between
therapist and client of mutual trust, confidence, and
acceptance
Goal: Objectives of therapy that both client and therapist
endorse
4. Measurement Concepts
b. Questionnaire Development
ii. Therapeutic alliance

Sample items:





I felt like we talked about the things that were important
to me
I felt like the therapist liked and understood me
I felt the session was helpful
I felt confident that the therapist and I worked well
together
Did you feel that the clinician understood what it was like
to be you?
4. Measurement Concepts
b. Questionnaire Development
iii. Reliability and validity

Reliability



It asks, “how consistent is this questionnaire?”
Cronbach’s coefficient alpha (α) estimates how
consistent the items within this questionnaire is
 α =.9 or higher preferred for measures
Validity



It asks, “does the questionnaire measure what it is
supposed to measure?”
Measures of global distress will correlate highly with
other similar measures (construct validity)
Items are representative of what is supposed to be
measured (content validity)
4. Measurement Concepts
b. Questionnaire Development
iv. Item Response Theory (IRT)


Evaluates how clients at varying
levels of distress reply to each items
Identifies items that are endorsed by
clients with high levels of distress



E.g., thoughts of suicide; worthlessness
Questionnaires can be uniquely
tailored to the users or setting
Questionnaires for outpatient
treatments are appropriately
calibrated for patients with moderate
to severe symptoms
4. Measurement Concepts
b. Questionnaire Development

Sample
ACORN
Questionnaire:
Questionnaires
faxed to data
center for cost
effective
scoring,
analysis &
data
warehousing
Global distress
Productivity
Global distress
Suicidal Ideation
Substance abuse
Alliance
4. Measurement Concepts
c. Effect Size

What is effect size?


Effect size is simply the magnitude of the
treatment effect
Of many effect size units, Cohen’s d is very
informative for treatments



An effect size of d= 1 means that the client improved
one standard deviation on the outcome measure
On a global distress scale, an effect size of d= 0.8 or
higher is considered large
Meta-analyses of large samples of psychotherapy studies
suggest the effect size for psychotherapy using a global
distress scale is approximately d= 0.8
4. Measurement Concepts
d. Analysis
i.
Case mix adjustment

Statistical adjustments are used to control for case
mix differences (severity adjusted effect size)



General linear model (GLM) is commonly used
GLM use both categorical (e.g., diagnosis) and
continuous (e.g., severity) variables in predictive model
Predictive model employs clinically relevant
variables collected at the start of treatment to
predict improvement during treatment


Initial severity, as measured by the intake score, is the
strongest predictor
Other predictors: diagnosis, treatment history, health
4. Measurement Concepts
d. Analysis
ii. Inadequacies of analyses used in clinical
trials

Clinical trials typically:




Randomly assign patients to treatment conditions
Conduct analysis of variance (ANOVA) to compare
among different treatment conditions to see whether or
not treatment results are “statistically significant”
Analysis assumes that the clinicians are all the same
ANOVA as commonly employed in clinical trials
cannot be used in assessing real-world treatment
effects because there are clear differences among
clinicians in their average treatment outcome
4. Measurement Concepts
d. Analysis
iii. Hierarchical linear modeling (HLM)

Clinicians are treated as a factor that influences
outcomes



Patients are “nested” within clinician
Estimates percentage of variance due to clinician
Use of HLM becomes a necessity because we
cannot ignore the effects due to the individual
clinicians

Ignoring effects due to clinicians can lead to erroneously
concluding that the type of treatment makes a difference
in outcomes
5. Clinician’s Use of Feedback

Monitoring trajectory of change:
Clinical Boundary:
Scores above this line
are in a clinical range
Expected change based
on normative sample

Signal score: patient “off track”
and at risk for poor outcome
Patient’s actual scores: probability of good
outcome remains high if patient in the “clinical”
range remains engaged in treatment.
Graphing change permits clinician to quickly
identify cases that are “off track”
5. Clinician’s Use of Feedback

Monitoring effect size:



Data warehouse and web based tool permits
health plans and managed care companies to
monitor effect size
Many health plans and managed care companies
permit clinicians to access their data and monitor
their own outcomes
Outcomes informed clinicians can use data to
market their services
5. Clinician’s Use of Feedback

Monitoring effect size:
Severity Adjusted Effect Size:
 d = 0.5~0.8 (effective)
 d > 0.8 (highly effective)
5. Clinician’s Use of Feedback

Therapeutic alliance scale

Items are heavily skewed in positive direction




I.e., clients report “perfect” or “near perfect” therapeutic
alliance with their therapist
Scale scores are not normally distributed
Cannot calculate reliability & validity using
parametric statistics that assume normality of
distribution
Items are only as “valid” as clinician’s ability
to illicit honest and frank responses!
5. Clinician’s Use of Feedback
Therapeutic alliance scale results

Alliance analyses performed by Jeb Brown, PhD
using data from the ACORN data repository
1.2
Highly effective range
1
Effect Size

0.8
0.6
Effective
Range
0.4
0.2
0
Alliance items completed No items alliance at start
at start of treatment
of treament (n=1192)
(n=1924)
5. Clinician’s Use of Feedback
Therapeutic alliance scale results

Measuring alliance clearly makes a difference
1.20
Highly effective range
1.00
Effect Size

0.80
Effective
Range
0.60
0.40
0.20
0.00
Alliance Change for
Worse
No Change
Alliance Change for
Better
6. Financial Implications

Clinicians vary in their “value”


Effective clinicians produce greater “return on
investment” of employers




Value Index estimates effect size per $1000
Increased productivity gains for treated employees
Reduced medical costs
Effective clinicians enhance the “value” of
medications
Clinicians with good outcome data could have
competitive advantage in attracting business
6. Financial Implications

Certificate of Effectiveness


Similar to concept of “Certified Organic”
Process



Data analyzed by independent party
Applies agreed upon criteria, including minimum effect
size and sample size
Purpose



Increase customer confidence of “value”
Enable clinicians to demonstrate effectiveness to referral
sources such as employers, health plans, and HMOs
Empower clinicians to compete for business and
negotiate contract based on demonstrated “value”
Psychotherapy Works. It Works!
“Monitoring combined with feedback is a simple method, divorced of
theoretical baggage, for providing accountability. The results are
apparent to all who have an interest in the outcome: therapists,
consumers, administrators, and payers. Accepting the premise that
therapeutic factors constitute the engine of change, then monitoring
and feedback offers the means to deliver them. Many are anxious
about the future and deservedly so. At the same time, the profession
has the opportunity to establish itself in its own right. Psychotherapy
works. It works. Therapists now have the ability to show it and the
means to banish the despair in the workforce. The challenge is to put
it into practice.”
—Barry Duncan, Scott Miller, Bruce Wampold, & Mark Hubble (2009) from
“Introduction” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of
Change (2nd ed.), p. 40