Download Draft progress report re the vic mental health project for Ian

A report to the Mental Health Branch,
Department of Human Services
Consumer self-rated outcome
measures in mental health
October 2003
Siggins Miller Consultants
Consumer self-rated outcome measures
Page ii
Table of contents
Introduction........................................................................................................................................... 1
Part 1: A review of the literature on consumer outcome measurement .......................................... 3
The scope of this review ................................................................................................................. 4
Self-rated measures......................................................................................................................... 5
Motives for measuring outcomes.................................................................................................... 5
The purposes of outcome measures ................................................................................................ 9
Health outcomes............................................................................................................................ 10
Consumer identified outcomes...................................................................................................... 14
Consumers’ capacity to assess outcome ....................................................................................... 18
Implementation and administration .............................................................................................. 20
The impact of measurement itself on outcome ............................................................................. 27
References..................................................................................................................................... 28
Part 2: The consultations and development of an evaluation framework ..................................... 33
The consultation process............................................................................................................... 33
The messages we heard from consumers and carers ........................................................................ 34
The value of listening to the consumer’s assessment of outcome................................................. 34
Key issues in outcome measurement identified by consumers and carers.................................... 35
The purpose of consumer self-rated outcome measures ............................................................... 35
Who can and or should use the information? It must be useful for whom?.................................. 36
The domains of outcome that are important to consumers ........................................................... 37
The impact of the continuing possibility of involuntary treatment............................................... 40
The impact of fluctuations in insight and capacity ....................................................................... 40
The need for a mixture of methods ............................................................................................... 41
The need for a set of complementary measures ............................................................................ 41
Consumer advice about implementation....................................................................................... 43
Clinicians’ views of purpose, use, and method ................................................................................. 44
The purpose of consumer self-rated outcome measures ............................................................... 44
Uses of the information................................................................................................................. 44
When the measure should be applied............................................................................................ 45
The impact of fluctuations in insight and capacity ....................................................................... 45
The need for a combination of methods........................................................................................ 45
The need for a set of complementary measures ............................................................................ 45
Features of the instrument............................................................................................................. 46
The domains of outcome clinicians think are important............................................................... 47
Process issues identified by clinicians .......................................................................................... 50
Domains of importance compared ................................................................................................ 51
Determinants of outcome .............................................................................................................. 52
Part 3: A framework for evaluation.................................................................................................. 53
The implications of purpose for choice of measures .................................................................... 53
Objectives of an evaluation framework ........................................................................................ 53
Development of a measurement evaluation framework................................................................ 54
The concept of outcome................................................................................................................ 56
Part 4: Application of the framework to measures.......................................................................... 58
Description.................................................................................................................................... 58
Domains of applicability............................................................................................................... 58
Acceptability ................................................................................................................................. 58
Practicality .................................................................................................................................... 58
Reliability...................................................................................................................................... 58
Sensitivity to change ..................................................................................................................... 58
Validity ......................................................................................................................................... 58
Siggins Miller Consultants
Consumer self-rated outcome measures
Page iii
Availability of normative data ...................................................................................................... 58
Summary ....................................................................................................................................... 58
References..................................................................................................................................... 58
Bases for selection of measures ........................................................................................................ 59
Measures in Current Use by Australian States ................................................................................ 60
Kessler 10 (K10), Kessler 6 (K6) and K 10 Plus .............................................................................. 60
The Mental Health Inventory ............................................................................................................ 63
Behaviour and Symptom Identification Scale [BASIS-32]................................................................ 66
Other consumer measures.................................................................................................................. 71
Short Form 36 (SF-36) and SF-12.................................................................................................... 71
General Health Questionnaire.......................................................................................................... 74
Depression Anxiety and Stress Scale (DASS) and DASS-21............................................................. 76
World Health Organization Quality of Life – BREF ........................................................................ 79
The Quality of Life Interview ............................................................................................................ 81
Emergent Measures ............................................................................................................................ 84
Health of the Nation Outcome Scales-Self Report (HoNOS-SR) ...................................................... 84
Manchester Short Assessment of Quality of Life (MANSA) .............................................................. 86
Recovery Assessment Scale............................................................................................................... 87
Other measures ................................................................................................................................. 89
Summary ........................................................................................................................................... 92
Recommended elements in a research strategy ................................................................................ 93
1. Collection of data on adopted measures ................................................................................... 93
2. Direct comparison of adopted measures ................................................................................... 93
3. Development of one or more brief measures tapping additional domains................................ 93
4. Further development of multi-ethnic versions of instruments and instruments applicable to
Aboriginal and Torres Strait Islanders .......................................................................................... 93
Part 5: Conclusions and recommendations ...................................................................................... 94
Appendix 1: Draft project outline for progressing to the next steps................................................. 96
Appendix 2: Interview protocols ....................................................................................................... 98
Appendix 3: Reference group and informants ................................................................................ 102
Siggins Miller Consultants
Consumer self-rated outcome measures
Page iv
Executive summary
In August 2002, the Mental Health Branch of the Victorian Department of Human Services appointed
Siggins Miller Consultants to undertake a comparative analysis of the consumer self-rating outcome
measures now in use in mental health services throughout Australia, identify international best
practice, and make recommendations towards a uniform national approach.
Literature review
Part 1 sets out the results of a comprehensive review of national and international literature to identify
best practice in the development and use of consumer self-reporting measures, and to analyse the
advantages and disadvantages of each measure from a clinical and a consumer perspective, including the
motives for measuring outcomes, the purposes of outcome measures, health outcomes and consumeridentified outcomes, consumers’ capacity to assess outcome, issues in implementing and
administering consumer self-rated outcome measures, and the impact of measurement on outcome. It
includes a comprehensive bibliography.
Consultations
Part 2 analyses the content of consultations with stakeholders in every State and Territory, including
consumers, carers, clinicians and service providers, mental health service management,
representatives of government authorities, and nominated groups and individuals. Face-to-face
consultations included 96 consumers of mental health services who participated in group discussions
in every State and Territory; 33 members of state-based Consumer Advisory Groups and consumer
bodies, 35 carers, mainly in group discussions; and 47 State and Territory mental health officials.
Phone or email consultations included 28 mental health clinicians and academic researchers.
Consumers and carers
The messages we heard from consumers and carers included these recurrent themes:
Consumers and carers alike believe it is essential to introduce the consumer’s view into how the
system is judged in its performance, the appropriateness of the service mix at the system level, and the
appropriateness of interventions at the individual and family level. Consumers and carers thought the
potential purposes for outcome measurement were to assist individual case management, help
consumers interact better with case managers, empower consumers, allow the government’s
evaluation of the system to be well grounded in consumer experience, help determine the outcome of
services at unit or service level, and the system at regional, state/territory, and national levels, and
encourage the system not to pay for things that make no difference to people’s quality of life.
The measure should be useful for consumers: The measure must be useful and useable at the personal
and clinical levels, or it is not an ethical use of consumer information and time. Instruments must be
useful to consumers in helping to make sense of, cope with and recover from mental illness.
The measure should be useful for case managers: Consumers hope the information will improve their
capacity to be involved in decision-making about treatment and support and future planning.
Clinicians and case managers can use the information in charting progress and tailoring treatment,
psychosocial rehabilitation, and health maintenance. Outcome measures should generate data that
could be used to change attitudes and reduce stigma.
The measure should be useful at service level: Outcome data should be useful for research purposes
so that the system better understands the true course and impact of a mental illness or the different
forms it takes, rather than merely how it presents to the treatment system. Some consumers say that, if
the aggregate data are used for research or evaluation purposes, consumers and carers - not just
researchers or clinicians - must interpret the information. Consumers also believe an important aspect
of outcome evaluation is to provide the system with timely data on the unintended negative
consequences of policy changes. Even policy decisions beyond the mental health system could have a
critical impact on outcome.
In all locations, the heart of the discussion with consumers concerned the domains of outcome they
hoped a self-rated measure would cover. Across all the groups we consulted, there was a high level of
Siggins Miller Consultants
Consumer self-rated outcome measures
Page v
consistency about what consumers and carers regarded as key issues in outcome measurement. The
domains of outcome that should be measured are the domains consumers themselves use to judge how
well they are going (or for carers, how the people they are caring for are going). In consumers’ own
words, the measure should ask the consumer about:
Coping and resilience: Strengths and resilience, coping skills and strategies; feeling confident to act
on danger signals in a positive way; knowing what to do when you feel you are getting ill; helpseeking behaviour, being positively inclined towards seeking help; the extent of return to your
own baseline; visualising a time in your life when you were OK; no longer in victim mode.
Hopefulness: Optimism, hope for the future; sense of connectedness or isolation; belief about the
likely trajectory of your life.
Empowerment: Social support from other people in similar circumstances; feeling of control over
your own life; a sense of purpose and value to the community; personal goal attainment and
meeting one’s own goals; improved self efficacy, belief that you can do things for yourself; how
often you do what you consider to be your own enjoyable pursuits.
Personal relationships: Your perception of the impact of your illness on others; other people’s
reactions to you – good, bad, and indifferent; social and behavioral outcomes; the strength of your
social network; socialising with family, and with peers; how you think others close to you think
you are going (housemate, sibling, parent, spouse); whether you are remaining sociable and not
isolating yourself; concern for others, participation, and leadership; whether people believe what
you say; your credibility and ability to influence others, or by contrast, your experience of
discrimination and loss of credibility; how you relate to family or carers and advocates; how you
relate to health professionals, and they to you in mutual respect; how you relate to the
professionals in making decisions about treatment and care; sexuality and relationships, the health
of the sexual part of your life; and relationships with pets
Functioning and well being: How you feel you are doing in relation to your various roles as a parent,
spouse, worker, housemate etc; being able to do the things you did before - normal things like
reading, studying; activities of daily living, like cooking; functioning at home and in general
social situations; ability to sustain your own standard of living; keeping commitments and
appointments; sleeping patterns - too much or too little; where you are living; are you able to meet
financial commitments; employment – paid and unpaid ( “a big area of outcome”); participation
in education; thinking of a time when you were well and functioning at your best, and how you
are you now in physical health, feelings, social connections, employment etc compared with that
time; balanced recreation and leisure; spirituality or religious activity.
Extra-treatment factors that influence outcome: Things happening in your life that affect how you are
feeling; the intervening events or things that affect how well you are doing; what else, other than
treatment, is going on in your life that may affect outcome.
Symptoms of illness: understanding signs of relapse; the onslaught of unconnected thoughts; mood
state fluctuations; finding fault with everything and everyone; domineering attitudes, overly
assertive or aggressive in responses to others; level of symptom control; social withdrawal.
Physical health: the impact of medication (“It can be worse than the illness.” “The side effects can be
so debilitating.” “Compliance doesn’t mean you feel well.”); nutrition and diet, caffeine intake;
drug and alcohol use, and drug and alcohol related problems; physical health; tiredness; personal
hygiene; minimal hospitalisations.
Health literacy –knowing how to negotiate the health and welfare system: knowledge of and use of
support services; if things go wrong, knowing what to do and where to go, and that help is
accessible; partnership in care - how empowered you feel to discuss care and medications and
needed support; carers, consumers and doctors working together; managing inpatient to outpatient
transitions in the system; involvement in decision-making.
Consumers and carers gave practical advice about implementation of self-rated measures:
Siggins Miller Consultants
Consumer self-rated outcome measures
Page vi
•
•
•
•
The continuing possibility of involuntary treatment has implications for how clinicians present the
survey to consumers and carers, and the need for consumer education about the benefits of
outcome measurement and how it will be used.
Both consumers and carers say that from time to time for all consumers insight and capacity
fluctuate owing to the chronic relapsing nature of most mental illness. For people with
unremitting severe illness, it may not ever be possible to get a valid consumer self-rating of
outcome.
All consumers and carers find the use of ‘tick flick’ check box rating scales too limiting. Outcome
measures must allow consumers to opt in and opt out from time to time.
Ideally, for service planning purposes, it would be useful to measure the health status of people
who have exited the system, not just those chronically in it.
•
Consumers and carers say that, whatever the instrument, permission must be given, and
encouragement and practical support for some people to complete the surveys. People filling out
the survey should finish it feeling better, and it must not damage the consumer.
•
While not directly related to outcome measurement, carers and consumers gave many examples of
perversities in the system that may be helped if the outcome and satisfaction measures chosen
allow for painting a full picture.
Consumers persistently report that mental health professionals fail to ask these sorts of questions
in many instances, and pathologise normal responses to difficult life events. There were repeated
comments that psychiatrists and nurses tended to pathologise everything a consumer or carer said.
Many consumers and carers are aware of the difference having a case manager makes to
outcomes. The system must ensure that both people who have a case manager and those who do
not complete the measures: if the measurement process cannot track that difference, it loses
valuable information for evaluation purposes.
Consumers and carers believe there is a need for ways to validate self-reports. They are pleased
that efforts are being made to include the consumer view, but believe that the true picture will
emerge only by looking at the consumer-rated outcomes alongside clinician and family/carer
proxy ratings.
Carers who have had elderly relatives in the mainstream health system are aware of efforts to
involve the family at the time of diagnosis or recuperation, and in discharge planning. Unlike the
mainstream, carers are not involved in discharge planning in the mental health system, but must
cope after discharge, and often there is no discharge planning at all. The system seems to
underrate this aspect of the experience of mental illness, and does not provide support for it.
•
•
•
•
Clinicians and academics
Clinicians and academics said a consumer self-rated measure should be useful:
•
•
•
•
for case managers: regular routine feedback is very important for both clinicians and consumers.
Information should be used to measure change and inform treatment. Information must be
available for use in these ways to ensure staff commitment to the process.
for inpatient settings: Information from consumers helps identify admission goals, review
progress towards these goals during the admission, and develop discharge plans.
for team meetings: Information from consumers is discussed in team meetings and provides a
platform for peer review, feeds into the case manager’s decision making, and validates or
modifies decisions.
at unit level: Aggregated consumer outcomes may be useful to determine an average base level of
progress for measuring treatment outcomes for a particular unit, and to allow mental health units
to view consumers in the setting of information supplied from other sources.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page vii
Clinicians can use consumer self-rated information for:
•
initial and continuing assessments, giving insight into their current thought processes, perspectives, and level
of functioning, relevant areas of concern that form the focus of intervention, and areas they want to change
(goal setting).
•
treatment planning, encouraging more consumer-focussed activities and helping consumers be
more goal directed.
dialogue and rapport building: as they discuss the items in consumer-rated instruments, clinicians
and consumers open up a dialogue that before they may have found difficult to broach or discuss.
Self-report measures give consumers the opportunity to state their own assessments that may vary
from the clinicians’, and facilitates the validation process for consumers.
measuring progress during treatment and influence planning. To monitor progress, consumers
need to provide information at different points over the course of their contact with the service.
screening to determine suitability for program attendance.
group work, to provide baseline measures, facilitate discussion of planning, and provide feedback
on change.
service evaluation: results from consumer feedback are invaluable in assessing the effectiveness
of services, and many consumers are eager to provide feedback to services. There are very few
other outcome measures of quality improvement if consumer feedback is not gathered.
planning in meetings where the future directions of a service are discussed.
short-term intensive rehabilitation and longer-term rehabilitation both use consumer self-report
measures, especially in identifying goals and reviewing progress over time.
reports: data from consumer self-reports may be included in assessment reports, progress reports
and discharge summaries.
•
•
•
•
•
•
•
•
Clinicians said self-report measures should be administered at significant points throughout the
consumer’s contact with a service, particularly on admission, three monthly review, and on discharge.
Clinicians and some academics said a combination of outcome measures was necessary to overcome
errors in self reporting, cognitive distortions, and social desirability distortions, and the points of
agreement and disagreement among the measures was important. Several clinicians said a set of
complementary measures caters for the differing perspectives of consumers, clinicians, and significant
others. Moreover, different stakeholders needed different sorts of information. Some recognised a
need for a combination of measures including one supplied by “an objective party” and a self-report
(one without the other is inadequate).
The domains of outcome clinicians identified as important were these:
Life circumstances: Stable and adequate accommodation; the financial situation: financial concerns
reduce recovery, but it helps when income is reliable and finances are managed well; involvement
in education and training; employment, increased involvement in work, loss of employment,
problems at work, volunteer work.
Roles and activities: Routines and changes in routines; satisfactory functioning in significant life
roles; level of functioning in daily activities; leisure activities increased or decreased - projects
and hobbies and recreation; decreased or increased activity levels (eg staying in bed: being
productive, ability to cope with daily activities.
Independence: Able to make decisions, to pay rent, etc, and maintain self-care: any changes in
appearance; dietary habits; weight
Behaviour: Appropriate or inappropriate use of prescribed medications; misuse of alcohol and other
drugs; sleep patterns; self-harming behaviour; inappropriate regressive or aggressive behaviour.
Use of services: Medical and support services appropriately accessed; increased contact with multiple
health practitioners.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page viii
Events: Contact with case manager increased or reduced by the consumer; missed or rescheduled
appointments, increased or reduced contact by family; medication regime stable or unstable;
hospital admissions: frequency and length of stay.
Interpersonal relationships: Communication: ability to report distressing feelings; honesty in
communication; expressing thoughts and feelings; communication with significant others;
“connectedness” with family and friends; capacity for confidences; the opportunity to nurture
some one, something, some project; contributing to the well-being of some person or thing;
expansion of the social network; relationships with significant others: partners, family and
friends; the strength of the social support network, the level of social interaction level and
isolation; frequency of seeking reassurance from others.
Mental state: Signs and symptoms becoming worse, experiencing negative emotions, anxiety, panic,
despair, sadness; deterioration in mood, level of dangers; signs and symptoms reduced or
managed well; emotional stability: psychological well-being: a sense of hope, a view to the future,
purpose and vision; a sense of control of circumstances; stress managed appropriately; positive
emotions; spontaneity; appropriate humour; vitality.
Illness awareness/ insight: Symptoms recognised; symptoms not the dominant feature of consumer’s
life from the o consumer’s standpoint
Expectations of others: Reports by friends/family of improvement or deterioration; reports by GP;
unsolicited feedback from other people and agencies; consumer self-evaluations positive or
negative: sense of self-esteem; self image; feeling productive; feelings of worthless and low selfesteem; suicidal ideation; consumer expectations of self: realistic expectations of level of mood:
acceptance of not having to feel ‘good’ all the time; future goal setting becomes appropriate;
consumer expectations of others: realistic or unrealistic.
Medical / health: Changes in physical health; pain; presence or absence of medication side-effects.
Cognitive functioning: Changes in cognitive functioning; goals: ability to plan for the future and set
goals; distorted or realistic thinking; blame; negative thoughts; misinterpretation of others’ input.
The domains where clinicians say consumers are the best or only source are how they feel; thoughts,
ideation, suicidal ideation, reflections; hallucinations and delusions; sleep, diet, exercise; substance
use; functioning; self-evaluation of health status, ability to cope, level of distress; self-esteem;
perceptions of their relationships with others; goals of treatment; quality of life and satisfaction.
Ratings that can be validly obtained from consumers include functional assessment of level of
activity; amount of socialisation; achievement of goals; sense of pleasure; routine: work duties,
interests, and periods of rest; identification of expectations and the extent they are achieved; level of
activity over time; level of wellness over time; consumer satisfaction survey; perceived contentment
with life; satisfaction with relationships including with treating professionals; and symptom ratings.
Consumers and carers should be involved from the start, clinicians believe – for example, having
input into the development of protocols. Information on instruments and the uses of information
presented to consumers by information sheets or brochures provided to all consumers, possibly by
inclusion in service information booklets, rights and responsibilities brochures and the like. Consumer
participation should not be mandatory and the process should be non-threatening. Introduction of any
new instruments should be conducted in a slow and progressive manner, and should be as simple as
possible. Successful implementation requires feedback to encourage compliance.
Implementation of outcome tools involves culture change in a system. This issue was a central
concern for some of the most experienced clinicians. One of them said that to engage consumers in
completing these instruments and for staff then not to use them to enhance the quality of treatment
would be ‘criminal’. The introduction of a suite of outcome measures was a challenge to the entire
system and its workforce. If staff observe the use of consumer feedback working in practice, they are
less cynical. The use of consumer provided information represents a significant shift for many
practitioners. A critical ingredient in acceptance of such measures is strong endorsement from
relevant State Government officials. Successful implementation requires managerial understanding
and support. Without a commitment to use the results, “outcomes measures will be viewed by
clinicians as a bureaucratic and clinically irrelevant task”.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page ix
Evaluation framework
In Part 3 of the report, we combine the results of the literature on outcome measurement and of the
consultations with consumers, carers and clinicians to develop a framework for evaluating existing
and emerging consumer self-rated outcome measures.
Competing purposes have implications for the choice and evaluation of particular instruments. To be
clear about purpose is an essential aspect of choosing an instrument (or instruments within a suite of
measures). All instruments do not display the same suitability to achieve diverse purposes. The
properties of measures that become important in an evaluation framework will differ according to the
intended purpose. Such potential purposes may include the usefulness of a measure at the population
level, at the system or service level for evaluation and accountability purposes, and at the individual
level, to allow consumers, carers and clinicians to track those dimensions of outcome they deem to be
important in the context of individual recovery and rehabilitation.
The framework for evaluation that emerges includes these valued objectives:
Usefulness at the individual level: Carers, consumers and clinicians all say they want consumer selfrated outcome measures that are useful at the individual clinical level.
Assurance that the mental health system is effective: All stakeholders ideally want a system where
they can be confident that it works - that is, those who need help can get it, what is provided
works in a holistic way, and the way services are offered is humane and supportive.
Complementary measures: While each sector of the mental health system in Australia already has
clinician-rated measures, routine use of consumer self-rated measures is in its early stages here,
but stakeholders identified a need to develop a collateral measure in the adult and aged suites that
families and carers may complete alongside the consumer and clinician ratings.
Resource and workforce implications: State and Territory officers express concern about the planning
and resource allocation, workforce development, and management issues involved in choice of a
measure or suite of measures.
Two widely recognised Australian studies offer guidance for developing an evaluation framework.
Andrews et al (1994, 29-33) say the criteria for use in routine clinical practice are that ‘consumer
outcome measures should be applicable, acceptable, practical, reliable, valid, and sensitive to change’.
Stedman et al (1997, 14) summarise these criteria. This report applies the results of the literature
review and consultations to build on these criteria.
“The measure must be applicable” that is, address dimensions important to consumers and useful for
clinicians in conducting treatment. It should also provide information that facilitates the management
of services.
Comments: It is clear that these dimensions go well beyond symptoms, disability, and consumer
satisfaction. The dimensions of greatest importance to consumers are their coping skills, and their
capacity for resilience in dealing with personal relationships and day-by-day functioning.
Consumers also want measures that address optimism and connectedness, social support, personal
goal attainment, physical health, factors outside treatment that influence outcome, and knowing
how to negotiate the health and welfare system, as well helping them to monitor the symptoms
and effects of their illness.
Other issues of applicability include these questions: Is the instrument useful to the consumer and
their carers to track how they are going over the course of an illness or treatment? Is the
instrument useful for individual clinical purposes - that is, could it be used as a tool to inform
clinical management and case management? Does the survey cover significant comorbidities - in
particular, alcohol and drug related comorbidity? Does the measure potentially add to or fit well
with the other sources of outcome data (clinician, population) needed to present a complete
picture of outcome at system and program levels? Does the instrument have a relationship with a
population measure to allow comparisons of the in-treatment population with people with a
mental illness who are not in treatment? Will the data at the aggregate level be useful for system
Siggins Miller Consultants
Consumer self-rated outcome measures
Page x
planners, evaluators and for resource allocation? Does the survey cover variables that could
inform culture and system changes?
“The measure must be acceptable” that is, brief, and clear in purpose, wording, and interpretation.
Comments: Consumers have differing views about brevity – some believe it must be brief to be
acceptable, but others prefer a longer one that is less vulnerable to short-term distress, and gives
them opportunity to monitor other aspects of their lives. The issue is closely related to the
consumer’s sense of the relevance of the measurement to his or her own purposes. Many
consumers would value an instrument that allows open-ended responses, and creates an
opportunity for the respondent to provide information about their context that would help to
interpret any observed changes.
A measure needs to have culturally and linguistically appropriate forms to cover the diversity of
the Australian population. The wording and content of the measure should be appropriate to the
cultural context being canvassed. Are there culturally appropriate versions of the measure? Does
the instrument require linguistic or cultural interpreters for its valid application? Are there
translations of the measure available in relevant languages?
“The measure must be practical” - the burden imposed on consumers and providers in time, costs,
training in use of the measure, and level of skill in scoring and interpreting the data.
Comments: From the perspective of clinicians and managers, the measure should be relatively
cheap, relatively brief given the limited time available for completion, and relatively easy to
administer, score and interpret cost effectively without a high level of training of staff, and easy to
report back on at individual and system levels. Other practical criteria include: Is computer
software available? Is the instrument applicable in both inpatient and outpatient settings? Do the
survey and its use have significant workload implications for the system? Do the protocols for
administering this instrument allow discretion to accommodate periods of ill health and lack of
capacity for informed consent?
A practical issue raised spontaneously in several consumer and carer focus groups is whether the
measure produces information of sufficient value to justify its cost of collection. Questions of
practical administration of particular interest to consumers and carers are these: Can this
instrument be done as an interview as well as self completed? Does the survey have a private
section that does not go on the file? Can it form part of a consumer-held record?
“The measure must be valid” that is, have sound psychometric properties and measure what it is
supposed to measure.
“The measure must be reliable. The measure should (within acceptable limits) provide the same
results when given to the same person on two occasions or by two different people.
Comments: We address these criteria in detail in Part 4, including the questions: Is there evidence
that a measure has well-tested and sound psychometric properties? Are the properties of an
instrument appropriate to the purposes for which it is to be used? Are population norms available
in Australia? Can the results of a measure be compared with results from measures widely used
nationally or internationally in other similar organisations and programs?
“The measure must be sensitive to change” - able to indicate whether a clinically significant change
has occurred for a consumer over consecutive administrations of the measure.
Comment: The property of sensitivity to change in a person’s condition is indisputably an
essential criterion for any outcome measure. However, at this stage of the implementation of
outcome measurement in the Australian mental health system, the link between clinically
significant change and the definition of ‘outcome’ requires deeper consideration.
As an important supplement to the criteria in the Andrews and Stedman framework, we propose an
additional question: “Can the measure function as part of a collective suite of outcome measures?”
Commentators point to the limitations of relying on self-reports, and advocate complementing the
information by using other informants. The clinician and consumer self-rated measures should
complement and add value to each other, and the system needs a carer collateral measure in the
Siggins Miller Consultants
Consumer self-rated outcome measures
Page xi
context of outcome measurement. Does the consumer self-rated measure have parallel clinician
and carer or family measures? Can this instrument function as part of a set of complementary
outcome measures?
Some definitions of outcome suggest a direct linear relationship between treatment intervention and
consumer outcomes. The views and experience of consumers, carers and clinicians paint a very
different picture. When consumer focus or advisory groups have been asked to choose among the
currently nominated self-rating measures, they have encountered the difficulty that what they want is
a different concept of outcome. The outcomes that are real to them and the outcomes they regard as
valuable to record include but go far beyond the responses sought by the existing choices.
The research literature suggests the need for models of outcome that address the complex, interactive,
two-way relationships among the characteristics the individual brings to the treatment encounter, the
characteristics of the treatment experience (quantity, location, intensity, quality of the therapeutic
relationship, congruity of therapist/ consumer expectations of treatment) and extra- or post-treatment
variables such as access to financial, social and physical resources post treatment. Outcome
measurement should be shaped by the understanding that develops out of the complex mix of
elements and experiences that are the determinants of outcome in either ambulatory or inpatient
treatment settings.
Application of the measurement evaluation framework
Part 4 of the report applies the evaluation framework to the three consumer self-rated instruments
currently in use in Australian jurisdictions (BASIS-32, K10, and MHI), and to a range of similar or
emerging instruments, under the headings: description; domains of applicability; acceptability;
practicality; reliability; sensitivity to change; validity; and availability of normative data. In each case,
a summary statement is accompanied by references to articles and websites dealing specifically with
the properties of the measure under consideration.
We conclude that there is an acute need for further development of a brief measure relating to coping,
resilience, recovery, empowerment, hope, and similar consumer-identified domains. The review has
identified some instruments that have potential for such development. Both consumers and carers are
also acutely aware that fluctuations in a person’s condition have an inescapable effect on self-rating,
and there is therefore a powerful argument for developing a collateral carer measure to complement
the consumer self-rated measure. Such a measure could be used in conjunction with existing measures
adopted by Australian states, providing a well-rounded assessment that captures many of the domains
of primary interest to both clinicians and consumers or carers.
There is also a need to determine which of the currently adopted measures appears to work most
effectively at tracking outcomes, given the longer-term advantage of having equivalent measurement
strategies across the States. While some independent studies of the measures in each jurisdiction
provide some information that would help in making such a decision in the future, a head-to-head
comparison of the three measures will be required make a considered judgment on a future common
measurement policy.
The elements of a coherent research strategy to advance this field are: collection of data on adopted
measures; direct comparison of adopted measures; development of one or more brief measures
tapping additional domains; and further development of multi-ethnic versions of instruments and
instruments applicable to Aboriginal and Torres Strait Islanders
Conclusions and recommendations
Part 5 concludes that there is clear support for including consumer self-rated outcome measures in the
suite of measures that are necessary in an accountable mental health system focused on consumers
and their families and carers. There is substantial consensus among consumers, families and carers,
and clinicians about the range of purposes for which consumer self rated outcome measurement can
and should be used. This consensus has implications not only for outcome measurement, but also for
the culture, workforce, and goals of the mental health system.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page xii
In order of priority for consumers and carers, the purposes for which consumer self-rated outcome
measures can or should be used encompass:
•
•
•
•
•
increasing the person’s capacity to self-monitor progress over time,
increasing the quality and appropriateness of the relationship between consumer and practitioner,
and the focus of treatment and support interventions,
informing planning and direction of the amount and nature of service to meet client and family /
carer needs better,
informing the evaluation and review of the individual’s management, and
informing the evaluation and review of the management of the service system over all.
There is a high level of consensus among consumers, families and carers about the domains of life the
system should be interested in knowing about in relation to consumer outcomes. There is substantial
overlap between what consumers think are the relevant and important domains, and what clinicians
think. In particular, consumers and their families or carers are keen that consumer self rated outcome
measures focus the attention of both the individual and the system on tracking resilience and coping
skills in the face of what are predominantly chronic relapsing disorders.
The adoption of more holistic measures is a necessary but not sufficient step in the culture change
necessary in the mental health system to bring it in line with changes in general health and aged care
systems. In particular, families and carers, and consumers themselves, experience a stark contrast in
how they are treated as a valuable unit in the management of other acute or chronic conditions in the
mainstream, and how families and carers are excluded or pathologised in the mental health system.
This disparity argues strongly for developing a collateral family/carer outcome measure.
All the self-rating measures that have been adopted by Australian states in recent years have
significant strengths. However, none of them covers all the criteria for measurement that were
important to consumers and carers, or fulfilled all the other criteria in the evaluation framework. In
the longer term, it is clearly preferable that Australia adopt a single consumer self-rating measure, so
that direct comparisons can be made across jurisdictions.
We propose that Victoria recommend to the Information Strategy Committee established under the
auspices of the National Mental Health Working Group that it consider commissioning research to
address the research priorities listed under the recommended research strategy to address the
following objectives:
1. Compare the performance of the measures currently in use by Australian states
2. Develop one or more additional measures that meet some of the key criteria of importance to
consumers and carers that are revealed by this project
3. Ensure that measures are applicable across ethnic groups and in Aboriginal and Torres Strait
Islander populations.
We recommend that these projects have substantial consumer and carer involvement and
collaboration to ensure the relevance of the work to the priorities expressed in the consultations during
this study. The ultimate aim should be to develop a consumer-rated measure (or an integrated set of
measures) that has sound psychometric characteristics, is sufficiently practical and financially viable
for widespread use, has aspects that link closely with clinician measures, and is relevant to the
Australian context.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page xiii
Siggins Miller Consultants
Consumer self-rated outcome measures
Page xiv
Consumer self-rated outcome measures
Introduction
In August 2002, the Mental Health Branch of the Victorian Department of Human Services appointed
Siggins Miller Consultants to undertake a comparative analysis of the consumer self-rating outcome
measures now in use in mental health services throughout Australia, identify international best
practice, and make recommendations towards a uniform national approach.
The investigators were:
Dr Mary-Ellen Miller, Director of Siggins Miller Consultants
Professor Ian Siggins, Director of Siggins Miller Consultants, and Adjunct Professor in the
School of Medicine at the University of Queensland
Associate Professor David Kavanagh, Director of Research and Associate Professor in Clinical
Psychology in the Department of Psychiatry at the University of Queensland
Dr Maria Donald, School of Population Health, University of Queensland
The brief required a project in three phases:
Phase 1: Literature review: A review of the literature to determine international best practice in the
field of consumer-self rating.
Phase 2: Development of evaluation framework and comparative analysis: Formal consultation across
all States and Territories with consumers, clinicians and service providers in both public and
private sector mental health services to develop a detailed evaluation framework. Apply this
framework in a comparative analysis of consumer self-rating outcome measures used or proposed
for use in each State and Territory in Australia, including but not limited to BASIS-32, MHI, K10,
SF36, HoNOS, LSP, RFS and FoC.
Phase 3: Reporting and recommendations: A report and recommendations arising from Phases 1 and
2, including recommendations on the most suitable measures now in use or proposed for use,
development of new or modified measures, key features of any new or improved measures, and
implications for existing infrastructure and resources. If development of a new measure or
modification is recommended, prepare a detailed proposal and methodology for this work to form
part of a Victorian submission to the Commonwealth.
Preparatory steps
Meeting with client
We met with representatives of DHS to finalise the contracting process, discuss the draft work plan,
and receive a briefing about the project and its national context.
The DHS project officer supplied us with key national and Victorian contact lists covering consumers,
carers, academics, service providers, and policy staff as a starting point for developing the list of
people who would be consulted during State and Territory visits and/or by telephone interview.
A completed list was prepared for discussion at the first meeting of the reference group.
Team meeting
A full team meeting finalised subcontracts with associates and established a protocol for allocating
tasks. The meeting discussed the search strategy for the literature review, and the structure of a
protocol for the site visits and informant interviews. These were presented to the reference group at its
first meeting.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 1
Contact with reference group members
We contacted reference group members by phone and or email to introduce the project and the
members of the team. We sought their help in developing a list of key people (consumers, carers,
clinicians, policy staff, and academics) they would like us to consult in their jurisdictions; advice
about any documents that would inform us where the member’s jurisdiction or organisation had
reached in thinking or practice about consumer self rating measures; and advice about any key articles
or unpublished papers they would recommend.
We met with the reference group in Melbourne on 23 September, and received feedback about the
draft protocols for officials, clinicians, academics and consumers and carers for use in focus groups
and telephone interviews. (The redrafted protocols appear in Appendix 1).
Literature review
The consultants undertook a comprehensive review of national and international literature to identify
best practice in the development and use of consumer self-reporting measures, and to analyse the
advantages and disadvantages of each measure from a clinical and a consumer perspective, including:
•
•
•
•
•
•
motives for measuring outcomes
the purposes of outcome measures
health outcomes and consumer-identified outcomes
consumers’ capacity to assess outcome
issues in implementing and administering consumer self-rated outcome measures
the impact of measurement on outcome.
The findings of the literature review are set out in Part 1 of the report, together with a comprehensive
bibliography.
Consultations
We consulted stakeholders in every State and Territory, including consumers, carers, clinicians and
service providers, mental health service management, representatives of Government authorities in each
State, Territory and the Commonwealth, and groups and individuals nominated by the Mental Health
Branch.
Face-to-face consultations included 96 consumers of mental health services who participated in group
discussions in every State and Territory; 33 members of state-based Consumer Advisory Groups and
consumer bodies, 35 carers, mainly in group discussions; and 47 State and Territory mental health
officials. Phone or email consultations included 28 mental health clinicians and academic researchers.
On the basis of these consultations and the literature review, we developed a comprehensive
evaluation framework for comparative analysis of measures. It was submitted to the national reference
group for approval before comparative analysis of individual measures began.
The results of the consultations are set out in Part 2 of this report.
A framework for evaluation
The implications of the literature review and the consultation for an evaluation framework are
discussed in Part 3 of this report.
Analysis of measures
The agreed framework was then applied to a comparative analysis of existing and proposed measures
to meet the client’s objectives and to formulate recommendations as required by the brief. The
findings of the analysis are set out in Part 4 of the report.
Findings and recommendations
Part 5 of the report contains findings and recommendations on the use or adoption of existing measures,
and the need for development of a new set of parallel measures.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 2
Part 1: A review of the literature on consumer outcome measurement
Australian mental health services are introducing a national outcome measurement protocol to
measure the effects of their activities, in order to inform stakeholders (consumers and their families
and friends, consumer groups, clinicians, managers, policy makers, funding bodies) about the quality
and cost-effectiveness of services, and possible future directions in service and policy development.
The national protocol includes a suite of measures for each age group (child and adolescent, adult, and
aged persons). Collectively, a suite gives a picture of the severity of illness and the degree of
functional status. Each suite of measures includes a self-rated instrument to elicit feedback from
consumers about the outcomes of mental health treatment.
From its adoption in 1992, the Australian National Mental Health Strategy placed a clear emphasis on
the measurement of outcomes. In 1994, Andrews, Peters & Teeson identified a range of existing
instruments that could possibly serve as useful outcome measures, including the MOS Short Form 36
(SF-36), the Mental Health Inventory (MHI), the Behaviour and Symptom Identification Scale
(BASIS), the Role Functioning Scale (RFS), the Health of the Nation Outcome Scale (HoNOS), and
the Life Skills Profile (LSP). The first three of these measures are consumer self-reports.
Building on the work of Andrews et al (1994), Stedman et al (1997) field tested the selected measures
and reported that there was “low convergence between consumers’ and service providers’
assessments”. Stedman et al attributed this finding to “the different contents and orientations of the
measures (the consumer measures generally assessed how the consumer was feeling, whereas the
service provider measures generally assessed functional ability). They may also reflect treatment
approaches to alternative diagnostic groups. There is also the possibility that consumers and service
providers undertake different processes in completing such measures”.
One aspect of the Stedman study was the use of focus groups to gather feedback from consumers. The
results included these observations:
-
“A consensus regarding the questionnaires was the inability of a general questionnaire to address
not just different diagnoses, but that they do not provide open-ended questions, as well as not
taking into account wellness and people’s individuality …
-
“It is taken for granted that the information gathered will benefit service providers, but how will it
benefit consumers unless a tailor-made questionnaire is developed? …
-
“The questions themselves do not address day-to-day influences that we are all prone to. The
scoring of each question is difficult on a number scale, which could be altered by having a sliding
scale instead …”.
The feedback consumers gave Stedman’s study was that the standardised measures (SF-36, MHI, and
BASIS-32) did not capture the full range of issues consumers of mental health services regarded as
relevant.
A study of Consumer perspectives on future directions for outcome self-assessment for the Victorian
Mental Health Outcomes Measurement Strategy (Graham et al 2001) concluded: “consumers are
seeking a different approach to implementation of consumer self-ratings than has been the case to
date. In particular, they want to see strong consumer involvement at all stages, commencing with the
design of a new instrument that reflects their views.”
Accordingly, one purpose of this project is to explore the views of Australian consumers of mental
health services and other stakeholders on the purposes, potential uses, and domains of outcome they
believe are most important in self-rated and self-reported outcome measures.
In 1999, the AHMAC Mental Health Working Group committed all States and Territories to the
introduction of routine consumer outcomes assessment, using standard clinical assessment scales and
a consumer self-report instrument (DHAC 1999). All jurisdictions have agreed to comply with the
National Outcomes and Casemix Collection protocol that sets out service settings, age groups, and
rating points for collecting outcome measures in all mental health services. The protocol requires that
the mental health and functional status of all consumers of mental health services are assessed at key
points in the treatment cycle for inpatient, ambulatory, and community residential services. There are
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 3
three suites of measures: child and adolescent, adult, and aged persons. Some or all of the measures in
a suite are usually rated at entry, exit, and review in both inpatient and other settings.
The protocol includes a consumer self-report tool in both the adult and the aged persons suites, and
jurisdictions could choose initially among three self-rated measures – Kessler (K10), MHI, or BASIS.
A choice among these measures was made in various ways.
In Victoria, reference groups developed age-specific outcome measure suites in 1997-99, and the
BASIS-32 was adopted as an interim measure for adult mental health services. It was trialed in
selected adult services from mid-2000, and after a consumer consultation project (Graham et al 2001)
and analysis of data from the trial, BASIS-32 was adopted as useful for consumers to participate in
outcome measurement, and for treatment plans to be modified as a result of consumer input.
In NSW, the Centre for Mental Health participated in field-testing of the K10 during its development.
The choice of this measure was made after piloting for validation in the Richmond District of NSW
under the auspices of the Mental Health Assessment and Outcomes Tools and Training (MHOAT)
initiative. The NSW Consumer Advisory Group (CAG) has also overseen a project to develop a fuller
approach to consumer feedback called MH-CoPES (Consumer Perceptions and Experiences of
Services), still in progress.
Several other jurisdictions invited their Consumer Advisory Groups to compare the three instruments,
and give advice on the instrument they found most acceptable. This process was under way when this
project began, and three States had not yet chosen a measure when the data collection for this project
ended; but each jurisdiction had made a choice by early 2003 (BASIS-32 in Victoria, Tasmania and
the ACT; K10+ in New South Wales, South Australia, and the Northern Territory; and MHI in
Queensland and Western Australia).
In the private sector, the Strategic Planning Group for Private Psychiatric Services (SPGPPS) formed
a working group representing the AMA, RANZCP, private health insurers, private hospitals, the
mental health and private health industry branches of the Department of Health & Aged Care, and the
Network of Australian Community Advisory Groups to produce a national model for data collection
and analysis, including outcome measures. On the basis that a consumer self-rated instrument “should
address both the disability and distress associated with mental disorders or problems, it must not be
specific to any particular disorder, it must be very brief, and it must be supported by a broad base of
benchmark data”, the SPGPPS adopted SF-14-M as the routine consumer self-rated measure. (MorrisYates et al 2000)
The Strengths and Difficulties Questionnaire (Goodman 1997; Goodman et al 1998; Mathai et al
2002) has recently been endorsed for inclusion in the child and adolescent suite. It allows for selfrating by adolescents, and also includes parent and teacher versions.
The NOCC Overview of Clinical Measure and Data Items V1.02 (NOCC 2002) “recognises that
limited Australian research has been undertaken on consumer rated measures, and additional
exploratory work in this area is important.”
The scope of this review
This review of the literature covers the concept of a consumer self-rated outcome measure; motives
for measuring outcomes (costs, accountability, clinical quality, and consumer perspectives), and the
purposes to which they are applied; the variety of perspectives on ‘health outcomes’, the choice of
what is measured, causal attribution, and methods of outcome measurement.
We then address consumer-identified outcomes – the domains of outcome valued by consumers, and
the dimensions of consumer-focused outcomes. We review research on consumers’ capacity to assess
outcome, factors that may affect that capacity, and the value of complementary measures.
Finally, we review research and opinion on practical issues in implementing consumer self-rated
outcome measures: how the data is used and stored, name-linked data and confidentiality, options for
individualising measurement, the combination of measures, psychometric properties, sampling error,
response biases, instructions, protocols and formats, feasibility and practicality issues, and the impact
of measurement itself on results.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 4
Self-rated measures
For the purposes of this review, we make little distinction between the expressions ‘self-rated’ and
‘self-reported’. However, the use of these two terms alerts us to the reality that, in reporting the
outcome of treatment, the way consumers rate their health has two dimensions. First, they use
behavioural criteria, often called ‘functional status measures’ – that is, how far they are or are not
limited in performing the activities of their daily routine as a result of illness. Secondly, they use
subjective ratings - the extent to which they feel their mental health is optimal or compromised
(Kaplan & Ware 1989).
Their reports about their health, for example, may give a Yes or No answer to the question ‘Can you
participate freely in everyday activity?’ or about their care, ‘Were the side-effects of your medications
discussed with you?’ Their ratings may range from Excellent to Poor about their mental shape or
condition, or about the quality of the provider’s explanations and advice during their care.
As we shall see, consumers of mental health services wish very strongly that both their reports and
ratings of outcome are taken into full account, and a suitable measure should elicit both aspects.
Sometimes a less important distinction is drawn between a self-report and a self-administered
measure. Some commentators (eg Eisen et al 1991) think that the only “true self reports” are
instruments administered by the consumer that collect information provided by the consumer. Kessler
et al (2000) define “consumer measures” as “methods that do not require clinician administration or
clinical judgement”.
While many self-report instruments are simply adaptations of instruments already administered by
clinicians, or seek information in domains of interest to clinicians, many clinician-administered
instruments simply record what the consumer reports, and can be considered self-report measures.
Other methods of administration (such as by non-clinical personnel) should not be taken to mean that
an instrument does not collect self-report data.
The administration of self-report measures is relevant to the practicality of applying them, but if a
useful consumer self-report instrument is to be developed, self-reports administered by means other
than self-completion should be considered, and aspects of administration dealt with in their own right.
Motives for measuring outcomes
In her address to the 1996 Australian Health Outcomes Clearing House Conference on Mental Health
Outcomes, Beverley Raphael concluded: “Key requirements must … include a conceptual framework
recognising the theoretical and real issues affecting those with mental health problems and disorders”
(Raphael, Hugh & Stewart 1997).
Commenting on the choice of outcomes for the UK National Health Service, Brown, McCartney and
Bell (1995) say: “Theoretically any service has measurable outcomes. However, what actually gets
measured will inevitably be affected by the varying degrees of influence of the different interest
groups concerned. The critical question is: who sets the measurement criteria? It is not obvious which
health service interest group (clinicians, managers, consumers or researchers) is best placed to define
appropriate outcome measures or to provide appropriate measurement tools. Attempting to reach
consensus may not provide a clear measurement tool satisfactory to all constituencies. And if one
group monopolises decision-making the result will probably be an unbalanced instrument that does
not measure comprehensively”.
In a report on the activities of the US National Outcomes Roundtable, Boothroyd et al (1998) say the
catalysts for outcomes assessment have included the changes in accountability that have arisen as
responsibility for the care of people with a mental illness has shifted at least partly from the state; the
need to monitor populations, treated and untreated, for access to care; and the need systematically to
assess treated populations.
Walter et al (1998) list the characteristics of what they regard as suitable outcomes databases: “First,
reliable and valid outcome measures; second, clinical information systems that give feedback to
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 5
clinicians in real time; third, benchmark data against which improvement can be assessed; and finally,
costs so that we know what we are paying for a given outcome”.
The Ohio Department of Mental Health (2003) says: “Measuring success in a large, complex mental
health system requires balanced attention to data in three critical areas: quality, access, and cost. …
the most obvious systematic gap has been in the lack of statewide data for consumer outcomes as an
indicator of quality.”
A continuing education course at the Harvard School of Public Health (2003) lists the important
purposes of outcomes evaluation and research as “improving patient outcomes, controlling costs and
allocating resources, implementing disease management programs, making effective clinical and
business decisions, and developing and marketing health care products and services”.
Published research and commentary on the growth of the outcomes movement reflect these diverse
motives for measuring outcomes. Possible starting points include the pressure on funders to contain
costs, accountability for the cost-effective use of resources, the desire of clinicians to improve quality
and of managers to minimise risks, the rights and satisfaction of consumers and their capacity to make
valid judgments about the quality of the healthcare services they receive, and the need to find
practicable and reliable ways of collecting and using useful information.
Costs
In the US, according to Smith et al (1996), the advent of managed care has forced the administrators
of mental health care to implement outcomes measurement systems that will provide information for
cost control and quality improvement. Similarly, Eisen et al (1999) say increasing pressure from third
party payers to assess client outcomes has made clinical programs want to know how to implement
outcomes systems. Dickey et al (1998) say the drive to contain the costs of health care in the United
States focuses attention on how quality of care is affected. Holloway (2002) attributes the emergence
of the outcomes movement to the “rapidly escalating costs [of health care], acceleration of the
introduction of new health technologies and evidence of massive regional variations in the delivery of
health care in the USA”.
There is a resulting danger that many of the outcome variables valued by third-party payers may seem
reductionist to clinicians (Dornelas et al 1996). With this in mind, Tobin and Hickie (1998) propose
an Australian model for a successful partnership between the academic and management arms of a
district health service in an integrated outcomes-based evaluation system.
Accountability
As a result of the diverse pressures to move toward greater accountability in delivering health care,
health departments in countries such as the US, UK, and Australia have adapted commercial business
practices in running their organisations, and constructed frameworks more amenable to evaluation.
In the US, the current approach to managing provision of health care is “managed behavioural health
care”. In theory, managed funds contract with service providers on the basis of performance:
“capitated prospective payment to preferred providers based on a performance contracting system;
whereby the provider assumes financial risk for the treatment of illness, … providers must
demonstrate quality and accessibility, and … the provider must earn the reimbursement (Dyer, 1992,
relying on Boland, 1991)” (Ross 2000). The primary driving force behind this approach is cost
containment and reduction, but accountability for the quality of service has grown in prominence as
consumers and providers deal with the negative outcomes that have emerged.
In the US, negative outcomes have included tension among competing service providers, hostility
towards managed care organisations, inappropriate discharge of individuals with serious mental health
problems from inpatient treatment, under-funding of service providers, closure of facilities that
provided high quality service but at too high a cost to compete, the diversion of consumers to
interventions not adequate to meet their needs, withholding of appropriate but costly medications, and
in general under-servicing of consumers with serious mental illnesses.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 6
The response to these negative impacts has been to strengthen demand for accountability for quality in
services. This is what prompted committee members of the USA Institute of Medicine (IOM) in 1996
to adopt an evidence-based approach. After examining available information, the IOM concluded:
“The research basis and the development of quality assurance and accreditation standards are far less
advanced in behavioural health care than in other areas of health care. Further, development of
analytical tasks is necessary and this evidence base needs to be expanded before detailed
recommendations can be made. … In their current forms, performance indicators are not specific for
particular treatment characteristics (organizational and clinical), and there is a lack of consensus of
clinical judgment with regard to the relationship to outcome”. (Ross 2000)
Graham et al (2001) report the results of consumer focus groups that identified service accountability
as a purpose to which outcome data (as distinct from satisfaction data) can be put, and outlines the
implications of this purpose: “The introduction of regular consumer self-reporting measures creates a
new form of service accountability when consumers’ feedback is aggregated and reviewed at the
agency level. Participants argued that the combined information provided by an agency’s consumers
should be reviewed regularly to identify and address service deficiencies”.
Clinical quality
Walter et al (1996) say health care has a tendency to resist change, but “this conservatism cannot be
decried until clear evidence becomes available that the rigmarole of outcome assessment actually
results in improved clinical outcomes.” There is a small but growing literature on the clinical utility of
the use of outcome measurement instruments in clinical practice (eg Callaly, Trauer and Hantz 1998,
Callaly and Hallebone 2001, Dickey 2002).
Holloway (2002) notes that attempts are being made in the UK to use outcome measures in research
to evaluate the merit of specific interventions, new technologies, and policies in mental health.
Similar calls for change have been made in Australia. Walter et al (1998) conclude their summary of
proceedings of a conference on Getting better: outcome measurement and resource allocation in
psychiatry with the statement, “The acid test will be to demonstrate that evaluating outcome can be
harnessed to improve treatment methods and, in turn, to produce better results”.
Consumer perspectives on quality
The consumer movement has significantly affected developments in health, including an increasing
demand for systems to embrace human rights issues, and a growing emphasis on financial
accountability and accountability for quality. It has stressed “the ethical principle that people have the
right to know what is happening to them and to make a choice about what will happen to them. In
addition there have been associated challenges to professional authority and control of knowledge”
(Hill 1998).
In 1946, the World Health Organization produced a definition of health that included a reference to
‘the patient’s quality of life’. At first the responsibility for ascertaining a consumer’s quality of life
was ascribed to medical practitioners. But when discrepancies between doctor and consumer reports
became apparent, it seemed consumers were better able to report on their own quality of life than their
doctors, necessitating the inclusion of consumer self-ratings in assessing health outcomes (Bowling
1991).
One response to the consumer movement has been the effort to evaluate performance by consumer
satisfaction and success in meeting consumer-identified needs. In the US, “managed behavioural
health care has been introduced to the concept of personal outcomes, in which the consumer and
family points of view are ascendant”. A recent example of this shift in emphasis is the support of the
USA Federal Centre for Mental Health Services (CMHS) for the “Outcome Roundtable for Children
and Families, which includes representatives from mental health and child welfare, as well as
consumer, family, and academic participants”. (Manderscheid, Henderson & Brown 2000)
Eisen et al (1991) argued that an emphasis on the consumer’s perspective on measuring treatment
outcomes had unique value for evaluators, and could be realised by (1) designing outcome measures
around treatment goals defined by consumers, (2) recruiting consumers as self-reporting evaluators of
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 7
their progress, and (3) assessing consumer satisfaction with various aspects of treatment. It would be
limiting, however, to attend exclusively to the consumer’s perspective.
California enacted legislation in 1991 (the Bronzan-McCorquodale Act) to enforce the rights of
consumers to be actively involved in the realignment of the mental health system, including being
involved “as an integral part of assessment, intervention, and evaluation”. As a result, the California
Department of Mental Health “adopted a performance outcome measurement system that brings
together the views of the clinical team, consumer, and where appropriate, family members of the
consumers. Not only is the consumer involved in providing factual information … but also in
providing his or her perception of the system and services with which he or she is involved. In this
model, the clinical service team, as well as the consumer, provides information about the consumer’s
medical and social functioning” (California DMH n.d.).
Pulice et al (1995) investigated the effect of a specific policy change on consumers, families, and
providers, and chose a qualitative approach for the following reasons: “First, quantitative studies
which have traditionally guided mental health policy, often gloss over factors that affect individuals
on a personal level. Second, qualitative results were sought to guide the development of instruments
and methods for a quantitative study of the policy, a strategy that had been shown to result in
improved quantitative research. Third, the authors were interested in using the empowerment model,
which encourages the active involvement of study participants in the design of the research project …
focus groups provided guidelines on the formulation of appropriate and meaningful research questions
for the larger, quantitative study of the policy”.
Campbell (1996) said that growing recognition that customers were important health partners had
created the need for collaborative mental health outcomes systems. However, Lelliott et al (2001) said
that no existing instrument measured all, or even most, of the issues considered important by users of
mental health services.
Booth et al (1997) emphasise the importance for outcome measures to be multidimensional and
include the economic costs of depressive illness, both the direct costs of treatment and the indirect
costs of the individual’s lost earnings and caregiver burden. Hill (1998) says that industrialised
nations have experienced rising demand for treatment of chronic illnesses. The need for ongoing
intervention for individuals with such illnesses, and the costs involved, have contributed to a
movement toward greater self-care and peer support, necessitating increased consumer education and
involvement in decision-making.
The US Council on Quality and Leadership in Support of People with Disabilities (CQLSPD) has
adopted consumer-identified outcomes to evaluate the performance of service providers, with
apparent success. The Council oversees the delivery of services to individuals with developmental
disabilities, chronic mental illness, brain injury, and physical disabilities (Burwell & Jackson, 1999).
Over the course of six years, “via interviews with people with disabilities, their families and friends,
and agencies”, the Council developed a Personal Outcomes Measure against which to evaluate the
performance of service providers. Paralleling this shift to measuring outcomes of personal relevance
to consumers, service delivery models have developed greater flexibility in structures and processes to
achieve more positive consumer outcomes.
Technological advances (the internet, computers, and telecommunications) have brought about an
exponential rise in the rate of information exchange. Consumers, providers, managers, and policy
makers now have more points of comparison, greater awareness of achievable standards, and greater
knowledge on which to judge performance and identify options. Advances in this area have also made
health care from a distance possible. Manderscheid & Henderson (2000) have argued that with further
advances in technology, consumers will begin to interact with artificial intelligence systems.
Accompanying these developments is a growing awareness of the fact that, if the benefits of these
new approaches to health care are to be maximized, it is necessary for consumers to develop skills in
making accurate self reports and ratings, and to become more actively involved in decision making.
Holloway (2002) says: “Paul Ellwood, in his 1988 Shattuck lecture … ushered in the modern
outcomes movement and called for the routine collection of outcome measures by clinicians”.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 8
Ellwood envisaged such collections as involving “a rich and clinically meaningful set of outcomes
[that] would be collected for all patients during their routine care”. They would “form a resource for
clinical and health service research … [and] eventually be used inter alia to compare existing
treatments and to evaluate new technologies”.
The purposes of outcome measures
One of the first questions consumers asked in the course of this project was, “Why are we supposed to
do this? What will the information be used for?” The purpose of outcome measurement, in the sense
of the ends to which it may be applied, adds another dimension to the motives outlined above.
Information from and about consumers may feed into decision-making at different levels. For
example, it may be used by the consumer to gain insight into how situational factors are affecting his
or her mood; it may be used jointly by the consumer and case manager to make changes to the
treatment plan; it may be used by clinicians to evaluate the effectiveness of a particular intervention
and generate options for further intervention; it may be used by managers to determine whether the
level of clinical supervision and training is sufficient to enhance and maintain consistent standards
across the service; and it may be used by policy makers to determine whether or not certain practices
are cost-effective. For example,
•
•
•
•
•
Individual consumers may decide whether to access treatment or not, what treatment to seek, or
whether to stay in treatment; the clinician may consider how to improve consumer care, offer a
particular treatment, or seek training in a particular treatment. The importance of using outcomes
data in individual case planning and management is stressed by Stedman et al (1997), Morris
Yates & Andrews (1997), and Callaly & Hallebone (2001)
Health service management and providers may consider how service outcomes can be improved;
health research funding bodies may decide whether to fund the development of new interventions
or treatments.
Information may be used in the decisions of public policy makers or consumer advocacy groups;
researchers may design, test, or refine standardised instruments appropriate for particular clinical
conditions being treated, measure outcomes in multiple relevant domains, add other data to
outcomes measures to support matching, and collect data at standardised intervals after a sentinel
event (such as entry to or discharge from hospital); payers, other funders of treatment (eg health
funds), and government departments will decide resource allocation, and use data comparable to
outcomes measures for interventions targeting somatic conditions.
Governments will make resource allocation decisions using data comparable to outcomes
measures for non-health programs (eg ‘years of productive life lost’).
Decisions and implications that cross the stakeholder groups include assessing the effectiveness of
various treatments, decisions about resource utilisation, and decisions about developing standards
(Dorwart et al 1994; Walter et al 1996; Holloway 2002).
Several authors suggest terminology to categorise the purposes of measurement according to the
interest groups involved and the decisions they make, for example borrowing the ‘micro/meso/macro’
terminology from economics (‘micro’ refers to the decision-making system of the clinician and
consumer, ‘macro’ to decision-making at the governmental or public policy level, and ‘meso’ to
levels of organisation in between (for example, at the level of the institution, service, or district).
Stanley (1992) uses the terms ‘individual, intermediate and aggregate levels’. Englehardt (1996) and
Kluge (1992) adopt similar systems. Sutherland & Till (1993) include an additional ‘meta’ level for
decisions that entail comparisons between the health field and other fields, or “where priorities for
health care are traded off against other priorities at the governmental level”. Green & Newman (1996)
describe and suggest the use of Ackoff’s ‘stakeholder analysis’ method (Ackoff 1975) and Lawrence
& Cook’s survey of stakeholders (1982) as ways to identify interest groups and their particular
purposes for information about the outcomes of mental health services.
The literature does not appear to deal specifically with the issue of assigning relative importance or
priority to the purposes to which a measure can be put. Unanswered questions in the literature suggest
that further work is needed to determine the various actions that need to be taken to improve quality in
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 9
mental health services, which are the most important, and what information these actions must be
based on. Determining the relevant interest groups is a necessary step in this process.
Health outcomes
The usefulness of consumer self-reports as outcome measures in mental health depends on knowing
what outcomes are to be measured. Brown et al (1995) comment: “In order to take an outcome
measure one needs to know what an outcome is, but there is no agreement about this”.
A health outcome, as defined by the Australian Health Ministers’ Advisory Council (the AHMAC
‘Sunshine Statement’) is “a change in the health of an individual or group of individuals which is
attributable to an intervention or a series of interventions” (Andrews et al 1994, Graham et al 2001).
In the case of mental health services, however, the literature does not appear to distinguish explicitly
between the ‘outcomes of mental health services’ and ‘mental health outcomes’. The literature often
appears to assume that when ‘mental health outcomes’ are discussed, it is ‘mental health outcomes of
mental health services’ that are of interest.
The variety of perspectives on outcome
The implications of poorly defining mental health outcomes may mean that various interest groups
(consumers, clinicians and policy makers) value the usefulness of self-rated measures against different
criteria. Different interest groups are likely to identify different changes in the consumer’s
circumstances, and will assign different weight or priority to these changes.
Brown et al (1995) say that the ‘outcomes’ of treatment, whether physical or mental, are most
commonly considered to be those phenomena that occur as a result of events that are in some way
under the direct control of the treatment provider. When clinicians make changes to clinical practice,
the clinician also determines the outcomes studied and the information collected about those
outcomes. When the scientific and practitioner communities promote an evidence-based medicine
philosophy, those groups will influence decisions about clinical practice. The implication is that the
extent to which a measure meets the demands of the evidence-based medicine system will be an
important criterion for evaluating its usefulness.
The tensions that may arise among the disparate perspectives of stakeholders may be illustrated as
follows. A treatment service may be designed and staffed to deal with limited aspects of a person’s
illness. A psychiatrist or a program may accept someone into treatment for symptom relief, to ensure
compliance with medication, or to deal with an episode of anxiety or psychosis. They may not wish to
be held accountable by consumers’ self-reports about whether they feel good about themselves.
Current treatment interventions do not affect a range of outcomes that consumers hold as important
outcome measures – that is, the symptoms may be relieved, the regime adhered to, the episode
survived, but the person may still be socially isolated, or unable to obtain work, or otherwise feel as
badly about their well being as before.
This mismatch may produce unintended effects. Brown et al (1995) discuss the relationship between
the choice of outcome measure and those affected by the outcomes of measurement. They note that
measures sometimes do not directly address the outcome in question, but measure a phenomenon
related to that outcome, relying on an established constant relationship between the measured
outcome and the outcome of interest. In these circumstances they suggest that groups whose interests
are affected by the outcome of measurement will act to achieve better results on the measure by
manipulating the intermediate phenomenon without achieving the desired outcomes. If the admission
of patients to an inpatient treatment program is taken as an indicator of the provision of treatment, and
consequently only the admission of patients is measured, then service managers will tend to increase
the admission of patients without increasing provision of treatment, thus changing the assumed
relationship between the number of admissions and the amount of treatment provided.
For those who manage mental health services, if the results demonstrate distance between what the
existing system offers and what its consumers judge most important to their lives, the implication may
be that the system needs to be redesigned, and in particular that the structure of the workforce may
need to match the full range of interventions consumers are seeking.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 10
A parallel may be found in recent policy developments in Australian aged care. The feasibility of
providing extended care in their own homes to older people classified as needing high care, rather
than admitting them to residential facilities, has been successfully tested in a pilot program, and is
now becoming available as an aged care option. While the usual range of health professionals remain
part of the Extended Age Care in the Home (EACH) package, the bulk of care is provided by
certificated personal care workers, who deal with the full range of the clients’ needs for social contact
and the normal activities of daily living. The assessed needs of the recipients directly shape the
composition of the workforce, the mode of employment, and the individualised credentialing of
workers (Siggins Miller 2001).
Limits on the effectiveness of outcome measurement
The mechanism by which data yielded by an outcome measure leads to action is complex, entailing a
number of influences over the decision-making and action-taking steps. For example, Walter et al
(1996) list professional autonomy as a factor limiting the potential for outcome data to lead to change.
Commenting on the UK experience with the “new culture” of the National Health Service (NHS),
Brown et al (1995) argue that where an outcome measure is adopted because in theory it measures the
predicted impact of a particular intervention, the relationship between the outcome measure and the
intervention disappears as service providers adjust practice and redefine constructs in order to meet
standards. In fact, the problem has not been addressed, but merely relocated. They conclude that
although “the search for ‘better’ outcome measures should not be abandoned … it should be
recognised that universal measures will never be achieved. Instead, it should be acknowledged that
outcome measures are imperfect instruments, and that the attention of management cannot be
concentrated on a few simple numbers, but must necessarily be diffused not only over a wide variety
of outcome measures, but over outcome unmeasurables as well”.
Holloway (2002) recently reviewed the worth of outcome measurement in the UK mental health
system. He comments on the parallel US situation, and says the “results to date…are disappointing”.
One of the reasons, Holloway argues, is that there are “problems with adequately taking account of
confounding variables (such as illness severity) [that] can lead to apparently bizarre findings, for
example, that depressed people who receive treatment for depression have worse outcomes than those
who have no treatment”.
Similarly, Gilbody et al (2002) express concern about the use of outcomes research to guide future
directions in mental health, noting these characteristics of such evaluations: “their observational
(rather than experimental) design; the poor quality of the data that are used; the inability to adjust
sufficiently for case mix and confounding; and the absence of clinically meaningful outcomes in
routinely collected data (Iezzoni, 1997)”.
If all effects of mental health services are to be measured, one must include both unintended positive
effects and unintended negative effects as objects of measurement. Some writers deal with unintended
negative effects (such as loss of freedom and dignity through involuntary commitment, drug side
effects, and “resultant feelings such as anger, fear, alienation and confusion” by including their
minimisation as a goal of treatment (eg Campbell 1996, cited in Graham et al 2001).
The Ohio Mental Health Consumer Outcomes System (2003) voices this caution:
“There are very few empirically established causal links between specific mental health system
services, specific agencies, or specific workers/clinicians and consumer outcomes. … While there
is likely to be some significant association between actions of a provider or clinician and a
consumer’s outcomes, it is not usually possible to prove a cause and effect relationship between
these factors. Data use should emphasize quality improvement at the clinical level at this time.”
Understanding the determinants of outcome
Increasingly in the literature on health outcomes the focus on treatment alone as the major
determinant of outcome is being questioned. Factors such as consumers’ belief in their capacity to
manage an illness or self-efficacy, and consumers’ participation in decision-making about their choice
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 11
of treatments were once written off as placebo effects, but are now the subject of study in their own
right (Miller 2000).
Similarly, the phenomenon of ‘spontaneous remission’ in depression and in alcohol dependence has
been shown to be the result of factors that a treatment system may influence if it adopts broader
interventions than medications and cognitive therapies to relieve distress and symptoms. Positive
changes in life circumstances, better housing, a paid or unpaid job, membership in a support group,
better relationships with family and friends may have more therapeutic importance than previously
recognised. This literature suggests that clinicians and system designers should consider the external
environment and the interpersonal milieu of apparent chronic treatment failures, and try to seek
amelioration of these factors (Miller 1984).
In their evaluation of six available outcome measures, Stedman et al (1997) commented: “A concern
for consumers and service providers was the extrapolation of data collected on these measures
[BASIS-32, MHI, SF36, HoNOS, LSP, and RFS] to judgements concerning the effectiveness of
service provision. That is, attributing change in a person’s mental health condition to the efficacy of
treatment … We simply do not know enough about the measures to determine their usefulness in
facilitating decisions concerning the attribution of change to specific factors, especially given the
potential impact that such information can have. … In order to make appropriate judgements
concerning clinically meaningful change for an individual, and to attribute such change to the
influence of various factors such as treatment efficacy, required more information than the six
selected measures appear able to provide”.
They go on to argue that due to concerns in the system that it should not be held to account for things
outside its focus in order to ensure good cooperation by clinicians and system managers with outcome
measurement that the ‘definition of outcomes” should be altered “to reduce the emphasis on
attribution” to “assist the promotion and use of routine consumer outcomes measures” (p93).
In short consumer outcomes are related to a multiplicity of factors. That many of these factors are
outside the current focus of the armamentarium of psychiatry and psychology could be used as the
basis for limiting the scope of the things that should be included in a measure used to assess consumer
outcomes. However, others argue that it is also essential to recognise that treatment itself is neither a
fixed nor simple independent variable, and that the effects of a treatment result from a combination of
the participants’ experiences both inside and outside the treatment system. When studies use the
“traditional patient input – ‘black box’ treatment outcome paradigm” they typically find that patient
characteristics and treatment factors together explain only a small proportion of the variance in patient
outcomes (Moos et al, 1982).
Too much of the discussion in the literature on the determinants of outcome and causal attribution
mixes up the difference between method and measurement (Andrews et al 1994, Stedman et al 1997).
The most valid and reliable measures used as indicators of outcome in a poorly designed study will
not improve the understanding of outcome. The most elegantly designed and executed randomised
controlled trial will not advance knowledge about how to improve mental health outcomes if the
outcome measures do not measure the right things.
It is argued therefore that if the purpose of measuring consumer outcomes is to improve the safety and
quality and effectiveness of the treatment system then measures need to be designed that reflect those
aspects of outcome most important to the consumers of such treatment. If we do not, we run the risk
of achieving outcomes such as ‘the operation was successful but the patient died’. To limit the scope
and content or indeed the purpose of an outcome measure to what is the current focus of treatments or
to limit the purpose to which the results of outcome measurement will be put for public relations
purposes is to limit the possibility that outcome measurement could contribute to improvements in the
quality and appropriateness of future treatments and the focus of treatment systems.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 12
Methods
A myriad of outcomes measures have been reported in the literature. Lambert, Oglis & Masters
(1992), for example, noted 1,430 different outcome measures reported in the literature from 1983 to
1988. Andrews et al (1995) considered 95 different scales. This proliferation probably results from
the ease of framing questions and constructing questionnaires with face validity, compared to the
more difficult task of systematically assessing the actual psychometric properties of scales and
evaluating their usefulness.
IsHak, Burt and Sederer (2002) have recently published Outcome measurement in psychiatry, a
comprehensive collection of papers on outcome assessment in psychiatry. The 48 contributors to the
book examine the underlying concepts and scientific bases of outcome measurement, and constructs
such as process and outcome, efficacy and effectiveness, basic statistics, and quality improvement.
They assess measurable domains (symptoms, functioning, quality of life, and perception of care), and
review the standards and instruments now used to judge the quality of care. They critically review the
wide range of instruments that produce data for assessing outcomes for individuals, populations, and
services, and discuss practical obstacles to assessment programs, such as regulatory and accreditation
demands, confidentiality and training, and the technology of measurement and proven solutions, and
the tools needed to incorporate quality measurement into clinical practice. Opler et al, Perlis et al,
Christopher & Sanlian, and Peselow contributed essays on the choice of measures for various mental
health categories (serious mental illness, geriatric psychiatry, personality disorders, and anxiety
disorders). A final chapter canvasses potential future developments.
Earlier, Seligman (1995) drew a distinction between efficacy studies and effectiveness studies,
identifying the characteristics of traditional efficacy studies as involving random assignment of
subjects, rigorous controls, standardised treatment, fixed duration of treatment, well operationalised
outcomes, “blind” raters and diagnosticians, patients with single diagnoses only, and use of a
thorough assessment battery. Effectiveness studies, on the other hand, may be used to investigate the
impact of interventions as they are applied in the field.
Seligman analysed data from the 1994 USA Consumer Reports Survey to investigate issues related to
the impact of psychotherapy. As well as responding to inquiries about appliances and services, the
180,000 readers were asked to answer questions regarding their mental health if they had “at any time
over the past three years…experience[d] stress or other emotional problems for which [they] had
sought help…” Seligman pointed to five characteristics of psychotherapy applied in the field that
were not addressed in efficacy studies: (a) it is not of a fixed duration; (b) it is self-correcting; (c)
consumers actively seek particular therapies and therapists; (d) consumers typically have multiple
problems; and (e) “psychotherapy in the field is almost always concerned with improvement in the
general functioning of patients” (1995).
The consumer survey covered (a) type of therapist; (b) type of intervention; (c) duration of therapy;
(d) frequency of contact; (e) improvement in symptoms; (f) improvement in work and social domains;
(g) satisfaction with intervention; and (h) global improvement. After analysing the data, Seligman
concluded that: “The CR study, then, is to be taken seriously – not only for its results and its credible
source, but for its method. It is large scale; it samples treatment as it is actually delivered in the field;
it samples without obvious bias those who seek out treatment; it measures multiple outcomes
including specific improvement and more global gains such as growth, insight, productivity, mood,
enjoyment of life, and interpersonal relations; it is statistically stringent and finds clinically
meaningful results. Furthermore, it is highly cost-effective”. Inclusion of a “well-normed
questionnaire” was one way Seligman felt such an investigation could be improved.
In 1998, the US Centre for Mental Health Services (CMHS) investigated the feasibility of using
consumer reports (via mail survey) to monitor access to and involvement in specific treatments, and
outcomes. Consumers provided information on such variables as demographic characteristics,
diagnosis, symptoms, involvement in treatment, quality of life, and satisfaction with services and
providers. The range of treatment covered in the survey included outpatient mental health programs,
day programs, residential treatment, emergency room or crisis centre, consumer-run programs,
vocational rehabilitation, individual psychotherapy, marriage or family therapy, and electroconvulsive
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 13
therapy. On the basis of information obtained solely from consumers, relationships between types of
interventions and outcomes were investigated (CMHS 1998).
The Center developed a consumer self-administered questionnaire “derived from several existing
instruments that were included in whole or in part: the SF-12, … Lehman’s Brief Quality of Life
Questionnaire, … the outcomes assessment modules for schizophrenia and major depression
developed by the Centers for Mental Healthcare Research at the University of Arkansas … the
consumer satisfaction instrument incorporated into the Consumer-Oriented Report Card of the Mental
Health Statistics Improvement Program, … and the patient questionnaire used in the Schizophrenia
PORT” (1998). After assessing the worth of this instrument through mail surveys, the Centre is now
“in the process of developing a standard set of reports and norms for the measures that were employed
in both the population survey and the disorder-specific study that can be used to characterize the
performance of a plan or mental health care system”.
The Center’s conclusion was that “given the importance of understanding the outcomes achieved in
mental health treatment systems, these monitoring techniques [consumer self reports via mail] seem
capable of producing unbiased population estimates at an affordable cost” (1998).
Individual and group approaches
A body of literature emphasises the importance and practicality of adopting an individual rather than
normative approach. Measurement models such as Goal Attainment Scaling (Kiresuk & Serman 1968,
Kiresuk et al 1994, Goodman et al 1993, Sutherland & Till 1993) and their implications for clinical
practice recommend desired outcomes or ‘goals’ are determined for individual consumers through a
process of collaboration between the clinician and consumer.
Sutherland & Till (1993) describe how generic quality of life measures can also be applied in an
individualised manner. They cite the Schedule for The Evaluation of Individual Quality Of Life
(SEIQOL) as one such example. In these approaches, respondents are asked to nominate those aspects
of their lives that are of interest to them (in the case of mental health service consumers, what
outcomes they would like to achieve), and then provide information about those life domains. Such
information can include descriptions of their external conditions, subjective responses to these
conditions (satisfaction, for example), and the relative importance the respondent places on each life
domain. The advantage of such approaches is that in these circumstances outcomes that are least
relevant for a particular consumer are given least attention and those that are not relevant are given no
attention at all.
However, aggregation of such data results in some limitations. For example, normative conclusions
are limited to statements such as “x percent of consumers were satisfied with what they saw as the
most important aspects of their lives”. The implication for measurement is that the ability of scales to
enable such individualised goal setting or outcome identification should also be considered when their
usefulness is being assessed.
Cummins & Baxter (1994) propose measuring the effects of service on “all people affected by the
service” and raise the possibility of including in the service model changes that occur for people other
than the person or social system that is the focus of treatment.
Mental health literature has long acknowledged that a system involving more than one individual can
be the object of a mental health intervention. The focus of family therapy on the family as a system is
perhaps the clearest example, but other mental health interventions discussed in the fields of
community psychology and social and preventive medicine also target complex systems rather than
individuals. The approach has been to focus on all the changes (internal and external) in the consumer
(be that consumer an individual or a system) that are attributable to a mental health intervention. The
outcomes measurement literature, however, largely overlooks this issue.
Consumer identified outcomes
Consumers are now also involved in determining the nature of the services they receive, both at an
institutional level and at an individual client-clinician level (Hill 1998, Manderscheid & Henderson
2000).
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 14
Manderscheid Henderson Brown (2000) say that little work was done in the field of consumerfocused outcomes before the 1990s. Earlier work tended to have a provider focus. More recently,
consumers and their families have been increasingly involved in identifying outcomes.
“In the past, work on outcomes usually has reflected only the provider’s point of view. More
recently, managed behavioural health care has been introduced to the concept of personal
outcomes, in which the consumer and family points of view are ascendant. Needless to say, the
elements of outcome emphasized by the provider and by the consumer are likely to be different.”
With respect to adult consumers, they refer to “person-centred outcomes” that “proceed from the point
of view of the consumer and what is desired from care”, and represent an approach that “has been
used in the developmental disability field with a high degree of success”.
In outlining the development of consumer involvement in health care, Hill (1998) identifies the
significant influences as evidence based medicine, the consumer movement, and the increase in
chronic conditions creating an interest in self-management.
In a review of the literature, Eisen et al (1991) reported that early studies (before 1970) were critical
of the validity of consumer self-reports, but confidence in these measures had risen more recently.
They listed a number of potential sources for consumer self report measures, including adaptations of
classification systems, adaptations of database items, adaptations of clinician administered scales,
adaptation of outcomes measures from other related areas such as quality of life, physical health
screening scales, assessment of need scales, and surveys of clinicians and consumers.
According to the Ohio Mental Health Outcomes Task Force, “Consumer outcomes are indicators of
health or well being for an individual or family, as measured by statements or characteristics of the
consumer/family, not the service system. Even though outcomes often are not attributable to one
service or program, it is our belief that these measures provide an overall ‘status report’ with which to
better understand people’s life situation” (Ohio Dept of Mental Health 2003).
Salzer et al (2000) point out that, with the “rising influence of consumers”, there has been an
increased drive to address quality-of-life issues for people with mental illnesses, including personal
relationships, living arrangements, education and employment. They comment:
“Traditional intervention approaches have focussed on reducing symptoms and increasing
functioning. Consumer advocacy in the mental health movement has led researchers and service
providers to focus on what consumers want to achieve as a result of mental health services. This
has resulted in a reorientation away from symptoms and functioning as the sine qua non of
treatment to the valuing of quality of life, recovery, hope, empowerment, and other related
outcomes. Focusing on the latter outcomes represents a shift in views of serious and persistent
mental illness from a focus on progressive deterioration, especially in schizophrenia, to recovery
and rehabilitation”.
Graham et al (2001) say: “For many [consumers], the quality of the services they receive is the
critical ingredient to whether they have a successful outcome. Thus, if they are going to be surveyed
regularly through a standard form, consumers have asserted that they want to see the inclusion of
questions concerned with service quality sitting alongside questions about the current status of their
illness”.
Lelliott et al (2001) comment: “People who use mental health services often emphasise and value
different aspects of their health and social function than do mental health care professionals. They
appear to place less emphasis on symptom reduction than they do on improvements in other areas of
their lives. These include: work or other meaningful daytime activity; financial security; suitable and
comfortable accommodation; choice and control over where they live; and the establishment and
maintenance of relationships.”
Domains of outcome valued by consumers
In the past few years, several attempts have been made to codify the domains of outcome that reflect
these emphases and values.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 15
Andrews et al (1994) conducted a survey showing that, while clinicians and consumers both felt the
measurement of outcome in terms of symptoms and satisfaction was very important, the measurement
of disability and quality of life was significantly more important.
More recently, the US Outcomes Roundtable for Children and Families, with consumer and provider
participation, prepared a framework for examining consumer outcomes, subdivided into five key
measurement domains: safety, health, functioning, life satisfaction and fulfilment, and satisfaction
with services (Center for Mental Health [US] 2000).
The US jurisdiction that has devoted the most sustained effort to consumer outcome measurement in
the past five years is Ohio. The vision statement of the Ohio Mental Health Outcomes Task Force
says: “All participants in Ohio’s publicly supported human care system are accountable to monitor
and continually improve outcomes for consumers. These outcomes, ...[based on] choice, respect,
dignity, and cultural and clinical competence, embrace the values of recovery for consumers and
families” (Ohio Dept. of Mental Health 2003).
The Ohio Task Force’s model includes four domains of consumer outcome measures: clinical status,
quality of life (including life satisfaction, fulfilment, and empowerment), functional status, and safety
and health. The specific outcomes that comprise each domain are as follows:
Clinical status: Level of symptom distress, number of psychiatric emergencies and
emotional/behavior crises, person/family ability to understand, recognize and manage or seek help
for symptoms, both physical and psychiatric.
Quality of life: Satisfaction with areas of life including family relationships, social involvement,
financial resources, physical health, control over life and choices, individual and family safety,
participation in community life, living situation, productive activity, and overall satisfaction with
life; a sense of overall fulfillment, purpose in life, hope for the future and personal or parental
empowerment; attainment of personal goals related to culture, spirituality, sexuality, individuality,
developmental stage and liberty; and the family's sense of balance between providing care and
participation in other life.
Functional status: Identifying, accessing, and using community resources to fulfil spiritual, social,
cultural, and recreational needs, by participation in activities that are not primarily mental health
organizations; developing and managing interpersonal relationships; managing money; managing
personal hygiene and appearance, skills such as use of public transportation, phone books, grocery
store, laundromats, etc to maintain oneself independently, and maintain a home environment in a
safe, healthy and manageable fashion; advocating successfully for oneself with mental health
professionals, or landlords; remaining in a home or family-like environment as measured by
stability and tenure; engaging in meaningful activity (work, school, volunteer activity, leisure
activity); and abiding by the law enough to avoid in involvement with the justice system.
Safety and health: The person does not want to or does not harm him- or herself, or die from suicide,
is free from physical and psychological harm or neglect in the person's social environment (home,
school, work, and service settings); physical health; treatment effects, including medication, are
more positive than negative; safety and health are not threatened by disabilities, lack of dignity, or
discrimination; services can be terminated safely and in a planned way; and a person who receives
few services has a secure sense they can obtain more services in a timely manner.
In Australia, the 2001 report to the Victorian Mental Health Outcomes Strategy, Consumer
perspectives on future directions for outcome self-assessment (Graham et al 2001), distilled the
following domains of outcomes from its consultations with consumers:
Quality of life: suitable accommodation∗, enjoyment of activities/life*, undertaking of meaningful
work (paid/unpaid)*, spiritual values and fulfilment*, meaningful occupation of time*, sense of
happiness, the way you spend your leisure time, creative activities, capacity to have choices, sense
∗ The outcomes within this domain most often identified by consumers in this study
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 16
of responsibility and control over one’s life, level of self esteem and self confidence, hope for the
future.
Functioning: maintaining your house*, handling money*, social activities*, cooking, maintaining
appearance/demeanour, meeting financial obligations/budgeting, ability to work, ability to
manage medication and side effects, maintaining friendships.
Physical health and health risks: medication - level and side effects*, sleeping habits*, eating habits
and diet*, energy levels*, level of cigarette smoking.
Relationships: quality of relationships with family and friends*, extent of social networks and
activities*, amount of friendships, level of support from family, friends.
Illness symptoms: anxiety levels*, memory*, ability to break habits/cycles*, depression*, level of
concentration, number of nightmares, understanding things going on around you, thought
disturbance.
Coping and recovering from illness: ability to recognise and take action on early warning signs*,
having a strategy to break habits/cycles*, minimising any detrimental effects of treatment and
care*, adjusting to the effects of medication*, coping with day-to-day stress and change*, dealing
with the impact of the illness, responding to how people treat you (stigma), coping in return to
work, mood swings, hearing voices, developing a recovery plan and ability to initiate action.
Satisfaction with service quality: the help you have been able to get from the service*, the information
given by the service about your mental illness, medication and your rights as a consumer of the
programs*, the attitude of staff toward you and whether you have been treated with dignity*, staff
listening to your opinion*, feeling confident to express yourself to your mental health worker*,
being able to advocate for yourself to the Mental Health Review Board.
In short, the recent literature reports that themes that appear consistently when consumers (and to a
substantial degree clinicians) are invited to identify outcome domains for consumer self-reports are
quality of life, functioning, personal relationships, living arrangements, education and employment,
coping and recovering from illness, satisfaction and fulfilment, personhood, empowerment, hope,
physical health and health risks, illness symptoms, safety, and satisfaction with services.
Dimensions of consumer-focused outcomes
Irrespective of which interest group identifies the outcomes of mental health services, a number of
relevant dimensions of these outcomes emerge from the literature and from the application of broader
health service theory. The World Health Organization definition of health notes that positive
functioning must be considered as well as deficits in functioning (WHO 1948). While various
published scales seek to measure only deficits, others purport to have the capacity to measure positive
functioning. Often this is equated with ‘well being’. Wilkin et al (1992) discuss the issue in depth.
They conclude that while the concept of positive health is an important one: “its application in
instruments designed to measure needs for and outcomes of, health care must be limited to those
which aim to tap subjective experiences”.
Lehnhoff (1993) and Tooth et al (2001) discuss positive health in terms of the positivity of one’s
approach to health issues and their measurement. “Health professionals often disrupt the
normalisation process (that emphasises abilities and recovery) by continually introducing a problem
saturated perspective which services the illness rather than providing the help one needs for getting on
with your life … the participants focussed on their strength and used them to help themselves recover.
A major problem with health professionals was their focus on deficits, on symptoms, on what the
person could not do, resulting in the stripping of hope”.
A theme that arises in the literature is acknowledging both internal and external states as outcomes of
mental health services. Objective aspects of quality of life fall into the external events category, as do
the status of relationships in social contexts, and offending behaviour. Implications for measurement
are twofold. First, external events are by their nature observable by others and hence may be more
amenable to measurement through sources other than the consumer. Secondly, if external events are
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 17
viewed as legitimate outcomes of mental health services, they will be treated in a different way from
when they are viewed as merely indicators of internal states. If they are viewed as indicators of
internal states, they will cease to be of interest if a more effective indicator is found.
The relative importance of changes in external conditions also requires further investigation. If action
in relation to a mental health service is based on the value of that service, and the value of a service is
gauged by its ability to achieve its ultimate intended outcomes, then the place of external events in the
mental health service model must be established. In particular, it must be established whether
attention is given to internal states so that people can operate effectively in their environment, or
living environments are of interest only insofar as they affect the way people think and feel.
Consumers’ capacity to assess outcome
In 1997, when 25 knowledgeable people were invited to the Institute for the Future to imagine the
future of health systems, one of their scenarios pictured the impact of the information revolution on
health care. In this model, ‘information age health care’ inverts the traditional ‘industrial medicine’ of
the present (Jennings et al 1997):
Information age health care
Cost
Primary
Self-care
Source: BMJ
Low
Communities of interest
Low
Multidisciplinary teams
Competence hierarchies
Clinicians as facilitators
Clinicians as partners
Clinicianss
Clinicianss as controllers
Human cost
High
Professional care
Encouraged
Tertiary
Secondary
Discouraged
Discouraged
Encouraged
Industrial age medicine
High
Source:
Source: BMJ
A BMJ editorial commented: “The centrality of consumers is the main characteristic of [this] scenario
–’the informed consumer.’ This is a form of health care that might operate within any other system.
Consumers will use information technology to access information and control their own health care,
consulting professionals much less often. … Instead of being viewed as the apex of a system of care
that hardly recognises the large amount of self-care that occurs now, professional care will be viewed
as the support to a system that emphasises self-care. Healthcare providers will progress in this world
from managing disease to promoting health, and they will do this through lifetime plans that are built
on intimate and detailed knowledge of customers” (Smith 1997).
Kaplan & Ware (1989) have done detailed high quality work on whether consumers can share reliably
and validly in assessing clinical outcome, functional status, and well being. They have shown:
•
•
•
•
•
•
Consumers assess different aspects of their care differently (access, delays, quality of
accommodation, doctors’ interpersonal and technical skills, and personal outcome). Therefore
these different dimensions should be measured separately.
Consumer evaluations are the most practical source of information on access to care, and the
nature of the provider-consumer relationship.
Consumer satisfaction is positively related to the quality of the relationship, to health status, and
to compliance with treatment.
Expanding the role of consumers in their own care enhances the treatment process to produce
better outcomes.
Consumers can provide information on tests ordered, treatment recommended, and medications at
least as accurately as other sources of information such as medical records.
A range of studies in different clinical settings has demonstrated relationships between clinical
measures of health and consumer reported health outcome.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 18
•
Consumers may or may not be able to distinguish between highly competent care and nice, warm
humane but incompetent care.
Factors that may affect capacity
One of the problems inherent in consumer self-reports arises from the changes in perception that
occur during the course of a mental illness. For example, symptoms of depression include anhedonia,
loss of interest, negative self-thoughts, and negative perceptions of the world (DSM-IV). The more
severe the depression, the more severe will be the perceptual distortion. In this sense, the subject of
change will affect the measurement of change.
Following its 1998 survey, the USA Center for Mental Health Services, Mental Health concluded that,
“[w]hen properly designed and rigorously executed…[such surveys] can be completed successfully
by…individuals with disabilities, including serious mental illness” (CMHS 1998). Of the respondents
with a diagnosis of schizophrenia, only 7% were too unwell to complete the questionnaire, while only
3% of those diagnosed with major depression could not complete it owing to illness.
Corrigan et al (1996) consider that thought disorders compromise the consumer’s ability to report on
their own circumstances.
Cummins & Baxter (1994), in their discussion of QOL scales applications in the field of intellectual
disability, note the role of intellectual functioning in limiting item comprehension. Other research
suggests that mental illness has less effect on the validity of self reports than would be expected.
Eisen et al (1991) comment that “unreliability or lack of validity may be attenuated in these patients
who have in essence already admitted that they have a problem”.
Kessler et al (2000) discuss the distinction between episodic memory where individual occurrences
are recalled, and semantic memory where a person has “a history of many instances of the same
experience that cannot be discriminated [and] the separate instances of the same experience tend to
blend together in memory to form a special kind of memory schema … a general memory for a
prototypical experience”. They argue that “when a survey question is designed to ask about a
particular instance or experience, it must be posed in such a way that the respondent knows he or she
is being asked to recover an episodic memory.”
Other characteristics of consumers of mental health services that may affect the usefulness of selfreports have also been identified in the literature. Importantly, these include literacy and health
literacy (eg Ventres & Gordon 1990; Elson 1993; Plimpton & Root 1994; Baker et al 1996; AMA Ad
Hoc Committee 1999; Miller 2001).
Complementary measures
The usefulness of self-reports in measuring the outcomes of mental health services can be established
only when self-reports are compared to other methods of data collection. Kaplan & Ware (1989) say:
“The most comprehensive measures of consumers’ self-assessed health tap at least five distinct health
concepts: physical health, mental health, social functioning, role functioning, and general well being.
Because each of these dimensions makes a unique contribution to the total picture of consumers’
overall health status, each must be measured and interpreted separately to understand consumers’
health status over time.” This means that, to assess the outcome of care, several complementary
measures are needed.
In their treatment of self-report measures in the field of intellectual disability, Cummins & Baxter
(1994) cite a number of studies that compare self-reports with reports by significant others such as
family, friends or other carers.
Other options for collecting data that measure outcomes of mental health services include objective
measures such as photographs (eg Nurcombe & Fitzhenry-Coor 1987) and reports by observers such
as carers or family members (eg Mirin et al 1991, Rosen et al 1989). Trauer (1998) discusses the
importance of gathering information from significant others, but only insofar as “the index patient is
not always the only person to receive services”. Trauer regards the index patient’s significant others
as also consumers of the mental health service.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 19
Implementation and administration
When to measure
Dorwart et al (1994) draw a distinction between measuring and monitoring outcomes of mental health
services, noting that monitoring introduces a time component (measurement over time). The literature
identifies a range of points in time when the status of both internal and external conditions may be
important. In order to assess change in these conditions, at least two points in time must be
considered, even if change is established in retrospect. Smith et al (1996) nominated intake, progress
at 3 monthly intervals, or 4-month or 6-month intervals from baseline until remission of the episode as
suitable times for measurement. Walter et al (1996) proposed no less than one month after “optimal
symptom reduction”. Other proposed milestones are when significant life events occur outside of
treatment, when the treatment setting alters (eg a change from inpatient to outpatient treatment), or
when a course of treatment is completed. However, it is often unclear whether the suggested times are
points at which outcomes are of interest, or times when measurements of outcomes should be made.
Such suggestions depend for their usefulness on the accuracy with which terms are defined and the
uniformity with which they are applied in the field. Some points are defined in relative rather than
absolute terms.
More specifically, Nurcombe & Fitzhenry-Coor (1987) suggest different times for different outcomes
for the same individual. They describe a case study in which multiple goals such “eradicate acne”,
“foster social skills” and “alleviate depressed mood” are identified and monitoring regimens ranging
from “weekly photographs of facial and shoulder acne”, “weekly reports from primary worker and
milieu staff concerning the patient’s conversation …” and “self concept test administered by
psychologist (baseline and 3 months)” are reported.
“If people are unwell, especially those who have been recently admitted to hospital or have just
commenced treatment, they may either be unable or unwilling to complete a questionnaire even if
they are assisted to do so. This raises questions of how to measure outcomes in such situations, and of
when it is appropriate to use self-report measures for outcomes assessment” (Stedman et al 1997).
At the 1998 Conference “Getting better: outcomes measurement and resource allocation in
psychiatry”, Trauer pointed out that “assessing outcome at significant points in the patient’s career,
rather than at fixed intervals, tends to bias the sampling in favour of the more unstable patient, and
produces spurious impressions of improvement” (Walter et al 1998).
How the data is used and stored
Much of the literature directly addressing the use of consumer self-reports in measuring the outcomes
of mental health services deals with limitations of the system by which these self-reports are
collected, stored, analysed, interpreted and provided to those who need the information for decision
making. Gilbody et al (2002) commend the work of Rosenheck et al (1999) as an example of rigorous
outcomes research. These researchers discuss how databases should be established, and argue that for
outcomes databases to be “capable of producing rigorous and informative research” they should
include large numbers of subjects; use standardised instruments that are appropriate for the clinical
condition being treated; measure outcomes in multiple relevant domains; include extensive data in
addition to outcomes measures in order to support matching; collect data at standardised intervals
after a sentinel event such as entry to or discharge from hospital; take aggressive steps to achieve the
highest possible follow-up rates”.
One of the significant issues raised by this research is the question of linking consumer self-reports
and ratings to factors that may be causally linked to outcomes – for example, involvement in different
types of treatment, significant life events, diagnosis, etc - information that does not directly describe
the outcomes of mental health services but is used in conjunction with measures of outcome to
enhance their validity or link them to elements of the treatment process.
The most valid information about factors such as treatment type and dosage can often be obtained
from treatment service records. However, some monitoring systems trade validity for feasibility or
convenience by including items that measure these factors in consumer completed instruments. While
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 20
the utility of the instrument may be reduced including additional items, eliminating the need to
establish interfaces between multiple data collection and management structures may enhance the
feasibility of the monitoring system.
Name linked data and confidentiality
Data collected in relation to mental health services is possibly more highly sensitive that other health
related data, given levels of stigma associated with mental illness. Stedman et al (1997) found that
confidentiality and the eventual uses of outcome data were real issues for consumers. The context or
monitoring system in which a consumer self-report measure is applied will have ramification for the
usefulness of that measure. Systems that preclude unit record identification for the sake of
confidentiality also preclude the use of such information in individual case planning and management.
Systems such as those for notifiable diseases have been developed that enable preservation of
confidentiality standards when data is aggregated, as well as the use of data in individual clinical
settings. This is achieved by the use of codes identifiable by the consumer and/or the clinician only.
Options for individualising measurement
A significant body of literature stresses the importance of consumer self-reports of outcomes in
individual clinical work. This body of literature also propounds the importance of tailoring
measurement instruments to the individual involved. Essentially the argument is that not all outcomes
are relevant to all consumers so rather than ask a large number of irrelevant questions about irrelevant
outcomes a more efficient approach to measurement is to have the consumer identify outcomes that
are relevant to them and to structure questioning around those items only. Measurement can be
individualised further by allowing the consumer to determine aspects of a measure’s scaling
properties, in particular the response options or scale anchors.
This procedure forms the basis of Goal Attainment Scaling (GAS) and is described well by Kiresuk et
al (1994). Quality of life measures have also been applied in this fashion, as described in Sutherland
& Till (1993).
The issue of individualised measurement appears to be of practical relevance to consumers. In an
investigation of the three self report measures BASIS-32, MHI and SF36, Stedman et al (1997) report
a response to questioning about “the actual outcome of filling out these questionnaires” thus: “It is
taken for granted that the information gathered will benefit service providers, but how will it benefit
consumers unless a tailor made questionnaire is developed?”
While individualised measures may prove more relevant to consumers and reduce completion
demands, it is likely that they would result in increased demands arising from the aggregation of data,
and would result in limitations on the interpretation of aggregated data. Unintended outcomes are not
the subject of measurement, and theoretically changes in phenomena that may be intended outcomes
of a mental health service but are not of interest to the individual would need to be scored as zero
before aggregation could take place. Additional coding difficulties would arise if the outcomes
identified by the individual were more specific than categories of outcomes of interest across a group
of consumers.
Combination of measures
While one measure, one type of measurement or one instrument format may appear more or less
valuable in isolation, its usefulness may well be enhanced when combined with others. Goal
attainment scales, for example, could be complemented by some open-ended questioning about
unintended positive outcomes at the completion of treatment.
Byrne (2000) says a single outcome indicator will not suit all purposes and all populations, but
multiple measures within one health system would reduce capacity to compare outcomes across
components of the system or between different intervention groups. It would also be likely to lead to
the choice of a very low level indicator so that it could encompass the largest number of subjects. He
therefore suggests that nested outcome measures should be used in a fashion where a single measure
is selected from a number of classes of measures namely; comprising self-rated scales, clinician rated
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 21
scales, fully structured interviews administered by trained lay interviewer or computer, and semi
structured diagnostic interviews by expert clinicians.
Typically there are discrepancies between information provided by consumers, and reports on
consumers by family, friends, neighbours, case managers and other service providers. Given the errors
that exist in information obtained by any one measure, it is advisable to seek information from more
than one source. A higher measure of certainty may be attached to items of information that are
consistent across different sources.
The availability of such collateral from carers may depend on the openness of the system to family
involvement. A literature review by Ellers (1993) found that, in mainstream healthcare, families
influence health status, clinical outcomes, and patterns of service use. “Family involvement in
programs of prevention, treatment, and rehabilitation has been demonstrated to speed patients’ rate of
recovery and lower their risk of mortality, decrease reliance on medical services and lower the rate of
rehospitalisation, improve patients’ adherence to medical regimens, and improve psychosocial and
family functioning.”
However, Jubb and Shanley (2002) say that, in the Australian mental health system, “treatment under
the traditional medical model has largely ignored the family’s knowledge, expertise, and potential
therapeutic effects …” They say there is a need for more education and support for families, greater
involvement of families in the consumer’s care, and an improvement in staff attitudes towards the
families of mental health consumers.
This is not dissimilar to comments about mental health services in some other countries. Brennan
(1998), describing the situation in the UK, says relatives have had an increasingly important role in
providing care in the community to people with mental health problems, yet numerous studies have
highlighted a lack of support from health care professionals. Similarly, Gall et al (2001) say existing
UK interventions for families often focus on the needs of the consumer rather than the carer, and
describe a nursing initiative to enhance a collaborative partnership with carers.
Stanhope (2002) contrasts India with the US in the extent of family involvement in psychosocial
rehabilitation. India emphasises interdependence, an externalise locus of control, and family
involvement as significant factors in caring for people with psychiatric disabilities, while
rehabilitation goals in the US focus on independence and individual productivity.
Assessing the psychometric properties of specific measures
Earlier sections of this review have discussed the nature of the outcomes of mental health services that
need to be measured by consumer self reports. In particular, it was concluded that outcome measures
need to measure a range of phenomena, but more specifically they need to measure changes in these
phenomena, and in particular changes that are attributable to interventions undertaken by the mental
health service. Establishing the validity of such measures determines the extent to which self-report
instruments succeed in doing this. Various methodological options exist for this process of validation
and have been reported in the literature.
The most common methods of validation appear to have been the investigation of internal consistency
– that is, the extent and manner in which responses to items in a scale correlate with each other, how a
new measure corresponds to existing measures of established validity, and how various related
measures correlate with each other. Dornelas et al (2001), for example, examined the validity of their
Outcome Assessment Program questionnaire by comparing a clinician report version compared with a
consumer report version. Studies have also assessed validity through an investigation of how changes
in mental health or living situations are reflected in changed scores returned by an instrument.
Reliability, or the extent to which a measure maintains its validity across differing administration
conditions (eg different times, testing situations and testing personnel), has also been assessed and is
relevant if administration conditions are expected to change.
A variety of different types of measures can be used in evaluation, such as questionnaires, checklists,
interviews, and direct observation. There is widespread agreement that reliable and valid quantitative
measures are essential in evaluation, but supplementary qualitative techniques such as focus groups,
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 22
unstructured interviews, open-ended questions and analysis of stories and scripts may add useful
information and generate new ideas.
Quantitative measures have the advantage of providing standardised information that permits
comparison across different studies, populations and over time. Quantitative measures are selected on
the basis of known reliability and validity and the availability of appropriate norms. Many quantitative
measures, such as standardised questionnaires, also have the advantage of being relatively brief, easy
to administer, and easy to score and interpret. They do not typically require the subjective
interpretation of the assessor in making judgements about the information presented.
Traditional criteria used in selection of measures include:
•
•
•
•
•
•
•
•
•
•
Reliability: The measure must produce accurate information. Elements of reliability include
internal consistency, test-retest reliability, inter-rater reliability, and cross-informant agreement.
Validity: The measure actually measures what it is intended to measure. Forms of validity that
should be considered in selecting a measure include face validity, content validity, and construct
validity.
Suitability: The measures must be suitable for use with the target population, with limited
budgets, and staff needs training. Measures need to be relatively brief, simple to complete by the
recipient, and easy to understand in language level with simple instructions and clear response
formats.
Relevancy: The measure must be clearly related to the objective, provide useful information, and
allow meaningful conclusions to be drawn. It should address dimensions that are important to the
various stakeholders, including consumers, carers, policy makers, program managers, program
staff, and the community at large.
Availability of norms: In order to interpret whether an individual’s response is high, low or
equivalent in comparison with the general population or a particular social group, it is important
to have normative data (mean values and standard deviations) from a representative sample.
Norms need to be relatively recent from a large sample size, and cover the same demographic
characteristics as the individuals being assessed.
Cultural relevancy: The wording and content of the measure should be appropriate to the
cultural context being canvassed. Measures should be used that are equally applicable across
cultures, and that do not show different psychometric properties for different cultures.
Developmental validity: Cognitive and social development factors influence the readability,
understandability, and meaning of test items. It is important that the content and format of a
measure is appropriate for the age level of the person being assessed.
Financial justifiability: The measure produces information of sufficient value to justify its cost
of collection, including the price of test materials, and the cost of staff time to administer,
score and interpret the results, in comparison to the value of the results.
Sensitivity to change: The measure must be sufficiently sensitive and accurate in its
measurement to detect relevant changes over time.
Comparability with other studies: An important consideration in selecting measures is whether
they have been widely used in similar evaluations. The use of measures that have been widely
used nationally or internationally permits comparison of results with other similar programs.
(adapted from Spence, Donald et al 2000)
The American Psychiatric Association has recently published a Handbook of Psychiatric Measures
containing assessments of the properties of a very wide range of measures, including helpful
summaries of a number of consumer self-rated instruments. (Rush et al 2000)
One approach to determining the nature of mental health service outcomes has been to examine how
responses to groups of individual items in scales correlate with each other when measures are
administered to large groups of consumers. For example, Eisen, Dill & Grob (1994) found five
clusters of items within the BASIS-32 that appeared to correspond to the five constructs (1) relation to
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 23
self and others, (2) daily living and role functioning, (3) depression and anxiety, (4) impulsive and
addictive behaviour, and (5) psychosis.
Veit & Ware (1983) describe the model of the Mental Health Inventory (MHI) as a higher order
structure defined by two correlated factors - psychological distress vs well being - and five lower
order factors: anxiety, depression, emotional ties, general positive affect, and loss of behavioural
emotional control. Results of field testing, they say, supported scoring options ranging from five
distinct constructs to reliance on one summary index.
Headley et al (1991) identified four dimensions: mental health, life satisfaction, positive affect, and
anxiety and depression. By contrast, others (eg Fordyce 1978, Grichting 1983, and Kamman et al
(1979) speak of a single unifying factor they called simply “happiness”, and Stones & Kozma (1986)
also found a single factor which they believed explained responses on individual scale items
adequately.
Berwick et al (1991) assessed the validity of the MHI-5, MHI-15, GHQ-30, SSI-28 scales by
comparing their results to the criterion measure of results from the Diagnostic Structured Interview
(DIS). A number of studies have used a global self-report of well being as a criterion measure. To a
certain extent these approaches are circular, in that comparisons are only as useful as the criterion to
which the measure in question is being compared. Assessments made by clinicians are not necessarily
a more valid indicator of internal states than are consumer self reports. Dornelas et al (2001) in a
study of post-hospital adjustment found good levels of concordance between clinician and consumer
ratings of anxiety-depression, but poor concordance in “functional adjustment”. If individual items or
subscales do not correlate well with a global self-assessment, the fault may be in the individual’s
ability to make a global summation of their condition. If, however, responses to questioning are
considered to be an outcome in themselves (eg an intended outcome of mental health services is that
consumers say that they feel better), then the task of validation becomes simpler.
Sensitivity, or the ability of a measure to pick up small changes in the phenomenon being measured,
has also been assessed, but the literature does not appear to deal with the level of sensitivity required
by decision makers. It could be expected that in evaluating mental health services, or even in
evaluating service to an individual consumer, some small changes in some aspects of outcomes might
not be considered worth the effort. Stedman et al (1997) discuss specificity in terms of clinically
meaningful levels of change attributable to treatment, and in their investigation of six outcome
measures conclude: “In order to make appropriate judgements concerning clinically meaningful
change for an individual, and to attribute such change to the influence of various factors such as
treatment efficacy, required more information than the six selected measures appear able to provide.”
Where consumer reports of external events are in question, criterion measures are much easier to
establish. The validity of such reports does not appear to have received a great deal of attention in the
literature when compared to the validation of measures that address internal states.
Sources of sampling error
Boothroyd et al (1998) compared interviewer administered and mailed outcome surveys of mental
health service consumers, and reported no significant differences between the methods in terms of the
content of responses, but did identify two important sampling biases. In their sample of Medicaid
eligible consumers in the US, they found an under representation of men and blacks, and an overrepresentation of “other races” in their sample. Overall response rates for mailed surveys were 75% of
the response rate for interviewer-administered surveys.
Eisen (1995) compared results of the BASIS-32 administered through a structured interview with
results obtained when it was self-administered, and concluded that self-administration was “a viable
alternative”. Weissman and Bothwell (1976) had found that consumers reported higher levels of
impairment on the Social Adjustment Scale when the scale was self-administered compared with
interview administered, but scores on the two versions of the scale correlated highly.
Techniques that assist in the estimation of completion rates can be expected to improve the usefulness
of consumer self reports. The facility to record either non-completion of treatment or non-completion
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 24
of outcome measures can be built into both the self-report measure and the system of measurement,
for example by allowing an open ended question where statements of why the full instrument was not
completed in full. Measurement systems can be organised to allow for telephone follow up of
consumers who withdraw from treatment unexpectedly.
Response biases
In any study in which responses of some sort (such as answers to set questions in an outcome
measure) are required of participants, response bias exists if for some reason they are more likely to
respond in one way than in another. For example, in a paper on women’s roles and their mental
health, Repetti (1987) says that “a correlation between individuals’ role evaluations and their selfreported mental health could be explained simply by response bias - some people are inclined to use
positive terms and others are inclined to use negative terms when describing themselves and their life
circumstances.” Similarly, Cummins & Baxter (1994) speak of the importance to outcome
measurement of accounting for the response biases of acquiescence and ‘nay-saying’. The implication
for the evaluation of self-report measures may be that their susceptibility to response biases should be
assessed.
Instructions, protocols, formats
The extensive literature on psychometric testing suggests that presentation aspects of instrument
construction can have implications for the usefulness of a measure in terms of both its psychometric
properties and its feasibility or practicality. A number of considerations in this regard have been
identified in the literature.
In their evaluation of six consumer self-report measures, Stedman et al (1997) reported: “Consumers
voiced that the language used in the questionnaires (BASIS-32, MHI, and SF36) was out of date.
More modern questionnaires with appropriate language are required”. It was not clear from this report
whether the contemporary quality of the language used in questionnaires relates to the usefulness of
these measures, or whether it is simply a preference expressed by consumers.
The issue of using open ended as opposed to closed question formats arises in the literature. Closed
questions are usually easier to administer and score, and the data they yield is usually easier to
aggregate and compare to other data, but open-ended questions can be more useful in eliciting
important unexpected information. They can also be more useful in eliciting information that the
respondent believes is relevant and hence may contribute to positive unintended effects of
measurement treated here in a later section. Graham et al (2001) say: “There was strong support
[amongst consumers] for including a minimum of one open-ended question at the end [of a consumerrated questionnaire] to tap concerns held by the consumer that are not covered in a standard
instrument. This might be worded along the lines of: ‘In what ways do you think that you have made
progress in the past six months?’”
Byrne (2000) discusses the applicability of various formats for responding to questions. In addition to
traditional Likert scaling where a series of points on a scale are marked with descriptive anchors such
as “Agree strongly”, “Agree somewhat” etc, Visual Analogue Scales (VAS) can be used (Aitken
1969). Such scales mark only the extremities of a scale, and ask respondents to mark a point on a line
between the extremities to describe their answer.
Byrne (2000) also notes that a variety of non-verbal scales have been used where respondents’
“literacy cannot be taken for granted, oral communication is not possible or proficiency in the English
Language is not assured.” This form of scale provide images or icons such as stylised faces that are
recognisable as mood states, and ask the respondent to point to the icon that best describes their own
mood state. Nelson et al (1987) describe the use of a sophisticated version of such scales by the
Dartmouth Primary Care Cooperative Information Project.
Feasibility/ practicality
Feasibility or practicality can be considered in both the individual administration of a measure and the
broader system of data collection, scoring, storage and management. In the words of Stedman et al
(1997), “A measure is not a method (Andrews et al, 1994). There is more to consumer outcomes
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 25
assessment than the regular administration of the HoNOS, LSP, MHI or any other questionnaire.
Using standardised outcomes measures may provide stakeholders with a ‘common language’ to report
on consumer outcomes (Johnson, 1995). However, they will be of limited utility if the methodology
for implementing and analysing the data are not considered in the same instance”.
The literature on outcome measures in mental health does not appear to have dealt directly with the
issue of balancing doing with measuring. Some authors note the burden of data collection and
management, but do not specifically deal with when this burden becomes excessive. The field of
project evaluation has long espoused a principle that devoting 10% of project budgets to evaluative
research is appropriate, but no such principle has been investigated in the mental health services field
specifically. While the act of measuring can be beneficial in itself, investing resources in
measurement can be at the expense of intervention.
Options for administration of self-report measures include electronic, computer or online
administration, clinician administered, personnel independent of the therapeutic relationship,
telephone administration, and paper instruments
Walter et al (1996) note two administration systems employing electronic communication technology
rather than human administration namely; the Computerised Outcome & Resource Measure (CORM)
and the Interactive Voice Response (IVR) system. Other recent studies have also compared computer
applications with other modes in several clinical settings (eg Locke & Gilbert 1995, Kobak et al 1996,
Kobak et al 1999, IsHak et al 2002 pp.383-91, and Ryan et al 2002).
Boothroyd et al (1998) studied aspect of administration systems by comparing mailed follow up
surveys of mental health service consumers with an interview-based approach. They described mail
surveys as “the typical way that population based monitoring has occurred in the past”. This US study
found that unit costs for mailed surveys were $28 per questionnaire compared with $80.50 per
questionnaire administered in an interview situation. This finding is comparable to that of Felton &
Shern (1995). Similar feasibility studies for the Australian context have not been reported.
A CMHS investigation of mailed self reports concluded that, “given the importance of understanding
the outcomes achieved in mental health treatment systems, these monitoring techniques seem capable
of producing unbiased population estimates at an affordable cost” (1998).
In their investigation of the Carers’ and Users’ Expectations of Services: User Version (CUES-U),
Lelliott et al (2001) found that “half of the people in this field trial sought help with its completion.
However, the type of help needed usually could be provided by a friend, carer or advocate.”
Dornelas et al (2001) reported attrition rates in their follow up study of mental health service
outcomes having found that only 38% of former consumers returned their questionnaires. While
multiple factors could have accounted for this return rate the acceptability of the questionnaire may
have been a significant one. The examination of return rates and perhaps more importantly completion
rates could be used to assess the acceptability of self-report measures.
Aspects of individual administration
The issue of completion time also arises in the literature but is rarely dealt with empirically. Rosen,
Hadzi-Pavlovic & Parker (1989), for example, suggest that a quality of an effective consumer selfreport measure is that “it should be brief” but do not discuss what “brief” means in practical terms.
Others report on the time taken to complete a measure, but do not relate this to information about
what proves to be an acceptable maximum duration. Kelley et al (1989), for example, report a mean
completion time of three minutes and thirty seconds for the full version of the Treatment Evaluation
Inventory and a mean completion time of two minutes ten seconds for a short form of the same
instrument. While 71% of their respondents reported preferring the short form, it is unclear whether
their preference reflected completion time or to other factors. The fact that a significant proportion of
respondents did not prefer the shorter form suggests that a completion time of three and a half minutes
was acceptable.
No normative statistics have been established in this field. Graham et al (2001) report a somewhat less
direct indicator of acceptable completion time arising from their research: “In general, participants
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 26
suggested that ‘no more than two pages’ was a useful guideline for any consumer self-rated measure
but the font size should be ‘big enough to see’”. Walter et al (1996) suggest that clinician measures
should take no more than five to ten minutes to rate, but cite no empirical evidence for this assertion.
Rost et al (1992) sought to assess the consumer-completed component of their “outcomes modules”
instruments against a number of criteria for feasibility including completion times. They reported a
completion time of 25 minutes, but were unable to make firm conclusions about completion time and
the acceptability of the measure since instruments were completed in a research setting, and in the
context of extensive additional questioning and payments to participants.
The impact of measurement itself on outcome
The unintended positive and negative effects of using a particular consumer self report measure needs
to be taken into consideration when assessing its usefulness. Unintended positive effects or byproducts such as the therapeutic effect of measurement can be used to choose between measures of
otherwise equal utility and steps can be taken to maximise valuable by-products once the ability of a
measure to achieve its intended purposes have been optimised. Negative effects however, must be
considered in determining the net value of a measure or measurement system.
Process aspects of the service may affect the validity of consumer self-reports as a measure. For
example, consumers may be more or less likely to attribute cause to the mental health intervention if
they are satisfied with the treatment process, and satisfaction or dissatisfaction with the process could
affect the estimations made in self-reports of the size or nature of outcomes.
Eisen et al (2000), for example, investigated the effect on consumers of providing self-reports (using
the BASIS-32), and reported that “patient’s perceived involvement in treatment decisions was
significantly greater for those who met individually with a psychiatric resident to review their selfreported symptoms and problems”. In this case a system of measurement where the results of
measurement are reviewed was seen as contributing to the effectiveness of treatment.
In the evaluation and quality management field, there is a commonly held view that what is easiest to
measure gets measured, and what is measured gets focused on. Failing to measure the outcomes of
mental health services in a balanced fashion could result in changing service priorities simply on the
basis of the practicality of measurement.
Callaly & Hallebone (2001) say that the introduction of a suite of outcome measures is a challenge to
the entire mental health system and its workforce. They report an action research project that
examined the introduction of an outcomes monitoring system in Victoria, and conclude that without
this commitment to use the results, “outcomes measures will be viewed by clinicians as a bureaucratic
and clinically irrelevant task”. It is crucial, they say, to engage management in reviewing case
management systems and continuity of care, redesigning consumer record systems so that outcome
measures are an essential ingredient in individual treatment plans, and developing technical support.
The knowledge and advice of clinicians is also crucial - the effectiveness of the initiative depends on
their agreement, enthusiasm, and willingness to encourage consumers to use outcome measures.
Engaging consumers in planning, delivering, and evaluating services is not only a matter of rights, but
a highly effective way to gather information and ideas from those who know and use the services.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 27
References
Ackoff R. 1975. Redesigning the future. New York: Academic Press
AMA [US] Ad Hoc Committee on Health Literacy. 1999. Health Literacy: Report of the Council of Scientific
Affairs. JAMA 6 552-7.
Andrews, G., L. Peters, M. Teeson. 1994. Measurement of consumer outcome in mental health: A report to the
National Mental Health Information Strategy Committee. Sydney: Clinical Research Unit for Anxiety
Disorders.
Baker D. W., R. M. Parker, and M. V. Williams. 1996. The health care experience of patients with low literacy.
Archives of Family Medicine 329-34.
Berwick D. M., Jane M. Murphy, P. A. Goldman, John E Jr. Ware, A. J. Barsky, and Milton C. Weinstein. 1991.
Performance of a five-item mental health screening test. Medical care 2 169-76.
Boland P. 1991. Making managed health work. New York: McGraw Hill, cited in Ross (2000).
Booth B. M., M. Zhang, K. M. Rost, J. A. Clardy, L. G. Smith, and G. R. Smith. 1997. Measuring outcomes and
costs for major depression. Psychopharmacology bulletin 4 653-8.
Boothroyd R. A., El. A. Skinner, D. L. Shern, and D. M. Steinwachs 1998. Feasibility of consumer-based
outcome monitoring: A report from the National Outcomes Roundtable. Chapter 3 in Center for Mental
Health Services, Mental Health, United States, 1998. Manderscheid and Henderson (eds). Washington DC:
US Government Printing Office.
Bowling A. 1991. Measuring health: A review of quality of life measurement scales. Buckingham and Bristol:
Open University Press, reprinted 1992.
Brown R. B., S. McCartney, and L. Bell. 1995. Why the NHS should abandon the search for the universal
outcome measure. Health care analysis HCA journal of health philosophy and policy 3 191-5.
Burt T. and W. W. IsHak. 2002. Outcome measurement in mood disorders. Chap. in Outcome measurement in
psychiatry: A critical review. Washington, DC, US: American Psychiatric Publishing, Inc. 155-90.
Burt T. and W. W. IsHak. 2002. Use of technology in outcome measurement. Chap. in Outcome measurement in
psychiatry: A critical review. Washington, DC, US: American Psychiatric Publishing, Inc. 383-91.
Burwell B. and B. Jackson. (1999). Personal outcomes measurement in home and community-based services
programs for persons with severe disabilities. Report to the Center for State Medicaid Operations.
Cambridge MA; The Medstat Group. http://www.ncor.org/pomhcbs.htm
Byrne G. 2001. Clinical outcome indicators and clinical practice guidelines for major depression. Brisbane:
Queensland Health (unpublished report)
Callaly T., T Trauer, P. Hantz. 1998. Integration of outcome measures into clinical practice. Australasian
Psychiatry 4 188-90.
California Department of Mental Health. Realignment Performance Outcome System [RPO]. Background,
history, & general information. Sacramento; DMH. http://www.dmh.cahwnet.gov/RPOD/background.asp
Callaly T. and E. L. Hallebone. 2001. Introducing the routine use of outcomes measurement to mental health
services. Australian Health Review 1 43-50.
Campbell J. 1996. Toward collaborative mental health outcomes systems. New directions for mental health
services 71 69-78.
Center for Mental Health Services [US]. 1998. Mental Health, United States, 1998. Manderscheid R. W. and M.
J. Henderson (eds). Substance Abuse and Mental Health Services Administration, US Department of Health
and Human Services. Washington DC: US Government Printing Office.
Center for Mental Health Services [US]. 2000. Mental Health, United States, 2000. Manderscheid R. W. and M.
J. Henderson (eds). Substance Abuse and Mental Health Services Administration, US Department of Health
and Human Services. Washington DC: US Government Printing Office.
Conte H. R., R. Plutchik, S. Picard, T. B. Karasu, and D. Vaccaro. 1988. Self-report measures as predictors of
psychotherapy outcome. Comprehensive Psychiatry 4 355-60.
Corrigan P. W., B. Buican, and S. McCracken. 1996. Can severely mentally ill adults reliably report their needs?
Journal of nervous and mental disease 9 523-9.
Department of Health & Aged Care. 1999. Mental Health Information Development: National Information
Priorities and Strategies under the Second National Mental Health Plan 1998-2003. Canberra: DHAC
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 28
Dickey B., R. C. Hermann, and Susan V. Eisen. 1998. Assessing the quality of psychiatric care: research
methods and application in clinical practice. Harvard review of psychiatry 2 88-96.
Dornelas E A., R E. Correll, L Lothstein, C Wilber, J. W. Goethe. 1996. Designing and implementing outcome
evaluations: Some guidelines for practitioners. Psychotherapy: Theory, Research, Practice, Training 2 23745.
Dornelas E. A., A. L. Botticello, .J. W. Goethe. 2001. Validity of a brief measure of post-hospital adjustment for
psychiatric patients. Comprehensive Psychiatry 5 410-5.
Dorwart R. A, D. Adler, J. Berlant, L. Dixon, J. Docherty, J. Ellison, H. Goldman, M. Sageman, S. Siris, W.A.
Sonis. 1994. Outcomes assessment and psychiatric services. Committee on Psychopathology, Group for the
Advancement of Psychiatry. Hospital and Community Psychiatry 45(12), 1165.
Dyer R.L. 1992. The many faces of public managed care in state mental health programs. A paper to the
summer meeting of the National Association of State Mental Health Program Directors [US], cited in Ross
(2000).
Eisen S. V., M. C. Grob, and D. L. Dill. 1991. Outcome measurement: Tapping the patient's perspective. Chap.
in Mirin, Steven M. (Ed); Gossett, John T. (Ed); et al. (1991). Psychiatric treatment: Advances in outcome
research.
Eisen S. V. 1995. Assessment of subjective distress by patients' self-report versus structured interview.
Psychological Reports 1 35-9.
Eisen S. V., M. Wilcox, H. S. Leff, E. Schaefer, .M. A. Culhane. 1999. Assessing behavioral health outcomes in
outpatient programs: reliability and validity of the BASIS-32. journal of behavioral health services and
research 1 5-17.
Eisen S. V., B. Dickey, L. I. Sederer. 2000. A self-report symptom and problem rating scale to increase
inpatients' involvement in treatment. Psychiatric Services 3 349-53.
Ellers B. and J.D. Walker 1993. Facilitating the transition out of the hospital. In: Gerteis, M., Edgman-Levitan,
S., Daley, J., Delbanco, T.L., (eds) Through the patient's eyes - understanding and promoting patientcented care, 204-223. San Francisco: Jossey-Bass Publishers.
Elson P. R. 1993. Literacy and health. Journal of the Canadian Medical Association 10 1382-3.
Englehardt H.T. Jr. 1996. The foundations of bioethics. New York: Oxford University Press.
Figueredo A-J. and L. Sechrest. 2001. Approaches used in conducting health outcomes and effectiveness
research. Evaluation and Program-Planning. 24(1), 41-59
Fordyce W.E. 1978. Psychology in disability. Archives of Physical Medicine Rehabilitation 1978 59(11), 516-7.
Gall S.H., L. Elliott, J.M. Atkinson, R. Johansen 2001. Training nurses to support carers of relatives with
schizophrenia. British Journal of Nursing 10(4): 238-41.
Gilbody S. M., A. O. House and T. A. Sheldon. Outcomes research in mental health; Systematic review. British
Journal of Psychiatry 181, 8-16.
Goodman R (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology
and Psychiatry. 38, 581–586.
Goodman R, Meltzer H and Bailey V (1998) The strengths and difficulties questionnaire: A pilot study on the
validity of the self-report version. European Child & Adolescent Psychiatry. 7, 125–130.
Goodman S. H., D. R. Sewell, E. L. Cooley, and N. Leavitt. 1993. Assessing levels of adaptive functioning: the
Role Functioning Scale. Community mental health journal 2, 119-31.
Graham C., T. Coombs, B. Buckingham, K. Eagar, T. Trauer, T. Callaly. 2001. The Victorian Mental Health
Outcomes Measurement Strategy: Consumer Perspectives on Future Directions for Outcome SelfAssessment. Report of the Consumer Consultation Project. Wollongong: University of Wollongong
Green R. S. and F. L. Newman. 1996. Criteria for selecting instruments to assess treatment outcomes. Chap. in
Outcome assessment in residential treatment. 29-48.
Hammermeister K. E., A. L. Schroyer, G. K. Sethi, and F. L. Grover. 1995. Why it is important to demonstrate
linkages between outcomes of care and processes and structures of care. Medical Care 33(10), Supplement
OS5-OS16.
Harvard School of Public Health 2003. Measurement, design, and analysis methods for health outcomes
research. Continuing education course outline. http://www.hsph.harvard.edu/ccpe/programs/MDA.shtml
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 29
Headey B. W., J. Kelley, A. J. Wearing. 1993. Dimensions of mental health: Life satisfaction, positive affect,
anxiety and depression. Social Indicators Research 1 63-82.
Hill S.. 1998. Using the evidence: empowering the consumers of health care.
Holloway F. 2002. Outcome measurement in mental health--welcome to the revolution. British Journal of
Psychiatry 181, 1-2.
Iezzoni L.I. 1997. Assessing quality using administrative data. Annals of Internal Medicine 127(8 Pt 2), 666-74.
IsHak W. W., A. I. Abad, M. Alpert. 2002. Innovative approaches in outcome measurement: A futuristic view.
Chap. in Outcome measurement in psychiatry: A critical review. Washington, DC, US: American
Psychiatric Publishing, Inc. 393-404.
IsHak W. W., T. Burt, L. I. Sederer (eds) 2002. Outcome measurement in psychiatry: A critical review.
Washington, DC, US: American Psychiatric Publishing, Inc.
Jennings K., K. Miller, and S. Materna. 1997. Changing health care. Santa Monica: Knowledge Exchange.
Jubb M. and E. Shanley 2002. Family involvement: the key to opening locked wards and closed minds.
International Journal of Mental Health Nursing 11(1): 47-53
Kaplan S. H. and J. E. Ware Jr. 1989. The patient's role in health care and quality assessment. Chap. in
Providing quality care: the challenge to clinicians. Philadelphia: American College of Physicians. 25-68.
Kelley M. L., R. W. Heffer, F. M. Gresham, .S. N. Elliott. 1989. Development of a modified Treatment
Evaluation Inventory. Journal of Psychopathology and Behavioral Assessment 3 235-47.
Kessler R. C., H-U. Wittchen, J. Abelson, Sanyang Zhao. 2000. Methodological issues in assessing psychiatry
disorders with self-reports. Chapter 14 in A.A. Stone et al (eds), The Science of Self-Report: Implications
for Research and Practice. Matwah NJ & London: Lawrence Erlbaum Associates
Kiresuk T. J. and R. E. Sherman. 1968. Goal attainment scaling: A general method for evaluating
comprehensive community mental health programs. Community mental health journal 6 443-53.
Kiresuk T. J., A. Smith, .J. E. Cardillo. 1994. Goal attainment scaling: Applications, theory, and measurement.
Hillsdale NJ: Lawrence Erlbaum Associates.
Kluge E. 1992. Biomedical ethics in a Canadian context. Scarborough, Ontario: Prentice Hall.
Kobak K. A., J. H. Greist, J. W. Jefferson, and D. J. Katzelnick. 1996. Computer-administered clinical rating
scales. A review. Psychopharmacology 4 291-301.
Kobak K. A., J. H. Greist, J. W. Jefferson, J. C. Mundt, and D. J. Katzelnick. 1999. Computerized assessment of
depression and anxiety over the telephone using interactive voice response. M.D. computing computers in
medical practice 3 64-8.
Lawrence J.E.S. and T.J. Cook. 1982. Designing useful evaluations: The stakeholder survey. Evaluation and
Program Planning. 5, 327-336.
Lehnhoff J. 1993. Assessment and utilization of patient strengths in acute care treatment planning. The
Psychiatric Hospital 22(1), 11-15.
Lelliott P., A. Beevor, G. Hogman, J. Hyslop, J Lathlean, and M. Ward. 2001. Carers' and users' expectations of
services. User version (CUES-U): a new instrument to measure the experience of users of mental health.
British Journal of Psychiatry 67-72.
Locke S D. and B O. Gilbert. 1995. Method of psychological assessment, self-disclosure, and experiential
differences: A study of computer, questionnaire, and interview assessment formats. Journal of Social
Behavior and Personality 1 255-63.
Manderscheid R. W., M. J. Henderson, D. Y. Brown 2000. Status of national accountability efforts at the
millenium. Chapter 6 in Center for Mental Health Services. Mental Health, United States, 2000.
Manderscheid and Henderson (eds). Washington DC: US Govt Printing Office.
Manderscheid R. W. and M. J. Henderson 2000. Where is mental health likely to be a century hence? An
editorial perspective. Chapter 1 in Center for Mental Health Services. Mental Health, United States, 2000.
Manderscheid and Henderson (eds). Washington DC: US Government Printing Office.
Mathai, J., P.Anderson, A. Bourne (2002). The Strengths and Difficulties Questionnaire (SDQ) as a screening
measure prior to admission to a Child and Adolescent Mental Health Service (CAMHS). Australian eJournal for the Advancement of Mental Health 1(3). www.auseinet.com/journal/vol1iss3/Mathai.pdf
Mellsop G and G O’Brien. 2000. Outcome Summary Report. A report to the Health Research Council of New
Zealand.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 30
Miller M-E. 2001 Doctor-patient communication and collaboration in elective surgery [dissertation]. St Lucia:
University of Queensland
Mirin S M., Jn T. Gossett, M C. Grob (eds). 1991. Psychiatric treatment: Advances in outcome research.
Washington DC: American Psychiatric Press.
Moos R.H., R Cronkite, J.W. Finney. 1982. A conceptual framework for alcoholism treatment evaluation. In
Pattison E & Kaufam E (eds), Encyclopedic Handbook of Alcoholism. New York: Gardner Press
Morris-Yates, A. & the Strategic Planning Group for Private Psychiatric Services Data Collection and Analysis
Working Group (2000). A national model for the collection and analysis of a minimum data set with
outcome measures for private, hospital–based psychiatric services. Canberra: Commonwealth of Australia
National Outcomes and Casemix Collection [NOCC] 2002. Overview of Clinical Measures and Data Items
V1.02. Canberra: Mental Health & Special Programs Branch, Department of Health & Ageing.
Nurcombe B and I. Fitzhenry Coor. 1987. Diagnostic reasoning and treatment planning: I. Diagnosis. Australian
and New Zealand journal of psychiatry 4 477-83.
Ohio Department of Mental Health 2003. The Ohio Mental Health Consumer Outcomes System Procedural
Manual Fourth Edition (Revised). Columbus OH: Ohio Department of Mental Health
Opler L A., P M Ramirez, d V M. Mougios. 2002. Outcome measurement in serious mental illness. Chap. in
Outcome measurement in psychiatry: A critical review. Washington, DC, US: American Psychiatric
Publishing, Inc. 139-54.
Perlis R H., D Davidoff, W E. Falk, D L. Round, S. Verma, L. I. Sederer. 2002. Outcome measurement in
geriatric psychiatry. Chap. in Outcome measurement in psychiatry: A critical review. Washington, DC, US:
American Psychiatric Publishing, Inc. 77-96.
Perry J. C. and N. H. Sanlian. 2002. Outcome measurement in personality disorders. Chap. in Outcome
measurement in psychiatry: A critical review. Washington, DC, US: American Psychiatric Publishing, Inc.
235-57.
Peselow E. D. 2002. Outcome measurement in anxiety disorders. Chap. in Outcome measurement in psychiatry:
A critical review. Washington, DC, US: American Psychiatric Publishing, Inc. 191-206.
Plimpton S. and J. Root. 1994. Materials and strategies that work in low literacy health communication. Public
Health Reports 1 86-92.
Pulice R. T., L. L. McCormick, and M. Dewees. 1995. A qualitative approach to assessing the effects of system
change on consumers, families, and providers. Psychiatric services Washington, D.C 6 575-579.
Repetti, R. L. (1987). Work and family roles and women's mental health. Institute for Social Science Research.
Volume III. 1986-87 - Women at Work: The Conference Papers, May 1987. Working Paper 6.
http://repositories.cdlib.org/issr/volume3/6
Rosen A., D. H. Pavlovic,. G. Parker. 1989. The life skills profile: a measure assessing function and disability in
schizophrenia. Schizophrenia Bulletin 2 325-37.
Rosenhech R.A., A. Fontana, and M. Stolar. (1999). Assessing quality of care: administrative indicators and
clinical outcomes in posttraumatic stress disorder. Medical Care 37(2), 180-188.
Ross E. C. The promise and reality of managed behavioural health care. Chapter 9 in Mental Health, United
States, 1998. Manderscheid and Henderson (eds). Center for Mental Health Services. Washington DC: US
Government Printing Office.
Rost K., G. R. Smith, A. A. Burnam, B. J. Burns. 1992. Measuring the outcomes of care for mental health
problems. Medical Care 30(5), supplement MS266-MS 273.
Rush A J. Jr et al 2000. Handbook of Psychiatric Measures. Washington DC: APA Task Force for the
Handbook of Psychiatric Measures. xxviii, 820 p. + 1 computer optical laser disc
Salzer M. S., M. Blank, A. Rothbard, T. Hadley. 2000. Adult mental health services in the 21st century. Chapter
11 in Mental Health, United States, 1998. Manderscheid and Henderson (eds). Center for Mental Health
Services. Washington DC: US Government Printing Office.
Santos M., M. Schwarz, A. Aladjem. 2002. Outcome measurement in somatoform disorders. Chap. in Outcome
measurement in psychiatry: A critical review. Washington, DC, US: American Psychiatric Publishing, Inc.
221-33.
Seligman M. E. P. 1995. The effectiveness of psychotherapy. The Consumer Reports study. American
Psychologist 50(12), 965-974
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 31
Siggins I. D. K., M-E Miller et al. 2001. Evaluation of the Extended Aged Care at Home (EACH) Pilot
Program. Aged and Community Care Service Development Reports No.40. Report prepared by Siggins
Miller Consultants for the Department of Health and ageing. Canberra: CoA.
Smith G. R., H. Waxman, D. Snyder, M. Raphael. 1996. Assessment tools in psychiatry. American journal of
medical quality the official journal of the American College of Medical Quality 1 46-9.
Smith R. 1997. The future of healthcare systems. Information technology and consumerism will transform
health care worldwide. Editorial. British Medical Journal 24 May 1495
Spence S. H, M. Donald M, J. Dower J, R. Woodward, P. Lacherez 2000. Performance indicators and
outcomes, measurement tools, and databases for the National Mental Health Promotion and Prevention
Action Plan. Volume 1 of a project by the University of Queensland, Griffith University, and Siggins Miller
Consultants for the Mental Health Branch, Commonwealth Department of Health and Aged Care.
Stanhope V. 2002. Culture, control, and family involvement: a comparison of psychosocial rehabilitation in
India and the United States. Psychiatric Rehabilitation 25(3): 273-80
Stanley J.M. 1992. Decisions to forgo life-sustaining treatment under conditions of scarcity. Journal of Medical
Ethics 185, 16-21
Stedman, T., P. Yellowlees, G. Mellsop, R. Clarke, and S. Drake. 1997. Measuring consumer outcomes in
mental health. Canberra ACT: Department of Health and Family Services.
Stones M.J. and A. Kozma. 1986. “Happy are they who are happy ...” a test between two causal models of
relationships between happiness and its correlates. Experimental Aging Research 12(1), 23-9.
Sutherland H. J. and J. E. Till. 1993. Quality of life assessments and levels of decision making: differentiating
objectives. Quality of life research an international journal of quality of life aspects of treatment, care and
rehabilitation 4 297-303.
Tobin M. and I. Hickie. 1998. Outcomes focused service delivery: developing an academic-management
partnership. Australian and New Zealand Journal of Psychiatry 3 327-36.
Tooth B.A., V. Kalyanansundaram, and H. Glover. (2001). Recovery from schizophrenia: A consumer
perspective. Final report to Health and Human Services Development Grants Program. Canberra:
Department of Health and Aged Care.
Trauer T. 1998. Issues in the assessment of outcome in mental health. Australian and New Zealand Journal of
Psychiatry 3 337-43.
Veit CT and Ware JE Jr 1983. The structure of psychological distress and well being in general populations.
Journal of consulting and clinical psychology 5, 730-42
Ventres W. and P. Gordon. 1990. Communication strategies in caring for the underserved. Journal of Health
Care for the Poor and Underserved 3 305-14.
Walter G., K. Kirkby, I. Marks. 1998. Getting better: outcome measurement and resource in psychiatry.
Australasian Psychiatry 6(5), 252-254.
Weissman M.M. and S. Bothwell. 1976. Assessment of social adjustment by patient self-report. Archives of
General Psychiatry 33(9), 1111-1115.
Wilkin D., L. Hallam, and M-A. Doggett. 1992. Measures of need and outcome for primary care. Oxford, New
York, Tokyo: Oxford University Press.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 32
Part 2: The consultations and development of an evaluation framework
The consultation process
The heart of this project entailed wide-ranging formal consultation in all States and Territories with
consumers, clinicians and service providers in public and private sector mental health services, to
develop an evaluation framework for analysing consumer self-rated outcome measures.
In August 2002, after ISC members were informed that the project was under way, we asked members
of the Reference Group to help us identify people for consultations in each State and Territory. We
wrote to the members asking them for a list of key people (consumers, carers, clinicians, policy staff,
and academics) whom we should consult in each jurisdiction, in focus groups or by telephone
interview. Reference Group members were very helpful in identifying key people for face-to-face or
telephone interviews, and an Informant list in progress was prepared for their information in
September 2002.
We approached all the people nominated by the jurisdictions, and added others nominated by
informants at interview. The response rate from our initial contacts with all the people nominated,
after follow-up reminders, was about 57%. In the case of a list of 32 clinicians in private hospitals
supplied by the Strategic Planning Group for Private Psychiatric Services (SPGPPS), we approached
16 people identified as having central responsibility for collecting self-report measures, and
interviewed or received emailed responses from eight of them. Several people contacted us to say they
were not in a position to respond.
Both the Reference Group and officers in the jurisdictions put us in touch with consumer and carer
organisations and peak bodies, who kindly undertook to organise the groups for face-to-face meetings.
These meetings were held in every State and Territory capital city, and in two regional centres (Alice
Springs and Ballarat). The eventual number of people consulted was as follows:
•
•
•
Face-to-face consultation with 96 consumers, 35 carers, and 33 members of state-based Consumer
Advisory Groups (CAGs) and consumer bodies (we met one CAG on two occasions, and held a
follow-up meeting with the chair of another State’s CAG)
Telephone interviews with or emailed responses from 27 clinicians and academics, several of
whom also supplied very useful materials
Meetings or telephone interviews with 47 officers in the public sectors of each jurisdiction
(including some who also had a service delivery role)
These categories overlapped to some extent, in that many academic informants also had a clinical
role, several of the officers we interviewed in State and Territory departments were clinicians, and
representatives of peak bodies included some consumers or carers.1
For reasons of privacy and respect, we made a conscious choice not to invite or record personal or
clinical details about the consumers who generously participated in the focus groups. However, from
information they volunteered, we can say that men and women were represented in almost equal
numbers, their ages ranged from 20 to 70, and there were approximately equal numbers of those under
and over 35 years of age. At least a third came from regional or rural areas. Contributions to group
discussions disclosed a very wide range of conditions, including bipolar disorder, depression and
anxiety, schizophrenia, and drug-related psychoses, at many points in the treatment cycle.
On the advice of consumer representatives, we took copies of consumer self-rated outcome measures
in use in Australia (Basis-32, K10, MHI) to each consumer consultation, and examples of some other
similar instruments (Ohio, SF-36), so that consumers who had not yet used such a measure could
better understand the ideas behind consumer self rated outcome measurement and inspect the kinds of
measure under consideration. At the time the project began, only two States (Victoria and NSW) had
begun the roll out of a measure, and only one in ten of the consumers consulted in this project had
previously completed such an instrument.
1
In the figures above, respondents have been counted in only one of these overlapping categories.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 33
The messages we heard from consumers and carers
96 consumers of mental health services and 35 carers (mainly parents or siblings of consumers) took
part in group discussions about their expectations and concerns in consumer outcome measurement.
The value of listening to the consumer’s assessment of outcome
Consumers and carers alike believe it is essential to introduce the consumer’s view into how the
system is judged in its performance, the appropriateness of the service mix at the system level, and the
appropriateness of interventions at the individual and family level.
They hope the collection of self-rated outcome data will lead to holistic treatment and coordination of
support, paint the full picture of need, guide service providers to deal with individuals more
respectfully, and lead to reconsideration of what the system should be like.
They hope it will be a major tool for culture change, and influence decision makers from the political
level down to increase the flexibility of the system to meet need and to generate compassionate,
non-discriminatory support for people with a mental illness and their families, carers and advocates.
But consumer outcome measures should never replace face-to-face contact. They fear that, with the
pressure now on the mental health system, providers might be tempted to think they could track
people merely through pencil and paper tests.
Consumer and carer concerns about outcome measurement
While not directly related to outcome measurement, carers and consumers gave many examples of
perversities in the system that may be helped if the outcome and satisfaction measures chosen allow
for painting a full picture.
For example, carers describe their interactions with the system as like a seesaw: do too much for the
consumer and the case worker backs off, then the carer becomes overburdened, the consumer loses
support and grows unwell, and the cycle begins again. While the family, carer, or advocate must cope
after a hospital discharge, unlike the mainstream of the health system carers are not involved in
discharge planning, and there is often no discharge planning at all.
Carers who have had elderly relatives in the mainstream health system (or have had recent experience
of it themselves as consumers) are aware of the efforts - albeit imperfect - to involve the family at the
time of diagnosis or post-operatively, and in discharge planning. They wonder why the mental health
area is different. If an elderly person with diminished capacity, or a fit young adult with diminished
capacity for consent needs treatment due to illness or injury, the family is always consulted.
A parent with two young adult children described the extraordinary contrast between two episodes of
illness. One of her children was admitted to hospital with acute leukaemia: she was made welcome on
the ward, treated with kindness and respect, engaged in decisions about her child’s treatment, kept
informed of their progress, and coached about how to support them both in hospital and after
discharge. Some time later, her other child was admitted to a mental health unit with schizophrenia: in
this case her access was not only sharply restricted, but she felt she was regarded as a threat to his
well being and recovery, and excluded from any discussion of his aftercare.
Some carers noted that they were often used as jailers by the system, asked to monitor medications
and ensure compliance, but were accorded no place or influence in the system.
Consumers and carers voiced these caveats about outcome measurement:
•
Interpretation of results at aggregate or individual level must be aware that consumers and carers
in rural and remote areas will be very cautious about making a complaint or criticism or admitting
to not doing well, since the one service or a key provider may be the only available source of help.
They fear the impact a negative judgment or admitting to not doing well may have on the level
and type of treatment. References for housing and job search come from the treatment providers,
and they potentially can affect all areas of the person’s or family’s life. People said they were
afraid of being critical, or they would be scared to say they were not doing well in a survey for
fear of these consequences.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 34
•
•
•
•
•
Unprompted, most of the focus groups throughout the country came around to discussing the risk
to consumers if all a survey and its content did was to reinforce and document failure, or required
the person to document their losses. If the approach taken was largely about distress or symptoms,
a considerable investment would be needed in the system to provide debriefing, and to support to
people who were vulnerable at the time they filled out the form.
Outcome measures must allow consumers to opt in and opt out from time to time. If a carer
measure is introduced, both consumers and carers agree the consumer should be able to opt out of
the carer measure as they see fit.
A number of carers in States where consumer self-rated measures are already in place said they
did not believe there was true consent or opportunity to opt in or out, since they as carers had been
approached to help providers convince a consumer to fill it out, because there was pressure on the
provider to get people to complete it. In another instance one consumer reported that when they
had filled in the questionnaire it was given back to them by the nurse who asked them to do it
again as it did not paint a bad enough picture to gain admission.
The interpretation of identified outcomes must be made using the person as their own baseline
and remembering that different outcomes are important at different stages of an illness. Very
severe cases may show quite marked improvement, while those with chronic and long-term
condition may not show any improvement or deterioration. Without thoughtful analysis at the
aggregate system level interpretation of the performance of a treatment system or program will
depend on the capacity to take these issues into account.
Collection of these measures must be worth the cost of doing it.
Key issues in outcome measurement identified by consumers and carers
Across all the groups we consulted, there was a high level of consistency about what consumers and
carers regarded as key issues in outcome measurement:
•
•
•
•
•
•
•
•
What is the purpose of self-rated outcome measurement?
Who can or should use the information? It must be useful for whom?
The domains of outcome that should be measured are the domains consumers themselves use to
judge how well they are going (or for carers, how the people they are caring for are going).
The impact of the possibility of involuntary treatment on the truthfulness of consumers’ answers
to symptom and distress based surveys.
When should measures be applied in the course of treatment?
The effect of fluctuations in wellness on levels of insight and capacity to make valid judgments.
The need for a combination of methods –qualitative and quantitative
The need for a suite of measures across clinician, consumer, and carer/family/advocate that are
consistent to the extent possible, and sensible to allow for maximum validity and usefulness.
These are the views they expressed about each of these issues in turn:
The purpose of consumer self-rated outcome measures
During the initial consultations, jurisdictional officers had little to say about the purpose of consumer
self-rated outcome measures or what such a measure aimed to achieve. Nevertheless, we found that
the consumers and carers we consulted are clear about the range of purposes they believe such
measures should meet. (As we shall see, clinicians also had similar views, even if they had somewhat
different reasons and hopes about ways the data and information could be used).
Consumers and carers believe the outcome measure should be the sort of instrument where the person
is their own baseline and the setter of their own standards as to what constitutes a good outcome.
Outcomes should be interpreted in relation to the person’s own baseline, and to what can be achieved
within the financially and socially constrained situations of people with a mental illness.
Consumers and carers canvassed the following potential purposes for outcome measurement:
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 35
•
•
•
•
•
•
•
•
•
•
•
•
To build a resource for the consumer over time by building a catalogue of strengths and coping
skills and a record of wellness and achievements, no matter how small.
To assist individual case management.
To help clinicians improve their practice.
To help consumers interact better with case managers.
To empower consumers.
To allow the government’s evaluation of the system to be well grounded in the experience of
consumers.
To help determine the outcome of services at the unit or service level, and the outcome of the
system at the regional, state/territory, and national levels.
To encourage the system not to pay for things that make no difference to people’s quality of life.
To help evaluate the positive and negative impacts of policy.
To meet funding requirements for QA measures.
To close the gap between consumers and carers and clinicians – they regard this as far more
important than generating data for program or state and national planning or accountability.
To influence the way health care providers are trained and how their attitudes are formed.
Consumers hope that, if people really know about what happens to people between appointments,
and what they do to cope and survive in the face of their illness and its social, emotional and
spiritual impacts, there would be greater respect for them and for what they say, in contrast to
constant reinterpretation of everything to support a diagnosis, or even to discount the credibility
of what consumers say, so that nothing has to change.
Who can and or should use the information? It must be useful for whom?
The measure should be useful for consumers
First of all, the information from self-rated measures must be used by and useful to the person with
the mental illness, to keep track of themselves over time and to help them articulate ‘insight’. Several
consumers said they would like to fill out outcome forms both as a prompt to comply with needed
supports when they were well, and as a prompt to go back to see the case manager when danger
signals emerged.
The consumer view is that the measure must be useful and used and useable at the personal and
clinical levels, or it is not an ethical use of consumer information and time. Instruments must be useful
to consumers in helping to make sense of, cope with and recover from mental illness. Carers also
affirmed this view. The chief value of consumer self-rating is to the individual, and to discuss
successive results with their case manager or therapist.
Some consumers said they would like a client-held record. Some information would be only for the
consumer, and the form could be carbonised so that some and not other items are copied. The data
should be de- identified at aggregate level.
The measure should be useful for case managers
Consumers also hope the information will improve their capacity to be involved in decision making
about treatment and support and future planning. Clinicians and case managers can use the
information in charting progress and tailoring treatment, psychosocial rehabilitation, and health
maintenance. It should be useful for relapse prevention, and managing a relapse, and promoting early
intervention in a relapse episode.
Knowing how consumers are going in a holistic sense would cause the system to be more reflective
about the service and support needs of consumers, not just concentrated on what the system currently
has to offer, how it is staffed and how it relates to consumers and carers/families and advocates.
Outcome measures should generate data that could be used to change attitudes and reduce stigma.
Treatment may not have a direct causal relationship will all the outcomes achieved by consumers, but
it is still important for treatment providers to know about the full range of outcomes important to
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 36
them. Without that knowledge individual treatment review or planning or system level planning will
continue to be done in relation to symptoms and distress and not those extra treatment factors that can
exacerbate or ameliorate them. Without up to date shared information about the reality of a consumer
and carers day to day world and functioning the interpretation of symptom levels, and the assessment
of need for support and the content of the talking therapies will be narrow and self fulfilling. A
number of consumers noted that without knowing what was going on in a person’s life distress might
be interpreted as a return of symptoms rather than a normal reaction to the inevitable losses in life or
the frustrations of a marginalised existence. The system they say tends to pathologies every reaction
because so often no one asks if the dog died or the hostel atmosphere is anxiety provoking.
The measure should be useful at service level
The data can be used by the system’s managers at various levels, from service through regional, state
and territory and national levels, for monitoring the effectiveness of mental health services and for
quality assurance.
Most but not all consumers and carers interviewed recognised the distinction between an outcome
measure and a measure of satisfaction with how services are delivered. Nevertheless, they said, if you
want to know how the system is working from the consumers’ point of view, you need to ask them
about satisfaction as well as outcome (but not in the same survey).
Several focus groups made the point that coverage by case managers varied highly across services in
all jurisdictions, and sometimes even within services. The question for outcome measurement is: if the
psychiatrist is not going to use the information, and there is no case manager, who will look at it?
There was some scepticism among consumers and carers whether psychiatry was interested in broader
health and well being outcomes, and whether psychiatrists’ time constraints and skill base limited
their capacity to invest in these areas.
Outcome data should be useful for research purposes so that the system better understands the true
course and impact of a mental illness or the different forms it takes, rather than merely how it presents
to the treatment system. Some consumers say that, if the aggregate data are used for research or
evaluation purposes, consumers and carers - not just researchers or clinicians - must interpret the
information.
Consumers believe an important aspect of outcome evaluation is to provide the system with timely
data on the unintended negative consequences of policy changes. Even policy decisions beyond the
mental health system could have a critical impact on outcome. A husband and wife who came to a
focus group in a regional area said the husband’s regime required him to go to their GP for medication
twice every week; but now that their doctor, and most of the GPs in their area, were abandoning bulk
billing, they would not be able to afford to pay repeated gap fees out of their pensions, and did not
know how they could continue to comply with what the system was telling them to do.
The domains of outcome that are important to consumers
In all locations, the heart of the discussion with consumers concerned the domains of outcome they
hoped a self-rated measure would cover. The issue that recurred more than any other was how they
coped with life – both in general, and in dealing with the effects of illness in particular - and their
capacity for resilience in dealing with personal relationships and day-by-day functioning. Here are
just some of the things the consumers said about these domains:
Coping and resilience
The measure should ask the consumer about:
•
Strengths and resilience, coping skills and strategies
•
Feeling confident to act on danger signals in a positive way
•
Knowing what to do when you feel you are getting ill
•
Help-seeking behaviour, being positively inclined towards seeking help
•
Extent of return to your own baseline
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 37
•
•
•
Visualising a time in your life when you were OK: ‘How do you feel today in relation to A, B
and C etc?’
No longer existing in victim mode
“An instrument should have a review section, looking at your last survey: what has changed
since, what has helped and hindered, and what new things might help in the future.”
Hopefulness
The measure should ask the consumer about:
•
Optimism, hope for the future
•
Sense of connectedness or isolation
•
Belief about the likely trajectory of your life
•
“How you fit into the picture”
Empowerment
The measure should ask the consumer about:
•
Social support from other people in similar circumstances
•
Feeling of control over your own life.
•
A sense of purpose and value to the community
•
Personal goal attainment - are you meeting your own goals?
•
Improved self efficacy, belief that you can do things for yourself
•
How often you are doing what you consider to be your own enjoyable pursuits
Personal relationships
The measure should ask the consumer about:
•
Your perception of the impact of your illness on others
•
Other people’s reactions to you – good, bad, and indifferent
•
Social and behavioral outcomes
•
The strength of your social network
•
Socialising with family, and with peers
•
Whether you are remaining sociable and not isolating yourself
•
Concern for others, participation, and leadership
•
Whether people believe what you say
•
Your credibility and ability to influence others, or by contrast, your experience of
discrimination and loss of credibility. (“How much do you believe that other people’s
knowledge of your illness has affected how you have been treated?”)
•
How you relate to family or carers and advocates
•
How you think others close to you think you are going (housemate, sibling, parent, spouse)
•
How you are relating to health professionals, and they to you. “Is there two-way, mutual
respect?”
•
How you relate to the professionals in making decisions about treatment and care (“The more
the participation, the healthier the relationship. Sometimes psychiatrists are not healthy
enough to allow consumer participation in decision making, even when the person is well.”)
•
Sexuality and relationships, the health of the sexual part of your life
•
Relationships with pets
Functioning and well being
The measure should ask the consumer about:
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 38
•
•
•
•
•
•
•
•
•
•
•
•
•
•
How you feel you are doing in relation to your various roles as a parent, spouse, worker,
housemate etc
Being able to do the things you did before - normal things like reading, studying
Activities of daily living, like cooking
Functioning at home and in general social situations
Ability to sustain your own standard of living
Keeping commitments and appointments
Sleeping patterns - too much or too little
Where you are living
How is the money? Are you able to meet financial commitments?
Employment – paid and unpaid (this is “a big area of outcome”)
Participation in education
Thinking of a time when you were well and functioning at your best. How are you now in
relation to your physical health, feelings, social connections, employment etc at that time?
Balanced recreation and leisure.
Spirituality –participation in religious activity (several consumers emphasised the need to
cover spirituality or religious activity, despite concerns about how mental health staff might
view religious beliefs).
Extra-treatment factors that influence outcome
The measure should ask the consumer about:
•
What things are happening in your life that affect how you are feeling?
•
What are the intervening events or things that affect how well you are doing?
•
What else other than treatment is going on in your life that may affect outcome?
Symptoms of illness
The measure should ask the consumer about:
•
Do you understand signal, signs of relapse?
•
The onslaught of unconnected thoughts
•
Mood state fluctuations
•
Finding fault with everything and everyone
•
Domineering attitudes, overly assertive or aggressive in responses to others
•
Level of symptom control
•
Social withdrawal
Physical health
The measure should ask the consumer about:
•
The impact of medication (“It can be worse than the illness.” “The side effects can be so
debilitating.” “Compliance doesn’t mean you feel well.”)
•
Nutrition and diet, caffeine intake
•
Drug and alcohol use, and drug and alcohol related problems
•
Physical health
•
Tiredness
•
Personal hygiene
•
Minimal hospitalisations
Health literacy –knowing how to negotiate the health and welfare system
The measure should ask the consumer about:
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 39
•
•
•
•
•
•
Knowledge of and use of support services
If things go wrong, do you know what to do and where to go, and that help is accessible?
Partnership in care - how empowered you feel to discuss care and medications and needed
support
Carers, consumers and doctors working together
Managing inpatient to outpatient transitions in the system
Involvement in decision-making
The impact of the continuing possibility of involuntary treatment
An important and consistent message from consumers and carers was that they would be very
cautious about using measures that concentrate on symptom severity or distress, or would always give
positive ratings in order to avoid the risk that a clinician may use the results as the basis or as part of
the basis to impose involuntary treatment orders or to increase medication.
People said they would be anxious about how they would answer any questions about parenting,
because they were constantly fearful of losing their kids. A young mother brought her baby to a focus
group. She had suffered severe postnatal depression. She told the group: “If they asked me how often
I had suicidal thoughts, the real answer would be ‘about 98% of the time’, but I wouldn’t tell them
that - they might put me in hospital away from my baby, or give me medications that would mean I
couldn’t go on breastfeeding.” She said: “I’d be happy to tell them about all the ways I keep on
functioning and being a good mother in spite of all those thoughts.”
The consistency of this unprompted response across jurisdictions indicates challenges for
measurement of outcome in this population that are not found in outcome measurement in mainstream
areas. Interpretation needs to be aware of the ‘socially desirable’ response, ‘What will get me the best
treatment or avoid mistreatment?’
This response has implications for the culture of mental health systems. It also has implications for
how clinicians present the survey to consumers and carers, and the need for consumer education about
the benefits of outcome measurement and how it will be used.
The impact of fluctuations in insight and capacity
Both consumers and carers acknowledge that from time to time for all consumers’ insight and
capacity fluctuate owing to the chronic relapsing nature of most mental illness. They acknowledge the
effect this will have on the ratings and reports given by consumers. This has implications for
measurement - timing of the application of the measure, and the issue of who can provide information
when the consumer cannot.
They believe this reality also points to the need for a carer/family/advocate outcome measure, to allow
for these fluctuations in capacity and insight, and to give a full picture of the consumer’s functioning.
This too must be with the consumer’s consent, and the right to opt out of both self and other rating
from time to time, as they see fit.
One consumer asked how you could find a test for someone like him that would have any test re-test
reliability when he has rapid cycling bipolar illness and would give different answers to the distress
related survey from one hour to the next. He believes that measuring coping strategies would be more
sensitive to real change
The system must accept the limitations some people have, and accept that for some people, limited
goals of rehabilitation will be all that is possible.
“I know now after a number of years of living with my condition that I must maintain a very tight
and closed social life… basically my son, my parents and that’s about it. Any time I try to broaden
that out and socialise, I get ill again. ... To others, that may not be an acceptable, positive or good
outcome. For me it’s the only possible outcome”.
There was considerable discussion in several groups about the impact of the use of illicit drugs on
capacity or lack of it.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 40
The need for a mixture of methods
All consumers and carers find the use of ‘tick flick’ check box rating scales too limiting. They believe
that without some capacity to provide text comment, the results will not be interpretable
Almost universally, they are also keen to see a place on all surveys that allows them to say whether
something has happened in their lives in the last five days (or 30 days, or whatever time frame the
survey uses) that has affected their self-ratings - for example, whether their rating of depression or
anxiety is the result of a death in the family, conflict in a hostel, exposure to violence. Consumers
persistently report that mental health professionals fail to ask these sorts of questions in many
instances, and pathologise normal responses to difficult life events.
Many consumers and carers would like to see routine focus groups held in clinics to complement their
reporting of outcome. A minority also feels that people could be asked to keep a diary from time to
time, rather than simply fill out retrospective forms with only ticks or circles.
Many consumers and carers are aware of the difference having a case manager makes to outcomes.
The system must ensure that both people who have a case manager and those who do not complete the
measures: if the measurement process cannot track that difference, it loses valuable information for
evaluation purposes.
The need for a set of complementary measures
Both consumers and carers believe there is a need for ways to validate self-reports. “You always need
at least two of consumer / clinician / carer to be sure of getting a valid read of the situation - any one
on its own is not good enough.”
Consumers and carers are pleased that efforts are being made to include the consumer view, but
believe that the true picture will emerge only by looking at the consumer-rated outcomes alongside
the clinician and family/carer proxy ratings.
This is the case for every consumer, and for all conditions and levels of severity, but is particularly the
case for the most ill who have rare or no symptom free periods. For people with unremitting severe
illness, it may not ever be possible to get a valid consumer self-rating of outcome. “A person may be
happy with delusions and or mania, but everyone around them is not.” A carer said,” My sister is a
severe paranoid schizophrenic - she is never well, but she thinks she is.” One consumer with a bipolar
disorder asked, “How will the system know if the very positive ratings of the person with mania are a
good or a bad thing?”
Identifying what is ‘normal’
Another reason for parallel clinician, consumer and carer measures is that different ways of working
with people whose behaviour is disturbed lead to different ways of viewing what is ‘normal’.
Psychiatrists, or carers, or people with a mental illness may become used to things that those outside
that subculture would find odd or unacceptable. Providers do not often ask carers or families or
relatives what is normal in their context. One group of carers discussed the extent to which health
professionals - especially psychiatrists - were socially distant from their clients - many of whom had
been downwardly mobile even if they started in the same social class - and their views of psychosocial outcome were often incongruent with how the person or the family saw things.
An antidote to pathologising
In all the focus groups with consumers and carers, there were repeated comments that psychiatrists
and nurses tended to pathologise everything a consumer or carer said. Providers they said often used
the jargon and concepts of analytic psychiatry to explain away the importance of things they did not
want to hear.
For example, concerned husbands advocating for their wives were defined as intrusive and
manipulative, and part of the problem. Even if there could sometimes be some truth in that analysis,
labeling and excluding carers and family members did not help progress. They surmised that in many
cases this may also be a mechanism service providers used to cope with work overload, or a belief
that including family or carers would be too time-consuming.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 41
Normal reactions to the constraints of living in a ward setting were also pathologised, especially for
young adults. Similarly, religious expression was often seen as a form of pathology, especially if the
person did not come from a form of religion known to the majority of white middle- or working-class
health professionals.
One mother described her son who, before he became ill, was an excellent high school student and an
ardent reader of the classics - in particular Goethe, whom he could quote at length. In hospital, he
learnt not to talk about his interest in the classics nor his knowledge of them, because it was fed back
to him as an indication that he was not getting well – displaying grandiosity and using complex
language. His mother said that, just because he was possibly brighter and better educated than most
psych nurses and many doctors, it did not mean he was not getting well. In fact, from her experience
the truth was that as he began to show an interest in his earlier intellectual interests, this was the sign
of his turning the corner. While he may have been displaying his intellect as a way of seeking
approval, the need for approval and positive feedback is nowhere felt more acutely than in a
psychiatric hospital, which often forgets to see consumers as people not illnesses, and also forgets to
ask what they did before they were old or ill, and what is important to them and their sense of self.
One mother said that, if she had been listened to about the effect of the medication on her child, what
was ultimately recognised as a medication error would have been picked up earlier, and the outcome
for her son would have been improved.
Another mother described a son who never watched television, even as a child. She described a time
when she asked why her son appeared to be very highly medicated. A nurse told her that he had not
been participating in ward activities and socialising with other consumers, but since they had
increased his medication he was now watching television with the others, and this was taken as the
indication that his behaviour was now ‘more appropriate’.
Most carers say that outcomes will be compromised if they do not know what a caseworker is telling
the consumer to do, and do not share information about cues for relapse, and relapse prevention is not
planned in conjunction with all those who have the possibility of seeing the cues or intervening early.
Some carers acknowledged that they had very unrealistic expectations of treatment at first, and after
20-25 years of experience of the person, the illness, and the treatment system, and the limits inherent
in each element of that mixture, they had markedly lower expectations. How the system is performing
may look successful from clinicians’ and consumers’ ratings, they say, but the carers may be dropping
because the system relies on them so heavily.
The system still appears to be struggling with how to incorporate families, carers, and advocates. With
deinstitutionalisation, the system has come to rely on families and carers more and more. There is a
significant level of concern about carer distress and burden in the community. An important part of
outcome evaluation is to have timely data on the unintended negative consequences of policy and
program shifts.
Many carers commented on the continuous state of loss and grief, both for them and the person with
the mental illness, as they come to grips with the probability that the person will never have a
sustainable quality relationship that could lead to grandchildren. “Watching your child come to grips
with that reality over the years, and then adjusting to it as a parent, is heartbreaking.” The system
seems to underrate this aspect of the experience of mental illness, and does not provide support for it.
One carer, supported by other members of the focus group, commented that there were some things
consumers would tell only their parents or those close to them. A mother of a boy with schizophrenia
told the story how her son one day told her proudly that he had met a girl who loved him, only to say
a few days later, “Mum, you know that girl I said loved me? She was only in my head”. This sadness
was never shared with the boy’s psychiatrist, who seemed only interested in his symptoms and
medication.
Some consumers qualified these carer views with the caution there could be a gap between the
expectations of family and the consumers’ own perspectives of how well they were doing. All the
carers acknowledged the rights of adult children to privacy, but also believed that the system should
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 42
encourage the consumer to include significant people in their treatment, outcome measurement, and
monitoring of how things are going.
The focus groups agreed that the ideal would be a set of complementary measures with a core set of
variables in common, and variables for each stakeholder group - things, for example, that only the
person in the skin can rate; or only the people close to the consumer’s day to day functioning can rate;
or only the clinician’s technical skill and expertise can rate.
Consumer advice about implementation
When the measure should be applied
Consumers said it was not clear when self-rated measure was best applied, but it was clear when it
should not be applied - not on admission, not during an acute episode - and the protocols as to when to
apply it should be sensitive to the state the person is in.
Some thought it should not be done after a recent acute episode when the person was hospitalised; but
others said that, if there could be an instrument that would make the system more likely to monitor
people carefully at transition points, that would be a good thing. For example, it could be used
regularly for the first few weeks after an inpatient stay, just as the mainstream does post-operatively
and in discharge planning to prevent readmissions. One consumer said, “Don’t do it immediately
before discharge – we’ll say whatever we think will get us out”. A more likely time was early post
discharge, and then at regular intervals in conjunction with case management reviews.
The process must measure people who have exited the system, not just those chronically in it.
Consumers and carers say that, whatever the instrument, permission must be given, and
encouragement and practical support for some people to complete the surveys. Some people may be
able to fill it out at some times and not at others. Sometimes it is not safe to have the person do it
alone. Consumers discussed the effect of personal attributes such as literacy, ability to concentrate,
motivation to complete the survey, and fear of failure on how they answered the form.
People filling out the survey should finish it feeling better, and it must not damage the consumer.
How to use, word, and interpret an instrument
Consumers make the following comments about practical issues of applying a measure.
The wording needs need to be careful not to imply ‘should’ and ‘ought’, and should leave judgment
out. If the instrument focuses on symptoms or even functioning, this could be highly variable, and
therefore difficult to interpret in the aggregate across conditions. If the focus is on coping skills and
self-rated strengths and personal goal attainment, then it will be easier to aggregate across conditions
even if they rehabilitate differently in symptoms or functioning.
The questionnaires should be readable, user friendly, and able to done assisted or alone. Most
consumers prefer a brief measure that would take no more than 10 minutes to complete, though some
consumers say they are happy to do a long one because the short one seems so vulnerable to shortterm distress, and may not be a true picture of how they are really feeling.
A few suggest the use of computer based interactive measures that the person can tailor to him- or
herself. Other suggestions include including an opportunity to say what else is going on in one’s life,
and to do a drawing or write a poem to illustrate how you feel.
Literacy will be a major issue for many people, and there is a need to develop multilingual and forms
free of cultural biases.
There may be a need to do some training of consumers who have not had much association with being
asked to rate things. For example, people who are not studying, or left school early, or do not have a
phone or a consistent address or are homeless and have not been surveyed, the whole concept of
rating abstract ideas about oneself may be so alien as to be impossible without guidance.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 43
Clinicians’ views of purpose, use, and method
In the course of the project, we interviewed or received written contributions from 27 clinicians,
chiefly psychiatrists and psychologists in both public and private sector mental health services,
including seven university researchers, most of whom also maintain a clinical practice.
The purpose of consumer self-rated outcome measures
We asked them first to address the purpose of consumer self-rated outcome measures. In summary,
this is what they said:
The measure should be useful for consumers
Clinicians said it was vitally important for both consumers and their case managers to have ready
feedback. Consumers should have ready access to information about themselves, and “should be able
to take the results with them, and say where the data ends up” Issues of confidentiality and privacy
were voiced, and concerns about these issues had been expressed by consumers when they were
completing self-report measures.
The measure should be useful for case managers
Regular routine feedback was very important for both clinicians and consumers. Information should
be used to measure change and inform treatment. Information must be available for use in these ways
to ensure staff commitment to the process.
The measure should be useful in inpatient settings
Information from consumers helps identify admission goals, review progress towards these goals
during the admission, and develop discharge plans.
The measure should be useful for team meetings
Information from consumers is discussed in team meetings and provides a platform for peer review,
feeds into the case manager’s decision making, and validates or modifies decisions.
The measure should be useful at unit level
Aggregated consumer outcomes may be useful to determine an average base level of progress for
measuring treatment outcomes for a particular unit, and to allow mental health units to view
consumers in the setting of information supplied from other sources.
Uses of the information
Clinicians said consumer self-rated measures could be used for initial and continuing assessments.
Information provided by consumers provides insight into their current situation (including current
thought processes, perspectives, and level of functioning), identifies relevant areas of concern that
form the focus of intervention, and outlines areas they want to change (that is, goal setting).
It can be used for treatment planning, encouraging more consumer-focussed activities and helping
consumers be more goal directed. Individual level data should therefore be available to the consumer
and his or her case manager.
It can be used for dialogue and rapport building: In discussing consumer rated instruments and the
items in them, clinicians and consumers can open up a dialogue that previously they may have found
difficult to broach or discuss. Self-report measures give consumers the opportunity to state their own
assessments that may vary from the clinicians’. It also facilitates the validation process for consumers.
Consumer provided information can be used to monitor individual progress during treatment, and
thus influence planning - for example, when deciding if the focus needs to be placed on a particular
aspect of therapy (medication, group therapy work, management of symptoms, etc.) To monitor
progress, consumers need to provide information at different points over the course of their contact
with the service.
Consumer self-reports are used for screening to determine suitability for program attendance.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 44
Consumer self-report measures can be used in group work to provide baseline measures, facilitate
discussion of planning, and provide feedback on change. Feedback from consumers is very useful in
developing group programs and determining needs for staff training.
Results from consumer feedback are invaluable in evaluating the effectiveness of services. Many
consumers are eager to provide feedback to services. In terms of quality improvement, there are very
few other outcome measures if consumer feedback is not gathered. “It also fosters respect in
clinicians for consumers’ opinions.”
Consumer outcome measures are usefulness in planning meetings where the future directions of a
service are discussed.
Short-term intensive rehabilitation and longer-term rehabilitation both use consumer self-report
measures, especially in identifying goals and reviewing progress over time.
Data from consumer self-reports may be included in assessment reports, progress reports and
discharge summaries.
When the measure should be applied
Clinicians say that self-report measures should be administered at significant points throughout the
consumer’s contact with a service – in particular, on admission, at three monthly review, and on
discharge. When self-report measures are included at these stages, they become readily integrated into
current processes. The current NOCC protocol was developed in collaboration with clinicians, and
already reflects these views.
The impact of fluctuations in insight and capacity
Clinicians varied in their opinions about the influence of consumers’ varying levels of insight and
capacity. Some think “patients’ level of functioning influences their capacity to participate in this
process” while others say that “the majority of consumers, if asked, are able to complete self-report
forms (and are willing to do so). Only a small number of consumers would be considered to be too
unwell to complete self report measures – for example, those lacking the required concentration, too
agitated, delirious, or medically unwell”.
There is therefore an argument for use of information provided by significant others in the consumer’s
life (partners, family, carers etc). One clinician commented that “sometimes by virtue of the
consumer’s condition the perceptions of significant others may be the most accessible gauge of
consumer progress. In other situations it could usefully add to the picture of consumer progress. This
could happen in many instances with consumer support that is, many consumers would be accepting
of families’ contributing to outcome measures”.
The need for a combination of methods
Clinicians and some academics said a combination of outcome measures was necessary to overcome
errors in self-reporting, cognitive distortions, and social desirability distortions, and the points of
agreement and disagreement among the measures were important. One clinician, however, was not
sure that comparable measures were the answer. He commented that: “a self-report is potentially
distorted by the human capacity for denial of reality and the need to report to the satisfaction of
others. I am cautious about spending too much time on self-reports, which should be balanced by
observation of ‘voting with one’s feet’ – that is, objective behavioural cues”.
The need for a set of complementary measures
Several clinicians said a set of complementary measures catered for the differing perspectives of
consumers, clinicians, and significant others. Moreover, different stakeholders needed different sorts
of information. Some recognised a need for a combination of measures including one supplied by “an
objective party” and a self-report (“one without the other is inadequate”). One psychologist in a
private hospital said an adequate result would be a refined HoNOS as the objective measure, and
(with adequate skill training) the SF-14 if Australian weightings could be assigned.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 45
A single outcome indicator cannot suit all purposes and all populations, but multiple measures would
reduce the ability to compare outcomes across parts of the system or between different intervention
groups. It would also lead to the choice of a very low level indicator to cover the largest number of
subjects. Complementary outcome measures with some common subscales could be used, and a single
measure could be chosen from a number of classes of measures - self-rated scales, clinician rated
scales, fully structured interviews, and semi structured diagnostic interviews by clinicians.
Features of the instrument
Structure: Clinicians and academic respondents made these suggestions:
•
•
•
•
•
•
A standard spiel to introduce the measure
Open-ended questions to obtain information from the consumer’s perspective
Some reverse-direction scoring to identify mindless completion
Yes/No responses
A broad rating scale (1 to 10 instead of 1 to 5) with clear labels
Clear protocols and procedures that cannot be misinterpreted through vagueness
Simplicity: The measure should be easy to read, follow, and complete. (“Patients will not complete it
if it’s too hard.”)
Length: Not overly long.
Language:
•
•
•
Avoid use of jargon
Questions should not be repetitive “so that consumers don’t conclude that they are answering the
same question twice”
Questions should not be too generalised as they will then have little meaning to consumers
Coverage of different domains:
•
•
•
The measures should be divided into subscales, so that you can assess a range of domains
Cover specific issues so that responses can be used to implement change
Items on the questionnaire should relate to interventions
Relevance: Must be seen as relevant to consumers to get compliance in completing it.
Sub-populations:
•
•
•
Measures of self-report are needed for older people.
We need to take into account people with sight impairment, impairment of the ability to write,
hear, see or otherwise communicate (for example by making forms brief (les than 10 items), and
available in large print).
We need to take cultural diversity into account.
Time: Define an upper limit for completion time (especially if the worker will be required to stay with
consumer to assist)
Norms: Australian standardised norms are lacking for many measures. It should be a well-recognised
and widely benchmarked tool
Validity and representativeness:
•
•
•
There is need for a valid and standardised measure in order to be able to interpret shifts in
“points” on a scale.
Any system that promotes or allows the views of one consumer group’s experience to dictate to
another is not appropriate.
Consumers who are psychotic are less likely to complete questionnaires, and those with
personality or affective disorders will.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 46
•
•
The diagnostic group impact in the completion rates needs to be investigated and allowed for if
data is to be interpreted above clinical level. (“As any consumer will tell you, there are marked
differences in illness presentation and treatments.”)
Use of this information in aggregated form to make comparisons across settings and/or to measure
service performance is inappropriate, as there are too many variables apart from systems and
service – you cannot measure outcomes without measuring inputs. They are best used as measures
of various aspects of health status
Processing of data: Information should be collected in such a way that it can be processed
immediately. This would be more beneficial than having results ‘sent away’, analysed then returned
which would be less immediate and have less impact.
Individual feedback: Any instrument should be constructed in such a way that it can accommodate the
following needs for feedback:
•
Questionnaires and the feedback derived from them should focus on the positives and positive
gains so as to instil hope or belief in gains.
•
Consumers should be given a copy of the completed assessment to help form part of their own
consumer health record.
•
Feedback should be able to be provided in written and verbal format.
•
Feedback should be able to be readily interpreted by the consumer.
•
Feedback should be immediate, easily understood, and appropriate to the consumer’s experience
of his/her illness
Presentation of information to consumers:
•
•
•
Too many numbers will not make life easy for interpretation and use with consumers.
Colourful graphical representation showing change and comparison is preferable. A visual result
such as this gives a prompt clear picture. Produce a poster each year and put it up in units so all
consumers can see that their opinions count.
General findings and trends could be announced to consumer advocates, regularly in community
meetings, and posted on communication boards.
The domains of outcome clinicians think are important
Clinicians’ comments on the relevant domains of outcome for consumer self-rating matched
consumers’ own concerns to a marked degree, though with a stronger emphasis on observable features
of treatment, and less explicit emphasis on coping mechanisms. They listed these domains:
Life circumstances
•
•
•
•
Stable and adequate accommodation
The financial situation: financial concerns reduce recovery, but it helps when income is
reliable and finances are managed well
Involvement in education and training
Employment, increased involvement in work, loss of employment, problems at work,
volunteer work
Roles and activities
•
•
•
•
•
•
Routines and changes in routines
Satisfactory functioning in significant life roles
Level of functioning in daily activities
Leisure activities increased or decreased - projects and hobbies and recreation
Decreased or increased activity levels (eg staying in bed)
Being productive, ability to cope with daily activities
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 47
Independence
•
•
Able to make decisions, to pay rent, etc, and maintain self-care
Any changes in appearance; dietary habits; weight
Behaviour
•
•
•
•
Appropriate or inappropriate use of prescribed medications
Misuse of alcohol and other drugs
Sleep patterns. self-harming behaviour
Inappropriate regressive or aggressive behaviour
Use of services
•
•
Medical and support services appropriately accessed
Increased contact with multiple health practitioners
Events
•
•
•
Contact with case manager increased or reduced by the consumer; missed or rescheduled
appointments, increased or reduced contact by family
Medication regime stable or unstable
Hospital admissions: frequency and length of stay
Interpersonal relationships
•
•
•
•
•
•
•
Communication
Ability to report distressing feelings; honesty in communication; expressing thoughts and
feelings; communication with significant others
“Connectedness” with family and friends
Capacity for confidences (as they require them); the opportunity to nurture some one, something,
some project; contributing to the well being of some person or thing; expansion of the social
network
Relationships with significant others: partners, family and friends
The strength of the social support network, the level of social interaction level and isolation
Frequency of seeking reassurance from others
Mental state
•
•
•
•
•
•
Signs and symptoms becoming worse, experiencing negative emotions, anxiety, panic, despair,
sadness; deterioration in mood, level of dangers
Signs and symptoms reduced or managed well
Emotional stability: psychological well being, a sense of hope, a view to the future, purpose and
vision
A sense of control of circumstances
Stress managed appropriately
Positive emotions; spontaneity; appropriate humour; vitality
Illness awareness and insight
•
Symptoms recognised (“though symptoms are not the dominant feature of the consumer’s life
from the consumer’s standpoint”)
Expectations of others
•
•
•
Reports by friends/family of improvement or deterioration
Reports by GP
Unsolicited feedback from other people and agencies
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 48
•
•
•
Consumer self-evaluations, positive or negative
Sense of self-esteem; self-image; feeling productive; feelings of worthless and low selfesteem.
Suicidal ideation
Consumer expectations of self
•
•
Realistic expectations of level of mood
Acceptance of not having to feel ‘good’ all the time; future goal setting becomes appropriate
Consumer expectations of others
•
Realistic or unrealistic
Medical / health
•
•
•
Changes in physical health
Pain
Presence or absence of medication side-effects
Cognitive functioning
•
•
•
Changes in cognitive functioning
Ability to plan for the future and set goals
Distorted or realistic thinking; blame; negative thoughts; misinterpretation of others’ input
The domains of information about outcome where consumers are the best or only source
How they feel - labelling emotions, mood:
•
Thoughts, ideation, suicidal ideation, reflections
•
Hallucinations
•
Delusions
•
Sleep
•
Diet
•
Exercise
•
Substance use
Functioning:
•
Self-evaluation of health status
•
Self-evaluation of ability to cope (eg with work, at home etc)
•
Self-evaluation of level of distress
•
Self-esteem
•
Consumer perceptions of their relationships with others (family, friends, professionals):
•
Consumer goals of treatment
•
Quality of life measures; satisfaction with life
The sorts of ratings that can be validly obtained from consumers
•
•
•
•
•
•
•
Functional assessment of level of activity
Amount of socialisation
Achievement of goals
Sense of pleasure
Routine: work duties, interests, and periods of rest
Identification of expectations and the extent they are achieved
Level of activity over time
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 49
•
•
•
•
•
Level of wellness over time
Consumer satisfaction
Perceived contentment with life
Satisfaction with relationships, including with treating professionals
Symptom ratings
Process issues identified by clinicians
Clinicians and academic respondents believe consumers and carers should be involved from the outset
in introducing self-rated outcome measures – for example, having input into the development of
protocols.
Information about the instruments and the uses of information should be presented to consumers
through information sheets or brochures provided to all consumers, and possibly by inclusion in
service information booklets, and rights and responsibilities brochures.
Consumer participation should not be mandatory, and the process should be non- threatening
Introducing the instrument: Introduction of any new instruments should be conducted in a slow and
progressive manner, and as simply as possible.
If possible, explain the measures to consumers in person rather than by letter or telephone. Allow time
for discussion, and let them know whom to contact if they require any assistance (supply the name
and contact details of the appropriate person).
Feedback: Successful implementation requires feedback. Appropriate processes for feedback to
consumers and staff are important to encourage compliance.
Culture: Some staff find the idea of eliciting consumer feedback threatening, as if it is a performance
appraisal - they feel a little persecuted when consumer feedback measures are first introduced. This
issue needs to be addressed in clear definitions of purpose and use.
Different disciplines have different priorities, backgrounds, and varying commitment to the use of
outcome measures. There is a very active paternalistic view among some clinicians - especially in
some medical areas and older clinicians - that it is pointless asking acutely psychiatrically unwell
consumers for any feedback
The use of consumer provided information will represent a significant shift for many practitioners.
Some staff do not regard the collection of this data as consumer participation. In service areas where
participation is embraced and promoted in treatment planning, goal setting, and actually having
consumers on staff, this attitude is voiced less often. If staff can observe the use of consumer feedback
working in practice, they are less cynical.
Implementation of outcome tools does involve culture change in a system. Successful implementation
requires managerial understanding and support. A critical ingredient in acceptance of such measures
is strong endorsement from relevant State Government officials.
This issue is a central concern for some of the most experienced clinicians we talked with. One of
them said that to engage consumers in completing these instruments and for staff then not to use them
to enhance the quality of treatment would be “criminal”. The introduction of a suite of outcome
measures was a challenge to the entire system and its workforce. Without a commitment to use the
results, “outcomes measures will be viewed by clinicians as a bureaucratic and clinically irrelevant
task”. It was crucial to engage management in reviewing case management systems and continuity of
care, redesigning consumer record systems so that outcome measures were an essential ingredient in
individual treatment plans, and developing technical support. The effectiveness of the initiative also
depended on clinicians’ knowledge and advice, and their agreement and enthusiasm to encourage
consumers to use outcome measures.
Resources: Successful implementation requires adequate resources. Staff commitment to an activity
for which no additional resources are allocated may be hard to maintain. The collection, use and
analysis of measures are time and energy demanding. Acute care settings are busy contexts and
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 50
completing or administering these forms may not be viewed as a priority. Resources are needed for
training time, follow-up refresher activities, orientation and support for clinicians, and then it will
work.
Time: Time is required to discuss and explain measures to consumers, and to input, discuss and utilise
the data. (In the minds of some clinical respondents, time spent with a person filling out a
questionnaire may detract from the impact of human interaction itself.)
Training of staff: Successful implementation of any new instrument requires that staff support its
introduction and understand it well. The introduction of any new instrument should therefore be
accompanied by appropriate training.
A critical ingredient in the successful introduction of a new instrument is a comprehensive training
package that addresses attitudinal issues (being more guided by consumer perceptions), as well as
practical issues around implementation, including practical constraints to routine use. Training needs
to reinforce the importance and value of outcome assessment, the appropriate ways to collect
information and administer forms to consumers. Details of the training will be determined by the
measures used. Elements of the training may include how to explain measures to consumers,
educating staff on the benefits of using self-report measures, and how to input and use the data.
Training is a continuous issue because of staff turnover, and the use of agency and casual staff.
Agency staff do not typically collect outcome data in some mental health units.
The implications of all these measures will involve efforts to ensure that the quality of the process is
the same or better five years after introduction. This means retraining, inter-rater reliability studies,
orientation for new staff, and alarm mechanisms in case the measures start producing positive harms.
If clinicians are not trained in how to discuss the results with consumers, then some of them will, and
some of them won't. (“Funders and academics tend to assume that clinicians are born knowing all of
these skills and have the innate ability to interpret and teach when, like most other people, they need
to learn.”) Good, well-resourced training will promote and effective and standardised approach to
interpretation and discussion
Cost: The system needs to take into account the cost involved in using some measures.
Uniformity across jurisdictions: One respondent said:
“The actual instruments are important. Individual states and territories have chosen different
consumer rated instruments with separate backgrounds. This will never work. We are making
things more complex and difficult for ourselves by using instruments that weren't developed to be
used together. It would be better to agree on a basic set of subscales for consumer and clinician
rated instruments and develop or adapt instruments to them.”
Selection of instruments: New instruments should not be introduced on top of currently used
instruments. The system should refine some effective existing instrument rather than re-invent the
wheel, and there would be nothing worse than a grab bag of a global scale plus a bunch of other
scales. In summary, clinicians said any new instrument must give consumers and clinicians
information that can be readily fed back to consumers and clinicians, addresses domains relevant to
treatment, is transparent, relevant, meaningful, and easily interpreted, can be presented in both verbal
and visual formats, and does not represent an additional paper-work burden.
Domains of importance compared
There is a close similarity between what Australian consumers identified to us as the outcome
domains of greatest importance to them, and reports of similar investigations in the international
literature. There is a remarkable degree of consistency between what consumers have told researchers
overseas about the outcomes they regard as crucial, and what Australian consumers have said in our
discussions with them throughout the country. The responses we obtained from clinicians and
academics understandably concentrate more on issues of clinical presentation, but in several important
respects also reflect the priorities expressed by consumers.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 51
Determinants of outcome
This clarity of direction has powerful implications not only for outcome measurement, but also for the
culture, workforce, and goals of the mental health system as a whole.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 52
Part 3: A framework for evaluation
In this project, we have been asked to review the literature on outcome measurement in mental health
treatment and to consult with consumers and carers and clinicians about what aspects of outcome are
important to them, in order to develop a framework for evaluating existing and emerging consumer
self-rated outcome measures.
The objective to develop an evaluation framework for consumer self-rated outcome measure is made
more difficult by the lack of an overarching evaluation framework for the mental health treatment
system - a framework built on evidence about the determinants of outcome in a population with
mental health problems, the characteristics of the treatment system, and its focus and methods. The
literature and these consultations both call for models of outcome that address the complex interaction
among characteristics the individual brings to the treatment encounter, characteristics of the treatment
experience, the relationship between the consumer and therapists and case managers, and resources
and other factors outside or following treatment.
The implications of purpose for choice of measures
Competing purposes have implications for the choice and evaluation of particular instruments. In the
absence of an evidence-based evaluation framework for the treatment system, assumptions about the
purpose of outcome measurement have overlooked these implications. To be clear about purpose is an
essential aspect of choosing an instrument (or instruments within a suite of measures).
Obviously, all instruments do not display the same suitability to achieve diverse purposes. The
properties of measures that become important in an evaluation framework will differ according to the
intended purpose. Such potential purposes may include the usefulness of a measure:
•
•
•
at the population level
at the system or service level, for evaluation and accountability purposes
at the individual level, to allow consumers, carers and clinicians to track those dimensions of
outcome they deem to be important in the context of individual recovery and rehabilitation.
For instance, if a measurement has the twofold aim (a) to be useful to the individual, and (b) to screen
the population for symptoms, population norms must be available for the instrument. If the measure is
intended to help screen for program suitability, its usefulness may vary from condition to condition,
from program to program, with implications for the choice of properties chosen as criteria for
evaluation. In each case, it is important to ask, ‘What is the appropriate set of psychometric and other
properties to apply to measures for each of these purposes?’
Objectives of an evaluation framework
The framework for evaluation that emerges from the literature, practice in outcome measurement here
and elsewhere, and our discussions with the consumers and carers, clinicians and academics who took
part in this project includes these valued objectives:
Usefulness at the individual level
Carers, consumers and clinicians all say they want consumer self-rated outcome measures that are
useful at the individual clinical level.
Assurance that the mental health system is effective
All the stakeholders ideally want a system where they can be confident that it works - that is,
those who need help can get it, what is provided works in a holistic way, and the way services are
offered is humane and supportive. Accordingly, they want built-in measures that track who is in
the system that should be in it, who is not in it that should be, who is in it that should not be, and
so on - the usual program evaluation questions.
Population data
Consumers, carers, and most clinicians are explicitly concerned about population data - not just
about those who make it in the front door of services.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 53
At a number of levels, people are very concerned about outcome measures that should be
available to the system. They want measures that can inform the community how the mental
health system plans for those people who do not make it into the system as well as those who do,
and is accountable for them. They want measures that offer information about how the nature and
incidence of mental illness are changing, and the implications for equity, access, and practice.
Complementary measures
The literature and some academics talk about the value of a set of complementary measures.
While each sector of the mental health system in Australia already has clinician-rated measures,
routine use of consumer self-rated measures is in its early stages in Australia. A gap that all the
stakeholders have identified is the apparent lack of an intention to develop a collateral measure in
the adult suite that families, carers and advocates may complete alongside the consumer and
clinician ratings. In child and adolescent mental health services, the Strengths and Difficulties
Questionnaire (SDQ) is now used in some jurisdictions as a self-rating measure. Parent and
teacher versions enable the views of others also to be considered. SDQ has recently been endorsed
for inclusion in version 1.5 of the NOCC protocol.
Resource and workforce implications
State and Territory officers charged with running mental health systems are concerned about the
planning and resource allocation and management issues involved, and the workforce
development implications of knowing and not knowing the answers to these questions.
Development of a measurement evaluation framework
Several approaches to the construction of a framework for the evaluation of consumer measures are
reported in the literature. Most are variants of a similar set of key criteria. Two widely recognised
Australian studies offer guidance for the development of an evaluation framework for consumer
outcome measurement. Andrews et al (1994, 29-33) say the criteria for use in routine clinical practice
are that ‘consumer outcome measures should be applicable, acceptable, practical, reliable, valid, and
sensitive to change’. We present here the summary of these criteria by Stedman et al (1997, 14)
together with comments based on the results of our literature review and consultations:
1. “The measure must be applicable. The measure should address dimensions that are important to
consumers (symptoms, disability, and consumer satisfaction) as well as being useful for clinicians
in formulating and conducting treatment. The measure should also provide information of a
nature, and in a format, that facilitates the management of services.”
Comments: Our findings wholeheartedly support the relevance of a measure to the domains of
outcome that are important to consumers. However, it is abundantly clear that these dimensions
go well beyond symptoms, disability, and consumer satisfaction. In particular, the dimensions of
greatest importance to consumers are their coping skills, and their capacity for resilience in
dealing with personal relationships and day-by-day functioning. Consumers also want measures
that address optimism and connectedness, social support, personal goal attainment, physical
health, factors outside treatment that influence outcome, and knowing how to negotiate the health
and welfare system, as well helping them to monitor the symptoms and effects of their illness.
Other issues of relevance of a measure or suite of measures that were canvassed in our
consultations with stakeholders (including consumers, carers, clinicians, policy makers, program
managers, and program staff) included these questions:
•
•
•
Is the instrument useful to the consumer and their carers to track how they are going over the
course of an illness or treatment?
Is the instrument useful for individual clinical purposes - that is, could it be used as a tool to
inform clinical management and case management?
Does the survey cover significant comorbidities - in particular, alcohol and drug related
comorbidity?
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 54
•
•
•
Does the measure potentially add to or fit well with the other sources of outcome data
(clinician, population) needed to present a complete picture of outcome at system and
program levels?
Does this instrument have a relationship with a broad population measure to allow
comparisons of the in-treatment population with the general population and with people with
a mental illness who are not in treatment?
Will the data generated at the aggregate level be useful for system planners, evaluators and
for resource allocation? Does the survey cover variables that could inform culture and system
changes?
2. “The measure must be acceptable. The measure should be brief, and the purpose, wording, and
interpretation should be clear (that is, it should be user-friendly).”
Comments: The participants in this project agreed that a measure should be user-friendly and
simple to complete, in language easy to understand, and offered with simple instructions and clear
response formats. Consumers have differing views about brevity – some believe it must be brief
to be acceptable (no more than 10 minutes to complete), but others prefer a longer one that is less
vulnerable to short-term distress, and gives them opportunity to monitor other aspects of their
lives. The issue of acceptability is closely related to the consumer’s sense of the relevance of the
measurement to his or her own purposes.
Many consumers would value an instrument that allows open-ended responses, and creates an
opportunity for the respondent to provide information about their context that would help to
interpret any observed changes.
A measure needs to have culturally and linguistically appropriate forms to cover the diversity of
the Australian population. Ideally, measures should be used that are equally applicable across
cultures, and do not show different psychometric properties (bias) for different cultures. Similarly,
information about the reliability and validity of the measure should have been collected for the
population concerned. The wording and content of the measure should be appropriate to the
culture being assessed.
•
•
•
Are there culturally appropriate versions of the measure?
Does the instrument require linguistic or cultural interpreters for its valid application?
Are there translations of the measure available in relevant languages?
3. “The measure must be practical. Issues of practicality relate to the burden imposed upon
consumers and service providers in terms of time, costs, training in the use of the measure, and
level of skill required in the scoring and interpretation of the data.”
Comments: Our discussions with clinicians and managers confirmed the importance of addressing
issues of practicality. From their perspective, the measure should be relatively cheap to purchase,
relatively brief given the limited time available for completion, and relatively easy to administer,
score and interpret cost effectively without a high level of training of staff, and easy to report back
on at individual and system levels. Other practical criteria canvassed by managers included:
•
•
•
•
Is computer software available?
Is the instrument applicable in both inpatient and outpatient settings?
Do the survey and its use have significant workload implications for the system?
Do the protocols for administering this instrument allow discretion to accommodate periods
of ill health and lack of capacity for informed consent?
A practical issue raised spontaneously in several consumer and carer focus groups is whether the
measure produces information of sufficient value to justify its cost of collection.
Questions of practical administration of particular interest to consumers and carers are these:
•
•
Can this instrument be done as an interview as well as self completed?
Does the survey have a private section that does not go on the file?
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 55
•
Can it form part of a consumer-held record?
4. “The measure must be valid. The measure should have sound psychometric properties and should
measure what it is supposed to measure.
5. “The measure must be reliable. The measure should (within acceptable limits) provide the same
results when given to the same person on two occasions or by two different people.
Comments: We shall address these two evaluation criteria in detail in the next section, including
the questions:
•
Is there established evidence that a measure has well-tested and sound psychometric
properties?
•
Are the properties of an instrument appropriate to the purposes for which it is to be used?
•
Are population norms available in Australia?
•
Can the results of a measure be compared with results from measures widely used nationally
or internationally in other similar organisations and programs?
6.
“The measure must be sensitive to change. The measure should be able to indicate whether a
clinically significant change has occurred for a consumer over consecutive administrations of the
measure. This is the definitive property of a measure of consumer outcomes. A measure may
provide information with respect to mental health status, but it is the extent to which it assesses
meaningful change in a person's condition that it can be called an outcomes measure.”
Comment: The property of sensitivity to change in a person’s condition is indisputably an
essential criterion for any outcome measure. However, at this stage of the implementation of
outcome measurement in the Australian mental health system, the link between clinically
significant change and the definition of ‘outcome’ requires deeper consideration, in light of what
consumers have told us during these consultations.
As an important supplement to the criteria in the Andrews and Stedman framework, we propose an
additional question: “Can the measure function as part of a collective suite of outcome measures?”
Commentators point to the limitations of relying on self-report, and advocate using
complementary information provided by other informants (in youth mental health, for example,
information from the young person, parents and teachers). The clinician and consumer self-rated
measures should complement and add value to each other, and we have no doubt that the system
needs a carer collateral measure in the context of outcome measurement. These considerations
prompt the questions:
•
•
Does the consumer self-rated measure have parallel clinician and carer or family measures?
Can this instrument, in its present or adapted form, function as part of a set of complementary
outcome measures?
The concept of outcome
The AHMAC definition of outcome (“a change in the health of an individual or group of individuals
which is attributable to an intervention or a series of interventions”) and the logic of the current
NOCC protocols for the timing and nature of outcome measurement seem to suggest a direct linear
relationship between treatment intervention and consumer outcomes.
The views and experience of consumers, carers and clinicians paint a very different picture. When
consumer focus or advisory groups have been asked to choose among the currently nominated selfrating measures, they have encountered the difficulty that what they want is a different concept of
outcome. The outcomes that are real to them and the outcomes they regard as valuable to record
include but go far beyond the responses sought by the existing choices.
The research literature in the fields of the mental health and alcohol treatment suggests the need for
models of outcome that address the complex, interactive, and often two-way relationships among the
characteristics the individual brings to the treatment encounter, the characteristics of the treatment
experience (quantity, location, intensity, quality of the therapeutic relationship, congruity of therapist/
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 56
consumer expectations of treatment) and extra- or post-treatment variables such as access to financial,
social and physical resources outside treatment.
What this literature and our consultations suggest is that what is measured, and when it is measured,
should be grounded in evidence about the determinants of outcome, the characteristics of the
treatment system, and the treatment system’s focus and methods.
Outcome measurement should be shaped by the understanding that develops out of the complex mix
of elements and experiences that are determinants of outcome in either ambulatory or inpatient
treatment settings. They include at least these components:
Determinants of outcome
Severity of
condition
Patient’s pre-treatment functioning
Patient’s post-treatment functioning
Access to
physical
resources
Quality of
therapist/patient
relationship
Quality of treatment
Access to resources outside
treatment
Therapist’s
interest in the
extra-treatment
environment
Consumer’s
level of comfort
on discharge
Improvement
in symptoms
Consumer
motivation
Coping skills
and resilience
Goodness of fit between treatment
database and the reality of the
consumer’s extra-treatment world
Formal
aftercare
Informal
treatment
Strength of
social
supports
The content of the domains also has significant implications for when they can logically be measured.
By definition, extra-treatment and post-treatment variables can be measured only between treatments,
or retrospectively at the time of later treatment. This has implications for protocols about when and
how and to whom to apply the measures.
The implications of these shared concerns for the design of an evaluation framework are these:
•
•
•
A consumer self-rated outcome measure cannot be recommended without reference to the rest of
the measurement system it will be part of.
A key performance indicator of a system is whether it provides equitable access, and is attractive
to the range of people for who it is designed.
A system also needs to have access to data that can tell it whether it has the right workforce and
skill mix and focus of intervention to meet consumers’ needs in a changing world (welfare,
employment, housing systems etc). To do that, it needs outcome measures that allow consumers
and carers to report the reality of their world and their living circumstances and how they affect
whether they get well and stay well, as much as their condition allows.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 57
Part 4: Application of the framework to measures
Application of the measurement evaluation framework
The measurement evaluation framework was applied by reference to the following headings:
Description
The length, nature of items and nature of ratings is briefly described. We also report whether any
open-ended questions are included.
The length of the measure contributes both to acceptability to consumers and to practicality. We have
included it in this opening section in order to avoid duplication. The presence of open-ended questions
has been mentioned as an aspect that contributes to acceptability, but it is also an important
descriptive feature of the measure and so is included here.
Domains of applicability
We list the domains, derived from the consumer and carer consultations and comparable work
elsewhere, which the instrument appears to address: coping and resilience, hopefulness,
empowerment, personal relationships, functioning and well being, extra-treatment factors, symptoms
of illness, physical health, and health literacy.
Acceptability
Where possible, we provide information on the acceptability of the measure to consumers.
Practicality
Under this heading, we take up financial justifiability and suitability aspects, including ease of
completion and scoring and availability of computer software. (Brevity is included under Description
above).
Reliability
We describe internal consistency and factorial structure and test-retest reliability.
Sensitivity to change
Stability over time is a desirable characteristic when the underlying trait is invariant. Conversely, the
measure needs to be sensitive to real changes in status (eg after treatment). We report on whether
average scores on the measure change after a group has received treatment. Where available, we
report correlations with changes on other measures, and the ability of the measure to detect change in
individuals.
Validity
We describe correlations with other measures in the same domain, and differences between patient
groups that might be expected to score differently on the measure.
Availability of normative data
We report on the nature and adequacy of normative data and the availability of any information on the
measure’s application across cultures (including Indigenous populations), age groups and levels of
linguistic ability.
Summary
We summarise the evidence across these criteria.
References
We list articles and websites, especially recent publications, dealing specifically with the properties of
the measure under consideration.
Throughout these reviews, a lack of reference to a particular issue should be taken to mean that we
have found no relevant data on the issue.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 58
Bases for selection of measures
For this review, we ensured that the three measures currently used in Australian states and territories
(MHI, BASIS-32, K-10) were included.
Other reviewed instruments must have had some published peer-reviewed research on them. We
omitted measures that focused on a single disorder such as depression or anxiety. While we included
some widely used screening instruments, we emphasised those with some data on sensitivity to
change. Similarly, we included some that were currently too long for practical application, especially
where they covered a number of consumer-identified domains, but for the most part we focussed on
instruments that were relatively brief. We particularly emphasised adult measures, but some measures
were also applicable to older or younger age groups.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 59
Measures in Current Use by Australian States
Kessler 10 (K10), Kessler 6 (K6) and K 10 Plus
Description
The K10 (10-question version) and K6 (6 questions) provide a measure of psychological distress. All
items ask respondents to rate the frequency of the symptom over the past 30 days, using the
alternatives: all of the time (1), most of the time (2), some of the time (3), a little of the time (4), or
none of the time (5). The Department of Health in NSW is also using a 3-day version of K10 (K10L3D) to examine symptoms on inpatient discharge.
The K10 was developed as a scale to screen psychological distress, for inclusion in National Health
Interview Surveys in USA (Kessler et al 2002), and in the 1997 Australian National Survey of Mental
Health and Well being (MHWB; Andrews et al 2001). The scales were primarily based on items from
existing screening scales that related to depression, generalized anxiety or positive mood. Items that
were rated as by an expert panel as clearly expressed were administered by mail or telephone survey
to pilot samples. These data were used to select items that loaded on a single dimension, that had the
greatest ability to discriminate in the highest scoring 10% of the population, and that performed best
in detecting psychiatric disorders (anxiety, mood or non-affective psychotic disorders on the
Structured Clinical Interview for DSM-IV—SCID). K10 and K6 correlate very highly (eg r = .97 in
Kessler et al 2003).
A recently developed “K10+” adds 5 items. These ask whether the feelings were different from usual
in the last 30 days, the number of days they made the person totally unable to work or carry out
normal activities, or able to do half or less than usual, the number of times they saw a health
professional because of the feelings, and whether physical problems were the main cause of the
feelings. The computerised version has 3 contingent items that are only asked if the person
experienced the feeling (if…nervous->…so nervous that nothing could calm you down; if
…depressed->….so depressed nothing could cheer you up; if…restless and fidgety-> …so restless
that you could not sit still). Similarly, the additional questions are only asked in the computerised
K10+ if the person scores above 15 on the first 10 questions (possible scores are 10-50). The nett
effect is that the K10+ is only 7 questions in cases where the person did not experience any of the
listed feelings. In the paper-and-pencil version, all 10 initial questions are administered, but the person
only completes the last 4 “Plus” questions if they experienced one or more feelings. A version of
K10+ without the first additional “change” question (K10+-LM) is currently being used in NSW.
Domains of applicability
The K10 is purely a measure of psychological distress or symptoms (particularly, of anxiety and
depression/dysphoria). As noted above, the K10+ also includes two questions about impact on
functioning and one on whether physical problems were the cause of distress.
Acceptability
The process of developing the K-10 included extensive checks on the clarity of wording, as judged by
clinicians. We are aware of no published qualitative data on the acceptability of the measure to
consumers, but it has been successfully completed by very large numbers of people in both population
surveys and consumers evaluations.
Practicality
K10 was designed for computer-assisted interviews, but can also be administered in paper and pencil
format. The scale is very brief, it is easy to score, and is available at no cost, provided that
acknowledgement is given.
Reliability
Internal structure and internal consistency. Both the K10 and K6 have very high internal consistency
(K10: .92-.93; K6: .89-.92; Kessler et al 2002, 2003).
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 60
Stability over time. No data on the stability of K6 or K10 over time has as yet been published. Current
NSW data could address the issue.
Sensitivity to change
There appears to be no published data as yet on the sensitivity of K6 or K10 to changes in symptoms
after treatment. Current data on the sensitivity of the test to severe symptoms together with data on the
distributions of scores by clinic attenders (G. Stewart, personal communication, 2003) provide some
optimism that it may be sensitive to changes after treatment.
Validity
Effectiveness as a screening instrument. In the test development sample, the K10 and K6 performed
very well at detecting community respondents who met criteria for a DSM-IV Axis I diagnosis on the
Structured Clinical Interview for DSM-IV (Kessler et al 2002). The measures were even more
effective at detecting people with high levels of symptoms and disability. (AUCs, or Areas under the
Receiver Operating Characteristics Curve—which plots the sensitivity of the scale in being able to
detect the disorder at each score against its tendency to falsely identify unaffected people as having
problems—were 0.876 and 0.879 in detecting DSM-IV diagnosis, and 0. 955 and 0.950 in detecting
severe cases, for K10 and K6 respectively).
In the 1997 MHWB sample (10.5k participants), data were available from the 12-month version of the
WHO Composite International Diagnostic Interview (CIDI; World Health Organization, 1997), K6,
K10, SF-12 and GHQ-12. The criterion was fulfilment of DSM-IV depression, dysthymia, or anxiety
disorders on the CIDI. The K10 (AUC = .90) and K6 (AUC = .89) had better discriminatory power
than GHQ-12 (AUC = .80) at the detection of disorder (Furukawa, Kessler, Slade & Andrews, 2003).
K10 performed slightly better than K6, but the discriminating ability of K6 was less affected by the
presence of physical disorder. Sensitivity and specificity for detecting CIDI anxiety or affective
disorders at each K 10 score are provided in Andrews and Slade (2001). K10 correlated moderately
with both GHQ-12 and SF-12 (rho = .5 and -.6 respectively; Andrews & Slade, 2001). Tables are
available for the probability of meeting types of non-psychotic disorders under DSM-IV or ICD-10
(Andrews & Slade, 2001).
Both K10 and K6 are moderately correlated (r’s = .71 and .67) with scores of self-rated functional
disability on the WHO Disability Assessment Schedule (Rehm et al 1999), using instructions that
asked about disabilities because of “emotions, nerves, or mental health” (Kessler et al 2003).
Availability of normative data
Normative data from a sample of 150 US community residents are published in Kessler et al (2003).
More substantial normative data on the general population is available from the application of K-10 in
Australian national surveys in 1997 (a sample of 10.6k) and 2001 (15k), NSW surveys in 1997 and
1998 (17k each) and 2002 (15k), and a Western Australian survey in 2000 (10k). US data on K6 from
1997-2000 in very large community samples is also available. These data tend to show scores of 1021 from 87-92%, 22-29 by 6-9%, and 30-50 in 2-4%. The routine use of K10+LM and K10-L3D in
NSW is now gathering a large data set on consumers of mental health services. These data of course
show much higher levels of average distress, but substantial proportions still exhibiting low scores (G.
Stewart, 2003, personal communication). Data is also being gathered in NSW on translations of
K10+-LM into some 15 languages, but there is no published information on its cross-cultural
applicability at this stage.
Summary
Provided that the K6 or K10 demonstrate sensitivity to change in clinical populations, these
instruments appear to have significant potential as consumer self-rated measures of symptoms (and
specifically of psychological distress). Neither measure covers psychotic symptoms, which of course
is a key concern with many consumers of mental health services, although these symptoms may be
difficult to address reliably in those consumers who have limited insight—see for example the section
below on HoNOS-SR.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 61
If K10+ is used, brief measures of functional impact are available, but there are no published data as
yet on the reliability or validity of these or other additional items in K10+. None of the Kessler scales
cover other domains of key interest to consumers, such as relationships, coping, well being, quality of
life, hopefulness or empowerment. However, they are sufficiently brief to allow a combination with
other brief measures that did cover these areas.
References
Andrews, G., Henderson, S., & Hall, W. (2001). Prevalence, comorbidity, disability and service utilisation.
Overview of the Australian National Survey of Mental Health and Well being. British Journal of
Psychiatry, 178, 145-153.
Andrews, G., & Slade, T. (2001). Interpreting scores on the K10. Australian and New Zealand Journal of Public
Health, 25, 494-497.
Furakawa, T. A., Kessler, R. C., Slade, T., & Andrews, G. (2003). The performance of the K6 and K10
screening scales for psychological distress in the Australian National Survey of Mental Health and Well
being. Psychological Medicine, 33, 357-362.
Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S.-L. T., Wlaters, E. E., &
Zaslavsky, A. M. (2002). Short screening scales to monitor population prevalences and trends in nonspecific psychological distress. Psychological Medicine, 32, 959-976.
Kessler, R.C., Barker, P. R., Colpe, L. J., Epstein, J. F., Gfroerer, J. C., Hiripi, E., Howes, M. J., Normand, S.L. T., Manderscheid, R.W., Walters, E. E., Zaslavsky, A. M. (2003). Screening for Serious Mental Illness in
the General Population. Archives of General Psychiatry, 60, 184-189.
World Health Organization (1997). Composite International Diagnostic Interview (CIDI): Version 2.1. Geneva:
World Health Organization.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 62
The Mental Health Inventory
Description
The MHI was developed for use in the Rand Health Insurance Study, which aimed to study the effects
of contrasting health funding systems on the general population. It can be administered as a self-report
scale or self-ratings at interview, and is available in 18-item and 5-item versions. The full form of
MHI has 38 items that provide a global Mental Health Index, together with component scores of
Psychological Distress and Psychological Well being, and sub-scale scores of anxiety, depression,
loss of behavioural or emotional control, general positive affect, and emotional ties. One item
provides an index of life satisfaction (or positivity of emotion). Items refer to the previous month, and
most refer to the frequency of the feeling over that period (eg always or all of the time to never or
none of the time). There are also items using other scales—eg positivity of emotion, degree of
concern over losing your mind, and degree of anxiety or stress. Questions were taken from MHI to
form the Mental Health Scale of SF-36.
Domains of applicability
The MHI places a large emphasis on symptoms (24 items), but does also include some items that
relate to a sense of well being (eg positive affect, life satisfaction) and personal relationships (14
items).
Acceptability
In one comparative study (Stedman et al 2000), a sample of 183 mental health consumers perceived it
to have significantly greater utility than the SF-36 or BASIS-32 and preferred the MHI and BASIS-32
to the SF-36 (preferences for the MHI and BASIS-32 were similar). Consumers reported little
difficulty in understanding the wording of items, and the response format of the MHI was reported to
be less confusing than that of SF-36. However, the 1-month period was seen as too long for accurate
recall of feelings (Stedman et al 1997; Stedman et al 2000).
Practicality
The MHI form can be downloaded free of charge from the Rand Corporation website:
http://www.rand.org/health/surveys/section5.pdf. Computerised versions of MHI have also been
developed. Its adoption and routine use by mental health services in some Australian states attests to
its practicality.
Reliability
Internal structure and internal consistency. The internal structure of the MHI was initially examined
in a sample of 1,755 respondents from the US general population using exploratory factor analysis.
Confirmatory factor analysis in this sample, and in a combined sample from several sites (n = 5,089)
suggested a hierarchical factor structure. A large bipolar mental health factor was observed, however,
there was a substantial gain of information with a two-factor model of psychological distress and well
being. Psychological distress was in turn defined by 3 lower-order factors, Anxiety, Depression and
Behavioural/Emotional Control, while well being was defined by General Positive Affect, and
Emotional Ties. Notably, 3 of the 38 MHI items loaded two or more lower-order factors (Veit &
Ware, 1983).
High levels of internal consistency were found for each of the factors (Mental health index: α = .96;
Psychological distress: α = .94; Psychological well being: α = .92; Lower-order factors: α = 0.81.92) (Veit & Ware, 1983). Good internal consistency of MHI subscales was also reported in
Australian study of people with mental illness (n = 183), (Mental health index: α = .97; Psychological
distress: α = .96; Psychological well being: α = .93; Lower-order factors: α = 0.80- .93) (Stedman, et
al 1997).
Stability over time. MHI factors exhibited moderate test-retest reliability in a large sample of the US
general population (n = 3,525) over a one-year period (Mental health index: r = .64; Psychological
distress: r = .62; Psychological well being: r = .63; Lower-order factors: r = .56 - .63) (Veit & Ware,
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 63
1983). Test-retest reliability of MHI subscales was reportedly high in an Australian study of people
with mental illness (n = 158), when tested approximately 1 day apart (Mental health index: r = .97;
Psychological distress: r = .96; Psychological well being: r = .94; Lower-order factors: r = .90- .94)
(Stedman et al 1997).
Sensitivity to change
The MHI had a moderate relationship with consumer’s and service provider’s reports of global
change in mental health in Stedman et al (1997). Consumers receiving psychiatric treatment with
either standard face-to-face consultation or with videoconferencing technology showed improved
MHI scores over a 12-month period in Kennedy and Yellowlees (2003).
Validity
The MHI has been cross-validated with two other self-report measures, the BASIS-32 and the SF-36,
in a study of Australian consumers (n = 183) (Stedman et al 1997). MHI scores were generally
correlated with the BASIS-32, with the strongest associations between MHI subscales and the BASIS
scores: relation to self/others, depression/anxiety, and daily living/role functioning. Overall a high
correlation (r = -.81) was found between the MHI Index and BASIS-32 Average, the two summary
scores of each measure.
MHI scores also correlated well with most SF-36 scores. Of the 5 subscales, MHI Depression
correlated best with the SF-36 Mental Health subscale (r =.88). A strong correlation was also reported
between the MHI Index and the SF-36 Mental Component Score (r = .86; Stedman, et al 1997).
The MHI discriminates between consumers with affective disorders and schizophrenia, and to a lesser
extent anxiety disorders. Consumers with affective disorders differed significantly from those with
schizophrenia, exhibiting higher levels of depression, reduced positive affect, and fewer feelings of
emotional/behavioural control, as well as higher levels of psychological distress, and higher overall
scores on the MHI. Consumers with affective disorders also had lower levels of psychological well
being than both consumers with schizophrenia and consumers with anxiety disorders, and fewer
emotional ties than those with anxiety disorders (Stedman et al 1997).
MHI-18 performs better than GHQ-30 at detecting mental disorders (Weinstein et al 1989).
Availability of normative data
Normative data on provider-nominated consumers of public and private mental health services in
Australia is provided by Stedman et al (1997). The MHI has been used as a screening tool for
depression in an elderly sample (Dorfman et al 1995).
Summary
While the MHI heavily emphasises symptoms, it also measures some aspects of well being, and was
more positively received by consumers in the Stedman et al. (1997) study than the BASIS-32 or SF36. It has strong psychometric characteristics and has very brief versions (although the data on those
versions is limited at present). Until its recent adoption for use as an outcome measure for mental
health services in some Australian states, the MHI has had relatively little data on its use for people
with serious mental disorders, especially as an outcome instrument in these populations.
References
Davies, A. R., Sherbourne, C. D., Peterson, J. R., & Ware, J. E. (1998). Scoring manual: Adult health status and
patient satisfaction measures used in RAND’s health insurance experiment. Sata Monica: RAND Corp.
Dorfman, R. A., Lubben, J. E., Mayer-Oakes, A., Atchison, K., Schweitzer, S. O., De-Jong, F. J., & Matthias, R.
E. (1995). Screening for depression among a well elderly population. Social Work, 40, 295-304.
Liang, J., Wu, S. C., Krause, N. M., Chiang, T. L., & Wu, H. Y. (1992). The structure of the Mental Health
Inventory among Chinese in Taiwan. Medical Care, 30, 659-76
Ostroff, Woolverton, Berry, & Lesko (1996).
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 64
Stedman, T., Yellowlees, P., Mellsop, G., Clarke, R., & Drake, S. (1997). Measuring consumer outcomes in
mental health. Canberra, ACT: Department of Mental Health and Family Services.
Veit, C. T., & Ware, J. E. (1083). The structure of psychological distress and well being in general populations.
Journal of Consulting and Clinical Psychology, 51, 730-742.
Weinstein, M. C., Berwick, D. M., Goldman, P. A., & Barsky, A. J. (1989). A comparison of three psychiatric
screening tests using receiver operating characteristic (ROC) analysis. Medical Care, 27, 593-607.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 65
Behaviour and Symptom Identification Scale [BASIS-32]
Description
The BASIS-32 is a brief outcome measurement scale designed to assess both psychiatric symptoms
and functional abilities and was originally intended for use with psychiatric inpatients. Eisen, Dill and
Grob (1994) stated that there are four elements that set the measure apart from other measures. First, it
was empirically derived from the consumers’ perspective; second, it was developed with an acutely ill
psychiatric inpatient population; third, it includes both symptoms and functioning; and fourth, it
combines individual and standardised approaches. The scale includes 32 items and measures the
degree of difficulty experienced by the consumer during a one-week period on a five-point scale: 0 No Difficulty; 1 - A Little Difficulty; 2 - Moderate Difficulty; 3 - Quite a Bit of Difficulty; 4 Extreme Difficulty. An overall BASIS-32 score can be obtained across all items, as well as scores on
each of five subscales.
Domains of applicability
The factor structure reported by Eisen et al (1994) included 5 subscales (1) relation to self and others
(2) daily living and role functioning (3) depression and anxiety (4) impulse and addictive behaviour
and (5) psychosis.
Acceptability
The BASIS-32 is relatively brief and is easy to understand in terms of its language level. If necessary
the BASIS-32 can also be administered by clinician interview, with the respondent indicating their
rating from a response card (Eisen, 1998; Russo, Roy-Byrne, Jaffe et al 1997).
The utility of the BASIS-32 as rated by consumers in Stedman et al. (2000) was a little lower than the
MHI and approximately equal to the SF-36. In that study, the BASIS-32 and MHI were preferred to
the SF-36, and there were no differences in preference between the BASIS-32 and SF-36.
Practicality
The BASIS-32 was designed for use with consumers of mental health services and it accommodates
many of the practical issues. It can be used in both inpatient and outpatient settings. The BASIS-32 is
available on computer-scannable forms with software that automatically scores the measures and
prints out individual scores. The instrument can be completed in 10-20 minutes depending on the
level of functioning of the client.
Eisen, Dickey and Sederer (2000) found that inpatients who completed the BASIS-32 at admission,
from which the clinical team then built a therapeutic alliance informing treatment planning, were
more likely to perceive that they were involved as much as they would like in decisions about their
treatment and to perceive that they were treated with respect and dignity by the staff than inpatients in
comparison groups with whom the BASIS-32 scores were not used to develop therapeutic alliance.
The findings highlight how consumer outcome measures can be used to engage clients in the
treatment process. Treatment outcomes however did not differ between those inpatients in the
intervention group and those in the comparison groups.
As the copyright holder of BASIS-32, McLean Hospital offers an annual site licence for psychiatric
and mental health providers and organisations. The annual fee for an end-user site licence is based on
the number of locations at which the BASIS-32 will be administered. The annual fee is USD 300 for
the 1st location, USD 100 for the second location, and USD 50 for each subsequent location. Contact
details are:
Attn: BASIS-32
McLean Hospital
115 Mill St.
Belmont, MA 02478
E-mail: Alex Speredelozzi ([email protected])
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 66
Detailed information in relation to the BASIS-32 can be located at the following website
http://www.basis-32.org/.
Higgins & Purvis (2002) recently assessed the utility of the BASIS-32 as a consumer self-rated
outcome measure in California’s statewide mental health performance outcome system. The authors
noted the escalation of costs involved in the use of the scale.
Reliability
The internal consistency or alpha coefficients for each of the 5 subscales have been reported to be
0.76 for relation to self and others; 0.80 for daily living and role functioning; 0.74 for depression and
anxiety; 0.71 for impulse and addictive behaviour and 0.63 for psychosis (Eisen et al 1994). The
authors also assessed the stability of the subscales using a second sample of inpatients and reported
similar alphas for the first four subscales however the fifth, namely psychosis, had a substantially
lower internal consistency of 0.43. Internal consistency for the entire scale was reported by Eisen et al
(1994) to be 0.89. Test-retest coefficients were also calculated; 0.80 0.81, 0.78, 0.65 and 0.76. The
average test-retest reliability across all items was 0.85.
Hoffman and Capelli (1997) also investigated the internal consistency of the BASIS-32 and reported
an overall coefficient of 0.91 for an adult inpatient population and 0.92 for an adolescent inpatient
population. Again the authors noted that the subscale showing the least internal consistency was the
psychosis subscale.
Eisen, Wilcox, Leff, Schaefer and Culhane (1999) explored reliability of the BASIS-32 with
outpatients and reported that the internal consistency reliability for the entire scale was again
extremely high at 0.95. The internal consistency for each of the 5 subscales were 0.89 for relation to
self and others; 0.88 for daily living and role functioning; 0.87 for depression and anxiety; 0.65 for
impulse and addictive behaviour and 0.66 for psychosis. The latter two had substantially lower
internal consistency than the other three subscales among this sample of outpatients.
Cameron, Eagles, Howie et al (2001) have recently conducted some preliminary validation of a UKmodified version of the BASIS-32 and briefly reported a five factor structure including (1) depression
and anxiety (2) lability (3) psychosis (4) substance misuse and (5) functioning.
Sensitivity to change
Eisen et al (1994) report that the BASIS-32 is adequately sensitive to change across a 6-month
follow-up with consumers reporting statistically significant decreases in difficulties. Hoffman and
Capelli (1997) investigated sensitivity to change at three points: admission, discharge and six months
following discharge and were able to replicate the sensitivity to change findings for adult inpatients
reported by Eisen et al (1994). Doerfler, Addis and Moran (2002) reported statistically significant
improvements among inpatients from admission to discharge and commented that the BASIS-32 was
useful for tracking consumer functioning across a range of treatment contexts.
Stedman et al., (1997) reported that in comparison to the MHI and the SF-36, the BASIS-32 does not
have as high a level of convergence with other ratings of change.
Reporting on the use of the BASIS-32 in people receiving at least 60 days of treatment by the
Californian mental health system, Higgins & Purvis (2002) noted that, while statistically significant
improvements over time were observed, the size of these changes tended to be small (as measured by
the scale). However it is not clear whether this effect was due to the actual change in status being
small, or the scale being unable to sufficiently detect the change that did occur.
Validity
The concurrent validity of the measures is good with the exception of the subscale of impulse and
addictive behaviour where no difference was found between consumers not hospitalised, consumers
hospitalised in the past 6 months and currently hospitalised (Eisen et al 1994). The authors explain
that since alcohol and drugs are generally not accessible to inpatients this lack of validity is not
surprising. Using BASIS-32 scores to predict diagnostic groups for the depression and anxiety,
impulse and addictive behaviour, and psychosis subscales assessed discriminant validity. The results
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 67
showed that the BASIS-32 scores successfully discriminated these diagnostic groups (Eisen et al
1994).
Hoffman and Capelli (1997) also report strong and significant correlations between the BASIS-32 and
the SCL-90 and reported that the relevant subscales of the BASIS-32 successfully discriminated
inpatients with diagnosis of depression (major depression, bipolar/dysthymia and other) and diagnosis
of substance abuse.
Eisen, Wilcox, Leff, Schaefer and Culhane (1999) also investigated the discriminant validity of the
BASIS-32 by comparing the previous inpatient sample with an outpatient sample and reported that the
BASIS-32 scores differentiate inpatients from outpatients. Also, as was reported above with the
inpatient sample, Eisen Wilcox, Leff, Schaefer and Culhane (1999) found that outpatients diagnosed
with depression or anxiety had more difficulty on the depression/anxiety subscale than did clients
with other diagnoses. However outpatients diagnosed with substance use disorders did not report
more difficulty on the impulsive/addictive behaviours subscale than those not diagnosed with a
substance abuse disorder. The authors suggest that for this sample substance abuse and impulsivity
may be separate factors. The analysis showed that the psychosis subscale failed to discriminate
outpatients with psychotic disorder diagnosis.
Higgins & Purvis (2002) recently presented data suggesting that a large sample of outpatients
receiving treatment for more than 60 days had a relatively low level of impairment compared to the
level expected, given their situation. In fact, a small sample of inpatients that was known to be
severely impaired actually rated themselves as less severe on depression/anxiety, daily living skills
and overall scores than the consumers in the larger sample. This may reflect that different consumers
may be using different anchors for ratings, or that impaired insight may produce under-reporting by
some severely impaired consumers. These issues may of course affect any self-report instrument.
In a response to the article by Higgins & Purvis (2002), Eisen & Speredelozzi (2003) note that (i) the
standard protocol to administer the BASIS-32 was not used and (ii) that the length of time study
participants had spent in treatment is not reported. If consumers had received treatment for some time,
their acute symptomatology may actually be low.
Availability of normative data
Means and standard deviations are available on both inpatients (eg Doerfler et al 2002; Russo et al
1997) and outpatients with mental disorders (Eisen et al 1999).
Summary
The BASIS-32 recognises the need to move beyond symptoms and was developed in conjunction with
consumers of mental health services. However the concepts of well being, personhood, recovery and
quality of life are not well represented. A factor potentially affecting its widespread use is its cost.
References
Eisen, S.V., Klinkenberg, W.D., Cho. D.W., Vieweg. B (1998). Reliability and validity of the interview and
self-report versions of the BASIS-32. Psychiatric Services 49:1229-1231
Eisen, S.V., Dickey, B, Sederer, L.I. (2000). A self-report symptom and problem rating scale to increase
inpatients' involvement in treatment. Psychiatric Services 51(3): 349-53
Eisen, S.V., Dill, D.L., Grob, M.C. (1994). Reliability and validity of a brief patient-report instrument for
psychiatric outcome evaluation. Hospital and Community Psychiatry 45(3): 242-7
Eisen, S.V. and Culhane, M.A. (1999). Behavior and Symptom Identification Scale (BASIS-32). In: Maruish,
M.E. (ed), The use of psychological testing for treatment planning and outcomes assessment (2nd ed.)
Eisen, S.V., Wilcox, M., Leff, H. S., Schaefer, E., Culhane, M. A. (1999). Assessing behavioral health outcomes
in outpatient programs: reliability and validity of the BASIS-32. Journal of Behavioral Health Services and
Research 26(1): 5-17
Eisen, S.V. and Speredelozzi. A. (2003). Letter: Usefulness of the BASIS-32 in evaluating program level
outcomes. Journal of Psychiatric Practice 9: 5-6
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 68
Higgins J. and Purvis K. (2002). Usefulness of the BASIS-32 in evaluating program level outcomes, Journal of
Psychiatric Practice8: 125-30
Hoffman, F.L., Capelli, K., Mastrianni, X. (1997). Measuring treatment outcome for adults and adolescents:
reliability and validity of BASIS-32. Journal of Mental Health Administration 24(3): 316-331
Klinkenberg, W.D., Cho, D.W., Vieweg, B. (1998). Reliability and validity of the interview and self-report
versions of the BASIS-32. Psychiatric Services 49(9): 1229-31
Russo, J., Roy-Byrne, P., Jaffe, C., Ries, R., Dagadakis, C., Dwyer O'Connor, E., Reeder, D (1997). The
relationship of patient-administered outcome assessments to quality of life and physician ratings: Validity
of the BASIS-32. Journal of Mental Health Administration, 24(2): 200-215
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 69
Summary comparison of consumer self-rating measures currently in use in Australia
Measure
K6/ K10 / K10+
(6-15 items, less skips)
MHI
(38 items)
(Also 5, 18 item versions)
BASIS-32
(32 items)
-
-
-
Criterion
Domains of applicability
Coping and resilience
-
-
-
Hopefulness
-
-
-
Empowerment
-
-
-
Personal relationships
-
Functioning & well being
Extra-treatment factors
Symptoms of illness
Physical health
Health literacy
Acceptability
Practicality
Reliability
K10+: Impact on functioning
Well being
Distress/anxiety/depression
-
Relation to self and others
Daily living and role
functioning
Impulsive and addictive
behaviour
-
Distress/anxiety/depression
Depression & anxiety
Behavioural/emotional control
Psychosis
Extent they caused distress
-
-
-
-
High acceptability in one
study, including equal
consumer preference to MHI
(but lower perceived utility)
Brief, wording clear
Translated into several
languages
Clear
Computer-assisted available.
Free
Computerised available. Use
of 38-item version is free
Computer scanning available.
Internal consistency of K10
and K6: high. No published
data on K10+
Internal consistency of factors
high
Test-retest very high over 1
day, moderate over 1 year
Internal consistency high for
total, moderate for most
subscales (psychosis low to
moderate)
Moderate correlation with
clinician ratings of change
Moderate sensitivity
established
Test-retest: No published data
Sensitivity to change
Emotional ties
No published data
Brief forms available.
High acceptability in one
study, including higher
perceived utility than BASIS32 and equal consumer
preference
Attracts cost (site licence)
Test-retest: high
Improves after treatment
Validity
Support as population screen.
Validity in clinical populations:
No published data
Availability of normative data
Community norms available
High correlations with BASIS32 and SF-36.
Discriminates between types
of disorders.
Australian norms available
Clinical samples: No
published data, but large
Australian samples collected
Siggins Miller Consultants
Consumer self-rated outcome measures
Discriminates between types
of disorders (one study
reported low discriminate
validity for psychotic
disorders).
Norms available for inpatients
and outpatients
Page 70
Other consumer measures
Short Form 36 (SF-36) and SF-12
Description
The SF-36, together with the SF-12, and SF-8 scales, are a collection of surveys that measure generic
health status. The SF-36, a 36-item short-form survey, was the first of the three scales to be developed
(Ware & Sherbourne, 1992). It was designed for use in the Medical Outcomes Study (MOS), a project
conducted by the Rand Corporation in the United States to devise tools for monitoring consumer
outcomes in routine clinical practice. The scale items were selected by the authors to measure eight
domains of health commonly represented in previous health surveys: physical functioning, role
limitations due to physical problems, social functioning, bodily pain, general mental health, role
limitations due to emotional problems, vitality, and general health perceptions (Ware & Sherbourne,
1992). Scale scores for each domain may in turn be used to calculate a Physical Component Score
(PCS) and a Mental Component Score (MCS).
The 12-item SF-12 survey was constructed four years later to provide a shorter scale for use in largescale health measurement and monitoring studies (Ware, Kosinski, & Keller, 1996). Results for the
SF-36 survey administered as part of the National Survey of Functional Health Status and MOS were
subjected to regression analysis to identify a subset of items able to estimate Physical and Mental
Component Scores. Ten items reproduced SF-36 PCS and MCS with an R2 of above 0.90. Two
further items were selected to ensure that each of the eight original health domains was represented.
The SF-12 and SF-36 share 12 items in common. In contrast, the SF-8 and SF-36 share only one item.
Nevertheless, for each of the eight health domains, the content of the SF-8 is very similar to that of the
SF-36 and SF-12, and the measures of each health domain are highly correlated scores across the
three forms. The psychometric properties of the SF-36 have been reviewed in detail elsewhere. The
SF-12, which is shorter and quicker to complete will be the focus of the current review.
Domains of applicability
All of the SF versions produce scale scores that relate to aspects of the consumer-identified domains
of personal relationships, functioning, and physical health and mental health and vitality/well being.
Practicality
The SF-12 can be either self-administered or administered within an interview, and takes
approximately 2-minutes to complete. Standard hand scoring of the SF-12 is time-consuming because
it involves weighting each answer to 5 decimal places. An alternative scoring method that rounds
scores to the nearest integer returns fairly comparable results (Andrews, 2002). The SF-12 may also
be scored using a computerised scoring program that creates two summary scores: physical health
(PCS12) and mental health (MCS12). The scale is available at no cost, but the manual is purchased.
Reliability
Internal structure and internal consistency. Factor analysis of the SF-12 has confirmed the scale’s
two-factor structure in both medical and psychiatric populations (Salyers et al 2000; Gandhi et al
2001; Amir et al 2002). SF-12 mental and physical component scores are weakly correlated (r= 0.06),
suggesting that the two scores are largely independent (Ware, et al 1996).
Test-retest reliability. The SF-12 shows high test-retest correlations over 2 weeks in general
populations from the US (n = 232), (PCS: r = 0.89, MCS: r = 0.76), and UK (n = 187) (PCS: r = 0.86,
MCS: r = 0.77; Ware, et al 1996). Good test-retest reliability has been found for consumers with
severe mental illness (n = 64) tested approximately one week apart (PCS: ICC = 0.79; MCS: ICC =
0.79).
Sensitivity to change
The SF-12 MCS and PCS has been shown to be sensitive to longitudinal changes in the health of
consumers receiving treatment for congestive heart failure, sleep apnoea, and inguinal hernia
(Jenkinson et al 1997). Over a six-week period, consumers receiving combined antidepressant therapy
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 71
and nurse telephone counselling for depression show improved MCS scores, compared to those
receiving antidepressant therapy alone (Hunkeler et al 2000). There is some evidence, however, that
SF-12 scores for the eight domains of health are less useful than those of the SF-36 for monitoring
longitudinal change in health status (Ware et al 1996; Schofield & Mishra, 1998).
Validity
A cross-validation study performed in the United States with 2,293 participants found that Physical
and Mental Component Scores obtained on the SF-12 are highly correlated with those obtained on the
SF-36 (PCS: r = 0.95; MCS: r = 0.97), and further that the SF-12 largely reproduced the SF-36
component scores, explaining 90% of the variance in PCS and 94% of the variance in MCS (Ware et
al 1996). These results have been replicated in studies of the general population in Europe (Gandek et
al 1998) and Australia (Sanderson & Andrews, 2002), and in a study of medical consumers in the UK
(Jenkinson et al1997).
SF-12 component scores are reportedly capable of discriminating between consumers with known
differences in physical and mental health (Ware et al 1996). The SF-12 and SF-36 scales have reached
the same statistical conclusions regarding group differences, however, validity coefficients were
typically 10% lower for SF-12 than the SF-36 (Ware et al 1996). In a study of Australian women, the
SF-12 PCS discriminated between women in poor versus good physical health, while the SF-12 MCS
discriminated between groups with or without psychological distress, as measured by the GHQ-12
(Schofield & Mishra, 1998).
The discriminate validity of the SF-12 has also been examined in consumers with severe mental
illness (n = 946). Both PCS and MCS correlated with three physical health indices; however, PCS was
more strongly related than MCS to two of the measures: chronic health problems (r = -0.31) and
doctor visits (r = -0.32). PCS and MCS were equally related to days hospitalised for a physical
condition. MCS correlated with two mental health indices: psychiatric hospitalisations (-0.32) and
current substance use disorder (-0.11), and was more strongly related than PCS to self-reported mental
health (0.49; Salyers et al 2000).
Availability of normative data
SF-12 standardized scores are norm-based with a mean of 50 and a standard deviation of 10 in US
(Ware et al (1998) and Australian (Schofield & Mishra, 1998) samples. The SF-12 has been translated
into more than 40 languages. It can be used with respondents 14 years and over. There is some
evidence that the format of the questionnaire may be confusing for older respondents (>70 years),
which has lead to a successful modification (Iglesias et al 2001). The SF-12 has been employed to
assess health status in a wide range of groups, including the general population, consumers with
medical conditions, people with mental illness, and the homeless.
Summary
The SF-12 has very sound psychometric characteristics, and it has been shown to be sensitive to
change. It provides both physical and mental status scores, and it includes items on the functional
impact of disorders. It has been used in a wide variety of linguistic contexts, and Australian norms are
available.
References
Amir, M., Lewin-Epstein, N., Becker, G., Buskila D. (2002), Psychometric properties of the SF-12 (Hebrew
version) in a primary care population in Israel. Medical Care 40(10): 918-28
Andrews, G. (2002). A brief integer scorer for the SF-12: validity of the brief scorer in Australian community
and clinic settings. Australian & New Zealand Journal of Public Health 26(6): 508-10
Gandek, B., Ware. J.E.Jr, Aaronson, N.K., Apolone, G., Bjorner, J.B., Brazier, J.E., Bullinger, M., Kaasa, S.,
Leplege, A., Prieto, L., Sullivan, M. (1998). Cross-validation of item selection and scoring for the SF-12
Health Survey in nine countries: results from the IQOLA Project. International Quality of Life Assessment.
Journal of Clinical Epidemiology 51(11): 1171-8
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 72
Gandhi, S.K., Salmon, J.W., Zhao, S.Z., Lambert, B.L., Gore, P.R., Conrad, K. (2001). Psychometric evaluation
of the 12-item short-form health survey (SF-12) in osteoarthritis and rheumatoid arthritis clinical trials. Clinical
Therapeutics 23(7): 1080-98
Hunkeler, E.M., Meresman, J.F., Hargreaves, W.A., Fireman, B., Berman, W.H., Kirsch, A.J., Groebe, J., Hurt,
S.W., Braden, P., Getzell, M., Feigenbaum, P.A., Peng, T., Salzer, M. (2000). Efficacy of nurse telehealth
care and peer support in augmenting treatment of depression in primary care. Archives of Family Medicine.
9(8): 700-8
Iglesias, C.P., Birks, Y.F., Torgerson, D.J. (2001). Improving the measurement of quality of life in older people:
the York SF-12. QJM 94(12):695-8
Jenkinson, C., Layte, R., Jenkinson, D., Lawrence, K., Petersen, S., Paice, C., Stradling, J. (1997). A shorter
form health survey: can the SF-12 replicate results from the SF-36 in longitudinal studies? Journal of
Public Health Medicine 19(2): 179-86
Salyers, M. P., Bosworth, H. B., Swanson, J. W., Lamb Pagone, J., Osher, F. C.(2000). Reliability and validity
of the SF-12 Health Survey among people with severe mental illness. Medical Care 38)11): 1141-50
Sanderson, K. and Andrews, G. ( 2002). The SF-12 in the Australian population: cross-validation of item
selection. Australian & New Zealand Journal of Public Health 26(4): 343-5
Schofield, M.J. and Mishra, G. (1998). Validity of the SF-12 compared with the SF-36 Health Survey in pilot
studies of the Australian Longitudinal Study on Women's Health. Journal of Health Psychology 3(2): 259271
Ware, Kosinski & Keller, (1998). SF-12: How to score the SF-12 Physical and Mental Health Summary Scores
(3rd Ed.). Lincoln, RI, QualityMetric Incorporated.
Ware, J.E.Jr and Sherbourne, C.D. (1992). The MOS 36-item short-form health survey (SF-36): I. Conceptual
framework and item selection. Medical Care 30(6): 473-483
Ware, J.E.Jr and Sherbourne, C.D. (1992). The MOS 36-item short-form health survey (SF-36): I. Conceptual
framework and item selection. Medical Care 30(6): 473-483
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 73
General Health Questionnaire
Description
The GHQ is a self-report screening instrument that was developed to detect psychiatric distress in
non-clinical populations and the general community (Goldberg and Williams, 1988). It was not
designed to provide a diagnosis. There are several versions of the measure, the GHQ-12, GHQ-28,
GHQ-30 and the GHQ-60. In recent years, the GHQ-28 and the GHQ-12 have been the most widely
used. Traditionally, the scoring involves measurement on a 4 point likert scale and total scores are
then subject to a threshold cut-off value to determine “caseness”.
Domains of applicability
The GHQ provides a broad measure of psychiatric distress and social dysfunction. For example, the
GHQ-28 comprises four subscales - somatic symptoms, anxiety and insomnia, social dysfunction, and
depression. The GHQ-30 and GHQ-12 do not include the somatic symptoms, since these symptoms
may be present for people with entirely physical illnesses who do not have psychiatric distress. The
GHQ could potentially be used in combination with measures that addresses the other areas of interest
to consumers.
Practicality
The GHQ has been used widely, particularly in research but also in clinical practice settings. It is
widely available and can be administered by a range of mental health professionals. It is available in
numerous languages, up to 38. While the GHQ-60 may be considered too long for use with consumers
of mental health services, the GHQ-12 can be completed in 2 to 5 minutes and the GHQ-28 in 5 to 10
minutes. The GHQ does not include qualitative information.
Use of the GHQ does incur a cost to the service.
Reliability
Studies of the reliability of the various versions of the GHQ have been widely undertaken across
numerous populations and cultural settings. In a review of psychiatric measures Goldman et al (2000)
provide a comprehensive discussion of the reliability and validity of the various versions of the GHQ.
Specifically, Cronbach’s alpha coefficients for the GHQ-60, GHQ-30 and GHQ-12 are reported to
typically range from 0.82 to 0.93 indicating good internal consistency. And while the authors noted
that test-retest scores are subject to variation given that the scale measures psychiatric distress rather
than diagnosed disorders, they report a coefficient of r =0.90 for the GHQ-28 and a range of r =0.50
to 0.90 for the other three versions of the scale.
Sensitivity to change
GHQ versions including the GHQ-12 have been extensively used in controlled outcome trials, and
appears sensitive to change after treatment.
Validity
Studies of the validity of the various versions of the GHQ have been widely undertaken across
numerous populations and cultural settings. In their review Goldman and colleagues (2000) reported
that GHQ scores generally correlate with scores from psychiatric structured interviews ranging from r
=0.65 to 0.70. The authors also comment on the specificity and sensitivity of the GHQ and state that
on the basis of more than 60 studies, all versions of the GHQ have sensitivity (probability of testing
positive when the disease is truly present) estimates for psychiatric distress of 80%-84%; estimates for
specificity (probability of testing negative when the disease is truly absent) are generally
commensurate with the figures for sensitivity if not slightly higher.
Availability of normative data
The manual reports detailed norms for the GHQ.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 74
Summary
The GHQ was developed for use as a screening instrument or as an epidemiologic tool for estimating
psychiatric morbidity, but it also appears to be sensitive to clinical improvement. The GHQ-12 is
likely to be the most useful of the various versions of the GHQ for current purposes. Cost may prove a
barrier to its widespread adoption.
References
Donath, S. (2001) The validity of the 12-item General health Questionnaire in Australia: a comparison between
three scoring methods. Australian and New Zealand Journal of Psychiatry, 35:231-235.
Goldberg, D.P., Gater, R., Sartorius, N., Ustun, T.B., Piccinelle, M., Gureje, O., and Rutter, C. (1997) The
validity of two versions of the GHQ in the WHO study of mental illness in general health care.
Psychological Medicine 27(1): 191-197.
Goldberg, D.P. and Williams, P. (1988) A User’s Guide to the General Health Questionnaire. NFER-Nelson
Publishing Co. Ltd. Windsor, UK.
Goldman, R.S., Robinson, D., Grube, B.S. (2000) General Psychiatric Symptom Measures. In A.J. Rush et al
(Eds) Handbook of Psychiatric Measures. American Psychiatric Association Washington, D.C.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 75
Depression Anxiety and Stress Scale (DASS) and DASS-21
Description
The DASS (Lovibond 1995a; Wilson, 1982) is a 42-item self-report questionnaire that provides scores
on the level of Depression, Anxiety and Stress. Responses refer to the degree, or the amount of time,
that items referred to the person over the previous week. Inspection of the depression scale suggests
that it focuses on cognitive-emotional features and inertia, that anxiety emphasises arousal, panic and
muscular tension, and stress covers areas such as difficulty relaxing, impatience, irritability and
agitation. A shorter version (DASS-21) has 7 items for each symptom type (Lovibond 1995a).
Domains of applicability
The DASS assesses symptoms of depression, anxiety and stress. It does not cover other forms of
psychopathology or other consumer-identified domains.
Practicality
DASS-21 is currently a paper-and-pencil questionnaire that is relatively quick to administer and is
rapidly scored.
Reliability
Internal structure and internal consistency. Repeated factor analyses attest to the 3-factor structure of
the DASS and DASS-21, both in clinical populations (Antony et al 1998; Brown et al 1997; Daza et
al 2002; Lovibond 1995b) and an unselected student sample (Lovibond 1995b). However, replication
studies have found that one item on the original DASS does not conform to its original attribution to
Anxiety (“I feared I would be ‘thrown’ by some trivial but unfamiliar task”). The 3 scales are
moderately and positively correlated with each other (Antony et al 1998; Brown et al 1997; Lovibond
1995a). Internal consistencies are consistently high for both the DASS (Depression: .91-.97; Anxiety:
.84-.89; Stress: .90-.93; Antony et al 1998; Brown et al 1997; Lovibond 1995b) and DASS-21
(Depression: .93-.94; Anxiety: .86-.87; Stress: .91; Antony et al 1998; Daza et al 2002).
Stability over time. The scales show high test-retest correlations over 2 weeks in clinical populations
(Depression: r = .71; Anxiety: r = .79; Stress: r = .81) and no significant average changes over the
period (Brown et al 1997). A 3-8 year follow-up study by Lovibond (1998) using undergraduate
students, found much lower test-retest correlations (Depression: r = .29; Anxiety: r = .41; Stress: r =
.39), even though most respondents completed the scale about a “typical week in the past 12 months”
rather than having the standard “past week” instructions.
Sensitivity to change
Using an earlier 2-factor version of the Scales (DAS), Wilson (1982) found that volunteers who
scored highly on the Beck Depression Inventory (BDI) and received antidepressant medication
(amitriptyline) showed a larger reduction in DAS Depression at 4 weeks than those who received
placebo. On both the BDI and DAS Anxiety, there was a significant reduction over time for all
groups, but no specific effect for treatment. In an uncontrolled case series study by Hooke and Page
(2002), all three scales from the 42-item DASS were sensitive to changes after an intensive 2-week
program of group cognitive-behavioural therapy.
Validity
People with DSM-III-R mood disorders score significantly higher on the Depression scale than people
with anxiety disorders, and those with Panic Disorder and Agoraphobia scored higher on Anxiety than
those in other groups (Antony et al 1997; Brown et al 1997). This is also true of DASS-21 (Antony et
al 1997). Depressed people and those with Generalised Anxiety Disorder (GAD) tend to score more
highly than non-GAD anxiety groups on the Stress scale (Antony et al 1997; Brown et al 1997). All
three DASS and DASS-21 scales also discriminate between people with anxiety or mood disorders
and nonclinical volunteers (Antony et al 1997).
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 76
A longitudinal study of final-year high school students found that scores immediately before the final
examination were significantly higher than an assessment 10 weeks before (Einstein et al 2000).
However in that study, there was no control re-assessment over a different 10-week period.
DASS and DASS-21 Anxiety correlate well with the Beck Anxiety Inventory (Beck & Steer, 1990) in
both clinical (Antony et al 1998: r = .84; Brown et al 1997: r = .75; Daza et al: r = .82) and nonclinical populations (Lovibond 1995b: r = .81), as does Depression with the Revised Beck
Depression Inventory (Beck & Steer, 1987; Clinical: Antony et al 1998: r = .77; Brown et al 1997:
r = .83; Daza et al: r = .86; Non-clinical: Lovibond 1995: r = .74). In Brown et al (1997), the Penn
State Worry Questionnaire (Meyer et al 1990) correlated .60 with Stress (r = .60; Brown et al 1997).
That study found that the criterion measures had a higher correlation with their relevant scales than
with the other scales. Together, these data strongly support the interpretation of the scales.
Availability of normative data
Normative data is available on Australian adults (Lovibond 1995a), and means and standard
deviations are available on a sample of final-year high school students (Einstein et al 2000). The latter
showed relatively high scores, and it is not clear whether this reflects a developmental change or an
effect of ambient stress during the final year of high school.
A study using a Spanish translation of DASS-21 found that it had high internal consistency and
validity in a bilingual group of US residents with anxiety disorders (Daza et al 2002). Einstein et al
(2000) found that scores on DASS Depression and Anxiety were higher in final year high school
students in Sydney whose parents were born overseas than for those whose parents were Australianborn. It is not clear whether this finding relates to immigration or to cultural issues, or whether it is
developmentally specific. The result suggests that caution should be used in applying DASS norms to
people from overseas backgrounds. Currently published data do not allow conclusions on its
applicability across older aged adults, to indigenous people, or to people with limited linguistic
abilities.
Summary
DASS and DASS-21 focus only on symptoms, but the 21-item version could be combined with a
measure of other domains of interest to consumers. While substantial data are now available on the
DASS, information on its ability to correctly identify people with disorders and we located no
published data on the sensitivity of DASS-21 to changes after treatment.
References
Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric properties of the
42-item and 21-item versions of the Depression Anxiety and Stress Scales in clinical groups and a
community sample. Psychological Assessment, 10, 176-181.
Beck, A. T., & Steer, R. A. (1987). Manual for the Revised Beck Depression Inventory. San Antonio, Tex:
Psychological Corporation.
Beck, A. T., & Steer, R. A. (1990). Manual for the Beck Anxiety Inventory. San Antonio, Tex: Psychological
Corporation.
Brown, T. A., Chorpita, B. F., Korotitsch, W., & Barlow, D. H. (1997). Psychometric qualities of the
Depression Anxiety and Stress Scales (DASS) in clinical samples. Behaviour Research and Therapy, 35,
79-89.
Daza, P., Novy, D. M., Stanley, M. A., & Averill, P. (2002). The Depression Anxiety and Stress Scale-21:
Spanish translation and validation with a hispanic sample Journal of Psychopathology and Behavioral
Assessment, 24, 195-205.
Einstein, D., Lovibond, P. F., & Gaston, J. E. (2000). Relationship between perfectionism and emotional
symptoms in an adolescent sample. Australian Journal of Psychology, 52, 89-93.
Hooke, G. R., & Page, A. C. (2002). Predicting outcomes of group cognitive behavior therapy for patients with
affective and neurotic disorders. Behavior Modification, 26, 648-658.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 77
Lovibond, S. H. & Lovibond, P. F. (1995a). Manual for the Depression Anxiety and Stress Scales (DASS),
2nd Ed. Sydney: Psychology Foundation.
Lovibond, P. F. & Lovibond, S. H. (1995b). The structure of negative emotional states: Comparison of the
Depression Anxiety and Stress Scales (DASS) with the Beck Depression and Anxiety Inventories.
Behaviour Research and Therapy, 33, 335-343.
Meyer, T.J., Miller, M.L., Metzger, R.L., Borkovec, T.D. (1990). Development and validation of the Penn State
Worry Questionnaire. Behavior Research and Therapy 28(6): 487-95
Wilson, P. H. (1982). Combined pharmacological and behavioural treatment of depression. Behavior Research
and Therapy 20: 173-184
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 78
World Health Organization Quality of Life – BREF
Description
The WHOQOL-BREF is an abbreviated version of the WHOQOL-100. The measure was developed
simultaneously in fifteen international field centres in an attempt to develop a cross-culturally relevant
measure of quality of life defined as “individuals’ perceptions of their position in life in the context of
the culture and value systems in which they live and in relation to their goals, expectations, standards
and concerns”. The WHOQOL-BREF is a self-report measure comprising of 26 items scored on a 5point likert scale. A time frame of two weeks is indicated in the instrument.
Domains of applicability
The scale has four domains: physical, psychological, social relationships and environment and also
includes two general items namely (1) How would you rate your overall quality of life and (2) How
satisfied are you with your health? The Facets incorporated within domains are detailed below.
Physical health:
Activities of daily living, Dependence on medicinal substances and medical aids, Energy and fatigue,
Mobility, Pain and discomfort, Sleep and rest, Work capacity.
Psychological
Bodily image and appearance, Negative feelings, Positive feelings, Self-esteem, Spirituality / Religion
/ Personal beliefs, Thinking, learning, memory and concentration.
Social relationships
Personal relationships, social support, sexual activity.
Environment
Financial resources, Freedom, physical safety and security, Health and social care: accessibility and
quality, Home environment, Opportunities for acquiring new information and skills, Participation in
and opportunities for recreation / leisure activities, Physical environment (pollution / noise traffic /
climate), Transport.
Practicality
The measure is freely available on the Internet and takes approximately 5 to 10 minutes to complete.
It is available in 19 different languages. The instrument can be administered by interview if necessary
and a standard set of instructions exists if this approach to administration is necessary.
Reliability
Internal consistency of the scale in the original analysis showed Cronbach alpha values for the four
domains ranged from 0.66 to 0.84 (WHOQOL Group, 1998). Test -retest reliability was conducted in
four of the fifteen scale development sites and the interval between test and re-test ranged form 2
weeks to 8 weeks. The correlations between items across time ranged from 0.56 to 0.84. Test-retest
reliabilities for the four domains were 0.66 for physical health, 0.72 for psychological, 0.76 for social
relationships and 0.87 for environment.
Sensitivity to change
Not addressed in the published literature.
Validity
The WHOQOL-BREF was been shown to have good validity in discriminating between ill and well
respondents (WHOQOL Group, 1998).
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 79
Availability of normative data
Means and standard deviations are available on a wide variety of populations. The very bases for the
development of the WHOQOL-BREF was to develop a cross-culturally useful measure of quality of
life and therefore can be used with people from a range of culturally and linguistically diverse
backgrounds.
Summary
One advantage of the WHOQOL-BREF is that it has been relatively widely used internationally and
therefore scores could potentially be compared across a range of countries. Perhaps its weakness in
the current context is its limited measurement of symptoms of serious mental illness.
References
The WHOQOL Group (1988). Development of the World Health Organization WHOQOL-BREF Quality of
Life Assessment. Psychological Medicine, 28(3); 551-558.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 80
The Quality of Life Interview
Description
The original Quality of Life Interview (QOLI, Lehman, 1988) is a 158-item scale that is administered
in a 45-minute structured interview, while a short version takes 15-20 mins. In all versions the
consumer reports personal characteristics, objective indicators and subjective ratings of quality of life.
Subjective ratings of global well being and life domains used a 7-item scale from Terrible to
Delighted.
The interview covers global well being, as well as the life domains of living situation, family
relations, social relations, leisure activities, work and school, finances, legal/safety and health. A
version of the longer scale also has a life domain of religion, and Levitt et al (1990) added an adult
education satisfaction scale and a self-fulfilment scale for use in a day treatment setting.
Domains of applicability
The QOLI does attempt to cover aspects of the listed consumer domains of personal relationships,
functioning, well being, quality of life and physical health. There is an opportunity to rate services in
the sense of the medical care available to the person, how often they see a doctor, and the chance to
talk to a therapist. However there is no rating on the perceived quality of medical or psychiatric care.
Segments of the original QOLI assumed that the person was living in a sheltered environment (eg
objective Living Situation, and initial objective questions on Social Relations), limiting the
comparability of those aspects across different living environments. The original QOLI does include
some open-ended questions about things that made the person feel good or unhappy during the last
week, and one thing they would like to change in their life.
Practicality
Application of even the brief version of the QOLI as part of a routine assessment of all clients may be
impractical because of the time required.
Although the QOLI is copyright, it may be reproduced free of charge with permission. A copy may be
obtained from The Evaluation Center, Health Services Research Institute, 2336 Massachusetts Ave,
Cambridge MA 02140 USA.
Reliability
Internal consistencies and test-retest reliabilities are usually reported for items or life domains rather
than for an overall total (Lehman, 1988, 1995). Internal consistencies for subjective ratings within
each life domain are satisfactory for both the original (.70-.87; Lehman, 1995) and the brief QOLI (
.82-.89 in Lehman, 1995; .82 - .90 in Russo et al 1997). The internal consistency of objective scales is
moderate to high (Original: .61-.80; Brief: .60-.82; Lehman, 1995). Similar internal consistency
results are seen in a Greek language version (Zissi et al 1998).
One-week test-retest reliabilities of the subjective ratings range from .41 (Work) to .95 (Safety), with
global well being at .71 (Lehman, 1988).
Subjective ratings of global well being tend to show moderate correlations with subjective ratings of
life domains (Lehman, 1983; Trauer et al 1998), although correlations with work satisfaction are
variable (Lehman, 1983). Objective measures do not tend to correlate highly with their subjective
counterparts (Trauer et al 1998; Russo et al 1997; Zissi et al 1998), although moderately strong
correlations between the number of social contacts and social satisfaction are often obtained (Russo
et al 1997; Trauer et al 1998). The lack of a relationship between subjective and objective quality of
life is also seen in QOLI ratings of consumers that are undertaken by collateral observers (Khatri et al
2001).
In terms of inter-rater reliability, a strong correlation (r = .72) between total objective QOLI scores of
consumers and collateral informants was obtained by Khatri et al (2001). Total subjective scores by
consumers and collaterals were not significantly related, suggesting that perspectives on subjective
quality of life by different observers may be very different.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 81
Sensitivity to change
The QOLI is sensitive to changes that occur over an inpatient admission averaging 2 weeks (Russo et
al 1997). However Cramer et al (2000) found that objective QOLI scales were less sensitive to change
than interviewer-rated scales.
Validity
Correlations between the QOLI and clinician-rated scales of functional status tend to be relatively low
(Barry & Crosby, 1996; Dickerson et al2000; Trauer et al 1998), even when objective functioning in a
similar life domain is the focus (Dickerson et al2000). However closer relationships can be sometimes
obtained with consumer-rated measures that tap similar life domains. In Dickerson et al (2000), some
QOLI objective scales correlated at least moderately with scores on the self-report Social Functioning
Scale (Birchwood et al 1990)—notably, QOLI Daily Activities with SFS Recreation (r = .64) and
SFS Prosocial (r = .46), QOLI Family with SFS Prosocial (r = .28), and QOLI Social Relations with
SFS Interpersonal Behaviour (r = .34). However with the exception of the first of these correlations,
the values were not sufficiently high to allow an assumption that the two measures were assessing
substantially the same construct. Sullivan et al (1992) found that self-rated global quality of life was
moderately correlated (r = -.52) with the consumer-rated level of criticism by family members,
consistent with the perceived living environment being an important element in global subjective
quality of life.
Subjective quality of life is lower in inpatients with depressive disorders than in those with mania,
schizophrenia or other disorders (Russo et al 1999), and it correlates negatively with levels of
depression and anxiety (Corrigan & Buican, 1995; Lehman, 1988). However correlations with levels
of depression are often relatively low, suggesting a separability of the constructs. There is little
relationship between subjective quality of life and being employed (van Dongen et al 1996).
Availability of normative data
Normative data on subscales completed by 1805 US participants with chronic mental disorders are
provided in the manual (Lehman, 1995). Means and standard deviations by diagnostic groups are also
available on large samples with data at admission and discharge (Russo et al 1997). Smaller samples
are provided by a range of other studies including Dickerson et al (2000—72 stable US outpatients),
Vandiver et al (1998—42 Canadian and 36 US outpatients), Trauer et al (1998—55 Australian
consumers). Data on psychiatric consumers who were also past and current substance misusers and
consumers without a substance use history are provided by Dixon et al (1998).
Means and standard deviations are available on QOLI versions in Greek (Zissi et al 1998—54 Greek
hostel residents) and Spanish (Vandiver, 1998—27 outpatients in Cuba; Duñó et al 2001—44
outpatients in Spain). Few demographic variables influence subjective QOLI scores: exceptions
appear to be marital status (Corrigan & Buican, 1995) and social relations ratings by men and women
(with women’s scores being slightly higher—Vandiver, 1998). Some studies show lower ratings by
non-Caucasian US respondents (Levitt et al 1990); others do not (Corrigan & Buican, 1995).
Summary
The QOLI covers a wide range of domains that are of interest to both consumers and clinicians, in
which objective and subjective responses are obtained. There is not strong support for the model
underpinning the development of the QOLI, that subjective well being is determined by the
combination of personal characteristics, objective indicators and subjective quality of life. Instead,
current research suggests a separation of life circumstances and life satisfaction. Subjective
satisfaction is related to emotional status, and especially to sadness or depression: we might expect
this association to reflect mutual influence. Self-reports of objective functioning on QOLI tend to
correlate moderately with self-reports of functioning on other instruments, but they do not reliably
relate to clinician ratings of functioning. While the latter result could be due to a limited awareness by
consumers of their own behaviour or of behavioural standards, it could also reflect differing
observational access.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 82
The 45-minute QOLI is too long for routine administration, but further development of a shortened
version may result in a practical instrument.
References
Barker, S., Barron, N., McFarland, B. H., & Bigelow, D. A. (1994). A community ability scale for chronically
mentally ill consumers: I. Reliability and validity. Community Mental Health Journal, 30, 363-379.
Barry, M. M., & Crosby, C. (1996). Quality of life as an evaluative measure in assessing the impact of
community care on people with long-term psychiatric disorders. British Journal of Psychiatry, 168, 210216.
Corrigan, P. W., & Buican, B. (1995). The construct validity of subjective quality of life for the severely
mentally ill. Journal of Nervous and Mental Disease, 183, 281-285.
Cramer, J. A., Rosenheck, R., Xu, W., Thomas, J., Henderson, W., & Charney, D. S. (2000). Quality of life in
schizophrenia: A comparison of instruments. Schizophrenia Bulletin, 26, 659-666.
Dickerson, F. B., Parente, F., & Ringel, N. (2000). The relationship among three measures of social functioning
in outpatients with schizophrenia. Journal of Clinical Psychology, 56, 1509-1519.
Dixon, L., McNary, S., & Lehman, A. F. (1998). Remission of substance use disorder among psychiatric
inpatients with mental illness. American Journal of Psychiatry, 155, 239-243.
Duñó, R., Pousa, E., Domènech, C., Díez, A., Ruiz, A., & Guillamat, R. (2001). Subjective quality of life in
schizophrenic outpatients in a Catalan urban site. The Journal of Nervous and Mental Disease, 189, 685690.
Khatri, N., Romney, D. M., & Pelletier, G. (2001). Validity of self-reports about quality of life among patients
with schizophrenia. Psychiatric Services, 52, 534-535.
Lehman, A. F. (1988). A Quality of Life Interview for the chronically mentally ill. Evaluation and Program
Planning, 11, 51-62.
Lehman, A. F. (1995). Toolkit for evaluating quality of life for persons with severe mental illness. Cambridge,
MA: Evaluation Center, Human Services Research Institute.
Levitt, A. J., Hogan, T. P., & Bucosky, C. M. (1990). Quality of life in chronically mentally ill patients in day
treatment. Psychological medicine, 20, 703-710.
Russo, J., Roy-Byrne, P., Reeder, D. et al (1997). Longitudinal assessment of quality of life in acute psychiatric
inpatients: reliability and validity. Journal of Nervous and Mental Disease, 185, 166-175.
Sullivan, G., Wells, K. B., & Leake, B. (1992). Clinical factors associated with better quality of life in a severely
mentally ill population. Hospital and Community Psychiatry, 43, 794-798.
Trauer, T., Duckmanton, R. A., & Chu, E. (1998). A study of the quality of life of the severely mentally ill.
International Journal of Social Psychiatry, 44, 79-91.
Van Dongen, C. J. (1996). Quality of life and self-esteem in working and nonworking persons with mental
illness. Community Mental Health Journal, 32, 535-548.
Vandiver, V. L. (1998). Quality of life, gender and schizophrenia: A cross-national survey in Canada, Cuba, and
USA. Community Mental Health Journal, 34, 501-511.
Zissi, A., Barry, M. M., & Cochrane, R. (1998). A mediational model of quality of life for individuals with
severe mental health problems. Psychological Medicine, 28, 1221-1230.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 83
Emergent Measures
This section briefly reviews other potential measures for which at this point in time there is little
published data regarding their psychometric properties but nevertheless could be considered as
contending measures of consumer outcomes.
Health of the Nation Outcome Scales-Self Report (HoNOS-SR)
Description
The HoNOS (Wing, Beevor, Curtis, Park, Hadden, & Burns, 1998; Wing, Lelliott & Beevor, 2000) is
a 12-item form that is completed by clinical raters, based on their knowledge of the consumer. Each
item is rated from 0—no problem to 4—severe problem.
The HoNOS has been tested and used extensively. It has good reliability and validity when used by
well trained raters, and it is sensitive to clinical change. The HoNOS exists in a number of versions,
including a version for child and adolescent mental health services (HoNOSCA) (Garralda & Yates,
2000), and one for older persons (HoNOS 65+). The HoNOS is a core component of the National
Outcomes & Casemix Collection, and extensive training of raters has been or is being conducted in
several states. A consumer rating measure that mapped onto this clinician measure would have
obvious utility, allowing a direct comparison of clinician and consumer views on status and progress.
The HoNOS-SR involves giving the HoNOS form to consumers for completion, with some changes in
wording of items to “make them suitable for self-rating by an individual with mental health problems”
(Trauer & Callaly, 2002, p. 25). Only one paper is currently available on this scale (Trauer & Callaly,
2002). A self-report form of the HoNOSCA (HoNOSCA-SR) also has been prepared (Gowers,
Levine, Bailey-Rogers, Shore & Burhouse, 2002) on a similar basis.
Domains of applicability
HoNOS items cover symptoms (Mood disturbance/depression; Problems associated with
hallucinations and delusions), problematic behaviours (Problems from aggressive or disruptive
behaviour; Suicidal thoughts or non-accidental self-injury; Problems associated with alcohol or drug
misuse) and impairment (Cognitive problems—involving memory, orientation or understanding;
Problems from physical illness or disability). There is also one item each on Relationship problems,
Activities of daily living, Accommodation, and Occupational problems.
Practicality
The scale has only 12 items, but in other respects there is insufficient information as yet to judge the
practicality of using these instruments.
Reliability
The internal consistency and factorial structure of the self-report HoNOS have not been reported. The
one-week test-retest reliability in an adolescent inpatient setting of the HoNOSCA-SR showed that 3
items had kappas exceeding .60 (concentration, language, somatic disturbance), and another 5 had
kappas above .50 (Gowers et al., 2002). It is unclear whether this was because of actual changes that
occurred (eg in alcohol and other drug use), or rating error.
Sensitivity to change
Adolescent sample in Gowers et al. (2002) appeared to show reductions in the absolute values of
ratings over the course of treatment, but no data on whether these changes reached statistical
significance were reported.
Consumer vs clinician ratings
A comparison of the ratings by consumers and those by clinicians can be seen either as a reflection of
inter-rater reliability or validity. The latter assumes that the clinician offers a “gold standard”, but this
may not of course be the case, especially in areas where the clinician does not have full observational
access.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 84
Trauer and Callaly (2002) found that consumers of an adult mental health service tended to report a
higher level of problem than their case managers in the areas of depressed mood, cognitive problems,
activities of daily living and accommodation. Case managers had significantly higher ratings than
consumers on hallucinations and delusions. The remaining 7 items showed no significant differences
between the clinicians and consumers. Intra-class correlations (ICCs) between ratings of aggression,
self-harm and alcohol or other drug use were .60-.65, and the ICC between total scores was .60.
However agreement on other items was much lower, and in particular, ratings of cognitive problems
and hallucinations or delusions were near zero. These results do not give confidence that ratings on
items other than aggression, self-harm and alcohol or other drug use can be relied on, and that in
particular, self-ratings of hallucinations or delusions may miss occurrences detected by clinicians. It is
not clear whether higher ratings by consumers on some items was due to a greater sensitivity to actual
problems or to a differential interpretation of the items. It is also not clear how to interpret the higher
self-ratings of depression, especially since this item had an ICC of -.17 with clinician ratings. Results
from this study should be interpreted with caution, given a low return rate (32.8%, and only 25.6%
fully complete).
Adolescent inpatients (n = 96) in Gowers et al. (2002) who completed the HoNOSCA-SR also had a
relatively low correspondence with clinician ratings at intake (r = .27), but the correlation rose to .58
at discharge. The authors noted that those with psychoses tended to have lower problem ratings than
clinicians, but that consumers with emotional disorders tended to see themselves as having more
severe problems than did their clinicians. Issues such as these are unlikely to be unique to this
particular instrument.
Availability of normative data
Mean ratings by the adolescents in Gower et al. (2002) were provided on both the total and item
scores, but no standard deviations were reported. Current information on the scales is insufficient to
use in a normative manner.
Summary
Only two studies on the self-report versions of HoNOS have been published at this time. Neither
provide confidence in the accuracy of the scale, especially in relation to psychotic symptoms.
However in principle, the use of a very similar self-report instrument to a widely adopted instrument
that is rated by clinicians has great appeal. Such an instrument may even be more sensitive than a
clinician rating in some domains (and current data suggest that depression may be one of these). The
instruments deserve further research attention.
References
Gowers, S., Levine, W., Bailey-Rogers, S., Shore, A., & Burhouse, E. (2002). Use of a routine, self-report
outcome measure (HoNOSCA-SR) in two adolescent mental health services. British Journal of Psychiatry, 180,
266-269.
Trauer, T., & Callaly, T. (2002). Concordance between patients and their case managers using the Health of the Nation
Scales (HoNOS). Psychiatric Services, 10, 24-28.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 85
Manchester Short Assessment of Quality of Life (MANSA)
Description
The Manchester Short Assessment of Quality of Life (MANSA) is a shorter modified version of the
Lancashire Quality of Life Profile (LQLP). The scale includes 25 questions, only 16 of which are
required to be completed each time the instrument is administered (the other 9 address primarily
demographic information considered to be relatively stable over time). Four objective items are
answered yes or no and twelve subjective items are rated on a 7-point satisfaction scale (1 = Couldn’t
be worse, 7 = Couldn’t be better). The MANSA, like the LQLP is based on Lehman’s Quality of Life
Interview (Lehman et al 1982) and subsequently the scale is generally administered by clinician
interview. The LQLP, which is used in the European Psychiatric Services: Inputs Linked to outcome
Domains and Need (EPSILON) study, consists of 105 items and therefore was considered too long to
be eligible for inclusion in the current review of consumer outcome measures.
Domains of applicability
The MANSA consists of three sections (Priebe et al 1999). The first includes personal details that are
supposed to be consistent over time (date of birth, gender, ethnic origin, and diagnosis). The second
includes personal details that may vary over time and have to be re-documented if change has
occurred specifically education; employment status including kind of occupation and working hours
per week; monthly income; state benefits; living situation including number of children, people the
consumer lives with, and type of residence. The four objective items assess the existence of a "close
friend", number of contacts with friends per week, accusation of a crime and victimisation of physical
violence. The twelve subjective questions obtain satisfaction with life as a whole, job (or sheltered
employment, or training/education, or unemployment/retirement), financial situation, number and
quality of friendships, leisure activities, accommodation, personal safety, people that the consumer
lives with (or living alone), sex life, relationship with family, physical health, and mental health.
Practicality
Administration of the MANSA takes approximately 5 minutes.
Reliability
Cronbach’s alpha for MANSA satisfaction ratings was 0.74 in Priebe et al (1999).
Sensitivity to change
Not addressed in the available literature.
Validity
Priebe et al (1999) state that high correlations of MANSA and LQLP scores suggest a concurrent
validity for the MANSA.
Availability of normative data
Not reported.
Summary
The MANSA provides a useful measure of quality of life. As with other measures of quality of life its
weakness in the current context is its inability to provide symptom specific outcome information.
References
Priebe, S., Huxley P, Knight S & Evans S. (1999). Application and results of the Manchester Short Assessment of quality of
life (MANSA). International Journal of Social Psychiatry, 45:1, pp 7-12.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 86
Recovery Assessment Scale
Description
The Recovery Assessment Scale (Corrigan et al 1999; Giffort et al 2000) is a 41-item self-report
questionnaire designed to measure recovery from severe mental illness. The scale was developed over
two stages. Initially, 4 consumers with severe mental illness recounted their stories of recovery, and
analysis of the resulting narrative identified 39 items. Subsequently, the items were reviewed by an
independent group of 12 consumers. The measure was then adjusted to produce the 41-item scale,
based on their feedback (Corrigan et al 1999). The scale can be administered using an interview
format or it can be self-administered. Consumers are required to indicate whether they agree with the
listed statements using a 5-point agreement scale (5 = strongly agree). Sample items include “I have a
desire to succeed” and “I can handle it if I get sick again”. In general, the items examine agency and
goal setting, self-esteem, ability to cope with symptoms, social support, and quality of life.
To date, only one published study has examined the psychometric properties of the Recovery
Assessment Scale (Corrigan et al 1999). The study included 35 participants diagnosed with severe
mental illness, with a minimum of 3 psychiatric hospitalisations in the preceding 2 years. The scale
appears to be well suited for assessing mental health outcomes, but its application in this context has
not yet been examined.
Domains of applicability
The Recovery Scale provides a single score, but its content appears on face value to cover aspects of
coping (eg “Although my symptoms may get worse, I know I can handle it”, “Coping with my mental
illness is no longer the main focus of my life”), empowerment and well being (eg “I like myself”, “I
have a purpose in life”) and personal relationships (eg I have people I can count on”, “Even when I
don’t believe in myself, other people do”).
Reliability
The Recovery Assessment Scale is reported to have high internal consistency (alpha = .93; Corrigan
et al 1999), and also shows high test-rested reliability when it is administered on two occasions within
2 weeks (r = .88; Corrigan et al 1999).
Sensitivity to change
There is no published data regarding instrument’s sensitivity to change.
Validity
The Recovery Scale score correlates strongly with the Rosenberg Self-Esteem Scale (r = .55);
Empowerment Scale - Self Orientation (r = -.71), the Social Support Questionnaire Size of Support
Network (r = .48); and Quality of Life (r = .62; Corrigan et al 1999). Higher scores on the Recovery
Scale are associated with fewer psychiatric symptoms as measured by the BPRS (r = -.44; Corrigan et
al 1999). Stepwise regression analysis indicates that three variables, self-esteem on the Rosenberg
Scale, self-orientation on the Empowerment Scale, and age, account for 61% of the variance in the
Recovery Scale score (Corrigan et al 1999).
Availability of normative data
The Recovery Assessment Scale has been used with a culturally diverse sample comprising African
Americans, European Americans and Asian Americans, and has been administered in English
(Corrigan et al 1999). There is no data regarding its suitability for other cultural groups. All members
of the sample were adults, and while there is no literature on the scale’s applicability for children and
adolescents, Recovery Scale scores were found to be lower in older participants. Since the scale has
not yet been translated, caution may be required when assessing those for whom English is a second
language. No published details of norms could be located.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 87
Practicality
Recovery Assessment Scale items were based on consumer accounts of the recovery process. Thus it
would appear to be an appropriate instrument for measuring this construct. The Recovery Assessment
Scale takes approximately 20 minutes to complete when administered in an interview format. Given
the absence of norms at this stage for either psychiatric or non-psychiatric populations, interpretation
of scores may be problematical. The scale is available in paper and pencil format only.
Summary
The Recovery Assessment Scale is designed to measure recovery from severe mental illness.
Currently there is only one published study regarding the psychometric properties of the scale, so the
results need to be independently replicated. While the scale may be an appropriate measure for
examining mental health outcomes, there is no data as yet on its sensitivity to change. Its length (41
items) makes it relatively impractical for inclusion in a routine assessment at present, but there may be
a possibility of developing a much briefer version in further research.
References
Corrigan, P.W., Giffort, D., Rashid, F., Leary, M., Okeke, I. (1999). Recovery as a psychological construct.
Community Mental Health Journal, 35, 231-239.
Giffort, D., Schmook, A., Woody, C., Vollendorf, C. & Gervain, M. (2000). Recovery Assessment Scale. In
R.O. Ralph & K. Kidder (Eds.), What is recovery? A compendium of recovery and recovery related instruments
(51-54). Cambridge, MA: Human Services Research Institute.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 88
Other measures
There are several consumer self-report measures currently under development in the USA. These
measures generally form part of a suite of measures making up a measuring system for the local
mental health services. A strength of the measures is that they have been developed in collaboration
with people with serious mental illness for the specific purpose of measuring consumer outcomes and
a weakness is that there is limited published empirical data regarding their psychometric properties.
The Mental Health Statistics Improvement Program (MHSIP) Consumer Survey was developed
with input from a wide range of stakeholders and is currently used widely in public mental health
systems in the USA. Four domains are assessed access, quality/appropriateness of care, outcome, and
general satisfaction. “Consumers surveyed in three states confirmed that the survey addressed
important concerns, was easy to understand, and was easy to complete” (Eisen et al 2001). The
measure is flexible in regards to the time frame covered and there is a space available at the end of the
measure for consumers to comment on their answers. There is also an inpatient version of the
measure, a youth version and a youth services survey for families version.
More detailed information regarding the measures can be found at http://www.mhsip.org/.
The Multnomah Community Ability Scale Self Report (MCAS-SR) is a self-report version of the
more widely known and used MCAS which is a clinician rated measure developed to measure
functioning among people with severe mental illness. The MCAS-SR is a 17-item measure scored on
a 5-point scale. O’Malia et al (2002) developed and have assessed the MCAS-SR. The authors report
the acceptability, reliability and construct validity for the scale using a sample of consumers of mental
health services. The scale was found to be acceptable and 80% of the consumers could complete the
scale without assistance. The test-retest correlation was 0.91 and Cronbach's alpha for measuring
internal consistency was 0.80. Construct validation showed high correlations with the Brief Symptom
Inventory and the Mental Component Scale of the SF-12. A modest relationship was found between
case manager ratings on the MCAS and the consumer self-report ratings on the MCAS-SR. The
authors conclude that the MCAS-SR is “a reliable instrument that is acceptable and relevant to
consumers” (330).
The Clients Assessment of Strengths, Interests and Goals (CASIG) is administered as a structured
interview and measures consumers goals for improvement in five areas of community functioning
(open ended questioning), their social and independent living skills (assessed with 4 to 9 yes/no items
from the Independent Living Skills Survey), medication and side effects (8 yes/no items), quality of
life (10 self-rating items), quality of treatment (5 self-rating items), symptoms (six symptoms assessed
as present/absent items that consist of probes and follow-up questions), and performance of
unacceptable community behaviours (10 yes/no items). There is also an informant-completed
counterpart to the CASIG called the Staff Observations and Client Information (SOCI). Wallace,
Lecomte, Wilde and Liberman, (2001) provide a detailed overview of the CASIG and report on the
reliability and validity of the measure.
There are several other scales that have potential for assessment of functioning or quality of life. For
example, the Social Functioning Scale (Birchwood et al 1990) is a self-report questionnaire to
examine the social behaviours and community functioning of people with schizophrenia. It covers
social engagement, interpersonal behaviour, independence-performance, recreation, prosocial,
independence-competence and employment-occupation. While this and other similar scales are both
reliable and valid, their length (71 items in this case) render them too time-consuming in their current
form, for use as outcome measures that are applied to all consumers.
The Quality of Life Inventory (QOLI-F; Frisch et al 1992) is a rating of the importance (0: not at all,
to 2: extremely important) and satisfaction (-3: very dissatisfied to 3: very satisfied), of 17 life
domains. Overall life satisfaction is scored as the average of the products of the ratings of importance
and satisfaction (omitting domains with zero importance to the person). The life domains are:
(physical) health, self-regard, philosophy of life, standard of living, work, recreation, learning,
creativity, social service, civic action, love relationship, friendships, relationships with children,
relationships with relatives, home, neighbourhood, community. The QOLI-F addresses the consumer-
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 89
identified domain of quality of life, together with satisfaction and importance of personal
relationships, together with aspects of functioning, well being, empowerment and physical health.
Time to administer the QOLI-F is not reported, but it consists of only 34 ratings. Very high internal
consistency and test-retest reliability (2-4 weeks) are reported (Frisch et al 1992). Moderate positive
correlations are seen with other measures of life satisfaction, and moderate negative correlations are
obtained with anxiety and depression measures and with SCL-90-R (Frisch et al 1992). Frisch et al
(1992) provided normative data on US samples of veterans participating in an inpatient alcohol
treatment program (n = 51), former consumers of the program (n = 51), undergraduates attending a
counselling centre (n = 127), unselected undergraduate students (n = 272), and small samples of
private inpatients with substance misuse and offenders on probation. The QOLI-F discriminated
between the current veteran inpatients and former consumers in Frisch et al (1992), suggesting that it
may also prove to be sensitive to change. The inventory appears to have potential for assessment of
life satisfaction in consumers of psychiatric services, but this application and a sensitivity to
individual improvement need to be established.
In relation to symptom assessment, the various versions of the Hopkins Symptom Checklist, which
are extensively used self-report measures of distress and psychopathology, also require mention.
These include the SCL-90, the Brief Symptom Inventory (BSI-53), the SCL-90-R, and the Hopkins
Symptom Checklist 25- and 21-item versions, together with recently developed 6- and 10-item
versions of SCL (SCL-6 and SCL-10; Rosen et al (2000)). The various versions of SCL, as implied by
the name, primarily measure symptoms and therefore do not adequately address the range of domains
that are of importance to consumers. Furthermore, most are too long to be eligible for consideration in
the current context. The shorter versions of the scale, specifically SCL-10 and SCL-6, have not been
broadly applied and therefore cannot be systematically reviewed at this time. However, Rosen et al
(2000) reported that the brief indexes had similar convergent validity, correlating .87 to .97 with the
SCL-90 and Brief Symptom Inventory, .49 to .76 with other symptom scales, and .46 to .73 with
changes in other symptom measures over time and the authors conclude that the results indicate the
concise, easily administered indexes are valid indicators of psychological distress. One of these
shorter scales might be incorporated as the symptom component of a multi-faceted consumer-rated
measure, but would not by itself offer a comprehensive assessment of all the areas seen as important
by consumers.
The Self-Reporting Questionnaire (SRQ: Beusenberg & Orley, 1994) has been developed by the
World Health Organization as a brief screening measure for psychological distress that can be
completed as a self-report instrument or self-rating interview and is widely applicable in different
cultures. The most common form (SRQ-20) has 20 questions answered Yes/No which predominantly
focus on symptoms in the previous 30 days (eg “Do you feel unhappy?”), but also include some
questions on functional impact (eg “Is your daily work suffering?”). The 1994 manual reported that it
was available in 17 languages at that time. The overall total on the SRQ has acceptable internal
consistency and it performs well as a screening instrument for people with psychological distress
(most commonly with a >7 cut off). The factor structure appears to be unstable across different
studies. We have not been able to find studies on the use of the SRQ as a measure of individual
change, so it is not included in the main reviews. Furthermore, like the SCL measures, it focuses only
on psychological distress.
References
Becker, M (1998). A US experience: Consumer responsive quality of life measurement. Canadian-Journal-ofCommunity-Mental-Health. Winter 1998; Suppl 3: 41-52.
Beusenberg, M., & Orley, J. (1994). A user’s guide to the Self-Reporting Questionnaire (SRQ). Geneva: World
Health Organization.
Eisen, S., Shaul, J., Leff, H., Stringfellow, V., Clarridge, B. and Cleary, P. (2001) Toward a National Consumer
Survey: Evaluation of the CABHS and MHSIP Instruments. Journal of Behavioral Health Services &
Research, 28(3), 347-369.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 90
Frisch, M. B., Cornell, J., Villaneuva, M., & Rtzlaff, P. J. (1992). Clinical validation of the quality of Life
Inventory: A measure of life satisfaction for use in treatment planning and outcome assessment.
Psychological Assessment, 4, 92-101.
O’Malia, L., McFarland, B., Barker, S and Barron, N. (2002) A Level-of-Functioning Self-Report Measure for
Consumers with Severe Mental Illness. Psychiatric Services, 53(3) 326-331.
Rosen, C.S., Drescher, K.D., Moos, R.H., Finney, J.W., Murphy, R.T. and Gusman, F. (2000) Six and ten-item
indexes of psychological distress based on the Symptom Checklist-90. Assessment 7(2):103-111.
Wallace, C., Lecomte, T., Wilde, J and Liberman, R. (2001) CASIG: a consumer centered assessment for
planning individualized treatment and evaluating program outcomes. Schizophrenia Research, 50(2001)
105-119.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 91
Summary
Each of these measures has significant strengths, notably in relation to its psychometric
characteristics. Some such as the K10 have inadequate published data at present on their sensitivity to
change, which is a significant issue if they are being used to evaluate change. No well-established,
brief measure covers the full range of consumer-identified domains or includes a qualitative
component.
For example, some measures are designed to measure the incidence of psychological distress or
symptoms, but most do not address the wish of consumers to describe their ways of coping or their
achievements in the face of ongoing symptoms. Hopefulness, empowerment, coping and health
literacy are not addressed by the three instruments in most common use in Australia, except to the
extent that the MHI covers a sense of well being.
Even in the area of symptoms, none of the well-established and brief self-report instruments that were
reviewed above adequately measures psychotic symptoms (although some including the BASIS-32,
MHI and HoNOS-SR do have some items on psychotic symptoms, reports of symptoms such as
hallucinations or delusions do not perform as well as those measuring anxiety, depression or distress).
One issue is whether acute psychotic symptoms, which are often associated with a loss of insight,
need to be captured through a consumer-rated instrument.
While some have data on multiple ethnic or language groups, we found no published data on the
applicability of any of these measures to Australian Indigenous communities.
Some instruments set out to measure outcomes against both psychiatric symptoms and empirical
functional abilities, and go some way towards the breadth of outcomes consumers and carers wish to
address, but are limited in the scope of the domains they cover. Others embrace a wide range of the
domains of interest to both consumers and clinicians, and solicit both objective and subjective
information, but correlate unreliably with clinical ratings of functioning or are limited in their capacity
to measure symptoms of serious mental illness.
An issue in relation to practicality and workload is that routine application of a measure or set of
measures will require brevity (given the reported preferences of informants, the limitations of time,
and the speed of item completion by some consumers). It may be impractical to measure all areas of
interest. In short, the descriptions of the instruments set out above indicate that most of the measures
already in use confront the system with a forced choice among the range of legitimate purposes for
consumer outcome measurement.
A question arises whether the priority for consumer-rated measures should be on symptom status (a
key focus of the self-rated instruments that have currently been adopted), given the recent adoption by
Australian states of clinician-rated instruments such as HoNOS, which cover this area. A similar point
could also be made about whether a priority for self-ratings should be the measurement of objective
functional attainments, given the inclusion of the LSP in the national protocol. One alternative would
be to leave symptom reports to clinician ratings, and instead to focus on one or more other areas of
prime interest to consumers.
On the other hand, it may be argued that consumer or carer measures should attempt to include some
similar domains and items to clinician measures, in order to obtain confirmation of therapist ratings
from the consumer’s perspective without the confounding problem of different item content. A set of
parallel measures, including clinician-, consumer-, and carer-rated scales with a core set of variables
in common and variables for each stakeholder group, would produce outcome measurement of great
value to consumers, and to an accountable mental health system. Consistent with this view is the
development of HoNOS-SR, although its underpinning research is very preliminary at present, and
only the behavioural items from that measure are currently seeing strong convergence with clinician
assessments.
We conclude that there is an acute need for further development of a brief measure relating to coping,
resilience, recovery, empowerment, hope, and similar consumer-identified domains. The review has
identified some instruments that have potential for such development. Both consumers and carers are
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 92
also acutely aware that fluctuations in a person’s condition has an inescapable effect on self-rating,
and there is therefore a powerful argument for developing a collateral carer measure to complement
the consumer self-rated measure.
Such a measure could be used in conjunction with existing measures adopted by Australian states,
providing a well-rounded assessment that captures many of the domains of primary interest to both
clinicians and consumers or carers.
There is also a need to determine which of the currently adopted measures appears to work most
effectively at tracking outcomes, given the longer term advantage of having equivalent measurement
strategies across the states (an important objective in order to achieve ongoing assessments of
consumers who move between jurisdictions and to allow unambiguous comparisons of consumer
populations across jurisdictions). While some independent studies of the measures in each jurisdiction
provide some information that would help in making such a decision in the future, a head-to-head
comparison of the three measures will ultimately be required in order to make a considered judgment
on a future common measurement policy.
Recommended elements in a research strategy
The following elements in a coherent research strategy to advance this field are therefore suggested.
The numbering does not necessarily denote priority or temporal order.
1. Collection of data on adopted measures
It is suggested that data continue to be collected and published on the performance of the measures in
the Australian context, including the provision of local data on consumer populations where this is
required. These studies focus on the collation and analysis of both the self-report and clinician rated
measures in each jurisdiction.
2. Direct comparison of adopted measures
We recommend that one or more studies be planned, in which a group of consumers are given all
three consumer-rated measures and are rated using the clinician-rated measures on more than one
occasion over the course of treatment. This will enable the relative sensitivity of the measures both to
clinical status and to treatment-induced changes to be assessed. The study should also re-examine the
relative acceptability and perceived utility of the three measures. Such a study is essential to future
decisions on the long-term use of the measures in Australia.
3. Development of one or more brief measures tapping additional domains
None of the measures currently in use by the state and territory health services satisfactorily captures
the majority of the domains of key interest to consumers and carers. While there are some other
existing measures that cover domains relating to coping, resilience, well being or recovery, some
further development is needed in order to obtain one or more instruments that validly assess domains
such as these within a measure of practical length for widespread application. The development of
such a measure could be done in combination with a direct comparison of existing adopted measures,
or as a separate research project.
4. Further development of multi-ethnic versions of instruments and instruments applicable to
Aboriginal and Torres Strait Islanders
While there are some existing data addressing this issue and there are studies planned or underway,
there is a need to ensure that both the adopted measures and any developed measure are applicable to
the Australian population as a whole, or that alternative measures are in place if a single measure is
impossible. Adaptations of measures will require attempted translations and investigations of the
acceptability, perceived meaning (including back-translation) and cultural relevance of items, together
with the collection of normative data within clinical populations and evaluations of the reliability and
validity of the resultant instruments.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 93
Part 5: Conclusions and recommendations
There is clear support for the inclusion of consumer self-rated outcome measures in the suite of
measures that are necessary in an accountable mental health system focused on consumers and their
families and carers. The support comes from several sources: academic literature on the management
of health care systems; research on the impact of consumer and carer participation on outcomes, and
on the reputation and standing of health care providers and facilities; Australian clinicians; and most
importantly, from the consumers themselves and their families and carers.
There is substantial consensus among consumers, families and carers, and clinicians in the mental
health system in Australia about the range of purposes for which consumer self rated outcome
measurement can and should be used. This consensus has implications not only for outcome
measurement, but also for the culture, workforce, and goals of the mental health system.
In order of priority for consumers and carers, the purposes for which consumer self-rated outcome
measures can or should be used encompass:
•
•
•
•
•
increasing the person’s capacity to self-monitor progress over time,
increasing the quality and appropriateness of the relationship between consumer and practitioner,
and the focus of treatment and support interventions,
informing planning and direction of the amount and nature of service to meet client and family /
carer needs better,
informing the evaluation and review of the individual’s management, and
informing the evaluation and review of the management of the service system over all.
There is a high level of consensus among consumers, families and carers about the domains of life the
system should be interested in knowing about in relation to consumer outcomes. There is substantial
overlap between what consumers think are the relevant and important domains, and what clinicians
think. In particular, consumers and their families or carers are keen that consumer self-rated outcome
measures focus the attention of both the individual and the system on tracking resilience and coping
skills in the face of what are predominantly chronic relapsing disorders.
We met no consumers or carers who believed that a focus on symptoms and distress was an adequate
basis of judging outcomes for either individuals or the system.
The adoption of more holistic measures is a necessary but not sufficient step in the culture change
necessary in the mental health system to bring it in line with changes in general health and aged care
systems. In particular, families and carers, and consumers themselves, experience a stark contrast in
how they are treated as a valuable unit in the management of other acute or chronic conditions in the
mainstream, and how families and carers are excluded or pathologised in the mental health system.
This disparity argues strongly for developing a collateral family/carer outcome measure, an action we
strongly recommend.
Ideally, in tracking consumer outcome, the literature and most stakeholders agree that in the mental
health field, owing to intermittent losses of capacity and insight, ideally there should be some
elements in common across consumer, carer and clinician assessments. From this point of view, the
HoNOS-SR merits further research attention.
All the self-rating measures that have been adopted by Australian states in recent years have
significant strengths. However, none of them covers all the criteria for measurement that were
important to consumers and carers, or fulfilled all the other criteria in the evaluation framework.
While each jurisdiction is collecting large datasets on the use of the NOCC measures that have been
adopted, there appears to be no plan to undertake comparative research applying more than one
measure to the same set of consumers. Without such data it will be difficult to make direct
comparisons of the current measures in terms of their widespread practicality and acceptability and
their sensitivity to change. In the longer term, it is clearly preferable that Australia adopt a single set
of consumer measures, so that direct comparisons can be made across jurisdictions. A decision to
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 94
select one measure would be easier to sustain if a set of comparative studies showed that one was
clearly superior.
We propose that Victoria recommend to the Information Strategy Committee established under the
auspices of the National Mental Health Working Group that it consider commissioning research to
address the research priorities listed under the recommended research strategy (immediately preceding
this section), in order to address the following objectives:
1. Compare the performance of the measures currently in use by Australian states
2. Develop one or more additional measures that meet some of the key criteria of importance to
consumers and carers that are revealed by this project
3. Ensure that measures are applicable across ethnic groups and in Aboriginal and Torres Strait
Islander populations.
We recommend that these projects have substantial consumer and carer involvement and
collaboration to ensure the relevance of the work to the priorities expressed in the consultations during
this study.
The ultimate aim should be to develop a consumer-rated measure (or an integrated set of measures)
that has sound psychometric characteristics, is sufficiently practical and financially viable for
widespread use, has aspects that link closely with clinician measures, and is relevant to the Australian
context.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 95
Appendix 1: Draft project outline for progressing to the next steps
The report proposes commissioned research to address the following objectives:
1. Compare the performance of the measures currently in use by Australian states.
2. Develop one or more additional measures that meet some of the key criteria of importance to
consumers and carers that are revealed by this project.
3. Ensure that measures are applicable across ethnic groups and in Aboriginal and Torres Strait
Islander populations.
We have framed the draft project to allow for the three research objectives to be addressed by three
separate tenders, but it is possible that at least the first and second might be combined into a single
tender. We suggest that the third objective be the subject of separate tenders, because of the different
research skills and key stakeholders that are likely to be involved.
The auspice of the projects
In the case of at least the second research objective (development of additional measures), to ensure
that the next stage of development of outcome measurement is consumer and family or carer focused,
we recommend that expressions of interest be sought from a respected national mental health
consumer or carer organisation to auspice the project. The consumer or carer organisation must be
able to demonstrate: (a) its capacity to direct and supervise a project of national significance, and
(b) its willingness to enter into consortium arrangements with one or more in collaboration with
research groups or organisations who have the requisite research expertise to ensure successful
completion of the task.
In relation to the first and third objectives, the successful tenderer would have to establish that the
project would incorporate collaboration with consumers and carers. In the case of the third objective,
a partnership with relevant community groups would be required.
Management of the projects
The management of the projects would require a steering committee to be convened comprising
consumer, family/carer, clinician, jurisdictional and (in the case of objective 3) community
representation. The members of the steering committee should be chosen for their capacity to provide
representative perspectives on the instruments at each stage of their development. They must also be
able to demonstrate that they are in a position to facilitate access to consumers, families/carers, and
mental health systems for the studies required to fulfil the research objectives.
Where possible, consumer, family/carer, and jurisdictional representatives should also have a
demonstrated record of interest and involvement or understanding and expertise in outcome
measurement.
In partnership with the sponsoring body, the auspicing body should call for tenders from suitably
qualified academics or consortia with a proven track record in outcome measurement in mental health,
and who can demonstrate the capacity to develop a cost-effective program of instrument development
and testing. They must be highly skilled in technical aspects of questionnaire and survey design, but
must also be open to the view that in the first instance the mental health system and therefore its
outcomes should be consumer and family focused, not provider or facility focused.
The work to be commissioned should not replicate the detailed qualitative work on domains of
interest to consumers and carers and clinicians, nor the development of evaluation frameworks
available from earlier work and refined by this project.
Terms of reference
The objective of the projects would be consistent with the research objectives outlined above. In each
case they would evaluate (and where necessary, develop or refine) a set of nationally acceptable
measures of consumer-rated and family- or carer-rated outcomes of mental health services, consistent
with the evaluation framework.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 96
In undertaking the first research objective, the successful tenderers would:
•
Incorporate consumer and carer evaluations of the currently adopted instruments
•
Undertake the comparative instrument testing in line with current international best practice
•
Cost the infrastructure requirements of the measures, as well as those of replacing them, including
dissemination, training, software development, and data management.
In undertaking the second research objective, the successful tenderers would also:
•
Develop items that cover the majority of documented key domains of interest to consumers and
carers (Siggins Miller 2003). In particular, the tenderers would incorporate items relating to
coping, resilience, hopefulness or empowerment. Items would reflect the differences in capacity
of the rater to access and evaluate information on the domains being examined (eg symptoms,
technical aspects of treatment outcome, or the subjective experience of living with a mental
illness). Consideration would be given to the incorporation of a common core of items for
completion by clinicians, consumers and carers, to permit comparison and clinically useful
discussion between consumers and their clinicians.
•
Refine the measures based on pilot testing in appropriate jurisdictions and clinical contexts.
In undertaking the third objective, the tenderers would also:
•
Develop culturally and linguistically appropriate variants on existing adopted measures or the new
measures developed under Objective 2, in collaboration with relevant community representatives.
Evaluation criteria for research tenders:
•
A demonstrated, effective partnership between consumers, carers, clinicians, jurisdictional
representatives, and (in the case of the third objective) relevant community groups
•
Demonstrated understanding of the relative contributions of and differences between providerand consumer-focused perspectives in relation to mental health outcomes
•
The extent to which the proposed program of research draws on but does not replicate the
extensive qualitative work done here and internationally
•
The extent to which the proposed program of research demonstrates understanding and experience
in all the relevant aspects of survey and questionnaire design and analysis.
•
The extent to which the research process recognises the importance of building stakeholder
confidence in the measures and their routine application
•
The capacity of the researchers to design a research program that can successfully meet the
relevant research objectives within in three years.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 97
Appendix 2: Interview protocols
Interview protocol for State and Territory Officers
Thankyou for your time and agreeing to help us with this project. We have already sent you a copy of
our terms of reference. In case you haven’t had a chance to read them, here is a copy [attached] so
you can refer to it if you need to as we talk.
[Give an update of how the project is going in terms of the phases in the Terms of Reference, and
expectations about end dates and so on.]
•
First, could you tell us where your jurisdiction is up to in the use of self-rating outcome measures
for mental health consumers?
•
What instrument[s] have you used?
•
What were the reasons for choosing this instrument? What was the process you used to make the
choice?
•
How do you think it has worked?
•
Have there been any particular problems with its use?
•
On the basis of your experience so far, what advice would you give the system about a framework
for evaluating consumer self-rating measures? What do you think an instrument should measure?
•
What would be the practical implications for your jurisdiction if national recommendations were
made about changing or adapting the instrument?
•
Where do you see this going in 5 years time?
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 98
Consumer or carer focus group protocol
We have been engaged by the Victorian Department of Human Services to do a study of national
significance for the mental health system. It will help inform the way the system gets consumers’
views about the outcome of mental health treatment and support services. During the next month or
so, we shall be talking to consumers and carers in every State and Territory.
There is range of pen-and-paper questionnaires that are given to consumers to fill out. They ask how
people are feeling about their health and well being. They’re sometimes called ‘outcome measures’.
Here are some examples (hand out examples of questionnaires - MHI, K-10, BASIS32). In this State,
you may have already been asked to fill out the XX.
What we want to do today is take a step back from the existing instruments, and talk to you about the
way you think about your own health and well being (or for carers, how you think about the health
and well being of the person you care for).
Focus group questions (to be appropriately modified for the carer groups)
If you think about the things that are important to you socially, emotionally, and in your physical
health, can you tell us?
•
What sorts of things are happening in your life when you feel like things are going well for you?
•
What sorts of things are you feeling or experiencing that make you worried about yourself?
•
How are other people in your life relating to you?
•
If you had to tell someone else your list of things that together would add up to you feeling good
about the way your life is going, what would you tell them?
•
What are the things that make you think you are not doing well
•
What do you think it would be useful for your mental health service providers to know about how
things are going for you?
•
What areas of your life did you expect treatment to help you with?
Facilitator summarises key themes: What you seem to be saying is that the dimensions of how you
judge your health and well being include X, Y and Z.
On reflection, is there anything we haven’t talked about today that you think would be important for
the mental health system to know about its consumers?
(For consumers only). We are also interested in how you feel about being asked to fill in
questionnaires that ask you how you think you are going. The intention is that the forms consumers
fill in go onto their clinical files, and will be used by the doctors and nurses and other health
professionals to help them work better with you to improve the treatment and support you receive.
Have you ever filled out one of these questionnaires? If so –
•
•
•
Which one was it?
Do you have any feelings about whether it was a useful tool?
How do you think it could be improved?
Is there any advice you want to give the system about:
•
•
•
How to introduce the questionnaires to consumers?
How to discuss the results with consumers?
Any concerns about how the results of many people’s questionnaires can be used anonymously to
evaluate how well the system works for consumers?
(Prompt when necessary for impact on consumers’ time, concerns about confidentiality, how the
information may be used, what is the feedback process, how the need to do the survey is presented to
people.)
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 99
Interview protocol for clinicians
Thankyou for your time and agreeing to help us with this project. We have already sent you a copy of
our terms of reference. In case you haven’t had a chance to read them, here is a copy [attached] so
you can refer to it if you need to as we talk.
[Give an update of how the project is going in terms of the phases in the Terms of Reference, and
expectations about end dates and so on.]
As you are aware, there are a number of pen and paper questionnaires, and each of them cover
different domains of health and well being. At this stage of the project, we’d like to talk with you
about what you would ideally like to see covered in a consumer self-rating instrument - how you think
the concept should be presented to clients, and how the results should be used in case management
and planning and at the service system level for monitoring and evaluation purposes.
If you are already using a self-rated instrument with your clients, you may like to answer these
questions by referring to your current practice.
•
Are you currently using a particular instrument? If so can you tell us which one?
•
How have used consumer self-rated measures at the clinical and or the system or program level,
what have been its strengths and weaknesses and so on?
Thinking about the things that are important to consumers socially, emotionally and in terms of
physical health, can you tell us:
•
What sorts of things would you see happening in a consumer’s life that would lead you to judge
that things are going well for them?
•
What sorts of things would you see or hear about from consumers that would make you worried
about them?
•
If you had to tell colleagues about the list of things that together add up to a picture of a consumer
doing well, what would you tell them?
•
What are the indicators that consumers are not doing well?
•
In your view what are the domains of information about outcome that consumers are the best or
only source?
•
In your view what sorts of ratings can be validly obtained from consumers?
We are also interested in how you feel about being asked to give consumers these sorts of
questionnaires.
•
How useful do you find these sorts of instruments in your clinical practice, do the results
influence how you work with consumers and with colleagues
•
Is there any advice you have to offer about the implications of introducing consumer self-rating
instruments in the system? (We would be grateful if you could think in terms of culture change,
training and development, cost.)
What advice would you give the system about:
•
how to introduce the questionnaires to consumers?
•
how to discuss the results with consumers?
•
any concerns about how the results could be aggregated at the system or program level to evaluate
how well the system works for consumers?
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 100
Interview protocol for academics
Thankyou for your time and agreeing to help us with this project. We have already sent you a copy of
our terms of reference. In case you haven’t had a chance to read them, here is a copy [attached] so
you can refer to it if you need to as we talk.
[Give an update of how the project is going in terms of the phases in the Terms of Reference, and
expectations about end dates and so on.]
At this stage of the project our job is to develop an evaluation framework for consumer self-rated
outcome measures. We will then assess the range of existing instruments identified by practitioners,
academics, and the literature against this framework.
Before we discuss with you your views on appropriate elements of an evaluation framework, can you
tell us if you are actively engaged in research or clinical research in this area, or if you are aware of
any instruments that are under development, and who we need to talk to about them?
Now, moving on to the elements of an evaluation framework for a self-rated outcome instrument.
Obviously there are the usual psychometric criteria – reliability, validity, suitability, and so on.
We would be grateful for your views on the relative importance of these dimensions for the possible
evaluation framework we shall draw up. (See outlines below)
What other dimensions would be important to include in the evaluation framework?
Apart from the need for translation of any instrument into relevant languages for people who are not
English speakers, are there in your view any cultural issues that would impact on the choice of
instrument?
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 101
Appendix 3: Reference group and informants
The heart of this project has been the invaluable personal insight and experience offered by the
96 consumers who participated in group discussions in every State and Territory; 33 members of
state-based Consumer Advisory Groups and consumer bodies; and 35 carers. For reasons of privacy
and respect, we have chosen not to disclose personal or clinical details about the consumers and carers
who generously participated, and their names are therefore not listed here.
We also acknowledge the support and advice of the Project’s Reference Group, and record our gratitude
to the stakeholders we consulted in every State and Territory, including clinicians and service providers,
mental health service management, representatives of government authorities, and groups and individuals
nominated by the Mental Health Branch.
Their names and affiliations are listed below. ∗
Consumer Self-Rating Reference Group
Ms Beth Bailey
Carer Representative
Ms Kristen Breed
Policy Officer, Quality and Effectiveness Section, Mental Health and
Special Programs Branch, Department of Health and Ageing
Ms Ruth Catchpoole
Manager, Systems and Outcomes, Mental Health Unit, Queensland Health
Ms Isabell Collins
Director, Victorian Mental Illness Awareness Council
Ms Helen Connor
Consumer Representative, Mental Health Council of Australia
Ms Karen Dickinson
Manager, Clinical Training Project, Mental Health Division, Department of
Health, Western Australia
Mr Andre Jenkins
Senior Consultant, Information Development & Knowledge Management,
Mental Health Unit, Department of Human Services, South Australia
Dr Graham Meadows
Senior Lecturer in Acute Psychiatry, University of Melbourne, Victoria
Mr Allen Morris Yates
Principal Information Officer, Strategic Planning Group for Private
Psychiatric Services (SPGPPS)
Mr Ian Munday
Team Leader, Northern District Adult Community Mental Health Team,
Tasmania
Ms Lorna Payne
Manager, Service Monitoring and Review, Mental Health Branch,
Department of Human Services, Victoria
Mr Graeme Purcell
Mental Health Business Information Officer, Department of Health and
Community Services, Northern Territory
Ms Deborah Shaw
Mental Health Services Staff Development Officer, ACT
Mr Gavin Stewart
Manager, Evaluation Program, Centre for Mental Health, NSW Health
Department
∗
The categories overlap to some extent, in that several of the officers we interviewed in State and Territory
departments were clinicians, and many academic informants also had a clinical role. In these cases, informants
are listed in only one category.
Siggins Miller Consultants
Consumer self-rated outcome measures
Page 102
Officers of State and Territory mental health services
ACT
Mr Ian Bull
Mr Brian Jacobs
Ms Deborah Shaw
NSW
Ms Caroline Muir
Prof Beverley Raphael
Mr Gavin Stewart
NT
Mr Terry Barker
Ms Helen Blunt
Ms Narelle DeMarte
Ms Colleen Doherty
Mr Michael Fletcher
Mr Trevor Galbraith
Mr James Jeffery
Mr Peter Mals
Ms Rebecca Orr
Mr Graeme Purcell
Ms Vicki Stanton
QLD
Ms Ruth Catchpoole
Ms Rhondda James
Ms Anne Louis
Ms Amanda Kivic
Ms Trudy Sebastio
Ms Alexis Stockwell
SA
Mr David Carey
Ms Bronwyn Hendry
Mr Malcolm Hugo
Mr Victor Wasylenko
TAS
Ms Chris Fox
Ms Annabel Hanke
Ms Sue Jones
Mr Ray Kemp
Ms Nicole Manktelow
Ms Irene Matthews
Mr Ian Munday
Dr Graeme Vaughan
Mr Jill Whyte
Siggins Miller Consultants
Manager, Health Informatics and Business Systems Unit
Acting Director, Mental Health Services
Staff development Officer, Mental Health Services
Principal Information Officer, Centre for Mental Health
Director, Centre for Mental Health
Manager, Evaluation Program, Centre for Mental Health
Senior Policy Officer, Mental Health
CNC, Mental Health Inpatient Unit, Alice Springs
Business Analyst, Community Care Information System
CNC, Mental Health Inpatient Unit, Alice Springs
Tennant Creek Mental Health
CNC Remote Mental Health Team
Tennant Creek Mental Health
Psychologist, Top End Mental Health Services
Mental Health Services Development, Top End Services Network
Mental Health Business Information Officer
Manager, Central Australian Mental Health Services
Manager, Systems and Outcomes, Mental Health Unit
Senior Project Officer QCAG, Mental Health Unit
Mental Health Unit
Team Leader, Inala AMH
Mental Health Unit
Outcomes Initiative, Mental Health Unit
Consumer Consultant, Mental Health Unit
Principal Project Officer, Country and Disability Services Division
Team Leader, MAC Team
NOCC Project Manager, Mental Health Unit
Team Leader, Adult Inpatient Services
Mental Health Services
Mental Health Services
Mental Health Services
Psychologist, Department of Psychiatry, Launceston
Mental Health Services
Team Leader, Northern District Adult Community Mental Health Team
Mental Health Program
Community Mental Health
Consumer self-rated outcome measures
Page 103
VIC
Ms Maria Bubnic
Ms Vrindée Edan
Mr Mark Smith
Ms Veronica Spillane
Mr Gilbert van
Hoeydonck
WA
Ms Karen Dickinson
Dr Aaron Groves
Ms Jennifer Hoffman
Ms Maureen Jubb
Ms Leanne Mirabella
Ms Danuta Pawalek
Ms Duane Pennebaker
Mr Neil Preston
Ms Jennifer Wright
Project Manager Quality, Mental Health Branch
Consumer Consultant, Eastern Health
Senior Project Officer, Mental Health Branch
Project Manager, Outcome Measurement, Mental Health Branch
Senior Project Officer, Mental Health Branch
Manager, Clinical Training Program
Director, Office of Mental Health
Training Consultant, Office of Mental Health
Training Consultant, Office of Mental Health
Training Consultant, Office of Mental Health
Director, Systems Development, Mental Health Division
Training Consultant, Office of Mental Health
Office of Mental Health
Office of Mental Health
Clinical and academic informants
Prof Gavin Andrews
Mr Edward Blacklock
Mr Peter Brann
Dr Gerard Byrne
Dr Tom Callaly
Ms Mary Cutajar
Mr David Dennis
Ms Laura Fisher
Ms Fran Gallagher
Dr Shane Gill
Dr Graeme Hawthorne
Ms Mary Hyland
Ms Janelle Killick
Dr David Lie
Ms Belinda Mackie
Prof Brett McDermott
Mr Brendan O’Hanlon
Dr Colin Riess
Dr Philip Robinson
Dr Rob Shields
Dr Terry Stedman
Dr Tom Trauer
Ms Elisabeth Webb
Mr John Woodhead
Siggins Miller Consultants
Professor of Psychiatry, UNSW, and Director of CRUFAD, NSW
Nurse Unit Manager, Wesley Private Hospital, Auchenflower Qld
Lecturer in Psychology, Monash University, Wundeela Centre, Vic
Psychogerontologist, Brisbane Qld
Director of Psychiatry, Barwon Health, Geelong Vic
Occupational Therapist, Northpark Private Hospital, Vic
Forensic Psychologist, Mental Health Executive, ACT
Research Psychologist, Adelaide Clinic, Gilberton SA
Clinical Care Coordinator, Belmont Private Hospital, Carina Qld
Director of Training, ECMHS, Royal Adelaide Hospital, SA
Deputy Director, Program Evaluation Unit, University of Melbourne, Vic
Quality Coordinator, Community and Mental Health, Barwon Health, Vic
Manager, Toowong Private Hospital, Toowong Qld
Director, Geriatric and rehabilitation Unit, Princess Alexandra Hospital, Qld
Quality and Outcomes Coordinator, Albert Road Clinical, Vic
Chief Psychiatrist, CYMHS Management Unit, Mater Hospital, Qld
Social Worker, Bouverie Centre, Flemington Vic
Psychiatrist and Director, Bouverie Centre, Flemington Vic
Chief, Division of Mental Health, Women’s & Children’s Hospital, SA
Director, Acute Inpatient Service, Alfred Healthcare Group, Prahran Vic
Director of Clinical Services, The Park Centre for Mental Health, Vic
Senior Lecturer, Psychology, Psychiatry & Psychological Medicine,
Monash University, Vic
Clinical Psychologist, Joondalup Health Campus, Joondalup WA
CEO, Pioneer Valley Private Hospital, Mackay Qld
Consumer self-rated outcome measures
Page 104