Download Measuring Service Level Outcome: how to use the Parent Concerns

Measuring Service Level Outcome: how to use the
Parent Concerns Questionnaire to Assess Outcome
in Many Cases
Summary
Measuring outcome is an important feature of looking at service effectiveness, and in
turn, planning services. It is important to supplement existing approaches to service
evaluation with robust additional methods, where available. The PCQ provides an
opportunity to contribute to service evaluation, with its capacity to provide
quantitative – numerical – measures of problem levels. Three broad approaches are
explained and illustrated here:



Measuring outcome/effectiveness using the PCQ scoring system with ‘before
‘and ‘after ‘service use measures
Measuring outcome/effectiveness using the presence or absence of at least
one severe problem identified by the PCQ before and after service use
Measuring outcome/effectiveness by comparing two different groups, one that
receives a service and another that does not
Introduction
One feature of the Parent Concerns Questionnaire (PCQ) is its capacity to measure
outcome – whether or not there was an improvement over a period in time in the
seriousness of a child and his or her family’s problems. This is a significant feature of
the work of any accountable child care agency, since it can provide hard information
about the degree of effectiveness of the agency.
We have seen elsewhere on this web site [in our examples of measuring outcomes
with individual cases] how this may be done with an individual case. However, this
may also be done with many cases, for example all attendees at a Children’s Centre,
or all families referred with additional needs.
Thus, for example, if Children’s Centres were to introduce a new resource – for
example parenting classes – it would be possible to see whether or not their
parenting had improved or whether child development was less problematic for those
who had taken those classes. We would do this by getting all participants in the
parenting classes to complete the PCQ immediately before they began the classes.
Then, when the classes were completed, we would then get them to complete the
PCQ.
We could then compare the scores for all the completed PCQs aggregated (that is
added together) at the initial assessment (just prior to beginning the parenting
classes) and at the follow up, or second completion (just after they had finished the
Parenting Classes). We could then see whether the situation for that group, as a
whole, had improved, got worse, or stayed pretty well the same.
How do we do this?
How to measure Outcome using the Parent Concerns
Questionnaire – Practical Features
There are two key ways in which outcome can be measured using the PCQ for the
whole group
(a) Using the PCQ Scoring System
First, we could aggregate (add together) the score for each participant’s PCQ and
then add them all together. We can explain this by reminding you how the scoring
system of the PCQ works and then showing how this works with a number of families.
The PCQ is divided into 37 Problem Areas or items. Each of these is scored as
follows
No problem
A problem that is not severe
A severe problem
=0
=1
=2
Scoring Each PCQ (for one family)
Consider the situation where you wish to make an initial assessment of families that
were about to go on a parenting course. You ask the first parent to complete the
PCQ.
So, let’s suppose a parent rated the following problems to be severe:
Child Behaviour
Parental Guidance and
Housing
And the following as problems that were not severe:
Marital/Partner
Financial
We would have 3 severe problems (3 x 2=6) and two problems that were not severe
(2 x 1=2). The overall total would be 8.
Aggregating scores for all families
Let us suppose now that there are six families involved in Parenting Classes and that
you wish to focus on them. You would add together the overall scores for each of the
PCQs completed. Let us suppose the scores were as follows
8
10
12
9
11
10
Adding these together would make an overall Total of 60 and the average (mean) for
each family would be 10.
Assessing Outcome
Let us now suppose that all six parents completed the parenting class. At the end of
the classes they were each asked to fill in the PCQ. This yielded scores (based on
the scoring system just outlined) as follows
0
15
5
3
5
2
Adding these together would yield a total of 30, and the average (mean) for each
family would be 5.
From this we could see that a clear improvement had occurred over all the
parents/families involved. On average, the problem score for each family had halved
(from 10 to 5).
However, this would not prevent us, on an individual basis to identify where
improvements had occurred. For example, we can see from the above figures that
one case deteriorated, from a score of 10 to one of 15. So within the general
improvement that was experienced by the group, there was one family that not only
had not benefited, but had got worse.
We might conclude that, overall, this group had improved, in some cases markedly,
but that was not the case for all service users.
Further features of the scoring system with many cases
It is possible to see whether an improvement by a group has been maintained. This
is straight-forwardly done by asking the service users to complete the PCQ a third
time, say at 3 or 6 months after the finish of the parenting classes
Suppose we got the following overall scores six months after the parenting classes
had finished
0
10
2
2
3
1
This would yield a total score of 18 and an average problem score for each family of
3 (18/6). This would indicate that the improvement had indeed been maintained, and
that there had even been some further improvement, as the average (mean) family
score had reduced from 5 to 3 **
(b) Measuring outcome through examining the presence of Severe
Problems
Our second approach would be to examine the presence and absence of severe
problems. Since the presence of a severe problem is the ‘trigger’ for an assessment,
this would indicate whether the family was finding at least one area of their life to be
severely problematic. Of course, the absence of a severe problem would indicate
that there was no area of their life (at least as measured by the PCQ) that was
severely problematic
How would this be done? Well in principle, there is much that is similar to the
process outlined above when using the PCQ to get a problem score. Ion this case,
though we concentrate on the presence or absence of severe problems.
Let us suppose, as above, we have a group of parents who will be taking parenting
classes. We will ask them to complete the PCQ just before the commencement of
the parenting class and just after it is finished. We might find at the initial
assessment that all families had at least one severe problem. At the follow up,
just after the parenting class finishes, we might find (for our six participants) the
following
No severe problems
Severe problems
Severe problems
No severe problems
Severe Problems
No severe problems
Here we find that where, at initial assessment, all 6 of the families had at least one
severe problem, following the parenting class only three of them had at least one
severe problem, while 3 had no severe problems at all.
The usefulness of this approach is apparent if we see the presence of at least one
severe problem as a reason for further assessment with a view to possible
intervention. It would be clear from these results that 3 of these families/parents no
longer needed any intervention because they had no severe problems.
The other three may need further intervention, and that could be determined by
assessing them further to take a closer look at their situation.
(c) Measuring Outcome by Comparing Groups
For most busy practitioners and managers this would provide good evidence for the
helpfulness of the parenting classes. This is perfectly reasonable, and where you do
have some hard evidence of improvements occurring in the face of some particular
intervention (such as a parenting class) this is a major step forwards from where no
such evidence is present.
However, there is always a danger that an improvement of this sort ‘just happened’
and that it had little to do with the parenting classes. How can we find out whether
those improvements are due to the parenting classes? We can do so by having a
comparison group of parents who did not have that class.
This may not be straight forward, but could be done in some situations. For example,
suppose we had a children’s centre, or a number of children’s centres, that wished to
offer parenting classes. Perhaps there were 12 parents who were lined up to go on
one. Where there were six parents in each parenting class and you were running
only one at a time, you might wish to choose parents (say) by the toss of a coin. You
could do so until you had 6 who were in the first parenting group and six who were
not.
Having got two groups you could go through the same process with the group not
initially receiving the parenting class as you had with the group who did. Both groups
would complete the PCQ immediately before the Parenting classes began and
immediately after they finished. Of course only one group would actually attend the
parenting class.
Suppose the following was the result. Suppose the overall score for the group not
receiving parenting classes (and hence also the average score for each family)
remained the same over the two completion of the PCQ. On the other hand the
score for those involved in the parenting class improved as outlined above. We
would be able to conclude that it was likely that the parenting class was the reason
why the improvement had occurred.
A Health Warning
It is important to recognise that it is difficult to be perfect in measuring outcomes
when you are busy practitioners and managers. When these evaluations are carried
out by researchers, they are aware of all sorts of pitfalls that they would seek to
guard against. For example, in this case there might be the danger of what is called
‘contamination’. If all the parents were chosen from one children’s centre (both those
receiving the Parenting Classes and those not receiving them), then they probably
know each other, and it is quite possible that the techniques learned by those
attending the parenting classes would be passed on to at least some of those not
going to the parenting classes. This could, of course, affect outcome.
A Realistic Choice
However, we can get too hung up on perfection. Even the most carefully
constructed experimental designs conducted by researchers on social interventions
invariably have ‘drop outs’ (people who start but do not complete an intervention).
Practitioners and managers operate with all the imperatives and pressures of day to
day work and need to be realistic about their evaluations. Using the PCQ to
measure outcomes, with or without a comparison group, provides important
information that can help practitioners and managers in determining which
services to provide and discovering which are effective.
This is a realistic and practical choice for measuring outcome, evaluating
interventions and helping with accountability.
An Example: Measuring Outcome using the PCQ Scoring System
We here present an example, using this method to see how things changed when a
group of families, who were applicants for children’s social care services were
assessed using the PCQ at the point of application, and at a second point six months
later. T
Change in Average Problem Score between Initial Assessment and
Assessment 6 months later (total sample=67)
Average (mean) PCQ Score at:
Initial
Assessment
6 months
later
Overall
change
Child Development
Parenting Capacity
Family and Environmental
factors
8.4
6.5
4.0
6.2
4.7
3.4
2.2
1.8
0.6
Total PCQ Score
18.9
14.3
4.6
We can see from this that these families did improve, but we can see that there was
considerable room for further improvement.
If we remember that a single severe problem scores 2, an overall score of 18.9 (the
score of the applicants at initial assessment) will almost certainly involve a
considerable number of severe problems across the majority of cases. The same
goes for the domains constituent of the PCQ – there was, at initial assessment,
almost certainly a number of severe problems in each of the domains.
At follow up we can see some improvement – the equivalent of an average of about
two severe problems per family. However, we can also see score which suggest, on
average, these families retained a considerable number of problems, and were, as a
whole considerably in need of support.
We can clarify this further by focusing on our second way of evaluating outcome, by
focusing on severe problems.
An Example: measuring outcome through the presence or
absence of severe problems
If we look at the PCQ as a whole, 56 families (out of 67) had at least one severe
problem at initial assessment. Hence eleven families had no severe problems.
Six months later 52 families had at least one severe problem, and therefore 15
families had no severe problems. We see, therefore, an improvement, in this
respect of only 4 – that is only 4 fewer families (of the 67) had at least one severe
problem six months after the initial assessment.
We can look at this in terms of the different domains as well. This is useful where we
want to look at specific aspects of the family’s functioning. We can see from this that
while 48 families had at least one severe child Development problem at initial
assessment, this had reduced to 38 families six months later. Put another way, the
number without at least one sever child development problem at initial assessment
was 19. This number increased, six months later to 29. This meant there was a
reduction of ten families in the number that had at least one severe child
development problem.
Number of Families with at least one severe problem: Each Domain and the
PCQ overall
Initial Assessment Six Months later
Number percent Number percent
Child Development
Parenting Capacity
Family and Environmental Factors
48
43
34
72
64
51
38
26
32
57
39
48
PCQ as a Whole
56
86
52
78
Looking in greater detail: an Example from the Child Development Domain
We can look at these changes in greater detail by focusing on the individual items or
areas that make up each domain.
Here we see that child emotional problems (where a child is upset distressed or
depressed) were severe in 31 of the 67 families at the initial assessment. Six
months later this had reduced – by ten – to 21
We see Child Behavioural Problems (where a child is frequently defiant or
aggressive or has tantrums) were severe in 39 of the 67 families at initial
assessment. This had reduced – by 14 – to 25 six months later.
Thirdly, we can see child cognitive problems (where they have difficulty
concentrating or learning as quickly as they should) were severe in 21 families at
initial assessment. This, though increased – by two – to 23 families six months
later.
Thus we see a more complex picture when we look at individual areas or items of
the PCQ. While child emotional and behavioural problems decreased over the six
months, child cognitive problems increased. Provided with this kind of information, a
manger would have a better idea about where resources might usefully be
concentrated (here we might argue that child cognitive problems provided one such
area).
However, this is always a judgement for the practitioner or manager. They may
consider, for example, that child behavioural problems merited more attention than
child cognitive problems. They may also feel that the improvement in this area was
not as great as s/he would have wanted. In that case, they might decide to put
further resources into tackling the problem of child behaviour precisely because of
the importance they ascribe to it.
Number of families with at least one severe problem in 3 individual problem
areas/items of the Child Development domain (number=67)
Initial Assessment
Child
Emotional
Child
Behavioural
Child Cognitive
Six Months later
Number
%
Number
%
31
46
21
31
39
58
25
37
21
31
23
34
An Example: Measuring Outcome by Comparing Groups
How to do – a hypothetical example
How can we compare one group with another? Well, to do this we make use of the
Parent Concerns Questionnaire scoring system, as described above. When we
compare groups we seek to compare the change in group A with the change in
group B. For example, we might look at Parenting classes.
Group A would be the group who went to/received the parenting class training.
Group B would be the group who did not go to/receive the parenting class training.
We are looking, then at the change in PCQ score for each group so that we can
compare them with each other.. So, we go from (a) to (b) to (c) as follows
(a)
Change Score for
Parenting Class
Group is
Average PCQ score just
before commencement of
classes
minus Average PCQ score
immediately after end
of classes
Average PCQ score just
before commencement of
classes
minus Average PCQ score
immediately after end
of classes
Change Score for
Parenting Class
minus Change Score for
Group not at Parenting
class
(b)
Change Score for
Group not at
Parenting class is
(c)
Comparison is
Thus suppose
Change Score for Parenting Class Group was
Average PCQ score just
before commencement of classes
minus
= 15
Average PCQ score immediately
after end of classes
Leaving Change Score for
Parenting Class Group
=10
= minus 3
Change Score for group NOT in parenting class was
Average PCQ score just
before commencement of classes
= 14
minus
Average PCQ score immediately
after end of classes
=13
Leaving Change Score for
Group Not in Parenting class
= minus 1
Comparison of the two groups would be
Change Score for
Parenting Class Group
= minus 3
Change Score for
Group Not in Parenting class
= minus 1
This means that the problem score for the Parenting class Group had reduced to a
greater degree (minus three) than the group not receiving parenting classes. Hence
the Parenting Class group showed greater improvement
How to do – an actual example from Children Services Applicants
Children’s Service Applicants were those who referred themselves or were referred
by someone else for social work support.
The following provides the results for a comparison between children’s service
applicants who received practical/instrumental support compared with service
applicants who did not.
Some initial information will help show the scale of change in each group.
The mean overall PCQ score at initial assessment for those who received practical
support was 18.6.
The mean overall PCQ score at initial assessment for those who did NOT receive
practical support was 16.3.
Hence those who received practical support started out with a larger problem score.
However, we find that a larger improvement – a better outcome – was achieved
by those who received practical support.
In this we can see that there was a markedly greater reduction in problem score
amongst those in receipt of practical support than those not in receipt of practical
support. Thus overall the PCQ score reduced for those receiving practical
support by 5.2, but only by 1.4 for those not receiving practical support.
Likewise we find the Child Development score reduced by 2.6 amongst those in
receipt of practical support, but there was no change amongst those not in
receipt of practical support
Also there was a reduction in Family and Environmental Factors problem score
of 0.9 amongst those in receipt of practical support, but an increase in Family
and Environmental Factors problem score of 0.4 amongst those not in receipt
of practical support
We can conclude from this actual example that receipt of practical support led to a
better outcome than the absence of practical support
The table showing the relevant figures is below [next page]
Comparison of outcomes between service applicants who received practical
support and those who did not receive that support
Problem score
Practical Support
(number=54)
No practical Support
(number=13)
Child Development
Parenting capacity
Family and Environmental
Factors
-2.6
-1.7
-0.9
0.0
-1.8
+ 0.4
PCQ Overall
-5.2
-1.4
Conclusion
We can see a number of ways of assessing outcome. We can look at one group
receiving a service, and see whether (a) their PCQ scores had reduced or (b) the
number of cases with no severe problems had increased. We can also compare one
group receiving a service with another not receiving that service. All these provide
aides to busy practitioners and managers who are seeking to evaluate the
effectiveness of service provision.
** [A technical note] We are seeking here to assess outcomes in a straightforward
and practical way. It is worth noting that the degree of variation between different
cases (and hence the distribution of scores), respectively, at the first assessment
and also at the follow up stage, is generally considered relevant when statistical tests
of significance are undertaken (although some will examine central tendency and so
on). However, we are here seeking to develop a practical approach to helping us
examine outcomes, and an emphasis on the mean provides busy agencies with that
opportunity. Where the resources and expertise are available, it is possible to use
the appropriate statistical tests top help further in outcome assessment.