Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Measuring Service Level Outcome: how to use the Parent Concerns Questionnaire to Assess Outcome in Many Cases Summary Measuring outcome is an important feature of looking at service effectiveness, and in turn, planning services. It is important to supplement existing approaches to service evaluation with robust additional methods, where available. The PCQ provides an opportunity to contribute to service evaluation, with its capacity to provide quantitative – numerical – measures of problem levels. Three broad approaches are explained and illustrated here: Measuring outcome/effectiveness using the PCQ scoring system with ‘before ‘and ‘after ‘service use measures Measuring outcome/effectiveness using the presence or absence of at least one severe problem identified by the PCQ before and after service use Measuring outcome/effectiveness by comparing two different groups, one that receives a service and another that does not Introduction One feature of the Parent Concerns Questionnaire (PCQ) is its capacity to measure outcome – whether or not there was an improvement over a period in time in the seriousness of a child and his or her family’s problems. This is a significant feature of the work of any accountable child care agency, since it can provide hard information about the degree of effectiveness of the agency. We have seen elsewhere on this web site [in our examples of measuring outcomes with individual cases] how this may be done with an individual case. However, this may also be done with many cases, for example all attendees at a Children’s Centre, or all families referred with additional needs. Thus, for example, if Children’s Centres were to introduce a new resource – for example parenting classes – it would be possible to see whether or not their parenting had improved or whether child development was less problematic for those who had taken those classes. We would do this by getting all participants in the parenting classes to complete the PCQ immediately before they began the classes. Then, when the classes were completed, we would then get them to complete the PCQ. We could then compare the scores for all the completed PCQs aggregated (that is added together) at the initial assessment (just prior to beginning the parenting classes) and at the follow up, or second completion (just after they had finished the Parenting Classes). We could then see whether the situation for that group, as a whole, had improved, got worse, or stayed pretty well the same. How do we do this? How to measure Outcome using the Parent Concerns Questionnaire – Practical Features There are two key ways in which outcome can be measured using the PCQ for the whole group (a) Using the PCQ Scoring System First, we could aggregate (add together) the score for each participant’s PCQ and then add them all together. We can explain this by reminding you how the scoring system of the PCQ works and then showing how this works with a number of families. The PCQ is divided into 37 Problem Areas or items. Each of these is scored as follows No problem A problem that is not severe A severe problem =0 =1 =2 Scoring Each PCQ (for one family) Consider the situation where you wish to make an initial assessment of families that were about to go on a parenting course. You ask the first parent to complete the PCQ. So, let’s suppose a parent rated the following problems to be severe: Child Behaviour Parental Guidance and Housing And the following as problems that were not severe: Marital/Partner Financial We would have 3 severe problems (3 x 2=6) and two problems that were not severe (2 x 1=2). The overall total would be 8. Aggregating scores for all families Let us suppose now that there are six families involved in Parenting Classes and that you wish to focus on them. You would add together the overall scores for each of the PCQs completed. Let us suppose the scores were as follows 8 10 12 9 11 10 Adding these together would make an overall Total of 60 and the average (mean) for each family would be 10. Assessing Outcome Let us now suppose that all six parents completed the parenting class. At the end of the classes they were each asked to fill in the PCQ. This yielded scores (based on the scoring system just outlined) as follows 0 15 5 3 5 2 Adding these together would yield a total of 30, and the average (mean) for each family would be 5. From this we could see that a clear improvement had occurred over all the parents/families involved. On average, the problem score for each family had halved (from 10 to 5). However, this would not prevent us, on an individual basis to identify where improvements had occurred. For example, we can see from the above figures that one case deteriorated, from a score of 10 to one of 15. So within the general improvement that was experienced by the group, there was one family that not only had not benefited, but had got worse. We might conclude that, overall, this group had improved, in some cases markedly, but that was not the case for all service users. Further features of the scoring system with many cases It is possible to see whether an improvement by a group has been maintained. This is straight-forwardly done by asking the service users to complete the PCQ a third time, say at 3 or 6 months after the finish of the parenting classes Suppose we got the following overall scores six months after the parenting classes had finished 0 10 2 2 3 1 This would yield a total score of 18 and an average problem score for each family of 3 (18/6). This would indicate that the improvement had indeed been maintained, and that there had even been some further improvement, as the average (mean) family score had reduced from 5 to 3 ** (b) Measuring outcome through examining the presence of Severe Problems Our second approach would be to examine the presence and absence of severe problems. Since the presence of a severe problem is the ‘trigger’ for an assessment, this would indicate whether the family was finding at least one area of their life to be severely problematic. Of course, the absence of a severe problem would indicate that there was no area of their life (at least as measured by the PCQ) that was severely problematic How would this be done? Well in principle, there is much that is similar to the process outlined above when using the PCQ to get a problem score. Ion this case, though we concentrate on the presence or absence of severe problems. Let us suppose, as above, we have a group of parents who will be taking parenting classes. We will ask them to complete the PCQ just before the commencement of the parenting class and just after it is finished. We might find at the initial assessment that all families had at least one severe problem. At the follow up, just after the parenting class finishes, we might find (for our six participants) the following No severe problems Severe problems Severe problems No severe problems Severe Problems No severe problems Here we find that where, at initial assessment, all 6 of the families had at least one severe problem, following the parenting class only three of them had at least one severe problem, while 3 had no severe problems at all. The usefulness of this approach is apparent if we see the presence of at least one severe problem as a reason for further assessment with a view to possible intervention. It would be clear from these results that 3 of these families/parents no longer needed any intervention because they had no severe problems. The other three may need further intervention, and that could be determined by assessing them further to take a closer look at their situation. (c) Measuring Outcome by Comparing Groups For most busy practitioners and managers this would provide good evidence for the helpfulness of the parenting classes. This is perfectly reasonable, and where you do have some hard evidence of improvements occurring in the face of some particular intervention (such as a parenting class) this is a major step forwards from where no such evidence is present. However, there is always a danger that an improvement of this sort ‘just happened’ and that it had little to do with the parenting classes. How can we find out whether those improvements are due to the parenting classes? We can do so by having a comparison group of parents who did not have that class. This may not be straight forward, but could be done in some situations. For example, suppose we had a children’s centre, or a number of children’s centres, that wished to offer parenting classes. Perhaps there were 12 parents who were lined up to go on one. Where there were six parents in each parenting class and you were running only one at a time, you might wish to choose parents (say) by the toss of a coin. You could do so until you had 6 who were in the first parenting group and six who were not. Having got two groups you could go through the same process with the group not initially receiving the parenting class as you had with the group who did. Both groups would complete the PCQ immediately before the Parenting classes began and immediately after they finished. Of course only one group would actually attend the parenting class. Suppose the following was the result. Suppose the overall score for the group not receiving parenting classes (and hence also the average score for each family) remained the same over the two completion of the PCQ. On the other hand the score for those involved in the parenting class improved as outlined above. We would be able to conclude that it was likely that the parenting class was the reason why the improvement had occurred. A Health Warning It is important to recognise that it is difficult to be perfect in measuring outcomes when you are busy practitioners and managers. When these evaluations are carried out by researchers, they are aware of all sorts of pitfalls that they would seek to guard against. For example, in this case there might be the danger of what is called ‘contamination’. If all the parents were chosen from one children’s centre (both those receiving the Parenting Classes and those not receiving them), then they probably know each other, and it is quite possible that the techniques learned by those attending the parenting classes would be passed on to at least some of those not going to the parenting classes. This could, of course, affect outcome. A Realistic Choice However, we can get too hung up on perfection. Even the most carefully constructed experimental designs conducted by researchers on social interventions invariably have ‘drop outs’ (people who start but do not complete an intervention). Practitioners and managers operate with all the imperatives and pressures of day to day work and need to be realistic about their evaluations. Using the PCQ to measure outcomes, with or without a comparison group, provides important information that can help practitioners and managers in determining which services to provide and discovering which are effective. This is a realistic and practical choice for measuring outcome, evaluating interventions and helping with accountability. An Example: Measuring Outcome using the PCQ Scoring System We here present an example, using this method to see how things changed when a group of families, who were applicants for children’s social care services were assessed using the PCQ at the point of application, and at a second point six months later. T Change in Average Problem Score between Initial Assessment and Assessment 6 months later (total sample=67) Average (mean) PCQ Score at: Initial Assessment 6 months later Overall change Child Development Parenting Capacity Family and Environmental factors 8.4 6.5 4.0 6.2 4.7 3.4 2.2 1.8 0.6 Total PCQ Score 18.9 14.3 4.6 We can see from this that these families did improve, but we can see that there was considerable room for further improvement. If we remember that a single severe problem scores 2, an overall score of 18.9 (the score of the applicants at initial assessment) will almost certainly involve a considerable number of severe problems across the majority of cases. The same goes for the domains constituent of the PCQ – there was, at initial assessment, almost certainly a number of severe problems in each of the domains. At follow up we can see some improvement – the equivalent of an average of about two severe problems per family. However, we can also see score which suggest, on average, these families retained a considerable number of problems, and were, as a whole considerably in need of support. We can clarify this further by focusing on our second way of evaluating outcome, by focusing on severe problems. An Example: measuring outcome through the presence or absence of severe problems If we look at the PCQ as a whole, 56 families (out of 67) had at least one severe problem at initial assessment. Hence eleven families had no severe problems. Six months later 52 families had at least one severe problem, and therefore 15 families had no severe problems. We see, therefore, an improvement, in this respect of only 4 – that is only 4 fewer families (of the 67) had at least one severe problem six months after the initial assessment. We can look at this in terms of the different domains as well. This is useful where we want to look at specific aspects of the family’s functioning. We can see from this that while 48 families had at least one severe child Development problem at initial assessment, this had reduced to 38 families six months later. Put another way, the number without at least one sever child development problem at initial assessment was 19. This number increased, six months later to 29. This meant there was a reduction of ten families in the number that had at least one severe child development problem. Number of Families with at least one severe problem: Each Domain and the PCQ overall Initial Assessment Six Months later Number percent Number percent Child Development Parenting Capacity Family and Environmental Factors 48 43 34 72 64 51 38 26 32 57 39 48 PCQ as a Whole 56 86 52 78 Looking in greater detail: an Example from the Child Development Domain We can look at these changes in greater detail by focusing on the individual items or areas that make up each domain. Here we see that child emotional problems (where a child is upset distressed or depressed) were severe in 31 of the 67 families at the initial assessment. Six months later this had reduced – by ten – to 21 We see Child Behavioural Problems (where a child is frequently defiant or aggressive or has tantrums) were severe in 39 of the 67 families at initial assessment. This had reduced – by 14 – to 25 six months later. Thirdly, we can see child cognitive problems (where they have difficulty concentrating or learning as quickly as they should) were severe in 21 families at initial assessment. This, though increased – by two – to 23 families six months later. Thus we see a more complex picture when we look at individual areas or items of the PCQ. While child emotional and behavioural problems decreased over the six months, child cognitive problems increased. Provided with this kind of information, a manger would have a better idea about where resources might usefully be concentrated (here we might argue that child cognitive problems provided one such area). However, this is always a judgement for the practitioner or manager. They may consider, for example, that child behavioural problems merited more attention than child cognitive problems. They may also feel that the improvement in this area was not as great as s/he would have wanted. In that case, they might decide to put further resources into tackling the problem of child behaviour precisely because of the importance they ascribe to it. Number of families with at least one severe problem in 3 individual problem areas/items of the Child Development domain (number=67) Initial Assessment Child Emotional Child Behavioural Child Cognitive Six Months later Number % Number % 31 46 21 31 39 58 25 37 21 31 23 34 An Example: Measuring Outcome by Comparing Groups How to do – a hypothetical example How can we compare one group with another? Well, to do this we make use of the Parent Concerns Questionnaire scoring system, as described above. When we compare groups we seek to compare the change in group A with the change in group B. For example, we might look at Parenting classes. Group A would be the group who went to/received the parenting class training. Group B would be the group who did not go to/receive the parenting class training. We are looking, then at the change in PCQ score for each group so that we can compare them with each other.. So, we go from (a) to (b) to (c) as follows (a) Change Score for Parenting Class Group is Average PCQ score just before commencement of classes minus Average PCQ score immediately after end of classes Average PCQ score just before commencement of classes minus Average PCQ score immediately after end of classes Change Score for Parenting Class minus Change Score for Group not at Parenting class (b) Change Score for Group not at Parenting class is (c) Comparison is Thus suppose Change Score for Parenting Class Group was Average PCQ score just before commencement of classes minus = 15 Average PCQ score immediately after end of classes Leaving Change Score for Parenting Class Group =10 = minus 3 Change Score for group NOT in parenting class was Average PCQ score just before commencement of classes = 14 minus Average PCQ score immediately after end of classes =13 Leaving Change Score for Group Not in Parenting class = minus 1 Comparison of the two groups would be Change Score for Parenting Class Group = minus 3 Change Score for Group Not in Parenting class = minus 1 This means that the problem score for the Parenting class Group had reduced to a greater degree (minus three) than the group not receiving parenting classes. Hence the Parenting Class group showed greater improvement How to do – an actual example from Children Services Applicants Children’s Service Applicants were those who referred themselves or were referred by someone else for social work support. The following provides the results for a comparison between children’s service applicants who received practical/instrumental support compared with service applicants who did not. Some initial information will help show the scale of change in each group. The mean overall PCQ score at initial assessment for those who received practical support was 18.6. The mean overall PCQ score at initial assessment for those who did NOT receive practical support was 16.3. Hence those who received practical support started out with a larger problem score. However, we find that a larger improvement – a better outcome – was achieved by those who received practical support. In this we can see that there was a markedly greater reduction in problem score amongst those in receipt of practical support than those not in receipt of practical support. Thus overall the PCQ score reduced for those receiving practical support by 5.2, but only by 1.4 for those not receiving practical support. Likewise we find the Child Development score reduced by 2.6 amongst those in receipt of practical support, but there was no change amongst those not in receipt of practical support Also there was a reduction in Family and Environmental Factors problem score of 0.9 amongst those in receipt of practical support, but an increase in Family and Environmental Factors problem score of 0.4 amongst those not in receipt of practical support We can conclude from this actual example that receipt of practical support led to a better outcome than the absence of practical support The table showing the relevant figures is below [next page] Comparison of outcomes between service applicants who received practical support and those who did not receive that support Problem score Practical Support (number=54) No practical Support (number=13) Child Development Parenting capacity Family and Environmental Factors -2.6 -1.7 -0.9 0.0 -1.8 + 0.4 PCQ Overall -5.2 -1.4 Conclusion We can see a number of ways of assessing outcome. We can look at one group receiving a service, and see whether (a) their PCQ scores had reduced or (b) the number of cases with no severe problems had increased. We can also compare one group receiving a service with another not receiving that service. All these provide aides to busy practitioners and managers who are seeking to evaluate the effectiveness of service provision. ** [A technical note] We are seeking here to assess outcomes in a straightforward and practical way. It is worth noting that the degree of variation between different cases (and hence the distribution of scores), respectively, at the first assessment and also at the follow up stage, is generally considered relevant when statistical tests of significance are undertaken (although some will examine central tendency and so on). However, we are here seeking to develop a practical approach to helping us examine outcomes, and an emphasis on the mean provides busy agencies with that opportunity. Where the resources and expertise are available, it is possible to use the appropriate statistical tests top help further in outcome assessment.