Download Establishing the principles: Quality Change and Complementarity

January 2007
Establishing the principles:
Quality Change and Complementarity
Seminars held on 30-31 October 2006 at Novotel West, Hammersmith,
London
Summary Notes
Covering note
Background:
The seminars hosted by ONS/UKCeMGA on Quality Change and
Complementarity, constitute a significant part of the public consultation on
“Establishing the Principles”. The seminars were organised to consult on the
substantive methodological issues of:
• accounting for changes in quality of non-market output and
• public sector and private sector complementarity
as presented in the consultation document published on the 18th of
September 2006 . This consultation process reflects the commitment made
by the National Statistician in his response to the Atkinson Review 2005
that radical changes to methodology would not be proposed before a
thorough consultation with external experts has taken place.
The “Establishing the Principles” consultation paper is based on two general
issues:
1. Taking account of quality in the measurement of public output.
The international guidance governing national accounts, the 1993 United
Nations System of National Accounts (SNA93), regards quality change as an
integral dimension of output. The Atkinson Report outlined this
methodological issue in Principle B:
“the measurement of government non-market output should, as far as
possible, follow a procedure parallel to that adopted in the National Accounts
for market output”
The principal argument for accounting for quality change is that if quality
changes are not measured, then the volume measure of output and thereby
productivity will be either under-stated or over-stated depending on whether
quality went up or down - Annex A contains more detail on the issue as
1
presented in the consultation paper.
2. Public sector and Private sector Complementarity as outlined in
Principle C:
“account should be taken of the complementarity between public and private
output, allowing for the increased real value of public services in an economy
with rising real GDP”.
This in turn derives from Principle A in the Atkinson Report which advocates
measuring government non-market output, as far as possible, following a
procedure parallel to that adopted in the national accounts for measuring
market output. The main proposition of the complementarity principle is that
as private sector productivity increases and the economy grows over time,
the value of the public services increase due to the private sector
circumstances. The argument is that the increase in value of the public
services is not merely a price or a relative preference effect but something
additional, which should be accounted for in the measurement of public
sector output and productivity (Annex B contains more detail on the issue as
presented in the consultation paper).
The seminars on quality change and complementarity formed part of the
wider consultation process on the document which closed on 11th December
2006. The main objective of the consultation process is to ensure that the
approach to be taken has benefited from professional and expert opinion.
The success of the consultation will depend on the degree to which the
results of the days provide:
• a clear steer for the next phase of development of the methodology;
• evidence that the consultation has benefited from relevant expert
opinion, both practitioners and analysts; and
• evidence that everyone who would wish to do so has had the
opportunity to contribute to the debate.
The seminars were facilitated by Robert Chote, a non-executive member of
UKCeMGA’s independent Advisory Board. They consisted of short
presentations of the issues as outlined in the consultation documents,
followed by a clarification session and discussion by the participants. A list of
attendees is attached in the Annex C.
Guide to the summary notes:
The objective of the summary notes is to provide an accurate written
documentation of the consultation seminar for the purpose of transparency
and accountability. The summary notes provide a flavour of the discussions
on the day which will feed into the outcome of the consultation process which
will be a ‘strategy paper’. The strategy paper will document how the work on
measurement of non-market output and productivity will be taken forward
based on the feedback from not just the ‘Establishing the Principles
consultation’ but also from two other consultations on education and health
2
scheduled in the early part of 2007. It will be published within 3 months of
the end of the whole consultation programme which closes on the 3rd April
2007.
The summary note is a high-level aggregation of the key discussion points of
the consultation seminars. The notes are structured under relevant topics for
the sake of clarity. They do not reflect the chronological order of the
discussion at the event nor do they reflect any order of importance.
3
High-level summary:
Quality change
1.
Is there a need for quality adjustment factors in non-market
output measures?
o There was broad agreement that non-market output should be
measured in a way which takes account of quality change. This was
reinforced by the proposal that complex public services generally have
more quality dimensions than many market outputs.
o There was concern to ensure that the weight of each quality dimension
genuinely reflects its valuation. However questions were raised with
regard to how the relative weights between different quality dimensions
could be determined. Contributors suggested that all the different quality
dimensions cannot have the same weights (for example patients will
prefer better survival rates in comparison to having cleaner beds in the
quality spectrum of NHS care).
o There was broad agreement on the need to include both positive and
negative quality changes when measuring non-market output, and to
avoid tracking only instances of quality improvements.
o Some contributors highlighted the need to limit the quality and
outcome dimensions in measuring government output to just a few. This
was suggested for practical reasons to avoid ‘death by million measures’.
However others voiced concern that limiting the dimensions could lead to
politically motivated bias.
2.
Should there
purposes?
be
different
output
measures
for
different
o There was broad agreement that different measures may be needed
for different purposes. For example an education output measure which
standardises for differences in socio-economic background might be
thought useful as a performance measure or welfare measure, but would
not be suitable for the national accounts.
3.
Should measurement of quality for the non-market sector
reflect what is done for the market sector?
o There was broad acceptance of this. But it was suggested that the
following should be borne in mind in applying this principle:
•
Quality change is not always measured adequately for market
output;
•
Where it is measured, it is not something that can be observed
directly but has to be estimated by the statistician.
o Some contributors said that the fundamental character of the nonmarket sector – absence of prices - made the measurement of public
services difficult. This contrasts with the market sector where prices can
provide rich information on quality changes.
o It was also pointed out that a public service provided free of charge
contains an element of insurance and this makes it something different in
nature to the equivalent market service. In the market sector, the
insurance element would be treated as a separate paid-for service.
4
4.
What is the relationship between output and outcome? Where
does the boundary between them lie?
o
o
o
o
o
5.
It was observed by some contributors that:
An output may make a contribution to a given outcome. The extent of its
contribution can vary according to the circumstances. The example given
was that a cataract operation is likely to have made a large contribution
to restoring the sight of a patient undergoing the treatment; a public
health programme, on the other hand, may or may not be the main cause
of a positive change in the health status of the population.
Each public service may be associated with many outcomes. A single
outcome may be associated – though not always exclusively – with the
production of one or more separate outputs.
It may be possible to measure an output sufficiently accurately by
working back from an associated outcome. But this will not always be the
case for various reasons: for instance, the link between the outcome and
the output may be too indistinct, or a time lag between output and
outcome could make estimation difficult.
Methods of deriving outputs from outcomes should be kept stable to
ensure comparability over time. However some contributors argued that,
in a changing world, quality characteristics and their valuation by
consumers could be expected to change over time; statistical measures
should therefore reflect this.
Ideally, negative contributions to outcomes should also be taken into
account in measuring output.
Should value weights or cost weights be used in aggregating
outputs?
o
o
o
o
A number of contributors saw the merit of using value weights especially
in cases where the unit cost of the service deviated from the unit value of
the service. Other theoretical reasons proffered were that
•
value weights enable output to be measured at a point closer to
outcomes, and that
•
value weights are more applicable in a direct approach to
measuring output whereas cost weights are a construct from the input
side.
There were contributors against the use of value weights especially when
it came to the National Accounts framework. Proponents of this view
argued that weights have to aggregate to one and this is not the case for
value weights. Furthermore it was their position that there is no logic in
weighting some components of a measure using cost weights and others
using value weights. The result would be a meaningless hybrid.
Some contributors observed that value weights may prove to be useful in
micro studies of public services when suitable, mutually-consistent values
are available. It was suggested that this approach could be useful, for
instance, in measuring preventive activities (e.g. public health
information).
It was also suggested by a contributor, that variables commonly
suggested as value weights could be used as measures of activity. QALYs
(quality adjusted life years) could, for instance, be used as a direct
measure of health output.
5
o
6.
Some issues raised on the valuation of public services output
o
o
o
o
7.
It was broadly agreed that services which are regarded, by some, as
ineffective (e.g. homeopathy) should nevertheless be included as output if
there is evidence that they are valued in the market or that clients do
value them.
Some contributors observed that the set of outcome and quality
dimensions relevant to valuation of a service needs to be determined from
research into the particular area.
It was suggested that when comparing the education of a student from a
“poor background” with that of a student from a more “privileged
background”, the former may justifiably represent more value added and
hence more output. But when such a value judgement is made, it is
difficult to accept it as anything other than a part of a micro study.
The following were suggested by various contributors as possible sources
for valuing output characteristics in the non-market sector: information
about consumers’ behaviour as observed in the market sector, revealed
preferences through choices made (e.g. elective health treatment),
hedonic estimation and consumer satisfaction surveys.
Differentiation, degree of success and “overall quality
adjustment”: is it acceptable to use these three Atkinson
approaches to measuring quality adjustment?
o
o
o
o
o
8.
Some contributors highlighted the issue that value weights are subjective
and posed the question of how and who will determine the suggested value
weights.
There was strong support for differentiation as a means of capturing
quality change where it is possible.
It was argued that differentiation should be carried out only to the point
where it led to a more precise result.
There appeared to be some confusion about the distinction between
degree of success and overall quality adjustment.
It was regarded as very difficult to identify the incremental contribution to
outcome from a particular service.
Threshold measures for measuring success (e.g. achieving a minimum of
5 GCSE passes) were considered by some contributors as inadequate.
Is it acceptable to use the judgement criteria suggested in the
consultation paper?
o
o
o
o
Some contributors were uncertain with respect to what ‘statistical
judgement’ meant and how it can be applied. Some contributors
suggested providing clear guidance on how ‘statistical judgement’ would
operate.
Some contributors requested transparent analysis of the data to avoid
ambiguous interpretations.
Some contributors cautioned that many indicators are chosen
subjectively.
Most contributors urged the ONS to be fully transparent as regards the
process and called for a clear articulation of how any new productivity
measures and numbers are arrived at.
6
Complementarity:
9.
Should government output measures take account of the
interlinkages between the private and public sectors?
o
o
o
o
The idea that the economy is interlinked and that the private and public
sectors influence and depend on each other was not disputed. Whether
these interlinkages justify a new approach to measuring non-market
output, as suggested by the complementarity principle, was contested.
Some argued that government output measures should reflect the
quantities of services delivered, as in the National Accounts, and not the
benefits derived from them. It was considered that prices were the correct
means for weighting items together as price reflects the relative utilities
of what is bought or supplied. Under this scenario, changes in consumers’
subjective valuations can only impact on measured output if the former
are reflected in observed price changes. This led to the conclusion that an
exhaustive measure of non-market output would provide a result that was
precise enough, leaving no need to adjust for complementarity.
Others argued that consumers’ valuations did vary for public services:
they varied from consumer to consumer and from place to place even
when the service provided cost the same amount. Examples were quoted:
education services are usually valued more highly when they are of good
quality; fire services are valued more highly the more wealth they
protect, and health treatments are valued more highly the more
successful they are. This variation in valuations is not captured in prices
paid (or, more precisely, in the production costs in the case of nonmarket services).
Some public services - health and education, for example - contribute to
human capital and influence market sector production functions. Again,
that form of complementarity does not suggest that something extra has
to be added to the analysis.
10. What would the implications be if any such private-public
interlinkages were reflected in non-market output measures?
Does the existence of these interlinkages validate the
application of Principle C as outlined in the Atkinson Report?
o
o
o
o
o
Some argued that changes in people’s valuations of public services could
have an impact on some measures of output
In that context, complementarity was generally seen as a value or price
change, not as a volume change. There was general support for the idea
that complementarity is about the ‘value’ of public services.
Some contributors considered that complementarity effects were
sometimes captured in current measures and no additional adjustments
were necessary. An example given was that complementarity is apparent
when the public sector benefits from efficiency gains made by the private
sector (as in the purchase of better drugs).
One proposal made was that only a discernible change in the services
themselves could justify recording a volume change.
Some contributors thought that complementarity highlights an important
aspect of output and its impact on well-being. It was suggested that
7
capturing the effect of complementarity would be useful but one measure
may not fit all purposes.
11. If there are different measures for different purposes, how
does complementarity fit it?
o
o
There was broad agreement that different sets of measures are needed
for different purposes, and that for some purposes complementarity as a
concept may be justified.
The introduction of complementarity into National Accounts measures was
looked at critically and generally not favoured. In other types of measure,
taking account of complementarity could usefully capture changes in the
value of public services to individuals and society. The types of measures
mentioned in this context were those designed to measure performance,
effectiveness, efficiency, outcomes, welfare and well-being.
12. How does complementarity feature in measurement of private
sector output?
o
o
Some contributors indicated that market output, as currently measured,
does not make separate adjustment for complementarity. But concerns
were expressed by some about how services in the private sector are
currently measured and how this relates to an ideal measure. One
example given was the measurement of private sector education output
using staff numbers.
In the private sector the price is taken to represent the marginal valuation
of consumers. However it was observed by some contributors that in the
market place the price is dictated by both consumer and supply side
factors, and that prices can therefore deviate from the marginal value of a
commodity.
13. Is complementarity a price change capturing value changes
over time? What merit is there for using value weights instead
of cost weights?
o
o
o
Several speakers felt that complementarity impacted on the ‘value’ of
services. Such value changes would be cancelled out in a ‘volume’
measure of output. An illustrative example given was: If the EU
introduced French as the sole official language overnight would
complementarity principle mean that such a change increases the output
of education sector. It is the likely case that the change would lead to an
overnight increase in the value of French lessons but not to an increase in
the volume of output.
The use of value weights instead of cost weights was suggested by many.
But there were concerns with regard to their applicability for National
Accounts measures.
Some contributors argued that the use of value weights alongside cost
weights would result in making GDP deflators meaningless.
8
14. Does including complementarity risk double counting? And is
the issue one of attributing non-market output correctly?
o
o
o
Some saw a risk that an additional complementarity adjustment would
lead to double counting of some output (e.g. measure the output of a new
road in the non-market sector and then double count by measuring the
increase in output of companies in market sector arising from the use of
the road).
There was some concern about what part of complementarity effects can
be rightly attributed to public service providers or the market sector.
It was suggested that when more than one public service provider is
involved, it would also be difficult to determine the relative contributions
of the two services to a productivity change.
What should ONS do now?
The responses from the quality and complementarity seminars
o
o
o
o
o
o
There was general agreement that ONS should put into practice the
quality framework proposed in the Consultation paper.
A wide range of views was expressed on the complementarity principle
and it was concluded that this diversity should be reflected, perhaps by
ONS publishing a range of different measures.
Because the complementarity principle is still weak, it was suggested that
ONS should not include the complementarity effect in National Accounts
measures in order not to compromise the credibility of the total work
programme on better output measures.
It was generally agreed that ONS should also look into providing
measures that go beyond National Accounts measures to inform about
productivity and performance.
Some contributors recommended that the measures resulting from this
new work – including the effects of complementarity – should be included
in satellite accounts or in the productivity articles.
Many contributors highlighted the need for ONS to ensure transparency in
introducing methodological changes and stressed the importance of
communicating the impact of this on the numbers. Some contributors also
expressed the need for ONS independence to ensure credibility of the
productivity figures produced.
9
ANNEX A
Taking account of Quality in the measurement of Public output
The international guidance governing national accounts, the 1993 United
Nations System of National Accounts (SNA93), regards quality change as an
integral dimension of output. The Atkinson Report outlined this
methodological issue in Principle B:
“the measurement of government non-market output should, as far as
possible, follow a procedure parallel to that adopted in the National Accounts
for market output”
The principal argument for accounting for quality change is that any
improvements (or deterioration) in the services delivered if not taken into
account, then will result in productivity measure that is under-stated (or
over-stated).’ Principle B is a corollary of principle A advocating similar
procedure for measurement of non-market output as in the case of private
sector output. The three methods proposed in the consultation document
through which changes in quality could be measured are:
•
differentiation of services;
•
defining the volume of output by reference to the degree of
success of the activity concerned;
•
basing the estimate of output volume according to the
attributable contribution to a desired outcome.
The first approach is based on defining ever more homogenous categories of
service whereby a change in the mix of services delivered would
automatically capture change in quality of the overall service. Greater
differentiation in the health sector has resulted in better accounting for the
vast differences in quality of treatment in NHS (for example, without
differentiation the output generated by procedure to correct an in-growing
toe-nail was equated to be same as that of a complex heart or lung
operation).
The first approach will only capture part of the quality changes and the
second and third method can be utilized to factor in the different dimensions
of quality and the contribution to the over-arching objective or outcome of
service. For example in the case of NHS services, the waiting time, patience
experience, the effectiveness of treatment to list some, will all form part of
the quality of service delivered.
Difficulties outlined in the methods described are that information on all
quality domains may not be available. And although it is important to
account for all the positive and negative movements in the different quality
domains, it may be very difficult to ascribe a weight to the different quality
dimension. Moreover when it comes to the third method, the outcomes
observed in the economy are often influenced by a wide range of factors, not
just the specific service under consideration. A country’s health status, for
example, will depend upon a range of influences such as diet, exercise
10
habits, the extent of smoking, and so on, as well as the output of the health
services. So it would be a complete mistake to attribute all of the change in
health status to the output of the health service.
In view of these measurement difficulties the consultation paper advocates
against adopting an impossible theorem. As a way forward a principled
framework is suggested along the following lines:
•
Identify relevant set of quality domains for each service (using
evidence from studies and international corroborations)
•
Adopt differentiation to the maximum
•
Adjust output by degree of success
•
Adjust output by contribution to outcome
•
Use corroborating evidence to verify plausibility of quality
change
•
Employ professional statistical judgement
The questions posed in the consultation paper for public debate in carrying
forward the work are reproduced below. These questions were not presented
as exhaustive but on the day any aspect of the issue was made wide open for
discussion.
•
Does the discussion in the consultation paper cover the key
points relevant to taking forward development work on the quality
adjustment of public service output measures?
•
Does the discussion the consultation paper reflect the correct
balance between the importance of producing quality measures for
public services and the need for a cautious approach?
•
Does the framework in the consultation paper cover the relevant
aspects of dealing with quality adjustments?
•
Is it agreed that the professional statistical judgment relating to
the quality adjustment should be made as transparently as possible
and within a framework such as that proposed in the consultation
paper?
11
ANNEX B
Public sector and Private sector Complementarity
The Principle C proposed in the Atkinson Report follows from the key
principle A, advocating a parallel procedure for measuring non-market output
as far as possible to that of market output. The main proposition of the
complementarity principle is that as private sector productivity increase and
the economy grows over time the value of the public services increase due to
the private sector circumstances. The argument is that the increase in value
of the public services is not merely a price or a relative preference effect but
something more, which should be accounted for in the measurement of
public sector output and productivity.
Principle C: “account should be taken of the complementarity between
public and private output, allowing for the increased real value of public
services in an economy with rising real GDP”
The consultation paper furthermore distinguishes the usefulness of the
productivity
measures
derived
by
adjusting
for
public/private
complementarity which is different from performance of a particular public
service. The performance indicators will capture own partial effects which
reflects the contribution to outcome which is within the control of the public
service provider. This is different from output measure using
complementarity principle which captures the total contribution to the
economy from the public service even though it is arising due to private
sector circumstances. There is a direct parallel to the private sector in using
such a methodology in that private sector productivity in the economy can be
increasing due to public sector activity. In the example of the provision of a
new road, the productivity of private sector firm can increase due to the
public sector circumstance. As such the public/private complementarity is a
symmetric two-way relationship.
The consultation document suggests that what measures and indicators to be
used to account for complementarity will depend on the channel of influence,
which in turn can differ in each case depending on the specific public service
and the circumstances. In the case of education, for example, the specific
hypothesis was that real earnings growth in the economy as a whole added
to the value of the outcome, taken at a conservative estimate to be growing
by 1.5 per cent on average each year. By contrast, in the case of the fire
service, the supposed specific mechanism involved the increase in the
average value of buildings and their contents. If the principle is applied it can
mean that in the limit, if the properties and their contents had no value, then
the value of the protection provided by the fire service would also be zero.
The questions posed in the consultation paper for public debate in carrying
forward the work are reproduced below. These questions were not presented
as exhaustive but on the day any aspect of the issue was made wide open for
discussion.
12
•
Is public sector output and productivity affected by private
sector factors and vice versa, and does this validate the application
of Principle C to the measurement of government output?
•
Can the channel and hence the effect of complementarity vary
for each public service area?
•
Should “own effect” performance measures be published
alongside productivity measures, excluding private sector influences?
•
Are price and preference changes not relevant for
complementarity? Should work be carried out taking forward effects
of changing relative prices and preferences in economies with rising
real incomes?
•
Is it agreed that, for purposes of measuring output, changes in
preferences and relative prices should not be taken into account? But
should work be carried out to design and provide such metrics within
a wider agenda of measuring societal wellbeing?
13
ANNEX C:
Attendance List to the seminar on ‘Establishing the Principles
Consultation’
LIST OF DELEGATES
First Name
Surname
Organisation
John
Stephen
Steven
Robert
Bates
Black
Boyes
Chote
Nick
Doyle
Margaret
Andrew
Eames
Evans
Antony
Andrew
Deborah
Eva
Roger
Ciaran
Birgitta
Anne
Geoff
Allen
Hilary
David
Katie
John
Aileen
Emmanuel
Ray
Martin
Fielding
Frost
Garniss
Hagsten
Hay
Hayes
Magnusson
McAllister
Reed
Ritchie
Rowell
Saal
Salt
Seddon
Simkins
Thanassoulis
Thomas
Weale
Alison
Wolf
Department of Health
PA Consulting
Department for Work and Pensions
Facilitator (Member of UKCeMGA’s
Advisory Board)
National Institute for Health and Clinical
Excellence (NICE)
Beds & Herts Public Health Intelligence
Office for National Statistics – National
Accounts Group
University of Birmingham
David Mason Consultancy
Department of Education and Skills
Statistics Sweden
Oxford Policy Institute
Department for Education and Skills
Statistics Sweden, National Accounts
Statistical Governance
Department of Trade and Industry
Statistics Commission
Dr Foster Intelligence
Aston University
Department of Work and Pensions
Vanguard
Department of Health
Aston University
Open University
National Institute of Economic and Social
Research
King’s College
The following members of the UK Centre for the Measurement of Government Activity at
the Office for National Statistics attended the seminar: Steve Almond, Jim Ebdon, Joe
Grice (Executive Director), Isabel Kohten.
14
1
15