Download There has been a long-standing debate over the contribution that

Working Paper 34
Andrew Webster, SATSU, University of York, UK
Do not quote without authors permission: email [email protected]
Evaluation, governance and moves to a socially robust assessment of health
technology
Introduction
One of the core questions that is raised by the Workshop is ‘how do new narratives,
either in the form of actual artefacts or in the form of expectations regarding new
artefacts, influence the sociotechnical landscape?’ Expectations about new technologies
are subject to the discursive and material narratives of evaluation as forms of governance
that review and reshape the ways in which new technologies are taken up and deployed.
There are attempts to create new narratives of evaluation that take seriously how
technologies occupy specific sites in a ‘socio-technical landscape’ (Lehoux, 2006). This
has been most noticeable in the field of medicine, and especially the debates surrounding
‘evidence-based medicine’. This paper explores the debates within EBM as a form of
socio-technical governance and argues for an approach to evaluation that is not simply
the producer of ‘reliable knowledge’ but also, in Helga Nowotny’s (2006) sense, one that
is more ‘socially robust’, that is that ‘there is something of a societal value that needs to
be integrated into the definition of good science’. How might such normative
expectations influence the evaluation of medical technologies? Let me first provide some
context for my argument.
There has been a long-standing debate over the contribution that has been made by
medical innovation to health improvement. Reference is often made, for example, to
McKeown’s (1976) analysis of the limited contribution that medicine per se has made to
population health arguing that more progress has been made over the past century
through general improvements in public health (especially in regard to sanitation and
hygiene) than through the changes wrought by increasingly sophisticated health
technologies (see Harris, 2004). Sociologists have argued in a complementary way that
health improvements are also about broader changes in social structure and a more
equitable distribution of income and wealth. Yet, clearly, technical development in
medicine has occurred and now enables clinicians to undertake quite routinely what have
been in the past highly dangerous procedures. Heart transplants or brain surgery are
cases in point.
Such innovations rely on ever greater technical sophistication and clinical skill, an
extensive range of support sciences (such as immunology or pathology), a supportive
regulatory environment, and perhaps most important of all, willing patients who in the
early stages of a technology act as ‘moral pioneers’ (Rapp, 1999) negotiating the choices,
risks and hazards of these new techniques on behalf of others.
On the other hand, the increasing sophistication of health technology to probe, rechart,
and redefine pathology may serve to generate new forms of disease, as Foucault (1988) has
argued in regard to the role of screening in health care: this can simply produce the
asymptomatically ill, as in screens for hypertension in assessing blood pressure levels.
1
Innovative technologies here create and define diseased bodies and do so for people who
have no sense, no lived experience of being ‘ill’: they are the ‘new ill’. To this extent,
health innovation is generative of new uncertainties that cannot, as risks, be easily
calculated. Much of the success of biomedicine has been in coping with acute medical
problems but in doing so has relocated patients from acute illness into the category of
the chronically sick.
Recent commentary from within medicine itself raises similarly strong doubts about the
social utility – or what the authors call ‘the fidelity of health care’ – of contemporary
medicine. Woolf and Johnson (2005), practitioners in US family medicine, argue that few
innovations make the ‘break-even point’ wherein they provide real gain to health care.
They note that ‘industry’s technological advancement finds support with the American
public, which marvels over scientific discovery and technological breakthroughs. Robotic
devices and genome mapping are more thrilling than bland quality improvement efforts,
such as reminder systems and organizational redesign, irrespective of whether the latter
saves more lives’ (p. 550).
And indeed, policy-makers themselves acknowledge the limits of medical innovation and
its ultimate social utility: as the Australian Productivity Commission recently concluded,
it is virtually impossible to conclude that a particular technology will always be cost
effective or, for that matter, not cost effective — this will depend on who is receiving it
and the cost effectiveness of available alternative treatments.’ (2005, p. xlv). Moreover,
the focus of health innovation and interventions deriving from it typically neglects some
of the major health problems at a global level, especially in respect to the effect of
infectious disease.
One might presume that the widespread adoption of a health technology is a reasonably
good indication that it must ‘work’, and is effective (in terms of relative cost/benefit
analysis) and efficacious in terms of actually performing a specific function well. But
what counts as failure or success of a clinical intervention can often be difficult to define,
especially in managing chronic disorders such as cancer: does the chemotherapy work,
and what sort of criteria should be used to determine this? Sometimes drugs that have
the desired biochemical effect on the body can produce side effects that can be
experienced as a worsening of the disease itself. This has been observed, for example,
with respect to the effect of drugs – a cocktail known as HAART (Highly Active AntiRetroviral Therapy) – used to manage AIDS.
The growth of medical technologies and their spread, as well as the higher expectations
that patients and primary care practitioners have of them - that they are safe and work has not meant that they have gone unregulated. On the contrary, precisely because
medical technologies have had the capacity to develop rapidly and expensively, the state
has complemented its policy for fostering innovation (such as that being implemented by
the HITF [Healthcare Industries’ Task Force] in the UK) with a regime that seeks to rein
them in, as well as the researchers or clinical practitioners using or hoping to develop
them. There are various ways in which this is done. One of the most commonplace
relates to the governance and thereby evaluation of research proposals or clinical trials
about new technologies through review and approval mechanisms associated with ethics
and research governance. The ‘material narratives’ of consent and research governance
forms1 in the UK health care system act as technologies that relocate a patient from the
1
Lodged with the Workshop Exhibition
2
landscape of therapy to that of research. Technology performs in a different way as part
of a trial or research experiment and is given a special provisional status as one of
becoming, inasmuch as its actual effect (in terms of a physical one) is uncertain or open to
redefinition such that what marks out its specific role in healthcare is yet to be stabilised.
The redefinition of existing drugs through pharmacogenetic clinical trials is a case in
point (Webster, et al 2005).
In contrast, those technologies that have been widely adopted through being licensed for
use are subject to different evaluative mechanisms, related more closely to governing
their economic and clinical efficiencies. These mechanisms can be said to fall broadly
under the umbrella of ‘evidence-based medicine’.
The evaluation discourse of EBM concerns expectations about a health intervention’s
efficacy (can something work) and effectiveness (does it work in specific situations –
especially if alternatives are available?) (see May et al, 2003). Of course STS asks how
efficacy and effectiveness are socially and politically constructed, and thereby why some
medical technologies are seen to succeed and some fail, or indeed why some – such as
drugs – do both. Innovation does not speak for itself in terms of its efficacy or benefit,
but has to be spoken for through, what I call in this paper, the ‘holy trinity’ of evidencebased medicine: randomised controlled trials, health technology assessment (HTA) and
Quality of Life measures (the latter’s purpose to evaluate health interventions from the
patient’s perspective). Indeed my religious analogy to the trinity has echoes in the way
EBM was said, by Sir David Weatherhall, to be ‘a gift from the gods’ (2001). Continuing
the metaphor, in a sardonic commentary Traynor (2000) observes that medics have now
‘found the light’ and been ‘converted’ to EBM.
In this paper I consider some criticisms made of EBM from within the social world of
EBM itself, as well as those from outside, identifying the limitations of this form of
governance not least in terms of the way it frames expectations about new technologies,
and suggest how we might begin to develop a different narrative of evaluation.
Evidence-based medicine: its promise and development
The principal rationale for EBM is that clinical decisions should be informed by the best
available evidence derived from systematic research. While the actual term, ‘evidencebased medicine’, was coined in 1992 by Canadian clinical epidemiologists based at
McMaster University, its origins lie in the work of Archie Cochrane (1909-88) who was a
strong advocate of medical intervention being based on as comprehensive a knowledge
of current research and practice as possible, and of the RCT as a means to establish the
merits of particular interventions. This has since created a global network – the Cochrane
Collaboration – reviewing RCTs, HTA reviews, clinical evaluations and so on.
Mykhalovskiy and Weir (2004) describe EBM as ‘…one of the most important and
successful initiatives to recompose contemporary biomedical reasoning and practice’
(p.1060). This recomposition, a sort of evidential triage that filters off good from poor
information and practice, depends on access to as wide a range of information sources as
possible. Indeed Nettleton et al. (2004) actually locate EBM within a wider sociotechnical frame, viz, that of ‘informatised’ medicine. They argue that:
‘ medical evidence is collated and disseminated not by clinically trained practitioners, but
by experts in information science. Again, we see that medical knowledge has moved
3
beyond the clinic and is available ‘over the wire’– primarily on databases available on the
internet. Clinical decision-making is rooted not in the intellect of the practitioner but
within health ‘intelligence’ which is again an information science.’
They go on to argue that:
‘There is an affinity between the technological form of ICTs and the nature and form of
medical practice, in that vast quantities of information are systematically retrieved,
communicated and disseminated in support of the aspiration that clinical interventions
should be informed by evidence rather than experience’.
Citing the work of Fraser and Greenhalgh (2001, 799) they note that:
‘The modern expert, it has been argued, is no longer a person with specialist knowledge,
but rather is “someone who knows how to access knowledge efficiently and judiciously
and who can form conceptual links between seemingly unrelated areas”’.
Not only, then, is evidence-based medicine an important narrative that frames the material
positioning and institutional place of medical technologies within health care it is one
that is itself increasingly mediated through and dependent upon e-based medicine. (see
Figure 1 below).
Figure 1:
There has been considerable effort expended over recent years to ensure that the ways in
which evidence might be seen to be meeting certain standards, and so act to govern the
selection of therapies of diagnostics etc, conforms to an agreed set of evaluative rules or
protocols. The Centre for Evidence-based Medicine at Oxford has played a key role in
4
this regard, developing a range of criteria against which evidence about new technologies
can be evaluated. These go from mere ‘expert opinion’ to ‘systematic reviews with
homogeneity’ of measures used to assess the value of new technologies. While these are
now routinely used to govern expectations about the merits of interventions, even the
Oxford Centre acknowledges their limitations. As they observe:
‘…these levels and grades speak only to the validity of evidence about prevention,
diagnosis, prognosis, therapy, and harm. Other strategies must be applied to the
evidence in order to generate clinically useful measures of its potential clinical
implications and to incorporate vital patient-values into the ultimate decisions’ (2001).
Here expert opinion is reinstated as of value as is the view of the patient, such that in
practice, clinical expertise should, it is argued, be governed by three forms of
information: information about the clinical state and condition of the patient, the
patient’s preferences, and evidence-based research. However, how these three are to
work together is rarely explained, and it is this failure that informs much of the critique
made of EBM by those working within the epistemic community of health evaluation
itself.
The internal critique
Thus, despite the recognition among EBM modellers of the role of clinical expertise and
patient values, there is little in the literature on this: as Kristiansen and Mooney (2004)
say with respect to the first,
‘EBM in reality offers no methods for improving clinical skills, or takes it for granted
that doctors obtain clinical expertise by practising medicine’ (p. 18).
Moreover, in regard to patient values, the economic rationale for EBM favours a
standard ‘currency’ in valuing outcomes (eg 3 deaths equivalent to 5 severe strokes’ [
Sinclair 2001] ): but this may not only bear little relation to patient experience or values
but also vary between countries and over time, undermining the universalist claims upon
which EBM is based.
The RCT is of course typically regarded as the (universal) gold standard upon which
EBM’s real foundations lie. The use of RCT’s test a specific intervention – say a new
drug, device, or diagnostic test – across a target patient group. RCTs are designed to
identify the safety and efficacy of interventions and are organised as a sequence of
‘phases’ from pre-clinical tests, through to a full trial on the target patient group. The
objective of such trials is not only to test a new (or indeed existing) intervention but
thereby to provide a more rigorous basis on which clinicians can be advised what to use,
when and with what groups of patients. The assumption is, that these results can be
generalised across equivalent populations beyond the trial group. Clearly, the warrant for
this depends on the ways in which the initial sample was derived, and how representative
this is of those for whom the intervention is ultimately aimed for.
Criticisms made of RCTs (Healy, 2006) have focused on this question of
representativeness, noting that drugs trials often begin with healthy, young, typically male
volunteers in determining the toxicity of new compounds. Not surprisingly, down the
line, following approval such drugs can have quite severe side effects on those prescribed
them who do not conform to the norm built into the sample population. Adverse drug
reactions – the side effects of taking drugs that have been approved by regulatory
5
agencies – are a major source of morbidity and mortality: in the US in 2004 they
accounted for 104,000 deaths (Martin et al., 2006). This in part is a result of the
combined effects of drugs – the problem of ‘polypharmacy’ especially significant among
older patients being treated for multiple, rather than one, condition. Moreover, RCTs
assume stable pathologies or diseases that can be addressed by standardised treatments,
inappropriate therefore for mental health or more behaviourally based interventions
(Slade, 2001).
Internal criticism has also noted that RCTs can be as flawed as ‘observational studies’
since there is no external gold standard through which to determine which of these two is
more or less biased. Advocates of RCTs will respond to this by claiming that the use of
meta-analysis of a range of published RCTs in the area acts as a validation for evidencebased claims. But critics (such as Lindbaek and Hjortdahl, 1999) have argued that criteria
used to include some and exclude other RCTs are often unclear, that comparisons may
be poor, and even that quite independent meta-analyses of the same subject may draw
very different conclusions. This should not be surprising to STS, not least as the very
process of conducting and completing an RCT is often a highly precarious social
endeavour: as Mort (2003) has put it,
‘Clinical research practitioners undergo heroic struggles to produce Official Evidence.
Most often they find that the RCTs they try to use will just not work, since nothing
stays in place. The technologies are constantly changing, or being modified; the
patients are constantly moving about criteria; and often clinical colleagues just cannot
be relied on to refer or recruit or enrol either themselves or their patients into the
studies constructed to produce Evidence’
Overall, the internalist critique argues that EBM acts as a narrative that frames
technology and its material expression in terms that are managerially and resource-driven,
and fosters a discursive culture of standardisation that is in practice probably unreal but
has the effect of marginalising the patient and his or her views of the value and utility of
clinical interventions.
More generally, the EBM enterprise has also been criticised for moving beyond its initial
scope and expertise – evaluating technologies and interventions to improve clinical
decision-making – to wider judgements about health priorities and the social agenda for
health. In addition, it has been criticised for oversimplifying the social and economic
context within which health care is chosen and delivered. As one critical paper has
observed, EBM ‘…involves reducing problems to the exploration of simple input-output
relationships – essentially a set of answers. But they are answers to questions that
decision-makers have not asked and relate to problems they do not face’ (Birch and
Gafni, 2004, p. 60). It is arguable whether EBM is addressing questions that have not
been asked by policy-makers, since this would make the huge government investment in
the field both surprising and inexplicable: government seeks the governance of EBM
precisely to manage health technology resources. However, it is more acceptable perhaps
to argue that it deals with ‘problems they do not face’, for such problems relate less to
input-output relations and more to the political management of health expectations, a
much more demanding form of governance beyond the capacity of EBM to deliver (as is
evident in the controversies surrounding recent NICE decisions).
6
The external critique made by STS
Beyond internal debate in the field, there are a number of challenges that can be made to
EBM from STS (see Timmermans and Berg, 2003). These relate to the relationship
between knowledge and outcomes, the meaning attributed to the value of things, and
how the latter relates to the importance of context.
In regard to knowledge and outcomes, the role of expertise increasingly contends with its
own reflexive embrace with uncertainty. Indeed, Beck (2000, p 217) sees this as an
inescapable condition of contemporary expertise:
‘The inability to know is becoming ever more important in modernity…[within] highly
developed expert rationality’.
There are a number of ways in which this inability is expressed, but one of the more
obvious is in terms of the production of unanticipated effects and risks. Expertise does
not address unknown unknowns (Wynne, 2005) that are inevitably involved in the
implementation of socio-technologies in health care. For example, while in theory ICTs
associated with telehealthcare may be cost-saving per unit basis they may generate
unanticipated costs as they help unwittingly to redefine risk thresholds associated with
health disorders and create new patient/carer demand for intervention and monitoring
(Royal Society, 2006). Such uncertainty explains the common preoccupation with risk
management through the scrutiny, audit, and surveillance cultures that prevail in
modernity (Bauman, 2000).
In terms of the value of technologies, Callon (2005) speaks of the metrological work that
is required to create an agreed value attributable to a product or service, such as a new
medical intervention. As he says,
‘The characteristics of a good are not properties which already exist and on which
information simply has to be produced so that everyone can be aware of them. Their
definition or in other words their objectification implies specific metrological work and
heavy investment in measuring equipment. The consequence is that agreement on the
characteristics is sometimes in fact often difficult to achieve’ (Callon et al. 2005)
EBM may indeed reflect ‘heavy investment in measuring equipment’ but agreeing a
technology’s value, its objectification, is especially difficult where the contingencies of
context play an important role. As Berg has shown in his own analysis of telehealthcare
and its value,
Asking the question whether Electronic Medication Systems can prevent medication
errors without integrally taking its practices of use into account is like evaluating the
efficacy of a drug without taking note of how and when and with what other
substances the drug is taken. (Berg, 2005)
To a large degree it is the need to go beyond trialing and to review technologies in their
‘practices of use’ after their more general adoption that led to the establishing of HTA as
a way of monitoring technologies. The state’s desire to manage the potential economic
and legal risks associated with innovation has seen priority being given to ‘early warning’
about the likely impact of a new development. HTA is now a major enterprise
commissioned from public and private sector providers: the University of York’s Centre
7
for Health Economics employs over 100 staff to undertake such work for the UK and
overseas governments.
Critique of HTA from within STS has been limited (see Faulkner, 1997). But some of the
questions that have been raised about HTA (see Lehoux and Blume, 2000; Webster,
2004; Berg et al., 2004) relate to its tendency to review technologies as discrete
innovations rather than, as is often the case, part of wider integrated systems; its
presumption that it can deploy a similar methodology in measuring ‘success’ across
diverse medical fields; its failure to engage with more qualitative measures of evaluation,
and more generally, a failure to recognise that evaluation is a contested terrain involving
different sorts of evidence related to different sorts of context (such as the experimental
derived from clinical trials, evidential, derived from existing clinical practice, and
experiential, based on patients’ experience of an intervention). There has been some
attempt within the HTA discipline to respond to these criticisms (Giacomini, 2005) and
develop more sophisticated methodologies that provide for more complex evaluation
attentive to context (Sculpher, et al 2006; Ades, et al, 2006; Gabbay and Walley, 2006).
But as Blume (2005) has argued, the HTA field developed within a specific political
context (that promoting evidence-based medicine [EBM] and the claim to be able to
generalise results from one study) and this has this limited what it can do. In many ways,
the sort of data deemed key to the HTA world reflects the interests of more powerful
actors within government and health delivery: certain ‘variables’ used for assessment are
deemed relevant others not so, and much of the latter relates to context of use, and how,
ultimately, the patient benefits. Blume has argued strongly that HTA neglects the
political, social and ethical dimensions of new technologies, and has argued for a more
socially and politically informed HTA.
However, we should not thereby assume that this attempt is always successful or indeed
based on a proper understanding of the target patient group. In fact, as Blume (1997) has
shown with respect to his study of the cochlear implant used to treat deafness in
children, the deaf community were highly critical of the implant technology not only
because of its costs and side effects, but also because designers presumed no value
should be accorded to the use of sign language and the distinctive deaf culture that the
community holds in high regard.
Furthermore, where the design of a technology inscribes (i.e. presumes) specific contexts
of use, but that technology is deployed in a different setting, we can expect that users will
struggle to give value to it in the way envisaged by the innovator. A good illustration of
this is the study by Heaton et al. (2006) of the use of life-sustaining technologies for
seriously and chronically sick children, devices originally developed for use in a hospital
setting. The meaning, combination, use, effectiveness and efficacy of the devices and
systems (such as assisted ventilation and intravenous feeding) are quite different in the
home, where complex care regimens have to be managed by the children’s families
(including siblings) in conjunction with statutory and voluntary services. Many such
technologies work only through constant monitoring and intervention which in the 24/7
nursing shifts of a hospital is more easily achieved. Parents were also responsible for
preparing the equipment and providing supplies: the responsibilities and demands on
parental time were extensive. As Heaton (in Seymour 2006) observes, ‘While families
were, to varying degrees, able to incorporate aspects of the care regimes into their
everyday routines, at the same time, they experienced a range of difficulties as a result of
the time demands of the care regimes being incompatible with other domestic,
institutional and social schedules.’ (p. xxx).
8
This localisation of evidence-based care regimes echoes a more general observation
relating to the globalising aspirations of HTA itself. As Orvain (2004) has argued, HTA
faces the problem of translating a global appraisal and set of recommendations to the
local level. He points to the tensions between the abstract, global and the concrete, local levels
and the multiplicity of social groups who shape the evaluation process; it is their very
diversity that creates a multidimensional object, a range of competing material narratives
of what it actually is and does:
‘…it is clear that viewing any object (e.g. a medical device) from more than one angle
(confrontation of expert opinions) often best reveals the multidimensional structure of
the object (the crux of the matter)’.
Prior to the arrival of HTA analysis of new technologies, patient experience was a matter
that had been taken up by clinicians, who recognised that though the biomedical
outcome had been achieved (say a lowering of blood pressure), the sense of wellness for
the patient had not changed. This led in the 1970s to the development of ‘Quality of life’
(QoL) measures, which as Armstrong (2006) has argued, recognised the need for more
sensitive therapeutic markers of the patients well-being than simply those of lives saved
or lives extended. That is, they were designed to try to capture the patient’s subjective
experience of health and the value of health interventions, with respect, for example, to
their being able to live as ‘normal’ a life as possible. However, as Armstrong also shows,
QoL measures (of which there are now over 800) are a contested area, since they act as
‘social evaluations of the “good life” and these will both change over time and are likely
to differ for different people and groups in the population’ (2006). Again, the social
construction of what ‘works’ or has beneficial effect is evident.
From a reliable to a socially robust evaluation
From the discussion above, derived from both the internal and external critique of EBM,
it would be reasonable to argue that three forms of evidence need to be brought together
to produce a more socially robust evidence base for clinical practice. These three are the
experimental (for we should not abandon the search for safety and efficacy, and indeed
effectiveness), the evidential (that which pertains to expert judgement for the inability to
know does not thereby imply the inability to advise and act), and the experiential (the
patients’ own narratives of the utility of existing and new medical technologies in the
highly indexical and contingent circumstances of their own lives).
I can illustrate briefly how these three construct quite different material narratives of a
specific technology, accounts that are both descriptive and evaluative and carry
expectations about the effect the technology will have on illness. The technology in
question is the ACE inhibitor used to manage cardiovascular disorder.
The material narrative of the experimenter runs as follows:
An RCT in asymptomatic people after myocardial infarction with documented left
ventricular systolic dysfunction found that an ACE inhibitor (captopril) reduced
the risk of all ischaemic events, all myocardial infarctions, and fatal myocardial
infarctions compared with placebo (all ischaemic events: 29% with captopril v
33% with placebo).
Source: Clinical Evidence (BMJ) 1/4/2006
9
The material narrative of the practitioner modifies this and gives the materiality a more
differentiated form:
ACE-Is are highly effective anti-hypertensive drugs that lower blood pressure and
help reverse left ventricular hypertrophy. They are particularly useful in young
patients because of the low incidence of impotence and have a special role when
hypertension is associated with co-morbidity such as heart failure, diabetes or
coronary artery disease. Patients of African origin appear relatively less responsive
to treatment and high doses may be necessary. The efficacy of ACE-Is is
enhanced by their co-prescription with diuretics.
Source: BHF Fact file 2002
In contrast to both, a patient’s material narrative characterises ACE-Is in terms of their
more immediate impact on the body and its symptoms:
‘…When you first start to take it you feel absolutely stinking actually, and
apparently it’s not uncommon...Just like the flu, flu symptoms, continual flu
symptoms. And then as soon as you got rid of the symptoms on two and a half
(milligrams), they’d give you five!...(but) it prevents strokes and one thing and
another, so at this moment in time it’s the best one there is.’
Source: Cowie, 2006.
These narratives move from the abstract, through the typical, to the personal and though
linked to a common material base – the ACE- Inhibitor – they configure the technology
in quite different ways. The question is whether we can bring these together, or at least
make them work together in a more holistic, rather than ‘holy’ trinity of evaluation?
I doubt that this can be done as a methodological exercise, that is as one through which
we construct some sort of STS-inspired algorithm through which evaluation decisions
about existing/new technologies can be made. As Nowotny has argued, social robustness
is not about an absolute concept of reliability but about social relationships through
which reliability can be secured.
In light of this, it seems that our starting point must be more about the sort of evaluative
approach we adopt that can attend to these distinct narratives in such a way as to build
relations that foster both a clinical and social reliability. There are at least three features
to this approach we can nominate in light of the discussion above:
First, consider the contrasting material narratives of three forms of evidence and how
these might be brought together: eg a move from an emphasis on outcomes to one on
process and context of use, recognising that the aggregative knowledge of EBM needs to be
complemented by the qualitative knowledge of prospective users of IHTs (Coulter,
2004). Another way of putting this is to ask whether and how far different forms of
evidence attend to social relationships: in terms of the ACE-I, the RCT evidence blanks
on social relationships assuming a context-free applicability of the compound (though
this comes adrift in the PIL accompanying the blister pack of pills when the contra
indications are specified); in contrast, the experiential evidence points to coping relations
and a personal risk biography that help to make sense of the drug and why it should
continue to be taken. What we should move towards is evaluative techniques that give
10
primacy to the impact of socio-technologies on relating and relationships (Bauman, 2000)
and how they can help enhance rather than disembed or disrupt both;
Second ask how material narratives of health technologies encompass social relationships
that are more broadly implicated in a technology that is simply part of a wider set of
socio-technical relationships: patients, carers, and health and social care agencies are to
differing degrees of granularity involved in the management of chronic heart disease;
Finally, recognise that dominant material narratives tend to hide the dual effects of health
technologies: as Lehoux (2006) observes, ‘health technology generates advantages and
disadvantages simultaneously, disclosing the dualistic effects of health technology becomes a key
avenue [of investigation]’ (p.199). This might be in terms of the immediate physical, say
side, effects of (ACE-I or other) technologies, or more complex effects that reflect the
‘novel’ relationships that they create (Webster, 2007)? As Lehoux (2006, p. 167) has
noted, ‘Medical innovations…create novel interdependencies’. What are the continuities
and discontinuities of technologies and how are these experienced within time and space
(often not at the same time or place)?
Patients, carers, clinicians, and regulators confront both of these continuities and
discontinuities though not necessarily at the same time. As Franklin and Roberts
(2005) have shown in their examination of PiGD, the frozen embryo which is selected
for pre-implantation in the IVF clinic evokes a sense of reproductive continuity when
it is seen the first step towards paternity, and in spatial and temporal terms
encompassed, embraced, by the rhythm and narrative of ‘having a child’. Yet, the
same prospective parents and clinicians frame a second embryo very differently when
donated for embryo research, outside of the reproductive domain. This second
embryo occupies a very different spatial and temporal universe that is populated by
different social actors and so creative of new arrangements not least through its
performing a role in conjunction with, in Barry’s (2001) phrase, ‘other objects and
activities’, such as research labs, biocapital, and the ‘tissue economy’. The evaluation
of IVF should, I am suggesting, encompass both continuity and discontinuity for in
doing so it is sensitised in a direct way to the importance of social relationships that
create the very basis on which continuity and change work.
It is unlikely, of course, that the epistemic culture that is EBM will adopt these
perspectives on evaluation, or indeed, even be able to embrace them as such. Indeed,
evaluation practices are, as I have noted, more about closing down options (and
resources) than opening up and problematising the meaning and use of technologies.
Moreover, despite the significance of EBM as a form of evidential triage, there are many
technologies that have not been subject to any of its (HTA or other) metrics: their
certainty (in terms of efficacy and effectiveness) might on the face of it be as uncertain as
their uncertainty!
However, there are some working within the field of HTA who are now beginning to
treat uncertainty and contingency seriously, building on Bayesian approaches to
assessment that presume rather than deny uncertainty. This may well offer STS an
opportunity to enter the health policy community and begin to construct a critical
assessment that draws on some of the models already built through the constructive
technology assessment work undertaken in the Netherlands (Rip et al. 2000).
11
While the sociological important of context of use is therefore key to understanding how
technologies are experienced, evaluated and seen to ‘work’, we should not lose sight of
the way that priorities served by medical innovation systems function more generally.
Here we need to attend to the structuring of health care delivery rather than simply to
specific issues about context per se. That is we can ask what broader social utility is
produced by a biomedical model of health innovation that, as Bury (1998) notes
produces a system devoted on the one hand to high tech biomedical care and on the
other management of chronic disease – from a regime of ‘treatment and cure’ to
‘management and care’ (p.267).
Despite the criticisms levelled at EBM and its component elements, they are used more
and more heavily by health agencies in affluent states to assess the merits of health
innovation. Its three different forms of evaluation (RCT, HTA and QoL) act not only to
evaluate existing health technologies, but also shape how new technologies are being
developed. Health innovators are aware that these evaluations will be used to determine
the relative merits of their product or procedure. An important STS question then arises
that impinges upon the material positioning of technologies in the healthcare system:
how far do designers endeavour to construct them in such a way as to anticipate and
meet the evaluative criteria against which they will be judged? In other words, how
might such criteria be ‘inscribed’ (Akrich, 1992) within the very innovation itself? As
Akrich argues,
“Designers thus define actors with specific tastes, competences, motives, aspirations,
political prejudices, and the rest, and they assume that morality, technology, science, and
economy will evolve in particular ways. A large part of the work of innovators is that of
‘inscribing’ this vision of (or prediction about) the world in the technical content of the
new object … an attempt to predetermine the settings that users are asked to imagine”
(1992, p. 208)
If there is this feedback at work, it is an empirical question to answer how far does such
inscription vary by technology field, if at all, how does it act as a material narrative to
shape that field, and how far does that then shape the evidential and experiential
narratives that engage with it? This is a key area for future research in STS in seeing how
evaluation as a form of governance is instantiated across different domains associated
with the production and use of health technology.
Conclusion
This paper has argued that evaluation regimes in contemporary health technology act as a
form of governance that as such produce material narratives that shape and steer the
existing and prospective use of technologies in health care systems. I have argued that
the principal metrics of EBM (RCTs, HTA and QoL) serve to rationalise the provision
of new technologies but do so without any real integration of the forms of evidence on
which they depend. However, I have also suggested that such an integration would be
highly problematic on both methodological and normative grounds since what is central
to a more socially robust form of evaluation that attends to the multidimensionality of
technologies is one that attends not to absolute measures or gold standards but to
everyday social relationships and the interdependencies that in the end make technologies
work and have utility. Social efficacy and effectiveness and the material narratives that
give them expression should become central concerns for an evidence-based medicine
that would thereby be based on a socially robust form of governance.
12
References
Ades, A.E., Sculpher M.J., Sutton, A., Abrams, A., Cooper, N., Welton, N, et al. (2006)
Bayesian methods for evidence synthesis in cost-effectiveness analysis. Pharmacoeconomics
24:2-29.
Akrich, M. (1992) The de-scription of technical objects, in W. Bijker & J. Law (eds.),
Shaping Technology / Building Society: Studies in Sociotechnical Change,
Cambridge, MA: MIT Press, pp. 205-224
Armstrong, D. (2006) Evaluation as an Innovative Health Technology, in Webster, A.
(ed.) New Technologies in Health Care: Challenge, Change and Innovation, Palgrave: Basingstoke,
pp 232-42.
Australian Productivity Commission (2005) Impacts of Advances in Medical technology in
Australia, Canberra.
Barry, A. (2001) Political Machines: Governing a Technological Society. London: The Athlone
Press/Continuum Books.
Bauman, Z. (2000) Liquid Modernity, Polity, Cambridge.
Beck, U. (2000) Risk Society Revisited: Theory, Politics and Research Programmes, in
Adam, B. et al. (2000) The Risk Society and Beyond: Critical Issues for Social Theory, Sage:
London.
Berg, M. et al. (2004) Technology assessment, priority setting and appropriate care in
Dutch health care, International Journal of Technology Assessment in Healthcare, 20:35-43.
Berg, M. (2005) Clinical Quality Development and the Emergence of the EHCR, EPJObservatoriets Arskonference, October 26.
Birch, S., and Gafni, A., (2003) Economics and the evaluation of health care programs:
generalizability of methods and the implications for generalizability of results, Health
Policy, 64:207-219.
Blume, S. (1997) The Rhetoric and Counter-Rhetoric of a “Bionic Technology.”
Science Technology and Human Values 22:31–56.
Blume, S. (2005) HTA: a critical re-appraisal, Innovative Health Technologies
International Conference, Rome, June
Bury, M. (1998) Postmodernity And Health in G. Scambler and P. Higgs (eds) Modernity,
Medicine and Health: Medical Sociology Towards 2000 (Routledge)
Callon, M. (2005) Laws of the Markets, Blackwells, Oxford.
Coulter, A. (2004) Perspectives on health technology assessment, International Journal of
Technology Assessment in Health Care, 14: 135-144.
13
Cowie, L. (2006) The Value of Medicines, Doctoral Thesis, Department of Sociology,
University of York.
Faulkner, A. (1997) Strange bedfellows’ in the laboratory of the NHS? An analysis of the
new science of health technology assessment in the United Kingdom. In: Elston MA (ed)
The sociology of medical science and technology. Sociology of Health and Illness Monograph.
Foucault, M. (1988) Politics, Philosophy, and Culture: Interviews and Other Writings,
1977-1984, edited by M. Morris and P. Patton, New York: Routledge.
Franklin, S. and C. Roberts (2005) Born and Made: An Ethnography of
Preimplantation Genetic Diagnosis, Princeton University Press, Princeton.
Gabbay, J. and Walley (2006) Introducing new health interventions, British Medical
Journal, 332:64-65.
Giacomini, M. (2005) Health technology as policy. IHT/HTAi International Conference,
Rome ( see http://www.york.ac.uk/res/iht/events/HTAiRome2005.htm )
Harris, B. (2004) Public Health, Nutrition, and the Decline of Mortality: The
McKeown Thesis Revisited, Social History of Medicine: 17: 379-407.
Healy, D. (2006) Manufacturing Consensus,
Heaton J, Noyes J, Sloper P and Shah R. (2006) The Experiences of Sleep Disruption in
Families of Technology-dependent Children Living at Home, Children & Society 20: 196–
208
Kristiansen, I.S. and Mooney, G. (eds.) (2004) Evidence-Based Medicine: In its Place,
Routledge, London.
Lehoux, P. (2006) The Problem of Health Technology: Policy Implications for
Modern Health Care Systems, London: Routledge
Lehoux, P. and S. Blume (2000) Technology assessment and the socio-politics of
health technologies, Journal of Health Politics, Policy and Law, 25:1083-1120.
Lindbaek, M and Hjortdahl, P. (1999) How do two meta-analyses of similar data reach
opposite conclusions? BMJ. 1999 March 27; 318(7187): 873.
May, C.R., Harrison, R., Finch, T., MacFarlane, A., Mair, F.S., and Wallace, P.
(2003) Understanding the normalization of telemedicine services through qualitative
evaluation. Journal of the American Medical Informatics Association 10: 596-604
Martin, P. et al., (2006) False Positive? The Clinical and Commercail Development of
Pharmacogenetics, IGBis/SATSU publication.
McKeown, T. (1976) The Role of Medicine – Dream, Mirage or Nemesis, London
14
Mort, M. (2003) Evidence in Action. The Craft of Medicine: A Dialogue Between
Practitioners and Researchers, Workshop, Lancaster University.
Mykhalovskiy, M. and Weir L. (2004) The problem of evidence-based medicine:
directions for social science, Social Science & Medicine, 59: 1059-1069
Nettleton, S., Burrows, R. and O’Malley, L. (2004) Health E-types: an analysis of the
everyday use of the internet for health Information, Communication and Society (special issue
on E-Health), vol. 7, no. 4.
Nowotny H. (2006) The Problem of Transdisciplinarity, , Interdisciplines
http://www.interdisciplines.org/interdisciplinarity/papers/5).
Orvain, J. et al.(2004) Overview of health technology assessment in France, International
journal of technology assessment in health care, 20, no1 : 25-34.
Rapp, R. (1999) Testing Women, Testing the Fetus, London: Routledge
Rip, A. et al. (1995) Managing Technology in Society: Approach of the CTA, Pinter, London.
Royal Society (2006) The impact of information and communication technologies on health and
healthcare, Royal Society Publications, London (forthcoming).
Sculpher, M.J., Claxton, K., Drummond, M., and McCabe, C. (2006) Whither trial-based
economic evaluation for health care decision making, Health Economics, 15:677-687.
Slade, M. 2001. Are randomised controlled trials the only gold that glitters?, The
British Journal of Psychiatry (2001) 179: 286-287.
Timmermans, S., and Berg, M. (2003). The gold standard: The challenge of evidence-based medicine
and standardization in health care, Philadelphia: Temple University Press.
Traynor, M. (2000) Purity, conversion and the evidence based movements. Health, 4,
139–158
Webster, A. (2007) Health, Technology and Society: A Sociological Critique, Palgrave,
Basingstoke (forthcoming).
Webster, A. et al. (2005) Integrating pharmacogenetics into society: in search of a model,
Nature Reviews: Genetics, vol 5: 7-14.
Webster, A. (2004) Health Technology Assessment: a Sociological Commentary on
Reflexive Innovation, International Journal of Technology Assessment in Health Care, 20, No 1:
1-6.
Woolf, S. H. and Johnson, R.E. (2005) The Break-Even Point: When Medical Advances
Are Less Important Than Improving the Fidelity With Which They Are Delivered Am J
Public Health, 95: 1306-1307
Wynne, B. (2005) Reflexing Complexity Post-genomic Knowledge and Reductionist
Returns in Public Science, Theory, Culture & Society, 22(5): 67–94
15