Download Introducing the Problem of Risk in Forensic Science

Christopher Lawless – Kings College Risk Research
Symposium 5th June 2009
Introducing the Problem of Risk in Forensic Science
Introduction
The last decades have seen a notable increase in the use of scientific evidence in criminal
investigation, a development which has captured the imagination of the media, publics
and policymakers alike. However, questions also continue to be raised about the judicial
failures which continue to arise due to the alleged faulty use of forensic science. 1
Concerns have continued to be expressed over the assumptions, techniques and practices
used by forensic scientists in a number of areas of work ((Kennedy 2003, Robertson
2004, Saks and Faigman 2008). Such concerns have contributed to a growing interest in
the adoption of more risk-conscious epistemological approaches, in an attempt to render
forensic science reasoning processes not only more commensurable with conventionally
regarded standards of scientific inquiry, but also in ways which acknowledge the risks to
justice from the erroneous interpretation of forensic evidence. According to some
scholars, a new paradigm has arisen which accepts the conditionality of forensic scientific
claims and the epistemic risks associated with the interpretation of evidence (Saks and
Kohler 2005). This ‘paradigm shift’ has involved a move away from categorical
assumptions of uniqueness of identification evidence2, to a position which accepts
epistemic risk in evidential interpretation, and advocates the use of probabilistic methods.
Bayes Theorem has become prominent as a philosophical basis for the development
of interpretation methods which reflect the conditions of uncertainty typical of
criminal investigation.
In this paper I introduce issues relating to risk in forensic science, by way of a notable
example. A focus on the Case Assessment and Interpretation (CAI) model
demonstrates how forensic scientists have attempted to balance epistemic risk
management concerns with the realities of the criminal justice system, which has
itself been subject to a degree of neoliberal reform. I proceed to show how attempts
to introduce this new sensibility have opened up areas of controversy regarding the
role of forensic science vis-à-vis the wider criminal justice system within which it is
embedded. I consider this problem in the context of theoretical debates which have
emerged from Science and Technology Studies (STS).
Origins of the ‘New Paradigm’ in Forensic Science
Attempts to introduce Bayes Theorem into forensic science date from around the
beginning of the 1960s, and can be traced back to the work of both forensic scientists
and legal scholars. In what follows I briefly trace the origins of this development.
In the early 1960s, Paul Kirk, a leading figure in US forensic science, published an
article entitled ‘The Ontogeny of Criminalistics’ in which Kirk reflected on the extent
to which forensic inquiry could be considered a science in its own right. Marking a
1
Recent high-profile examples in the UK include cases involving Barry George, Sally Clark, and the
rejection, and subsequent heavy criticism, of DNA evidence involved in the Omagh bomb trial.
2
Which has been referred to as the ‘positivity doctrine’ by Broeders (2007)
1
particularly notable attempt to delineate the disciplinary boundaries of forensic
inquiry, Kirk defined criminalistics as the ‘science of individualization’ (Kirk 1963,
p.236).3 In a series of papers Kirk and his student Charles Kingston outlined the
way in which an advanced appreciation of statistics and probability could be used to
enhance ‘interpretive areas of criminalistics’, the first significant attempts to provide a
systematic probabilistic approach to evidence interpretation (Kirk and Kingston 1964,
p.514). In two papers concerning the ‘Applications of Probability Theory to
Criminalistics’ (1965a, 1965b), Kingston discussed the use of probabilistic models for
the assessment of certain forms of transfer evidence, and included a favourable
consideration of an explicitly Bayesian model (Kirk 1965b).
Around the same time, legal scholars also began to consider how Bayesian probability
could be used to overcome the risks of misinterpretation of evidence, as exemplified
in controversial cases such as People vs Collins.4 Kaplan (1968) argued that decision
theory, which had largely been developed in the fields of economics and business, and
which utilised Bayesian probability, enabled a better understanding of the
epistemological processes involved in the judicial arena (Kaplan 1968). In another
influential article, Finkelstein and Fairley (1970) questioned the uniqueness principle
which had predominated in forensic science.5 They argued that it was both wrong and
dangerous to attribute uniqueness to evidential data, and that in many cases,
identificatory evidence cannot be claimed to be unique, merely rare (Finkelstein and
Fairley 1970, p.516). In order to assess and express weight of evidence, they cited
Bayes Theorem as an appropriate means for translating frequencies of rare events into
a probability assessment. At the heart of their argument was the point that ‘unique’
characteristics could not be definitively attributed to an individual, but that certain
infrequently occurring forms of transfer evidence could actually demonstrate
significant probative weight if incorporated into a Bayesian formula. Hence they
argued that individualisation could be quantitatively assessed, if not categorically
proven, via the use of Bayesian reasoning.
The Bayesian approach to evidential reasoning began to receive further support, most
notably via the influential work of Ian Evett, a statistician based at the UK Forensic
Science Service (FSS). Evett collaborated with the distinguished Bayesian scholar
Denis Lindley (Evett and Joyce 2005), and made significant advances in applying
Bayesian approaches to trace evidence analyses (Evett 1984, 1986, 1987). These and
other studies significantly contributed to a growing perception of the potential of
Bayes Theorem to provide a more scientifically robust basis for the assertions of
The term ‘criminalistics’ is largely synonymous with forensic science, although it is more frequently
used to refer to the analysis of trace evidence.
4
This case involved the trial of an inter-racial couple for a robbery in California of an elderly woman.
The victim claimed that her assailant had been a young Caucasian woman with blond hair tied in a
ponytail, who escaped in a yellow car driven by an African-American man who wore a beard and
moustache. The prosecution case rested on the argument that the chances of selecting any couple
possessing these characteristics would be one in twelve million, a figure that was obtained by using the
product rule, under the assumption that these characteristics occurred independently of each other, and
hence could be regarded as providing a unique description of the offenders. The California Supreme
Court ruled against this prosecution argument, viewing the use of probability theory in such a manner
as erroneous. The assumptions of independence for each characteristic were ruled as erroneous; for
example, the probability of a man possessing a beard and a moustache was not seen as encompassing
two strictly independent features.
5
See Tribe (1971) for a critique of Finkelstein and Fairley.
3
2
forensic scientists. Evett saw the use of Bayes as an ideal solution to the unique
problems experienced in forensic scientific work:
That framework—call it Bayesian, call it logical—is just so perfect for forensic science. All
the statisticians I know who have come into this field, and have looked at the problem of
interpreting evidence within the context of the criminal trial, have come to see it as centring
around Bayes’s Theorem.’ (Evett and Joyce 2005, p.37).
The use of Bayes was seen as aiding the transition from a categorical epistemological
position (the uniqueness assumption), to a conditional paradigm which aspired to
improved standards of scientific propriety. In enabling the reasoner to update one’s
beliefs in a sequential, iterative manner, Bayes Theorem became viewed as being well
suited to the contingencies of a criminal investigation. The adoption of a quantitative
measure of subjective belief was also understood to provide a more accountable form
of reasoning than the opaque, intuitive methods that had traditionally been associated
with several forensic disciplines.
Bayesian reasoning now forms the basis of a wide range of forensic technologies and
initiatives. A particularly notable development in this regard concerns the Case
Assessment and Interpretation (CAI) model, which represents an attempt to construct
an all-encompassing method of evidential interpretation for use in the course of
criminal investigations. Developed by a team of FSS scientists, the CAI seeks to
continue to shape the process of evidential reasoning in a way that logically accounts
for epistemic risk, and hence promotes the new paradigm.
Furthermore, the CAI also reflects new commercial realities. UK forensic science has
been subject to an increasing degree of commercialisation in recent decades. In 1991,
in response to government reviews which urged the reform of forensic scientific
support in line with the ‘New Public Management’ ethos, 6 the FSS introduced a
system of direct charging to forces and other clients (McFarland 2003). Forensic
science providers such as the FSS hence entered into a new relationship with police,
in which the latter became customers. This development occurred at a time when
responsibility for force budgets became increasingly devolved. Hence the CAI was
intended to reflect this new orientation, ‘a radical change of culture’ where the costs
of forensic science were ‘no longer invisible to operational policemen’ (Cook et al
1998a, p.151). The aim of this initiative is therefore to provide better ‘value for
money’ to customers, and to also give the customer greater say ‘about what work is
done in the laboratory’ (Cook et al 1998a, p.151). At the same time, it is intended to
increase the contribution of forensic scientists to criminal investigations, as pointed
out by one of the authors of the CAI:
‘There should be a focused, directed use of resources to help make progress with the matters
that are of relevance and importance to the customer and for which forensic science can
actually provide useful information. Wherever possible, scientists should be addressing
questions that are a stage advanced beyond the simply analytical.’ (Jackson 2000, p.84,
emphasis added).
Hence the CAI is also intended to promote a more expansive form of scientific
support.
6
See Garland (2001) and chapter 6, Williams and Johnson (2008).
3
Producing and Delivering Bayesian Forensic Inquiry
In broad terms, the CAI is delivered to police users in three overarching phases:
‘customer requirement’, actual ‘case assessment’, and ‘service delivery’ of written
statements of test results together with an assessment of their significance (Cook et al
1998a, p.152). This process is sensitive to feedback, and continually subject to
review with the customer; new lines of inquiry generated by subsequent developments
can be readily incorporated into the framework.
The customer requirement phase involves a rigorous consideration of the client’s
specific needs. Here, the construction of an appropriate ‘framework of circumstances’
is taken to be a necessary pre-requisite for the development of propositions relevant to
the evidence and the case. The authors stress the need for scientists to take a ‘balanced
view’ of each case, in line with what they regard as the principles of ‘the Bayesian
view of evidence, that it is not sensible for a scientist to attempt to concentrate on the
validity of a particular proposition without considering at least one alternative’ (Cook
et al 1998a, p.153).
In the case assessment phase, the scientist is expected to further clarify the
propositions they wish to investigate by re-describing them in more quantitative
terms, via the generation of likelihood ratios, through which the scientist seeks to
organise the logical processing of evidential propositions using Bayes Theorem.
These take the form:
Probability of the evidence if prosecution proposition is true
Probability of the evidence if defence proposition is true
The likelihood ratio (LR) enables the scientist to represent his/her expectations in a more
precise and quantifiable manner. The LR also functions as a key device for enabling
providers and customers to reach agreement on the most efficient course of action to take
in the pursuit of an investigation. Cook et al (1998a) describe how the LR approach can
be used to assess the usefulness of scientific tests in relation to specific cases. In one
example they describe how this framework can be used to assess the significance of
finding certain quantities of glass on the garment of an individual implicated in an alleged
break-in, based on estimated probability distributions. Using these distributions, pairs of
propositions, relating to the case, can be assessed, for example
a) ‘the suspect is the person who committed the burglary’
b) ‘the suspect has had no connection with the crime scene’
Tables 1-3 demonstrate how the probability estimates for finding certain amounts of
glass may affect the value of the LR returned. In doing so, Bayesian reasoning
provides a means of assessing the potential probative value of conducting glass
analysis.
4
Table 1. Probabilities of Finding Quantities of Matching Glass (Reproduced from Cook et al
1998a, p.155)
Quantity (Q)
None
Few
Many
Probability Pr (Q|a)
0.05
0.30
0.65
Probability Pr (Q|b)
0.95
0.04
0.01
Table 2. Likelihood Ratios for the Three Values of Q (Reproduced from Cook et al 1998a, p.155)
Quantity (Q)
None
Few
Many
Likelihood Ratio (LR)
0.053
7.5
65.0
Table 3. Probabilities of Possible Outcomes From An Examination for Matching Glass
(Reproduced from Cook et al 1998a, p.155)
Opinion Expressed
Moderate Support for b
Weak Support for a
Moderate Support for a
Probability of Opinion if a is
true
0.05
0.30
0.65
Probability of Opinion if b is
true
0.95
0.04
0.01
From the information in Table 1, it is possible to obtain LR values by dividing the
proposition pairs corresponding to each matching fragment group, as reproduced in
Table 2. These can be further summarised in terms of the support they show for each
of the propositions (Table 3). This leads to the assessment that there is a 65% chance
that the examination will provide moderate support for prosecution proposition 1a,
and 30% chance that it will only provide weak support. If the suspect is innocent
however, then there is a 95% chance of moderate support for innocence, although
there is also 5% chance that the evidence will falsely incriminate him.
The Hierarchy of Propositions
In the above example, the analytical process is wholly the preserve of the scientist.
Although the decision to conduct glass analysis would remain with the customer, the
circumstances of this specific case do not enter into the consideration of the evidence. As
indicated by the concern to ‘add value’ to investigations however (Jackson 2000, p.84),
the CAI seeks to link these kinds of analysis with the wider framework of circumstances
of a case, so that the investigative process can, in theory at least, continue to proceed
along Bayesian lines. A key element of the CAI is the hierarchy of propositions, which
plays a central role in organising the way in which propositions are constructed and
assessed. This hierarchy classifies propositions in order of their relevance to the ultimate
issue under consideration by the court (Cook et al. 1998b).
Level I, or the source level, relates to propositions concerning the origins of the
evidentiary material, and.will exclusively involve the measurement and comparison of
quantitative data (Cook et al 1998b, p.232-233). Level II in the hierarchy, the activity
level, involves a greater element of reconstruction of the events in each case. Re-visiting
the above example, one may wish to pose a prosecution hypothesis, concerning the
probability that a given suspect smashed the window, against a defence hypothesis
seeking to ascertain the probability that the suspect was not present when the window was
5
broken. If the investigators wish to address the probability of finding a matching quantity
of fragments on a suspect given that it was they who had smashed the window, they
would require further circumstantial information. In this case it would be of use to be
able to ascertain how the window was smashed, and the time interval between the incident
and the removal of the clothing from the suspect for analysis. In order to assess a
corresponding defence hypothesis, in this case the probability of finding matching glass
on the suspect if they were unconnected with the smashing of the window, information
would be sought which would predispose the suspect to have glass present on his
clothing, such as lifestyle or profession (Cook et al 1998b, p.233). Hence the more
information a scientist may possess in relation to the circumstances of the case, the more
fully Level II propositions will be able to considered, and the more bearing that insights
generated at Level I will have on consideration of Level II propositions.
The emphasis on the role of forensic scientist in generating Level II propositions raises
some questions about the division of ratiocinative labour in the course of investigating a
criminal case. The kind of level II propositions outlined in the literature appear to be the
kind of lines of enquiry that one would expect the police to pursue. Hence the remit of
what could be viewed as ‘forensic science’ is possibly stretched; what the CAI can be
seen to represent in this regard is a possible extension of what may be seen as constituting
forensic scientific inquiry. Rather than a concern with the recovery, rendition and
analysis of items of possible evidential interest and importance, ‘forensic science’ under
the realm of the CAI, is extended to constitute a possible method for the advancement of
criminal investigation as a whole.
The final level in the hierarchy, Level III or the offence level, concerns the probability that
a suspect has committed a criminal offence. Level III propositions are the domain of the
jury, assisted by the judge (Cook et al 1998b, p.233). Offence-level propositions may
also be construed as activity-type propositions; however the key difference is that Level
III propositions concern the question of whether an actual crime has occurred.7 Although
Level III propositions are assessed by the jury, and ultimately outside the domain of the
forensic scientist, the latter may well be able to assist the court in their deliberations, by
reporting information concerning likelihoods in a suitably comprehendable form (Cook et
al 1998b, p.233).
Contestations
The previous section has sought to indicate how the CAI has been specifically designed to
embed Bayesian reasoning within the wider context of policing. As I describe below
however, attempts to apply CAI have opened up new spaces of contestation between and
within elements of the criminal justice system, including forensic science itself.
Within the framework of the CAI, the scientists are meant to act in a facilitative role,
helping police to clarify the important questions to ask in the course of a case. For the
CAI to function effectively, and for evidence to exert optimal impact on the course of a
case if considered within this Bayesian approach, forensic scientists need potentially high
levels of information from the police. The CAI also emphasises the desire for scientists to
enjoy a greater degree of influence with regard to which pieces of evidence are seen as
most pertinent to a case. However, it became clear through field research that this
For example, whilst it may possible to prove a Level II proposition such as ‘the suspect had
intercourse with the victim’, more persuasive work is required before a corresponding Level III
proposition, namely ‘the suspect is guilty of rape’ (ie no consent), is proven.
7
6
potential re-positioning of forensic scientists within the decision-making hierarchy of
investigative frameworks was not necessarily in keeping with traditional police views.
Instead, it appeared that police investigators viewed the role of forensic scientists as
somewhat more limited: to simply provide scientific data about evidence, which may
occur in relative isolation to the deliberations made by police concerning the progress of a
criminal case. Moreover, there appears to have been suspicion on the part of the police
with regard to the need to consider defence propositions within the framework of the CAI:
‘We had another group of officers who kind of mis-interpreted really…we had to
re-assure them we weren’t out to help the defendant…’ (Interview, 2007)
Tensions also became apparent with another set of actors – the FSS management
themselves. Although the CAI emphasised engagement with customers as facilitating an
efficient use of the latter’s resources, it appears that the CAI clashed with management
interest in maximising revenue from the same set of actors:
‘…I think [the FSS management] felt some commercial problems… the way we
were working in terms of earning our incomes was items examined, the more
items we examined the more income we got. So there was almost a counterpressure not to apply CAI, because CAI in some ways, said ‘lets just look at the
items that are gonna be really effective, really efficient, in addressing this
question, and if you decide with the customer these are the key issues in the case,
the strategy to address these key issues is this, this and this, it’s not these items or
these tests... and I think therein lay some of the difficulties from the managers
and leaders, because you could see the natural consequences, if we apply
CAI…we’re gonna lose a lot of income, potentially.’ (Interview, 2007)
It was clear that the FSS management saw commercialisation in a different way. Rather
than providing tests in a selective and targeted manner specific to customer needs, FSS
management viewed forensic scientific ‘products’ as being provided to customers on a
wholesale basis. In this view, a ‘product’ was packaged as a specific type of test, (e.g.
DNA analysis), to be sold to customers in multiple ‘units’.
‘In the FSS now, they’ve gone away from experienced case scientists looking at
multiple evidence types and synthesising all of the evidence together to neat little
boxes of ‘products’ where ‘you want the hair analysed?’ (Interview, 2008)
Here it is possible to identify different ideas on what a ‘commercialised’ science might
entail. The CAI developer’s interest in maximising the customer’s own value for money,
runs contrary to the need for the business leaders to maximise readily measurable profit.
The CAI was also found to have created controversy within the forensic science
community itself. Discussions highlighted a certain reluctance on the part of many
forensic scientists to open up their methods to the kind of epistemological scrutiny which
Bayesianism demands. Some practitioners appeared to be reluctant to do so:
‘Another area for objection might be because we were challenging conventional
wisdom, and we were almost challenging the basis of their expertise.’ (Interview 2007)
Other forensic scientists disagreed with the manner in which Bayesianism had been
adopted as a reasoning system. For example, one individual criticised the developers for
conflating an adjunctive technique with the scientific method itself:
7
‘Statistics is a tool. Its not science’ (Interview, 2008)
Discussion
In this paper I have sought to describe the development of epistemic risk management
practices in forensic science. By marking the redefinition of forensic science via the
discourse of epistemic risk, the CAI acts as a potential disciplining device, producing a new
disciplinary identity for forensic science, supposedly closer to the epistemic standards
common to ‘pure’ scientific disciplines, but also compatible with new economic rationalities,
prompted by Government policy, which have led to a re-structuring of the relationship
between forensic scientists and police. My account has sought to demonstrate how the CAI
serves these two concerns. Hence I have aimed to show how the CAI potentially constructs
new notions of ‘forensic’ practices, simultaneously compatible with particular views of
‘scientific’ and ‘commercial’ propriety.
As I have subsequently outlined however, attempts to apply the CAI have exposed areas of
tension in terms of the role and purpose of forensic science, as well as prompting further
debate on the nature of ‘scientific’ behaviours. These areas of contestation reflect diverging
views as to what is regarded by scientific support to criminal investigations, and on what
commercialisation means for how this support is provided.
In being shaped to reflect a new, commercially-influenced orientation to the disciplining of
forensic science, the CAI can be regarded as means of regulating this reconfigured relationship
between forensic science providers and their customers in the criminal justice system. However,
attempts to apply this regulatory model serve also to expose the fragility of its construction. To
use the terminology of Latour and Woolgar (1979), the application of the CAI threatens to open
up a series of ‘black boxes’ found within the substantial network which constitutes the wider
criminal justice system (Latour and Woolgar 1979). In this case, the ‘black boxes’ relate to
particular understandings of ‘science’ within this configuration of actors, which encompasses
police, commercial organisations and the apparatus of the judiciary. By challenging these
understandings, and opening up spaces of controversy, the CAI re-opens debates about the
identity of forensic science, and its position in relation to the other elements which constitute the
criminal justice system as it is organised in the neoliberal era. Taking on board Bruno Latour’s
(1996) arguments concerning the need to look for the hybrid entities submerged beneath
constructions of the ‘scientific’ and the ‘social’, the CAI can be regarded itself as just such a
hybrid, reflecting the interests of criminal justice, but also (and more importantly), those of
an neoliberal-compatible rendition of scientific practice sensitive to the emerging marketplace in which it emerges and whose typical commercial exchange it seeks to shape. It is
too early to say however whether the CAI will serve to support these newly emerging
forms or will instead expose the faultlines that exist below them.
References
Broeders, A. P. A. (2007) Principles of Forensic Identification Science. Handbook of
Criminal Investigation. Newburn, T., Williamson, T, and Wright, A. Portland,
Willan:303-337
Cook, R., I. Evett, et al. (1998a). "A Model For Case Assessment And Interpretation."
Science and Justice 38(3): 151-156.
Cook, R., I. Evett, et al. (1998b). "A Hierarchy of Propositions: Deciding Which
Level to Address in Casework." Science and Justice 38(4): 231-239.
Evett, I. (1984). "A Quantitative Theory for Interpreting Transfer Evidence in
Criminal Cases." Journal of the Royal Statistical Society - Series C 33: 25-32.
Evett, I. (1986). "A Bayesian Approach to the Problem of Interpreting Glass
Evidence." Journal of the Forensic Science Society 26: 3-18.
8
Evett, I. (1987). "Bayesian Inference and Forensic Science: Problems and
Perspectives." The Statistician 36(2/3): 99-105.
Evett, I. and Joyce, H. (2005). "Career Story: Consultant Forensic Statistician."
Significance: 34-37.
Finkelstein, M. O. and W. B. Fairley (1970). "A Bayesian Approach to Identification
Evidence." Harvard Law Review 83(3): 489-517.
Garland, D. (2001) The Culture of Control:Crime and Social Order in Contemporary
Society. New York, Oxford University Press US.
Jackson, G. (2000) “The Scientist and the Scales of Justice.” Science and Justice
40(2):81-85.
Kaplan, J. (1968). "Decision Theory and the Fact-Finding Process." Stanford Law
Review 20(6): 1065-1092.
Kennedy, D. (2003) “Forensic Science: Oxymoron?” Science 302(5651): 1625.
Kingston, C. R. (1965a). "Applications of Probability Theory in Criminalistics 1."
The Journal of Criminal Law, Criminology and Police Science 60(309): 70-80.
Kingston, C. R. (1965b). "Applications of Probability Theory in Criminalistics, 2."
The Journal of Criminal Law, Criminology and Police Science 60(312): 1028-1034.
Kingston, C.R. and Kirk, P. (1964) “The Use Of Statistics in Criminalistics.” The
Journal of Criminal Law, Criminology and Police Science 55:514-521.
Kirk, P. L. (1963). "The Ontogeny of Criminalistics." The Journal of Criminal Law,
Criminology and Police Science 54(2): 235-238.
Latour, B. (1996) Aramis, or The Love of Technology. Cambridge, MA, Harvard
University Press.
Latour, B. and S. Woolgar (1979). Laboratory Life : The Social Construction of
Scientific Facts. Los Angeles, CA, Sage.
McFarland, R. (2003) Review of the Forensic Science Service. London, HMSO.
Robertson, J. (2004) “Forensic Science, Oxymoron or Real Science?” Australian
Journal of Forensic Sciences 36(1): 1-2.
Saks, M. J. and J. J. Koehler (2005). "The Coming Paradigm Shift in Forensic
Science." Science 309(5736): 892-895.
Saks, M. and Faigman, D. (2008) “Failed Forensics: How Forensic Science Lost Its Way and
How It Might Yet Find It.” Annual Review of Law and Social Science 4:149-171.
Williams, R. And Johnson, P. (2008) Genetic Policing: The Use of DNA in Criminal
Investigations. Uffculme, Willan.
9