Download Introduction to Biostatistics - Annals of Emergency Medicine

SPECIAL CONTRIBUTION
biostatistics
Introduction to Biostatistics: Part 3,
Sensitivity, Specificity, Predictive Value,
and Hypothesis Testing
Diagnostic tests guide physicians in assessment of clinical disease states,
just as statistical tests guide scientists in the testing of scientific hypotheses. Sensitivity and specificity are properties of diagnostic tests and are
not predictive,of, disease in individual patients. Positive and negative predictive values are predictive of disease in patients and are dependent on
both the diagnostic test used and the prevalence of disease in the population studied. These concepts are best illustrated by study of a two by two
table of possible outcomes of testing, which shows that diagnostic tests
m a y lead to correct or erroneous clinical conclusions. In a similar manner,
hypothesis testing m a y or m a y not yield correct conclusions. A two by two
table of possible outcomes shows that two types of errors in hypothesis
testing are possible. One can falsely conclude that a significant difference
exists between groups (type I error). The probability of a type I error is ~.
One can falsely conclude that no difference exists between groups (type II
error). The probability of a type II error is ~. The consequence and probability of these errors depend on the nature of the research study. Statistical
power indicates the ability of a research study to detect a significant difference between populations, when a significant difference truly exists.
Power equals 1 - ~. Because hypothesis testing yields "yes" or "no" answers, confidence intervals can be calculated to complement the results of
hypothesis testing. Finally, just as some abnormal laboratory values can
be ignored clinically, some statistical differences m a y not be relevant clinically. [Gaddis GM, Gaddis ML: Introduction to biostatistics: Part 3, sensitivity, specificity, predictive value, and hypothesis testing. Ann Emerg
Med May 1990;19:591-597.]
Gary M Gaddis, MD, PhD*
Monica L Gaddis, PhDt
Kansas City, Missouri
From the Departments of Emergency
Health Services* and Surgery,t University
of Missouri -- Kansas City School of
Medicine, Truman Medical Center, Kansas
City.
Received for publication September 1,
1989. Accepted for publication
January 30, 1990.
Address for reprints: Gary M Gaddis, MD,
PhD, Department of Emergency Health
Services, University of Missouri -- Kansas
City School of Medicine, Truman Medical
Center, 2301 Holmes, Kansas City,
Missouri 64108.
INTRODUCTION
Diagnostic tests guide the physician in assessment of clinical disease
entities. In a similar manner, statistical inference theory guides the scientist in the testing of scientific hypotheses. Before discussing inferential
techniques (parts 4 and 5 of this series), it is necessary to understand the
basis of hypothesis testing, to gain an appreciation of the type of questions
inferential statistics help answer. Clinical diagnostic testing and hypothesis testing have many parallels, but most clinicians are more familiar
with diagnostic than hypothesis testing. Therefore, this article will focus
on the components of diagnostic testing theory, including sensitivity, specificity, and predictive value. This will be followed by analogies to facilitate
understanding of hypothesis testing.
EVALUATION OF DIAGNOSTIC TESTS
Sensitivity and Specificity
Physicians make medical diagnoses with the aid of the patient history,
physical examination, and diagnostic testing. Numerous new diagnostic
tests are presented each year in the medical literature, and each must be
evaluated before it is introduced into the clinical setting. Most new diagnostic tests are evaluated in relation to another older, previously accepted,
often more invasive, and historically reliable test (the "gold standard"
test). Common examples of gold standards include the use of ECG changes
plus cardiac enzyme levels to diagnose acute myocardial infarction, or pulmonary angiography to diagnose pulmonary embolism. For the purposes of
19:5 May 1990
Annals of Emergency Medicine
591/145
BIOSTATISTICS
Gaddis &Gaddis
our discussion, it will be assumed
that results obtained by the gold
standard test are always correct.
Hypothetically, imagine that a new
magnetic resonance imaging (MRI)
venogram has been proposed as a
noninvasive means of evaluating patients suspected by clinical criteria of
having a deep venous thrombosis.
The MRI venogram, the proposed
new diagnostic test, will be evaluated a g a i n s t the t r a d i t i o n a l and
widely used gold standard, the intravenous contrast venogram. Table 1
shows that there are four possible
outcomes of diagnostic testing. Patients can be diagnosed as having
deep venous thrombosis or not having deep venous thrombosis by both
the gold standard test and by the new
MRI diagnostic test, if patients undergo both tests.
In Table 1, 250 patients clinically
s u s p e c t e d of h a v i n g deep v e n o u s
thrombosis undergo both tests. Of
the 250 patients clinically suspected
to have deep venous thrombosis, 150
actually do have deep venous thrombosis, with 130 shown to have deep
venous thrombosis by both the gold
standard test and by the new MRI
test. This group of 130 is termed the
true positive (TP) group by the new
d i a g n o s t i c test because t h e y are
shown to have disease by the new
test and are also proven to have disease by the gold standard test. However, 20 of the 150 patients who are
proven by the gold standard test to
have deep venous thrombosis had a
negative MRI diagnostic test. These
20 are termed the false negative (FN)
group because they were classified
incorrectly as disease free by the new
MRI test.
Similarly, 100 of the patients were
judged disease free by the contrast
venogram, but of these, only 87 had a
negative MRI test. This group of 87
constitutes the true negative (TN)
group. The remaining 13 were incorrectly classified by the new MRI test
as having a deep venous thrombosis,
when in fact they did not have the
disease. This constitutes the false
positive (FP) group.
The two by two outcome table in
Table 1 can now be used to help us
evaluate how well the new MRI test
does in detecting deep venous thrombosis. We want to know the answers
to two questions: Is the test sensitive
enough to detect the presence of a
deep venous thrombosis in a diseased
146/592
TABLE 1. Gold standard versus diagnostic test
Gold Standard Test
(Contrast Venogram)
Disease
Evident
No Disease
Evident
Diagnostic
Test
(MRI
Venogram)
Disease
Evident
No Disease
Evident
Total
TP (130)
FP (13)
143
FN (20)
TN (87)
107
150
100
250
TABLE 2. Gold standard versus diagnostic test
Gold Standard Test
(Contrast Venogram)
Diagnostic
Test
(MRI
Venogram)
Disease
Evident
No Disease
Evident
Disease
Evident
No Disease
Evident
Total
TP (35)
FP (21)
56
FN (5)
TN (139)
144
160
200
40
,
TABLE 3. Possible outcomes of hypothesis testing
Reality
Decision From
Statistical Test
Reject Ho,
Accept H~
Ho False,
H1 True
Ho True,
H1 False
Correct,
No Error
Incorrect,
Type l Error
(A)
Accept Ho,
Reject H 1
Incorrect,
Type II Error
(C)
patient? Is the test specific enough to
indicate the absence of deep venous
thrombosis disease only in patients
who in fact are not afflicted by it?
Sensitivity, which can be thought
of as "positivity (of the test) in disease," is derived by working down
the first column of Table 1:
Sensitivity (%) =
100 x TP/(TP + FN)
In this example, sensitivity equals
100 x 130/(130 + 20), or 86.7%.
Annals of Emergency Medicine
(B)
Correct,
No Error
(U)
Specificity, which can be thought
of as " n e g a t i v i t y (of the test) in
health," is also derived by working
vertically, in the second column of
Table 1:
Specificity (%) =
100 x TN/(TN + FP)
Here, specificity equals 100 x 87/(87
+ 13), or 87.0%.
The ideal diagnostic test would be
100% sensitive and 100% specific,
and thus would have no FP or FN
19:5 May 1990
1,0
••
m
F I G U R E I. Operating characteristic
curve. ~ is dependent on ~, n, and i .
In this example, ct is fixed at .05. All
else held constant, increasing i or
increasing n decreases ~.
c~ 0.05
n2>nl
=
/
~
fl
0,5
-
•7
0.0
low
~
high
1
TABLE 4. Prior probability and chance of error
Prior Probability
Chance of Error
Type l
Type II
outcomes. Because v i r t u a l l y all diagn o s t i c tests have s o m e FP and F N
o u t c o m e s , t h e y do n o t h a v e 100%
sensitivity and specificity.
Unfortunately, m a n y clinicians believe that s e n s i t i v i t y and specificity
can be used to predict w h e t h e r an individual patient is diseased or disease
free. This is an error. Sensitivity and
specificity are m e r e l y properties of a
test. Sensitivity and specificity
should not be used to m a k e predictive s t a t e m e n t s a b o u t an i n d i v i d u a l
patient.
Predictive Value
P r e d i c t i v e v a l u e s can be u s e d to
help predict the l i k e l i h o o d of disease
in an i n d i v i d u a l . A p o s i t i v e predictive value (PPV) is useful to indicate
the proportion of individuals who actually have the disease w h e n the dia g n o s t i c test i n d i c a t e s the presence
19:5 May 1990
Low
High
High
Low
Low
High
of that disease. A negative predictive
v a l u e (NPV) is useful to d e t e r m i n e
the proportion of individuals who are
t r u l y free of t h e d i s e a s e t e s t e d for
when the diagnostic test indicates
the absence of that disease.
P r e d i c t i v e v a l u e s are d e r i v e d by
w o r k i n g h o r i z o n t a l l y on the two by
two o u t c o m e table in Table 1:
PPV (%) = 100 x TP/(TP + FP)
NPV (%) = 100 x T N / ( T N + FN)
From the e x a m p l e in Table 1, PPV
= 100 x 130/(130 + 13), or 90.9%,
and NPV = 100 x 87/(87 + 20), or
81.3%.
PPV and NPV are affected by the
prevalence of disease in the population. Prevalence is defined as the proportion of the p o p u l a t i o n afflicted by
the disease in question. In the example in Table 1, the prevalence of deep
venous t h r o m b o s i s w h e n it was clinAnnals of Emergency Medicine
i c a l l y s u s p e c t e d w a s 60% b e c a u s e
the total n u m b e r of patients studied
was 250, and the n u m b e r of patients
w h o a c t u a l l y h a d a c o n t r a s t venogram (the gold standard test) indicative of deep venous t h r o m b o s i s was
150.
Next, the effects of decreased prevalence of deep venous thrombosis on
the predictive value of the MRI venogram test will be examined. Imagine
a sample of 200 patients, only 20% of
w h o m h a v e a deep v e n o u s t h r o m bosis (prevalence, 20% ). This group is
d e p i c t e d (Table 2). Because 20% of
the patients have a deep venous
thrombosis, the s u m of TP + FN in
c o l u m n 1 m u s t be 0.2 x 200, or 40. Of
these, a b o u t 35 w i l l c o n s t i t u t e t h e
TP group b e c a u s e t h e s e n s i t i v i t y of
the test has already been shown to be
86.7% (0.867 x 40 = 34.7). The rem a i n i n g five can be expected to be in
the F N group because sensitivity is a
p r o p e r t y of the test i n d e p e n d e n t of
disease prevalence. Because the prevalence of deep venous t h r o m b o s i s is
only 20%, the r e m a i n i n g 0.8 x 200,
or 160, will not have a deep venous
thrombosis, so the s u m of T N + FP
results in c o l u m n 2 will be 160. Of
this set of 160, 87%, or a b o u t 139,
will be in the T N group, and the rem a i n i n g 21 will be in the FP group
because specificity is also a property
of t h e test, i n d e p e n d e n t of d i s e a s e
prevalence.
T h e c h a n g e of p r e v a l e n c e m a r k e d l y i n f l u e n c e s t h e PPV and N P V
v a l u e s o b t a i n e d (Table 2). W i t h a
20% prevalence, the PPV falls to 100
x 35/(35 + 21), or 62.5%, w h i l e the
NPV increases to 100 x 139/(139 + 5),
or 96.5%. N o t e that as disease prevalence falls, the PPV of any test will
fall and the NPV of any test will increase.
F r o m this, it is easy to see w h y
m a n y n e w diagnostic tests that seem
from initial reports to be useful m a y
not represent a diagnostic improvem e n t w h e n in c o m m o n use. M a n y
diagnostic tests are validated in sett i n g s on p o p u l a t i o n s w i t h a h i g h
p r e v a l e n c e of the disease for w h i c h
testing is done. However, w h e n the
n e w test is used in different clinical
settings w i t h a l o w e r p r e v a l e n c e of
593/147
BIOSTATISTICS
Gaddis & Gaddis
FIGURE 2. Clinical testing.
that disease, the test does not perform up to reported expectations. A
clinical example of the interrelationship between prevalence of disease
and predictive value is the use of amylase levels to screen for pancreatitis.
An elevated amylase level is more
likely indicative of panereatitis in
persons previously afflicted with
pancreatitis than it is predictive of
pancreatitis among all patients with
a b d o m i n a l pain or o t h e r possible
causes of an elevated serum amylase
level.
In summary, sensitivity and specificity are properties that indicate the
degree of reliability of a diagnostic
test. Sensitivity and specificity do
not indicate predictive value. Predictive values can be applied to an individual patient's test result and are affected by the prevalence of the disease in the population to which the
test is applied. The PPV will fall and
the NPV will rise as the prevalence
of disease decreases.
HYPOTHESIS TESTING
Formulation of the Hypothesis
Statistical inference involves the
testing of hypotheses. A hypothesis
is a numerical s t a t e m e n t about an
u n k n o w n p a r a m e t e r 3 Just as a two
by two table can be constructed for
the four possible outcomes of a clinical diagnostic test, a two by two table can be constructed for the four
possible outcomes of hypothesis testing.
Before constructing this table, it is
necessary to understand what a hypothesis states. The first step in hypothesis testing is a s t a t e m e n t of a
h y p o t h e s i s in positive terms. This
defines the " r e s e a r c h " or "alternative" hypothesis, H1.2 For example,
one could h y p o t h e s i z e t h a t experienced e m e r g e n c y p h y s i c i a n s (those
w i t h m o r e t h a n five years of fulltime postgraduate emergency departm e n t experience) can examine, diagnose, and treat m o r e p a t i e n t s per
hour than inexperienced emergency
p h y s i c i a n s (less t h a n five years of
full-time ED experience).
The next step is to state the "null"
or " s t a t i s t i c a l " h y p o t h e s i s , Ho,
w h i c h follows logically from H1.1,2
The hypothesis tested statistically is
H o. In this example, H o would state
"Experienced emergency physicians
and inexperienced emergency physi148/594
Sensitivity
The ability of a test to reliably detect the presence of disease
(positivity in disease).
Sensitivity (%) = 100 x TP/(TP + FN)
Specificity
The ability of a test to reliably detect the absence of disease
(negativity in health).
Specificity (%) = 100 x TN/(TN + FP)
Prevalence
The proportion of the population with disease.
Prevalence (%) = 100 x (TP + FN)/(n)
Positive
Predictive
Value
The proportion of individuals with disease when the presence
of disease is indicated by the diagnostic test.
PPV = 100 x TP/(TP + FP)
Negative
The proportion of individuals free of disease when the abPredictive
sence of disease is indicated by the diagnostic test.
Value
NPV = 100 x TN/(TN + FN)
TN, true negative; FN, false negative; TP, true positive; FP, false positive.
2
cians do n o t differ significantly in
the n u m b e r of patients they can examine, diagnose, and treat per hour."
We "reject" or "fail to reject" ("accept") H o based on our inferential
statistical testing. 1-3 Ho hypothesizes
a difference of zero between population samples tested, while H 1 hypothesizes a nonzero difference bet w e e n p o p u l a t i o n s a m p l e s tested.
There exist an infinite n u m b e r of
possible nonzero differences between
populations. Therefore, the reason
that H o rather than H 1 is tested is
that mathematically, H o theorizes a
single m a g n i t u d e of difference between populations studied, and it is
possible to statistically assess this
single hypothesis. In contrast, H 1 is
a c t u a l l y an infinite n u m b e r of hypotheses because there exist an infinite n u m b e r of possible magnitudes
of difference between populations. 4 It
would be impossible to calculate the
required statistics for each of the infinite n u m b e r of possible magnitudes
of d i f f e r e n c e b e t w e e n p o p u l a t i o n
samples H 1 hypothesizes.
If H 0 is "accepted" as tenable, then
H 1 m u s t be " r e j e c t e d , " and v i c e
versa, because the two h y p o t h e s e s
are mutually exclusive. When H o is
tested, the probability that numerical
differences between population samples are not due strictly to chance is
assessed. 2 H 0 does r e c o g n i z e t h a t
nonzero differences between groups
are possible, even if two samples of
the same population are tested, simply due to r a n d o m s c a t t e r of t h e
data. 2 If H o is "accepted" as tenable,
this signifies the likelihood that no
significant difference exists between
Annals of Emergency Medicine
the populations studied and that any
numerical differences between
groups are due to chance alone. If H o
is rejected, this signifies that a significant difference does exist between
the populations studied and that the
n u m e r i c a l differences b e t w e e n the
groups are not due to chance alone.
Errors in Hypothesis Testing
Hypothesis testing m a y lead to erroneous inferential statistical conclusions, just as diagnostic testing m a y
lead to erroneous diagnostic conclusions. Just as a two by two table of
possible outcomes of diagnostic tests
can be constructed, so can a two by
two table of possible outcomes of inf e r e n t i a l s t a t i s t i c a l t e s t s be c o n structed (Table 3). Two types of incorrect conclusions are possible. Box
B of Table 3 indicates cases in which
the statistical test falsely indicates
that a significant difference exists between groups, when in fact no true
difference exists. It is analogous to a
false-positive diagnostic test result.
In other words, box B shows cases
where H o is rejected, w h e n it is in
fact true. This rejection of H o when
H o is true is arbitrarily called a type I
e r r o r . 1-3
Box C of Table 3 indicates cases in
which the statistical test falsely indicates the lack of a significant difference between groups, w h e n in fact a
true difference exists (H 1 is true).
This is analogous to a false-negative
diagnostic test result. In other words,
box C shows cases in which H e is accepted when it is in fact false. The
acceptance of H o when H o is false is
arbitrarily called a type II error. 1-3
19:5 May 1990
FIGURE 3.
Research
(Alternative)
Hypothesis
(H1)
Null
(Statistical)
Hypothesis
An hypothesis that states a difference exists between two
(or more) populations studied. H 1 is a positive statement
that a difference exists between groups.
An hypothesis of no difference between two or more populations studied. H o is a negative statement, that no difference exists between groups.
(Ho)
Type I
Error
To reject the null hypothesis (Ho), when in fact H o is true.
To falsely conclude that a significant difference exists between populations.
Type II
Error
To accept the null hypothesis (Ho), when in fact H o is false.
To falsely conclude that no significant difference exists
between populations.
The probability of making a type I error.
Statistical calculations from the experimental data indicate
that the probability of making a type I error is less than
5%.
The probability of making a type II error.
The ability of an experiment to find a significant difference
exists between populations, when in fact a significant difference truly exists. Power = 1 - 13
The degree of difference between populations tested.
Alpha (o0 " •
P < .05
Beta (13)
Power
Delta (A)
Operating
Characteristic
Curve
Prior
Probability
A function that relates the dependent variable !8 that results from independent values of ~, A, and n.
The likelihood that an hypothesized difference between
populations is in fact correct.
Box A and box D of Table 3 denote
c o r r e c t c o n c l u s i o n s , a n a l o g o u s to
true-positive and true-negative diagnostic test results. Thus, Table 3
s h o w s t h a t there exist t w o correct
and t w o incorrect conclusions possible w h e n e v e r H o is tested.
Next, the probability of m a k i n g inc o r r e c t c o n c l u s i o n s m u s t b e assessed. T h e probability of m a k i n g a
type I error is defined as alpha (~).1,2,4
is derived from the raw data, statistical c a l c u l a t i o n s , and s t a t i s t i c a l tables a p p r o p r i a t e for t h e i n f e r e n t i a l
s t a t i s t i c a l t e s t used. By c o n v e n t i o n ,
s t a t i s t i c a l s i g n i f i c a n c e is g e n e r a l l y
accepted if the probability a of m a k ing a t y p e I error is less t h a n 0.05,
w h i c h is c o m m o n l y denoted on figures and tables as P < .05.3, 4
T h o u g h conventional, selection of
an a l p h a level of .05 as t h e crucial
level of significance is arbitrary. Acc e p t i n g s i g n i f i c a n c e a t et = .05
m e a n s that it is recognized that one
t i m e o u t of 20, a type I error will be
committed, a consequence that the
i n v e s t i g a t o r is w i l l i n g to accept. If
the consequences of malting a type I
error are judged to b e sufficiently se19:5 May 1990
vere, it m a y be appropriate to select
m o r e s t r i n g e n t levels of % such as
.01, as the cutoff for statistical significance. W h e n a caption or text indicates t h a t for s o m e s t a t i s t i c a l c o m parison, P = .XY, the probability of a
type I error, based on the calculations
performed for that inferential statistical test, is 0.XY, and the reader is
left to judge w h e t h e r this level of ot is
i n d i c a t i v e of a t r u e d i f f e r e n c e between populations tested. Another
advantage of the reporting of P values
is t h a t the a r b i t r a r y d e s i g n a t i o n of
significance at .05, and the i m p r o p e r
and arbitrary designation of a trend if
.10 > P > .05, can be avoided.
T h e probability of m a k i n g a type II
error is defined as beta (13).1,~,4 ~ is
m o r e difficult to derive t h a n a, and
u n l i k e ~, a c t u a l l y is n o t one single
probability value. [3 is often ignored
by researchers, s However, it is imp o r t a n t . If s o m e t r e a t m e n t yields a
10% increase in survival or a 10% decrease in some complication, it
w o u l d l i k e l y be readily incorporated
into m e d i c a l practice. Unfortunately,
n u m e r o u s c l i n i c a l t r i a l s h a v e suffered from errors of e x p e r i m e n t a l deAnnals of Emergency Medicine
Hypothesis testing.
sign that cause 13 to be u n a c c e p t a b l y
high, such t h a t type II errors are easily made, and t r e a t m e n t s that are significantly better t h a n older m e t h o d s
are rejected because of statistical artifact resulting from poor e x p e r i m e n t a l
design .s By convention, [3 should be
less t h a n .20, and i d e a l l y less t h a n
.10, to m i n i m i z e the chance of m a k ing a type II error. 6
and f3 are i n t e r r e l a t e d . A l l else
h e l d c o n s t a n t (such as t h e p o p u l a tions studied, the n u m b e r of subjects,
and t h e m e t h o d of testing), as a is arbitrarily decreased, 13 is increased. As
is increased, 13 is decreased.i, 2
S t a t i s t i c a l p o w e r is d e f i n e d as
(1-~).1,2, 4 B e c a u s e [3 i n d i c a t e s t h e
probability of m a k i n g a type II error,
power indicates mathematically the
probability of not m a k i n g a type II error. Power is analogous to sensitivity
in hypothesis testing. Sensitivity indicates the probability that the diagnostic test can detect disease w h e n it
is present. Power indicates the probability that the statistical test can detect s i g n i f i c a n t differences b e t w e e n
populations, w h e n in fact such differences truly exist.
Power depends on several variables: 1,2,4,7
c~: As a increases, 13 decreases, and
power increases.
n ( s a m p l e size): A s n i n c r e a s e s ,
power increases.
T h e m a g n i t u d e of t h e d i f f e r e n c e
actually present b e t w e e n the populations tested, delta (A): Just as it is
easier to find a pitchfork t h a n a needle in a h a y s t a c k , so it is easier to
find a large difference t h a n it is to
find a small difference b e t w e e n populations tested.
One-tailed versus two-tailed tests:
O n e - t a i l e d tests are m o r e p o w e r f u l
than two-tailed tests, because a statistical test result m u s t n o t vary as
m u c h from the m e a n to achieve significance at any level of c~ chosen. (If
c~ is .05, for a two-tailed test, a result
m u s t fall in either the top or b o t t o m
21/2% of r e s u l t s to a c h i e v e significance, b u t for a o n e - t a i l e d test, t h e
result m u s t m e r e l y fall in either the
top or b o t t o m 5% of a distribution.)
In the original h y p o t h e s i s e x a m p l e
about h o w quickly e m e r g e n c y physicians can treat patients, the appropriate test w o u l d be one-tailed, because
H 1 specifies the direction of the difference between groups hypothe595/149
BIOSTATISTICS
Gaddis & Gaddis
sized.
Parametric versus nonparametric
statistical testing: Parametric tests
are g e n e r a l l y m o r e p o w e r f u l . (This
will be further discussed in Part 4 of
this series.)
Use of proper e x p e r i m e n t a l design
and s t a t i s t i c s : Errors in t h e s e areas
decrease power.
Because so m a n y variables can affect I3, ~ is not one single value. This
follows from t h e fact t h a t ~ is t h e
probability of erroneously concluding
t h a t H o is false, and H o specifies a
single m a g n i t u d e of d i f f e r e n c e bet w e e n populations. However, as has
been explained, ~ is t h e p r o b a b i l i t y
of erroneously concluding that H I is
false, and H 1 h y p o t h e s i z e s an infinite
n u m b e r of p o s s i b l e m a g n i t u d e s of
difference between populations
tested. ~ is expressed as a function of
A, n, and a by a function called the
operating characteristic curve of the
test s (Figure 1).
T h e m o s t c o m m o n use of ~ is in
t h e c a l c u l a t i o n of t h e a p p r o x i m a t e
n u m b e r of s u b j e c t s t h a t m u s t be
s t u d i e d to keep R and [3 a c c e p t a b l y
small. This calculation uses estim a t e s of p o p u l a t i o n standard deviations and e s t i m a t e s o f / k , acceptable
values of a and ~, and n u m b e r s from
statistical tables, to derive a value of
n of sufficient size. T h e d e t e r m i n a t i o n of a d e q u a t e s a m p l e size for an
e x p e r i m e n t is readily referenced. 8 lo
P Values Versus Confidence
Intervals
H y p o t h e s i s testing yields yes or no
answers about statistical significance, answers t h a t can be fraught
w i t h errors, and a n s w e r s t h a t m a y
represent oversimplifications. P
values i m p l y l i t t l e about the magnit u d e of d i f f e r e n c e p r e s e n t b e t w e e n
p o p u l a t i o n s . T h e r e f o r e , s o m e feel
that the use of confidence intervals
(CIs) is c o m p l e m e n t a r y or even prefe r a b l e to t h e u s e of P v a l u e s i n
reporting clinical data.11 (Confidence
intervals were discussed in part 2 of
this series. 12) It is correct to r e p o r t
b o t h CI and P v a l u e s for s c i e n t i f i c
data, and the two are often complementary. 1,11
Clinical Versus Statistical
Significance
Statistically significant numerical
differences between study groups
m a y n o t be c l i n i c a l l y significant or
relevant. A n analogy to clinical test150/596
ing is again useful. It is c o m m o n experience to ignore or place little emphasis on a single diagnostic test res u l t t h a t lies o u t s i d e t h e e x p e c t e d
range for that test w h e n large n u m bers of tests are done. A n example is
the i n t e r p r e t a t i o n of an isolated elevated a m y l a s e level in a p a t i e n t having otherwise n o r m a l routine laborat o r y d a t a after a n o r m a l s c r e e n i n g
p h y s i c a l e x a m i n a t i o n at his f a m i l y
physician's office. M a n y experienced
clinicians can i n t u i t i v e l y sense w h e n
to place l i t t l e e m p h a s i s on i s o l a t e d
laboratory test results outside the
n o r m a l range w h e n an abnormal res u l t is n o t e x p e c t e d . A l t e r n a t i v e l y
stated, w h e n there is very little prior
probability of disease, an isolated abn o r m a l laboratory value is generally
n o t cause for great concern, and the
clinician avoids a clinical error analogous to a type I error by a v o i d i n g
concluding that disease is present in
a disease-free patient.
S i m i l a r l y , if e n o u g h s t a t i s t i c a l
c o m p a r i s o n s are m a d e , e v e n t u a l l y
type I and type II statistical errors are
i n e v i t a b l e . T h e p r o b l e m c o m e s in
discerning w h i c h statistically significant differences are m e a n i n g f u l and
w h i c h are meaningless. Just as prevalence affects the predictive value of a
positive diagnostic test, so the prior
probability of a difference affects the
predictive value of a statistical test.
Prior probability is an expression of
h o w likely an hypothesis will be true
w h e n assessed b e f o r e doing s t a t i s t i cal c a l c u l a t i o n s . Prior p r o b a b i l i t y is
derived from previously available
k n o w l e d g e t h a t led to the f o r m u l a tion of the hypothesis being tested.
W h e n a hypothesis has a low prior
probability
of b e i n g t r u e , y e t
achieves statistical significance, such
as a l i n k b e t w e e n coffee c o n s u m p tion and pancreatic cancer, 13 a significant result m u s t be interpreted cautiously. Furthermore, if a type I error
is being made, repetitive study will
p r o b a b l y n o t r e p l i c a t e a significant
difference, as subsequently occurred
in t h e case of t h e a l l e g e d l i n k bet w e e n coffee c o n s u m p t i o n and pancreatic c a n c e r J 4 However, in cases of
high prior p r o b a b i l i t y , a s i g n i f i c a n t
s t a t i s t i c a l difference is u s u a l l y correct, just as in cases of high disease
prevalence, a positive clinical test result is m o r e l i k e l y to be correct.
Table 4 s u m m a r i z e s the interrelationship between prior probability
and the chance of m a k i n g a type I or
Annals of Emergency Medicine
type II error. This relationship is further explained by Bayes theorem,
w h i c h t h e r e a d e r is i n v i t e d to explore.
SUMMARY
A n u n d e r s t a n d i n g of t h e i n t e r p r e t a t i o n of d i a g n o s t i c t e s t s facilitates an u n d e r s t a n d i n g of hypothesis
testing. A diagnostic test result m a y
be a t r u e - p o s i t i v e , t r u e - n e g a t i v e ,
f a l s e - p o s i t i v e , or f a l s e - n e g a t i v e result. For diagnostic tests, s e n s i t i v i t y
and s p e c i f i c i t y are p r o p e r t i e s of t h e
d i a g n o s t i c test and do n o t i n d i c a t e
p r e d i c t i v e value. P r e v a l e n c e of disease is a d e t e r m i n a n t of the predictive value of b o t h positive and negative test results.
Similarly, hypothesis testing can
yield erroneous results. A false-positive result, w h i c h a c c e p t s t h e presence of a significant difference bet w e e n p o p u l a t i o n s w h e n in fact no
significant difference exists (type I error}, occurs w i t h a probability of a. A
false-negative result, rejecting the
p r e s e n c e of a significant difference
b e t w e e n p o p u l a t i o n s , w h e n in fact
t h e y actually do differ (type II error),
occurs w i t h a probability of ~.
P o w e r is i-p, and is analogous to
the sensitivity of a diagnostic test in
t h a t b o t h s e n s i t i v i t y and p o w e r address w h e t h e r a test can detect w h a t
it is designed to detect. As s e n s i t i v i t y
and specificity are n o t predictive, so
also power is n o t predictive. As prevalence of disease affects the predictive value of a positive test result, so
the prior p r o b a b i l i t y of a difference
being p r e s e n t affects the p r e d i c t i v e
value of a significant statistical test
result. Figures 2 and 3 summarize
these points.
REFERENCES
1. Hopkins KD, Glass GV: Basic Statistics for
the Behavioral Sciences. Englewood Cliffs, New
Jersey, Prentice-Hall, Inc, 1978.
2. Keppel G: Design and Analysis. A Researcher's Handbook. Englewood Cliffs, New
Jersey, Prentice-Hall, Inc, 1978.
3. Elenbaas RM, Elenbaas JK, Cuddy PG: Evaluating the medical literature Part II: Statistical
analysis. Ann Emerg Med 1983;12:610-620.
4. Sokal RR, Rohlf FJ. Biometry (ed 2). New
York, WH Freeman and Co, 1981.
5. Freiman JA, Chalmers TC, Smith H, et al:
The importance of beta, the type II error, and
sample size in the design and interpretation of
the randomized clinical trial. N Engl J Med
1978;299:690-694.
6. Reed JF, Slaichert W: Statistical proof in inconclusive "negative" trials. Arch Intern Med
1981;141:1307-1310.
19:5 May 1990
7. Cohen J: Differences between proportions,
in: Statistics in Medicine. Boston, Little, Brown,
& Co, 1974.
8. Arkin CG WachtelMS: Howm.any patients
are necessary to assess test pefformance?JAMA
1990;263:275-278.
9. Fleiss JL: Statistical Methods for Rates and
Proportions (ed 2). New York, John Wiley &
19:5 May 1990
Sons, 1981.
10. Young MJ, Bresnitz EA, Strom BL: Sample
size nomograms for interpreting negative clinical studies. Ann Intern Med 1983;99:248-251.
11. Gardner MJ, Altman DG: Confidence intervals rather than P values: Estimation rather
than hypothesis testing. Br Meal J 1986;292:
746-750.
Annals of Emergency Medicine
12. Gaddis GM, Gaddis ML: Introduction to
biostatistics: Part 2, descriptive statistics. Ann
Emerg Med 1990;19:309-315.
13. MacMahon B, Yen S, Trichopoulos D, et al:
Coffee and cancer of the pancreas. N Engl J Med
1981;304:630-633.
14. Gorham ED, Garland CF, Garland FL, et al:
Coffee and pancreatic cancer in a rural California county. West J Med 1988;148:48-51.
597/151