Download Significance levels for multiple tests

STATISTICS&
PROBABILITY
LETTERS
ELSEVIER
Statistics & Probability Letters 35 (1997) 43-48
Significance levels for multiple tests
Sergiu Hart a'*, Benjamin
Weiss b
Department of Mathematics; Department of Economics; and CenterJbr Rationality and Interactive Decision Theom',
The Hebrew University of Jerusalem, Feldman building, Givat-Ram, 91904Jerusalem, Israel
bDepartment of Mathematics, The Hebrew University of Jerusalem, Givat-Ram 91904 Jerusalem, Israel
Received 1 May 1996
Abstract
Let X 1. . . . . X, be n random variables, with cumulative distribution functions F1, ..., F,. Define ¢i := Fi(Xi) for all i,
and let ~(1) ~< ... ~< ~(,) be the order statistics of the (~)~. Let cq ~< ... ~< ~. be n numbers in the interval [0, 1]. We show
that the probability of the event R := {(") ~< ct~for all 1 ~< i ~< n} is at most min~n~Ji }. Moreover, this bound is exact: for
any given n marginal distributions (F~)~, there exists a joint distribution with these marginals such that the probability of
R is exactly min~nct~/i}. This result is used in analyzing the significance level of multiple hypotheses testing. In particular,
it implies that the ROger tests dominate all tests with rejection regions of type R as above. © 1997 Elsevier Science B.V.
Keywords: Rtiger tests; Order statistics
Let X1, X 2 . . . . . X n be n r a n d o m variables, with cumulative distribution functions F1, F2, . . . , F,. Define
~i:= Fi(X~) for all i, and let ~(1) ~< ~(2) ~< ... ~< ~(,) be the order statistics of the (~)~. Let ~1 ~< c~2 ~< ' " <~ 7,
be n numbers in the interval [0, 1], and consider the event
R : = {~li~ ~< 7i for all 1 ~< i ~< n}.
(1)
F o r n = 2, this event is depicted in Fig. 1 (as the corresponding dark area). The problem is to evaluate the
probability of R, for arbitrary joint distribution of the Xi's.
Events of this kind arise in "multiple hypotheses testing" (see, e.g., Bonferroni, 1936; Rtiger, 1978; H o m m e l ,
1983; H o c h b e r g and H o m m e l , 1996). The setup is as follows: There are n statistical tests: for each i = 1 . . . . . n,
let Hol be the null hypothesis, Xi the test statistic, and {Xi ~ ci} the rejection region. The tests are applied to
the same data, but the dependence between them is not known. Let H o : = 0~=1 ...... H0~ be the null
hypothesis that is the conjunction of the n null hypotheses. The problem is to construct an "overall test" for
Ho that combines the n tests, i.e., it is based on the rejection regions of the n individual tests. Since Ho is not
a simple hypothesis, the significance level a of a test with rejection region R is defined as the worst-case
* Corresponding author. E-mail: [email protected].
i March 1996. Most of these results are from 1992.
0167-7152/97/$17.00 ~) 1997 Elsevier Science B.V. All rights reserved
PII S0 1 67-7 1 52(96)002 1 4-3
S. Hart, B. Weiss / Statistics & Probability Letters 35 (1997) 4 3 - 4 8
44
~2
Oe2
Oil
~I
0~2
Fig. 1. The event R.
probability of type-I error; i.e., ~(R) := sup0~eo Po(R), where 0 o is the set of all probability distributions that
are consistent with Ho. In the setup above, where Ho is the intersection of the Hofs, the set 0 0 contains all the
joint distributions with marginals F I , . . . , F , .
A rejection region R of type (1) means that Ho is rejected if the n test statistics are m o r e significant than
~1, ~2 . . . . . ~,, regardless of which test has passed which significant level.
Assume that all the ~ ' s are possible values of each Fi (this clearly holds for continuous distributions, and, if
there are atoms, assume they are "split" accordingly; recall that these r a n d o m variables represent statistical
tests); thus P(~i ~< ctj) = ctj for all i, j.
Theorem: P({~I) <<.~i for all 1 <<.i <<.n} ) <~ min i ~<i ~<, n~i/ i. Moreover, this bound is exact:for any n marginal
distributions (F~)~= 1....... there exists a joint distribution P with these marginals such that the inequality becomes
an equality.
In terms of the multiple test procedure discussed above, the result is that the significance level of R as
defined in (1) is a(R) = mini{n~i/i}. Equivalently, the p-value p of a test of type (1) m a y be expressed as
a function of the p-values Pl,P2 . . . . . p. of the n individual tests, namely p = mini{np~°/i}, where
ptl) ~< p(2) ~< ... ~< p(n) are the ordered pi's.
We come now to an interesting corollary of the theorem. Consider two tests with rejection regions R1 and
R 2, respectively. We say that R 1 is dominated by R 2 if ~(R1) = ~(R 2) and R 1 ~ R2; thus the two tests have the
same significance level, but the probability of type-II error is always at least as high for R 1 as for R 2. F o r each
i = 1 . . . . . n and each ~ ~ (0, 1), define RI,, := {i t/) ~< i~/n}; i.e., at least i a m o n g ~a, --- , 4, are <~ i~/n. N o t e
that R~,, obtains by taking ~j = i~/n f o r j ~< i and ~j = 1 f o r j > 1 in (1). The Ri., are the Rtiger (1978) tests.
Corollary: Let R be the rejection region of a test of type (1), with significance level ~ = ~(R). Then there exists
an integer i, 1 ~ i <<.n, such that R is dominated by Ri,~.
S. Hart, B. Weiss / Statistics & Probability Letters 35 (1997) 43-48
45
Proof. Let i be such that the minimum in the formula for ct(R) is attained at i, i.e., e = e(R) = n~//i; thus
ct~ = ie/n. By the result of the theorem, one can increase ej for all j # i without increasing the significance
level; the largest set is thus obtained by taking 2 ~j = e / = i~/n for j < i and ~j = 1 for j > i - which yields
precisely R/.,. []
Thus, for each • there are exactly 3 n tests of type (1) to consider, namely the Riiger (1978) tests R/,,. Note
that R L , is the Bonferroni (1936) test - reject if at least one of the ~i is ~< ~/n - and that the Simes (1986) test
is the union Ui=a ...... R~,, of all the 4 Ri,,.
We come now to the proof of the theorem. A few notations first: Put I : = {1,2 . . . . . n} and
J : = {1,2 . . . . . n + 1}. It will be convenient to define Co:= 0 and ~,+~:=1, and 6i:= ~ - ~_~ for all i E J .
Partition [0,1] into n + l
disjoint intervals Ol := [0, el], D 2 : = ( 0 ~ 1 , 0 ~ 2 ] . . . . , D j : = ( o ~ j - I , O ~ j ] , . . . ,
D,+ 1:= (~,, 1]. Define random variables Y~, Y2, .-., Y. by Yi =J whenever ~ e Dj, and put Y := (Yi)i~1.
Note that the event R is actually defined in terms of Y.
Proof of Theorem. For each vector y = (Y~)i~i in j i , write p(y) for the probability P ( Y = y). Denote by Q the
set of all values of Y corresponding to the event R; i.e., y belongs to Q if, for each j = 1, 2, ... , n (note: j ~< n),
there are at least j indices i such that y/~< j. Thus, the upper bound of Theorem 1 is obtained from the
following linear programming problem:
maximize ~' p(y)
y~Q
~,
subject to:
P(Y) =
fit for all i e I and j e J,
(LP)
ys.t. yi=j
p(y)>>,O
for ally.
The dual of (LP) is
minimize ~ ~ 6j21.j
i~l j~J
subject to: ~ 2i, y, ~> 1 for all y ~ Q,
(DLP)
iEI
~', 21,r, >~ 0
for all yCQ.
iel
We will show that the optimal value of ( D L P ) - and thus also of (LP) - is indeed min i{n~i/i}. First, note
that if 2 = (2/,~)i~i. j~s is feasible in (DLP), then, for any permutation n of I, 2' := (2~t0.j)i~x ' j~j is also feasible
in (DLP). Moreover, the value of the objective function is the same for 2' as for 2. Averaging over all
permutations n o f / i m p l i e s that we can assume, without loss of generality, that 2 is "symmetric", i.e., 2~,j = #j
2 Recall that 0 ~< ctl ~< ct2 ~< ... ~< cq ~< 1.
3 It is clear that the Ri., for different i do not dominate one another.
4 T h e significance level of the Simes test is m{~t(1 + ~ + ... + x), 1}, as s h o w n by H o m m e l (1983).
s. Hart, B. Weiss / Statistics & Probability Letters 35 (1997) 43-48
46
for all i ~ I and j ~ J. Thus (DLP) becomes
minimize n ~ 6jp r
j~J
subject to: ~ / ~ y , / > 1
for all y e Q,
(DLP')
i~l
,uy, >~ 0
for all y6Q.
iEl
Let/~ = (PJ)r~s be an optimal point for (DLP'); recall that all the fir are non-negative. We have:
(1) /it ~> 1/n (the vector y = (1, 1 . . . . ,1) belongs to Q).
(2) pj ~> 0 for all j e J (the vector y = (j,j, ... ,j) does not belong to Q when j / > 2).
(3) Without loss of generality 5, p, + 1 = 0 (indeed #, + 1 does not a p p e a r in the constraints for y e Q; by (2),
decreasing p,+ ~ to 0 does not violate the constraints for y¢Q, and can only decrease the value of the objective
function).
(4) W i t h o u t loss of generality, Pr ~> P J÷ 1 for all j e J, j ~< n (let j be the highest index such that pj < Pr+ 1;
put/~)+ 1 := ]Aj and Pk''.= Pk for k C j, then #' is also feasible - if y belongs to Q and, for some i, we have
Yl = j + 1, then y' belongs to Q too, where y~ := j and y~ := Yk for k ~ i - and, again, the value of the objective
function could have only decreased).
n
(5) Without loss of generality, Y~r~J #r = Y~r= a/~r = 1 (the vector y = (1, 2 . . . . . n) belongs to Q, therefore
~ j = ~/~j/> 1; by (4), this inequality implies that the constraints for all y in Q are satisfied; if the inequality were
strict, we could therefore decrease some of the/tj's, remain feasible and decrease the objective function).
n
(6) W i t h o u t loss of generality, # is a convex c o m b i n a t i o n of the n vectors {mi}i~i, defined by
m i : = (l/i, 1/i . . . . . l/i, 0,0 . . . . . O)
t
"v"_ _ J k . . . . . . _
i
)
n+l-i
(indeed, p = ~i~i i(pi - pi+ l)mi).
Therefore, the minimal value o f ( D L P ' ) is obtained as the m i n i m u m over the points {mi}i~ i, hence it equals
m i n i , ! {n Ej <,i fir/i } = mini~i {noel~ i }, as claimed.
T o complete the proof, one needs to exhibit, for any given marginals (Fi)i~i, a joint distribution where the
b o u n d is attained. This is done as follows: Let p* = (p*(y))y be an optimal point for the primal p r o b l e m (LP).
This determines the distribution of Y, by P ( Y = y ) : - - p * ( y ) for all y. Define the Xi's to be conditionally
independent given Y, specifically P ( X i <~ xi for all i[ Y = y ) : = tqi~! P ( X i <~ xl and ~i ~ Dr,)/6y, for6 all y. It
can now be checked that, indeed, P(~i ~ Dyi for all i) = p*(y) for all y, and that the marginal of each Xi is
precisely Fi. []
A few remarks are now in order.
R e m a r k s (i) The inequality P(R) <~ mini {no~/i} follows directly from R c Ri.~, and P(Ri, a) <~ nfl/i for all
fl and all i. The probabilistic inequality is proved as follows: Let B j : = {~j ~< fl} and Ek:= {exactly k of the
5If 6.+ 1 > 0 then necessarily #.+ 1 = 0; if 6,+ 1 = 0, then #.+ 1 may be taken to be 0. Similarly for the following statements.
6 Note that the ith term in the product equals (Fi(xi) - ~r,- 1)/(c% - ~y,_~) if Fi(xl) ~ Dr,, it is 0 if Fi(xi) ~<~r,- ~ and 1 if F~(x,) >1~y,.
S. Hart, B. Weiss / Statistics & Probability Letters 35 (1997) 43-48
47
{Bj}j~}, then
2
jEI
1B~ =
k l E k >/
~
k= 1
~ kl~ >~~
k=i
ilE~, =
ilR,.
k=i
Here lc denotes the indicator function of the event C; the first equality is due to the fact that each point in E k
is counted exactly k times in the left-most sum. Therefore,
i P(Ri.~) <~ ~" P(Bj) = nil.
jel
(ii) The linear programming approach is needed to prove the exactness of the bound for arbitrary
marginals (F~)~. If all the marginal distributions are continuous, thus all the ~i are uniformly distributed on
[0, 1], a joint distribution P with P ( R ) = mini{nai/i} may be explicitly constructed as follows: Put
7 := mini {~i/i}. Let Ao be a one-dimensional "diagonal" in [0, 1]" of length 1-nT:
Ao:= {x = (xi)~, e [0, 1]"Ix, = n7 + t for all i e I , t e [0,1 - nT)}.
For each permutation ~z of I, let A (~z) be a diagonal of length 7:
A(rt):= {x = (xi)i~ie [0, 1]"l xi = (Tr(i) - 1)7 + t for all i e I , t e [0,7)}.
We then put probability mass 1 - n7 on Ao and, for every one of the n! permutations n, mass 7/(n - 1)! on
A (z~);the mass is uniformly spread over each one of these diagonals. It is now easy to check that the resulting
probability distribution P on [0, 1]" has uniform marginals, and that P(R) = P ( ~ ) A ( r t ) ) = nT.
This construction however breaks down if there are atoms (recall that we have assumed that the possible
values of each ~i only include the ctj but not necessarily the multiples of 7 used above).
(iii) One may consider other types of rejection regions. For instance, R' := {~i) ~< ai for some i}. Such tests
have been considered by Hommel (1983), Simes (1986), R6hmel and Streitberg (1987), Hochberg (1988), and
others. The significance level is c~(R') = min {}2~=1 n(cti - cq_ 1)/i, 1} - see Hommel (1983). One may again use
the linear programming approach to obtain this bound. However, for such R' the direct proof (parallel to (i)
and (ii) above) works. Still, if additional linear constraints were added (which express restrictions on the set of
distributions in Ho), then the linear programming approach may be useful.
(iv) Our result implies a lower bound for the probability of the rejection region R' (cf. (iii) above). Indeed,
R for ( - Xi)i is essentially 7 the complementary event of R' for (X~)~. Therefore,
P({~i' ~< ~i for some i}) >~ 1 -
min
n(1 - e , + l - i )
l~i<~n
-
i
n~,+ l-i - n + i
max
l~i<~n
i
In the case of Simes' test, this can be shown to simplify to
P( { ~ll) <~ i~/n for some i } >~ m a x {~ , l - n ( 1 -
~) }.
Similarly, the result of Hommel (1983) yields a lower bound for the region R in (1), namely,
rora./,, max{1.
i=1
i
'
0}
'
v F o r i = 1, ... , n + 1, d e f i n e t h e i n t e r v a l s D~ : = D . + 2 ,; t h u s 6~ = 6 . + 2 - ~ a n d ct~ = 1 -
ct. + ~ _ ~.
48
S. Hart, B. Weiss / Statistics & Probability Letters 35 (1997) 43-48
One possible use for these lower bounds is to evaluate the power of the tests. However, this requires the
alternative Ha to be of a special form, namely, that all the Xi's are identically distributed under Ha (with an
arbitrary joint distribution; actually, it suffices to require that, for each 0 in Ha, all the probabilities
Po(Xi <~~j) do not depend on i}.
Acknowledgements
We thank Robert J. Aumann, Yoav Benjamini, Gerhard Hommel, Yossef Hochberg and Ester SamuelCahn, for useful comments.
References
Bonferroni, C.E., 1936. Theoria statistica classi e calcolo delle probabilita. Pubbl. R. Int. Super. Sci. Econ. Comm. Firenze 8, 1 62.
Hochberg, Y., 1988. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800-802.
Hochberg, Y., Hommel, G., 1996. Simes' test and generalizations, in: Kotz, S., Johnson N.L., (Eds.), Encyclopedia of Statistical Sciences.
Wiley, New York, forthcoming.
Hommel, G., 1983. Tests of the overall hypothesis for arbitrary dependence structures. Biometrical J. 25, 423~430.
R6hmel, J., Streitberg, B., 1987. Zur Konstruktion globaler Tests. EDV in Medizin und Biologie 18, 7-11.
RiJger, B., 1978. Das Maximale Signifikanzniveau des Tests "Lehne Ho ab, wenn k unter n gegebenen Tests zur Ablehnung fiihren'"
Metrika 25, 171-178.
Simes, R.J., 1986. An improved Bonferroni procedure for multiple tests of significance. Biometrika 73, 751-754.