Download HYPOTHESIS TESTING 1. Introduction 1.1. Hypothesis testing. Let {f

HYPOTHESIS TESTING
1. Introduction
1.1. Hypothesis testing. Let {fθ }θ∈Θ be a family of pdfs. Let θ ∈ Θ,
where θ is unknown. Let ΘN ⊂ Θ. A statement of the form ‘θ ∈ ΘN ’
is a hypothesis. Let X = (X1 , . . . , Xn ) be a random sample from fθ .
A hypothesis test consists a hypothesis called the null hypothesis,
of the form θ ∈ ΘN , and a function φ such that φ(X) ∈ {0, 1}; in
a hypothesis test if φ(X) = 0, then we retain the null hypothesis, otherwise we reject the null hypothesis in favor of the alternate hypothesis given by θ ∈ ΘcN . Sometimes, we allow for extrarandomization: we let U be uniformly distributed in [0, 1] and independent of X and we allow φ(X) ∈ [0, 1], and reject the null hypothesis if
U ≤ φ(X). Thus in context of a randomized test, φ(X) is the probability that we reject the null-hypothesis. Randomized test are useful
in stating nice theorems, especially in the case of discrete distributions,
but in practice are not used very often, since two experimenters with
the same data set may end up with different conclusions.
The null hypothesis is often denoted as H0 and the alternate hypothesis is often denoted as H1 or Ha .
1.2. Critical regions, errors, significance, and power. Sometimes
φ is called the critical function, and the critical event is the event
{φ(X) = 1} and the critical region is the subset C in the range of
X such that {X ∈ C} = {φ(X) = 1}. Sometimes C is also called the
rejection region. A type-I error occurs if H0 is rejected when it is
true, and a type-II error occurs if H0 is retained, when H1 is true. An
experimenter aims to design a test that will balance the minimization
of these two errors, depending on the application. The (exact) size (or
significance) of a test is the probability is given by
α := sup Pθ (X ∈ C);
θ∈ΘN
in the case where ΘN is a single point, α is exactly the probability of
a type-I error. A test is of (significance) level α ∈ (0, 1) if
α ≥ sup Pθ (X ∈ C).
θ∈ΘN
1
2
HYPOTHESIS TESTING
Note, sometimes the words size and level are freely interchanged; in
this course, we will try to keep this distinction. The power function
β : Θ → [0, 1], is defined by
β(θ) := Pθ (X ∈ C).
Thus the size of a test is given by supθ∈ΘN β(θ). A good test will have
a power function that is close to 0 for all θ ∈ ΘN , and close to 1 for all
θ ∈ ΘcN .
2. Likelihood ratio tests
The mle did not always provide the best estimators, but in many
cases it was a good first start. Here, we will derive tests based on
likelihood methods. Let {fθ }θ∈Θ be a family of pdfs. Let θ ∈ Θ, where
θ is unknown. Let ΘN ⊂ Θ. We consider the following hypothesis:
H0 : θ ∈ ΘN . Let X = (X1 , . . . , Xn ) be a random sample from fθ .
Consider the ratio given by
Λ = Λ(X) :=
supθ∈ΘN L(X; θ)
.
supθ∈Θ L(X; θ)
By definition, we have that Λ ≤ 1. However, if H0 is true, it seems
that Λ should be close to 1. Thus for c ∈ (0, 1), we define the following
likelihood ratio test given by
φc (X) = 1[Λ ≤ c].
Here the exact size the test if given by
sup Pθ (Λ ≤ c).
θ∈ΘN
Suppose that T = Tn is the maximum likelihood estimator for θ,
then
supθ∈ΘN L(X; θ)
.
L(X; T )
In the next subsections we will calculate the likelihood ratio test for
some familiar distributions.
Λ = Λ(X) :=
2.1. The mean of a normal distribution (known variance). Let
X = (X1 , . . . , Xn ) is a random sample from the normal distribution
with mean µ and known variance σ 2 . Let µ0 ∈ R. Consider the likelihood ratio test with H0 : µ = µ0 , and critical region given by Λ ≤ c,
where c ∈ (0, 1).
HYPOTHESIS TESTING
3
We know that the mle is given the sample mean. Recall we have the
following useful identity:
n
n
X
X
2
2
(Xi − µ0 ) = n(X̄ − µ0 ) +
(Xi − X̄)2 .
i=1
i=1
This easily implies that:
Λ(X) = exp(−n(X̄ − µ0 )2 /2σ 2 ).
We want to transform this random variable to make it look more familiar. Set
X̄ − µ
√
Zµ :=
σ/ n
and Zµ0 = Z. We have that
X̄ − µ 2
√ 0 = Z 2.
Y := −2 log(Λ) =
σ/ n
moreover, Λ ≤ c if and only if Y ≥ k(c). In this case, we know that
k(c) = −2 log(c) > 0.
But Y is nice, since if H0 is true, then Z ∼ N (0, 1) and Y ∼ χ2 (1)–a
known distribution. Thus if we want a test of size α, we need only find
k, such that Pµ0 (Y ≥ k) = α; this is undergraduate.
Next, we want to make this test look like a familiar undergraduate
test.
Clearly, for d > 0, we have
Y ≥ d2 = {Z ≥ d} ∪ {Z ≤ −d} .
Also,
µ − µ0
√ ≥d .
{Z ≥ d} = Zµ +
σ/ n
Thus we can compute β(µ), since Zµ ∼ N (0, 1) under Pµ .
2.2. The Gamma distribution. In the previous subsection, −2 log(Λ)
had the χ2 distribution. This was no accident, and we will see how the
χ2 distribution occurs again here. Let X = (X1 , . . . , Xn ) is a random
sample from the gamma distribution, where X1 ∼ Γ(a, β), where a is
known and fixed. Thus the pdf of X1 is given by
f (x1 ; β) =
1
xa−1 e−x1 /β 1[x1
a
β Γ(a) 1
≥ 0].
4
HYPOTHESIS TESTING
Let β0 > 0. Consider the likelihood ratio test with H0 : β = β0 , and
critical region given by Λ ≤ c, where c ∈ (0, 1). We have that
n
1 −nX̄/β0 Y 1
L(X; β0 ) = an e
Xia−1 .
β0
Γ(a)
i=1
We also know that X̄/a is the mle for β, so that
X̄ na
Λ = ena
e−nX̄/β0
aβ0
and
X̄
.
β0
Notice that by the law of large numbers, under H0 , we have
log(Λ) = na + na log(X̄/aβ0 ) − n
X̄/aβ0 → 1
and
X̄/β0 → a.
In order to compute the limit, we need to be more clever. Recall that
by the central limit theorem
√
n(X̄ − aβ0 ) →d N (0, aβ02 ).
Thus applying the second order delta method with f (z) = a(log(z) −
z/aβ0 ), f 0 (z) = a(1/z − 1/aβ0 ), and f 00 (z) = −a/z 2 we have that
2n(f (X̄) − f (aβ0 )) →d f 00 (aβ0 )aβ02 χ2 .
From which we have that
−2 log(Λ) →d χ2 .
2.3. The asymptotic distribution of Λ. Let X = (X1 , . . . , Xn ) be
a random sample from fθ0 . We proved, under suitable regularity conditions, that if Tn is the mle, then Tn → θ0 in probability and
√
n(Tn − θ0 ) →d N (0, 1/I(θ0 )),
with respect to Pθ0 . The same ideas give the following theorem.
Theorem 1. Let X = (X1 , . . . , Xn ) be a random sample from fθ . Let
θ0 ∈ Θ. Consider a likelihood ratio test for the hypothesis H0 : θ = θ0 .
Under the same regularity conditions as we required for the asymptotic efficiency of the mle, under H0 , we have that
−2 log(Λ) →d χ2 .
In particular, we have that for c ∈ (0, 1), and k(c) = −2 log(c) that
Pθ0 (Λ ≤ c) ≈ P(χ2 ≥ k(c)).
HYPOTHESIS TESTING
5
Proof. We recall some notation and facts. Let Tn be the mle. Set
`(θ) = `(X; θ). Recall that almost surely, we have
`00 (θ0 )
→ −I(θ0 ),
n
where I(θ0 ) is the Fisher information. We also had that
`0 (θ0 ) d
√ → N (0, I(θ0 )).
n
We took used Taylor’s theorem on `0 at the point θ0 and obtained that
√
√
√
(−1/ n)`0 (θ0 ) = (1/n)`00 (θ0 ) · ( n)[Tn − θ0 ] + (1/ n)r(Tn )[Tn − θ0 ],
where r(Tn ) → 0 in probability, since Tn → θ0 in probability; thus
√
√
√
(1/ n)`0 (θ0 ) = Jn · n[Tn − θ0 ] − (1/ n)r(Tn )[Tn − θ0 ],
where Jn = −(1/n)`00 (θ0 ).
With this notation,
−2 log(Λ) = 2[`(Tn ) − `(θ0 )].
An application of Talyor’s theorem on ` at the point θ0 gives
1
`(Tn ) = `(θ0 ) + (Tn − θ0 )`0 (θ0 ) + (Tn − θ0 )2 `00 (θ0 ) + s(Tn )(Tn − θ0 )2
2
where s(Tn ) → 0 in probability, since Tn → θ0 in probability.
Rearranging gives
−2 log(Λ) = −Jn n[Tn − θ0 ]2 + 2nJn [Tn − θ0 ]2 − 2r(Tn )[Tn − θ0 ]2 + 2s(Tn )[Tn − θ0 ]2
= Jn n[Tn − θ0 ]2 + Un ,
where Un → 0 in probability. Since Jn → I(θ0 ) in probability, and
√
n[Tn − θ0 ] →d N (0, 1/I(θ0 )),
the desired result follows by Slutsky’s theorem.