Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
HYPOTHESIS TESTING 1. Introduction 1.1. Hypothesis testing. Let {fθ }θ∈Θ be a family of pdfs. Let θ ∈ Θ, where θ is unknown. Let ΘN ⊂ Θ. A statement of the form ‘θ ∈ ΘN ’ is a hypothesis. Let X = (X1 , . . . , Xn ) be a random sample from fθ . A hypothesis test consists a hypothesis called the null hypothesis, of the form θ ∈ ΘN , and a function φ such that φ(X) ∈ {0, 1}; in a hypothesis test if φ(X) = 0, then we retain the null hypothesis, otherwise we reject the null hypothesis in favor of the alternate hypothesis given by θ ∈ ΘcN . Sometimes, we allow for extrarandomization: we let U be uniformly distributed in [0, 1] and independent of X and we allow φ(X) ∈ [0, 1], and reject the null hypothesis if U ≤ φ(X). Thus in context of a randomized test, φ(X) is the probability that we reject the null-hypothesis. Randomized test are useful in stating nice theorems, especially in the case of discrete distributions, but in practice are not used very often, since two experimenters with the same data set may end up with different conclusions. The null hypothesis is often denoted as H0 and the alternate hypothesis is often denoted as H1 or Ha . 1.2. Critical regions, errors, significance, and power. Sometimes φ is called the critical function, and the critical event is the event {φ(X) = 1} and the critical region is the subset C in the range of X such that {X ∈ C} = {φ(X) = 1}. Sometimes C is also called the rejection region. A type-I error occurs if H0 is rejected when it is true, and a type-II error occurs if H0 is retained, when H1 is true. An experimenter aims to design a test that will balance the minimization of these two errors, depending on the application. The (exact) size (or significance) of a test is the probability is given by α := sup Pθ (X ∈ C); θ∈ΘN in the case where ΘN is a single point, α is exactly the probability of a type-I error. A test is of (significance) level α ∈ (0, 1) if α ≥ sup Pθ (X ∈ C). θ∈ΘN 1 2 HYPOTHESIS TESTING Note, sometimes the words size and level are freely interchanged; in this course, we will try to keep this distinction. The power function β : Θ → [0, 1], is defined by β(θ) := Pθ (X ∈ C). Thus the size of a test is given by supθ∈ΘN β(θ). A good test will have a power function that is close to 0 for all θ ∈ ΘN , and close to 1 for all θ ∈ ΘcN . 2. Likelihood ratio tests The mle did not always provide the best estimators, but in many cases it was a good first start. Here, we will derive tests based on likelihood methods. Let {fθ }θ∈Θ be a family of pdfs. Let θ ∈ Θ, where θ is unknown. Let ΘN ⊂ Θ. We consider the following hypothesis: H0 : θ ∈ ΘN . Let X = (X1 , . . . , Xn ) be a random sample from fθ . Consider the ratio given by Λ = Λ(X) := supθ∈ΘN L(X; θ) . supθ∈Θ L(X; θ) By definition, we have that Λ ≤ 1. However, if H0 is true, it seems that Λ should be close to 1. Thus for c ∈ (0, 1), we define the following likelihood ratio test given by φc (X) = 1[Λ ≤ c]. Here the exact size the test if given by sup Pθ (Λ ≤ c). θ∈ΘN Suppose that T = Tn is the maximum likelihood estimator for θ, then supθ∈ΘN L(X; θ) . L(X; T ) In the next subsections we will calculate the likelihood ratio test for some familiar distributions. Λ = Λ(X) := 2.1. The mean of a normal distribution (known variance). Let X = (X1 , . . . , Xn ) is a random sample from the normal distribution with mean µ and known variance σ 2 . Let µ0 ∈ R. Consider the likelihood ratio test with H0 : µ = µ0 , and critical region given by Λ ≤ c, where c ∈ (0, 1). HYPOTHESIS TESTING 3 We know that the mle is given the sample mean. Recall we have the following useful identity: n n X X 2 2 (Xi − µ0 ) = n(X̄ − µ0 ) + (Xi − X̄)2 . i=1 i=1 This easily implies that: Λ(X) = exp(−n(X̄ − µ0 )2 /2σ 2 ). We want to transform this random variable to make it look more familiar. Set X̄ − µ √ Zµ := σ/ n and Zµ0 = Z. We have that X̄ − µ 2 √ 0 = Z 2. Y := −2 log(Λ) = σ/ n moreover, Λ ≤ c if and only if Y ≥ k(c). In this case, we know that k(c) = −2 log(c) > 0. But Y is nice, since if H0 is true, then Z ∼ N (0, 1) and Y ∼ χ2 (1)–a known distribution. Thus if we want a test of size α, we need only find k, such that Pµ0 (Y ≥ k) = α; this is undergraduate. Next, we want to make this test look like a familiar undergraduate test. Clearly, for d > 0, we have Y ≥ d2 = {Z ≥ d} ∪ {Z ≤ −d} . Also, µ − µ0 √ ≥d . {Z ≥ d} = Zµ + σ/ n Thus we can compute β(µ), since Zµ ∼ N (0, 1) under Pµ . 2.2. The Gamma distribution. In the previous subsection, −2 log(Λ) had the χ2 distribution. This was no accident, and we will see how the χ2 distribution occurs again here. Let X = (X1 , . . . , Xn ) is a random sample from the gamma distribution, where X1 ∼ Γ(a, β), where a is known and fixed. Thus the pdf of X1 is given by f (x1 ; β) = 1 xa−1 e−x1 /β 1[x1 a β Γ(a) 1 ≥ 0]. 4 HYPOTHESIS TESTING Let β0 > 0. Consider the likelihood ratio test with H0 : β = β0 , and critical region given by Λ ≤ c, where c ∈ (0, 1). We have that n 1 −nX̄/β0 Y 1 L(X; β0 ) = an e Xia−1 . β0 Γ(a) i=1 We also know that X̄/a is the mle for β, so that X̄ na Λ = ena e−nX̄/β0 aβ0 and X̄ . β0 Notice that by the law of large numbers, under H0 , we have log(Λ) = na + na log(X̄/aβ0 ) − n X̄/aβ0 → 1 and X̄/β0 → a. In order to compute the limit, we need to be more clever. Recall that by the central limit theorem √ n(X̄ − aβ0 ) →d N (0, aβ02 ). Thus applying the second order delta method with f (z) = a(log(z) − z/aβ0 ), f 0 (z) = a(1/z − 1/aβ0 ), and f 00 (z) = −a/z 2 we have that 2n(f (X̄) − f (aβ0 )) →d f 00 (aβ0 )aβ02 χ2 . From which we have that −2 log(Λ) →d χ2 . 2.3. The asymptotic distribution of Λ. Let X = (X1 , . . . , Xn ) be a random sample from fθ0 . We proved, under suitable regularity conditions, that if Tn is the mle, then Tn → θ0 in probability and √ n(Tn − θ0 ) →d N (0, 1/I(θ0 )), with respect to Pθ0 . The same ideas give the following theorem. Theorem 1. Let X = (X1 , . . . , Xn ) be a random sample from fθ . Let θ0 ∈ Θ. Consider a likelihood ratio test for the hypothesis H0 : θ = θ0 . Under the same regularity conditions as we required for the asymptotic efficiency of the mle, under H0 , we have that −2 log(Λ) →d χ2 . In particular, we have that for c ∈ (0, 1), and k(c) = −2 log(c) that Pθ0 (Λ ≤ c) ≈ P(χ2 ≥ k(c)). HYPOTHESIS TESTING 5 Proof. We recall some notation and facts. Let Tn be the mle. Set `(θ) = `(X; θ). Recall that almost surely, we have `00 (θ0 ) → −I(θ0 ), n where I(θ0 ) is the Fisher information. We also had that `0 (θ0 ) d √ → N (0, I(θ0 )). n We took used Taylor’s theorem on `0 at the point θ0 and obtained that √ √ √ (−1/ n)`0 (θ0 ) = (1/n)`00 (θ0 ) · ( n)[Tn − θ0 ] + (1/ n)r(Tn )[Tn − θ0 ], where r(Tn ) → 0 in probability, since Tn → θ0 in probability; thus √ √ √ (1/ n)`0 (θ0 ) = Jn · n[Tn − θ0 ] − (1/ n)r(Tn )[Tn − θ0 ], where Jn = −(1/n)`00 (θ0 ). With this notation, −2 log(Λ) = 2[`(Tn ) − `(θ0 )]. An application of Talyor’s theorem on ` at the point θ0 gives 1 `(Tn ) = `(θ0 ) + (Tn − θ0 )`0 (θ0 ) + (Tn − θ0 )2 `00 (θ0 ) + s(Tn )(Tn − θ0 )2 2 where s(Tn ) → 0 in probability, since Tn → θ0 in probability. Rearranging gives −2 log(Λ) = −Jn n[Tn − θ0 ]2 + 2nJn [Tn − θ0 ]2 − 2r(Tn )[Tn − θ0 ]2 + 2s(Tn )[Tn − θ0 ]2 = Jn n[Tn − θ0 ]2 + Un , where Un → 0 in probability. Since Jn → I(θ0 ) in probability, and √ n[Tn − θ0 ] →d N (0, 1/I(θ0 )), the desired result follows by Slutsky’s theorem.