Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
CONSISTENCY OF MLE Abstract. We prove that under some regularity conditions that the mle is consistent. We will only consider the case where θ is a real number. 1. Some Regularity conditions Let fθ : R → [0, ∞) be a family of pdfs indexed by θ ∈ Θ ⊂ R. The support of f is the set {x ∈ R : f (x) > 0}. Consider the following conditions on the family. (A) The pdfs are distinct; that is, if θ 6= θ0 , thenfθ 6= fθ0 . (B) The pdfs ahve common support. (C) The true value θ0 is an interior point of Θ; that is, there is an open interval contaning θ0 that is a subset of Θ. (D) The pdf f (x; θ) is differentiable as a function of θ. Lemma 1. Let fθ be a family of pdfs indexed by θ ∈ Θ. Let X = (X1 , . . . , Xn ) be a random sample from fθ0 , where θ0 is the true parameter. Assuming conditons (A) and (B), for all θ 6= θ0 , we have that lim Pθ0 (L(θ0 ; X) > L(θ; X)) = 1 n→∞ In order to prove Lemma 1, we need to recall Jensen’s inequality. Theorem 2. Let X be a real-valued random variable. Suppose φ is convex function, then E(φ(X)) ≥ φ(EX). In addition, if φ is strictly convex, then if E(φ(X)) = φ(EX), then in fact, EX = X. One way to remember Theorem 2 is that it with f (x) = x2 , it gives that EX 2 ≥ (EX)2 , which is consistent with that fact that 0 ≤ Var(X) = EX 2 − (EX)2 . Proof of Lemma 1. Consider the random variables Yi = f (Xi ; θ) . f (Xi ; θ0 ) Note that Yi is well defined by (B) and not a constant almost surely by assumption (A). Thus by applying Jensen’s inequality with the strictly 1 2 CONSISTENCY OF MLE concave function φ(x) = log x, we have that, (in the case that Xi are discrete) Eθ0 φ(Yi ) < φ(Eθ0 Yi ) X f (x; θ) = φ × f (x; θ0 ) f (x; θ0 ) x = φ(1) = 0 On the other hand, we have that L(θ0 ; X) > L(θ; X) if and only if n 1X Zn := φ(Yi ) < 0. n i=1 We know that P(Zn > 0) → 1, since we know by the law of large numbers that Zn converges to Eθ0 φ(Y1 ) < 0 in probability. Exercise 3. Let F (θ) := Eθ0 log(f (X1 ; θ)). Show that F (θ) < F (θ0 ) under assumptions (A) and (B). Exercise 4. Show that 1 `(X; θ) → F (θ), n in almost surely, as n → ∞. Theorem 5. Let fθ be a family of pdfs indexed by θ ∈ Θ. Let X = (X1 , . . . , Xn ) be a random sample from fθ0 , where θ0 is the true parameter. Assuming conditons (A), (B), (C), and (D), then if likelhood equation L0 (θ) = 0 has a unique solution Tn , then Tn → θ0 in probability as n → ∞. Proof. Let ε > 0. By assumption (C), also assume ε is sufficiently small so that I := [θ0 − ε, θ0 + ε] ⊂ Θ. Consider the event En := {L(X; θ0 ) > L(X; θ0 − ε)} ∩ {L(X; θ0 ) > L(X; θ0 + ε)} . Notice that on the event En , as a function θ, the likelihood must have a local maximum on I, thus it is obtained at Tn by assumption and elementary calculus. Thus we have that En ⊂ {|Tn − θ0 | < ε} . We know by Lemma 1 that P(En ) → 1 as n → ∞, so it follows that Tn → θ0 in probability.