Download CONSISTENCY OF MLE 1. Some Regularity conditions Let fθ : R

CONSISTENCY OF MLE
Abstract. We prove that under some regularity conditions that
the mle is consistent. We will only consider the case where θ is a
real number.
1. Some Regularity conditions
Let fθ : R → [0, ∞) be a family of pdfs indexed by θ ∈ Θ ⊂ R. The
support of f is the set {x ∈ R : f (x) > 0}. Consider the following
conditions on the family.
(A) The pdfs are distinct; that is, if θ 6= θ0 , thenfθ 6= fθ0 .
(B) The pdfs ahve common support.
(C) The true value θ0 is an interior point of Θ; that is, there is an open
interval contaning θ0 that is a subset of Θ.
(D) The pdf f (x; θ) is differentiable as a function of θ.
Lemma 1. Let fθ be a family of pdfs indexed by θ ∈ Θ. Let X =
(X1 , . . . , Xn ) be a random sample from fθ0 , where θ0 is the true parameter. Assuming conditons (A) and (B), for all θ 6= θ0 , we have
that
lim Pθ0 (L(θ0 ; X) > L(θ; X)) = 1
n→∞
In order to prove Lemma 1, we need to recall Jensen’s inequality.
Theorem 2. Let X be a real-valued random variable. Suppose φ is
convex function, then E(φ(X)) ≥ φ(EX). In addition, if φ is strictly
convex, then if E(φ(X)) = φ(EX), then in fact, EX = X.
One way to remember Theorem 2 is that it with f (x) = x2 , it gives
that EX 2 ≥ (EX)2 , which is consistent with that fact that
0 ≤ Var(X) = EX 2 − (EX)2 .
Proof of Lemma 1. Consider the random variables
Yi =
f (Xi ; θ)
.
f (Xi ; θ0 )
Note that Yi is well defined by (B) and not a constant almost surely by
assumption (A). Thus by applying Jensen’s inequality with the strictly
1
2
CONSISTENCY OF MLE
concave function φ(x) = log x, we have that, (in the case that Xi are
discrete)
Eθ0 φ(Yi ) < φ(Eθ0 Yi )
X f (x; θ)
= φ
× f (x; θ0 )
f (x; θ0 )
x
= φ(1) = 0
On the other hand, we have that L(θ0 ; X) > L(θ; X) if and only if
n
1X
Zn :=
φ(Yi ) < 0.
n i=1
We know that P(Zn > 0) → 1, since we know by the law of large
numbers that Zn converges to Eθ0 φ(Y1 ) < 0 in probability.
Exercise 3. Let F (θ) := Eθ0 log(f (X1 ; θ)). Show that F (θ) < F (θ0 )
under assumptions (A) and (B).
Exercise 4. Show that
1
`(X; θ) → F (θ),
n
in almost surely, as n → ∞.
Theorem 5. Let fθ be a family of pdfs indexed by θ ∈ Θ. Let X =
(X1 , . . . , Xn ) be a random sample from fθ0 , where θ0 is the true parameter. Assuming conditons (A), (B), (C), and (D), then if likelhood
equation L0 (θ) = 0 has a unique solution Tn , then Tn → θ0 in probability
as n → ∞.
Proof. Let ε > 0. By assumption (C), also assume ε is sufficiently small
so that I := [θ0 − ε, θ0 + ε] ⊂ Θ. Consider the event
En := {L(X; θ0 ) > L(X; θ0 − ε)} ∩ {L(X; θ0 ) > L(X; θ0 + ε)} .
Notice that on the event En , as a function θ, the likelihood must have
a local maximum on I, thus it is obtained at Tn by assumption and
elementary calculus. Thus we have that
En ⊂ {|Tn − θ0 | < ε} .
We know by Lemma 1 that P(En ) → 1 as n → ∞, so it follows that
Tn → θ0 in probability.