Download A Proof of the Central Limit Theorem

A Proof of the Central Limit Theorem
Preliminaries: Two Definitions and Two Theorems
Definition One: The rth moment of a random variable X about the mean is
𝜇𝑟 = 𝐸[(𝑋 − 𝜇)𝑟 ] , 𝑟 = 0,1,2,3, …
[Note: It follows that 𝜇0 = 1, 𝜇1 = 0, and 𝜇2 = 𝜎 2 .]
Thus, we have
𝑟
𝜇𝑟 = ∑𝑛𝑗=1(𝑥𝑗 − 𝜇) 𝑝(𝑥𝑗 ) = ∑(𝑥 − 𝜇)𝑟 𝑝(𝑥)
∞
𝜇𝑟 = ∫−∞(𝑥 − 𝜇)𝑟 𝑓(𝑥)𝑑𝑥
(discrete variable)
(continuous variable)
Definition Two: The moment-generating function (MGF) of X is
𝑀𝑋 (𝑡) = 𝐸(𝑒 𝑡𝑋 )
Thus, we have
𝑀𝑋 (𝑡) = ∑𝑛𝑗=1 𝑒 𝑡𝑥𝑗 𝑝(𝑥𝑗 ) = ∑ 𝑒 𝑡𝑥 𝑝(𝑥)
∞
𝑀𝑋 (𝑡) = ∫−∞ 𝑒 𝑡𝑥 𝑓(𝑥)𝑑𝑥
(discrete case)
(continuous case)
[Note: The name moment-generating function comes from the fact that
𝑀𝑋 (𝑡) = 𝐸(𝑒 𝑡𝑋 ) = 𝐸 (1 + 𝑡𝑋 +
𝑡2𝑋2
2!
+
𝑡3𝑋3
3!
𝑡2
+⋯)
𝑡3
= 1 + 𝐸(𝑋)𝑡 + 𝐸(𝑋 2 ) 2! + 𝐸(𝑋 3 ) 3! + ⋯
which shows the moments about the origin 𝜇 = 0 [𝐸(𝑋), 𝐸(𝑋 2 ), 𝐸(𝑋 3 ), …] are
generated as the coefficients in this expansion in 𝑡.]
Proof of Central Limit Theorem
Page Two
Theorem One: The MGF for the general normal distribution is
𝑀(𝑡) = 𝑒 𝜇𝑡+(𝜎
2 𝑡 2 /2)
Proof: By definition, we have
∞
1
𝑀(𝑡) = 𝐸(𝑒 𝑡𝑥 ) = 𝜎√2𝜋 ∫−∞ 𝑒 𝑡𝑥 𝑒 −(𝑥−𝜇)
2 /2𝜎 2
𝑑𝑥
Letting (𝑥 − 𝜇) / 𝜎 = 𝑣 so that 𝑥 = 𝜇 + 𝜎𝑣 and 𝑑𝑥 = 𝜎𝑑𝑣, this becomes
𝑀(𝑡) =
1
√2𝜋
𝑒 𝜇𝑡+(𝜎
2 𝑡 2 /2)
∞
∫−∞ 𝑒 −(𝑣−𝜎𝑡)
2 /2
𝑑𝑣
Now letting 𝑣 − 𝜎𝑡 = 𝑤, we have
𝑀(𝑡) = 𝑒 𝜇𝑡+(𝜎
2 𝑡 2 /2)
(
∞
1
∫ 𝑒 −𝑤
√2𝜋 −∞
2 /2
𝑑𝑤) = 𝑒 𝜇𝑡+(𝜎
2 𝑡 2 /2)
Corollary: The MGF for the standard normal distribution is
𝑀(𝑡) = 𝑒 𝑡
2 /2
Theorem Two (Uniqueness Theorem): Two random variables have the same probability
distribution if and only if their moment-generating
functions are identical.
We are now ready to prove the Central Limit Theorem.
Proof of Central Limit Theorem (CLT)
Theorem (CLT): Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be independent random variables that are identically
distributed (that is, all have the same probability mass function in the discrete
case or probability density function in the continuous case) and have finite
mean 𝜇 and variance 𝜎 2 . Then if 𝑆𝑛 = 𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛 (n = 1, 2, …),
lim 𝑃(𝑎 ≤
𝑛→∞
𝑆𝑛 −𝑛𝜇
𝜎 √𝑛
≤ 𝑏) =
1
𝑏
∫ 𝑒 −𝑢
√2𝜋 𝑎
2 /2
𝑑𝑢
that is, the random variable (𝑆𝑛 − 𝑛𝜇)/(𝜎√𝑛 ), which is the standardized
variable corresponding to 𝑆𝑛 , is asymptotically normal.
Note: Since (𝑆𝑛 − 𝑛𝜇)/(𝜎√𝑛 ) = (𝑋̅ − 𝜇)/(𝜎⁄√𝑛 ) = 𝑍, the more familiar form of the CLT
follows as a corollary.
Proof of Central Limit Theorem
Page Three
Proof: For 𝑛 = 1, 2, …, we have 𝑆𝑛 = 𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛 . Now 𝑋1 , 𝑋2 , … , 𝑋𝑛 each have mean 𝜇
and variance 𝜎 2 . Thus,
𝐸(𝑆𝑛 ) = 𝐸(𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛 ) = 𝐸(𝑋1 ) + 𝐸(𝑋2 ) + ⋯ 𝐸(𝑋𝑛 ) = 𝑛𝜇
and, because the 𝑋𝑘 are independent,
𝑉𝑎𝑟(𝑆𝑛 ) = 𝑉𝑎𝑟(𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛 ) = 𝑉𝑎𝑟(𝑋1 ) + 𝑉𝑎𝑟(𝑋2 ) + ⋯ + 𝑉𝑎𝑟(𝑋𝑛 ) = 𝑛𝜎 2
It follows from this that the standardized random variable corresponding to 𝑆𝑛 is
𝑆𝑛 ∗ =
𝑆𝑛 −𝑛𝜇
𝜎 √𝑛
The MGF for 𝑆𝑛 ∗ is thus
∗
𝐸(𝑒 𝑡𝑆𝑛 ) = 𝐸[𝑒 𝑡(𝑆𝑛 −𝑛𝜇)/(𝜎√𝑛) ]
= E[et(X1 −μ)/(σ√n) et(X2 −μ)/(σ√n) ⋯ et(Xn−μ)/(σ√n) ]
= 𝐸[et(X1 −μ)/(σ√n) ]∙ 𝐸[et(X2−μ)/(σ√n) ]⋯ 𝐸[et(Xn−μ)/(σ√n) ]
t(X1 −μ)
σ√n
= {𝐸[e
𝑛
]}
[The last step is a result of the 𝑋𝑘 being identically distributed and the preceding step
because the 𝑋𝑘 are independent.]
Now, by a Taylor series expansion, we have
t(X1 −μ)
σ√n
𝐸 [e
] = 𝐸[1 +
𝑡(𝑋1 −𝜇)
𝜎 √𝑛
= 𝐸(1) +
= 1+
𝑡
𝜎 √𝑛
𝑡
𝜎 √𝑛
+
𝑡 2 (𝑋1 −𝜇)2
2!𝜎 2 𝑛
+ ⋯]
𝐸(𝑋1 − 𝜇) +
(0) +
𝑡2
2𝜎 2 𝑛
𝑡2
2𝜎 2 𝑛
𝐸(𝑋1 − 𝜇)2 + ⋯
(𝜎 2 ) + ⋯ = 1 +
𝑡2
2𝑛
+⋯
Thus, we have
∗
𝑡2
𝑛
𝐸(𝑒 𝑡𝑆𝑛 ) = {1 + + ⋯ } = (1 +
2𝑛
2
𝑡 2 ⁄2
𝑛
+ ⋯ )𝑛
But the limit of this as 𝑛 → ∞ is 𝑒 𝑡 ⁄2 , which is the MGF of the standard normal
distribution. Hence, by Theorem Two above, the required result follows.
Proof of Central Limit Theorem
Page Four
Corollary: Suppose the population from which samples are taken has some probability
distribution with mean 𝜇 and variance 𝜎 2 . Then the standardized variable associated
with 𝑋̅, given by
𝑍=
𝑋̅−𝜇
𝜎 ⁄√ 𝑛
is asymptotically normal, that is
lim 𝑃(𝑍 ≤ 𝑧) =
𝑛→∞
1
𝑧
∫ 𝑒 −𝑤
√2𝜋 −∞
2 ⁄2
𝑑𝑤
Note: The above proof of the CLT, slightly edited, was given by the late Murray R. Spiegel of RPI
in his 1975 Schaum’s Outline Series book Probability and Statistics in Problem 4.25, which
appears as such in the current Third Edition of the book, with John J. Schiller and R. Alu
Srinivasan of Temple University now as coauthors.
DB