Download Markov, Chebyshev, and the Weak Law of Large Numbers

Markov, Chebyshev, and the Weak Law of Large Numbers
The Law of Large Numbers is one of the fundamental theorems of statistics. One
version of this theorem, The Weak Law of Large Numbers, can be proven in a fairly
straightforward manner using Chebyshev's Theorem, which is, in turn, a special case of the
Markov Inequality.
Markov's Inequality: If X is a random variable that takes only nonnegative values, then
E X
for any value c > 0 , P X ≥ c ≤
.
c
g b g
b
The discrete and continuous forms of this theorem are quite similar, varying only in
using sums
FG Eb X g = ∑ x pb x gIJ or integrals F Eb X g = z x pb xgI.
H
K
H
K
∞
i
i
−∞
i
Here, we give the
continuous form.
bg
If X is a continuous random variable with density function p x , then
z
b g
∞
E X =
0
bg
x p x dx .
We can break this interval into two pieces, those values of x less than or equal to c and
those values of x greater than or equal to c. So
b g
E X =
z
c
0
bg
x p x dx +
z
∞
c
bg
x p x dx .
If we take only those values of x greater than or equal to c, we have
z
b g
∞
E X ≥
c
bg
x p x dx .
Since the values of x in this integrand have the property that x ≥ c , if we replace x with c,
we have
b g
E X ≥
z
∞
c
bg
x p x dx ≥
z
∞
c
bg
c p x dx = c
z
∞
c
bg
p x dx .
This last integrand is just the probability that X is greater than or equal to c, so
b g
b
g
b
g EbcX g.
E X ≥ a P X ≥ c so P X ≥ c ≤
This means that for any random variable with nonnegative values which has a mean of 10,
10
the probability that X > 15 is less than or equal to
= 0.67 , regardless of the variance
15
of the random variable.
Chebyshev's Theorem: Let X be a random variable with mean µ and standard deviation
c
h
σ . Let k be any positive number. Then P X − µ > k <
σ2
.
k2
b
nonnegative random variable and apply the inequality to E X − µ
b X − µg
2
Now that we have Markov's Inequality, we recognize that
g
2
is a
with c = k 2 .
Instead, we will use a similar argument to that used for the Markov Inequality, but
consider a discrete random variable.
By definition, we have
n
b
σ 2 = ∑ xi − µ
i =1
g pb x g.
2
i
If we delete some terms in the sum, the sum will get smaller. So, delete all terms for
which xi − µ ≤k . We will denote the terms remaining with x ∗ . Now sum only those
values x ∗ x ∗ where x ∗ − µ > k to find that
d
σ 2 ≥ ∑ x ∗j − µ
i pd x i .
2
∗
j
j
The subscript j is used to indicate that this summation may not be as extensive as the
summation on i. Since we are using only those values x ∗ where x ∗ − µ > k , if we
d
i
replace each x ∗j − µ with k, we have
d
σ 2 ≥ ∑ x ∗j − µ
i d i
2
p x ∗j >
j
∑
j
d i
d i
k 2 p x ∗j = k 2 ∑ p x ∗j and
j
σ2
>
k2
∑
d i
p x ∗j
j
σ2
As before, the summation ∑ p x is just P X − µ > k and so P X − µ > k < 2 .
k
j
Thus, for example, in any population with mean 10 and standard deviation 2, at most onefourth of the values will be more than 4 units from 10. Consequently, at least 75% of the
values will be between 6 and 14.
d i
∗
j
c
h
c
h
The Weak Form of the Law of Large Numbers: Let a population be specified by a
random variable X with mean µ X and standard deviation σ X . Let X be the mean of a
random sample of size n drawn with replacement from this population. Let c be any
positive number. Then lim P µ X − c ≤ X ≤µ X + c = 1.
n→ ∞
c
h
To prove this form of the Law of Large Numbers, invoke Chebyshev's Theorem on
the random variable X with mean µ X and standard deviation σ X . Thus,
σ 2X
.
c2
We need only rewrite the mean µ X and standard deviation σ X in terms of the population
mean µ X and population standard deviation σ X . We know that µ X = µ X and
d
i
P X − µX > c <
σ 2X =
σ 2X
. So we have
n
σ 2X
σ 2X
and
so
P
X
−
µ
≤
c
>
−
.
1
X
nc 2
nc 2
σ 2X
Thus, P µ X − c ≤ X ≤µ X + c is bound between 1 −
and 1. As n → ∞ , the
nc 2
probability approaches 1. So lim P µ X − c ≤ X ≤µ X + c = 1.
d
i
h
d
P X − µX > c <
c
n→ ∞
c
i
h
References:
Goldberg, Samuel, Probability: An Introduction, Dover Publications, New York, New
York, 1960.
Ross, Sheldon, A First Course in Probability, Macmillan Publishing Company, New
York, New York, 1976.