Download Independence of random variables

Law of Large Numbers
• Toss a coin n times.
• Suppose

1
Xi  

0
if i th toss came up H
if i th toss came up T
• Xi’s are Bernoulli random variables with p = ½ and E(Xi) = ½.
1 n
• The proportion of heads is X n   X i .
n i 1
• Intuitively X n approaches ½ as n  ∞ .
week 12
1
Markov’s Inequality
• If X is a non-negative random variable with E(X) < ∞ and a >0 then,
P X  a  
EX 
a
week 12
2
Chebyshev’s Inequality
• For a random variable X with E(X) < ∞ and V(X) < ∞, for any a >0
P X  E  X   a  
V X 
a2
• Proof:
week 12
3
Back to the Law of Large Numbers
• Interested in sequence of random variables X1, X2, X3,… such that the
random variables are independent and identically distributed (i.i.d).
Let
1 n
Xn   Xi
n i 1
Suppose E(Xi) = μ , V(Xi) = σ2, then
1 n
 1 n
E X n   E   X i    E  X i   
 n i 1  n i 1
and
1 n
 1
V X n   V   X i   2
 n i 1  n
n
V  X  
i 1
i
2
n
• Intuitively, as n  ∞, V  X n   0 so X n  E  X n   
week 12
4
• Formally, the Weak Law of Large Numbers (WLLN) states the following:
• Suppose X1, X2, X3,…are i.i.d with E(Xi) = μ < ∞ , V(Xi) = σ2 < ∞, then for
any positive number a


P Xn    a  0
as n  ∞ .
This is called Convergence in Probability.
Proof:
week 12
5
Example
• Flip a coin 10,000 times. Let

1
Xi  

0
if i th toss came up H
if i th toss came up T
• E(Xi) = ½ and V(Xi) = ¼ .
• Take a = 0.01, then by Chebyshev’s Inequality


1
1
1
1
P X n   0.01 


2
2
4

 410,000 0.01
• Chebyshev Inequality gives a very weak upper bound.
• Chebyshev Inequality works regardless of the distribution of the Xi’s.
week 12
6
Strong Law of Large Number
• Suppose X1, X2, X3,…are i.i.d with E(Xi) = μ < ∞ , then X n converges to μ
as n  ∞ with probability 1. That is
1


P lim  X 1  X 2    X n      1
 n n

• This is called convergence almost surely.
week 12
7
Continuity Theorem for MGFs
• Let X be a random variable such that for some t0 > 0 we have mX(t) < ∞ for
t   t0 , t0  . Further, if X1, X2,…is a sequence of random variables with
m X n t    and lim m X n t   m X t  for all t   t0 , t0 
n 
then {Xn} converges in distribution to X.
• This theorem can also be stated as follows:
Let Fn be a sequence of cdfs with corresponding mgf mn. Let F be a cdf with
mgf m. If mn(t)  m(t) for all t in an open interval containing zero, then
Fn(x)  F(x) at all continuity points of F.
• Example:
Poisson distribution can be approximated by a Normal distribution for large λ.
week 12
8
Example to illustrate the Continuity Theorem
• Let λ1, λ2,…be an increasing sequence with λn ∞ as n  ∞ and let {Xi} be
a sequence of Poisson random variables with the corresponding parameters.
We know that E(Xn) = λn = V(Xn).
X  E  X n  X n  n
• Let Z n  n
then we have that E(Zn) = 0, V(Zn) = 1.

V X n 
n
• We can show that the mgf of Zn is the mgf of a Standard Normal random
variable.
• We say that Zn convergence in distribution to Z ~ N(0,1).
week 12
9
Example
• Suppose X is Poisson(900) random variable. Find P(X > 950).
week 12
10
Central Limit Theorem
• The central limit theorem is concerned with the limiting property of sums
of random variables.
• If X1, X2,…is a sequencen of i.i.d random variables with mean μ and
variance σ2 and , S   X
n
i 1
i
then by the WLLN we have that
Sn
  in probability.
n
• The CLT concerned not just with the fact of convergence but how Sn /n
fluctuates around μ.
• Note that E(Sn) = nμ and V(Sn) = nσ2. The standardized version of Sn is
S n  n
Zn 
and we have that E(Zn) = 0, V(Zn) = 1.
 n
week 12
11
The Central Limit Theorem
• Let X1, X2,…be a sequence of i.i.d random variables with E(Xi) = μ < ∞
and Var(Xi) = σ2 < ∞. Suppose the common distribution function FX(x) and
the common moment generating function mX(t) are defined in a
neighborhood of 0. Let
n
Sn   X i
i 1
Then, lim P S n  n  x   x  for - ∞ < x < ∞
n
  n

where Ф(x) is the cdf for the standard normal distribution.
• This is equivalent to saying that Z n  S n  n converges in distribution to
 n
Z ~ N(0,1).
•
 Xn  


P
 x     x 
Also, lim
n   
n


i.e. Z n  X n   converges in distribution to Z ~ N(0,1).
 n
week 12
12
Example
• Suppose X1, X2,…are i.i.d random variables and each has the Poisson(3)
distribution. So E(Xi) = V(Xi) = 3.


• The CLT says that P X 1    X n  3n  x 3n  x as n  ∞.
week 12
13
Examples
• A very common application of the CLT is the Normal approximation to the
Binomial distribution.
• Suppose X1, X2,…are i.i.d random variables and each has the Bernoulli(p)
distribution. So E(Xi) = p and V(Xi) = p(1- p).


• The CLT says that P X 1    X n  np  x np1  p  x as n  ∞.
• Let Yn = X1 + … + Xn then Yn has a Binomial(n, p) distribution.

So for large n, PYn  y   P Yn  np

 np1  p 

 y  np 
y  np 

 



np1  p  
 np1  p  
• Suppose we flip a biased coin 1000 times and the probability of heads on
any one toss is 0.6. Find the probability of getting at least 550 heads.
• Suppose we toss a coin 100 times and observed 60 heads. Is the coin fair?
week 12
14