Download Convergence in Probability and Convergence in Distribution

STAT 516: Multivariate Distributions
Lecture 7: Convergence in Probability and Convergence in
Distribution
Prof. Michael Levine
November 12, 2015
Levine
STAT 516: Multivariate Distributions
Convergence in Probability
I
p
A sequence Xn → X (converges in probability to X ) if, for
any ε > 0
lim P(|Xn − X | ≥ ε) = 0
n→∞
I
In this context, X may be a constant a - a degenerate random
variable
I
Chebyshev’s inequality is a common way of showing
convergence in probability
Levine
STAT 516: Multivariate Distributions
Examples
1. Let Xn = X +
1
n
where X ∼ N(0, 1)
p
2. Easy to verify that (by Chebyshev’s inequality) Xn → X
1. For {Xn } s.t. the mean µ and variance σ 2 are finite
p
X̄n → µ
2. The weak law of large numbers - second moment must exist;
strong law does not require that - will not be proved in this
course
Levine
STAT 516: Multivariate Distributions
Convergence in probability is closed under linearity
p
p
p
I
Xn → X and Yn → Y implies Xn + Yn → X + Y
I
If a is a constant and Xn → X aXn → aX
I
Conclusion: convergence in probability is closed under linearity
p
Levine
p
STAT 516: Multivariate Distributions
Continuous Mapping Theorem for Convergence in
Probability
p
p
I
If g is a continuous function, Xn → X then g (Xn ) → g (X )
I
We only prove a more limited version: if, for some constant a,
p
g (x) is continuous at a, g (Xn ) → g (a)
I
Can be viewed as one of the statements of Slutsky theorem the full theorem to be stated later
Levine
STAT 516: Multivariate Distributions
Another useful property
p
p
p
I
If Xn → X and Yn → Y , Xn Yn → XY
I
Only prove in this form but can be generalized to
p
g (Xn , Yn ) → g (X , Y )
Levine
STAT 516: Multivariate Distributions
Consistency and convergence in probability
I
For X ∼ F (x; θ), θ ∈ Ω a statistic Tn is a consistent
estimator of θ if
p
Tn → θ
I
I
Weak Law of Large Numbers → X̄ is a consistent estimator of
µ
1 Pn
2
2 p
A sample variance S 2 = n−1
i=1 (Xi − X̄ ) → σ
I
By continuous mapping theorem S → σ
p
Levine
STAT 516: Multivariate Distributions
Example
I
I
I
I
Let X1 , . . . , Xn ∼ Unif [0, 1] and Yn = max{X1 , . . . , Xn }
n
The cdf of Yn is FYn (t) = θt for 0 < t ≤ θ
n
θ - Yn is a biased estimator
Check that EYn = n+1
Direct computation implies that
Yn is consistent...and so is
the unbiased estimator n+1
Yn
n
Levine
STAT 516: Multivariate Distributions
Convergence in Distribution
I
If for a sequence {Xn } with cdf FXn (x) , and a random
variable X ∼ FX (x)
lim FXn (x) = FX (x)
n→∞
D
for all points of continuity of FX (x), Xn → X - Xn converges
in distribution or in law to X
I
We say that FX (x) is the asymptotic distribution or the
limiting distribution of Xn
I
Occasional abuse of notation: Xn → N(0, 1)
Levine
STAT 516: Multivariate Distributions
Convergence in distribution and convergence in probability
I
Convergence in distribution is only concerned with
distributions and not at all with random variables
I
For a symmetric fX (x), X and −X have the same distribution
I
Let
Xn =
X
−X
if n is odd
if n is even
D
I
Clearly, Xn → X but there is no convergence in probability!
Levine
STAT 516: Multivariate Distributions
Example
I
Let X̄ ∼ N(0, σ 2 /n)
I
Check that
I
Conclude that Fn (x̄) converges to the point mass at zero

 0 x̄ < 0
1
lim Fn (x̄) =
x̄ = 0
n→∞
 2
1 x̄ > 0
Levine
STAT 516: Multivariate Distributions
Example
I
Convergence of pdfs/pmfs does NOT mean convergence in
distribution!
I
Define the pmf
pn (x) =
1
0
x = 2 + n1
elsewhere
I
limn→∞ pn (x) = 0 for any x
I
However, the limiting function of cdf’s is F (x) = 0 if x < 2
and F (x) = 1 if x ≥ 2 which is a cdf!
I
Convergence in distribution does take place!
Levine
STAT 516: Multivariate Distributions
Example
I
However...for Tn ∼ tn we have
Z t
Γ[(n + 12 )]
1
dy
Fn (t) =
√
n
2 /n)(n+1)/2
(1
+
y
πnΓ
−∞
2
I
Stirling’s formula:
Γ(k + 1) ≈
√
2πk k+1/2 exp (−k)
I
The limit under the sign of integral is the normal pdf...so
Z t
1
√ exp (−y 2 /2) dy
lim Fn (t) =
n→∞
2π
−∞
I
The limiting distribution of tn is N(0, 1)
Levine
STAT 516: Multivariate Distributions
Example
I
Recall that for X1 , . . . , Xn ∼ Unif [0, θ] Yn = max1≤i≤n is the
consistent estimator of θ
I
Now we can say more...let Zn = n(θ − Yn )
I
For any t ∈ (0, nθ)
t/θ n
P(Zn ≤ t) = P(Yn ≥ θ − (t/θ)) = 1 − 1 −
n
I
Since limn→∞ P(Zn ≤ t) = 1 − exp (−t/θ) for some
D
Z ∼ exp(θ) Zn → Z
Levine
STAT 516: Multivariate Distributions
Relationship between convergence in probability and
convergence in distribution
p
D
I
If Xn → X , Xn → X
I
The converse is not true in general - see an earlier example!
I
However, if for a constant b Xn → b it also true that Xn → b
D
Levine
STAT 516: Multivariate Distributions
p
Basic properties of convergence in distribution
D
p
D
I
If Xn → X and Yn → 0, Xn + Yn → X
I
This is the magic wand if it is hard to show that Xn → X but
p
D
easy to show that some other Yn → X and Xn − Yn → 0
I
If Xn → X and g (x) is a continuous function on the support
of X ,
D
g (Xn ) → g (X )
I
If a and b are constants, Xn → X , An → a, and Bn → b,
D
D
p
D
p
D
An + Bn Xn → a + bX
Levine
STAT 516: Multivariate Distributions
Boundedness in probability
I
For any X ∼ FX (x), we can always find η1 and η2 s.t.
FX (x) < ε/2 for x ≤ η1 and FX (x) > 1 − ε/2 for x ≥ η2
I
Thus, for η = max{|η1 |, |η2 |}
P[|X | ≤ η] ≥ 1 − ε
I
Formal definition: {Xn } is bounded in probability if for any
ε > 0 there exist Bε > 0 and an integer Nε s.t. if n ≥ Nε
P[Xn ≤ Bε ] ≥ 1 − ε
I
Can show immediately that if Xn → X then {Xn } is bounded
in probability...the converse is not always true
D
Levine
STAT 516: Multivariate Distributions
Why a sequence that is bounded in probability may not
converge
1
2m
and X2m−1 = 1 +
1
2m
w.p.1
I
Define X2m = 2 +
I
All of the mass of this sequence is concentrated in [1, 2.5] and
so it is bounded in probability
I
Xn consists of two subsequences that converge to degenerate
RV’s Y = 2 and W = 1 in distribution
Levine
STAT 516: Multivariate Distributions
A useful property
I
Xn a sequence of random variables bounded in prob. and Yn a
sequence that converges to zero in probability
I
Then, Xn Yn → 0
I
Analog from the world of calculus: if A is a constant and
ξn = n1 then limn→∞ n1 = 0 and limn→∞ An = 0
p
Levine
STAT 516: Multivariate Distributions
MGF technique
I
If Xn has mgf MXn (t) for |t| ≤ h, X has mgf MX (t) for
|t| ≤ h1 < h, and limn→∞ MXn (t) = M(t) for |t| ≤ h1 , then
D
Xn → X
I
I
Take Yn ∼ b(n, p) with fixed µ = np for every n
h
in
t
Check that MYn (t) = [(1 − p) + pe t ]n = 1 + µ(e n−1)
Levine
STAT 516: Multivariate Distributions
Poisson approximation of the binomial: an example
I
1
Y ∼ b 50, 25
;
I
P(Y ≤ 1) =
I
Since µ = np = 2, we have the Poisson approximation
24 50
25
+ 50
1
25
24 49
25
= 0.4000
e −2 + 2e −2 = 0.406
Levine
STAT 516: Multivariate Distributions
Central Limit Theorem (CLT)
I
I
2
If X1 , . . . , Xn ∼ N(µ, σ 2 ) we know that X̄ ∼ N µ, σn
CLT: if X1 , . . . , Xn are independent, bE Xi = µ and
Var Xi = σ 2 , we have
√
n(X̄ − µ)
∼ N(0, 1)
σ
Levine
STAT 516: Multivariate Distributions