Download Convergence Modes and Limit Theorems

THE UNIVERSITY OF HONG KONG
DEPARTMENT OF STATISTICS AND ACTUARIAL SCIENCE
STAT1301 PROBABILITY AND STATISTICS I, FALL 2010
EXAMPLE CLASS 10
Convergence Modes and Limit Theorems
Elements of Theory
1. Infinite Sequence of random variables and Its Partial Sum Sequence
+. An infinite sequence of random variables is an infinite sequence
Infinite sequence of random variables : *
+ where
of functions. The partial sum sequence associated with the infinite sequence : *
∑
is called the -th partial sum of the sequence *
functions.
+. The partial sum sequence is also an infinite sequence of
2. The Three Modes of Probabilistic Convergence
(
)
( )
( )
Special Notations:
→
where
should be replaced by one of the following symbols or abbreviations:
mode notation
meaning
(sure) convergence
→
almost surely, with probability 1
in probability
in law, in distribution, or weakly
→
mathematical definition
.{
}/
,|
|
( )
( )
Convergence in Probability to a constant and Convergence in Distribution to a constant (singleton)
for some constant .
3. Independent and Identically Distributed Sequence and Basic Statistical Notions
If the ’s are independent and identically distributed as a same distribution (known or unknown), then *
is called an independent and identically distributed (i.i.d.) sequence of random variables.
+
An iid collection of random variables is the prime device of statistics. In the population-sample language of statistics,

the
collection *
+ is called a sampling frame of size ;


+, is called a sample;
the realized values of the collection, denoted as *
the common distribution that all ’s follow is called the population.
The fundamental problem of statistics is to infer properties of the unknown population
+ obtained under the sampling frame *
properties of the many samples *
The sample mean ̅ is a random variable based on the sampling frame *
by investigating the
+.
+, defined as
̅
The sample mean depends on the size . As the size increases, we obtain a sequence of sample means vividly
depicted as the “running sample mean”. The Laws of large numbers concern the convergence of the running sample
mean. The Central limit theorem concerns the limit distribution of the (standardized) sample mean. All under the iid
condition.
4. Weak and Strong Laws of Large Numbers (for iid r.v. sequence)
Both versions of LLN state that the limit of the rolling sample mean is the population expectation.


Weak Law of Large Numbers: Running Sample Mean Converges to Population Expectation with high probability.
Strong Law of Large Numbers: Running Sample Mean Converges to Population Expectation with probability 1
(almost surely).
The Strong Law implies the Weak Law because a.s. convergence implies in-probability convergence. The Weak law
nevertheless occupies important position in probability and statistics and can be called “the fundamental theorem of
statistics” for the reason next:The difference between the two laws could be heuristically understood as how long one should wait to see the effect
of the respective law to manifest: After fixing an , you need to wait much longer to see the strong law effect than to
see the weak law effect.
Formal representation of the Laws of Large Numbers
+ be an i.i.d. sequence of random variables. The requirement ( )
(
Let *
Now the condition is enough for us to summon both versions of the law of large numbers:
) is necessary.
Weak version
(*| ̅
̅
+)
|
Remark: The Weak Law is about convergence in probability to the constant , thus it is equivalent to convergence in
distribution to that constant (singleton).
Strong version
.{
̅
}/
̅ →
5. Central Limit Theorem (for iid r.v. sequence)
CLT states that the standardized rolling sample mean’s limiting distribution is the standard normal distribution.
Formal representation of CLT
Let *
(
+ be an i.i.d. sequence of random variables following any distribution. The conditions of ( )
) and ( )
are enough to summon the CLT:
̅
(
)
√
Problems
.
1. (December 2009 Exam Q4) Let
(a) Compute (
) and
(b) What distribution does
(c) Compute ( ) and
(d) For
(
/. Then
converges in distribution to ; i.e.,
.
)? [5 marks]
have? (For full credit, be sure to provide the values of all parameters.) [5 marks]
( ) [5 marks]
relatively large, give an approximation for (
) that does not depend on . [5 marks]
2. Markov’s and Chebyshev’s Inequalities and WLLN
1) An archer is aiming at a circular target of radius 20 inches. Her arrows hit on average 5 inches away from the
center, each shot being independent. Show that the next arrow will miss the target with probability at most .
2) Let
be a sequence of iid random variables and assume the population expectation and variance are
finite and equals to and , respectively. Prove WLLN using the Chebyshev’s Inequality, that is, show the following:
3) A fair coin is tossed independently
times. Let
be the number of heads obtained. Use the Chebyshev
inequality to find a lower bound of the probability that
, and (iii)
differs from by less than
when (i)
, (ii)
.
3. True or False
a) Convergence in distribution concerns the convergence of a function sequence while almost sure convergence
concerns only the convergence of a point sequence.
b) Almost sure Convergence is also known as almost everywhere convergence.
c) Convergence in distribution implies almost sure convergence.
d) Convergence in probability implies convergence in distribution.
e) Convergence in distribution to a constant implies convergence in probability to that same constant.
f) If a sequence of functions does not converge, the sequence must not converge almost surely.
g) If a sequence of functions does not converge in distribution, the sequence must not converge.
h) Both the Strong Law of Large Numbers and the Central Limit Theorem imply the Weak Law of Large Number.
i) In most practical repetitive random phenomena, the effect of the Weak Law of Large Numbers manifests much
earlier than does the effect of the Strong Law of Large Numbers.
j) The Central Limit Theorem is a consequence of the de Moivre-Laplace theorem; this means the central limit
theorem describes a nature of a sequence of Bernoulli trials.
k) The Central Limit Theorem is a theorem concerning the limiting distribution of iid sum.
l) When the population variance is finite, we can use Markov’s inequality to prove the Weak Law of Large Numbers
and in that case the independence requirement for the sequence cannot be relaxed to the requirement of pairwisely
uncorrelated sequence.
m) The Poisson distribution cannot be the distribution of any iid sum and therefore we cannot use the Normal
distribution to approximate a Poisson distribution.
n) We can use the Normal distribution to approximate a Gamma distribution; this is directly because the Gamma
distribution is independently additive in its first parameter.
o) The Gamma distribution is the inter-arrival time distribution of a Poisson process. The exponential distribution is
the waiting time distribution of a Poisson process.
p) We can use the Normal distribution to approximate a Negative Binomial distribution because the Negative
Binomial distribution can be expressed as the iid sum of Geometric distribution.
q) The
4. CLT
distribution equals in distribution the sum of independent
’s
WLLN
Show that CLT implies WLLN, under iid and finite population expectation and variance conditions.
5. CLT formal exercise
Demonstrate that
,
.{
}/
(
√
)
∫
6. Elementary Statistics
Estimate the probability that the number of heads in 10,000 independent tosses of a fair coin differs by less than 1%
from 5000.
What is the probability that the number of heads will be greater than 5100?
How many independent tosses of a fair coin are required for the probability that the average number of heads
differs from 0.5 by less than 1% to be at least 0.99?