Download Sampling Distributions and the Central Limit Theorem

Sampling Distributions and the Central Limit
Theorem
Dr. Tai-kuang Ho, National Tsing Hua University
The slides draw from the textbooks of Wackerly, Mendenhall, and
Schea¤er (2008) & Devore and Berk (2012)
1
7.1
Introduction
We will be working with functions of the variables Y1; Y2; : : : ; Yn observed in a
random sample selected from a population of interest.
The random variables Y1; Y2; : : : ; Yn are independent and have the same distribution.
Certain function of the random variables are used to estimate unknown population parameters.
Population mean
2
A random sample of n observations y1; y2; : : : ; yn
It seems reasonable to estimate
with the sample mean
n
1X
y=
yi
n i=1
y depends on Y1; Y2; : : : ; Yn and the function form.
y is an example of a statistic.
3
DEFINITION: statistic
A statistic is a function of the observable random variables in a sample and
known constants.
Examples of statistic
sample mean: Y
sample variance: S 2
maximum: max (Y1; Y2; : : : ; Yn)
4
minimum: min (Y1; Y2; : : : ; Yn)
range: R = max (Y1; Y2; : : : ; Yn)
min (Y1; Y2; : : : ; Yn)
Statistics are used to make inferences about unknown population parameters.
Because all statistics are functions of the random variables observed in a sample,
all statistics are random variables.
Consequently, all statistics have probability distributions, which we will call their
sampling distributions.
5
The sampling distribution of a statistic provides a theoretical model for the relative frequency histogram of the possible values of the statistic that we would
observe through repeated sampling.
Example 7.1
A balanced die is tossed three times.
Y1; Y2; Y3
Y = 13 (Y1 + Y2 + Y3)
6
Y
E Y
= 13 (EY1 + EY2 + EY3) = 31 (3 3:5) = 3:5
V ar Y
0:9722
= V ar 13 (Y1 + Y2 + Y3)
= 91 (3 V ar (Y1)) = 13
1
P Y = 33 = p (1; 1; 1) = 6 61 6 = 216
3
P Y = 43 = p (1; 1; 2) + p (1; 2; 1) + p (2; 1; 1) = 216
7
2:9167 =
...
P Y = 18
3 =
Figure 7.1
Program to simulate Figure 7.1, Eviews
’ simulate.figure.7.prg
create u 1 4000
8
series ergib=na
for !i=1 to 4000
scalar toss1=@floor(@runif(0,6))+1
scalar toss2=@floor(@runif(0,6))+1
scalar toss3=@floor(@runif(0,6))+1
ergib(!i)=(toss1+toss2+toss3)/3
9
next
Before we start the introduction of central limit theorem, we will introduction
some additional distributions.
7.2
Sampling Distributions Related to the Normal Distribution
We start from the case where there is an exact expression for the sampling
distribution (for the statistics).
Usually, this comes from speci…c assumption about the random variables Y1; Y2; : : : ; Yn,
such as normal distribution.
10
In most application, obtaining exact expression for the sampling distribution is
not possible; therefore we have to resort to central limit theorem.
In many applied problems, it is reasonable to assume that the observable random
variables in a random sample Y1; Y2; : : : ; Yn are independent with the same
normal density function.
THEOREM: normal sampling distribution
Let Y1; Y2; :::; Yn be a random sample of size n from a normal distribution with mean
and variance 2. Then
11
n
1X
Yi
Y =
n i=1
and variance 2 = 2=n.
is normally distributed with mean Y =
Y
Proof
n
Y1 Y2
1X
Y =
Yi =
+
+
n i=1
n
n
E (Yi) =
12
Yn
+
n
V (Yi) = 2
E Y
0
n
X
1
1
Y1 Y2
Yn
YiA = E
+
+
+
n i=1
n
n
n
Y1
Y2
1
Yn
= E
+E
+
+E
=
n
n
n
n
n
= E@
13
=
Y1 Y2
Yn
+
+
+
n
n
n
1
1
=
V
(
Y
+
V (Y2) +
)
1
2
2
n
n
2
1
2
n
=
=
2
n
n
= V
V Y
1
+ 2 V (Yn)
n
Notice that the variance of each of the random variable Y1; Y2; : : : ; Yn is 2 and
2
the variance of the sampling distribution of the random variable Y is n .
Y
N
2 ;
;
Y
Y
2
Y
Y = ;
14
= 2=n
Normalization
Z has a standard normal distribution.
Z=
Y
Y
Y
=
Y
p
=
n
p
n
Y
!
This theorem provides the basis for development of inference procedures about
the mean of a normal distribution with known variance 2.
p
n(Y
)
N (0; 1)
15
Example 7.2
Y1; Y2; : : : ; Y9
Yi
Y
N
N
; 2=1
2
; 9 = 19
P
0:3 =?
Y
16
P
Y
0:3
= P
0
0:3
0:3
= P@
0
B
= P@
p
17
p
n
Y
p1
9
= P ( 0:9
Example 7.3
Y
n
0:3
0:3
Y
p1
9
Z
0:9)
1
0:3 A
p
n
1
0:3 C
A
p1
9
P
0:3 = 0:95
Y
n =?
P
Y
0:3
= P
0
= P@
= P
0:3
0:3
p
n
p
0:3 n
18
0:3
Y
Y
p
n
Z
1
0:3 A
p
n
p
0:3 n = 0:95
P ( 1:96
1:96) = 0:95
Z
p
0:3 n = 1:96
THEOREM: 2 distribution
Let Y1; Y2; :::; Yn be de…ned as in the above theorem. Then Zi =
independent, standard normal random variables,i = 1; 2; :::; n, and
19
(Yi
)
are
n
X
i=1
Zi2 =
n
X
Yi
i=1
has a 2 distribution with n degrees of freedom (df).
Proof
Figure 7.2
P
2>
2
=
20
2
P
2
2
2
is the (1
=1
) quantile of the 2 random variable.
Example 7.4
The 2 distribution is important in the inference about population variance.
Suppose we wish to make an inference about the population variance 2 based
on a random sample Y1; Y2; : : : ; Yn from a normal distribution.
A good estimator of 2 is the sample variance:
21
S2
=
1
n
X
1 i=1
n
Yi
Y
2
THEOREM: use of 2 distribution
Let Y1; Y2; :::; Yn be a random sample from a normal distribution with mean
variance 2. Then
(n
1)S 2
2
n
1 X
= 2
22
i=1
(Yi
Y )2
and
has a 2 distribution with (n
variables.
We know that
p
n(Y
)
1) df. Also, Y and S 2 are independent random
N (0; 1).
When 2 is known, we can use
.
p
n(Y
)
N (0; 1) to make inference about
When 2 is unknown, it can be estimated by S 2.
We use the quantity
p
n(Y
S
)
to make inference about .
23
What is the distribution of
!
p
n(Y
S
)
t (n
p
n(Y
S
)
?
1)
DEFINITION: Student’s t distribution
Let Z be a standard normal random variable and let W be a 2-distributed variable
with v df. Then, if Z and W are independent,
T =q
is said to have a t distribution with v df.
24
Z
W=v
Some properties
Y1; Y2; : : : ; Yn
Z = Yp
W =
N
; 2
N (0; 1)
n
(n 1)S 2
2
2 (n
1)
Z and W are independent (because Y and S 2 are independent).
According to the above de…nition
25
Y
T =q
Z
W=v
=r
p
(n
n
=
1)S 2
= (n
2
p
n
1)
Figure 7.3
Standard normal distribution and t distribution.
E (T ) = 0;
V (T ) = v v 2 ;
v>1
v>2
26
Y
S
!
t (n
1)
Example 7.6
Y1; Y2; : : : ; Y6
N
=?; 2 =?
P
2S
p
n
Y
27
!
=?
P
Y
2S
p
n
!
= P
0
2S
p
n
Y
B
= P@ 2
pS
n
= P( 2
P
Y
28
n
!
=?
1
C
2A
T (df = 5)
If 2 were known,
2
p
2S
p
n
Y
2)
!
P
Y
2
p
n
!
= P
0
2
p
= P@ 2
= P( 2
2
p
Y
n
Y
p
Z
n
2)
1
n
!
2A
Suppose we want to compare the variances of two normal populations based on
information contained in independent random samples from the two populations.
2 ! n ! S2
1
1
1
29
2 ! n ! S2
2
2
2
S12
!
S22
S12 =
S22 =
2
1
2
2
2
1
2
2
F (n1
1; n2
1)
DEFINITION: F distribution
Let W1 and W2 be independent 2-distributed random variables with v1 and v2 df,
respectively. Then
30
W1=v1
F =
W2=v2
is said to have an F distribution with v1 numerator degrees of freedom and v2
denominator degrees of freedom.
Figure 7.4
E (F ) = v v2 2 ;
2
V (F ) =
v2 > 2
2v22 (v1 +v2 2)
;
v1 (v2 2)2 (v2 4)
v2 > 4
31
W1 =
W2 =
(n1 1)S12
2
1
(n2 1)S22
2
2
(n1 1)S12
= (n1
2
W =v
1
F = 1 1=
(n2 1)S22
W2=v2
= (n2
2
1)
S12=
= 2
S2 =
1)
2
Example 7.7
32
2
1
2
2
F (n1
1; n2
1)
(Y1; Y2; : : : ; Y6) ! n1 = 6
(Y1; Y2; : : : ; Y10) ! n2 = 10
2
1
= 22
P
P
S12
S22
S12
S22
2
2
2
1
!
b = 0:95
2
b 22
1
33
!
= 0:95
P [F (5; 9)
b] = 0:95
Summary
If Y1; Y2; : : : ; Yn are a random sample from a normal distribution with mean
and variance 2.
p
n(Y
(n 1)S 2
2
)
N (0; 1)
2 (n
1)
34
p
n(Y
S
S12 =
S22 =
2
1
2
2
)
t (n
F (n1
1)
1; n2
1)
These sampling distributions will enable us to evaluate the properties of inferential procedures in later chapters.
35
7.3
The Central Limit Theorem
Y1; Y2; : : : ; Yn represents a random sample from any distribution with mean
and variance 2.
E Y
= ;
V Y
2
= n
This tells us only the mean and variance of Y , but the distribution of Y is
unknown.
If Y is normally distributed, then Y is also normally distributed.
36
If Y is not normally distributed, what is the distribution of Y ?
We will develop an approximation for the sampling distribution of Y that can
be used regardless of the distribution of the population from which the sample
is taken.
Fortunately, Y will have a sampling distribution that is approximately normal if
the sample size is large.
The formal statement of this result is called the central limit theorem.
An experiment
37
Generate a random sample of size n from an exponential density function with
mean 10 and variance 100.
f (y ) =
(
y
1 e 10
10
0
Figure 7.5
Figure 7.6, n = 5
Figure 7.7, n = 25
38
;
y>0
; elsewhere
Number of simulation is 1000.
Table 7.1 make sure that we have enough number of simulation.
THEOREM: Central Limit Theorem
Let Y1; Y2; :::; Yn be independent and identically distributed random variables with
E (Yi) = and V (Yi) = 2 < 1. De…ne
Pn
Yi
i=1
Un =
p
n
n
=
Y
p ;
= n
39
n
1X
Yi:
Y =
n i=1
Then the distribution function of Un converges to the standard normal distribution
function as n ! 1. That is,
lim P (Un
n!1
u) =
Z u
2
1
p e t =2dt;
1 2
f or all u:
DEFINITION: normal probability distribution
A random variable Y is said to have a normal probability distribution if and only if,
for > 0 and 1 < < 1, the density function of Y is
40
1
f (y ) = p exp
2
"
(y
)
2 2
2#
;
1<y<1
The theorem states that probability statement about Un can be approximated
by corresponding probabilities for the standard random normal variable if n is
large.
n
30
Equivalent statements:
41
Yp
= n
is asymptotically normally distributed with mean 0 and variance 1.
Y is asymptotically normally distributed with mean
2
and variance n .
The central limit theorem can be applied to a random sample Y1; Y2; : : : ; Yn
from any distribution as long as E (Yi) = and V (Yi) = 2 are both …nite
and the sample size is large.
Example 7.8
Y1; Y2; : : : ; Yn
42
= 60
2
= 64
n = 100
P Y
58 ?
P Y
58
0
60
BY
p8
100
= P@
' P (Z
43
2:5)
58
1
60 C
p8
100
A
7.4
A Proof of the Central Limit Theorem
Proof
Un =
Y
p
Pn
Yi
= i=1 p
= n
n
1 X
Zi;
= p
n i=1
n
n
Y
Zi = i
1
=p
n
mPn Zi (t) = mZ1 (t) mZ2 (t)
i=1
44
Pn
i=1 Yi
mZn (t)
n
!
Y = aX + b
mY (t) = ebt mX (at)
!
1
mUn (t) = mPn Zi p t
i=1
n
!
!
1
1
= mZ1 p t mZ2 p t
n
n
"
!#n
1
= mZ1 p t
n
45
!
1
mZn p t
n
00 ( )
m
Z1
mZ1 (t) = mZ1 (0) + m0Z1 (0) t +
t2;
2
0<
mZ1 (0) = 1
m0Z1 (0) = E (Z1) = 0
m00Z ( ) 2
1
t ;
mZ1 (t) = 1 +
2
46
0<
<t
<t
t
t!p
n
mUn (t) =
=
n!1)
"
"
!#n
1
p t
n
mZ1
1+
1 t2
n2
m00Z1 ( )
2
= 41 +
#n
;
m00Z (
1
0<
)
2
t
<p
n
!23n
t
5 ;
p
n
0<
t
<p
n
!0
2
2
2
! 0 ) t2 m00Z ( ) ! t2 m00Z (0) = t2 ;
1
1
47
* m00Z (0) = E Z12 = 1
1
lim bn = b ) lim
n!1
n!1
1+
bn n
n
= eb
lim mUn (t) = lim
n!1
mZ (t) = exp 12 t2
n!1
"
1+
1 t2
n2
m00Z1 (
N (0; 1)
lim mUn (t)
n!1
48
N (0; 1)
)
#n
=
t2
e2
7.5
The Normal Approximation of the Binomial Distribution
The central limit theorem can be used to approximate probabilities for some
discrete random variables when the exact probabilities are tedious to compute.
An example is binomial distribution
Y
P (Y
b (n; p)
b)
Method 1: use table and exact values
49
Method 2: approximation using CLT
Y =
n
X
Xi
i=1
Xi =
(
1 ; success
;
0 ; otherwise
E (Xi) = p
50
i = 0; 1; 2; : : : ; n
V (Xi) = p (1
Un =
Y
p
= n
Y
p
=
p
Y
p(1 p)
p
n
Y
N
p)
p;
n
X
=r
p(1 p)
n
p (1 p)
n
1
Y
Xi = X ! N
=
n
n i=1
51
p
N (0; 1)
!
p (1 p)
p;
n
!
Y ! N (np; np (1
52
p))