Download Central Limit Theorem

BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
So what exactly is special about sample averages?
Let’s play a hypothetical game. Let’s suppose that the population is genuinely normal,
with mean μ = 400 and standard deviation σ = 80. We are playing this as hypothetical,
so that we use the actual values 400 and 80 when setting up the population but we
recognize that the person taking the sample might have no idea about μ and σ.
Here is a graph that represents the density of the population:
Distribution Plot
Normal, Mean=400, StDev=80
0.005
Density
0.004
0.003
0.002
0.001
0.000
100
200
300
400
X
500
600
700
Suppose that we took a sample of 25 values. It might look like this:
342
365
491
546
513
439
341
362
381
359
457
324
434
302
418
373
221
531
390
349
460
441
427
417
621
The values here came as rounded to the nearest integer.
This sample has x = 412.2 and s = 86.5. These are reasonably close to the true
population values, so it looks like a “good” sample. Of course, the person doing the
sampling has no idea whether it’s good or not, as there is no access to the true values of μ
and σ.
1
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
The histogram for this sample suggests that it’s well-behaved.
Histogram of Sample1
7
6
Frequency
5
4
3
2
1
0
200
300
400
Sample1
500
600
Here’s another sample of 25:
452
397
383
389
369
538
415
504
322
423
349
428
396
515
489
335
349
256
343
260
349
336
385
357
287
For this sample of 25, x = 385.0 and s = 74.1. The histogram also looks reasonable:
Histogram of Sample2
6
Frequency
5
4
3
2
1
0
240
280
320
360
400
Sample2
440
480
520
2
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
We can do this indefinitely, drawing sample after sample. It is possible to show
mathematically that
E X =μ
SD( X ) =
(here μ is 400)
σ
=
n
σ
σ
=
5
25
(here σ is 80, so
σ
is 16)
5
This population was set up as normal, so the distribution of X is exactly normal with
mean μ = 400 and standard deviation equal to 16.
Examine these two logical statements:
If
then
X1, X2, …, Xn is a sample from a population with mean μ and standard
deviation σ,
σ
E X = μ and SD( X ) =
.
n
X1, X2, …, Xn is a sample from a population with mean μ and standard
deviation σ,
and if the population distribution is normal,
σ
then E X = μ , SD( X ) =
, and the distribution of X is itself normal.
n
If
The second is more useful, as it has the more powerful conclusion. This document will
end with yet another logical statement. Check carefully!
3
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
In fact, we can play with this, since we are running simulations. Let’s generate 10,000
samples, each of size 25, and let’s compute X for each. Here is a histogram of those
10,000 sample averages (with the best-fitting normal curve superimposed):
Histogram of C3
Normal
Mean 400.2
StDev 15.88
N
10000
500
Frequency
400
300
200
100
0
352
368
384
400
C3
416
432
448
This is an excellent picture. The inset box from Minitab identifies the average of these
10,000 averages as 400.2, very close to the target value of 400. It also identifies the
standard deviation as 15.88, close to the target value of 16.
Please remember….we are playing a hypothetical game. No one would really draw
10,000 separate samples, each of size 25. That’s 250,000 values, and we would probably
just assemble them into a single very large sample.
4
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
Let’s now add a wrinkle to this problem. Suppose that the population is not exactly
normal – and in fact is badly non-normal. Below is the probability density for a
population with mean μ = 800 and standard deviation σ = 565.69.
Skewed population, mean = 800, standard deviation = 565.69
0.0009
0.0008
0.0007
Density
0.0006
0.0005
0.0004
0.0003
0.0002
0.0001
0.0000
0
500
1000
1500
2000
2500
3000
3500
X
Technical side note. These were actually generated as a gamma distribution
with shape parameter = 2 and scale parameter = 400. The mean is 2 × 400
= 800 and the standard deviation is 2 × 400 ≈ 565.69.
Let’s suppose that we took a sample of size 5 from this population, and that we let X
represent the average of these 5 values. We can promise that E X = 800 and SD( X )
565.69
=
≈ 252.98. We can make no promise about the distribution of X , however.
5
Technical side note, continued. The distribution of X is gamma with shape
parameter = 10 and scale parameter = 80.
5
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
Here is a picture of 10,000 tries at drawing a sample of 5. The histogram shows values of
the sample averages.
Averages of 5 from previous skewed distribution
Mean 794.0
StDev 251.5
N
10000
400
Frequency
300
200
100
0
250
500
750
1000
1250
GammaAveOf5
1500
1750
2000
The curve is an approximating normal. It’s a bad fit to normal, but somehow the
averages of 5 are not as badly behaved as averages of originals.
Suppose we did samples of size 10 and let X be the average of 10. This will have
565.69
E X = 800 and SD( X ) =
≈ 178.89. Here is a plot of the averages from 10,000
10
samples of size 10:
Averages of 10 from previous skewed distribution
500
Mean 798.8
StDev 178.2
N
10000
Frequency
400
300
200
100
0
360
540
720
900
1080
GammaAveOf10
1260
1440
The normal approximating curve here (for averages of samples of 10) looks a little closer
to the normal approximating curve than for the previous (for averages of samples of
size 5). Notice too that the horizontal axis is more compact, reflecting the lower
standard deviation.
6
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
So . . . what if we took larger samples? Let’s try samples of size 50. Here is a repeat of
the previous.
Average of 50 from previous skewed distribution
Mean 800.1
StDev 79.37
N
10000
500
Frequency
400
300
200
100
0
560
640
720
800
880
GammaAveOf50
This should have mean 800 and standard deviation
960
1040
1120
565.69
≈ 80. The fit to the
50
approximating normal curve is excellent.
What we’ve seen here, in a completely empirical way, is that averages from larger
samples seem to have distributions that are closer to normal. In the case shown here,
averages of 50 look closer to normally distributed than averages of 10
averages of 10 look closer to normally distributed than averages of 5
This can in fact be mathematically proved. It’s called the Central Limit theorem, one of
the most important results in statistics.
Here is that statement:
X1, X2, …, Xn is a sample from a population with mean μ and standard
deviation σ,
and if the sample size is large,
σ
then E X = μ , SD( X ) =
, and the distribution of X is itself
n
approximately normal.
If
7
a a
©gs 2011
BEHAVIOR OF SAMPLE AVERAGES . . . CENTRAL LIMIT THEOREM
a a a a a a a a a a a a a a a a a a a a a a a a a a
This leaves open exactly what we mean by “sample size is large.” Most practitioners
will invoke the rule n ≥ 30. If the original population was only mildly non-normal (say
not badly skewed), then the Central Limit theorem should work with smaller n, perhaps
down to 10. For even badly non-normal populations, like the one used here, the cutoff of
30 is quite reasonable.
The example here certainly had bad skewness; the values were positive and the
standard deviation of 565.69 was almost as large as the mean 800.
We also should mention what we want to mean by “the distribution of X is itself
approximately normal.” This decision hangs on the level of precision that we want. If
a true probability of 0.4248 is approximated as 0.4189, we would not be upset.
However, approximating a true probability of 0.0014 by 0.0031 would be a disaster. If
this were a disease probability, we’d be doubling a risk!
Of course, we never get to know the true probability; we only have the approximation.
The troubles come in answers that are very close to zero or to 1. Thus, we should worry
about the Central Limit theorem approximation in cases where our calculations produce
values like 0.01 or 0.99. It’s also hard to operationalize the “worry” notion.
8
a a
©gs 2011