Download Statistical Foundations: The Normal Distribution The Central Limit

Statistical Foundations:
The Normal Distribution
The Central Limit Theorem
Other Fun Things...
Lecture 7
September 14, 2006
Psychology 790
Lecture #7 - 9/14/2006
Slide 1 of 32
Today’s Lecture
●
Overview
➤ Today’s Lecture
Homework questions?
✦
Any questions on homework #2?
Normal Distribution
●
The Normal Distribution (Chapter 6.1-6.6).
Assessing
Normality
●
The Central Limit Theorem (Chapter 6.7).
Central Limit
Theorem
●
Information about Maximum Likelihood Estimators.
Maximum
Likelihood
●
Biasedness and Unbiasedness.
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 2 of 32
Univariate Normal Distribution
●
The univariate normal distribution function is:
Overview
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
2
e
f (x; µ, σ ) = √
2
2πσ
●
Assessing
Normality
Bias
The mean sets the center of the distribution on the x axis.
The variance is σ 2 .
✦
Central Limit
Theorem
−(x−µ)2
2σ 2
The mean is µ.
✦
●
Maximum
Likelihood
1
The standard deviation is σ.
The variance sets the spread/dispersion/width of the
distribution.
Standard notation for normal distributions is N (µ, σ 2 ).
✦
●
✦
Hence, the letter N from Wonder Showzen.
Wrapping Up
Lecture #7 - 9/14/2006
Slide 3 of 32
Univariate Normal Distribution
N (0, 1)
Overview
Univariate Normal Distribution
0.2
f(x)
0.3
0.4
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
Central Limit
Theorem
0.1
Assessing
Normality
0.0
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
−6
−4
−2
0
2
4
6
x
Slide 4 of 32
Univariate Normal Distribution
N (0, 2)
Overview
Univariate Normal Distribution
0.2
f(x)
0.3
0.4
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
Central Limit
Theorem
0.1
Assessing
Normality
0.0
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
−6
−4
−2
0
2
4
6
x
Slide 5 of 32
Univariate Normal Distribution
N (3, 1)
Overview
Univariate Normal Distribution
0.2
f(x)
0.3
0.4
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
Central Limit
Theorem
0.1
Assessing
Normality
0.0
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
−6
−4
−2
0
2
4
6
x
Slide 6 of 32
Normal Distribution Notes
●
The area under the curve for the normal distribution is equal
to one (recall our probability lecture about P (S)).
●
Furthermore, we know that:
Overview
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
P (µ − σ ≤ X ≤ µ + σ) = 0.683
P (µ − 2σ ≤ X ≤ µ + 2σ) = 0.954
●
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
●
Bias
Also note the term in the exponent:
−(x − µ)
2σ 2
2
This is the square of the distance from x to µ in standard
deviation units.
Wrapping Up
Lecture #7 - 9/14/2006
Slide 7 of 32
Cumulative Normal Distribution
●
The cumulative normal distribution (denoted F (x)) gives the
probability a observation falls at or below x.
Overview
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
●
Assessing
Normality
Central Limit
Theorem
Wrapping Up
Lecture #7 - 9/14/2006
In probability notation: F (x) = p(a ≤ x)
✦
This probability is formed by taking the area under the
curve to the left of the point (found by integration).
For instance, the probability of finding a point less than or
equal to the mean would be given by:
Z µ
−(x−µ)2
1
√
F (µ) = p(a ≤ µ) =
e 2σ2 dx = 0.5
2πσ 2
∞
R
●
You can replace with what you find from a table in the book
(or the =normdist function in Excel).
●
Another name for the cumulative distribution is the
cumulative density function (abbreviated CDF).
Maximum
Likelihood
Bias
✦
Slide 8 of 32
Normal Distribution Functions
N (0, 1)
Overview
Probability Density Function
F(x)
0.2
f(x)
0.6
0.3
0.8
1.0
0.4
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
0.2
0.1
0.4
Assessing
Normality
Central Limit
Theorem
Cumulative Density Function
0.0
0.0
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
−4
−2
0
x
2
4
−4
−2
0
2
4
x
Slide 9 of 32
Normal Distribution Functions
N (0, 2)
Overview
0.4
Assessing
Normality
0.10
f(x)
F(x)
0.6
0.15
0.8
0.20
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
0.0
0.05
0.2
Central Limit
Theorem
Maximum
Likelihood
Cumulative Density Function
1.0
Probability Density Function
Bias
Wrapping Up
Lecture #7 - 9/14/2006
−4
−2
0
x
2
4
−4
−2
0
2
4
x
Slide 10 of 32
Normal Distribution Functions
N (3, 1)
Overview
Probability Density Function
F(x)
0.2
f(x)
0.6
0.3
0.8
1.0
0.4
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
0.2
0.1
0.4
Assessing
Normality
Central Limit
Theorem
Cumulative Density Function
Bias
Wrapping Up
Lecture #7 - 9/14/2006
0.0
0.0
Maximum
Likelihood
−4
−2
0
2
x
4
6
−4
−2
0
2
4
6
x
Slide 11 of 32
Finding Probabilities
●
We can use the Normal CDF to assess the probability an
observation falls within a given range.
●
For instance, imagine we had a distribution of variables that
we knew was N(0,1) - standard normal.
●
What is p(−0.75 ≤ 1.0)?
●
How would you figure that out?
Overview
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 12 of 32
More Normal Notes
●
The normal distribution is frequently used in mathematical
statistics because it is very flexible.
●
We will assume normality of many things in this course.
●
For instance, we will come to say the amount of error in
prediction we have in a regression will be normally
distributed.
Overview
Normal Distribution
➤ Normal
Distribution
➤ Normal
Distribution Notes
➤ Cumulative
Distribution
➤ Finding
Probabilities
➤ More Notes
Assessing
Normality
✦
●
This leads to hypothesis tests
Then there is the Central Limit Theorem...
Central Limit
Theorem
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 13 of 32
Assessing Normality
●
We will find that there are two ways to assess
normality/MVN.
Overview
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
Central Limit
Theorem
1. By comparing the distribution of your observations (or
some transformation of your observations) to some known
distribution. (These are commonly called Q-Q plots)
2. By computing some set of statistics and obtaining a
p-value (i.e., compute a statistic with a known distribution
and determine how extreme the statistic is compared to a
null hypothesis).
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 14 of 32
Assessing Univariate Normality
Overview
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
Central Limit
Theorem
There are situations where we would like to assess whether a
variable is normally distributed using a Q-Q plot.
● A Q-Q plot is a plot that matches the Quantiles of the
observed data with the Quantiles of a specific distribution.
●
✦
●
Maximum
Likelihood
Lecture #7 - 9/14/2006
●
For example the .5 quantile of a N(0,1) is 0.
In our case the Quantiles of a specific distribution will be a
normal, N (0, 1).
✦
Bias
Wrapping Up
A Quantile (commonly called a percentile) is that value such
that a specific proportion p of the population will score at or
below.
It could be a N (x̄, s2x ), if preferred.
There should be a linear relationship between the quantiles
of the observed data with their theoretical quantiles
(assuming the distribution) if they follow the same
distribution.
Slide 15 of 32
Constructing a Q-Q plot
Lets assume that we have n observations x1 , x2 , . . . , xn . To
construct a Q-Q plot we:
Overview
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
1. Order the observations from smallest to largest (i.e.,
x(1) ≤ y(2) ≤ . . . ≤ x(n) ).
2. Next we define the ith point, x(i) , as the (i − .5)/n quantile.
●
Central Limit
Theorem
Maximum
Likelihood
Bias
We could use i/n but can cause problems.
3. Based on a N (0, 1) distribution we compute the quantile
values q1 , q2 , . . . , qn (this is typically done using a table or
computer).
Wrapping Up
4. Finally plot (x(i) , qi ), and if they follow the same distribution
(Normal) they should form a line.
Lecture #7 - 9/14/2006
Slide 16 of 32
Example Q-Q plot
Lets assume that we have 5 observations: 3, 6, 4, 5, 2:
Overview
First we order them
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
Central Limit
Theorem
y(i)
2
3
4
5
6
(i − .5)/n
qi
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 17 of 32
Example Q-Q plot
Lets assume that we have 5 observations: 3, 6, 4, 5, 2:
Overview
Next compute quantiles
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
Central Limit
Theorem
Maximum
Likelihood
y(i)
2
3
4
5
6
(i − .5)/n
(1 − .5)/5 = .1
(2 − .5)/5 = .3
(3 − .5)/5 = .5
(4 − .5)/5 = .7
(5 − .5)/5 = .9
qi
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 18 of 32
Example Q-Q plot
Lets assume that we have 5 observations: 3, 6, 4, 5, 2:
Overview
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
Central Limit
Theorem
Maximum
Likelihood
Finally compute quantiles values assuming N (0, 1) (i.e., this is
a z-score)
y(i)
(i − .5)/n
2
(1 − .5)/5 = .1
3
(2 − .5)/5 = .3
4
(3 − .5)/5 = .5
5
(4 − .5)/5 = .7
6
(5 − .5)/5 = .9
and plot
qi
-1.28
-0.52
0.00
0.52
1.28
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 19 of 32
Example Q-Q plot
Notice how it follows nearly a straight line
Overview
Figure 1: Q-Q plot
Normal Distribution
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
7
6
5
4
Central Limit
Theorem
3
Maximum
Likelihood
Y
Bias
2
1
-1.5
Wrapping Up
Lecture #7 - 9/14/2006
-1.0
-.5
0.0
.5
1.0
1.5
Q
Slide 20 of 32
Other Tests
Other tests can be gathered in SAS (note the null hypothesis is
always that the data is normally distributed).
Overview
Normal Distribution
We will discuss this in later classes (especially in regression
diagnostics).
Assessing
Normality
➤ Uni Norm
➤ Make a Q-Q plot
➤ Example Q-Q plot
➤ Other Tests
proc univariate data=mydata normal plot;
var x1-x5;
run;
Central Limit
Theorem
Maximum
Likelihood
Bias
Wrapping Up
Lecture #7 - 9/14/2006
Slide 21 of 32
Central Limit Theorem
From Hays:
If a population has a finite variance σ 2 and a finite mean
µ, the distribution of sample means of N independent
observations approaches the form of a normal
distribution with variance σ 2 /N and mean µ as the
sample size N increases. When N is very large, the
sampling distribution of x̄ is approximately N (µ, σ 2 /N ).
(p. 251)
Overview
Normal Distribution
Assessing
Normality
Central Limit
Theorem
➤ CLT
Maximum
Likelihood
●
So...The distribution of x̄ converges to normal with mean
equal to µ and variance σ 2 /N .
Bias
✦
This is true no matter how X is distributed.
Wrapping Up
●
●
Lecture #7 - 9/14/2006
If X is normal, (N − 1)s2 /σ 2 has a Chi-Square distribution
with N − 1 degrees of freedom
Note: We will end up using these pieces of information for
hypothesis testing such as t-test and ANOVA.
Slide 22 of 32
Maximum Likelihood
●
Now that we know a thing or two about the PDF of the
normal distribution, it makes sense to talk about maximum
likelihood.
●
As we said last time, a MLE is an estimator which has a
value that maximizes something called a likelihood function.
●
In the statistics you will encounter in your career, MLEs will
frequently be used, so I present this to give you background.
Overview
Normal Distribution
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
➤ ML
➤ Likelihood
Functions
➤ MLE Properties
Bias
✦
●
In this class, you will never have to find what an MLE is,
only to know what types of properties MLEs have.
We said last time that x̄ and S 2 were MLEs - just know that.
Wrapping Up
Lecture #7 - 9/14/2006
Slide 23 of 32
Likelihood Functions
●
Let’s start our discussion by talking about a likelihood
function.
●
A likelihood function is the statistical model for the data
formed by the distribution the data follows.
●
Let’s imagine we have a sample of independent observations
x1 , x2 , . . . , xN we know come from N(0,1).
Overview
Normal Distribution
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
➤ ML
➤ Likelihood
Functions
➤ MLE Properties
Bias
Wrapping Up
Lecture #7 - 9/14/2006
✦
●
Statisticians would say that our sample is iid or
Independent and Identically Distributed.
Because of independence, the joint distribution of the data is
formed by taking the product of the individual distribution
functions of the data (like independence in our probability
chapter):
QN
L(x1 , x2 , . . . , xN ) = f (x1 ) × f (x2 ) × . . . × f (xN ) = i=1 f (xi )
Slide 24 of 32
Likelihood Functions
L(x1 , x2 , . . . , xN ) = f (x1 ) × f (x2 ) × . . . × f (xN ) =
Overview
Normal Distribution
Assessing
Normality
L(x1 , x2 , . . . , xN ) =
i=1
Central Limit
Theorem
Maximum
Likelihood
➤ ML
➤ Likelihood
Functions
➤ MLE Properties
N
Y
●
√
1
2πσ 2
e
N
Y
f (xi )
i=1
−(xi −µ)2
2σ 2
The likelihood function above is the function of the data that
needs to be maximized with respect to µ or σ 2 .
✦
By calculus, we know that:
■
x̄ happens to be where L is maximized for µ.
■
S 2 happens to be where L is maximized for σ 2 .
Bias
Wrapping Up
●
Lecture #7 - 9/14/2006
Let’s have an example...we have five observations: 2,3,4,5,6,
which are iid N (µ, 2.5).
Slide 25 of 32
Likelihood Functions
Our likelihood function:
N
Y
−(xi −µ)2
1
√
L(µ) =
e 2×2.5
2π2.5
i=1
Overview
Normal Distribution
1.40e−05
Assessing
Normality
1.20e−05
L(mu)
Maximum
Likelihood
➤ ML
➤ Likelihood
Functions
➤ MLE Properties
1.30e−05
Central Limit
Theorem
Wrapping Up
1.10e−05
Bias
3.6
3.8
4.0
4.2
4.4
mu
Lecture #7 - 9/14/2006
Slide 26 of 32
MLE Properties
●
Overview
MLEs are used frequently because of their properties:
✦
Functional invariance: any function of the MLE results in
the MLE for a function.
✦
Asymptotic behavior: when N is very large, the variance
of the MLE hits an important lower limit (implications for
consistency and relative efficiency).
Normal Distribution
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
➤ ML
➤ Likelihood
Functions
➤ MLE Properties
Bias
●
What does all of this mean to you?
✦
MLEs are your friends.
✦
You now know the basics, so do not shy from talking MLEs
(or reading them).
Wrapping Up
Lecture #7 - 9/14/2006
Slide 27 of 32
Bias Recap
●
Overview
Last time we said that x̄ is unbiased for µ.
✦
This means that E(x̄) = µ.
Normal Distribution
●
Assessing
Normality
We also said that if we use:
s2 =
Central Limit
Theorem
For variance, then:
Maximum
Likelihood
●
Lecture #7 - 9/14/2006
2
(x
−
x̄)
i
i=1
N −1
E(s2 ) = σ 2
Bias
➤ Mean
➤ Variance
Wrapping Up
PN
For your information, these slides show you how...
Slide 28 of 32
Mean
E (x̄) = E
Overview
N
X
i=1
xi
N
!
=E
x1 + x2 + . . . + xN
N
Normal Distribution
Assessing
Normality
Central Limit
Theorem
E(x1 ) + E(x2 ) + . . . + E(xN )
µ + µ + ...+ µ
Nµ
=
=
=
N
N
N
=µ
Maximum
Likelihood
Bias
➤ Mean
➤ Variance
Wrapping Up
Lecture #7 - 9/14/2006
Slide 29 of 32
Variance
From http://en.wikipedia.org/wiki/Variance
Overview
Normal Distribution
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
Bias
➤ Mean
➤ Variance
Wrapping Up
Lecture #7 - 9/14/2006
Slide 30 of 32
Final Thought
●
Overview
Normal Distribution
✦
Assessing
Normality
Central Limit
Theorem
Wrapping Up
➤ Final Thought
➤ Next Class
Lecture #7 - 9/14/2006
Part of that frequency is
due to the CLT.
●
MLEs are something to
understand but their
determination is not the
focus of this course.
●
Biased and unbiased parameters are not talked about much
after this lecture.
Maximum
Likelihood
Bias
The normal distribution is
something we will come to
use quite often.
Slide 31 of 32
Next Time
Overview
Normal Distribution
●
Hypothesis testing, Part I (Chapter 7.1 to 7.10)
●
We will recap this week’s Wonder Showzen: MTV2 Friday at
8:30pm.
Assessing
Normality
Central Limit
Theorem
Maximum
Likelihood
Bias
Wrapping Up
➤ Final Thought
➤ Next Class
Lecture #7 - 9/14/2006
Slide 32 of 32