Download Type I error - Nuffield College

Quantitative Methods – Week 7:
Inductive Statistics II:
Hypothesis Testing
Roman Studer
Nuffield College
[email protected]
Introduction
In last week’s class on inductive statistics, we learned how to
determine and interpret confidence intervals...
A standard normal
distribution is a normal
distribution N(0,1) with
mean =0 and standard
deviation =1
95% of cases
=1
2,5% of cases
2,5% of cases
with
-1,96
0
 Explain!
+1,96
Introduction (II)
… where the t-value tells us how many standard errors either side of the
mean we have to add to achieve a certain degree of confidence (90%, 95%,
99%)
X  t0.025 SE ( X )
E.g. 95% confidence level:
With confidence intervals, the implicit hypothesis was that the specific
intervals contains a certain value
With hypothesis testing, the hypothesis is now made explicit; so we
explicitly want to know whether…
•
Can the result from one sample be considered different from the result
of another sample?
Example: Is the mean relief payment in Kent different from the one in Sussex?
•
Is there really a connection between one variable and another one?
Example: Is the regression coefficient on unemployment statistically significant
and can therefore help to explain the voting share of the Nazi party?
 Why do we actually need hypothesis testing ??
The 5 Steps of Hypothesis Testing
1. Specify the hypothesis in an appropriate form for
statistical testing
2. Set a level of probability on the basis of which the
hypothesis should be rejected
3. Select the relevant test statistic
4. Calculate the test statistic and compare this calculated
value with the tabulated theoretical probability
distribution in order to reach a decision about the
hypothesis
5. Interpret the results of the decision
Step 1: Null and Alternative Hypothesis
For testing a hypothesis, the first step is set up a null
hypothesis (H0) that can be rejected
The null hypothesis is presumed to be true until the data
strongly suggests otherwise (like a defendant on trial)
The alternative hypothesis H1 specifies the opposite
Examples:
•
•
•
H0: The defendant is innocent
H1: The defendant is guilty
H0: Unemployment rates had no impact on Nazi votes
H1: Unemployment had an impact on Nazi votes
H0: bUnemployment=0
H1: bUnemployment≠0
Step 2: The Level of Probability
Type I and Type II error
In hypothesis testing we can make two kinds of mistakes
 Type I error: Rejecting the null hypothesis when it is in fact true
 Type II error: Failing to reject the null hypothesis when it is
actually false
Statistical decision
True state of
nature
Reject H0
Do not reject
H0
H0 is true
Type I error
Correct
H0 is false
Correct
Type II error
• What is the risk we are willing to take of making a Type I error?
Step 2: The Level of Probability (II)
The Significance level, a, is the probability of making a Type I
error
 A small probability of a type I error is preferred
To what extent we are willing to take a risk of making wrong
conclusion?
Common choices for a are…
• 10%
• 5% (most common)
• 1%
5% level means that we are taking a risk of being wrong five
times per 100 trials
Trade-off: If we reduce the probability of a type I error, the
probability for a type II error will increase
Step 3: The Test Statistic
What is the probability, that H0 is true given the observed
outcome?
For every sample statistic there is a corresponding sampling
distribution
Test statistics and critical values in order to test a null hypothesis
against an alternative
 For the time being, the test statistic we are using is the tstatistic
CAUTION!
 Always remember the underlying assumptions when testing:
a) The distribution is approximately normal
b) Our sample doesn’t suffer from a serious sample bias
Step 4: Calculate and Assess the Test Statistic
Test statistic for a regression coefficient
If we could repeat the “social” experiment, we would obtain
different values for the error term and consequently for the
dependent variable  impact on the slope coefficient
How reliable is it to conclude that there is a relationship?
Example:
2nd sample
y
1st sample
b<b
x
Step 4: Calculate and Assess the Test Statistic (II)
Is b significantly different from 0?
H0: b=0
 t-statistics
b
tb 
e / x
The nominator b is the regression coefficient derived from the
OLS method
The denominator corresponds to the estimated standard
deviation of the sampling distribution of the coefficient, i.e.
the SE(b)
Step 4: Calculate and Assess the Test Statistic (III)
Statistical significance using the t-statistics
The t-statistic indicates how many standard deviations the
sample regression coefficient is from 0 (we can also express it
as deviation from some certain value of interest if we want)
Central Limit Theorem applies
• Shape of the sampling distribution becomes normal
• With increasing sample size the t-distribution
approximates a standard normal distribution
 If t-value >1.96, the estimated regression coefficient
is more than 1.96 standard deviations from 0 and the
probability for such an outcome is less than 5%, if H0 is
true
Step 4: Calculate and Assess the Test Statistic (IV)
Sampling distribution and critical value
95% of cases
2.5%
of
cases
2.5% of
cases
-1.96
Reject H0
Accept H0
+1.96
Reject H0
Confidence level a=0.95
Step 4: Calculate and Assess the Test Statistic (V)
Significance level and p-level
P-value is the probability that the outcome observed would be
present if the null hypothesis is true
Small p-values are evidence against H0
P-value<a  Reject H0 , accept H1
Failing to reject the null hypothesis does not necessarily
constitute support for H0; it just means that the data or pattern
is not sufficiently strong to reject H0
Step 4: Calculate and Assess the Test Statistic (VI)
One- and two-tailed tests
Two-tailed test: The parameter value is calculated for both
tails of the sampling distribution
 The critical region is divided equally between the left- and
the right-hand tails
If the hypothesis is about the directions:
One-tailed test: H0: μ1 = μ2; H1: μ1 > μ 2
Critical region is in the left- or the right-hand tail
 a one-tailed test increases the critical region at one tail
Strong a priori reasons must exist!
Step 5: Interpreting the Results
Statistical versus historical significance
Statistical significance refers to the probability of type I error
(rejecting the null hypothesis when it is in fact true)
 Statistical significance is influenced by
• magnitude of the parameter
• magnitude of the standard error, i.e. sample size (because
the standard error decreases with N everything else being
equal)
Historical significance: What is the practical significance of
rejecting the hypothesis?
 The aim is to find statistically significant results that are
relevant from a historical perspective.
Computer Class:
•
Repetition & Hypothesis Testing
Exercises
Relief dataset
Get the “Relief” dataset at
http://www.nuff.ox.ac.uk/users/studer/teaching.htm
• Note: In the column “county”, 1 stands for Kent and 2 stands for Sussex.
• Look at the dataset: what is the basic unit of measurement here? Is it a time-series, a
cross-sectional or a panel dataset?
• Look at the variables relief and unemployment (unemp)
•
Get a first overview by visualising the data: Are the variables approximately normally distributed?
•
Compute the mean, median, standard deviation, coefficient of variation, kurtosis and skewness for the
variables
•
Are the level of relief and the level of employment statistically different in Kent and Sussex? (Hint:
create new variables and test for difference of means.) What is the null hypothesis? Are the means
different at a 95% significance level?
•
Work again with the whole dataset, not just with Kent and Sussex. Compute the correlation coefficient
between unemployment and the relief payments. How do you interpret the result? Is the correlation
coefficient significant at a 95% and at a 99% level? State the null hypothesis, and test it
•
Estimate a regression using relief and unemployment. Which one is the dependent, which one the
independent variable? Report the results of the regression (a, b, R 2). Is the regression coefficient (b)
statistically significant at a 95% level? Formulate the null hypothesis and test it. What is the critical tvalue shown in the t-statistics table? What is the actual t-value of b? What is the p-value? What does
all that mean for the interpretation of the regression results?
Homework
Readings:
• Feinstein & Thomas, Ch. 8 & 9
Problem Set 5:
 Finish the exercises from today’s computer class if you haven’t done so already.
Include all the results and answers in the file you send me.
 Do exercise 1 from Feinstein & Thomas, p. 181.