Download lecture 5

Introduction to Data
Analysis.
Hypothesis Testing for means
and proportions.
Today’s lecture
Hypothesis testing (A&F 6)
 What’s a hypothesis?

Probabilities of hypotheses being correct.
 Type I and type II errors.

2
What’s a hypothesis?


Hypotheses = testable statements about the world.
Hypotheses = falsifiable.



We test hypotheses by attempting to see if they could be false,
rather than ‘proving’ them to be true.
All swans are white…? Popper said you cannot prove that all swans
are white by counting white swans, but you can prove that not all
swans are white by counting one black swan.
We normally generate hypotheses from a
combination of theory, past empirical work, common
sense and anecdotal observations about the world;

e.g. young people are more leftwing than their elders. Observations
abut my friends, work showing this relationship in other countries,
theory suggesting that social ageing makes people more rightwing.
3
Some hypotheses (or not)

My girlfriend spends too much on make-up.


My girlfriend spends more than me on make-up.


NO. Falsifiable, but not really social science.
Women spend more than men on make-up.


NO. It’s a normative claim about ‘too much’.
HOORAY, a social scientific hypothesis. It’s falsifiable, and tells
us something about the world.
Make-up is used to marginalize the Other as a form of
contemporary patriarchialist nihilism.

NO. This is too vague to be falsifiable.
4
Hypothesis testing

Two different types of hypothesis.
Descriptive inference (e.g. old people are more religious
than young people).
 Causal inference (e.g. old people are more religious than
young people because they think about death more).


We test statistical hypotheses using
significance testing.

This is a way of statistically testing a hypothesis by
comparing the data we have to values predicted by the
hypothesis.
5
Null and alternative hypotheses


When we’re testing hypotheses, we want to choose
between two conflicting statements.
Null hypothesis (H0) is directly tested.



This is a statement that the parameter we are interested in has a
value similar to no effect.
e.g. regarding religiosity, old people are the same as young people.
Alternative hypothesis (Ha) contradicts the null
hypothesis.


This is a statement that the parameter falls into a different set of
values than those predicted by H0.
e.g. regarding religiosity, old people are different to young people.
6
A (simple) hypothesis

We’re interested in whether new measures to
curb MEPs fraudulently claiming for flights
are effective.
Previously, the population of MEPs managed to claim
16 flights each on expenses (per month).
 After the new measures are introduced, we sample 100
MEPs and find that they are now managing to only
charge 13½ flights a month each to expenses. The
standard deviation for this sample is 10 flights.
 Are EU (well British and German anyway) taxpayers
really getting better value for money though?

7
Aviatophobia or less fraud?

So in this case the null hypothesis is that there
has been no change.


H0 is that MEPs are claiming the same amount per
month as they were before, and the difference is just
because our sample happens to include a lot of
aviatophobic MEPs (or something like that).
The alternative hypothesis is that MEPs
spending has been curbed.

Ha is that MEPs are claiming less than they were
previously.
8
More formally

Slightly more formally:
The population we are interested is MEPs after the
changes.
 We want to know whether the population mean ‘number
of flights claimed per month’ is different from 16.
 H0 is that this population mean is equal to 16.
 Ha is that this population mean is less than 16.
 The info we have is from one sample mean.


Now we imagine that H0 is true…
9
If H0 is true…(1)



The population mean will be equal to 16.
Since n is large(ish) the sampling distribution will be
normal and centred around 16.
We can calculate the standard error of the sampling
distribution.
 
Standard error X 
s
10

1
n
100
10
If H0 is true…? (2)
H0 mean = 16
Sample mean = 13.5
10
11
12
13
14
15
16
17
18
19
20
21
22
Flights claimed (per month)
2 ½ % of the distribution
11
P-value (1)

So we know that H0 looks pretty unlikely (certainly
less than a 2½ per cent chance), but we can actually
give a more precise probability.

We work out how many standard errors the sample mean is away
from H0 to produce a z-score, and then calculate a p-value from
this with reference to the normal probability distribution.
Sample mean  Null hypothesis mean
Standard error
13.5  16
z
 2.5
1
Pr (sample mean being more than 2.5 SEs lower
than the population mean)  0.006
z
12
P-value (2)


If H0 were correct it looks unlikely that we would get
a sample that had a mean of 13½.
In fact there is just a 0.006 (or 0.6%) chance that our
sample could have come from a population postulated
by the null hypothesis.


We could set an (arbitrary) ‘significance level’ that our test must
meet. Maybe we need to be 99% confident that we can reject the
null hypothesis (i.e. the p-value is less than 1%).
Best practice when we perform significance tests is simply to report
the p-value, and to make the judgement that p-values of, say, 5%
and below are probably good evidence the null hypothesis can be
rejected.
13
Steps for Hypothesis test
Check assumptions (i.e. normality, sample
size, level of measurement)
 State hypotheses—Null and Alternative
 Calculate appropriate test statistic (e.g z-score)
 Calculate associated p-value
 Interpret the result

14
Interpreting hypothesis test

We NEVER accept the null hypothesis.
We either reject or fail to reject based on our
p-value.
 May fail to reject null hypothesis due to:


small sample size
 inappropriate research design
 biased sample, etc..
15
Type I and type II errors

A type I error occurs when we reject H0, even though
it is true.


A type II error occurs when we do not reject H0, even
though it is false.


This is going to happen 5% of the time if we choose to reject H0
when the p-value is less than 0.05.
If our ‘significance level’ is 0.05, then sometimes there will be a
real difference and we won’t detect it.
The more stringent the significance level the more
difficult to detect a real effect is, but the more
confident we can be that when we find an effect it is
real.
16
Making errors…

There is a trade-off between the two types of
error. Depending on what we’re doing we may
be more willing to accept one sort or the other.
Think of this as analogous to a legal trial, we don’t want
the guilty to go free (a type II error), but we’d be even
unhappier if we execute an innocent person (a type I
error).
 In this case, we might want the significance level to be
very low to minimize executing innocent people (but at
the same time allowing lots of the guilty to go free)

17
Differences between 2 sample means
More normally we’ve sampled two groups and
wish to see if they differ.
 Let’s go back to our religion example.

We might be interested in whether women’s churchgoing differs from men.
 We have 2 samples, 45 men and 55 women.
 The men have a mean attendance of 7 days a year, with
standard deviation of 15.
 The women have a mean attendance of 10 days a year
with standard deviation of 15.

18
Are women more or less religious than
men?

Essentially, to answer this we estimate the
difference between the populations (the
parameter) using the difference between the
sample means (the statistic).
We can run a significance test on this statistic, and work
out whether our samples are likely to represent real
differences between the populations of men and women.
 The null hypothesis is therefore that there is no
difference between men’s mean churchgoing and
women’s mean churchgoing.

19
Z-score

So we work out the z-score as before.
Estimate of parameter  null hypothesis value
Standard error of estimator
( X women  X men )  0 X women  X men
z

2
2
SE ( X women  X men )
swomen
smen

nwomen nmen
z
z
10 - 7
152 152

55 45

3
 1.00
4.09  5
20
P-value



The standard error of the estimator is ~3 and the Zscore is ~1 then.
We have no prior ideas of whether we think women
are more or less religious than men, so we just want
to test the possibility that they are not the same.
i.e. we want to know how likely it would be to get an
individual estimate of the difference between the
sample means that is either 1 SE greater than the null
hypothesis (i.e. zero) or 1 SE less than the null
hypothesis.
21
Two-sided tests (1)
-12
-9
-6
-3
0
H0 mean = 0.
SE = ~3.
3
6
9
12
Churchgoing for women - churchgoing for men (in days per year)
68% of the distribution
Difference between sample means = 3
i.e. SampleWomen is 3 greater than sampleMen
22
Two-sided tests (2)




The probability of a difference between the two
sample means being 3 or greater is 0.16.
The probability of a difference between the two
sample means being 3 or less is 0.16.
So the p-value for a 2-sided test is 2*(0.16) or 0.32.
This value is high (much higher than our 5% cut off
value), so we fail to reject H0 that men and women do
not differ in their church attendance.
23
Two-sided tests (3)

Regardless of our theoretical expectations, the
convention is to use two-tailed tests. Why?
In essence making it even more difficult to find results
just due to chance.
 We normally don’t have very strong prior information
about the difference.
 One tailed tests are often the hallmark of someone
trying to make something out of nothing.
 What does it mean to use a one-tailed test??


Not necessarily bad—in fact arguably MUCH smarter.
24
Significance tests and CIs (1)
Notice that our significance test has ended up
looking rather similar to the CIs.
 We could use a CI around the difference
between the two sample means to test the
hypothesis that they are the same.
 A 95% CI would just be 1.96*SE. We’ve just
worked out the SE (it’s approximately 3).

25
Significance tests and CIs (2)
(  women   men )  ( X women  X men )  1.96  SE
(  women   men )  3  1.96  3
(  women   men )  3  5.88


The 95% CI encloses zero (which was our null
hypothesis, that women are the same as men).
CIs and significance tests are doing the same job, just
presenting the information in a slightly different way.
26
Proportions and significance tests
This means of course that all I said about
proportions and CIs, applies to proportions and
significance tests.
 A hypothesis related to a previous example is
that one of the candidates does have a lead.
 Therefore the null hypothesis is that both
candidates have a vote share of 50%.

27
Proportions example – the return



Sample is 1000, proportion voting Democrat is 0.45 (or 45%).
Null hypothesis is that the population proportion is 0.5.
Thus, according to my sample, it is very likely that the
Presidential race is not a dead-heat.
z 
0  P

 0 (1   0 )
n
0.5  0.45
0.5(1  0.5)
1000
0.05
 3.16
0.0158
Pr(sample mean being 3.16 SEs or more different to the
z
population mean)  0.001* 2  0.002
Note: The SE is
calculated
with the null hypothesis
proportion, not with the
sample proportion.
Remember, we are
testing the null
hypothesis, so the mean
is the null hypothesis
mean .
28
Summary

Because we know the shape of the sampling
distribution, we can work out:
Ranges around an individual sample mean that will
enclose the population mean X per cent of the time.
 The probability that a hypothesis about the population
mean is true, given a particular sample mean.
 The probability that population means for different
groups are different, given two sample means.
 All of the above for proportions.


Although there are some small complications...
29
Things I haven’t mentioned

The reality of statistical hypothesis testing is slightly
more complicated than this in some ways particularly:



If your sample sizes are small. If the sample < 30 or < 40 then we
need to use a different distribution called the t distribution. To use
this we need to assume that the population distribution we are
interested in is normal.
As sample size increases, the t-distribution looks more and more
similar to the z-distribution. This is why significance tests (even for
big sample sizes) are often called t-tests.
When looking at proportions we often use slightly different tests as
well to accommodate the fact that a proportion cannot be greater
than 1, or less than 0.
30