Download P-value - ReSAKSS Asia

Statistical inference: distribution,
hypothesis testing
Distribution of a correlation coefficient?
Computer simulation…
1. Specify the true correlation coefficient
◦
Correlation coefficient = 0.15
2. Select a random sample of 100 virtual men from
the population.
3. Calculate the correlation coefficient for the
sample.
4. Repeat steps (2) and (3) 15,000 times
5. Explore the distribution of the 15,000 correlation
coefficients.
Distribution of a correlation
coefficient…
Normally distributed!
Mean = 0.15 (true correlation)
Standard error = 0.10
Distribution of a correlation
coefficient in general…
1. Shape of the distribution
◦
Normally distributed for large samples
◦
T-distribution for small samples (n<100)
2. Mean = true correlation coefficient (r)
3. Standard error 
1 r
n
2
Many statistics follow normal
(or t-distributions)…
Means/difference in means
◦ T-distribution for small samples
Proportions/difference in proportions
Regression coefficients
◦ T-distribution for small samples
Natural log of the odds ratio
Recall: 68-95-99.7 rule for normal distributions! These is a 95%
chance that the sample mean will fall within two standard errors of
the true mean
Mean - 2 Std error=55.4
Mean
Mean + 2 Std error =68.6
To be precise, 95%
of observations fall
between Z=-1.96
and Z= +1.96 (so
the “2” is a rounded
number)…
95% confidence interval
Thus, for normally distributed statistics, the
formula for the 95% confidence interval is:
sample statistic  2 x (standard error)
Simulation of 20 studies of
100 men…
Vertical line indicates the true mean (62)
95% confidence
intervals for the mean
vitamin D for each of the
simulated studies.
Only 1 confidence
interval missed the true
mean.
The P-value
P-value is the probability that we would have seen our
data (or something more unexpected) just by chance if
the null hypothesis (null value) is true.
Small p-values mean the null value is unlikely given
our data.
Our data are so unlikely given the null hypothesis
(<<1/10,000) that I’m going to reject the null
hypothesis! (Don’t want to reject our data!)
P-value<.0001 means:
The probability of seeing what you saw or
something more extreme if the null hypothesis is
true (due to chance)<.0001
P(empirical data/null hypothesis) <.0001
The P-value
By convention, p-values of <.05 are often accepted
as “statistically significant” in the medical
literature; but this is an arbitrary cut-off.
A cut-off of p<.05 means that in about 5 of 100
experiments, a result would appear significant just
by chance (“Type I error”).
Summary: Hypothesis Testing
The Steps:
1.
Define your hypotheses (null, alternative)
2.
Specify your null distribution
3.
Do an experiment
4.
Calculate the p-value of what you observed
5.
Reject or fail to reject (~accept) the null hypothesis
Hypothesis Testing
◦ Null hypothesis - Statement regarding the value(s) of unknown
parameter(s). Typically will imply no association between
explanatory and response variables in our applications (will always
contain an equality)
◦ Alternative hypothesis - Statement contradictory to the null
hypothesis (will always contain an inequality)
◦ Test statistic - Quantity based on sample data and null hypothesis
used to test between null and alternative hypotheses
◦ Rejection region - Values of the test statistic for which we reject the
null in favor of the alternative hypothesis
Hypothesis Testing
Test Result –
True State
H0 True
H0 False
H0 True
H0 False
Correct
Decision
Type I Error
Type II Error
Correct
Decision
  P(Type I Error )   P(Type II Error )
• Goal: Keep ,  reasonably small
Sampling Distribution of Difference in Means
In large samples, the difference in two sample means is
approximately normally distributed:
2
2 



1
Y 1  Y 2 ~ N  1   2 ,
 2 


n
n
1
2


• Under the null hypothesis, 1-2=0 and:
Z
Y1 Y 2

2
1
n1


2
2
n2
~ N (0,1)
• 12 and 22 are unknown and estimated by s12 and s22
Elements of a Hypothesis Test
Test Statistic - Difference between the Sample means, scaled
to number of standard deviations (standard errors) from the
null difference of 0 for the Population means:
T .S . : zobs 
y1  y 2
s12 s22

n1 n2
• Rejection Region - Set of values of the test statistic that
are consistent with HA, such that the probability it falls in
this region when H0 is true is  (we will always set =0.05)
R.R. : zobs  z
  0.05  z  1.645
P-value (aka Observed Significance Level)
P-value - Measure of the strength of evidence the sample
data provides against the null hypothesis:
P(Evidence This strong or stronger against H0 | H0 is true)
P  val : p  P(Z  zobs )
2-Sided Tests
H0: 1-2 = 0
HA: 1-2  0
Test statistic is the same as before
Decision Rule:
◦ Conclude 1-2 > 0 if zobs  z/2 (=0.05  z/2=1.96)
◦ Conclude 1-2 < 0 if zobs  -z/2 (=0.05  -z/2= -1.96)
◦ Do not reject 1-2 = 0 if -z/2  zobs  z/2
P-value: 2P(Z |zobs|)
Power of a Test
Power - Probability a test rejects H0 (depends on 1- 2)
◦ H0 True: Power = P(Type I error) = 
◦ H0 False: Power = 1-P(Type II error) = 1-
·Example:
· H0: 1- 2 = 0 HA: 1- 2 > 0
 12 = 22  25 n1 = n2 = 25
· Decision Rule: Reject H0 (at =0.05 significance level) if:
zobs 
y1  y 2

2
1
n1


2
2
n2

y1  y 2
 1.645 
2
y1  y 2  2.326