Download An Introduction to Statistical Significance

An Simple Introduction to
Statistical Significance
Julie Graves
MST Conference
June 2014
Consider a study evaluating a new weight loss drug.
Group A received the drug and lost an average of four
kilograms (kg) in seven weeks. Group B didn't receive
the drug but still lost an average of one kg over the
same period.
Did the drug produce this three-kg difference in
weight loss? Could some unmeasured influence have
led to the three-kg difference in weight loss? Or could
it be that Group A lost more weight simply by chance?
It is possible that the drug given to group A did not
actually have any influence on weight loss. A
statistical analysis could be used to determine how
likely it is that the observed difference — in this
case, the three-kg difference in average weight loss
— might have occurred by chance alone.
Statistical analysis may show that the observed
difference has a very small probability of
occurring by chance if the treatment did not
really have an effect on weight loss. If the
probability is small (less than 5%), the
researchers would conclude that the observed
differences in weight loss should be attributed
to the treatment, not to chance.
An observed difference that is unlikely to have
occurred by chance is called
statistically significant.
Are drivers more distracted when using a cell phone
than when talking to a passenger in the car?
Researchers wanted to find out, so they designed
an experiment. Here are the details:
.
In a study involving 48 people, 24 people were
randomly assigned to drive in a driving simulator
while using a cell phone. The remaining 24 were
assigned to drive in the driving simulator while
talking to a passenger in the simulator. Part of the
driving simulation for both groups involved asking
drivers to exit the freeway at a particular exit. In the
study, 7 of the 24 cell phone users missed the exit,
while 2 of the 24 talking to a passenger missed the
exit.
Note that 9 out of 48 drivers missed the exit.
Maybe the exit-missers were going to miss the exit
regardless of which group they were assigned to.
Maybe cell phone or passenger conversations don’t
have anything to do with missing exits. Of the 48
drivers in the study, maybe 9 are just inherent exitmissers. If this is the case, we would have expected
to see 4 or 5 exit-missers among the 24 cell phone
users. The fact that we saw 7 exit-missers among
the 24 cell phone users could be just a chance
occurrence.
We need to choose between two possibilities
• Drivers on cell phones are more likely to
miss an exit than are other drivers.
• Drivers on cell phones are not more likely to
miss an exit. It was due to chance that we
observed 7 exit-missers among the 24 cell
phone users.
Statisticians can calculate the probability that
an observed difference occurred by chance.
Rather than doing calculations, we will use a
simulation to estimate this probability.
Remove 4 spades from a standard deck of
playing cards. The modified deck will contain
13 hearts, 13 clubs, 13 diamonds and 9 spades.
The 39 non-spades will represent the drivers
who did not miss an exit, and the 9 spades will
represent the exit-missers.
Here is how our simulation will work
• From the 48 cards, randomly select 24 cards
to represent the 24 cell phone users.
• Count how many exit-missers were cell
phone users.
• Repeat.
• Pool results with other simulators.
Is it unlikely that the group of 24 cell phone users
would contain 7or more exit-missers by chance?
If the answer is yes, then the difference between 7
exit-missers in the cell phone group and 2 exit
missers in the passenger group is statistically
significant.
If the answer is no, then the difference between 7
exit-missers in the cell phone group and 2 exit
missers in the passenger group is not statistically
significant.
Suppose that in a clinical trial patients were
randomly assigned to either Treatment A or
Treatment B. Suppose that so far, Treatment A
has a cure rate of 100% and Treatment B has a
cure rate of only 50%. That's a pretty dramatic
sounding difference. Is it a significant
difference?
It is possible that only two patients have been
assigned to each treatment and both were cured
with treatment A, but only one of the two was
cured with treatment B. Treatment A would have a
100% cure rate and B would have a 50% cure rate.
Maybe the next patient assigned to treatment A
will not be cured and the next patient assigned
to treatment B will be.
With three patients assigned to each group, the
cure rate is 66% for both treatments. What
does all this tell you about the "real" cure rate
for each treatment? Can we conclude that the
cure rate is the same for both, 66%?
After more patients are assigned to each
treatment group, we might observe that
Treatment A has a 52% cure rate and B has a
51% cure rate.
Researchers would want to know if the
observed difference between cure rates is a
statistically significant difference.
What do we mean if we say that the difference
between two cure rates is statistically
significant?
We do not mean that the difference is
• A dramatic difference
• A large difference
• An important difference
We mean that an observed difference is
unlikely to have occurred by chance.