Download p values - Personal Web Pages

What a p Value Can and Can Not Tell You
Richard G. Lambert, UNC Charlotte, RSCH 6110
Unfortunately, practitioners and students of statistics often place more value on p values
than statisticians do. There seems to be an artificial reverence for p values that may not
always be justified. The social sciences seem at times to have there own doxology:
“Praise God from whom all p values flow,
praise alpha, beta, and .05,
praise statistics all ye researchers here below,
praise alpha, beta, and .05.”
While this handout is not intended to diminish the importance and usefulness of the
proper application of statistical methods, it may be helpful to reflect about exactly what
information a p value is providing.
First, let’s review some basic definitions:
p Value – A p value indicates the probability that effects, of the size observed or greater,
are due to sampling error under the condition that the null hypothesis is true. A p value
answers the question: If this treatment really does not work, how likely is it that effects of
at least the size observed could have been due to sampling error?
Statistical Significance - Are the results beyond what would be expected due to
sampling error alone? Are the results large enough that it is very unlikely that they are
due to sampling error? A finding is said to be statistically significant when the p value
associated with the test statistic is smaller than the predetermined alpha level. Typically,
alpha is set at .05. Therefore, when a p value less than .05 is observed the null hypothesis
is rejected.
Practical Significance - Are the effects found in a particular study large enough to be of
any practical value? Do the results translate into any change in actual practice? Do they
translate into any change in policy?
What a p Value Can Tell You
1. A p value can tell you if your results are typical or rare in the context of the sampling
distribution of the test statistic under the condition that the null hypothesis is true.
2. A p value tells you the probability that sampling error alone is responsible for your
findings.
3. A p value helps you to make a decision at the end of the study about which hypothesis,
the null or the alternative, is more reasonable to accept. A p value simply tells you
whether the findings you obtained are likely to be members of the sampling distribution
for the condition when the null hypothesis is true. If this probability is very low, less
than alpha, it is more reasonable to conclude that the observed results are members of
some sampling distribution that represents a case when the alternative hypothesis is true.
What a p Value Can Not Tell You
1. A small p value does not warrant use of the word “prove”. A single study, small p
value or large, merely demonstrates evidence for the truefulness or falsehood of a
hypothesis. It does not prove anything. Furthermore, a single study presents evidence
that is specific to the conditions under which the study is conducted. Theoretical
concerns, replication, validation, and judgements about the generalizability of findings
are just as important to the scientific process when a p value is small as they are when a p
value is large.
2. A p value can not tell you that you have an important finding. A small p value does not
exempt the researcher from using judgement to arrive at the correct interpretation of a
research finding. All the issues regarding theoretical, practical, or clinical significance of
a research finding are always present.
3. Correlation is still not necessarily causality, even when a p value is small.
4. A small p value does not indicate an effect of a particular size. Studies with large
sample sizes may yield small p values associated with very small effects, while studies
with small sample sizes may yield large effects that are not statistically significant.
5. A p value cannot tell you that a statistical significance test was necessary at all.
Statistical significance testing applies when researchers have used random sampling from
a defined population, or when a researcher has obtained a sample that is believed to be
reasonably similar to what would have happened if a scientifically credible sampling
strategy had been employed. Statistical significance testing applies to situation in which
researchers are testing hypotheses about population conditions by using information
obtained from random samples drawn from those population conditions, or from samples
that reasonably approximate random and representative samples. They do not apply to
population data, to some single subject research situations, or to batches of data obtained
by convenience.
6. A small p value does not indicate that the correct statistical procedure was used, nor
does it indicate that the assumptions of the procedure were met.
7. A small p value does not inform the researcher about the cost benefit or cost
effectiveness issues related to the treatment nor does it make any statement about the side
effects of the treatment.
8. A small p value does not make any statement about the presence or absence of
confounding variables in a study. Statistically significant results must still be qualified
with respect to measurement error, potential flaws in the research design, or context
specific factors that may influence the results.