Download Outline for Topic 1

Part of the Web Site of Dr. Jeffrey Oescher
EDF 802 Index Page
Risks and Errors,
Power and Sample Size, and
Effect Size
Revised – 30 January 2014
Formative Exercises - Topic 2
I.
Errors and risks
A.
Statistical decisions to reject or accept the null hypothesis
1.
Correct decisions
a.
Accept H0 when it is true
b.
Reject H0 when it is false
2.
Errors
a.
Type I – reject H0 when it is true
b.
Type II – accept H0 when it is false
Table 1
Type I and Type II Errors
Reject H0
Statistical
Conclusion
B.
Accept H0
Reality
H0 is True
H0 is False
Type I Error
Correct
(α)
Decision (1-β)
Correct
Type II Error
Decision (1-α)
(β)
Consequences of Type I and Type II errors are a function of the
researcher’s values
1.
Educational example
a.
Consider the use of an expensive computerized
writing program. If H0 is rejected and it is true
(i.e., Type I error), the program is selected even
though it is very expensive and will not result in
increased achievement. If H0 is accepted but a
difference really does exist (i.e., Type II error),
the program will not be used and the students will
not have the advantage of using an effective
program.
b.
Which error would you like to protect against if
the program director wants to showcase your
schools and is willing to provide them with all of
II.
the hardware, software, and training necessary to
use this program effectively?
c.
Which error would you like to protect against if
you want the students to benefit from an
exemplary program?
2.
Medical example
a.
Consider the use of a new, very costly drug. If H0
is rejected and it is true (i.e., Type I error), the
new drug is used even though it is costly and not
helpful. If H0 is accepted but a difference really
does exist (i.e., Type II error), the drug will not be
used and the patients will not receive the benefits
from it.
b.
Which error would you like to protect against if
several effective drugs are available already?
c.
Which error would you like to protect against if
there are no known effective drugs and the health
problems prompting the use of this drug are
increasing at an alarming rate?
3.
Urban study example
a.
Consider the consequences of an impact study
related to the revitalization of a major urban
recreation area. The predicative models suggest
the changes to the park will increase usage and
revenues to a level that the facilities would
become self-sustaining. If H0 is rejected and it is
true (i.e., Type I error), the renovations are
completed even though they are very expensive
and will not result in increased usage or revenue.
If H0 is accepted but the changes really would
create a self-sustaining recreational area (i.e.,
Type II error), the renovation will not be
undertaken and the citizens will not have the
advantage of using an improved, financially
viable recreation park.
b.
Which error would you like to protect against if
the city is in desperate need of a recreation area
that would support its youth and older citizens
and, perhaps more importantly, is willing to pay
whatever costs are involved in developing it?
c.
Which error would you like to protect against if
the city has indicated they cannot provide any
financial support for this project, although the
need exists.
Controlling errors
A.
The goal of any researcher is to minimize Type I and Type II
B.
C.
III.
errors
1.
As Type I errors are minimized, Type II errors are
maximized
2.
As Type II errors are minimized, Type I errors are
maximized
An error can be judged only relative to reality
1.
Reality is never known
2.
This is the reason statisticians work in probabilities
No error exists until a decision is made
1.
The only time a Type I error is made is when H0 is
rejected
a.
The probability of this error is alpha (α)
b.
Protecting against a Type I error
i. Decrease alpha (α) – that is, α=.05 rather
than .10 or α=.01 rather than .10)
ii. As a consequence beta (β) increases
c.
See Figures 1A, 1B, and 1C
2.
The only time a Type II error is made is when H0 is
accepted
a.
The probability of this error is beta (β)
b.
Beta (β) is not as straightforward as alpha (α)
c.
Protecting against a Type II error
i. Increase alpha (α) – that is α=.05 rather
than .01 or α=.10 rather than .05)
ii. As a consequence β decreases
iii. Increase sample size
d.
See Figures 2A and 2B
Power
A.
Correctly rejecting H0 when it is false (i.e., making a correct
decision)
B.
Algebraically noted as (1-β)
C.
Consider an appropriate level of power to be a 4:1 ratio of α to β
D.
Factors affecting power
1.
Directional nature of H0
2.
Level of significance
a.
If you increase alpha (α), you increase power (1β)
b.
If you decrease alpha (α), you decrease power (1β)
3.
Effect size
a.
The difference between the value specified in H0
and the value specified in H1
b.
Increasing effect size separates the two sampling
distributions
E.
Sample size and power
1.
Larger samples mean smaller standard errors which
IV.
effectively reduces the overlap between distributions and
thus power increases
2.
This is the only factor affecting power over which the
researcher has a reasonable level of control
a.
The directional nature of H0 is determined by the
nature of the research question
b.
Alpha level is set a-priori based on the
researcher’s concern for Type I and Type II errors
c.
Effect size is directly related to the success of the
treatment
3.
Two perspectives
a.
A-priori – Tables for sample sizes based on
assumption about effect size, power, and alpha
(See Hinkle, Wiersma, and Jurs or Cohen)
b.
Post-hoc – See specific SPSS Windows output
from procedures like ANOVA, GLM, etc.
Practical significance
A.
Distinguishing statistical and practical significance
1.
The effect of large samples on statistical significance
2.
Concern for the “meaning“ of results
B.
Estimates of practical significance
1.
Strength of association – the proportion of the variance in
the dependent variable that can be explained by the
independent variable
a.
r2
b.
Eta squared
c.
Omega squared
2.
Standardized effect size – a method of expressing mean
differences in standard deviation units
3.
See Table 2
Table 4
Interpreting Selected Measures of Effect Size
Analysis
Measure of Effect Size
Correlations
Means
r
Eta squared
Partial Eta Squared
Omega squared
Partial Omega Squared
Cohen’s f
Mean Difference (d)
Small
.10
.01
.01
.01
.01
.10
.20
Effect Size
Moderate
.30
.06
.06
.06
.06
.25
.50
Large
.50
.14
.14
.14
.14
.40
.80