Download Experimental Design

lab exam
• when: Nov 27 - Dec 1
• length = 1 hour
– each lab section divided in two
• register for the exam in your section so
there is a computer reserved for you
• If you write in the 1st hour, you can’t leave
early! If you write in the second hour, you
can’t arrive late!
lab exam
• format:
– open book!
– similar to questions in lab manual
– last section in the lab manual has review
questions
– show all your work: hypotheses, tests of
assumptions, test statistics, p-values and
conclusions
Experimental Design
Experimental Design
• Experimental design is the part of statistics
that happens before you carry out an
experiment
• Proper planning can save many
headaches
• You should design your experiments with a
particular statistical test in mind
Why do experiments?
• Contrast: observational study vs.
experiments
• Example:
– Observational studies show a positive
association between ice cream sales and
levels of violent crime
– What does this mean?
Why do experiments?
• Contrast: observational study vs.
experiments
• Example:
– Observational studies show a positive
association between ice cream sales and
levels of violent crime
– What does this mean?
Alternative explanation
Ice cream
Hot weather
Violent
crime
Alternative explanation
Ice cream
Correlation is
not causation
Hot weather
Violent
crime
Why do experiments?
• Observational studies are prone to
confounding variables: Variables that
mask or distort the association between
measured variables in a study
– Example: hot weather
• In an experiment, you can use random
assignments of treatments to individuals to
avoid confounding variables
Goals of Experimental Design
•
•
Avoid experimental artifacts
Eliminate bias
1. Use a simultaneous control group
2. Randomization
3. Blinding
•
Reduce sampling error
1. Replication
2. Balance
3. Blocking
Goals of Experimental Design
•
•
Avoid experimental artifacts
Eliminate bias
1. Use a simultaneous control group
2. Randomization
3. Blinding
•
Reduce sampling error
1. Replication
2. Balance
3. Blocking
Experimental Artifacts
• Experimental artifacts: a bias in a
measurement produced by unintended
consequences of experimental procedures
• Conduct your experiments under as
natural of conditions as possible to avoid
artifacts
Experimental Artifacts
• Example: diving birds
Goals of Experimental Design
•
•
Avoid experimental artifacts
Eliminate bias
1. Use a simultaneous control group
2. Randomization
3. Blinding
•
Reduce sampling error
1. Replication
2. Balance
3. Blocking
Control Group
• A control group is a group of subjects left
untreated for the treatment of interest but
otherwise experiencing the same
conditions as the treated subjects
• Example: one group of patients is given an
inert placebo
The Placebo Effect
• Patients treated with placebos, including
sugar pills, often report improvement
• Example: up to 40% of patients with
chronic back pain report improvement
when treated with a placebo
• Even “sham surgeries” can have a positive
effect
• This is why you need a control group!
Randomization
• Randomization is the random assignment
of treatments to units in an experimental
study
• Breaks the association between potential
confounding variables and the explanatory
variables
Confounding variable
Experimental units
Confounding variable
Experimental units
Treatments
Confounding variable
Experimental units
Treatments
Without
randomization,
the confounding
variable differs
among
treatments
Confounding variable
Experimental units
Treatments
Confounding variable
Experimental units
Treatments
With
randomization,
the confounding
variable does
not differ among
treatments
Blinding
• Blinding is the concealment of information
from the participants and/or researchers
about which subjects are receiving which
treatments
• Single blind: subjects are unaware of
treatments
• Double blind: subjects and researchers
are unaware of treatments
Blinding
• Example: testing heart medication
• Two treatments: drug and placebo
• Single blind: the patients don’t know which
group they are in, but the doctors do
• Double blind: neither the patients nor the
doctors administering the drug know which
group the patients are in
Goals of Experimental Design
•
•
Avoid experimental artifacts
Eliminate bias
1. Use a simultaneous control group
2. Randomization
3. Blinding
•
Reduce sampling error
1. Replication
2. Balance
3. Blocking
Replication
• Experimental unit: the individual unit to
which treatments are assigned
Experiment 1
Experiment 2
Tank 1
Tank 2
Experiment 3
All separate tanks
Replication
• Experimental unit: the individual unit to
which treatments are assigned
2 Experimental
Units
Experiment 1
2 Experimental
Units
Experiment 2
Tank 1
8 Experimental
Units
Tank 2
Experiment 3
All separate tanks
Replication
• Experimental unit: the individual unit to
which treatments are assigned
2 Experimental
Units
2 Experimental
Units
Experiment 1
Pseudoreplication
Tank 1
8 Experimental
Units
Experiment 2
Tank 2
Experiment 3
All separate tanks
Why is pseudoreplication bad?
Experiment 2
Tank 1
Tank 2
• problem with confounding and replication!
• Imagine that something strange happened, by
chance, to tank 2 but not to tank 1
• Example: light burns out
• All four lizards in tank 2 would be smaller
• You might then think that the difference was due
to the treatment, but it’s actually just random
chance
Why is replication good?
• Consider the formula for standard error of
the mean:
s
SE Y 
n
Larger n
Smaller SE
Balance
• In a balanced experimental design, all
treatments have equal sample size
Better than
Balanced
Unbalanced
Balance
• In a balanced experimental design, all
treatments have equal sample size
• This maximizes power
• Also makes tests more robust to violating
assumptions
Blocking
• Blocking is the grouping of experimental
units that have similar properties
• Within each block, treatments are
randomly assigned to experimental
treatments
• Randomized block design
Randomized Block Design
Randomized Block Design
• Example: cattle tanks in a field
Very sunny
Not So Sunny
Block 1
Block 2
Block 3
Block 4
What good is blocking?
• Blocking allows you to remove extraneous
variation from the data
• Like replicating the whole experiment
multiple times, once in each block
• Paired design is an example of blocking
Experiments with 2 Factors
• Factorial design – investigates all
treatment combinations of two or more
variables
• Factorial design allows us to test for
interactions between treatment variables
Factorial Design
Temperature
pH
5.5
6.5
7.5
25
n=2
n=2
n=2
30
n=2
n=2
n=2
35
n=2
n=2
n=2
40
n=2
n=2
n=2
Interaction Effects
• An interaction between two (or more)
explanatory variables means that the
effect of one variable depends upon the
state of the other variable
Interpretations of 2-way ANOVA
Terms
70
Effect of pH and Temperature,
No interaction
60
pH 5.5
pH 6.5
pH 7.5
Growth Rate
50
40
30
20
10
0
25
30
35
Temperature
40
Interpretations of 2-way ANOVA
Terms
45
40
Effect of pH and Temperature,
with interaction
35
pH 5.5
pH 6.5
pH 7.5
Growth Rate
30
25
20
15
10
5
0
25
30
35
Temperature
40
Goals of Experimental Design
•
•
Avoid experimental artifacts
Eliminate bias
1. Use a simultaneous control group
2. Randomization
3. Blinding
•
Reduce sampling error
1. Replication
2. Balance
3. Blocking
What if you can’t do experiments?
• Sometimes you can’t do experiments
• One strategy:
– Matching
– Every individual in the treatment group is
matched to a control individual having the
same or closely similar values for known
confounding variables
What if you can’t do experiments?
• Example: Do species on islands change
their body size compared to species in
mainland habitats?
• For each island species, identify a closely
related species living on a nearby
mainland area
Power Analysis
• Before carrying out an experiment you
must choose a sample size
• Too small: no chance to detect treatment
effect
• Too large: too expensive
• We can use power analysis to choose our
sample size
Power Analysis
• Example: confidence interval
• For a two-sample t-test, the approximate
width of a 95% confidence interval for the
difference in means is:
2
precision = 4 
n
(assuming that the data are a random
sample from a normal distribution)
Power Analysis
• Example: confidence interval
• The sample size needed for a particular
level of precision is:

n = 32 Precision
2
Power Analysis
• Assume that the standard deviation of exam scores for a class is 10.
I want to compare scores between two lab sections. A. How many
exams do I need to mark to obtain a confidence limit for the
difference in mean exam scores between two classes that has a
width (precision) of 5?

n = 32 Precision
n = 32
10
5
2
2
=128
Power Analysis
•
•
•
•
•
Example: power
Remember, power = 1 - 
 = Pr[Type II error]
Typical goal is power = 0.80
For a two-sample t-test, the sample size
needed for a power of 80% to detect a
difference of D is:

n = 16
D
2
Power Analysis
• Assume that the standard deviation of exam scores for a class is 10.
I want to compare scores between two lab sections. B. How many
exams do I need to mark to have sufficient power (80%) to detect a
mean difference of 10 points between the sections?

n = 16
D
10
n = 16
10
2
2
= 16