Download CLR

6. Methods of
Experimental Control
Chapter 6: Control Problems in Experimental Research
1
Goals
•
Understand:
•
Advantages/disadvantages of within- and
between-subjects experimental designs
•
Methods of controlling for group differences in
between-subjects experimental designs.
•
Counter-balancing techniques for controlling
sequence effects in within-subjects experimental
designs
2
Between-Subjects Experimental Design
(single factor, 2 levels)
Population
Sampling
Sample
Random Assignment
Condition 1
Condition 2
Group 1
Group 2
3
Within-Subjects Experimental Designs
(single factor, 2 levels)
Population
Sampling
Sample
1/2 1/2
Condition 1
Condition 2
Sample
Sample
Sample
Sample
4
Between subjects designs
•
Different groups of people assigned to each
level of IV
•
Requires more participants, but avoids
sequence effects (e.g., practice or fatigue).
•
Potential Validity Problem: Are the people in
the groups the same to begin with?
•
If not, group differences = a confound
5
Simple Random Assignment
•
•
Technique for minimizing group differences.
•
Simply divide P’s among levels of IV in a random
(not arbitrary!) way.
•
NOT to be confused with random sampling!
Works simultaneously on ALL variables that
might lead to group differences.Very powerful.
6
An Aside: Randomness
• Random is not the same as arbitrary.
• Randomness can be thought of as
“systematically non-systematic”. That is,
you set up a procedure to eliminate any
possible biases.
• Arbitrary procedures, such as deciding
haphazardly who goes in which group, may
contain unknown biases
7
An Aside: Randomness
• Two flavours of randomness:
• Random with replacement: All options
are there on every trial (dice, coin tosses)
• Random without replacement: When an
option is picked on a given trial, it is no
longer available for later ones (cards)
• Use the latter with random sampling and
random assignment
8
Simple Random Assignment
•
Typically, subjects are “shuffled” randomly using
computer-generated random numbers.
•
•
Physical mixing can also be used.
Careful! Must use a method that is “random without
replacement”.
Example: drawing cards from a deck without
putting them back in the deck for the next P.
Counter-Example: Flipping a coin is right out!
That is “random with replacement”
9
Random Assignment
in Excel: One IV
•
List all levels of IV in column A, with n repeats, where n is
the # of individuals who will be in each group.
•
Create a list of random numbers, using =rand() in column
B, then sort according to column B (cut and paste values)
•
If all participants are known ahead of time, just paste the
list into column C
•
If participants are not known ahead of time, test them as
they come in, in the order of the list.
10
Random Assignment
in Excel: Two IVs
•
List all levels of IV1 in column A, with n repeats, where n is
the # of individuals who will be in each level of IV1.
•
List all levels of IV2 in column B, with n repeats within each
level of IV1, where n is the number of individuals in each
condition.
•
Rest is as for one IV.
11
Discussion / Questions
12
Block Random Assignment
•
Technique used in between-subjects designs to avoid
“clumping” of conditions at particular times.
•
(Also used in within-subjects designs, but more on this
later... )
•
In sequential testings, simple RA may create a confound:
Example: One might end up doing most of Level 1
before any of Level 2.
•
Whether such clumping is a problem depends on the
likelihood of history confounds, and the size of the
sample, but it’s never a bad idea to avoid it.
13
Block Random Assignment
•
In block RA, the set of all conditions is shuffled several times, and
a series of shuffled sets of conditions is created
Example: Experiment with three conditions might produce a
sequence like
3 1 2 1 3 2 2 1 3 3 2 1 1 3 2 ...
Counter-example: With simple RA, same experiment
might produce a sequence like
3 3 1 1 3 2 3 3 1 2 2 1 1 2 2 ...
•
Note that, when using block RA, number of participants should
ideally be an even multiple of the number of conditions
14
Example of Block RA
•
•
•
E.g., Four conditions with n=10 each.
•
Block RA: Instead create 10 “block decks” of four
cards each: 1♥, 1♦, 1♣, 1♠ in each deck.
•
Shuffle each block deck, then stack all the block
decks on top of one-another.
In simple RA: Shuffle 10♥, 10♦, 10♣, 10♠
But might (just by chance) end up drawing most of
the ♥’s before any ♣’s are drawn (for example).
15
Block Random Assignment
• With simple randomization, you might
end up with a sequence like this:
♥ ♥ ♦♦ ♥ ♥ ♥ ♠ ♣ ♥ ♣ ♦ ♠ ♦ ♥ ♠ ♠ ♥ ♠ ♥ ♦ ♠ ♥ ♣ ♠ ♦ ♠ ♣ ♦ ♣ ♣ ♣ ♦ ♣ ♠ ♦ ♠ ♦ ♣ ♣
Start of Study
Middle of Study
End of Study
• But with block randomization, you end
up with a sequence like so:
♥♣♦♠ ♠♦♣♥ ♣♥♦♠ ♣♥♦♠ ♥♣♠♦ ♣♠♦♥ ♥♠♦♣ ♦♣♠♥ ♠♦♣♥ ♣♠♥♦
Start of Study
Middle of Study
End of Study
16
Random Assignment &
Number of Participants
• RA works well with large N. What is
“large”? Ideally 30, but as little as 10 is
acceptable for small studies.
• But, chance of non-equivalent groups rises
as N drops.
• What to do if you’re stuck with small N?
17
Matching
•
Instead of random assignment, test participants on
matching variable(s)
•
Then assign P’s to groups such that groups have
equal means (or frequency distributions if MVs are
nominal) on the matching variable(s)
•
Must have theoretical reason to expect an effect of
matching variable
•
Matching variable must be testable practically and
without introducing testing effects.
18
Matching
•
Easiest to do with variabless that can be
assessed without lengthy testing
Examples: Age, Gender, Weight...
•
How many factors to match on? Can get
complicated. May result in having to turn
participants away if no match can be made
•
May be simpler to test more subjects and let
random assignment do its magic.
19
Step 1:
Order Values
Step 2: Create pairs of adjacent values
9.1
9.0
GPA
8.5
8.0
9.1
7.3
7.1
9.0
7.0
6.8
8.5
6.5
5.5
8.0
7.3
7.1
7.0
6.8
6.5
5.5
Step 3: From each pair, randomly
assign one to each group
Group 1
Group 2
9.0
9.1
8.5
8.0
7.3
7.1
6.8
7.0
5.5
6.5
µ = 7.42
µ = 7.54
20
Discussion / Questions
21
Instructions:
A Between-Subjects
Experiment
In a moment I’m going to show you a video. It shows 6 people playing basketball. I want you to
watch the video and keep a silent mental count of the number of passes between players. But to
make it a little more difficult, I want you to keep two separate counts, one for the number of
passes through the air, and another for number of bounce-passes, that is, the number of times
they bounce the ball to one another.
If you’re on the right side of the class (your right, my left), do this for the white-shirted
players only.
If you’re on the left side of the class (your left, my right), do this for the black-shirted players
only.
When we’re done, I’ll ask you to write down the two numbers (number of bounce passes and
number of air passes) and give the data to me.
Remember, just keep a silent count, don’t make any noise or marks with your pen or anything like
that.
22
23
What to Take
Away From This?
•
Perception:You don’t actually see what’s out there, just a
reconstruction
•
•
Cognition: There are limits to human attentional load
•
RM&E: Some things can’t be repeated within-subjects
Phil. of Science: It pays to observe the same thing several
times, sometimes looking at details (=experiment),
sometimes looking at the “big picture” (= naturalistic
observation)
24
Within-Subjects Experimental Designs
Population
Sampling
Sample
1/2 1/2
Condition 1
Condition 2
Sample
Sample
Sample
Sample
25
Within-Subjects Designs
•
•
•
a.k.a. “Repeated measures designs”
•
Fewer participants needed, no group effects, but
may be impractical for some tasks.
Same group goes through all levels of the IV.
Often used when time to do one condition is
small, or when available population is small.
26
Within-subjects Designs
• Allows more statistical power
• More participants per condition for a
given grand N
• Don’t have to deal with between-groups
variance (even with RA, there’s always some difference between
groups that can obscure experimental effects)
• BUT, must be careful of sequence effects
27
Sequence Effects
• Going through level A of the IV may affect
performance on level B.
• Progressive Sequence Effects:
• Practice effect: Participant gains
knowledge, warms up, focuses, etc.
• Fatigue effect: Participant gets tired,
bored, overwhelmed, etc.
28
Sequence Effects
•
Carry-over effects: Non-symmetrical sequence
effects. Doing Level A then Level B not the same
as doing Level B then Level A.
•
Common when levels vary in difficulty: “simple
then hard” is easier than “hard then simple”.
•
In this case, best to switch to between-subjects
design.
29
Counterbalancing
•
Group of techniques for minimizing progressive sequence
effects in within-subjects experiments
•
•
•
Complete counterbalancing
Partial counterbalancing
Sequence randomization
Sequence randomization with constraints
Latin square
•
•
•
General idea is to equalize the number of participants who
do each level in each order
30
Complete
Counterbalancing
•
Equal number of participants goes through
each possible order of conditions
Example: With two condition, half of Ps
do 1-then-2, other half do 2-then-1
•
The ultimate form of counterbalancing, but not
always practical.
•
Number of orders of levels is “n factorial” or
“n!”, where n is number of levels.
31
Factorial
N! = N × N-1 × N-2 ...
# of
Levels
# of
Orders
2
2
3
6
4
24
5
120
6
720
32
×
1
Factorial
• Why is it N! ?
• Consider a case where you have 4 chairs
and need to seat 4 people. How many
people can you choose from to go in the...
...1st chair? 4
...2nd chair? 3 (because one is in the 1st)
...3rd chair? 2 (other 2 already seated)
...4th chair? 1
33
Discussion / Questions
34
Partial Counterbalancing
•
Sequence randomization: Necessary when large
number of levels, but adds noise.
•
Sequence randomization with constraints:
•
Fellows Numbers: Ensure that correct answer is
not the same more than X times in a row
•
Same stimulus does not appear on sequential trials
35
Latin Square
•
From an ancient roman game:
Given an X by X grid, and X different symbols, can
you place the symbols in the grid so that each appears
only once per row and once per column? Similar to
Sudoku puzzles.
•
Even harder:
Can you make it so that each symbol appears
directly to the right of each other symbol once and
only once? (only possible when X is even). This is
called a balanced latin square
36
Latin Square
•
A completed Balanced Latin Square can be used
as a form of partial counter-balancing
•
Each participant runs through the conditions in
the order indicated by one row of the BLS
•
Number of participants must be evenly divisible
by number of levels
37
6x6 Balanced Latin Square
Order #
Sequential Position
1
2
3
4
5
6
1
A
B
F
C
E
D
2
B
C
A
D
F
E
3
C
D
B
E
A
F
4
D
E
C
F
B
A
5
E
F
D
A
C
B
6
F
A
E
B
D
C
38
Latin Square Design
•
•
If you run one participant through each of the 6 orders,
then:
•
Each of the 6 levels will have been done once in each of
the 6 possible sequential positions.
•
Each of the 6 levels will have been immediately preceded
by each of the other 5 levels once and only once.
If you run 60 people, then 10 will have gone through each
order
39
Latin Square Design
•
•
•
•
For example, if you test 60 people (6 x 10), then:
10 will have done A first, 10 B first, 10 C first...
10 will have done A second, 10 B second, ...
10 will have done level A immediately preceded by
B, 10 will have done A immediately preceded by C,
etc...
40
Creating a Balanced Latin
Square (if X is even)
•
Build the first row according to the pattern:
A B (x) C (x-1) D (x-2) E (x-3)
etc...
where x is the highest letter you’re using (e.g., F if doing a
6×6). With 6 levels, row 1 is:
ABFCED
•
Build the remaining rows by incrementing the letters by 1
(i.e., A becomes B, B becomes C...). Row 2 is
B CA D F E
Note that we “wrap back” to A when incrementing F
41
Latin Square With Odd
Number of Levels
•
•
Previous system only works
with even # of levels. A 5×5
or 7×7 latin square cannot be
balanced.
With uneven # of levels,
create latin square plus a leftright mirror of it.
Run an equal number of
participants through each of
the orders in these two latin
squares
42
Order
•
Sequential Position
1
2
3
4
5
1
A
B
E
C
D
2
B
C
A
D
E
3
C
D
B
E
A
4
D
E
C
A
B
5
E
A
D
B
C
1
2
3
4
5
6
D
C
E
B
A
7
E
D
A
C
B
8
A
E
B
D
C
9
B
A
C
E
D
10 C
B
D
A
E
Summary:
Counterbalancing
• 2-3 levels: Complete counterbalancing
• 4-8 levels: Latin square
• 4+ levels: Sequence randomization, possibly
with constraints
43
Counterbalancing w/
Multiple Exposures
• What if subjects experience each condition
more than once?
• Reverse counterbalancing:
• ABCD-DCBA-ABCD-DCBA...
• Block randomization:
• BADC-CBAD-DCAB-ADCB...
• Block randomization with constraints
44
Within vs. Between in
Developmental Psych
•
•
Cross-sectional study
•
•
•
Faster than following from 5-9
Problem: Cohort effects.
Longitudinal study
•
•
•
•
Between groups: Test 5, 7, 9 years olds
Within groups: Follow 5 year olds until 9 years old.
Takes a long time!
Problem: Attrition
Other methods combine the two
45
Summary:
Confounds & Controls
•
Participant differences: random assignment, block
randomization, matching
•
•
Order effects: Full & partial counterbalancing
•
•
Experimenter bias: Automation, double-blind procedures
Participant bias: Blind procedures. Removal of demand
characteristics
Floor & ceiling effects: Use procedures that are neither
too difficult nor too easy.
46
Think Twice...
•
•
•
Carpenter’s adage: “Measure twice, cut once”
Scientist’s adage: “Think twice, measure once”
Do not rush experimental design, there are many
pitfalls to be avoided and careful design will save
time in the long run.
47
Discussion / Questions
• What is the difficulty with reverse
counterbalancing?
• What method of counterbalancing would
you use for an experiment with 5 levels?
48