Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
6. Methods of Experimental Control Chapter 6: Control Problems in Experimental Research 1 Goals • Understand: • Advantages/disadvantages of within- and between-subjects experimental designs • Methods of controlling for group differences in between-subjects experimental designs. • Counter-balancing techniques for controlling sequence effects in within-subjects experimental designs 2 Between-Subjects Experimental Design (single factor, 2 levels) Population Sampling Sample Random Assignment Condition 1 Condition 2 Group 1 Group 2 3 Within-Subjects Experimental Designs (single factor, 2 levels) Population Sampling Sample 1/2 1/2 Condition 1 Condition 2 Sample Sample Sample Sample 4 Between subjects designs • Different groups of people assigned to each level of IV • Requires more participants, but avoids sequence effects (e.g., practice or fatigue). • Potential Validity Problem: Are the people in the groups the same to begin with? • If not, group differences = a confound 5 Simple Random Assignment • • Technique for minimizing group differences. • Simply divide P’s among levels of IV in a random (not arbitrary!) way. • NOT to be confused with random sampling! Works simultaneously on ALL variables that might lead to group differences.Very powerful. 6 An Aside: Randomness • Random is not the same as arbitrary. • Randomness can be thought of as “systematically non-systematic”. That is, you set up a procedure to eliminate any possible biases. • Arbitrary procedures, such as deciding haphazardly who goes in which group, may contain unknown biases 7 An Aside: Randomness • Two flavours of randomness: • Random with replacement: All options are there on every trial (dice, coin tosses) • Random without replacement: When an option is picked on a given trial, it is no longer available for later ones (cards) • Use the latter with random sampling and random assignment 8 Simple Random Assignment • Typically, subjects are “shuffled” randomly using computer-generated random numbers. • • Physical mixing can also be used. Careful! Must use a method that is “random without replacement”. Example: drawing cards from a deck without putting them back in the deck for the next P. Counter-Example: Flipping a coin is right out! That is “random with replacement” 9 Random Assignment in Excel: One IV • List all levels of IV in column A, with n repeats, where n is the # of individuals who will be in each group. • Create a list of random numbers, using =rand() in column B, then sort according to column B (cut and paste values) • If all participants are known ahead of time, just paste the list into column C • If participants are not known ahead of time, test them as they come in, in the order of the list. 10 Random Assignment in Excel: Two IVs • List all levels of IV1 in column A, with n repeats, where n is the # of individuals who will be in each level of IV1. • List all levels of IV2 in column B, with n repeats within each level of IV1, where n is the number of individuals in each condition. • Rest is as for one IV. 11 Discussion / Questions 12 Block Random Assignment • Technique used in between-subjects designs to avoid “clumping” of conditions at particular times. • (Also used in within-subjects designs, but more on this later... ) • In sequential testings, simple RA may create a confound: Example: One might end up doing most of Level 1 before any of Level 2. • Whether such clumping is a problem depends on the likelihood of history confounds, and the size of the sample, but it’s never a bad idea to avoid it. 13 Block Random Assignment • In block RA, the set of all conditions is shuffled several times, and a series of shuffled sets of conditions is created Example: Experiment with three conditions might produce a sequence like 3 1 2 1 3 2 2 1 3 3 2 1 1 3 2 ... Counter-example: With simple RA, same experiment might produce a sequence like 3 3 1 1 3 2 3 3 1 2 2 1 1 2 2 ... • Note that, when using block RA, number of participants should ideally be an even multiple of the number of conditions 14 Example of Block RA • • • E.g., Four conditions with n=10 each. • Block RA: Instead create 10 “block decks” of four cards each: 1♥, 1♦, 1♣, 1♠ in each deck. • Shuffle each block deck, then stack all the block decks on top of one-another. In simple RA: Shuffle 10♥, 10♦, 10♣, 10♠ But might (just by chance) end up drawing most of the ♥’s before any ♣’s are drawn (for example). 15 Block Random Assignment • With simple randomization, you might end up with a sequence like this: ♥ ♥ ♦♦ ♥ ♥ ♥ ♠ ♣ ♥ ♣ ♦ ♠ ♦ ♥ ♠ ♠ ♥ ♠ ♥ ♦ ♠ ♥ ♣ ♠ ♦ ♠ ♣ ♦ ♣ ♣ ♣ ♦ ♣ ♠ ♦ ♠ ♦ ♣ ♣ Start of Study Middle of Study End of Study • But with block randomization, you end up with a sequence like so: ♥♣♦♠ ♠♦♣♥ ♣♥♦♠ ♣♥♦♠ ♥♣♠♦ ♣♠♦♥ ♥♠♦♣ ♦♣♠♥ ♠♦♣♥ ♣♠♥♦ Start of Study Middle of Study End of Study 16 Random Assignment & Number of Participants • RA works well with large N. What is “large”? Ideally 30, but as little as 10 is acceptable for small studies. • But, chance of non-equivalent groups rises as N drops. • What to do if you’re stuck with small N? 17 Matching • Instead of random assignment, test participants on matching variable(s) • Then assign P’s to groups such that groups have equal means (or frequency distributions if MVs are nominal) on the matching variable(s) • Must have theoretical reason to expect an effect of matching variable • Matching variable must be testable practically and without introducing testing effects. 18 Matching • Easiest to do with variabless that can be assessed without lengthy testing Examples: Age, Gender, Weight... • How many factors to match on? Can get complicated. May result in having to turn participants away if no match can be made • May be simpler to test more subjects and let random assignment do its magic. 19 Step 1: Order Values Step 2: Create pairs of adjacent values 9.1 9.0 GPA 8.5 8.0 9.1 7.3 7.1 9.0 7.0 6.8 8.5 6.5 5.5 8.0 7.3 7.1 7.0 6.8 6.5 5.5 Step 3: From each pair, randomly assign one to each group Group 1 Group 2 9.0 9.1 8.5 8.0 7.3 7.1 6.8 7.0 5.5 6.5 µ = 7.42 µ = 7.54 20 Discussion / Questions 21 Instructions: A Between-Subjects Experiment In a moment I’m going to show you a video. It shows 6 people playing basketball. I want you to watch the video and keep a silent mental count of the number of passes between players. But to make it a little more difficult, I want you to keep two separate counts, one for the number of passes through the air, and another for number of bounce-passes, that is, the number of times they bounce the ball to one another. If you’re on the right side of the class (your right, my left), do this for the white-shirted players only. If you’re on the left side of the class (your left, my right), do this for the black-shirted players only. When we’re done, I’ll ask you to write down the two numbers (number of bounce passes and number of air passes) and give the data to me. Remember, just keep a silent count, don’t make any noise or marks with your pen or anything like that. 22 23 What to Take Away From This? • Perception:You don’t actually see what’s out there, just a reconstruction • • Cognition: There are limits to human attentional load • RM&E: Some things can’t be repeated within-subjects Phil. of Science: It pays to observe the same thing several times, sometimes looking at details (=experiment), sometimes looking at the “big picture” (= naturalistic observation) 24 Within-Subjects Experimental Designs Population Sampling Sample 1/2 1/2 Condition 1 Condition 2 Sample Sample Sample Sample 25 Within-Subjects Designs • • • a.k.a. “Repeated measures designs” • Fewer participants needed, no group effects, but may be impractical for some tasks. Same group goes through all levels of the IV. Often used when time to do one condition is small, or when available population is small. 26 Within-subjects Designs • Allows more statistical power • More participants per condition for a given grand N • Don’t have to deal with between-groups variance (even with RA, there’s always some difference between groups that can obscure experimental effects) • BUT, must be careful of sequence effects 27 Sequence Effects • Going through level A of the IV may affect performance on level B. • Progressive Sequence Effects: • Practice effect: Participant gains knowledge, warms up, focuses, etc. • Fatigue effect: Participant gets tired, bored, overwhelmed, etc. 28 Sequence Effects • Carry-over effects: Non-symmetrical sequence effects. Doing Level A then Level B not the same as doing Level B then Level A. • Common when levels vary in difficulty: “simple then hard” is easier than “hard then simple”. • In this case, best to switch to between-subjects design. 29 Counterbalancing • Group of techniques for minimizing progressive sequence effects in within-subjects experiments • • • Complete counterbalancing Partial counterbalancing Sequence randomization Sequence randomization with constraints Latin square • • • General idea is to equalize the number of participants who do each level in each order 30 Complete Counterbalancing • Equal number of participants goes through each possible order of conditions Example: With two condition, half of Ps do 1-then-2, other half do 2-then-1 • The ultimate form of counterbalancing, but not always practical. • Number of orders of levels is “n factorial” or “n!”, where n is number of levels. 31 Factorial N! = N × N-1 × N-2 ... # of Levels # of Orders 2 2 3 6 4 24 5 120 6 720 32 × 1 Factorial • Why is it N! ? • Consider a case where you have 4 chairs and need to seat 4 people. How many people can you choose from to go in the... ...1st chair? 4 ...2nd chair? 3 (because one is in the 1st) ...3rd chair? 2 (other 2 already seated) ...4th chair? 1 33 Discussion / Questions 34 Partial Counterbalancing • Sequence randomization: Necessary when large number of levels, but adds noise. • Sequence randomization with constraints: • Fellows Numbers: Ensure that correct answer is not the same more than X times in a row • Same stimulus does not appear on sequential trials 35 Latin Square • From an ancient roman game: Given an X by X grid, and X different symbols, can you place the symbols in the grid so that each appears only once per row and once per column? Similar to Sudoku puzzles. • Even harder: Can you make it so that each symbol appears directly to the right of each other symbol once and only once? (only possible when X is even). This is called a balanced latin square 36 Latin Square • A completed Balanced Latin Square can be used as a form of partial counter-balancing • Each participant runs through the conditions in the order indicated by one row of the BLS • Number of participants must be evenly divisible by number of levels 37 6x6 Balanced Latin Square Order # Sequential Position 1 2 3 4 5 6 1 A B F C E D 2 B C A D F E 3 C D B E A F 4 D E C F B A 5 E F D A C B 6 F A E B D C 38 Latin Square Design • • If you run one participant through each of the 6 orders, then: • Each of the 6 levels will have been done once in each of the 6 possible sequential positions. • Each of the 6 levels will have been immediately preceded by each of the other 5 levels once and only once. If you run 60 people, then 10 will have gone through each order 39 Latin Square Design • • • • For example, if you test 60 people (6 x 10), then: 10 will have done A first, 10 B first, 10 C first... 10 will have done A second, 10 B second, ... 10 will have done level A immediately preceded by B, 10 will have done A immediately preceded by C, etc... 40 Creating a Balanced Latin Square (if X is even) • Build the first row according to the pattern: A B (x) C (x-1) D (x-2) E (x-3) etc... where x is the highest letter you’re using (e.g., F if doing a 6×6). With 6 levels, row 1 is: ABFCED • Build the remaining rows by incrementing the letters by 1 (i.e., A becomes B, B becomes C...). Row 2 is B CA D F E Note that we “wrap back” to A when incrementing F 41 Latin Square With Odd Number of Levels • • Previous system only works with even # of levels. A 5×5 or 7×7 latin square cannot be balanced. With uneven # of levels, create latin square plus a leftright mirror of it. Run an equal number of participants through each of the orders in these two latin squares 42 Order • Sequential Position 1 2 3 4 5 1 A B E C D 2 B C A D E 3 C D B E A 4 D E C A B 5 E A D B C 1 2 3 4 5 6 D C E B A 7 E D A C B 8 A E B D C 9 B A C E D 10 C B D A E Summary: Counterbalancing • 2-3 levels: Complete counterbalancing • 4-8 levels: Latin square • 4+ levels: Sequence randomization, possibly with constraints 43 Counterbalancing w/ Multiple Exposures • What if subjects experience each condition more than once? • Reverse counterbalancing: • ABCD-DCBA-ABCD-DCBA... • Block randomization: • BADC-CBAD-DCAB-ADCB... • Block randomization with constraints 44 Within vs. Between in Developmental Psych • • Cross-sectional study • • • Faster than following from 5-9 Problem: Cohort effects. Longitudinal study • • • • Between groups: Test 5, 7, 9 years olds Within groups: Follow 5 year olds until 9 years old. Takes a long time! Problem: Attrition Other methods combine the two 45 Summary: Confounds & Controls • Participant differences: random assignment, block randomization, matching • • Order effects: Full & partial counterbalancing • • Experimenter bias: Automation, double-blind procedures Participant bias: Blind procedures. Removal of demand characteristics Floor & ceiling effects: Use procedures that are neither too difficult nor too easy. 46 Think Twice... • • • Carpenter’s adage: “Measure twice, cut once” Scientist’s adage: “Think twice, measure once” Do not rush experimental design, there are many pitfalls to be avoided and careful design will save time in the long run. 47 Discussion / Questions • What is the difficulty with reverse counterbalancing? • What method of counterbalancing would you use for an experiment with 5 levels? 48