Survey

Survey

Document related concepts

Transcript

lab exam • when: Nov 27 - Dec 1 • length = 1 hour – each lab section divided in two • register for the exam in your section so there is a computer reserved for you • If you write in the 1st hour, you can’t leave early! If you write in the second hour, you can’t arrive late! lab exam • format: – open book! – similar to questions in lab manual – last section in the lab manual has review questions – show all your work: hypotheses, tests of assumptions, test statistics, p-values and conclusions Experimental Design Experimental Design • Experimental design is the part of statistics that happens before you carry out an experiment • Proper planning can save many headaches • You should design your experiments with a particular statistical test in mind Why do experiments? • Contrast: observational study vs. experiments • Example: – Observational studies show a positive association between ice cream sales and levels of violent crime – What does this mean? Why do experiments? • Contrast: observational study vs. experiments • Example: – Observational studies show a positive association between ice cream sales and levels of violent crime – What does this mean? Alternative explanation Ice cream Hot weather Violent crime Alternative explanation Ice cream Correlation is not causation Hot weather Violent crime Why do experiments? • Observational studies are prone to confounding variables: Variables that mask or distort the association between measured variables in a study – Example: hot weather • In an experiment, you can use random assignments of treatments to individuals to avoid confounding variables Goals of Experimental Design • • Avoid experimental artifacts Eliminate bias 1. Use a simultaneous control group 2. Randomization 3. Blinding • Reduce sampling error 1. Replication 2. Balance 3. Blocking Goals of Experimental Design • • Avoid experimental artifacts Eliminate bias 1. Use a simultaneous control group 2. Randomization 3. Blinding • Reduce sampling error 1. Replication 2. Balance 3. Blocking Experimental Artifacts • Experimental artifacts: a bias in a measurement produced by unintended consequences of experimental procedures • Conduct your experiments under as natural of conditions as possible to avoid artifacts Experimental Artifacts • Example: diving birds Goals of Experimental Design • • Avoid experimental artifacts Eliminate bias 1. Use a simultaneous control group 2. Randomization 3. Blinding • Reduce sampling error 1. Replication 2. Balance 3. Blocking Control Group • A control group is a group of subjects left untreated for the treatment of interest but otherwise experiencing the same conditions as the treated subjects • Example: one group of patients is given an inert placebo The Placebo Effect • Patients treated with placebos, including sugar pills, often report improvement • Example: up to 40% of patients with chronic back pain report improvement when treated with a placebo • Even “sham surgeries” can have a positive effect • This is why you need a control group! Randomization • Randomization is the random assignment of treatments to units in an experimental study • Breaks the association between potential confounding variables and the explanatory variables Confounding variable Experimental units Confounding variable Experimental units Treatments Confounding variable Experimental units Treatments Without randomization, the confounding variable differs among treatments Confounding variable Experimental units Treatments Confounding variable Experimental units Treatments With randomization, the confounding variable does not differ among treatments Blinding • Blinding is the concealment of information from the participants and/or researchers about which subjects are receiving which treatments • Single blind: subjects are unaware of treatments • Double blind: subjects and researchers are unaware of treatments Blinding • Example: testing heart medication • Two treatments: drug and placebo • Single blind: the patients don’t know which group they are in, but the doctors do • Double blind: neither the patients nor the doctors administering the drug know which group the patients are in Goals of Experimental Design • • Avoid experimental artifacts Eliminate bias 1. Use a simultaneous control group 2. Randomization 3. Blinding • Reduce sampling error 1. Replication 2. Balance 3. Blocking Replication • Experimental unit: the individual unit to which treatments are assigned Experiment 1 Experiment 2 Tank 1 Tank 2 Experiment 3 All separate tanks Replication • Experimental unit: the individual unit to which treatments are assigned 2 Experimental Units Experiment 1 2 Experimental Units Experiment 2 Tank 1 8 Experimental Units Tank 2 Experiment 3 All separate tanks Replication • Experimental unit: the individual unit to which treatments are assigned 2 Experimental Units 2 Experimental Units Experiment 1 Pseudoreplication Tank 1 8 Experimental Units Experiment 2 Tank 2 Experiment 3 All separate tanks Why is pseudoreplication bad? Experiment 2 Tank 1 Tank 2 • problem with confounding and replication! • Imagine that something strange happened, by chance, to tank 2 but not to tank 1 • Example: light burns out • All four lizards in tank 2 would be smaller • You might then think that the difference was due to the treatment, but it’s actually just random chance Why is replication good? • Consider the formula for standard error of the mean: s SE Y n Larger n Smaller SE Balance • In a balanced experimental design, all treatments have equal sample size Better than Balanced Unbalanced Balance • In a balanced experimental design, all treatments have equal sample size • This maximizes power • Also makes tests more robust to violating assumptions Blocking • Blocking is the grouping of experimental units that have similar properties • Within each block, treatments are randomly assigned to experimental treatments • Randomized block design Randomized Block Design Randomized Block Design • Example: cattle tanks in a field Very sunny Not So Sunny Block 1 Block 2 Block 3 Block 4 What good is blocking? • Blocking allows you to remove extraneous variation from the data • Like replicating the whole experiment multiple times, once in each block • Paired design is an example of blocking Experiments with 2 Factors • Factorial design – investigates all treatment combinations of two or more variables • Factorial design allows us to test for interactions between treatment variables Factorial Design Temperature pH 5.5 6.5 7.5 25 n=2 n=2 n=2 30 n=2 n=2 n=2 35 n=2 n=2 n=2 40 n=2 n=2 n=2 Interaction Effects • An interaction between two (or more) explanatory variables means that the effect of one variable depends upon the state of the other variable Interpretations of 2-way ANOVA Terms 70 Effect of pH and Temperature, No interaction 60 pH 5.5 pH 6.5 pH 7.5 Growth Rate 50 40 30 20 10 0 25 30 35 Temperature 40 Interpretations of 2-way ANOVA Terms 45 40 Effect of pH and Temperature, with interaction 35 pH 5.5 pH 6.5 pH 7.5 Growth Rate 30 25 20 15 10 5 0 25 30 35 Temperature 40 Goals of Experimental Design • • Avoid experimental artifacts Eliminate bias 1. Use a simultaneous control group 2. Randomization 3. Blinding • Reduce sampling error 1. Replication 2. Balance 3. Blocking What if you can’t do experiments? • Sometimes you can’t do experiments • One strategy: – Matching – Every individual in the treatment group is matched to a control individual having the same or closely similar values for known confounding variables What if you can’t do experiments? • Example: Do species on islands change their body size compared to species in mainland habitats? • For each island species, identify a closely related species living on a nearby mainland area Power Analysis • Before carrying out an experiment you must choose a sample size • Too small: no chance to detect treatment effect • Too large: too expensive • We can use power analysis to choose our sample size Power Analysis • Example: confidence interval • For a two-sample t-test, the approximate width of a 95% confidence interval for the difference in means is: 2 precision = 4 n (assuming that the data are a random sample from a normal distribution) Power Analysis • Example: confidence interval • The sample size needed for a particular level of precision is: n = 32 Precision 2 Power Analysis • Assume that the standard deviation of exam scores for a class is 10. I want to compare scores between two lab sections. A. How many exams do I need to mark to obtain a confidence limit for the difference in mean exam scores between two classes that has a width (precision) of 5? n = 32 Precision n = 32 10 5 2 2 =128 Power Analysis • • • • • Example: power Remember, power = 1 - = Pr[Type II error] Typical goal is power = 0.80 For a two-sample t-test, the sample size needed for a power of 80% to detect a difference of D is: n = 16 D 2 Power Analysis • Assume that the standard deviation of exam scores for a class is 10. I want to compare scores between two lab sections. B. How many exams do I need to mark to have sufficient power (80%) to detect a mean difference of 10 points between the sections? n = 16 D 10 n = 16 10 2 2 = 16