Survey

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Transcript

Introduction to Chance Models (Section 1.1) Introduction A key step in the Statistical Investigation Method is drawing conclusions beyond the observed data. Statisticians often call this “statistical inference.” There are four main types of conclusions (inferences) that statisticians can draw from data: significance, estimation, generalization, and causation. In the remainder of this chapter we will focus on statistical significance. If you think back to the Organ Donor study from the Preliminaries, there were three groups: those in the neutral group were asked to make a yes/no choice about becoming a donor; those in the “opt in” group were told their default was not to be a donor, but they could choose to become a donor; and those in the “opt out” group were told their default was they were a donor, but they could choose not to be one if they wished. Let’s further examine two of those groups. We saw that 23 of 55 or 41.8% in the “opt in” group elected to become donors, compared with 44/56 or 78.6% in the neutral group. A key question here is whether we believe that the 78.6% is far enough away from the 41.8% to be considered statistically significant, meaning unlikely to have occurred by random chance alone. True, 78.6% looks very different from 41.8%, but it is at least possible that the wording of the solicitation to donate actually makes no difference, and that the difference we observed happened by random chance. To answer the question “Is our result unlikely to happen by random chance?,” our general strategy will be to consider what we expect the results to look like if any differences we are seeing are solely due to random chance. Exploring the random chance results is critical to our ability to draw meaningful conclusions from data. In this section we will provide a framework for assessing random chance explanations. Example 1.1: Can Dolphins Communicate? A famous study from the 1960s explored whether two dolphins (Doris and Buzz) could communicate abstract ideas. Researchers believed dolphins could communicate simple feelings like “Watch out!” or “I’m happy,” but Dr. Jarvis Bastian wanted to explore whether they could also communicate in a more abstract way, much like humans do. To investigate this, Dr. Bastian spent many years training Doris and Buzz and exploring the limits of their communicative ability. During a training period lasting many months, Dr. Bastian placed buttons underwater on each end of a large pool—two buttons for Doris and two buttons for Buzz. He then used an old automobile headlight as his signal. When he turned on the headlight and let it shine steadily, he intended for this signal to mean “push the button on the right.” When he let the headlight blink on and off, this was meant as a signal to “push the button on the left.” Every time the dolphins pushed the correct button, Dr. Bastian gave the dolphins a reward of some fish. Over time Doris and Buzz caught on and could earn their fish reward every time. Then Dr. Bastian made things a bit harder. Now, Buzz had to push his button before Doris. If they didn’t push the buttons in the correct order—no fish. After a bit more training, the dolphins caught on again and could earn their fish reward every time. The dolphins were now ready to participate in the real study to examine whether they could communicate with each other. Dr. Bastian placed a large canvas curtain in the middle of the pool. (See Figure 1.1.) Doris was on one side of the curtain and could see the headlight, whereas Buzz was on the other side of the curtain and could not see the headlight. Dr. Bastian turned on the headlight and let it shine June 27, 2014 MAA PREP workshop 1 steadily. He then watched to see what Doris would do. After looking at the light, Doris swam near the curtain and began to whistle loudly. Shortly after that, Buzz whistled back and then pressed the button on the right—he got it correct and so both dolphins got a fish. But this single attempt was not enough to convince Dr. Bastian that Doris had communicated with Buzz through her whistling. Dr. Bastian repeated the process several times, sometimes having the light blink (so Doris needed to let Buzz know to push the left button) and other times having it glow steadily (so Doris needed to let Buzz know to push the right button). He kept track of how often Buzz pushed the correct button. In this scenario, even if Buzz and Doris can communicate, we don’t necessarily expect Buzz to push the correct button every time. We allow for some “randomness” in the process; maybe on one trial Doris was a bit more underwater when she whistled and the signal wasn’t as clear for Buzz. Or maybe Buzz and Doris aren’t communicating at all and Buzz guesses which button to push every time and just happens to guess correctly once in a while. Our goal is to get an idea of how likely Buzz is to push the correct button in the long run. Figure 1.1: Depending whether or not the light was blinking or shown steadily, Doris had to communicate to Buzz as to which button to push. Let’s see how Dr. Bastian was applying the Six-Step Statistical Investigation Method. Step 1: Ask a research question. Can dolphins communicate in an abstract manner? Step 2: Design a study and collect data. Notice Dr. Bastian took some time to train the dolphins in order to get them to a point where he could test a specific research conjecture. The research conjecture is that Buzz pushes the correct button more often than he would if he and Doris could not communicate. If Buzz and Doris could not communicate, Buzz would just be guessing which button to push. The observational units are Buzz’s attempts and the variable for each attempt is whether or not on each attempt, Buzz pushes the correct button (a categorical variable). Step 3: Explore the data. In one phase of the study, Dr. Bastian had Buzz attempt to push the correct button a total of 16 different times. In this sample of 16 attempts, Buzz pushed the correct button 15 out of 16 times. To summarize these results, we report the statistic, a numerical summary of the sample. For this example, we could report either 15, the number of correct pushes, or 15/16 = 0.9375, as the statistic. Definitions: The set of observational units on which we collect data is called the sample. The number of observational units in the sample is the sample size. A statistic is a number summarizing the results in the sample. June 27, 2014 MAA PREP workshop 2 The sample size in this example is 16. Note that the word “sample” is used as both a noun (the set of observational units being studied) and as an adjective, for example to mean “computed from the observed data,” as, for example, “sample statistic.” Step 4: Draw inferences beyond the data. These 16 observations are a mere snapshot of Buzz’s overall selection process. We will consider this a random process. We are interested in Buzz’s actual long-run probability of pushing the correct button based on Doris’ whistles. This unknown long-run probability is called a parameter. Definition: For a random process, a parameter is a long-run numerical property of the process. Note that we are assuming this parameter is not changing over time, at least for the process used by Buzz in this phase of the study. Because we can’t observe Buzz pushing the button forever, we need to draw conclusions (possibly incorrect, but hopefully not) about the value of the parameter based only on these 16 attempts. Buzz certainly pushed the correct button most of the time, so we might consider either: Buzz is doing something other than just guessing (his probability of a correct button push is larger than 0.50). Buzz is just guessing (his probability of a correct button push is 0.50) and he got lucky in these 16 attempts. These are the two possible explanations to be evaluated. Because we can’t collect more data, we have to base our conclusions only on the data we have. It’s certainly possible that Buzz was just guessing and got lucky! But does this seem like a reasonable explanation to you? How would you argue against someone who thought this was the case? Think about it: Based on these data, do you think Buzz somehow knew which button to push? Is 15 out of 16 correct pushes convincing to you? Or do you think that Buzz could have just been guessing? How might you justify your answer? So how are we going to decide between these two possible explanations? One approach is to choose a model for the random process (repeated attempts to push the correct button) and then see whether our model is consistent with the observed data. If it is, then we will conclude that we have a reasonable model and we will use that model to answer our questions. The Chance Model Scientists use models to help understand complicated real world phenomena. Statisticians often employ chance models to generate data from random processes to help them investigate such processes. You did this with the Monty Hall Exploration (P.3) to investigate properties of the two strategies, switching and staying with your original choice of door. In that exploration it was clear how the underlying chance process worked, even though the probabilities themselves were not obvious. But here we don’t know for sure what the underlying real world process is. We are trying to decide whether the process could be Buzz simply guessing or whether the process is something else, such as Buzz and Doris being able to communicate. Let us first investigate the “Buzz was simply guessing” process. Because Buzz is choosing between two options, the simplest chance model to consider is a coin flip. We can flip a coin to represent or simulate Buzz’s choice assuming he is just guessing which button to push. To June 27, 2014 MAA PREP workshop 3 generate this artificial data, we can let “heads” represent the outcome that Buzz pushes the correct button and let “tails” be the outcome that Buzz pushes the incorrect button. This gives Buzz a 50% chance of pushing the correct button. This can be used to represent the “Buzz was just guessing” or the “random chance alone” explanation. The correspondence between the real study and the physical simulation is shown in Table 1.1. Table 1.1: Parallels between real study and physical simulation Coin flip Heads Tails = = = guess by Buzz correct guess wrong guess probability of correct button when Buzz is just Chance of Heads = 1/2 = guessing One repetition = one set of 16 simulated attempts by Buzz Now that we see how flipping a coin can simulate Buzz guessing, let’s flip some coins to simulate Buzz’s performance. Imagine that we get heads on the first flip. What does this mean? This would correspond to Buzz pushing the correct button! But, why did he push the correct button? In this chance model, the only reason he pushed the correct button is because he happened to guess correctly—remember the coin is simulating what happens when Buzz is just guessing which button to push. What if we keep flipping the coin? Each time we flip the coin we are simulating another attempt where Buzz guesses which button to push. Remember that heads represents Buzz guessing correctly and tails represents Buzz guessing incorrectly. How many times do we flip the coin? Sixteen, to match Buzz’s 16 attempts in the actual study. After 16 tosses, Figure 1.2 shows our simulated data. Figure 1.2: Results from tossing a coin 16 times Here we got 11 heads and 5 tails (11 out of 16 or 0.6875 is the simulated statistic). This gives us an idea of what could have happened in the study if Buzz had been randomly guessing which button to push each time. Will we get this same result every time we flip a coin 16 times? Let’s flip our coin another 16 times and see what happens. When we did this we got 7 heads and 9 tails as shown in Figure 1.3 (7 out of 16 or 0.4375 is the simulated statistic). Figure 1.3: Results from a second repetition of tossing a coin 16 times So can we learn anything from these coin tosses when the results vary between the sets of 16 tosses? Using and evaluating the coin flip chance model Because coin flipping is a random process, we know that we won’t obtain the same number of heads with every set of 16 flips. But are some numbers of heads more likely than others? If we June 27, 2014 MAA PREP workshop 4 continue our repetitions of 16 tosses, we can start to see how the outcomes for the number of heads are distributed. Does the distribution of the number of heads that result in 16 flips have a predictable long-run pattern? In particular, how much variability is there in our simulated statistics between repetitions (sets of 16 flips) just by random chance? In order to investigate these questions, we need to continue to flip our coin to get many, many sets of 16 flips (or many repetitions of the 16 choices where we are modeling Buzz simply guessing each time). We did this, and Figure 1.4 shows what we found when we graphed the number of heads from each set of 16 coin flips. Here, the process of flipping a coin 16 times was repeated 1,000 times—this number was chosen for convenience, but also appears to be large enough to give us a fairly accurate sense of the long-run behavior for the number of heads in 16 tosses. Figure 1.4: A dotplot showing 1000 repetitions of flipping a coin 16 times and counting the number of heads Each dot represents one set of 16 attempts by Buzz Let’s think carefully about what the graph in Figure 1.4 shows. For this graph, each dot represents the number of heads in one set of 16 coin tosses. We see that the resulting number of heads follows a clear pattern: 7, 8, and 9 heads happened quite a lot, 10 was pretty common also (though less so than 8), 6 happened some of the time, 1 happened once. But we never got 15 heads in any set of 16 tosses! We might consider any outcome between about 5 and 11 heads to be typical, but getting fewer than 5 heads or more than 11 heads happened rarely enough we can consider it a bit unusual. We refer to these unusual results as being out in the ‘tails’ of the distribution. Think about it: How does the analysis above help us address the strength of evidence for our research conjecture that Buzz was doing something other than just guessing? What does this have to do with the dolphin communication study? We said that we would flip a coin to simulate what could happen if Buzz was really just guessing each time he pushed the button in 16 attempts. We saw that getting results like 15 heads out of 16 never happened in our 1000 repetitions. This shows us that 15 is a very unusual outcome— far out in the tail of the distribution of the simulated statistics—if Buzz is guessing. In short, even though we expect some variability in the results for different sets of 16 tosses, the pattern shown in this distribution indicates that an outcome of 15 heads is outside the typical chance variability we would expect to see when Buzz is simply guessing. June 27, 2014 MAA PREP workshop 5 In the actual study, Buzz really did push the correct button 15 times out of 16, an outcome that we just determined would rarely occur if Buzz was just guessing. So, our coin flip chance model tells us that we have very strong evidence that Buzz was not just tossing a coin to make his choices. This means we have strong evidence that Buzz wasn’t just guessing. Therefore, we don’t believe the “by chance alone” explanation is a good one for Buzz. The results mean we have strong enough evidence to be statistically significant, not a result that happened by chance alone, and to convince us that something other than “random chance” was at play. Definition: A result is statistically significant if it is unlikely to occur just by random chance. If our observed result appears to be consistent with the chance model, we say that the chance model is plausible or believable. What next? A glimpse into Steps 5 and 6 The steps we went through above have helped us evaluate how strong the evidence is that Buzz is not guessing (Step 4 of the Statistical Investigation Method). In this case, the evidence provided by this sample is fairly strong that Buzz isn’t guessing. Still, there are some important questions you should be asking right now, such as: If Buzz isn’t guessing what is he doing? Step 5: Formulate conclusions. We should also ask ourselves: if Buzz wasn’t guessing, does this prove that Buzz and Doris can communicate? And if so, what does this say about other dolphins? As we’ll find in later chapters, the answers to these questions hinge mainly on how the study was designed and how we view the 16 attempts that we observed (e.g., we assume Buzz couldn’t see the light himself, the light signal displayed each time was chosen randomly so Buzz couldn’t figure out a pattern to help him decide which button to push; Buzz’s 16 attempts are a good representation of what Buzz would do given many more attempts under identical conditions; but we might still wonder whether Buzz’s behavior representative of dolphin behavior in general or are there key differences among individual dolphins). Step 6: Look back and ahead. After completing Steps 1-5 of the Statistical Investigation Method, we need to revisit the big picture of the initial research question. First, we reflect on the limitations of the analysis, and think about future studies. In short, we are now stepping back and thinking about the initial research question more than the specific research conjecture being tested in the study. In some ways, this is the most important step of the whole study because it is where we think about the true implications of the scientific study we’ve conducted. For this study, we would reflect on Dr. Bastian’s methods, summarize the results for Buzz, and reflect on ways to improve the study to enhance the conclusions we can draw. The 3S Strategy Let us summarize the overall approach to assessing statistical significance that we have been taking in this section. We observed a sample statistic (e.g., the number of “successes” or the proportion of “successes” in the sample). Then we simulated ‘could-have-been’ outcomes for that statistic under a specific chance model (just guessing). Then we used the information we gained about the random variation in the ‘by-chance’ values of the statistics to help us judge whether the observed value of the statistic is an unusual or a typical outcome. If it is unusual– we say the observed statistic is statistically significant—it provides strong evidence that the chance-alone explanation is wrong. If it is typical, we consider the chance model plausible. You may have noticed that we only simulated results for one specific model. When we saw that the sample statistic observed in the study was not consistent with these simulated results, we June 27, 2014 MAA PREP workshop 6 rejected the chance-alone explanation. Often, research analyses stop here. Instead of trying to simulate results from other models (in particular we may not really don’t have an initial idea what a more appropriate model might be), we are content to say there is something other than random chance at play here. This might lead the researchers to reformulate their conjectures and collect more data in order to investigate different models. We will call the process of simulating could-have-been statistics under a specific chance model the 3S Strategy. After forming our research conjecture and collecting the sample data, we will use the 3S strategy to weigh the evidence against the chance model. This 3S Strategy will serve as the foundation for addressing the question of statistical significance in Step 4 of the Statistical Investigation Method. 3S Strategy for Measuring Strength of Evidence 1. Statistic: Compute the statistic from the observed sample data. 2. Simulate: Identify a “by chance alone” explanation for the data. Repeatedly simulate values of the statistic that could have happened when the chance model is true. 3. Strength of evidence: Consider whether the value of the observed statistic from the research study is unlikely to occur if the chance model is true. If we decide the observed statistic is unlikely to occur by chance alone, then we can conclude that the observed data provide strong evidence against the plausibility of the chance model. If not, then we consider the chance model to be a plausible (believable) explanation for the observed data; in other words what we observed could plausibly have happened just by random chance. Let’s illustrate how we implemented the 3S Strategy for the Doris and Buzz example. 1. Statistic: Our observed statistic was 15, the number of times Buzz pushed the correct button in 16 attempts. 2. Simulate: If Buzz was actually guessing, the parameter (the probability he would push the correct button) would equal 0.5. In other words, he would push the correct button 50% of the time in the long run. We used a coin flip to model what could have happened in 16 attempts when Buzz is just guessing. We flip the coin 16 times and count how many of the 16 flips are heads, meaning how many times Buzz pressed the correct button (“success”). We then repeat this process many more times, each time keeping track of the number of the 16 attempts that Buzz pushed the correct button. We end up with a distribution of could-have-been statistics representing typical values for the number of correct pushes when Buzz is just guessing. 3. Strength of evidence: Because 15 successes of 16 rarely happens by chance alone, we conclude that we have strong evidence that, in the long-run, Buzz is not just guessing. Notice that we have used the result of 15 out of 16 correct attempts to infer that Buzz’s actual long-run probability of pushing the correct button was not simply 0.5. June 27, 2014 MAA PREP workshop 7 Another Doris and Buzz study One goal of statistical significance is to rule out random chance as a plausible (believable) explanation for what we have observed. We still need to worry about how well the study was conducted. For example, are we absolutely sure Buzz couldn’t see the headlight around the curtain? Are we sure there was no pattern to which headlight setting was displayed that he might have detected? And of course we haven’t completely ruled out random chance, he may have had an incredibly lucky day. But the chance of him being that lucky is so small, that we conclude that other explanations are more plausible or credible. One option that Dr. Bastian pursued was to re-do the study except now he replaced the curtain with a wooden barrier between the two sides of the tank in order to ensure a more complete separation between the dolphins to see whether that would diminish the effectiveness of their communication. Step1: Ask a research question. The research question remains the same: Can dolphins communicate in a deep abstract manner? Step 2: Design a study and collect data. The study design is similar with some adjustments to the barrier between Doris and Buzz. The canvas curtain is replaced by a plywood board. The research conjecture, observational units, and variable remain the same. In this case, Buzz pushed the correct button only 16 out of 28 times. The variable is the same (whether or not Buzz pushed the correct button), but the number of observational units (sample size) has changed to 28 (the number of attempts). Think about it: Based on the results for this phase of the study, do you think that Doris could tell Buzz which button to push, even under these conditions? Or is it believable that Buzz could have just been guessing? Step 3: Explore the data. So our observed statistic is 16 out of 28 correct attempts, which is 16/28 × 100% = 57.1% of Buzz’s attempts. This is again more than half the time, but not much larger than 50%. A simple bar graph of these results is shown in Figure 1.5. Figure 1.5: Bar graph for Buzz’s 28 attempts June 27, 2014 MAA PREP workshop 8 Step 4: Draw inferences. Is it plausible (believable) that Buzz was simply guessing in this set of attempts? How do we measure how much evidence these results provide against the chance model? Let’s use the same chance model as we used earlier to see what could have happened if Buzz was just guessing. We will apply the 3S Strategy to this new study. 1. Statistic: The new observed sample statistic is 16 out of 28, or about 0.571. Think about it: Consider again our simulation of the chance model assuming Buzz is guessing. What do we need to change for this new phase of the study? 2. Simulation: This time we need to do repetitions of 28 coin flips, not just 16. A distribution of the number of heads in 1000 repetitions of 28 coin flips is shown in Figure 1.6. This models 1000 repetitions of 28 attempts with Buzz randomly pushing one of the buttons (guessing) each time. Figure 1.6: A graph showing 1000 repetitions of flipping a coin 28 times and counting the number of heads. This models the number of correct pushes in 28 attempts when Buzz is guessing each time. 3. Strength of evidence: Now we need to consider the new observed statistic (16 out of 28 or 0.571). We see from the graph that 16 out of 28 is a fairly typical outcome if Buzz is just randomly guessing. What does this tell us? It tells us that the results of this study are something that could easily have happened if Buzz was just randomly guessing. So what can we conclude? We can say his 16 successes are not convincing evidence against the “by chance alone” model. The graph in Figure 1.6 shows what happens for the hypothetical Buzz who just guesses. An actual outcome far out in the tail of that distribution would be strong evidence against the “just guessing” hypothesis. But be careful: The opposite result – an actual outcome near the center – is not strong evidence in support of the guessing hypothesis. Yes, the result is consistent with that hypothesis, but it is also consistent with many other hypotheses as well. Bottom line: In this second study we conclude that there is not enough evidence that the “by chance alone” model is wrong. That model is still a plausible explanation for the statistic we observed in the study (16 out of 28). Based on this set of attempts, we do not have convincing evidence against the possibility that Buzz is just guessing, but other explanations also remain plausible. For example, the results are consistent with very weak communication between the dolphins. All we know from this analysis is that one plausible explanation for the observed data is that Buzz was guessing. June 27, 2014 MAA PREP workshop 9 In fact, Dr. Bastian soon discovered that in this set of attempts, the equipment malfunctioned and the food dispenser for Doris did not operate and so Doris was not receiving her fish rewards during the study. Because of this malfunction, it’s not so surprising that removing the incentive hindered the communication between the dolphins and we cannot refute that Buzz was just guessing for these attempts. Dr. Bastian fixed the equipment and ran the study again. This time he found convincing evidence that Buzz was not guessing. For a bit more discussion on processes and parameters, see FAQ 1.1 Exploration 1.1: Can Dogs Understand Human Cues? Dogs have been domesticated for about 14,000 years. In that time, have they been able to develop an understanding of human gestures such as pointing or glancing? How about similar non-human cues? Researchers Udell, Giglio, and Wynne tested a small number of dogs in order to answer these questions. In this exploration, we will first see whether dogs can understand human gestures as well as non-human gestures. To test this, the researchers positioned the dogs about 2.5 meters from the experimenter. On each side of the experimenter were two cups. The experimenter would perform some sort of gesture (pointing, bowing, looking) toward one of the cups or there would be some other non-human gesture (a mechanical arm pointing, a doll pointing, or a stuffed animal looking) toward one of the cups. The researchers would then see whether the dog would go to the cup that was indicated. There were six dogs tested. We will look at one of the dogs in two of his sets of trials. This dog, a 4 year-old mixed breed, was named Harley. Each trial involved one gesture and one pair of cups, with a total of ten trials in a set. We will start out by looking at one set of trials where the experimenter bowed toward one of the cups to see whether Harley would go to that cup. Step 1. State the research question. 1. Based on the description of the study, state the research question. Step 2. Design a study and collect data. Harley was tested 10 times and 9 of those times he chose the correct cup. 2. What are the observational units? 3. Identify the variable in the study. What are the possible outcomes of this variable? Is this variable quantitative or categorical? June 27, 2014 MAA PREP workshop 10 Step 3. Explore the data. Definitions: The set of observational units on which we collect data is called the sample. The number of observational units in the sample is the sample size. A statistic is a number summarizing the results in the sample. With categorical data, we typically report the number of “successes” or the proportion of successes as the statistic. 4. What is the number of observational units (sample size) in this study? 5. Determine the observed statistic and produce a simple bar graph of the data (have one bar for the proportion of times Harley picked the correct cup, and another for the proportion of times he picked the wrong cup) 6. If the research conjecture is that Harley can understand what the experimenter means when they bow toward an object, is the statistic in the direction suggested by the research conjecture? 7. Could Harley have gotten 9 out of 10 correct even if he really didn’t understand the human gesture and so was randomly guessing between the two cups? 8. Do you think it is likely Harley would have gotten 9 out of 10 correct if he was just guessing randomly each time? Step 4. Draw inferences beyond the data. There are two possibilities for why Harley chose the correct cup nine out of ten times: He is merely picking a cup at random, and in these 10 trials happened to guess correctly in 9 of them. That is, he got more than half correct just by random chance alone. He is doing something other than merely guessing and perhaps understands what the experimenter means when they bow towards the cup. The unknown long-run probability that Harley will chose the correct cup is called a parameter. Definition: For a random process, a parameter is a long-run numerical property of the process. June 27, 2014 MAA PREP workshop 11 We don’t know the value of the parameter, but the two possibilities listed above suggest two different possibilities. 9. What is the value of the parameter if Harley is picking a cup at random? Give a specific value. 10. What is the possible range of values (greater than or less than some value) for the parameter Harley is not just guessing and perhaps understands the experimenter? We will know show you how statisticians use simulation to make a statement about the strength of evidence for these two possible statements about the parameter’s value. The Chance Model Statisticians often use chance models to generate data from random processes to help them investigate the process. In particular, they can see whether the observed statistic is consistent with the values of the statistic simulated by the chance model. If we determine that Harley’s results are not consistent with the results from the chance model, we will consider this to be evidence against the chance model and in favor of the research conjecture, that he understands the bowing gesture. In this case, we would say Harley’s results are statistically significant, meaning unlikely to have occurred by chance alone. We can’t perform the actual study more times in order to assess the second possibility, but we can simulate the behavior of Harley’s choices if we were to assume the first possibility (that he is simply guessing every time). 11. Explain how you could use a coin toss to represent Harley’s choices if he is guessing between the two cups each time. How many times do you have to flip the coin to represent one set of Harley’s attempts? What does heads represent? 12. If Harley was guessing randomly each time, on average, how many out of the 10 times would you expect him to choose the correct cup? 13. Simulate one repetition of Harley guessing randomly by flipping a coin 10 times (why 10?) and letting heads represent selecting the correct cup (“success”) and tails represent selecting the incorrect cup (“failure”). Count the number of heads in your 10 flips. Combine your results with the rest of the class to create a dotplot of the distribution for the number of heads out of 10 flips of a coin. a. Where does 9 heads fall in the distribution? Would you consider it an unusual outcome or a fairly typical outcome for the number of heads in 10 flips? June 27, 2014 MAA PREP workshop 12 b. Based on your answer to the previous question, do you think it is plausible (believable) that Harley was just guessing which cup to choose? Using an applet to simulate flipping a coin many times To really assess the typical values for the number of heads in 10 coin tosses (number of correct picks by Harley assuming he is guessing at random), we need to simulate many more outcomes of the chance model. Open the One Proportion applet from the textbook webpage. Notice that the probability of heads has been set to be 0.50, representing the guessing between the two cups. Set the number of tosses to 10 and press the Toss Coins button. What was the resulting number of heads? Notice that the number of heads in this set of 10 tosses is then displayed by a dot on the graph. Uncheck the Animate box and press the Toss Coins button 9 more times. This will demonstrate the how the number of heads varies randomly across each set of 10 tosses. Nine more dots have been added to your dotplot. Is a pattern starting to emerge? Now change the Number of repetitions from 1 to 990 and press Toss Coins. The applet will now show the results for the number of heads in 1000 different sets of 10 coin tosses. So each dot represents the number of times Harley chooses the correct cup out of ten attempts assuming he is just guessing. Remember why we conducted this simulation: to assess whether Harley’s result (9 correct in 10 attempts) would be unlikely to occur by chance alone if he were just guessing between the pair of cups for each attempt. 14. Locate the result of getting 9 heads in the dotplot created by the applet. Would you consider this an unlikely result, in the tail of the distribution of the number of heads? 15. Based on the results of 1000 simulated sets of 10 coin flips each, would you conclude that Harley would be very unlikely to have picked the correct cup in 9 times in 10 attempts if he was randomly guessing between the two cups each time? Explain how your answer relates to the applet’s dotplot. Definition: A result is statistically significant if it is unlikely to occur just by random chance. If our observed result appears to be consistent with the chance model, we say that the chance model is plausible or believable. 16. Do the results of this study appear to be statistically significant? June 27, 2014 MAA PREP workshop 13 17. Do the results of this study suggest that Harley just guessing is a plausible explanation for Harley picking the correct cup 9 out of 10 times? Summarizing your understanding 18. To make sure that you understand the coin flipping chance model fill in the following table indicating what parts of the real study correspond to the physical (coin-flipping) simulation. Table 1.2: Parallels between real study and physical simulation Coin flip = Heads = Tails = Chance of Heads = 1/2 = One repetition = one set of ___ simulated attempts by Harley The 3S Strategy We will call the process of simulating could-have-been statistics under a specific chance model the 3S Strategy. After forming our research conjecture and collecting the sample data, we will use the 3S strategy to weigh the evidence against the chance model. This 3S Strategy will serve as the foundation for addressing the question of statistical significance in Step 4 of the Statistical Investigation Method. 3S Strategy for Measuring Strength of Evidence 1. Statistic: Compute the statistic from the observed sample data. 2. Simulate: Identify a “by chance alone” explanation for the data. Repeatedly simulate values of the statistic that could have happened when the chance model is true. 3. Strength of evidence: Consider whether the value of the observed statistic from the research study is unlikely to occur if the chance model is true. If we decide the observed statistic is unlikely to occur by chance alone, then we can conclude that the observed data provide strong evidence against the plausibility of the chance model. If not, then we consider the chance model to be a plausible (believable) explanation for the observed data; in other words what we observed could plausibly have happened just by random chance. Let’s review how we have already applied the 3S strategy to this study. 19. Statistic. What is the statistic in this study? 20. Simulate. Fill in the blanks to describe the simulation. June 27, 2014 MAA PREP workshop 14 We flipped a coin ____ times and kept track of how many times it came up heads. We then repeated this process _____ more times, each time keeping track of how many heads were obtained in each of the ______ flips. 21. Strength of evidence. Fill in the blanks to summarize how we are assessing the strength of evidence for this study. Because we rarely obtained a value of ________ heads when flipping the coin _______ times, this means that it is ____________ (believable/unlikely) that Harley is just guessing, since if Harley was just guessing he ____________ (rarely/often) would get a value like _________ correct out of ______ attempts. Step 5: Formulate conclusions. 22. Based on this analysis, are you convinced that Harley can understand human cues? Why or why not? Another Study One important step in a statistical investigation is to consider other models and whether the results can be confirmed in other settings. 23. In a different study, the researchers used a mechanical arm (roughly the size of a human arm) to point at one of the two cups. The researchers tested this to see whether dogs understood non-human gestures. In 10 trials, Harley chose the correct cup 6 times. a. Using the dotplot you obtained when you simulated 1000 sets of 10 coin flips assuming Harley was just guessing, locate the result of getting 6 heads. Would you consider this an unlikely result, in the tail of the distribution? b. Based on the results of 1000 simulated sets of 10 coin flips each, would you conclude that Harley would be very unlikely to have picked the correct cup in 6 times in 10 attempts, if he was randomly guessing between the two cups each time? Explain how your answer relates to the applet’s dotplot. June 27, 2014 MAA PREP workshop 15 c. Is this study’s result statistically significant? d. Do the results of this study suggest that Harley just guessing is a plausible explanation for Harley picking the correct cup 6 out of 10 times? e. Does this study prove that Harley cannot understand the mechanical arm? Step 6: Look back and ahead. 24. Compare the analyses between the two studies. How does the unusualness of the observed statistic compare between the two studies? Does this make sense based on the value of the observed statistic in the two studies? Does this make sense based on how the two studies were designed? Explain. (Hint: Why might the results differ for human and mechanical arms? Why would this matter?) 25. A single study will not provide all of the information needed to fully understand a broad, complex research question. Thinking back to the original research question, what additional studies would you suggest conducting next? Section 1.1 Summary The set of observational units on which we collect data is called a sample. The number of observational units is the sample size. A number computed to summarize the variable measured on a sample is called a statistic. For a chance process, a parameter is a long-run numerical property of that process, such as a probability (long-run proportion). A simulation analysis based on a chance model can assess the strength of evidence provided by sample data against a particular claim about the chance model. The logic of assessing statistical significance employs what we call the 3S Strategy: Statistic: Compute an observed statistic from the data. Simulate: Identify a model for the “by chance alone” explanation. Repeatedly simulate values of the statistic that could have occurred from that chance model. Strength of evidence: Examine how unusual the observed value of the statistic would be under repeated application of the chance model. o If the observed value of the sample statistic is unlikely to have occurred from the chance model, then the data provide strong evidence against the chance model as the explanation. June 27, 2014 MAA PREP workshop 16 o If the observed value of the sample statistic is not unlikely to have occurred from the chance model, then the chance model is a plausible explanation for the observed data. The chance model considered in this section involved tossing a fair coin. This chance model allowed for assessing whether an observed number of “successes” in a study provided strong evidence that the two outcomes of a categorical variable were not equally likely. In the next section you will consider other chance models, but the reasoning process will remain the same. June 27, 2014 MAA PREP workshop 17 FAQ 1.1: What is a random process? Q: So a parameter is a numerical property of a process… but what is the process? All I have is Buzz’s 16 attempts. A: We can think of Buzz’s 16 attempts as a sample from his long-run selection process. He has some underlying probability of pushing the correct button, but he’s unlikely to be correct every time, even if the dolphins are communicating. There is randomness in his choices. His choices might be affected by Doris’ squeaks, but also by how hungry he is, how tired he is, how warm the water is. We are trying to see how much Doris’ communication is influencing him. Everything else gets lumped together into “random chance.” Any one outcome from a random process is unknown to us in advance (like tossing a coin), but if we observe outcomes from the process for long enough, we should start to see some patterns amidst the randomness. So you can think of a random process as an unlimited source of potential observations for your sample. Q: Can I use any old set of observations from a process as a sample? A: Some samples are good and others are terrible. Q: How do I tell the good apples from the rotten ones? A: For a Yes/No process like Buzz’s selections, we need to be willing to make certain assumptions along the way. For example, we assumed Buzz’s probability of a correct guess was the same every time. In fact, under the null hypothesis, we assumed it was 50/50 every time. But even if the probability is not 0.5, we are assuming it is not changing. In other words, we assume that Buzz doesn’t get better through learning and that he doesn’t get tired or bored or quit trying. We are also assuming that his previous trials don’t influence his future guesses – like a coin, he would have no memory of his past guesses. Q: So how do I know if I have a good sample? A: You need to believe that nothing about the process is changing and that each outcome does not depend on past outcomes. If you’re a coin, your chance of heads doesn’t change, and you have no memory of how you’ve landed in the past. If this is true about the process, then you will have a good sample! On the other hand, if Buzz’s probability of success is different in the morning than in the afternoon, then only observing him in the morning will not give you a good representation of his overall probability of success. Q: So then how does the parameter come into play? A: Parameter is a hard concept, precisely because we never see it! For the dolphin example, we want to know whether Buzz is just guessing. We think of his guesses as a potentially never-ending process (like coin tossing), and the parameter is the probability that Buzz will be correct in his next attempt. As you saw in the Preliminaries, this means if we were to observe the random process forever, this probability is the long-run proportion of times that Buzz pushes the correct button. Q: So I see why the parameter isn’t just 15/16, that is his sample proportion and we may not have hit that long-run probability yet. So the parameter is just 0.5? A: Not necessarily. The parameter equals 0.5 if he is just guessing. That’s the chance model we simulated. But his probability may actually be different from 0.5. Q: So the parameter is his actual probability of pushing the correct button. It’s a number, but we don’t know its value. A: Correct! But keep in mind that we will define lots of different kinds of parameters in future chapters. The “probability of success” parameter only applies in this type of Yes/No random process. June 27, 2014 MAA PREP workshop 18 Formalizing Terms (adapted from Exploration 1.2: Tasting Water and Exploration 1.3: Do People Use Facial Prototyping?) A study in Psychonomic Bulletin and Review (Lea, Thomas, Lamkin, & Bell, 2007) presented evidence that “people use facial prototypes when they encounter different names.” Participants were given two faces and asked to identify which one was Tim and which one was Bob. The researchers wrote that their participants “overwhelmingly agreed” on which face belonged to Tim and which face belonged to Bob, but did not provide the exact results of their study. Step 1: Ask a research question. We will gather data from your class to investigate the research question of whether students have a tendency to associate certain facial features with a name. Step 2: Design a study and collect data. 1. Identify the observational units in this study. 2. Identify the variable. Is the variable quantitative or categorical? 3. How many outcomes can the variable have? Definition: A binary variable is a categorical variable with only two outcomes. We also (arbitrarily) define one outcome to be a success and one to be a failure. 4. Describe the parameter of interest (in words). (Hint: The parameter is the long-run probability of …?) 5. One possibility here is that subjects are equally likely to associate “Tim” with the face on the left as they are to associate “Bob” with the face on the left and so are essentially guessing which name is associated with the face on the left. If this is the case, what is the long-run probability that a subject would associate “Tim” with the face on the left? 6. Another possibility is that subjects are more likely to associate “Tim” with the face on the left than they are to associate “Bob” with the face on the left. If this is the case, what can you say about the long-run probability that a subject would associate “Tim” with the face on the left? (Hint: You are not to specify a particular value this time; instead indicate a direction from a particular value.) June 27, 2014 MAA PREP workshop 19 Definition: The null hypothesis typically represents the “by chance alone” explanation. The chance model (or “null model”) is chosen to reflect this hypothesis. The alternative hypothesis typically represents the “there is an effect” explanation that contradicts the null hypothesis. It represents what the researchers are hoping to gather evidence to support. 7. Your answers to #5 and #6 should be the null and alternative hypotheses for this study. Which is which? Step 3: Explore the data. 8. How many students put Tim as the name on the left? How many students participated in this study (sample size)? What proportion put Tim’s name on the left? Number (Tim on left): Sample size: Proportion (Tim on left): Use of symbols We can use mathematical symbols to represent quantities and simplify our writing. Throughout the book we will emphasize written explanations, but will also show you mathematical symbols which you are free to use as a short-hand once you are comfortable with the material. The distinction between parameter and statistic is so important that we always use different symbols to refer to them. When dealing with a parameter that is a long-run probability, such as the probability that a (future) subject in this study would associate “Tim” with the face on the left, we use the Greek letter (pronounced “pie”). But when working with a statistic that is the proportion of “successes” in a sample, such as the proportion of subjects who associated “Tim” with the face on the left, we use the symbol 𝑝̂ (pronounced “p-hat”). Finally, we use the symbol n to represent the sample size. 9. What is the value of 𝑝̂ in this study? 10. What is the value of n in this study? 11. Hypotheses are always conjectures about the unknown parameter, . You can also use H0 and Ha as short-hand notation for the null and alternative hypotheses, respectively. A colon, ‘:’, is used to represent the word ‘is.’ Restate the null and alternative hypotheses using . H0: Ha: June 27, 2014 MAA PREP workshop 20 Step 4: Draw Inferences. 12. Is the sample proportion who associated “Tim” with the face on the left greater than the probability specified in the null hypothesis? 13. Is it possible that the sample proportion could turn out to be this large even if the null hypothesis were true? As we did with Buzz and Doris in Section 1.1, we will use simulation to investigate how surprising the observed sample result would be, if in fact subjects were just as likely to associate “Tim” with the face on the left as they were to associate “Bob” with the face on the left. (Note also that our null model assumes the same probability for all subjects.) 14. We will now use the One Proportion applet to conduct this simulation analysis. a. First enter the probability of success value specified in the null hypothesis. b. Enter the appropriate sample size (number of subjects in this study). c. Enter 1 for the number of samples, and press Draw Samples. Report the number of “successes” in this simulated sample. d. Now, select the radio button for “Proportion of successes.” Report the proportion of successes in this simulated sample. Use your answer to “c” to verify how this value is calculated. e. Leaving the “Proportion of successes” radio button selected, click on Draw Samples four more times. Do you get the same results each time? f. Now enter 995 for the number of samples and click on Draw Samples, bringing the number of simulated samples to 1000. Comment on the center, variability, and shape of the resulting distribution of sample proportions. This distribution of simulated sample proportions is called the null distribution, because it is based on assuming the null hypothesis to be true. 15. Restate the observed value of the sample proportion, p̂ , who associated “Tim” with the face on the left. June 27, 2014 MAA PREP workshop 21 16. Looking at the null distribution you have simulated, is this a very unlikely result, when the null hypothesis is true? In other words, is this value far in the tail of the null distribution? You might find that #16 is a bit of a tough call if the observed sample statistic is not far in the tail of the distribution. To help make a judgment about strength of evidence in this case, we can count how many (and what proportion) of the simulated sample proportions are as extreme, or more extreme, than the observed value. 17. Use the applet to count how many (and what proportion) of the simulated sample proportions are more extreme than the observed value, p̂ . To do this, enter the observed p̂ (the observed sample proportion who associated “Tim” with the face on the left) in the box to the left of the Count button. Then press the Count button. Record the number and proportion of simulated sample proportions that are as extreme, or more than, than the observed p̂ . Definition: The p-value is estimated as the proportion of simulated statistics in the null distribution that are at least as extreme (in the direction of the alternative hypothesis) as the value of the statistic actually observed in the research study. How do we evaluate this p-value as a judgment about strength of evidence provided by the sample data against the null hypothesis? One answer is: The smaller the p-value, the stronger the evidence against the null hypothesis and in favor of the alternative hypothesis. But how small is small enough to regard as convincing? There is no definitive answer, but here are some guidelines: Guidelines for evaluating strength of evidence from p-values 0.10 < p-value not much evidence against null hypothesis; null is plausible 0.05 < p-value < 0.10 moderate evidence against the null hypothesis 0.01 < p-value < 0.05 strong evidence against the null hypothesis p-value < 0.01 very strong evidence against the null hypothesis The smaller the p-value, the stronger the evidence against the null hypothesis. 18. Is the approximate p-value from your simulation analysis (your answer to #17) small enough to provide much evidence against the null hypothesis that subjects are equally likely to associate “Tim” with the face on the left as they are to associate “Bob” with the face on the left? If so, how strong is this evidence? Explain. June 27, 2014 MAA PREP workshop 22 Step 5: Formulate conclusions. 19. Do you consider the observed sample result to be statistically significant? Recall that this means that the observed result is unlikely to have occurred by chance alone. 20. How broadly are you willing to generalize your conclusions? Would you be willing to generalize your conclusions beyond the subjects in this study? How broadly? Explain your reasoning. Step 6: Look back and ahead. 21. Suggest a new research question that you might investigate next, building on what you learned in this study. June 27, 2014 MAA PREP workshop 23 June 27, 2014 MAA PREP workshop 24 Alternate Measures for Strength of Evidence (adapted from Exploration 1.3: Do People Use Facial Prototyping?) A study in Psychonomic Bulletin and Review (Lea, Thomas, Lamkin, & Bell, 2007) presented evidence that “people use facial prototypes when they encounter different names.” Participants were given two faces and asked to identify which one was Tim and which one was Bob. The researchers wrote that their participants “overwhelmingly agreed” on which face belonged to Tim and which face belonged to Bob, but did not provide the exact results of their study. Step 1: Ask a research question. We will gather data from your class to investigate the research question of whether students have a tendency to associate certain facial features with a name. Step 2: Design a study and collect data. Each student in your class will be shown the same two pictures of men’s faces used in the research study. You will be asked to assign the name Bob to one photo and the name Tim to the other. Each student will then submit the name that he or she assigned to the picture on the left. Then the name that the researchers identify with the face on the left will be revealed. The parameter of interest here is the long-run probability that a student in your class would assign the same name to the face on the left. 1. State the null and alternative hypotheses to be tested when the data are collected. Express these both in words and symbols. (Hint: Think about the parameter and the research question of interest here.) Step 3: Explore the data. 2. How many students put Tim as the name on the left? How many students participated in this study (sample size)? What proportion put Tim’s name on the left? Number (Tim on left): Sample size: Proportion (Tim on left): When we conduct analyses with binary variables, we often call one of the outcomes a success and the other a failure, and then focus the analysis on the success outcome. It is arbitrary which outcome is defined to be a success, but you need to make sure you do so consistently throughout the analysis. In this case we’ll call “Tim on left” a success because that’s what previous studies have found to be a popular choice. June 27, 2014 MAA PREP workshop 25 Step 4: Draw inferences. You will use the One Proportion applet to investigate how surprising the observed class statistic would be, if students were just randomly selecting which name to put with which face. 3. Before you use the applet, indicate what you will enter for the following values: a. Probability of success: b. Sample size: c. Number of repetitions: 4. Conduct this simulation analysis. Make sure the “Proportion of heads” button is selected in the applet and not the “Number of heads.” a. Indicate how to calculate the approximate p-value (count the number of simulated statistics that equal __________ or are ___________). b. Report the approximate p-value. c. Use the p-value to evaluate the strength of evidence provided by the sample data against the null hypothesis, in favor of the alternative that students really do tend to assign the name Tim (as the researchers predicted) to the face on the left. The p-value is the most common way to evaluate strength of evidence, but now we will explore a common alternative way to evaluate strength of evidence. The goal of any measure of strength of evidence is to use a number to assess whether the observed statistic falls in the tail of the null distribution (and is therefore surprising if the null hypothesis is true) or among the typical values we see when the null hypothesis is true. 5. Check the Summary Stats box in the applet. a. Report the mean (average) value of the simulated statistics. b. Explain why it makes sense that this mean is close to 0.5. c. Report the standard deviation (SD) of the simulated statistics. June 27, 2014 MAA PREP workshop 26 d. Report (again) the observed class value of the statistic. (What proportion of students in your class put Tim’s name on the left?) p̂ = e. Calculate how many standard deviations the observed class value of the statistic is from the hypothesized mean of the null distribution, 0.5. In other words, subtract the 0.5 from the observed value, and then divide by the standard deviation. This is the standardized statistic z = (observed statistic p̂ – 0.5) / SD of null distribution. f. Once you calculate this value, you interpret it as “how many standard deviations the observed statistic falls from the hypothesized parameter value.” What strength of evidence against the null does your standardized statistic provide? g. How closely does your evaluation of strength of evidence based on the standardized statistic compare to the strength of evidence based on the p-value in #4c? Guidelines for evaluating strength of evidence from standardized values of statistics Standardizing gives us a quick, informal way to evaluate the strength of evidence against the null hypothesis: between -1.5 and 1.5: below -1.5 or above 1.5: below -2 or above 2: below -3 or above 3: little or no evidence against the null hypothesis; moderate evidence against the null hypothesis; strong evidence against the null hypothesis; very strong evidence against the null hypothesis. Step 5: Formulate conclusions. 6. Now, let’s step back a bit further and think about the scope of inference. We have found that in most classes, the observed data provide strong evidence that students do better than random guessing which face is Tim’s and which is Bob’s. In that case, do you think that most students at your school would agree on which face is Tim’s? Do you think this means that most people can agree on which face belongs to Tim? Furthermore, does this mean that all people do ascribe to the same facial prototyping? We will look more at the scope of inference in chapters 2 and 4. June 27, 2014 MAA PREP workshop 27 Step 6: Look back and ahead. 7. Discuss limitations to this study? 8. Based on the limitations of this study, suggest a new research question that you would investigate next. Extensions 9. In #2 you recorded the proportion of students in your class who put Tim’s name with the photo on the left. Imagine that the proportion was actually larger than that (e.g., if your class was 60%, imagine it was 70%). a. How would this have affected the p-value: Larger Same Smaller b. How would this have affected the standardized value: Larger Same Smaller c. How would this have affected the strength of evidence against the null hypothesis: Stronger Same Weaker 10. What if the proportion who put Tim’s name on the left was the same, but 100 students were in your class. a. How would this have affected the variability in the null distribution? Larger Same Smaller b. How would this have affected the p-value? Larger Same Smaller c. How would this have affected the standardized value: Larger Same Smaller d. How would this have affected the strength of evidence against the null hypothesis: Stronger June 27, 2014 Same Weaker MAA PREP workshop 28 11. Suppose that less than half of the students in your class had put Tim’s name on the left, so your class result was in the opposite direction of the alternative hypothesis. What can you say about the standardized value of the statistic in this case? Explain. (Hint: You cannot give a value for the standardized statistic, but you can say something specific about its value.) June 27, 2014 MAA PREP workshop 29 Alternative Values for Probability of Success (adapted from Chapter 1 Investigation) Background: A legendary story on college campuses concerns two students who miss a chemistry exam because of excessive partying but blame their absence on a flat tire. The professor allowed them to take a make-up exam, and he sent them to separate rooms to take it. The first question, worth five points, was quite easy. The second question, worth ninety-five points, asked: Which tire was it? Wednesday we collected data from you, asking which of the four tires you would pick if you were caught in this situation (and had to make up a tire on the spot). For some reason, I conjecture that college students appear to have a tendency to pick the tire. You will use these data to conduct a test of whether the sample data provide convincing evidence that this tire is picked more often than we would expect if the four tires were picked equally often (purely at random). Step 1: Ask a research question (a) Identify the research conjecture. Step 2: Design a study and collect data (b) What are the observational units? (c) What is the variable that is measured/recorded on each observational unit? (Hint: State as a binary variable.) (d) Describe the parameter of interest in words. (e) State the null and alternative hypotheses to be tested. Null hypothesis (H0) >> Alternative hypothesis (Ha) >> Step 3: Explore the data Fill in the tallies of your responses. Front left Front right Back left Back right Total (f) Calculate the observed statistic: What proportion of students selected the left front tire? How many students participated? (Include the symbols you would use to represent these value (copy and paste one of these symbols below: p̂ , 𝜋, 𝑥̅ , 𝜇, 𝜎, s, n) June 27, 2014 MAA PREP workshop 30 Step 4: Draw inferences beyond the data Because I want to focus on the right front tire, I will make the variable binary (right front or not) and the parameter of interest in this study is the long-run probability that a Cal Poly student picks the left front tire. What symbol should I use to refer to this unknown parameter value: p̂ , 𝜋, 𝑥̅ , 𝜇, 𝜎, s, n – copy and paste below. (g) Restate the hypotheses in terms of this parameter. (h) Is the sample proportion who selected the left front tire greater than one-fourth? (If not, there’s no need to conduct a simulation analysis. You will have no evidence that this tire is selected more than one-fourth of the time in the long run.) (i) Simulate: Use the One Proportion applet to simulate 1000 repetitions of this study, assuming that every student in class has a 0.25 probability of selecting the right front tire. For your simulation “Success” represents: “Chance of success” = One repetition = (j) Using the proportion of successes for the values on the horizontal axis, what is the center of your null distribution? Does it make sense that this is the center? Explain. (k) Strength of evidence: Determine the p-value from your simulation results. Also interpret what this p-value probability represents (i.e., the long-run proportion of what that do what assuming what?!). (l) Evaluate this p-value: Does this p-value provide strong evidence against the null hypothesis? (m) Determine and interpret the standardized statistic from your simulation analysis. Standardized statistic = observed statistic hypothesized parameter = standard deviation of null distributi on (n) According to this standardized statistic, is the observed value of the sample proportion surprising when the null hypothesis is true? Is this consistent with what the p-value told you? June 27, 2014 MAA PREP workshop 31 Step 5: Formulate Conclusions (o) Write a paragraph, as if to a member of your family who has not taken statistics, summarizing the conclusion that you would draw from this study and your simulation analysis. Clearly explain to this intelligent but not-statistical family member how you are coming to this conclusion/the reasoning behind your conclusion. Step 6: Look back and ahead (p) Now, let’s step back a bit and think about the scope of our inference. For example: Have we shown that are equally likely to pick among the four tires? Does it follow that people in general don’t have a tendency to pick the left front? Are there any improvements you would suggest for how the data were collected? Suggest a future research study that you would be interested in investigating based on these results. Limitations in conclusions? >> Improvements would make to study? >> Future research questions? >> June 27, 2014 MAA PREP workshop 32 Two-Sided Tests (adapted from Example 1.5: Halloween Treats) Stemming from concern over the nation’s obesity epidemic, researchers investigated whether children might be as tempted by toys as by candy for Halloween treats. Test households in five Connecticut neighborhoods offered children two plates: one with lollipops or fruit candy and one containing small, inexpensive Halloween toys, like plastic bugs that glow in the dark. The researchers observed the selections of 283 trick-or-treaters between the ages of 3 and 14 (Schwartz, Chen, and Brownell, 2003). 1. Identify the parameter of interest in this study. 2. State the null and alternative hypotheses to be tested. (Note that no direction is anticipated in advance for this study, so the alternative should be two-sided.) The researchers found that 148 trick-or-treaters selected the candy, and 135 chose a toy. 3. Calculate the value of the statistic for this study. 4. Use the applet to conduct a simulation analysis with 10,000 repetitions, assuming the null hypothesis. Comment on the shape, center, and variability of the simulated distribution of sample proportions. 5. Use the simulation results to approximate the p-value for this study. Be sure to click on “Two-sided.” Also describe which values the applet counts to determine this p-value, and explain why these are the correct values to count. June 27, 2014 MAA PREP workshop 33 6. Calculate the standardized value of the statistic for this study. 7. Interpret what this standardized values reveals. 8. Summarize your conclusion from the simulation analysis and standardized statistic from this study. 9. Look back and ahead: What limitations do you see with this study? How broadly would you feel comfortable generalizing your conclusions? What would you like to see in a follow-up study? June 27, 2014 MAA PREP workshop 34 Theory-Based Inference (adapted from Examples 1.3 and 1.4) Example 1.4: Halloween Treats Recall that stemming from concern over the nation’s obesity epidemic, researchers investigated whether children might be as tempted by toys as by candy for Halloween treats. Test households in five Connecticut neighborhoods offered children two plates: one with lollipops or fruit candy and one containing small, inexpensive Halloween toys, like plastic bugs that glow in the dark. The researchers observed the selections of 283 trick-or-treaters between the ages of 3 and 14 (Schwartz, Chen, and Brownell, 2003). To investigate whether children show a preference for either the candy or the toys, we test the following hypotheses. Null hypothesis: The probability a trick-or-treater would choose candy is 0.5. Alternative hypothesis: The probability a trick-or-treater would choose candy is not 0.5. Note that our null model assumes that the probability of choosing candy (π) is the same for all children. In symbols, these hypotheses translate to H0: 𝜋 = 0.5 Ha: 𝜋 ≠ 0.5. The researchers collect the reactions of 283 children for the study. With a sample size of 283, under our null hypothesis, we simulated the null distribution (using 10,000 simulated samples) shown in Figure 1.1. Figure 1.1: A null distribution representing 10,000 simulated samples of 283 children where the probability that an individual child would choose candy is 0.5 1. With regard to the above null distribution: a. What is the numeric value of the center of this null distribution? Does that make sense? b. What is the SD of this null distribution? June 27, 2014 MAA PREP workshop 35 c. Would it be appropriate to describe the overall shape of this null distribution as bell-shaped? 2. As you did in the previous session, use the One Proportion applet to run a simulation (with 10000 repetitions) to find the approximate p-value for this study. Be sure to click on “Two-sided.” Record the p-value: ___________ Theory-based approach (One proportion z test) In the early 1900s, and even earlier, computers weren’t available to do simulations, and as people didn’t want to sit around and flip coins all day long, they focused their attention on mathematical and probabilistic rules and theories that could predict what would happen if someone did simulate. They proved the following key result (often called the Central Limit Theorem): Central Limit Theorem (CLT): The distribution of sample proportions will be centered at the long-run probability (𝜋), with a standard deviation of √𝜋(1 − 𝜋)/𝑛. If the sample size (n) is large enough, then the shape of the distribution is bell-shaped. One bit of ambiguity in the statement is how large is large enough for the sample size? As it turns out, the larger the sample size is, the better the prediction of bell-shaped behavior in the null distribution is, but there is not a sample size where all of the sudden the prediction is good. However, some people have used the convention that you should have at least 10 successes and at least 10 failures in the sample. Validity conditions: The normal approximation can be thought of as a prediction of what would occur if simulation was done. Many times this prediction is valid, but not always, only when the validity condition (at least 10 successes and at least 10 failures) is met. 3. How well does the CLT predict the center, SD, and shape of the null distribution of sample proportion of children who choose candy, as displayed in Figure 1.1? a. Center: b. SD: c. Shape: 4. Now go back to the One Proportion applet, and check the box next to “Normal Approximation." Be sure that the “Two-sided” option is also selected. Record the normal approximation-based p-value: ___________ 5. Compare the simulation-based p-value (from #2) to the normal approximation-based pvalue (from #4). Are the very different or about the same? June 27, 2014 MAA PREP workshop 36 6. In case of discrepancy between the simulation-based p-value and the normal approximation-based p-value, which do you think would be the more reliable? 7. (Optional) Notice that the applet provides you with an option to report the Exact Binomial p-value. Check on the option, and report the resulting p-value: ___________________. 8. (Optional) Which p-value is closer to the Exact Binomial p-value: the simulation-based pvalue (from #2) or the normal approximation-based p-value (from #4)? Figure 1.2 shows a few examples of null distributions. Figure 1.2: Null distributions for various studies. A bell-shaped distribution, called a normal distribution, can be used to nicely approximate some of them, but will not work well for others. St. George’s (n = 10; 𝜋 = 0.15) St. George’s (n = 361; 𝜋 = 0.15) Halloween treats (n = 283; 𝜋 = 0.5) The St. George’s (n = 361) and Halloween treats (n = 283) simulations show bell-shaped curves. However, the first sample of St. George’s heart patients (n = 10) is a not bell-shaped curve, in part because the distribution is not symmetric and in part because there are so few values that the sample proportion can be. June 27, 2014 MAA PREP workshop 37 A situation where a theory-based approach doesn’t work Example 1.3: Heart Transplant Operations In an article published in the British Medical Journal (2004), researchers Poloniecki, Sismanidis, Bland, and Jones reported that heart transplantations at St. George’s Hospital in London had been suspended in September 2000, after a sudden spike in mortality rate. Of the last 10 heart transplants, 80% had resulted in deaths within 30 days of the transplant. Newspapers reported that this mortality rate was over five times the national average. Based on historical national data, the researchers used 15% as a reasonable value for comparison. Think about it: Why do you think the Theory-based (One proportion z-test; Normal approximation) approach will not work well for this data? Applying the 3S Strategy Using the 3S Strategy from the previous section, we observed 0.80 as our statistic and now we will simulate one thousand repetitions from a process where 𝜋 = 0.15 (under the null hypothesis). We did this using the One Proportion applet as shown in Figure 1.2. Figure 1.2: Null distribution (could-have-been simulated sample proportions) for 1000 repetitions of drawing samples of 10 “patients” from a process where the probability of death is equal to 0.15. “Success” has been defined to be patient death. 9. How well does the CLT predict the center, SD, and shape of the null distribution of sample proportion of deaths among a sample of 10 heart transplant cases, as displayed in Figure 1.2? a. Center: Does it make sense? b. SD: c. Shape: June 27, 2014 MAA PREP workshop 38 Another situation where a theory-based approach doesn’t work Recall the study about the legendary story on college campuses concerns two students who miss a chemistry exam because of excessive partying but blame their absence on a flat tire. The professor allowed them to take a make-up exam, and he sent them to separate rooms to take it. The first question, worth five points, was quite easy. The second question, worth ninetyfive points, asked: Which tire was it? When I collected data in my class of 36 students, 18 picked the ____________ tire. To explore whether this is unlikely to have happened by random chance, I generated the following null distribution. 10. As you can see, I forgot to include the mean of the null distribution in the applet output. Use the CLT to predict what the mean of this null distribution should be. 11. Now use the CLT to predict the SD of this null distribution. 12. Do you think the One proportion z-test is a valid approach to find the p-value in the context of this study? Why or why not? Explain. (Hint: Is it appropriate to describe the overall shape of the null distribution as bell-shaped? Is the validity condition of at least 10 successes and at least 10 failures helpful in deciding whether the theory-based approach is valid in the context of this study?) 13. Use the One Proportion applet to find the following: a. Simulation-based p-value: __________ b. Normal approximation-based (theory-based) p-value: c. Which should you trust more: (a) or (b)? June 27, 2014 MAA PREP workshop 39 Assessment examples from Chapter 1 Sample quiz #1 1. Which definition of a p-value is the most accurate? a. If the alternative hypothesis is true, the p-value is the probability of observing an outcome as extreme or more extreme than the one actually observed in the study b. The p-value is the probability that the alternative hypothesis is true c. If the null hypothesis is true, the p-value is the probability of observing an outcome as extreme or more extreme than the one actually observed in the study d. The p-value is the probability that the null hypothesis is true Circle TRUE or FALSE each of the following (#2 through #5) 2. A p-value can be negative TRUE FALSE 3. A standardized statistic can be negative TRUE FALSE 4. We run tests of significance to determine whether 𝑝̂ is larger or smaller than some value. TRUE FALSE 5. We run tests of significance to determine whether is larger or smaller than some value. TRUE FALSE 6. Dwayne Wade of the Miami Heat hit 569 of his 1093 field goal attempts in the 2012/2013 season for a shooting percentage of 52.1%. Over the lifetime of Dwayne’s career, can we say that Dwayne is more likely than not to make a field goal? a. Is the long-run probability of Dwayne making a field goal a parameter or a statistic? b. Is 52.1% a parameter or a statistic? c. When simulating possible outcomes assuming the chance model, how many times would you flip a coin for one repetition of the 2012/2013 season? d. With each repetition, what would you keep track of? e. What would be a typical value from a single repetition? June 27, 2014 MAA PREP workshop 40 7. Following up on question #6. Suppose at the beginning of the 2013/2014 season Dwayne Wade made 12 of his first 20 field goal attempts Is there evidence that in the long run Dwayne will make his field goal a majority of the time in the 2013/2014 season? a. Let π be the long-run probability that Dwayne makes his field goal. State the null and alternative hypotheses using symbols. b. Based on the simulated null distribution, what is the p-value for your test? c. Based on the simulated null distribution, what is the standardized statistic for your test? d. Based on the p-value or standardized statistic, do you have strong evidence that in the long-run Dwayne will make a majority of his field goals? Explain. June 27, 2014 MAA PREP workshop 41 Sample quiz #2 Use the following situation to answer all questions. “Rock-Paper-Scissors” is a game in which two players simultaneously show one of three hand symbols (denoted rock, paper, or scissors). Rock beats scissors, scissors beats paper, and paper beats rock. An article published in College Mathematics Journal (Eyler, Shalla, Doumaux, and McDevitt, 2009) found that players tend to not prefer scissors, choosing it less often than expected if the 3 choices were made equally. They collected data from 119 people who played games against a computer. In the study, 55.5% started with rock, 32.8% started with paper, and 11.8% with scissors in their first round. Researchers wanted to know whether these data provided convincing evidence that scissors are chosen less than one-third of the time in the long run. 1. What are the observational units? 2. What is the variable? 3. State the null hypothesis 4. State the alternative hypothesis. 5. Identify the parameter of interest in this study A. People playing the game against a computer B. Whether or not someone chooses scissors C. 11.8% D. The long-run probability someone plays scissors E. 1/3 June 27, 2014 MAA PREP workshop 42 When using the One-proportion inference applet, indicate which values you would enter into each of the boxes 6. Box A_______ 7. Box B_______ 8. Box C_______ 9. Box D_______ (Hint: Answer is ≤ or ≥) 10. Box E_______ 11. Write a one-sentence interpretation of the p-value for this study (Note: you do not need to provide the actual VALUE of the p-value). The p-value is the proportion of simulations which are _______________or ____________ (more/less) assuming ___________________________________ (name what is assumed to be true) Here is output from the simulation for use in the following questions: June 27, 2014 MAA PREP workshop 43 12. What is the approximate value of the standardized statistic in this case? A. Z = 0 B. Z = -1 C. Z = 3 D. Z = -5 13. Would you consider the results of this study to be convincing evidence that scissors are chosen less often in the long run than expected? A. B. C. D. E. No, the p-value is going to be large No, the p-value is going to be small Yes, the p-value is going to be small Yes, the p-value is going to be large No, the distribution is centered at 1/3. June 27, 2014 MAA PREP workshop 44 Sample quiz #3 (for after Chapters 1 and 2) 1. Which of the following is true about sampling? a) Larger samples are always better than smaller samples, regardless of how the sample were collected. b) Larger samples will have less sampling variability than smaller samples. c) Larger samples give less biased estimates than smaller samples. d) Larger samples are needed with larger populations. 2. Recall the Gettysburg Address exploration. Which of the following sampling methods would be unbiased? a) Select your sample of words by closing your eyes and dropping a pencil down on the page, record the word, repeat until you have 10 words. b) Select your sample of words by cutting each word out of the address, mixing them up in a hat and drawing out 10 slips of paper. c) Select your sample of words by typing each word onto the same sized slips of paper, mixing them up in a hat and drawing out 10 slips of paper. d) Select your sample of words by randomly choosing words you think represent all the words in the Gettysburg Address. 3. The following dotplot is the time estimates for a 10 second song snippet we looked at in class. The highest time estimates were 30 seconds. Suppose that one of the observations recorded as 30 seconds was a typo, and was meant to be 20. If we corrected this entry in the dataset, how would the following numerical statistics change, if at all? Mean: SMALLER SAME LARGER Median: SMALLER SAME LARGER Standard Deviation: SMALLER SAME LARGER June 27, 2014 MAA PREP workshop 45 4. You have heard that in sports like boxing there might be some competitive advantage to those wearing red uniforms. You want to test this with your new favorite sport of chessboxing. You randomly assign blue and red uniforms to contestants in 20 matches and find that those wearing red won 14 times (or 70%). You conduct a test of significance using simulation and get the following null distribution. (Note this null distribution uses only 100 simulated samples and not the usual 1000 or more.) a) Suppose you want to see if competitors wearing red win more than 50% of the matches in the long run, so you test H0: 𝜋 = 0.5 versus Ha: 𝜋 > 0.5. What is your p-value based on the above null distribution? b) Suppose you now want to see if competitors wearing either red or blue have an advantage, so you test H0: 𝜋 = 0.5 versus Ha: 𝜋 ≠ 0.5. What is your p-value now based on the above null distribution? 5. Suppose you are testing the hypotheses H0: 𝜋 = 0.5 versus Ha: 𝜋 > 0.5. You get a sample proportion of 0.54 and find that your p-value is 0.08. Now suppose you redid your study with each of the following changes, will your new p-value be larger or smaller than the 0.08 your first obtained? a) You increase the sample size and still find a sample proportion of 0.54. LARGER SMALLER b) Keeping the sample size the same, you take a new sample and find a sample proportion of 0.55. LARGER SMALLER c) With your original sample, you decided to test a two-sided alternative instead of Ha: 𝜋 > 0.5. LARGER June 27, 2014 SMALLER MAA PREP workshop 46 A sample exam question (for after Chapters 1 and 2) Studying at Dordt A sample of twenty Dordt students is taken, four of whom (4/20=20%) say they study at least 35 hours per week during the academic year. At most state universities, the proportion of students who report studying at least 35 hours a week is 10%. We wish to see if the Dordt sample provides strong evidence that the true proportion of Dordt students who study more than 35 hours a week is more than 10%. Two different approaches were taken in order to yield a p-value. Option #1. 1000 sets of 20 spins on a spinner were generated where the probability of the spinner pointing to the blue region was 10%.Out of the 1000 sets of 20 spins, 148 sets had at least 4 spins yield blue, and so a p-value of 0.148 is obtained, showing no evidence that more than 10% of Dordt students study more than 35 hours a week. Option #2. The theory-based inference applet was used, generating a standardized zstatistic of 1.49 with a p-value of 0.068, yielding moderate evidence that more than 10% of Dordt students study more than 35 hours a week. Option #1 Option #2 Briefly explain which p-value (Option #1 or Option #2) is more valid and why. Note: While the results obtained from the coin tossing applet are subject to some variation because it is based on simulation that is NOT the main reason for the discrepancies between the two p-values. June 27, 2014 MAA PREP workshop 47 Generalization (adapted from Exploration 2.1A: Sampling Words) 1. Select a representative set of 10 words from the following passage by circling them. Four score and seven years ago, our fathers brought forth upon this continent a new nation: conceived in liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battlefield of that war. We have come to dedicate a portion of that field as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. But, in a larger sense, we cannot dedicate, we cannot consecrate, we cannot hallow this ground. The brave men, living and dead, who struggled here have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember, what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us, that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion, that we here highly resolve that these dead shall not have died in vain, that this nation, under God, shall have a new birth of freedom, and that government of the people, by the people, for the people, shall not perish from the earth. The authorship of literary works is often a topic for debate. For example, researchers have tried to determine whether some of the works attributed to Shakespeare were actually written by Bacon or Marlow. The field of “literary computing” examines ways of numerically analyzing authors’ works, looking at variables such as sentence length and rates of occurrence of specific words. Definitions: The population is the entire collection of observational units we are interested in, and the sample is a subgroup of the population on which we record data. Numerical summaries about a population are called parameters, and numerical summaries calculated from a sample are called statistics. June 27, 2014 MAA PREP workshop 48 The above passage is, of course, Lincoln’s Gettysburg Address, given November 19, 1863 on the battlefield near Gettysburg, PA. We are considering this passage a population of words, and the 10 words you selected are considered a sample from this population. In most studies, we do not have access to the entire population and can only consider results for a sample from that population, but to learn more about the process of sampling and its implications we will now deal with a somewhat artificial scenario where we sample from this known population. 2. Record each word from your sample, and then indicate the length of the word (number of letters) and whether or not the word contains at least one letter e. Word Length (no. of letters) Contains e? (Y or N) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 3. Identify the observational units and the variables you have recorded on those observational units. (Keep in mind that observational units do not have to be people!) 4. Is the variable “length of word” quantitative or categorical? 5. Calculate the average length of the 10 words in your sample. Is this number a parameter or a statistic? Explain how you know. What symbol would you use to refer to this value? 6. The average length of the 268 words in entire speech equals 4.29 letters. Is this number a parameter or a statistic? Explain how you know. What symbol would you use to refer to this value? June 27, 2014 MAA PREP workshop 49 7. Do you think the words you selected are representative of the 268 words in this passage? Suggest a method for deciding whether you have a representative sample. (Hint: Whereas any one sample may not produce statistics that exactly equal the population parameters, what would we like to be true in general?) 8. Combine your results with your classmates’ by producing a dotplot of the distribution of average word lengths in your samples. Be sure to label the axis of this dotplot appropriately. Describe the shape, center, and variability of the distribution of average word lengths as revealed in the dotplot. 9. Let’s compare your sample statistics to the population parameter. How many and what proportion of students in your class obtained a sample average word lengths larger than 4.29 letters, the average word length in the population? Definition: A sampling method is biased if, when using that sampling method, statistics from different samples consistently overestimate or consistently underestimate the population parameter of interest. Note that bias is a property of a sampling method, not a property of an individual sample. Also note that the sampling method must consistently produce non-representative results in order to be considered biased. 10. What does your answer to #9 tell you about whether the sampling method of asking students to quickly pick 10 representative words is biased or unbiased? If biased, what is the direction of the bias (tendency to overestimate or to underestimate)? 11. Do you think asking each of you to take 20 words instead of 10 words have helped with this issue? Explain. June 27, 2014 MAA PREP workshop 50 Now consider a different sampling method: What if you were to close your eyes and point blindly at the page with a pencil 10 times, taking for your sample the 10 words that your pencil lands on. 12. Do you think this sampling method is likely to be biased? If so, in which direction? Explain. A sample is only useful to us if the data we collect on our sample is similar to the results we would find in the entire population. In this sense, we say the sample is representative of the population. 13. Suggest another technique for selecting 10 words from this population in order for the sample to be representative of the population with regard to word length and e-words. Taking a simple random sample Key idea: A simple random sample ensures that every sample of size n is equally likely to be the sample selected from the population. In particular, each observational unit has the same change of being selected as every other observational unit. The key to obtaining a representative sample is using some type of random mechanism to select the observational units from the population, rather than relying on convenience samples or any type of human judgment. Instead of having you choose “random” words using your own judgment, we will now ask you to take a simple random sample of words and evaluate your results. The first step is to obtain a sampling frame - a complete list of every member of the population where each member of the population can be assigned a number. Below is a copy of the Gettysburg address that includes numbers in front of every word. For example, the 43rd word is nation. 14. Go to the Random Numbers generator at http://www.random.org Generate 10 Numbers in the range from 1 to 268. What should you do if you get a duplicate number generated? Enter the random numbers in the table below: Randomly generated ten ID values from 1-268 Using your randomly generated values, look up the corresponding word from the sampling frame. Fill in the data table below. June 27, 2014 MAA PREP workshop 51 Word Length (no. of letters) e-word? (Y or N) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Notice that each simple random sample gave us different values for the statistics. That is, there is variability from sample to sample (sampling variability). This makes sense because the samples are chosen randomly and consist of different words each time, so we expect some differences from sample to sample. Gettysburg Address Sampling Frame 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Four score and seven years ago Our fathers brought forth upon this continent A new nation conceived In liberty and dedicated To The proposition that All men Are created June 27, 2014 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 are met on a great battlefield of that war We have come To dedicate A portion Of that field as a final resting place For those Who here gave 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 we cannot consecrate we cannot hallow this ground The brave men living and dead who struggled here have consecrated It far above our poor power to add or detract MAA PREP workshop 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 living rather to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced It is rather for us to be here 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 full measure of devotion that we here highly resolve that these dead shall not have died in vain that this nation under God shall have a new birth of 52 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 equal Now We Are engaged In A great Civil War testing whether That nation Or Any nation So conceived And So dedicated Can Long Endure We 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 their lives that that nation might live It Is altogether fitting and proper that we should do this But in a larger sense we cannot dedicate 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 The world will little note nor long remember what we say here but it can never forget what they did here It is for us the 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 dedicated to the great task remaining before us that from these honored dead we take increased devotion to that cause for which they gave the last 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 freedom and that government of the people by the people for the people shall not perish from the earth 15. Let’s examine your sample and those of your classmates. a. Calculate the average word length in your random sample. b. Again produce a dotplot of the distribution of sample average word lengths for yourself and your classmates. c. Comment on how this distribution compares to the one from #8 based on nonrandom sampling. 16. Let’s compare your sample statistics to the population parameter. a. How many and what proportion of students in your class obtained a sample average word length larger than the population average (4.29 letters)? June 27, 2014 MAA PREP workshop 53 b. What does this answer reveal about whether simple random sampling is an unbiased sampling method? Explain. Key idea: When we use simple random sampling, the average of the sample means from different samples equals the population mean and the average of the sample proportions from different samples equals the population proportion. (This means that these statistics are unbiased estimators for the corresponding parameter when we use simple random sampling.) To really see the long-run pattern in these results, let’s look at many more random samples from this population. Open the Sampling Words java applet. You will see that the population distribution contains three variables: the lengths of the words, whether or not a word contains at least one letter e, and whether or not the word is a noun. For now, select only the length variable. 17. Check the Show Sampling Options box. Specify 10 in the Sample Size box and press the Draw Samples button. You will see the 10 selected words appear in the box below. You will also see 10 darkened dots within the population distribution representing the 10 lengths that you have sampled. The value of sample mean is displayed in the population box. The sample mean is also plotted in Sample Mean Length plot. Press Draw Samples again. Did you get the same sample this time? Did you get the same sample mean? You should now see two dots in the Sample Mean Length plot, one for each sample mean. The mean of these two sample means is displayed in the upper right corner of this plot. Confirm this calculation. First sample mean: Second sample mean: Average: Change the Number of Samples to 998 (for a total of 1000 samples). Uncheck Animate. Press the Draw Samples button. (This should generate a dotplot of 1000 values of the sample mean for 1000 different simple random samples, each consisting of 10 words, selected randomly from the population of all 268 words in the Gettysburg address.) 18. Describe the resulting distribution of sample means: What are roughly the largest and smallest values? What is the average of the distribution? What is the standard deviation? How does the average of the 1000 sample means compare to the population mean 𝜇 = 4.29? June 27, 2014 MAA PREP workshop 54 19. Now change the Sample Size from 10 to 30 and the Number of Samples to 1000. Press the Draw Samples button. The applet allows you to compare the two distributions (n = 10 to n = 30). How do these two distributions compare? Be sure to identify a feature that is similar and a feature that is different. 20. Did changing the sample size change the center of the distribution? If we used a biased sampling method, would increasing the sample size remove the bias? Explain. You should see that whereas the convenience samples tended to consistently produce samples that overestimate the length of words, the simple random sampling method does not have that tendency. In fact, the average of all the statistics from all possible simple random samples will exactly equal the population parameter value. For this reason, in the real world, where we don’t have access to the entire population, when we select a simple random sample, we will be willing to believe that sample is representative of the population. In fact, we would prefer a small random sample, than a large convenience sample. That doesn’t mean our sample result will match the population result exactly, but we will be able to predict how far off it might be. Simple random sampling (and probability sampling methods in general) allow us to believe that our sample is representative of the larger population and that the observed sample statistic is “in the ball park” of the population parameter. For this reason, when we use simple random sampling we will be willing to generalize the characteristics of our sample to the entire population. June 27, 2014 MAA PREP workshop 55 Interval Estimation (one proportion/probability) (adapted from Exploration 3.1: Kissing Right?) Most people are right-handed, and even the right eye is dominant for most people. Developmental biologists have suggested that late-stage human embryos tend to turn their heads to the right. In a study reported in Nature (2003), German bio-psychologist Onur Güntürkün conjectured that this tendency to turn to the right manifests itself in other ways as well, so he studied kissing couples to see which side they tended to lean their heads while kissing. He and his researchers observed kissing couples in public places such as airports, train stations, beaches, and parks. They were careful not to include couples who were holding objects such as luggage that might have affected which direction they turned. For each kissing couple observed, the researchers noted whether the couple leaned their heads to the right or to the left. They observed 124 couples, age 13-70 years. You will first use Güntürkün’s data to test his conjecture that kissing couples tend to lean their heads to the right. Use the symbol 𝜋 to denote the proportion of all kissing couples in these countries that lean their heads to the right. 1. Is 𝜋 a parameter or a statistic? Explain how you are deciding. 2. Do we know the exact value of 𝜋 based on the observed data? Explain. 3. State the appropriate null and alternative hypotheses, both in words and in terms of the parameter 𝜋, for testing the conjecture that kissing couples tend to lean their heads to the right more often than by random chance. Of the 124 couples observed, 80 leaned their heads to the right while kissing. 4. Calculate the sample proportion of the observed couples who leaned their heads to the right while kissing. Also indicate the symbol used to denote this value. June 27, 2014 MAA PREP workshop 56 5. Conduct a simulation analysis (using the One Proportion applet) to assess the strength of evidence that the sample data provide for Güntürkün’s conjecture that kissing couples tend to lean their heads to the right more often than they would by random chance. Report the approximate p-value and summarize your conclusion about this strength of evidence. Your simulation analysis should convince you that the sample data provide very strong evidence to believe that kissing couples lean their heads to the right more than half the time in the long run. That leads to a natural follow-up question: How much more than half the time? In other words, we have strong evidence that the long-run probability of leaning to the right is greater than one-half, but can we now estimate the value for that probability? We will do this by testing many different (null) values for the probability that a couple leans to the right when kissing. 6. Now test whether the data provide evidence that the long-run probability that a couple leans their heads to the right while kissing (𝜋) is different from 0.60. Use the One Proportion applet to determine the p-value for testing the null value of 0.60. Report what you changed in the applet and report your p-value. Think about it: What kind of test are we conducting if we want to determine whether the population parameter is different from a hypothesized value - a one-sided test or a two-sided test? Remember from Section 1.4 that if we don’t have a specific direction we are testing, greater than or less than, we need to use a two-sided test of significance. If you used a one-sided alternative in #8, determine the two-sided p-value now by checking the Two-sided box. Recall that a p-value of 0.05 or less indicates that the sample data provide strong evidence against the null hypothesis and in favor of the alternative hypothesis. Thus, we can reject the null hypothesis when the p-value is less than or equal to 0.05. Otherwise, when the p-value is greater than 0.05, we do not have strong enough evidence against the null hypothesis and so we consider the null value to be plausible for the parameter. Key Idea: We will consider a value of the parameter to be plausible if the two-sided p-value for testing that parameter value is larger than the level of significance. 7. Is the p-value for testing the null value of 0.60 less than 0.05? Can the value 0.60 be rejected, or is the value 0.60 plausible for the long-run probability that a couple leans their heads to the right while kissing? June 27, 2014 MAA PREP workshop 57 Recall from Section 2.3, the 0.05 criterion we are using is called the significance level. The pvalue you found in the previous question should not have been smaller than 0.05. Hence, you do not reject the null hypothesis at the 0.05 level of significance and therefore you do not reject 0.60 as a plausible value for 𝜋. Thus, it is plausible (i.e., believable) that the long-run probability that a kissing couple leans their heads to the right is 0.60. 8. Does this mean that you’ve proven that exactly 60% of kissing couples lean right? Why or why not? Because there are still other plausible values, now we want to “zoom in” on which values for the long-run probability are plausible and which can be rejected at the 0.05 significance level. 9. Use the applet to test the probability values given in the following table. Each time, change the Probability of success to match the value that you are testing (keeping the observed sample proportion that you count beyond the same). Everything else should stay the same; press Draw Samples and then Count to see the new two-sided p-value (with the Two-sided box checked). Probability under H0 (Two-sided) p-value Reject or plausible? Probability under H0 (Two-sided) p-value Reject or plausible? 0.54 0.55 0.56 0.57 0.58 0.59 0.60 0.70 0.71 0.72 0.73 0.74 0.75 0.76 10. Using a 0.05 significance level and your results from #9, provide a list of plausible values for 𝜋, the long-run probability that a kissing couple leans their heads to the right. This list of values represents an interval containing all values between two endpoints. Key idea: This interval of plausible values that you have written down in #10 is also called a 95% confidence interval for 𝝅, the long-run probability that a kissing couple leans their heads to the right. Why is this called 95% confidence? This corresponds to the 5% (0.05) significance level that was used to decide whether there was enough evidence against a hypothesized value. Notice that 95% and 5% add up to give 100%. The 95% is called the confidence level, and is a measure of how confident we are about our interval estimate of the parameter, in this case, we are 95% confident that the population proportion of all couples who lean their heads to the right while kissing is in the interval that you found. June 27, 2014 MAA PREP workshop 58 11. Does the 95% confidence interval include the value 0.5? Does it include the value 0.6? Explain how your answers relate to the significance test and p-value that you calculated in #5 and #6. 12. Now suppose we were to use a significance level of 0.01 instead of 0.05 to decide whether or not to reject the corresponding null hypothesis for each listed value of . How would you expect the interval of plausible values to change: wider, narrower, or no change? Explain your reasoning. 13. Implement the 0.01 significance level to determine plausible values for the population proportion of all kissing couples that lean their heads to the right. (Hint: Start with the table from #9, although you might have to test more null values as well.) Report the interval of plausible values. 14. How confident are you about the interval of plausible values that you listed in #13? 15. What is the primary difference between the 95% confidence interval reported in #10 and the 99% confidence interval reported in #13? Which interval is wider? Is this consistent with what you predicted in #12? 16. How would you expect a 90% confidence interval to compare to the 95% and 99% confidence intervals? Explain. Key idea: Using a higher confidence level produces a wider confidence interval. You should have found very strong evidence, based on a small p-value, that a majority of kissing couples in the population lean their heads to the right. Moreover, your confidence intervals, which provide plausible values for of the population proportion that lean to the right, contain only values above 0.5. June 27, 2014 MAA PREP workshop 59 Confounding, Causation, and Experiments (adapted from Exploration 4.1: Home Court Disadvantage?) Sports teams prefer to play in front of their own fans rather than at the opposing team’s site. Having a sell-out crowd should provide even more excitement and lead to an even better performance, right? Well, consider the Oklahoma City Thunder, a National Basketball Association team, in its second season (2008-09) after moving from Seattle. This team had a win-loss record that was actually worse for home games with a sell-out crowd (3 wins and 15 losses) than for home games without a sell-out crowd (12 wins and 11 losses). (These data were noted in the April 20, 2009 issue of Sports Illustrated in the Go Figure column.) 1. Identify the observational units and variables in this study. Also classify each variable as categorical (also binary?) or quantitative. 2. When did the Thunder have a higher winning percentage: in front of a sell-out crowd or a smaller crowd? Support your answer by calculating the proportion of sell-out games that they won and also the proportion of non-sell-out games that they won. (Write both these proportions as decimals.) Sell-out crowd: Smaller crowd: Definition: Two variables are associated or related if the value of one variable gives you information about the value of the other variable. When comparing two groups this means that the proportions or means take different values in the two groups. 3. Do the two variables appear to be associated? Explain. Often when a study involves two associated variables, it is natural to consider one the explanatory variable and the other the response variable. Definitions: The explanatory variable is the variable we think is “explaining” the change in the response variable and the response variable is the variable we think is being impacted or changed by the explanatory variable. The explanatory variable is sometimes called the independent variable and the response variable is sometimes called the dependent variable. June 27, 2014 MAA PREP workshop 60 4. Which would you consider the explanatory variable in this study? Which is the response? (That is, what are the roles of these variables in this study?) There are two possible explanations for this odd finding that the team had a better winning percentage with a smaller crowd: The sell-out crowd caused the Thunder to play worse, perhaps because of pressure or nervousness. The sell-out crowd did not cause a worse performance, and some other issue (variable) explains why they had a worse winning percentage with a sell-out crowd. In other words, a third variable is at play, which is related to both the crowd size and the game outcome. (Of course, another explanation is random chance. Using methods you will learn later, we’ve determined that you can essentially rule out random chance in this case.) 5. Consider the second explanation. Suggest a plausible alternative variable that would explain why the team would be less likely to win in front of a sell-out crowd than in front of a smaller crowd. (Make sure it’s clear not just that your explanation would affect the team’s likelihood of winning, but that the team would be less likely to win in front of a sell-out crowd compared to a smaller crowd.) Definition: A confounding variable is a variable that is related both to the explanatory and to the response variable in such a way that its effects on the response variable cannot be separated from the effects of the explanatory variable. 6. Identify the confounding variable based on your suggested explanation in #5. Explain how it is confounding– what is the link between this third variable and the response variable, and what is the link between this third variable and the explanatory variable? (Hint: Remember that this variable has to be recorded on the observational units: home games for the Thunder.) Another variable recorded for these data was whether or not the opponent had a winning record the previous season. Of the Thunder’s 41 home games, 22 were against teams that won more than half of their games. Let’s refer to those 22 teams as strong opponents. Of these 22 games, 13 were sell-outs. Of the 19 games against opponents that won less than half of their games that season (weak opponents), only 5 of those games were sell-outs. June 27, 2014 MAA PREP workshop 61 7. Was the Thunder more likely to have a sell-out crowd against a strong opponent or a weak opponent? Calculate the relevant (conditional) proportions to support your answer. When the Thunder played a strong opponent, they won only 4 of 22 games. When they played a weak opponent, the Thunder won 11 of 19 games. 8. Was the Thunder less likely to win against a strong opponent than a weak one? Again calculate the relevant (conditional) proportions to support your answer. 9. Explain how your answers to #7 and #8 establish that strength of opponent is a confounding variable that prevents drawing a cause-and-effect conclusion between crowd size and game outcome. 10. Summarize your conclusion about whether these data provide evidence that a sell-out crowd caused the Thunder to play worse. Write as if to a friend who has never studied statistics. Be sure to address the fact that the Thunder had a much smaller winning percentage in front of a sell-out crowd. June 27, 2014 MAA PREP workshop 62 Follow-ups: 1. A recent study classified children according to the type of lighting used in their room (full light, night light, no light) and their eye condition (near-sighted, normal vision, and far-sighted). The two-way table of counts is: Full light Night light No light Total Near-sighted 41 78 18 137 Normal vision 22 115 114 251 Far-sighted 12 39 40 91 Total 75 232 172 479 a) Is this an observational study or an experiment? Explain briefly. b) What proportion of the children who slept in a room with full light were near-sighted? The following segmented bar graph displays these data: 100% 80% 60% Far-sighted 40% Normal vision Near-sighted 20% 0% Full light Night light No light c) Based on this graph, describe the association (if any) between lighting condition and eyesight. d) Provide an argument that parents’ eyesight could be a confounding variable in this study. e) Would you conclude that this study provides strong evidence of a cause-and-effect relationship between lighting condition and eyesight? Explain briefly. 2. In a study published in a 2007 issue of the journal Preventive Medicine, researchers found that smokers were more likely to have used candy cigarettes as children than non-smokers were. When hearing about this study, John said: “But isn’t the smoking status of the person’s parents a confounding variable here?” When Karen asked what he meant, John said: “Children whose parents smoke are more likely to become smokers themselves when they become adults.” What else does John need to say in order to explain how the parents’ smoking status can be a confounding variable in this study? June 27, 2014 MAA PREP workshop 63 June 27, 2014 MAA PREP workshop 64 Randomization Test – Comparing Two Proportions (adapted from Example 5.1: Dolphin Therapy) Swimming with dolphins can certainly be fun, but is it also therapeutic for patients suffering from clinical depression? To investigate this possibility, researchers recruited 30 subjects aged 1865 with a clinical diagnosis of mild to moderate depression. Subjects were required to discontinue use of any antidepressant drugs or psychotherapy four weeks prior to the experiment, and throughout the experiment. These 30 subjects went to an island off the coast of Honduras, where they were randomly assigned to one of two treatment groups. Both groups engaged in the same amount of swimming and snorkeling each day, but one group (the animal care program) did so in the presence of bottlenose dolphins and the other group (outdoor nature program) did not. At the end of two weeks, each subjects’ level of depression was evaluated, as it had been at the beginning of the study (Antonioli and Reveley, 2005). The response variable is defined as whether or not the subject achieved substantial reduction (improvement) of their depression. 1. Why did the researchers include a comparison group in this study? Why didn’t they just see how many patients showed substantial improvement when given the dolphin therapy? The null hypothesis for this study should basically say that there is no association between swimming with dolphins and treatment of depression while the alternative should state that there is an association. We are going to write our hypotheses in terms of probabilities. Null: The probability a subject will show improvement in their depression symptoms is the same between those that swim with dolphins and those that don’t. Alternative: The probability a subject will show improvement in their depression symptoms is greater for those that swim with dolphins compared to those that don’t. The results that the researchers found are shown in the following table: Dolphin therapy Control group Total Showed substantial improvement 10 3 13 Did not show substantial improvement 5 12 17 Total 15 15 30 2. Identify which of the variables in this study is explanatory and which is response. 3. A useful first step begin to see whether the data appear to support the claim that dolphin therapy is effective is to calculate the conditional proportion that improved in each group. Calculate these conditional proportions. Did the dolphin therapy group have a higher conditional proportion that showed substantial improvement than the control group? June 27, 2014 MAA PREP workshop 65 4. Is it possible that this difference could happen even if dolphin therapy was not effective, simply due simply to the random nature of putting subjects into groups (i.e., the luck of the draw)? Sure, this is possible. Consider the following scenario: Assume that the 13 people in the study whose depression symptoms improved would have improved whether they swam with dolphins or not, let’s call them the “improvers”. Now, what if, by chance, of the 13 improvers (people who improve no matter what), 10 randomly ended up in the “swim with dolphins” group. Recall the important fact that subjects were randomly assigned to swim with dolphins or not. So, randomly, it is possible that 10 improvers end up in the swim with dolphins group and only 3 in the not swim with dolphins group. Thus, it is possible we would see 67% (10/15) of dolphin swimmers improving and only 20% (3/15) non-dolphin swimmers improving even if swimming with dolphins doesn’t actually make a difference. So, it is possible….but how unlikely is it? In order to answer this question we will once again turn to simulation. Remember that simulation is a method we have used to estimate probabilities. So far you used coin flipping and an applet to simulate the p-value (the probability we would obtain the observed statistic or something more extreme if the null hypothesis was true) for different tests of significance. Because the dolphin experiment is more complex than the previous studies we looked at, we can’t simply flip a coin any longer. Instead, to do our simulation, we’re going to use playing cards and a computer applet. To estimate the p-value for this study you will need 30 index cards, 13 of which are blue to represent the “improvers” and 17 of which are green to represent the “non-improvers.” The null hypothesis assumes that 13 people (the blue cards) will get better no matter whether they swim with dolphins or not. So those that improved were going to do so (perhaps just from getting to fly to Honduras and swim in the water) regardless of which treatment group they were assigned to. 5. Now shuffle your 30 cards and deal them into two stacks of fifteen. One of the two stacks represents the people who got to swim with dolphins and the other stack represents people who didn’t. Decide which stack is which and then fill in the table below. Showed substantial improvement Did not show substantial improvement Total Dolphin therapy Control group Total 13 17 15 15 30 Difference in proportions of improvement (dolphin group minus control group): Repeat this shuffling and dealing process a second time: Dolphin therapy Control group Total Showed substantial improvement 13 Did not show substantial improvement 17 Total 15 15 30 Difference in proportions of improvement (dolphin group minus control group): June 27, 2014 MAA PREP workshop 66 6. Combine your results with your classmates by constructing a dotplot of the differences in improvement proportions (dolphin group minus control group). 7. Does it appear that the observed value of this statistic (namely, 0.467) would be unlikely to occur if there really were no effect of the dolphin therapy? Explain. Remember the p-value is the probability we would observe a statistic at least as large as we did in our study, assuming the null hypothesis was true. By shuffling and dealing the cards, we are simulating a true null hypothesis. We are simulating a random assignment of the 13 subjects that actually improved to each group. 8. What proportion of the results from your class had a difference in the proportion of improvers of 0.467 or larger? This is your p-value. Provide a detailed interpretation of this p-value. Be sure to explain what is meant by “by chance alone,” keeping in mind what was randomized in this study. 9. Based on the class results, what is your preliminary conclusion about the effectiveness of swimming with dolphins in improving peoples’ depression symptoms? Explain the reasoning process behind this conclusion. We really need to carry out this simulated random assignment process hundreds, preferably thousands of times. This would be very tedious and time-consuming with cards, so let’s turn to technology. To do this, use the “Dolphin study” applet found at: http://www.rossmanchance.com/ISIapplets.html. Check the “Show Shuffle Options” x boand then press the “Shuffle” button. Notice that the applet does what you have done: shuffle the 30 cards and deal out 15 for the “dolphin therapy” group, separating cards by color. The applet also determines the 2×2 table for the simulated results (check the Show table box on the left) and creates a dotplot of the shuffled difference in conditional proportions. Now press the “Shuffle” button four more times. Then increase the “Number of Shuffles” to 995 and press the “Shuffle” button. This produces a total of 1000 repetitions of the simulated random assignment process. June 27, 2014 MAA PREP workshop 67 10. Use the applet to determine what proportion of your 1000 simulated random assignments were the results as (or more) extreme as the actual study (which, you’ll recall, saw a 0.467 difference in improvers). This is your p-value. Interpret this p-value, including what is meant by the phrase “by chance alone” in this context. 11. Is your p-value small enough so that you would consider an difference in the observed proportion of improvers of 0.467 or more surprising under the null model that dolphin therapy is not effective? Does this give strong evidence that the dolphin therapy is effective in treating depression in patients similar to those in the study? 12. Is it reasonable to draw a cause-and-effect conclusion between dolphin therapy and improvement or not in depression symptoms? Explain why or why not. 13. Is it reasonable to generalize your conclusion to all people who suffer from depression? Explain why or why not. Further Exploration What if the study had found the same overall proportion of improvers, but with the group proportions closer together? More specifically, suppose that the actual data had turned out with 2 fewer improvers in the dolphin therapy group and 2 more improvers in the control group. 14. How will this affect our analysis, and the p-value, and the conclusion? First make a conjecture, and explain your reasoning. 15. Then investigate this question with a simulation. [Hints: Think about how the two-way table will change. Then use your earlier simulation results but explain what you are doing differently now to find the approximate p-value.] Explain why your answers (about how the strength of evidence against the null model changes when the group difference is not as large) make intuitive sense. June 27, 2014 MAA PREP workshop 68 Randomization Tests Comparing Two Groups on a Quantitative Response (adapted from Exploration 6.2: Lingering Effects of Sleep Deprivation) Step 1: Ask a research question. Many students pull “all-nighters” when they have an important exam or a pressing assignment. Concerns that may arise include: Can you really function well the next day after a sleepless night? What about several days later: Can you recover from a sleepless night by getting a full night’s sleep on the following nights? Step 2: Design a study and collect data. Researchers Stickgold, James, and Hobson investigated delayed effects of sleep deprivation on learning in a study published in Nature Neuroscience (2000). Twenty-one volunteers, aged 18 to 25 years, were first trained on a visual discrimination task that involved watching stimuli appear on a computer screen and reporting what was seen. After the training period, subjects were tested. Performance was recorded as the minimum time (in milliseconds) between the appearance of stimuli and an accurate response. Following these “baseline” measurements, one group was randomly assigned to be deprived of sleep for 30 hours, followed by two full nights of unrestricted sleep, whereas the other group was allowed to get unrestricted sleep on all three nights. Following this, both groups were retested on the task to see how well they remembered the training from the first day. Researchers recorded the improvement in performance as the decrease in time required at retest compared to training. (Note: For example, if someone took 5 milliseconds (ms) to respond at the beginning of the study and then 2 ms, to respond at the end, the improvement score is 3 ms. But if someone took 2 ms at the beginning and then 5 ms at the end, the improvement score is -3 ms.) The goal of the study was to see whether the improvement scores tend to be higher for the unrestricted sleep treatment than for the sleep deprivation treatment. 1. Identify the explanatory and response variables in this study. Also classify them as either categorical or quantitative. Explanatory: Type: Response: Type: 2. Was this an experiment or an observational study? Explain how you are deciding. June 27, 2014 MAA PREP workshop 69 3. Let µunrestricted be the long-run mean improvement on this task three days later when someone has had unrestricted sleep and let µdeprived denote the long-run mean improvement when someone is sleep deprived on the first night. In words and symbols, state the null and the alternative hypotheses to investigate whether sleep deprivation has a negative effect on improvement in performance on visual discrimination tasks. (Hint for the alternative hypothesis: Do you expect the people to do better or worse when sleep deprived? Based on your answer, what sign/direction should you choose for the alternative hypothesis?) Here are the data, with positive values indicating better performance at retest than at training, and negative values indicating worse performance at retest than at training: Unrestricted-sleep group’s improvement scores (milliseconds): 25.2, 14.5, -7.0, 12.6, 34.5, 45.6, 11.6, 18.6, 12.1, 30.5 Sleep-deprived group’s improvement scores (milliseconds): -10.7, 4.5, 2.2, 21.3, -14.7, -10.7, 9.6, 2.4, 21.8, 7.2, 10.0 Step 3: Explore the Data. 4. To look at graphical and numerical summaries of the data from the study, go to the Multiple Means applet. The sleep deprivation data have already been entered into the applet. a. Notice that the applet creates parallel dotplots, one for each study group. Based on these dotplots alone, which group (unrestricted or deprived) appears to have had the higher mean improvement? How are you deciding? b. Based on the dotplots alone, which group (unrestricted or deprived) appears to have had more variability in improvement? How are you deciding? c. Notice also that the applet also computes numerical summaries of the data, such as, the mean and standard deviation (SD) for the improvements in each group. i. For the unrestricted group, record the sample size (n), mean, and SD. nunrestricted = ii. SDunrestricted = For the deprived group, record the sample size (n), mean, and SD. ndeprived= June 27, 2014 𝑥̅ unrestricted = 𝑥̅ deprived= MAA PREP workshop SDdeprived = 70 Recall from earlier in the course, that standard deviation is a measure of variability. Relatively speaking, smaller standard deviation values indicate less variability and a distribution whose data values tend to cluster more closely together, compared to a distribution with a larger standard deviation. d. Based on the numerical summaries reported in #4c, which group (unrestricted or deprived) had the higher mean improvement? e. Based on the numerical summaries reported in #4c, which group (unrestricted or deprived) had the higher variability in improvement? f. Notice that the applet also reports the observed difference in means for the improvements of the two groups. Record this value (and its measurement units). 𝑥̅ unrestricted - 𝑥̅ deprived = g. Before you conduct an inferential analysis, does this difference in sample means (as reported in #4f) strike you as a meaningful difference? Explain your answer. Step 4: Draw Inferences. 5. What are two possible explanations for why we observed the two groups to have different sample means for improvement in performance? Next you will be asked to describe how you might go about deciding whether the observed difference between the two sample means is statistically significant. (Hint: Think about how you assessed whether an observed difference between two sample proportions was statistically significant in Chapter 5. Use the same strategy, with an appropriate modification for working with means instead of proportions.) Once again the key question is how often random assignment alone would produce a difference in the groups at least as extreme as the difference observed in this study, if there really were no effect of sleep condition on improvement score. You addressed similar questions in Chapter 5 when you analyzed the dolphin therapy and yawing studies. The only change is that now the response variable is quantitative rather than categorical, so the relevant statistic is the difference in group means rather than the difference in group proportions. Also once again, we use simulation to investigate how often such an extreme difference would occur by chance (random assignment) alone (if the null hypothesis of no difference / no effect / no association were true). In other words, we will again employ the 3S Strategy. June 27, 2014 MAA PREP workshop 71 Statistic: 6. A natural statistic for measuring how different the observed group means are from each other is the difference in the mean improvement scores between the two groups. Report the value of this statistic, as you did in #4f. Simulate: You will start by using index cards to perform a tactile simulation of randomly assigning the 21 subjects between the two groups, assuming that sleep condition has no impact on improvement. Because the null hypothesis asserts that improvement score is not associated with sleep condition, we will assume that the 21 subjects would have had exactly the same improvement scores as they did, regardless of which sleep condition group (unrestricted or deprived) the subject had been assigned. 7. a. How many index cards do you need to conduct this simulation? b. What will you write on each index card? To conduct one repetition of this simulation: Shuffle the stack of 21 cards well, and then randomly distribute cards into two stacks: one stack with 10 cards (the unrestricted group) and one with 11 (the sleep deprived group). Calculate and report the sample means for each re-randomized group: Re-randomized unrestricted group’s mean: Re-randomized deprived group’s mean: Calculate the difference in group means, subtracting unrestricted mean minus sleep deprived mean. Report this value. Combine this result with your classmates’ to create a dotplot that shows the distribution of several possible values of difference in sample means that could have happened due to pure chance, if sleep condition has no impact on improvement. Sketch the dotplot below, being sure to label the horizontal axis. Label: June 27, 2014 MAA PREP workshop 72 c. At about what value is the dotplot centered? Explain why this makes sense. (Hint: What are we assuming to be true when we conduct the simulation?) d. Where is the observed difference in means from the original study (as reported in #6) on the dotplot? Did this value happen often, somewhat rarely, or very rarely? How are you deciding? 8. As before with simulation-based analyses, you would now like to conduct many, many more repetitions to determine what is typical and what is not for the difference in group means, assuming that sleep condition has no impact on improvement score. We think you would prefer to use a computer applet to do this rather than continue to shuffle cards for a very long time, calculating difference of group means by hand. Go back to the Multiple Means applet, check the Show Shuffle Options box, select the Plot display, and press Shuffle Responses. a. Describe what the applet is doing and how this relates to your null hypothesis from #3. (Note: You can check the Show Original Colors box to see which groups the observations originally belonged to.) b. Record the simulated difference in sample means for the re-randomized groups, as given in the applet output. Is this difference more extreme than the observed difference from the study (as reported in #6)? How are you deciding? c. Click on Shuffle Responses again, and record the simulated difference in sample means for the re-randomized groups. Did it change from #8b? d. Click on Re-Randomize again, and record the simulated difference in sample means for the re-randomized groups. Did it change from #8b and #8c? e. Now to see many more possible values of the difference in sample means, assuming sleep condition has no impact on improvement, do the following in the Multiple Means applet: Change Number of Shuffles from 1 to 997. Press Shuffle Responses, to produce a total of 1000 shuffles and rerandomized statistics. June 27, 2014 MAA PREP workshop 73 f. Consider the histogram of the 1000 could-have-been values of difference in sample means, assuming that sleep condition has no effect on improvement. i. What does one dot on the dotplot represent? (Hint: Think about what you would have to do to put another dot on the graph.) ii. Describe the overall shape of the null distribution displayed in this dotplot. iii. Where does the observed difference in sample means (as reported in #6) fall in this dotplot: near the middle or out in a tail? Are there a lot of dots that are even more extreme than the observed difference, assuming sleep condition has no impact on improvement score? How are you deciding? g. To estimate a p-value, continue with the Multiple Means applet. Type in the observed difference in group means (as reported in #6) in the Count Samples box (for the one-sided alternative hypothesis), and press Count. Record the approximate p-value. h. Fill in the blanks of the following sentence to complete the interpretation of the pvalue. The p-value of _______ is the probability of observing _________________________ _______________ assuming __________________________________________. Strength of Evidence: 9. Based on the p-value, evaluate the strength of evidence provided by the experimental data against the null hypothesis that sleep condition has no effect on improvement score: not much evidence, moderate evidence, strong evidence, or very strong evidence? How are you deciding? June 27, 2014 MAA PREP workshop 74 10. Significance: Summarize your conclusion with regard to strength of evidence in the context of this study. 11. Estimation: a. Use the 2SD Method to approximate a 95% confidence interval for the difference in long-run mean improvement score for subjects who get unrestricted sleep minus the long-run mean improvement score for subjects who are sleep deprived. (Hints: Remember the observed value of the difference in group means, and obtain the SD of the difference in group means from the applet’s simulation results. The interval should be observed difference in means ± 2 SD, where SD represents the standard deviation of the null distribution of the difference in group means.) b. Interpret what this confidence interval reveals, paying particular attention to whether the interval is entirely positive, entirely negative, or contains zero. (Hint: Be sure to mention “direction” in your interpretation by saying how much larger improvement scores are on average for the treatment you find to have the larger long-run mean: I’m 95% confident that the long-run mean improvement score is higher with the __ to ________ treatment as opposed to the ____________ treatment.) Step 5: Formulate Conclusions. 12. Generalization: Were the participants in this study randomly selected from a larger population? Describe the population to which you would feel comfortable generalizing the results of this study. June 27, 2014 MAA PREP workshop 75 13. Causation: Were the participants in the study randomly assigned to a sleep condition? How does this affect the scope of conclusion that you can draw? Another statistic: Could we have chosen a statistic other than the difference in group means to summarize how different the two groups’ improvement scores were? Yes, for example we could have used the difference in group medians. Why might we do this? For one reason, the median is less affected by outliers than the mean (see Section 3.2). And, we can use the Multiple Means applet to do this. 14. Step 6: Look back and ahead. Looking back: Did anything about the design and conclusions of this study concern you? Issues you may want to critique include: Any mismatch between the research question and the study design How the experimental units were selected How the treatments were assigned to the experimental units How the measurements were recorded The number of experimental units in the study Whether what we observed is of practical value Looking ahead: What should the researchers’ next steps be to fix the limitations or build on this knowledge? June 27, 2014 MAA PREP workshop 76 Common exam questions One proportion (Unit 1) Questions 1 – 5, refer to the following story. With movie-viewing-at-home made so convenient by services such as Netflix, Pay-per-view, and Video-on-demand, do a majority of city residents now prefer watching movies at home rather than going to the theater? To investigate, a local high school student, Lori, decides to conduct a poll of adult residents in her city. She selects a random sample of 100 adult residents from the city and gives each participant the choice between watching a movie at home or the same movie at the theater. She records how many choose to watch the movie at home. 1. Pick the correct null hypothesis. (Circle only one.) a) Adult residents of the city are equally likely to choose to watch the movie at home as to watch the movie at the theater. b) Adult residents of the city are more likely to choose to watch the movie at home than to watch the movie at the theater. c) Adult residents of the city are less likely to choose to watch the movie at home than to watch at the theater. d) Other: 2. Pick the correct alternative hypothesis. (Circle only one.) a) Adult residents of the city are equally likely to choose to watch the movie at home as to watch the movie at the theater. b) Adult residents of the city are more likely to choose to watch the movie at home than to watch the movie at the theater. c) Adult residents of the city are less likely to choose to watch the movie at home than to watch at the theater. d) Other: 3. After analyzing her data, Lori finds that significantly more than half of the sample (p-value 0.012) preferred to watch the movie at home. Below are two possible explanations for the observed majority. Based on her analysis, which explanation do you find the most plausible (i.e., reasonable, believable)? (Circle only one.) a) More than half of the adult residents in her city prefer to watch the movie at home. b) There is no overall preference for movie-watching-at-home in her city, but by pure chance her sample just happened to have an unusually high number of people choose to watch the movie at home. c) (a) and (b) are equally plausible explanations. June 27, 2014 MAA PREP workshop 77 4. Which of the following is the most valid interpretation of Lori’s p-value of 0.012? (Circle only one.) a) A sample proportion as large as or larger than hers would rarely occur. b) A sample proportion as large as or larger than hers would rarely occur if the study had been conducted properly. c) A sample proportion as large as or larger than hers would rarely occur if 50% of adults in the population prefer to watch the movie at home. d) A sample proportion as large as or larger than hers would rarely occur if more than 50% of adults in the population prefer to watch the movie at home. 5. Reconsider Lori’s results. If we were to construct a 95% confidence interval for the proportion of all adult residents of this city that would choose to watch the movie at home, will this interval contain the value of 0.5? (Circle only one.) a) Yes b) No c) Don’t have enough information to decide. Explain: June 27, 2014 MAA PREP workshop 78 Two proportions (Unit 2) You want to investigate a claim that women are more likely than men to dream in color. You take a random sample of men and a random sample of women (in your community) and ask whether they dream in color, and compare the proportions of each gender that dream in color. (Questions 1-6 are based on this study.) 1. If the difference in the proportions (who dream in color) between the two samples turns out not to be statistically significant (p-value was not small), which of the following would be the best conclusion to draw? a) You have found strong evidence that there is no difference between the proportions of men and women in your community that dream in color. b) You have not found enough evidence to conclude that there is a difference between the proportions of men and women in your community that dream in color. c) You have found strong evidence against the claim that there is a difference between the proportions of men and women that dream in color. d) Because the result is not significant, we can’t conclude anything from this study. Instructors may want to give partial credit for (d) 2. If the difference in the proportions (who dream in color) between the two samples does turn out to have a small p-value, which one of the following would be the best interpretation? a) It would not be very surprising to obtain the observed sample results if there is really no difference between the proportions of men and women in your community that dream in color. b) It would be very surprising to obtain the observed sample results if there is really no difference between the proportions of men and women in your community that dream in color. c) It would be very surprising to obtain the observed sample results if there is really a difference between the proportion of men and women in your community that dream in color. d) The probability is very small that there is no difference between the proportions of men and women in your community that dream in color. e) The probability is very small that there is a difference between the proportions of men and women in your community that dream in color. 3. Your review of the research suggests that there is a genuine difference between men and women on this variable. Suppose that the difference between the sample groups turns out not to be statistically significant (not a small p-value). Which conclusion would be most reasonable? a) Something went wrong with the analysis, and the results of this study cannot be trusted. b) There must not be a difference after all and the other research studies were flawed. c) The sample size might have been too small to detect a difference even if there is one. June 27, 2014 MAA PREP workshop 79 4. Suppose that two different studies are conducted on this issue. Study A finds that 40 of 100 women sampled dream in color, compared to 20 of 100 men. Study B finds that 35 of 100 women dream in color, compared to 25 of 100 men. Which study (A or B) would provide stronger evidence that there is a genuine difference between men and women on this issue? a) Study A b) Study B c) The strength of evidence would be similar for these two studies 5. Suppose two more studies are conducted on this issue. Both studies find 30% of women sampled dream in color, compared to 20% of men. But Study C consists of 100 people of each sex, whereas Study D consists of 40 people of each gender. Which study would provide stronger evidence that there is a genuine difference between men and women on this issue? a) Study C b) Study D c) The strength of evidence would be similar for these two studies 6. Suppose the difference in the sample proportions who dream in color does turn out to be statistically significant (small p-value). Below are two possible explanations for this observed difference. Which do you consider to be the more plausible (i.e., believable or reasonable) explanation for the observed result? a) Men and women in your community do not differ on this issue but by chance alone the random sampling led to the difference we observed between the two groups. b) Men and women in your community differ on this issue. c) (a) and (b) are equally plausible explanations. Possible alternative to Q5 and Q6: Suppose I were to use a simulation-based method to randomly assign yes/no responses about dreaming in color to the males and females in the study under the assumption that the likelihood of dreaming in color is the same for both groups. What would be the main purpose of this use of randomness in this simulation? a. To allow me to draw a cause-and-effect conclusion from the study. b. To allow me to generalize my results to a larger population. c. To simulate values of the statistic under the null hypothesis. d. To replicate the study and increase the accuracy of the results. . e. Other: June 27, 2014 MAA PREP workshop 80 Two means (Unit 2) A psychology student conducted a research study in which the research question is whether financial incentives can improve performance on video games. The student prepares an experiment in which 40 subjects are randomly assigned to one of two groups. The first group was offered $5 for a score above 100 and the other group was simply told to “do your best.” Each of the 40 students played the video game and achieved a certain score, from which the student calculated the following summary statistics: $5 incentive “do your best” Mean score 98 80 Sample size 20 20 These average scores differ by 98 – 80 = 18 points, but the student is concerned that this difference might be just due to random chance. To examine the question of random chance the student conducted the following simulation: She took 40 index cards. She wrote each subject’s score on one card. o She then shuffled the cards and randomly placed the cards into two stacks of 20 cards each. One stack represents $5 incentive and the other “do your best.” o She computed the mean score for each group. o She found the difference in means for the two groups by taking average for the $5 incentive group minus the average for the “do your best” group. She repeated the previous three steps 1000 times. She graphed the 1000 statistics (differences in group means) she calculated from these trials to produce the following histogram: June 27, 2014 MAA PREP workshop 81 Use this information to circle the best answer among the options to the following questions: 1. What is the motivation for the process the student followed? a) This process allows her to compare her actual result to what could have happened by chance if gamers’ performances were not affected by whether they were asked to do their best or offered an incentive. b) This process allows her to determine the percentage of time the $5 incentive strategy would outperform the “do your best" strategy for all possible scenarios. c) This process allows her to determine how many times she needs to replicate the experiment for valid results. d) This process allows her to determine whether the normal distribution fits the data. 2. Which of the following was assumed to be true in carrying out the simulation? a) The $5 incentive is more effective than the “do your best” incentive for improving performance. b) The $5 incentive and the “do your best” incentive are equally effective at improving performance. c) The “do your best” incentive is more effective than a $5 incentive for improving performance. d) Both (a) and (b) but not (c). 3. Which of the following could be the approximate p-value in this situation? Recall that the research question is “Does the $5 incentive improve performance?” a) 0.501 c) 0.022 b) 0.047 d) 0.001 4. What does the histogram tell you about whether $5 incentives are effective in improving performance on the video game? a) The $5 incentive is not effective because the distribution of differences generated is centered at zero. b) The $5 incentive is effective because distribution of differences generated is centered at zero. c) The $5 incentive is not effective because the p-value is greater than 0.05. d) The $5 incentive is effective because the p-value is less than 0.05. 5. Which of the following is the appropriate interpretation of the p-value? a) The p-value is the probability that the $5 incentive is not really helpful. b) The p-value is the probability that the $5 incentive is really helpful. c) The p-value is the probability that she would get a result as least as extreme as the one she actually found, if the $5 incentive is really not helpful. d) The p-value is the probability that a student wins on the video game. June 27, 2014 MAA PREP workshop 82 Transferability (Final exam) A consumer organization was concerned that a computer manufacturer was misleading customers by overstating the battery life of its laptop batteries (measured in hours). The manufacturer states the batteries will last 3 hours. To investigate this claim, researchers selected a random sample of 10 laptops from the current production process. The same exact operations were performed on each laptop (watching a movie, writing a term paper, surfing the internet, etc.), recording the number of hours each battery lasted. Consider the data displayed on the lifetimes of the sample of 10 laptop batteries (in hours). Suppose we want to investigate whether there is convincing evidence that the distribution of all battery lifetimes is skewed to the right. One possible statistic for measuring skewness is the ratio: (sample mean sample median) (a) Calculate the value of this statistic (sample mean divided by the sample median) for this sample. (b) If the distribution of battery lifetimes in the population is symmetric, then we would expect this skewness ratio statistic to be LESS THAN / CLOSE TO / GREATER THAN (circle one) the value of (fill in a number). (c) If the distribution of battery lifetimes in the population is skewed to the right, then we would expect this skewness ratio statistic to be LESS THAN / CLOSE TO / GREATER THAN (circle one) the value of (fill in a number). Even when the battery lifetimes in a sample of 10 batteries is skewed to the right, it is still possible that the actual population distribution of battery lifetimes is symmetric, and that the skewness observed in the sample was due to random chance alone. To simulate the selection of random samples from a hypothetical population for which the null hypothesis is true, 100 random samples, each of size 10, were taken from a perfectly symmetrically distributed population with the same mean and standard deviation as the original sample. For each of those 100 samples, the statistic sample mean divided by the sample median was calculated. A dotplot of the 100 simulated statistics is shown below. June 27, 2014 MAA PREP workshop 83 (d) (2 pts) Select the best response below to finish this statement: “Based on this simulation output, our observed sample statistic… does not provide strong evidence that the actual population distribution is skewed to the right because this null distribution is symmetric. provides strong evidence that the actual population distribution is skewed to the right because this null distribution is symmetric. does not provide strong evidence that the actual population distribution is skewed to the right because this null distribution is centered around one. provides strong evidence that the actual population distribution is skewed to the right because this null distribution is centered around one. does not provide strong evidence that the actual population distribution is skewed to the right because this null distribution ranges from .95 to 1.05. provides strong evidence that the actual population distribution is skewed to the right because this null distribution ranges from .95 to 1.05. June 27, 2014 MAA PREP workshop 84 Paired Tests (adapted from Exploration 7.1: Rounding First Base) Imagine you are at the plate in baseball and have hit a hard line drive. You want to try to stretch your hit from a single to a double. Does the path that you take to “round” first base make much of a difference? For example, is it better to take a “narrow angle” or a “wide angle” around first base? 1. Suppose 22 baseball players volunteered to participate in an experiment. Suppose also that you plan to assign a single angle, either wide or narrow, to each player. How would you decide which player ran with which base running angle? A reasonable experimental design would be to randomly assign 11 of the 22 players to run with the wide angle and the other 11 to run with the narrow angle. 2. Some runners are faster than others. Explain how random assignment controls for this, so that speed is not likely to be a confounding variable in this study. Even though random assignment tends to balance out other variables (such as speed) between the two groups, there’s still a chance that most of the fast runners could be in one group and most of the slow runners in the other group. More importantly, there’s likely to be a good bit of variability in the runners’ speeds, and that variability would make it harder to spot a difference between the base running angles even if one angle really is superior to the other. 3. Suggest a different way of conducting the experiment, to make sure that speed is completely balanced between the two groups. The data below are the times (seconds) for the first 10 off 22 runners in a matched-pairs experiment to study this question, and the graph shows dotplots of the times for all 22 runners. Subject 1 2 3 4 5 6 7 8 9 10 narrow angle 5.50 5.70 5.60 5.50 5.85 5.55 5.40 5.50 5.15 5.80 … wide angle 5.55 5.75 5.50 5.40 5.70 5.60 5.35 5.35 5.00 5.70 … June 27, 2014 MAA PREP workshop 85 One thing to notice in these distributions is that there is a lot of overlap in the two distributions (narrow mean: 5.534 sec, wide mean: 5.459 sec) and a fair bit of variability (narrow SD: 0.260 sec, wide SD: 0.273 sec). The slowest times took over six seconds, whereas the fastest times were just under five seconds. As you saw in Chapter 6, it will be difficult to detect a difference between the two methods when there is so much natural variation in the running times. However, these data are clearly paired. As discussed earlier, such data should not be treated as coming from two independent samples. Instead, we will focus our analysis on the difference in times for each runner between the two methods. Thus, we can define our parameter of interest to be µd = long-run mean difference in running speed when rounding first base with a wide angle versus a narrow angle in the population of interest. 4. State the null and alternative hypotheses (using µd), to test whether the mean difference in running speeds between the two running angles is not 0. 5. Go to the Matched Pairs applet. The default dataset is the baseball running data. Comment on the distribution of differences as revealed in the dotplot. 6. Report the mean and SD of the differences, with appropriate symbols. 7. Describe how you could flip a coin repeatedly to simulate the randomness in the way this experiment was conducted. 8. The Matched Pairs applet will perform the simulation similar to what you could do with flipping a coin. Set the number of times to Randomize to 1 and click on Randomize. One the coin tosses have determined which time will be in which column, the applet displays the re-randomized data (the colors show you the original column for each observation, so you should see a mix in each group now). The could-have-been value for the mean difference is added to the Average Difference graph. June 27, 2014 MAA PREP workshop 86 a. How does the simulated mean difference compare to observed mean difference. Circle one. More extreme Less extreme Similar b. How are you deciding? 9. Update the number of Randomizations to 99 (for a total of 100 repetitions), uncheck Animate, and press Randomize. Consider the graph “Average Difference” that the applet has created... a. How many dots are in this graph? b. What does each dot represent? 10. To see many more possible values of mean difference in sample means that could have been, IF the two running angles were swappable, update the Number of Randomizations to 900, and press Randomize (for a total of 1000 repetitions). Describe the updated “Average Difference” graph with the 1000 samples or repetitions, with regard to the following features. a. Shape: b. About what number is this graph centered? Explain why you were expecting this. c. This graph also reports a value for standard deviation, SD. Report this value, and give a simple interpretation of this value. As in, what is this value measuring? 11. You now should have generated 1000 possible values of the mean difference in running times between the two angles, assuming the null hypothesis was true and these times were the same, on average. How does the observed mean difference from your data compare to these simulated values? Is an average difference in running times like that observed in the actual study unlikely to happen by chance alone, if the two angles are the same, on average? How are you deciding? June 27, 2014 MAA PREP workshop 87 12. To quantify the strength of evidence against the null hypothesis, you can find the pvalue. Go back to the Matched Pairs applet. In the Count Samples box, make an appropriate selection from the drop down menu (Hint: what direction does your alternative hypothesis look), and enter the appropriate number in the box (Hint: at least as extreme as what number). Report the approximate p-value. 13. Use the p-value to state a conclusion in the context of the study. Be sure to comment on statistical significance. Can you conclude that there is strong evidence the wide angle and the narrow angle differ? Why or why not? Can you draw a cause-and-effect conclusion? To what population are you willing to generalize the results? 14. Alternatively, you can summarize the strength of evidence using a standardized statistic. Find the standardized statistic and confirm that the strength of evidence you receive from the p-value is approximately the same as with the standardized statistic. 15. We can again use the 2SD Method to approximate a 95% confidence interval for the mean difference in running angles. The overall structure of the formula is the same: estimate ± 2(SD) where the estimate is the sample mean difference in narrow angle minus wide angle and SD is the standard deviation of your null distribution when you did 1000 repetitions in the applet. Use these numbers to find an approximate 95% confidence interval for the population mean difference in running times among all baseball players who could use these two running angles. 16. Provide an interpretation of this confidence interval, being sure to explain the parameter in this context. 17. Describe what would change in your analysis if you had calculated differences in running times by subtracting in the other order. June 27, 2014 MAA PREP workshop 88 Randomization Tests (comparing many groups – categorical response) (adapted from Exploration 8.1: Recruiting Organ Donors) In Example P.1 we considered a study that investigated how to encourage people to be more likely to become organ donors. At the time, we did not formally analyze the data from the study. Now we are studying methods for comparing three groups, so we can further analyze the data and draw inferential conclusions from this study. Recall that researchers asked volunteer subjects to imagine that they had moved to a new state, applied for a driver’s license, and needed to decide whether or not to become an organ donor. Researchers created three different versions of the survey, each with a different donor recruiting strategy. Subjects were randomly assigned to one of the three types of recruiting strategies: The default option is to be an organ donor, and individuals have to “opt-out” of organ donation (Opt-out group) The default option is to not be an organ donor, and individuals have to “opt-in” to organ donation (Opt-in group) There is no default option, and individuals have to choose whether or not they will be an organ donor (Neutral group) 1. Write the appropriate null and alternative hypotheses in words or in symbols. 2. Researchers found that 23 of 55 subjects in the opt-in group chose to be organ donors, 41 of 50 in the opt-out group chose to be organ donors, and 44 of 56 in the neutral group chose to be organ donors. To see these data in the Multiple Proportions applet, open the OrganDonor data file and paste the two columns (with column names) into the applet and press Use Data. (Or enter/paste in the two-way table of counts in the Sample data window and press Use Table.) a. Check the Show table box to have the applet calculate the conditional proportions who agreed to become an organ donor across the three groups and produce a segmented bar graph. What are the three proportions? b. Comment on what your calculations and graph reveal about whether the default option used in the question appears to affect how likely the person is to become an organ donor. June 27, 2014 MAA PREP workshop 89 We see some differences in the sample proportions of individuals who chose to be organ donors across the three treatments, but are these differences large enough to be statistically significant? In other words, do these data provide strong evidence of an association between donor recruiting strategy (opt-in, opt-out, neutral) and choosing to become an organ donor in the population? 3. Suggest two possible explanations for how these observed proportions could have turned out to be as different from each other as they are. (Hint: Think about what our null and alternative hypotheses are.) Applying the 3S Strategy To investigate we will see how we can apply the 3S Strategy to these data. Because researchers are attempting to minimize (or at least control) the probability of Type I Errors, in cases with multiple groups they first do an overall test (see the Unit 3 introduction for more discussion on this). However, this requires computing a single statistic and considering the null distribution of that statistic. Let’s look at this approach now. 1. Statistic 4. a. Propose a formula for a statistic that could be used to measure how different the three sample proportions are. (Hint: The key here is that the statistic needs to provide a single number that somehow focuses on differences between/among the observed sample proportions.) b. For the statistic you propose, would a large value or small value give evidence for the alternative hypothesis that at least one proportion is different in the population? When we compared two proportions in Chapter 5, we used the difference in two proportions as the statistic. In this chapter, though, there are multiple proportions. There are actually several reasonable ways of summarizing the differences among the groups in one number. Once you have settled on a statistic, you apply the 3S process procedures as before – simulate the distribution of the statistic under the null hypothesis and then see where the observed value of the statistic falls in that distribution. June 27, 2014 MAA PREP workshop 90 5. One possible statistic is max p̂ – min p̂ . Review the conditional proportions you determined in #2b and calculate the observed value of this statistic for these data. Why might this not be a less than satisfactory choice? However, this statistic ignores some of the groups. Another reasonable statistic to calculate is the mean of the absolute values of differences for each pair. (We could call this statistic “MAD” for mean of absolute differences.) 6. Let’s construct the MAD statistic for these data by going through the following steps. a. Review the conditional proportions of organ donors you determined in #5b and calculate the differences in these proportions for each pair: Opt-out minus opt-in: Opt-out minus neutral: Opt-in minus neutral: b. Calculate the mean of the absolute values of these differences. So which statistic should you use? And how can you use the statistic to estimate a p-value? We’ll consider those questions in the following sections. 2. Simulation In Chapter 5 we first saw how we can simulate a null hypothesis of no association by shuffling the response variable outcomes across the explanatory variable groups. This models the random assignment process used in the data collection process, and, assuming the null hypothesis is true, “breaks any potential association” between the response and explanatory variables. We modeled this with playing cards in Chapter 5. 7. Describe how you would model a simulation of the null hypothesis with red and black playing cards now that there are three explanatory variable groups instead of two. Make sure to tell how many black cards you would need and how many red ones. Also tell what each color of cards would represent. How many cards would you deal out to each group? What would you calculate after dealing them out? 8. Now let’s use the Multiple Proportions applet to simulate a null distribution of these MAD statistics. This null distribution is again used to measure the strength of evidence provided by the observed statistic against the null hypothesis. June 27, 2014 MAA PREP workshop 91 a. Now check Show Shuffle Options, and leave Number of Shuffles at 1. Press Shuffle Response to perform one shuffle of the response variable values. You should see the shuffled response variable, the new two-way table for the simulated data, and the value of the simulated statistic (Most Recent Shuffled MAD) which is also placed in blue on the graph on the right. Select the Plot radio button to see the shuffled segmented bar graph. How does the distribution across the groups for the shuffled data compare to the original data? Is the simulated MAD statistic value closer to zero than the observed value of the MAD statistic? b. Now enter 999 for the Number of Shuffles, and press Shuffle Responses. This repeats the shuffling of the response variable 999 more times for a total of 1000 repetitions. You should see a graph of a null distribution of the MAD statistics. What is the shape of this distribution? Why is it not a bell-shaped curve centered at zero? 3. Strength of Evidence 9. To estimate a p-value, determine how often the observed value of the statistic, or something even larger, occurred in the null distribution. (Hint: Enter the observed value for the MAD statistic from the research study in the Count Samples box and press Count.) 10. Based on the p-value, summarize the strength of evidence that the sample data provide against the null hypothesis. June 27, 2014 MAA PREP workshop 92 Randomization Tests Comparing Many Groups on a Quantitative Response Research Question: An investigation of strategies that may affect memory scores. Data collection: You will be asked to study a sequence of letters for 20 seconds and then to write down as many as you can remember, in order. Your score will be the number that you remember correctly before your first error of any kind. 1) Note your score here: ____________ 2) Note your treatment group here: ______________ (the instructor will tell you the answer to this) 3) On the instructor’s computer, add your score to the appropriate column. Sketch the dotplots of the class data here. Make sure to draw a rough sketch of the three graphs in the space below, with labeled axes. 4) Did all the students in the study get the same memory score? YES NO What are a few reasons why the memory scores vary? 5) Record the averages (means) and SDs for each group. Record these below. Group Sample size Sample average Sample SD 6) Do the graphical and numerical summaries suggest that the three “treatment” groups tended to produce different responses? Explain. June 27, 2014 MAA PREP workshop 93 7) Write out the null and alternative hypotheses of what we are trying to test about memory in this experiment. Null hypothesis, H0: Alternative hypothesis, Ha: Drawing Inference Once again, we arrive at the question: how surprising are the study results to have happened by chance alone? And, we can answer that question by finding a p-value. Let us now use the 3S Strategy to find a p-value in the context of our study. That is, let us find the probability of seeing the kind of differences we are seeing between the three groups or something even more extreme, if the grouping of letters had no effect on memory scores. 8) First though, how can we measure or quantify the observed difference between the three groups? Suggest one or two ways to do this. In answering #8, you have identified a few possible statistics for the “statistic” step of the 3S Strategy. One option is to use the following statistic to quantify the observed difference in mean scores between the three groups, and call it the MAD statistic: MAD = Mean Absolute Difference = | avg1 avg2 | | avg1 avg3 | | avg2 avg3 | 3 Using the 3S Strategy: Statistic 9) For our data, find the numeric value for the MAD statistic. June 27, 2014 MAA PREP workshop 94 10) What kind of numeric values would you expect the MAD statistic to take if the type of grouping of letters had no effect on memory scores? Why? Recall that the issue of Statistical Significance asks whether the observed results would be surprising to occur by random chance alone, if there were really no effect/difference between the groups. In this case, we are exploring whether the observed MAD statistic value is statistically significant by investigating how often a MAD value as large as ours would occur by random chance alone, if there really were no effect/difference between the groups. 11) How was randomness (or random chance) used in this study: random sampling or random assignment? 12) Is it possible that even with the random assignment, we ended up with three groups where the good memorizers ended up in one group, and the not-so-good ones in the others? 13) Is it possible that the grouping of letters has absolutely no effect on memory score, and the difference observed here in our experiment is merely the result of an unlucky random assignment? Note that the more interesting question is, if the grouping of letters has absolutely no effect on memory score, what is the probability that an unlucky random assignment alone leads to the difference observed between groups in our study (as measured by the MAD statistic) or an even greater difference? We investigate this issue by assuming that all the subjects would have obtained the same memory score regardless of which group they had been in, and we repeat the random assignment process over and over, and calculate a could-have-been MAD value for each repetition. Then, we note how surprising the observed MAD value from our study would be in the long run, if the type of grouping of letters had no effect on memory scores. Of course, like we did in Chapter 6, we could shuffle index cards with the scores on them, to produce several possible values of the MAD statistic that could have happened by random chance (that is, random assignment) if the type of grouping of letters had no effect on memory scores. But we will directly go to an applet from here. NOTE: In the interest of time, we have presented the results from data collected previously. June 27, 2014 MAA PREP workshop 95 14) The data collected were copied and pasted the data into the Multiple Means applet. Pressing Use My Data on the applet produces the following output: Thus, the observed MAD statistic is: ____________________ Simulate 15) Similar to how the Multiple Proportions applet was used, we used the Multiple Means applet to produce the following randomization-based null distribution of the MAD statistic. With regard to the randomization-based null distribution of the MAD statistic: a. Is the null distribution centered at zero? Explain why your answer makes sense. June 27, 2014 MAA PREP workshop 96 b. Describe the shape of the null distribution. Is it symmetric or skewed? If it is skewed, in which direction is it skewed? Explain why your answer makes sense. Strength of evidence 16) Indicate the region of the null distribution that denotes the p-value. How did you decide where to look? Did the center and shape of the null distribution have a bearing on how you decided about what region of the null distribution denoted the p-value? 17) The (approximate) p-value was calculated to be: 0.570. Write a conclusion for the result of your hypothesis test in the context of this problem. In your conclusion, be sure to address the following: Can we conclude that type of letter grouping affects average memory scores? Why or why not? To whom can we say that these results are applicable? How do you decide? June 27, 2014 MAA PREP workshop 97 June 27, 2014 MAA PREP workshop 98 Correlation Test (adapted from Exploration 10.2: Draft Lottery) In 1970, the United States Selective Service conducted a lottery to decide which young men would be drafted into the armed forces (Fienberg, 1971). Each of the 366 birthdays in a year (including February 29) was assigned a draft number. Young men born on days assigned low draft numbers were drafted. We will regard the 366 dates of the year as observational units. We will consider two variables recorded on each date: draft number assigned to the date, and sequential date in the year (so January 31 is sequential date 31, February 1 is sequential date 32, and so on). 1. In a perfectly fair, random lottery, what should be the value of the correlation coefficient between draft number and sequential date of birthday? The graph below displays a scatterplot of the assigned draft numbers and the sequential dates. There are 366 dots, one for each day of the (leap) year. 2. a. Does the scatterplot reveal much of an association between draft number and sequential date? b. Based on the scatterplot, guess the value of the correlation coefficient. c. Does it appear that this was a fair, random lottery? It’s difficult to see much of a pattern or association in the scatterplot, so it seems reasonable to conclude that this was a fair, random lottery with a correlation coefficient near zero. But let’s dig a little deeper … June 27, 2014 MAA PREP workshop 99 3. The 1970 Draft Lottery data sheet (at the end of the exploration) shows the draft number assigned to each of the 366 birthdays. a. Find and report the draft number assigned to your birthday. b. Is your draft number in the bottom third (1-122), middle third (123-244), or top third (245-366)? 4. The second table at the end of the exploration has ordered the draft numbers within each month. a. Use this table to determine the median draft number for your birth month. b. Collaborate with your classmates to determine and report the median draft number for all twelve months. Month Median draft number Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec c. Do you notice any pattern or trend in the median draft numbers over the course of the year? (Hint: If you do not see a trend, compare the six medians from Jan – June with the 6 medians from July – Dec.) 5. The correlation coefficient for these data is r = -0.226. What does this number reveal? Is it consistent with the scatterplot? 6. a. Suggest two possible explanations (hypotheses) which could have generated the value of the observed correlation coefficient. June 27, 2014 MAA PREP workshop 100 b. In your own words, how could we go about determining whether random chance is a plausible explanation for the observed correlation value between sequential date and draft number? Explain how the 3S strategy could be applied here, in particular identifying a simulation strategy you could conduct “by hand.” Note: You do not need to actually carry out a simulation analysis. i. What is the statistic? ii. How would you simulate? iii. How would you evaluate the strength of evidence? The null hypothesis to be tested is that the lottery was conducted with a fair, random process. The null hypothesis would therefore mean that there is no association between sequential date and draft number for this process. The alternative hypothesis is that this lottery was not conducted with a fair, random process, so there is an association between sequential date and draft number. How can we assess whether the observed correlation coefficient of r = -0.226 is far enough from zero to provide convincing evidence that the lottery process was not random? Like always, we ask how unlikely it would be for a fair, random lottery to produce a correlation value as far from zero as -0.226. Also like always, we answer that question by simulating a large number of fair random lotteries, calculating the correlation coefficient for each one, and seeing how often we obtain a correlation coefficient as or more extreme (as far from zero) as -0.226. 7. Open the Corr/Regression applet. Copy the data from DraftLottery into the applet (remember to include the column titles). a. Check the Correlation Coefficient box and confirm that the correlation coefficient is -0.226. b. Check the Show Shuffle Options box and select the Correlation radio button to the right to keep track of that statistic. Then press Shuffle Y-values to simulate one fair, random lottery. (Select the Plot radio button to see the corresponding scatterplot.) Record the value of the correlation coefficient between the shuffled draft numbers and sequential date (in blue). c. Press Shuffle Y-values four more times to generate results of four more fair, random lotteries. Record the values of the correlation coefficients in the table below. June 27, 2014 MAA PREP workshop 101 Repetition Correlation coefficient 1 2 3 4 5 d. Did any of these simulated statistics from fair, random lotteries produce a correlation coefficient as extreme (far from zero) as the observed -0.226? e. Change the Number of Shuffles from 1 to 995 and press Shuffle Y-values to simulate 995 more fair, random lotteries. Look at the null distribution of these 1000 correlation coefficients. Where is this distribution centered? Why does this make sense? f. Use Count Samples pull-down menu to select Beyond. Specify the observed correlation coefficient (-.226) and press Count. What proportion of the 1000 simulated random lotteries produced a correlation coefficient at least as extreme (as far from zero in either direction) as -0.226? Report the approximate p-value. g. Interpret this p-value: This is the probability of what, assuming what? h. What conclusion would you draw from this p-value? Do you have strong evidence that the 1970 draft lottery was not conducted with a fair, random process? Explain the reasoning behind your conclusion. Once they saw these results, statisticians were quick to point out that something fishy happened with the 1970 draft lottery. The irregularity can be attributed to improper mixing of the balls used in the lottery drawing process. (Balls with birthdays early in the year were placed in the bin first, and balls with birthdays late in the year were placed in the bin last. Without thorough mixing, balls with birthdays late in the year settled near the top of the bin and so tended to be selected earlier.) The mixing process was changed for the 1971 draft lottery (e.g., two bins, one for the draft numbers and one for the birthdays), for which the correlation coefficient turned out to be r = 0.014. 8. Use your simulation results to approximate the p-value for the 1971 draft lottery. Is there any reason to suspect that this 1971 draft lottery was not conducted with a fair, random process? Explain the reasoning behind your conclusion. Also explain why you don’t need to paste in the data from the 1971 lottery first. June 27, 2014 MAA PREP workshop 102 1970 Draft Lottery Data date 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Jan 305 159 251 215 101 224 306 199 194 325 329 221 318 238 17 121 235 140 58 280 186 337 118 59 52 92 355 77 349 164 211 June 27, 2014 Feb 86 144 297 210 214 347 91 181 338 216 150 68 152 4 89 212 189 292 25 302 363 290 57 236 179 365 205 299 285 Mar 108 29 267 275 293 139 122 213 317 323 136 300 259 354 169 166 33 332 200 239 334 265 256 258 343 170 268 223 362 217 30 Apr 32 271 83 81 269 253 147 312 219 218 14 346 124 231 273 148 260 90 336 345 62 316 252 2 351 340 74 262 191 208 May 330 298 40 276 364 155 35 321 197 65 37 133 295 178 130 55 112 278 75 183 250 326 319 31 361 357 296 308 226 103 313 Jun 249 228 301 20 28 110 85 366 335 206 134 272 69 356 180 274 73 341 104 360 60 247 109 358 137 22 64 222 353 209 Jul 93 350 115 279 188 327 50 13 277 284 248 15 42 331 322 120 98 190 227 187 27 153 172 23 67 303 289 88 270 287 193 MAA PREP workshop Aug 111 45 261 145 54 114 168 48 106 21 324 142 307 198 102 44 154 141 311 344 291 339 116 36 286 245 352 167 61 333 11 Sep 225 161 49 232 82 6 8 184 263 71 158 242 175 1 113 207 255 246 177 63 204 160 119 195 149 18 233 257 151 315 Oct 359 125 244 202 24 87 234 283 342 220 237 72 138 294 171 254 288 5 241 192 243 117 201 196 176 7 264 94 229 38 79 Nov 19 34 348 266 310 76 51 97 80 282 46 66 126 127 131 107 143 146 203 185 156 9 182 230 132 309 47 281 99 174 Dec 129 328 157 165 56 10 12 105 43 41 39 314 163 26 320 96 304 128 240 135 70 53 162 95 84 173 78 123 16 3 100 103 The following table arranges the draft numbers in order for each month: rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Jan 17 52 58 59 77 92 101 118 121 140 159 164 186 194 199 211 215 221 224 235 238 251 280 305 306 318 325 329 337 349 355 June 27, 2014 Feb 4 25 57 68 86 89 91 144 150 152 179 181 189 205 210 212 214 216 236 285 290 292 297 299 302 338 347 363 365 Mar 29 30 33 108 122 136 139 166 169 170 200 213 217 223 239 256 258 259 265 267 268 275 293 300 317 323 332 334 343 354 362 Apr 2 14 32 62 74 81 83 90 124 147 148 191 208 218 219 231 252 253 260 262 269 271 273 312 316 336 340 345 346 351 May 31 35 37 40 55 65 75 103 112 130 133 155 178 183 197 226 250 276 278 295 296 298 308 313 319 321 326 330 357 361 364 Jun 20 22 28 60 64 69 73 85 104 109 110 134 137 180 206 209 222 228 247 249 272 274 301 335 341 353 356 358 360 366 Jul 13 15 23 27 42 50 67 88 93 98 115 120 153 172 187 188 190 193 227 248 270 277 279 284 287 289 303 322 327 331 350 MAA PREP workshop Aug 11 21 36 44 45 48 54 61 102 106 111 114 116 141 142 145 154 167 168 198 245 261 286 291 307 311 324 333 339 344 352 Sep 1 6 8 18 49 63 71 82 113 119 149 151 158 160 161 175 177 184 195 204 207 225 232 233 242 246 255 257 263 315 Oct 5 7 24 38 72 79 87 94 117 125 138 171 176 192 196 201 202 220 229 234 237 241 243 244 254 264 283 288 294 342 359 Nov 9 19 34 46 47 51 66 76 80 97 99 107 126 127 131 132 143 146 156 174 182 185 203 230 266 281 282 309 310 348 Dec 3 10 12 16 26 39 41 43 53 56 70 78 84 95 96 100 105 123 128 129 135 157 162 163 165 173 240 304 314 320 328 104 FAQ 10.3.1 Why least squares? Buzz: Why least squares? Why do we square the deviations? Why not absolute values? Doris: Not only is your question a thoughtful one, it’s also a question that invites us back in history. In the 1700s, scientists tried to do what you suggest – to minimize the sum of absolute deviations: they wanted to use least MAD (deviation) instead of least sum of squared deviations. The scientific logic was good, but the math turned out to be too messy. Sometimes it worked, and there was a single “best” line that minimized the MAD statistic. But sometimes there would be infinitely many “best” lines, with no way to choose from among them. “Best” and “infinitely many” don’t go together. Even when there was a single best line, it was devilishly hard to find. Overall, the math was just too hard. Buzz: So least squares is pretty much just the fallback Plan B after least absolute deviations failed? Doris: Yes. The justification for least absolute deviations makes good sense, but the theory is much simpler for least squares. As Fred Mosteller, one of the greatest statisticians of the last century, used to tell his students at Harvard, “Mathematicians love calculus, and calculus loves squares.” Buzz: Does the choice – absolute values or squares -- make much difference in the line you get? Doris: Another good question. Sometimes not, but sometimes it does. Buzz: Say more. Doris: Remember SD and IQR and outliers? Buzz: You mean SD is sensitive to outliers and IQR is resistant? Doris: Exactly. Remember why? Buzz: Because the SD squares the deviations, and the IQR does not. Squaring gives extra weight to large deviations. Doris: Apply the same logic to the least squares line. Buzz: Outliers have a big effect because their large deviations get extra weight from the squaring. Doris: Bravo! This is why some observations can become influential points – the least squares line won’t let their residuals get too large. If this were second grade in the 1950s you’d get to stay after class and dust the erasers. If this were the 1970s and you were a statistician, you’d be able to get a big research grant from the National Science Foundation. If this was Dr. Bastian’s lab and you were a dolphin, you’d have just earned a fish. June 27, 2014 MAA PREP workshop 105 Regression (adapted from Exploration 10.3: Predicting Height from Footprints) Can a footprint taken at the scene of a crime help to predict the height of the criminal? In other words, is there an association between height and foot length? To investigate this question, a sample of 20 statistics students measured their height (in inches) and their foot length (in centimeters). The sample data for the 20 students appear in the Correlation/Regression applet. 1. Open this applet, and look at the scatterplot for the data already there. Check the Show Movable Line box to add a blue line to the scatterplot. If you place your mouse over one of the green squares at the ends of the line and drag, you can change the slope of the line and move it. You can also use the mouse to move the green dot up and down vertically to change the intercept of the line. a. Move the line until you believe your line “best” summarizes the relationship between height and foot length for these data. Write down the resulting equation for your line (using traditional statistical notation). b. Why do you believe that your line is “best?” c. Did all students in your class obtain the same line/equation? d. How can we decide whether your line provides a better fit to the data than other students’ lines? Suggest a criterion for deciding which line “best” summarizes the relationship. One way to draw the best fit line is to minimize the distance of the points to the line (these distances are called residuals). Key idea: A residual is the difference between an observed response and the corresponding prediction made by the least squares regression line (residual = observed – predicted). Thus, negative residuals occur when points are below the best fit line and positive residuals occur when points are above the best fit line. June 27, 2014 MAA PREP workshop 106 2. Check the Show Residuals box to visually represent these residuals for your line on the scatterplot. The applet also reports the sum of the values of the residuals (SAE). SAE stands for “Sum of the Absolute Errors.” The acronym indicates that we need to make residuals positive before we add them up and that sometimes people call residuals “errors.” Record the SAE value for your line _____________ What is the best (lowest) SAE in the class?_________________ 3. It turns out that a more common criterion for determining the “best” line is to instead look at the sum of the squared residuals (SSE). This approach is similar to simply adding up the residuals, but is even more strict in not letting individual residuals get too large. Check the Show Squared Residuals box to visually represent the squared residual for each observation. Note that we can visually represent the squared residual as the area of a square where each side of the square has length equal to the residual. a. What is the SSE (sum of squared residuals) for your line?____________ What is the best (lowest) SSE in the class?________ b. Now continue to adjust your line until you think you have minimized the sum of the squared residuals. Report your new equation_________________________________ Report your new SSE value_____________________________ What is the best SSE in the class?_________________________ Key Idea: The least squares regression line minimizes the sum of squared residuals. 4. Now check the Show Regression Line box to determine and display the equation for the line that actually does minimize (as can be shown using calculus) the sum of the squared residuals. a. Record the equation of the least squares regression line by indicating the appropriate slope and intercept of the line. Note that we’ve used variable names in the equation, not generic x and y. And put a carat (“hat”) over the y variable name to emphasize that the line gives predicted values of the y (response) variable. ̂ = _______ + _______(𝐹𝑜𝑜𝑡𝐿𝑒𝑛𝑔𝑡ℎ) 𝐻𝑒𝑖𝑔ℎ𝑡 Notation: The equation of the best fit line is written as 𝑦̂ = 𝑎 + 𝑏(𝑥) where 𝑎 is the y-intercept 𝑏 is the slope 𝑥 is a value of the explanatory variable 𝑦̂ is the predicted value for the response variable June 27, 2014 MAA PREP workshop 107 b. Did everyone in your class obtain the same equation? 5. Uncheck the Show Movable Line box to remove it from the display and check the Move observations box. a. Now click on one student’s point in the scatterplot and drag the point up and down (changing the height, without changing the foot length). Does the regression line change much as you change this student’s height? b. Repeat the previous question, using a student with a very small x (foot size) value and then a point with an x value near the middle and then a point with a very large x value. Which of these seem(s) to have the most influence on the least squares regression line? Explain. Key idea: An observation or set of observations is considered influential if removing the observation from the dataset substantially changes the values of the correlation coefficient and/or the least squares regression equation. Typically, observations that have extreme explanatory variable values (far below or far above x ) are potentially influential. They may not have large residuals, having pulled the line close to them. Residuals also help us measure how accurate our predictions are from using the regression line. In particular, we can compare the “prediction errors” from the regression line to the prediction errors if we made no use of the explanatory variable. 6. Press the Revert button to reload the original data set. Recheck the Show Movable Line box to redisplay the blue line. Notice that this line is flat at the mean of the y (height) values. a. Check the Show Squared Residuals box (under the Movable Line information) to determine the SSE if we were to use the average height ( y ) as the predicted value for every x (foot size). Record this value. b. What is the slope of this line? c. If the slope of the best fit line is zero, our data shows __________________ (positive/negative/no) association between the explanatory and response variables. June 27, 2014 MAA PREP workshop 108 7. Now compare this to the SSE value from the regression line. Which is smaller? Why does that make sense? Coefficient of Determination (r2) 8. A quantity related to the correlation coefficient is called the coefficient of determination, or r-squared (r2). a. To measure how much the regression line decreases the “unexplained variability” in the response variable, we can calculate the percentage reduction in SSE. SSE( y ) – SSE(regression line) x 100% = SSE( y ) Key Idea: The coefficient of determination indicates the percentage of variation in the response variable (height) that is explained by changes in the explanatory variable (foot length). b. Complete the following statement: The coefficient of determination is ______________percent, this means that _____________ percent of the variation in people’s ______________________ is attributable to changes in their ______________________________. If we think of 𝑆𝑆𝐸(𝑦̅) as the “worst” line (no association), then we are seeing how much better the actual line is than this worst line. The best possible line would get rid of all of the residuals, and would have an r-squared of 100%. c. Find the value of the correlation coefficient using the applet, and confirm that when you square the correlation coefficient you get the same number as the applet reports for the coefficient of determination. June 27, 2014 MAA PREP workshop 109