Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Applied Data Analysis Spring 2017 Yemen, 2016 Karen Albert [email protected] Thursdays, 4-5 PM (Hark 302) Lecture outline 1. Percentages 2. Central limit theorem 3. Inference Let’s do some political science McClatchy-Marist poll released yesterday: 17% said that they trust President Trump to "deliver accurate and factual information to the public." • The percentage that is reported is a mean. • More specifically, it is a mean of 1’s and 0’s. • So the percentage is the expected value. • We can then figure out the standard error. The margin of error is twice the standard error. (Why twice?) Let’s do some political science McClatchy-Marist poll released yesterday: 17% said that they trust President Trump to "deliver accurate and factual information to the public." • The percentage that is reported is a mean. • More specifically, it is a mean of 1’s and 0’s. • So the percentage is the expected value. • We can then figure out the standard error. The margin of error is twice the standard error. (Why twice?) The poll number are therefore: expected value ± (2xS.E.) A short-cut to the SD for a special case For boxes with only two kinds of tickets, there is short-cut to finding the standard deviation. S = (big − small) × p big fraction × small fraction A short-cut to the SD for a special case For boxes with only two kinds of tickets, there is short-cut to finding the standard deviation. S = (big − small) × p big fraction × small fraction Consider a box with the values: 1, 1, 1, 5. The standard deviation is r S = (5 − 1) × 1 3 × ≈ 1.73 4 4 The short-cut x <- c(1,1,1,5) sd(x)*(3/4) (5-1)*sqrt(.25*.75) The short-cut x <- c(1,1,1,5) sd(x)*sqrt(3/4) ## [1] 1.732051 (5-1)*sqrt(.25*.75) ## [1] 1.732051 Another example If you bet a dollar on a single number at Nevada roulette, and that number comes up, you get the $1 back together with winnings of $35. If any other number comes up, you lose the dollar. Suppose you play 100 times, betting a dollar on the number 17 each time. Another example If you bet a dollar on a single number at Nevada roulette, and that number comes up, you get the $1 back together with winnings of $35. If any other number comes up, you lose the dollar. Suppose you play 100 times, betting a dollar on the number 17 each time. What is your chance of breaking even? Another example If you bet a dollar on a single number at Nevada roulette, and that number comes up, you get the $1 back together with winnings of $35. If any other number comes up, you lose the dollar. Suppose you play 100 times, betting a dollar on the number 17 each time. What is your chance of breaking even? Hint: there are 38 spaces on a roulette wheel. The box model and the expected value The box has 38 tickets in it: 1 worth 35 and 37 worth -1. The mean of the box is 35*(1/38)+(-1)*(37/38) ## [1] -0.05263158 The standard deviation Since there are only two kinds of tickets in the box, we can use the short-cut. (35-(-1))*sqrt((1/38)*(37/38)) ## [1] 5.762617 So the standard error is 5.76/sqrt(100) ## [1] 0.576 End of the example What is the probability that you break even? The mean (expected value) is -0.0526. The standard deviation (standard error) is 0.576. End of the example What is the probability that you break even? The mean (expected value) is -0.0526. The standard deviation (standard error) is 0.576. pnorm(0,-.0526,.576,lower.tail=FALSE) ## [1] 0.4636194 So the probability of breaking even is roughly 46%. SRS and chance error Let’s consider the percentage of Democrats in a sample. Assume that we took a simple random sample. With an SRS, the expected value for the sample percentage equals the population percentage. It won’t be exact, of course. The amount it is off is governed by chance, and we can use a box model to understand it. How BIG? % of 1’s in the sample = % of 1’s in the box + chance error How BIG? % of 1’s in the sample = % of 1’s in the box + chance error How big is the error likely to be? How BIG? % of 1’s in the sample = % of 1’s in the box + chance error How big is the error likely to be? The standard error tells us. How BIG? % of 1’s in the sample = % of 1’s in the box + chance error How big is the error likely to be? The standard error tells us. Calculation: 1. Find the standard deviation for the number of 1’s in the sample. 2. Find the standard error. 3. Convert to a percentage. Calculating the standard error (part one) Consider a box with 3,091 1’s and 3581 0’s. Assume that we take 100 draws from the box. Calculating the standard error (part one) Consider a box with 3,091 1’s and 3581 0’s. Assume that we take 100 draws from the box. The fraction of 1’s in the box is 0.46. Calculating the standard error (part one) Consider a box with 3,091 1’s and 3581 0’s. Assume that we take 100 draws from the box. The fraction of 1’s in the box is 0.46. The standard deviation is therefore, sqrt(.46*.54) ## [1] 0.4983974 Calculating the standard error (part deux) Now that we have the SD, the standard error is, sqrt(.46*.54/100) ## [1] 0.04983974 The percentage of 1’s in the sample is 46% give or take 5% or so. What if the sample size increases? First, what happens to the standard error of the sum? What if the sample size increases? First, what happens to the standard error of the sum? √ 100 × 0.5 = 5 What if the sample size increases? First, what happens to the standard error of the sum? √ √ 100 × 0.5 = 5 400 × 0.5 = 10 What if the sample size increases? First, what happens to the standard error of the sum? √ √ 100 × 0.5 = 5 400 × 0.5 = 10 Now, what about the standard error for the percentage? What if the sample size increases? First, what happens to the standard error of the sum? √ √ 100 × 0.5 = 5 400 × 0.5 = 10 Now, what about the standard error for the percentage? 0.5 √ 100 = 0.05 What if the sample size increases? First, what happens to the standard error of the sum? √ √ 100 × 0.5 = 5 400 × 0.5 = 10 Now, what about the standard error for the percentage? 0.5 √ 100 0.5 √ 400 = 0.05 = 0.025 The normal curve yet again Remember a box filled with 3,091 1’s and 3,581 0’s. The percentage of 1’s in the sample is 46% give or take 5% or so. Can we estimate the chances of the percentage being greater than 56%? The normal curve yet again Remember a box filled with 3,091 1’s and 3,581 0’s. The percentage of 1’s in the sample is 46% give or take 5% or so. Can we estimate the chances of the percentage being greater than 56%? We did this with the normal curve before, but the data this time are not normal.... The Central Limit Theorem When drawing at random with replacement from a box, the sampling distribution of the sample mean will follow the normal curve, even if the contents of the box do not. The Central Limit Theorem When drawing at random with replacement from a box, the sampling distribution of the sample mean will follow the normal curve, even if the contents of the box do not. There are conditions: 1. the sampling distribution must be put into standard units, 2. the number of draws must be reasonably large. Central Limit Theorem • Consider a variable uniformly distributed between 0 and 10. • Suppose that we take a random sample of size n from this distribution and compute the mean. • Next, we put the mean in standard units by dividing by the square root of n. • Repeat this process 4000 times. • What would the resulting sampling distribution look like? Sample size=1 Sample size=5 Sample size=10 Sample size=30 Actual inference OK, because I’m a nice guy, I keep telling you the contents of the box. Actual inference OK, because I’m a nice guy, I keep telling you the contents of the box. What if I don’t tell you the contents of the box, but only the draws from the box? Actual inference OK, because I’m a nice guy, I keep telling you the contents of the box. What if I don’t tell you the contents of the box, but only the draws from the box? This is the essence of inference. Actual inference OK, because I’m a nice guy, I keep telling you the contents of the box. What if I don’t tell you the contents of the box, but only the draws from the box? This is the essence of inference. Using the draws to learn about the box is just like using a sample to learn about a population. Box to sample If we don’t know the contents of the box, we have to substitute the fractions observed in the sample for the unknown fractions in the box. Example In a certain city, there are 100,000 persons age 18 to 24. A simple random sample of 500 such persons is drawn, of whom 194 turn out to be currently enrolled in college. Estimate the percentage of all persons age 18 to 24 in that city who are currently enrolled in college. Example In a certain city, there are 100,000 persons age 18 to 24. A simple random sample of 500 such persons is drawn, of whom 194 turn out to be currently enrolled in college. Estimate the percentage of all persons age 18 to 24 in that city who are currently enrolled in college. The sample percentage is, 194 = 0.388 500 The standard deviation The SD of the box, using the observed fractions, is, p % of 10 s × % of 00 s p = 0.388(1 − 0.388) s = = 0.487 The standard error The standard error is therefore 0.487 √ 500 = 0.0218 s.e. = The standard error The standard error is therefore 0.487 √ 500 = 0.0218 s.e. = The solution is therefore 38.8% give or take 2.18%. Standard error for a percentage r s.e. = p̂(1 − p̂) n • p̂ is the observed fraction of “successes” in the sample. • n is the number of draws (the size of the sample). Standard error for a percentage r s.e. = p̂(1 − p̂) n • p̂ is the observed fraction of “successes” in the sample. • n is the number of draws (the size of the sample). Interpretation: average distance of the p̂ values from the true, but unknown, percentage. Another example There are 4000 undergraduates at the University of Rochester. A simple random sample of 400 students is drawn, of whom 20 turn out to like Nickelback. Estimate the percentage of all students at UR that like Nickelback. Solution The sample percentage is p̂ = 20/400 = 0.05. Solution The sample percentage is p̂ = 20/400 = 0.05. The standard error is r 0.05(1 − 0.05) 400 = 0.0109 s.e. = Solution The sample percentage is p̂ = 20/400 = 0.05. The standard error is r 0.05(1 − 0.05) 400 = 0.0109 s.e. = The solution is therefore 5% give or take 1.09%. What did we learn? • Percentages • Central limit theorem • Inference