Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Statistics for Business and Economics Module 1:Probability Theory and Statistical Inference Spring 2010 Lecture 1 Priyantha Wijayatunga, Department of Statistics, Umeå University [email protected] These materials are altered ones from copyrighted lecture slides (© 2009 W.H. Freeman and Company) from the homepage of the book: The Practice of Business Statistics Using Data for Decisions :Second Edition by Moore, McCabe, Duckworth and Alwan. Introduction What is statistics? Use of data Population and sample Types of variables Graphical display of data Numerical summeries of data Obtaining data What is statistics? Use of data Science of Statistics provides collection of methods and tools for collection, organization, analysis, interpretion and presention of data that are usually numerical figures. The goal of statistics is to gain understanding of the phenomenon from data. Therefore we use data to understand the phenomenon Collection (design): planning how to gather data Presentation (description): summerizing data Analysis and interpretation (inference): making prediction based on data Population and sample Popuation is the collection of all the individuals/items under study. Sample is a subcollection (a part) individuals/items of the population. Eg: If we study the life age structure of the Swedes then Population: all the Swedes A sample: all the Swedes living in Umeå Parameter: a parameter is a numerical summary of the population. Statistic: a numerical summary of a data sample. – when parameters are not known they are estimated by corresponding sample statistics Population versus sample Population: The entire group of individuals in which we are interested but can’t usually assess directly. Sample: The part of the population we actually examine and for which we do have data. How well the sample represents the population depends on the sample design. Example: All humans, all working-age people in California, all crickets Population Sample A parameter is a number describing a characteristic of the population. A statistic is a number describing a characteristic of a sample. Variables In a study, we collect information—data—from individuals/items. They can be people, animals, plants, or any object of interest. A variable is any characteristic of an individual/item. A variable usually varies among individuals/items. Example: age, height, blood pressure, ethnicity, leaf length, first language The distribution of a variable tells us what values the variable takes and how often it takes these values. Two types of variables Variables can be either quantitative… Something that takes numerical values for which arithmetic operations such as adding and averaging make sense Example: How tall you are, your age, your blood cholesterol level, the number of credit cards you own … or categorical (qualitative). Something that falls into one of several categories. What can be counted is the count or proportion of individuals/items in each category. Example: Your blood type (A, B, AB, O), your hair color, your ethnicity, whether you paid income tax last tax year or not How do you know if a variable is categorical or quantitative? Ask: What are the n individuals/units in the sample (of size “n”)? What is being recorded about those n individuals/units? Is that a number ( quantitative) or a statement ( categorical)? Categorical Quantitative Each individual is assigned to one of several categories. Each individual is attributed a numerical value. Individuals in sample DIAGNOSIS AGE AT DEATH Patient A Heart disease 56 Patient B Stroke 70 Patient C Stroke 75 Patient D Lung cancer 60 Patient E Heart disease 80 Patient F Accident 73 Patient G Diabetes 69 Two types of variables 1. Qualitative (Categorical) : Measuremnet scale is a set of categories. 1. Nominal: unordered categories [discrete] (residence {Umeå, Stockholm,Uppsala} ) 2. Ordinal: ordered categories [discrete] ( income {Lower, Middle, Upper} ) 2. Quantitative: Measurement scale has numerical values and they represent the magnitude of the variable 1. Interval: has a numerical distance between values (prison term {<5, 5–10, 10–15, 15–20, 20>} or number of times married {0, 1, 2, 3, 4, >5}) 2. Continuous: Weight Variables are either 1. Discrete: has a set of separate numbers (number of children) (values cannot be subdivided) or 2. Continuous: has infinite continuum of real numbers (weight) (between any two possible values, there áre many possible values) Displaying distributions/data with graphs (Graphical summary of data) Ways to chart categorical data Bar graphs Pie charts Ways to chart quantitative data Histograms Interpreting histograms Stemplots Stemplots versus histograms Time plots Ways to chart categorical data Because the variable is categorical, the data in the graph can be ordered any way we want (alphabetical, by increasing value, by year, by personal preference, etc.) Bar graphs Each category is represented by a bar. Pie charts The slices must represent the parts of one whole. Bar graphs Each category is represented by one bar. The bar’s height shows the count (or sometimes the percentage) for that particular category. Accidents involving Firestone tire models Bar graph sorted by rank (Pareto Chart) Easy to analyze Sorted alphabetically Much less useful Pie charts Each slice represents a piece of one whole. The size of a slice depends on what percent of the whole this category represents. Child poverty before and after government intervention—UNICEF, 1996 What does this chart tell you? •The United States has the highest rate of child poverty among developed nations (22% of under 18). •Its government does the least—through taxes and subsidies—to remedy the problem (size of orange bars and percent difference between orange/blue bars). Could you transform this bar graph to fit in 1 pie chart? In two pie charts? Why? The poverty line is defined as 50% of national median income. Ways to chart quantitative data Histograms and stemplots These are summary graphs for a single variable. They are very useful to understand the pattern of variability in the data. Line graphs: time plots Use when there is a meaningful sequence, like time. The line connecting the points helps emphasize any change over time. Histograms The range of values that a variable can take is divided into equal size intervals. The histogram shows the number of individual data points that fall in each interval. Example: Histogram of the December 2004 unemployment rates in the 50 states and Puerto Rico. Interpreting histograms When describing the distribution of a quantitative variable, we look for the overall pattern and for striking deviations from that pattern. We can describe the overall pattern of a histogram by its shape, center, and spread. Histogram with a line connecting each column too detailed Histogram with a smoothed curve highlighting the overall pattern of the distribution Most common distribution shapes Symmetric distribution A distribution is symmetric if the right and left sides of the histogram are approximately mirror images of each other. A distribution is skewed to the right if the right side of the histogram (side with larger values) extends much farther out than the left side. It is skewed to the left if the left side of the histogram Skewed distribution extends much farther out than the right side. Complex, multimodal distribution Not all distributions have a simple overall shape, especially when there are few observations. Outliers An important kind of deviation is an outlier. Outliers are observations that lie outside the overall pattern of a distribution. Always look for outliers and try to explain them. The overall pattern is fairly symmetrical except for 2 states clearly not belonging to the main trend. Alaska and Florida have unusual representation of the elderly in their population. A large gap in the distribution is typically a sign of an outlier. Alaska Florida How to create a histogram It is an iterative process – try and try again. What bin size should you use? Not too many bins with either 0 or 1 counts Not overly summarized that you loose all the information Not so detailed that it is no longer summary rule of thumb: start with 5 to10 bins Look at the distribution and refine your bins (There isn’t a unique or “perfect” solution) Histogram of Drydays in 1995 IMPORTANT NOTE: Your data are the way they are. Do not try to force them into a particular shape. It is a common misconception that if you have a large enough data set, the data will eventually turn out nice and symmetrical. Stemplots (Stem–and–leaf plots) How to make a stemplot: 1) Separate each observation into a stem, consisting of all but the final (rightmost) digit, and a leaf, which is that remaining final digit. Stems may have as many digits as needed, but each leaf contains only a single digit. 2) Write the stems in a vertical column with the smallest value at the top, and draw a vertical line at the right of this column. 3) Write each leaf in the row to the right of its stem, in increasing order out from the stem. STEM LEAVES Stemplot To compare two related distributions, a back-to-back stemplot with common stems is useful. Stemplots do not work well for large datasets. When the observed values have too many digits, trim the numbers before making a stemplot. When plotting a moderate number of observations, you can split each stem. Stemplots of the December 2004 unemployment rates in the 50 states. (b) uses split stems. Stemplots versus histograms Stemplots are quick and dirty histograms that can easily be done by hand, therefore very convenient for back of the envelope calculations. However, they are rarely found in scientific or laymen publications. Line graphs: time plots In a time plot, time always goes on the horizontal, x axis. We describe time series by looking for an overall pattern and for striking deviations from that pattern. In a time series: A trend is a rise or fall that persists over time, despite small irregularities. A pattern that repeats itself at regular intervals of time is called seasonal variation. Retail price of fresh oranges over time Time is on the horizontal, x axis. The variable of interest—here “retail price of fresh oranges”— goes on the vertical, y axis. This time plot shows a regular pattern of yearly variations. These are seasonal variations in fresh orange pricing most likely due to similar seasonal variations in the production of fresh oranges. There is also an overall upward trend in pricing over time. It could simply be reflecting inflation trends or a more fundamental change in this industry. A time plot can be used to compare two or more data sets covering the same time period. 1918 influenza epidemic # Cases # Deaths Date 1918 influenza epidemic 800 700 700 600 600 500 500 400 400 300 300 200 200 100 100 0 0 wek ee1 we k 1 wek ee3 we k 3 wek ee5 we k 5 wek ee7 we k 7 wek we ee9k ewk 9 ee11 we k 1 ewk 1 e1 we ek3 ewk 13 e1 we ek5 ewk 15 ee17 k 17 0 # deaths reported 800 10000 9000 10000 8000 9000 7000 8000 6000 7000 5000 6000 4000 5000 3000 4000 3000 2000 2000 1000 1000 0 we 0 0 130 552 738 414 198 90 56 50 71 137 178 194 290 310 149 Incidence 36 531 4233 8682 7164 2229 600 164 57 722 1517 1828 1539 2416 3148 3465 1440 # cases diagnosed week 1 week 2 week 3 week 4 week 5 week 6 week 7 week 8 week 9 week 10 week 11 week 12 week 13 week 14 week 15 week 16 week 17 1918 influenza epidemic # Cases # Cases # Deaths # Deaths The pattern over time for the number of flu diagnoses closely resembles that for the number of deaths from the flu, indicating that about 8% to 10% of the people diagnosed that year died shortly afterward from complications of the flu. Scales matter Death rates from cancer (US, 1945-95) Death rates from cancer (US, 1945-95) Death rate (per thousand) 250 200 150 100 250 Death rate (per thousand) How you stretch the axes and choose your scales can give a different impression. 200 150 100 50 50 0 1940 1950 1960 1970 1980 1990 0 1940 2000 1960 1980 2000 Years Years Death rates from cancer (US, 1945-95) 250 Death rates from cancer (US, 1945-95) 220 Death rate (per thousand) Death rate (per thousand) 200 150 100 50 0 1940 1960 Years 1980 2000 A picture is worth a thousand words, 200 BUT 180 160 There is nothing like hard numbers. Look at the scales. 140 120 1940 1960 1980 Years 2000 Describing distributions/data with numbers (Numerical summary of data) Measures of center (central location): mean, median, mode Comparing mean, median and mode Measures of spread: range, interquartile range, standard deviation, Five-number summary and boxplots Choosing measures of center and spread Measure of center : the mean The mean or arithmetic average To calculate the average, or mean, add all values, then divide by the number of individuals. It is the “center of mass.” Sum of heights is 1598.3 divided by 25 women = 63.9 inches 58 .2 59 .5 60 .7 60 .9 61 .9 61 .9 62 .2 62 .2 62 .4 62 .9 63 .9 63 .1 63 .9 64 .0 64 .5 64 .1 64 .8 65 .2 65 .7 66 .2 66 .7 67 .1 67 .8 68 .9 69 .6 x1 x2 ... xn x n Example: Mean earnings of Black females 1 n x xi n i 1 262,934 x $17,528.93 15 Measure of center: the median The median is the midpoint of a distribution—the number such that half of the observations are smaller and half are larger. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 0.6 1.2 1.6 1.9 1.5 2.1 2.3 2.3 2.5 2.8 2.9 3.3 3.4 3.6 3.7 3.8 3.9 4.1 4.2 4.5 4.7 4.9 5.3 5.6 25 12 6.1 1. Sort observations by size. n = number of observations ______________________________ 2.a. If n is odd, the median is observation (n+1)/2 down the list n = 25 (n+1)/2 = 26/2 = 13 Median = 3.4 2.b. If n is even, the median is the mean of the two middle observations. n = 24 n/2 = 12 Median = (3.3+3.4) /2 = 3.35 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 8 9 10 11 0.6 1.2 1.6 1.9 1.5 2.1 2.3 2.3 2.5 2.8 2.9 3.3 3.4 3.6 3.7 3.8 3.9 4.1 4.2 4.5 4.7 4.9 5.3 5.6 Measure of center : the Mode The mode is the most frequent observation Data: 5, 7, 6, 7, 6, 6, 4, 6, 5, 4 Mode is 6 Comparing the mean, the median and the mode The mean and the median are the same only if the distribution is symmetrical. The median is a measure of center that is resistant to skew and outliers. The mean and mode are not. Mean, median and mode for a symmetric distribution Mean Median Mode Left skew Mean Median Mode Mean, median and mode for skewed distributions Mean Median Mode Right skew Mean and median of a distribution with outliers Percent of people dying x 3.4 x 4.2 Without the outliers With the outliers The mean is pulled to the The median, on the other hand, right a lot by the outliers is only slightly pulled to the right (from 3.4 to 4.2). by the outliers (from 3.4 to 3.6). Impact of skewed data Mean and median of a symmetric Disease X: x 3.4 M 3.4 Mean and median are the same. … and a right-skewed distribution Multiple myeloma: x 3.4 M 2.5 The mean is pulled toward the skew. Measures of spread: range & Minimum = 0.6 interquartile range The first quartile, Q1, is the value in the sample that has 25% of the data at or below it ( it is the median of the lower half of the sorted data, excluding M). The third quartile, Q3, is the value in the sample that has 75% of the data at or below it ( it is the median of the upper half of the sorted data, excluding M). Range = maximum – minimum Interquartile range = Q3 – Q1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 2 3 4 5 6 7 1 2 3 4 5 1 2 3 4 5 6 7 1 2 3 4 5 0.6 1.2 1.6 1.9 1.5 2.1 2.3 2.3 2.5 2.8 2.9 3.3 3.4 3.6 3.7 3.8 3.9 4.1 4.2 4.5 4.7 4.9 5.3 5.6 6.1 Q1= first quartile = 2.2 (n+1)x25%th data M = median = 3.4 Q3= third quartile = 4.35 (n+1)x75%th data Maximum = 6.1 Five-number summary and boxplot 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 6.1 5.6 5.3 4.9 4.7 4.5 4.2 4.1 3.9 3.8 3.7 3.6 3.4 3.3 2.9 2.8 2.5 2.3 2.3 2.1 1.5 1.9 1.6 1.2 0.6 Largest = max = 6.1 BOXPLOT 7 Q3= third quartile = 4.35 M = median = 3.4 6 Years until death 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 5 4 3 2 1 Q1= first quartile = 2.2 Smallest = min = 0.6 0 Disease X Five-number summary: min Q1 M Q3 max Calculating quartiles 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 6.1 5.6 5.3 4.9 4.7 4.5 4.2 4.1 3.9 3.8 3.7 3.6 3.4 3.3 2.9 2.8 2.5 2.3 2.3 2.1 1.5 1.9 1.6 1.2 0.6 Here the sample size is n=25 Q3= third quartile = (25+1)x75%th data = 4.35 M = median = (25+1)x50%th data= 3.4 Q1= first quartile = (25+1)x25%th data=2.2 Boxplots for skewed data Years until death Comparing box plots for a normal and a right-skewed distribution 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 Boxplots remain true to the data and depict clearly symmetry or skew. Disease X Multiple Myeloma Side-by-side boxplots Side-by-side boxplots comparing the earnings of four groups of hourly workers at National Bank Suspected outliers Outliers are troublesome data points, and it is important to be able to identify them. One way to raise the flag for a suspected outlier is to compare the distance from the suspicious data point to the nearest quartile (Q1 or Q3). We then compare this distance to the interquartile range (distance between Q1 and Q3). We call an observation a suspected outlier if it falls more than 1.5 times the size of the interquartile range (IQR) above the first quartile or below the third quartile. This is called the “1.5 * IQR rule for outliers.” 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 6 5 4 3 2 1 7.9 6.1 5.3 4.9 4.7 4.5 4.2 4.1 3.9 3.8 3.7 3.6 3.4 3.3 2.9 2.8 2.5 2.3 2.3 2.1 1.5 1.9 1.6 1.2 0.6 8 7 Q3 = 4.35 Distance to Q3 7.9 − 4.35 = 3.55 6 Years until death 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 5 Interquartile range Q3 – Q 1 4.35 − 2.2 = 2.15 4 3 2 1 Q1 = 2.2 0 Disease X Individual #25 has a value of 7.9 years, which is 3.55 years above the third quartile. This is more than 3.225 years, 1.5 * IQR. Thus, individual #25 is a suspected outlier. Measure of spread: the standard deviation The standard deviation “s” is used to describe the variation around the mean. Like the mean, it is not resistant to skew or outliers. 1. First calculate the variance s2. n 1 2 s2 ( x x ) i n 1 1 2. Then take the square root to get the standard deviation s. x Mean ± 1 s.d. 1 n 2 s ( x x ) i n 1 1 Calculations … Women height (inches) i xi x (xi-x) (xi-x)2 1 59 63.4 -4.4 19.0 2 60 63.4 -3.4 11.3 3 61 63.4 -2.4 5.6 4 62 63.4 -1.4 1.8 5 62 63.4 -1.4 1.8 6 63 63.4 -0.4 0.1 7 63 63.4 -0.4 0.1 8 63 63.4 -0.4 0.1 9 64 63.4 0.6 0.4 10 64 63.4 0.6 0.4 11 65 63.4 1.6 2.7 Degrees freedom (df) = (n − 1) = 13 12 66 63.4 2.6 7.0 s2 = variance = 85.2/13 = 6.55 inches squared 13 67 63.4 3.6 13.3 14 68 63.4 4.6 21.6 Sum 0.0 Sum 85.2 1 s df n (x i x) 2 1 Mean = 63.4 Sum of squared deviations from mean = 85.2 s = standard deviation = √6.55 = 2.56 inches Mean 63.4 Properties of standard deviation s measures spread about the mean and should be used only when the mean is the measure of center. s = 0 only when all observations have the same value and there is no spread. Otherwise, s > 0. s is not resistant to outliers. s has the same units of measurement as the original observations. Software output for summary statistics: Excel - From Menu: Tools/Data Analysis/ Descriptive Statistics Give common statistics of your sample data. Minitab Choosing measures of center and spread Because the mean is not Height of 30 Women resistant to outliers or skew, use 69 it to describe distributions that are 68 fairly symmetrical and don’t have Plot the mean and use the standard deviation for error bars. Otherwise use the median in the five number summary which can be plotted as a boxplot. Height in Inches outliers. 67 66 65 64 63 62 61 60 59 58 Box Plot Boxplot Mean ± +/-SD SD Mean What should you use, when, and why? Arithmetic mean or median? Middletown is considering imposing an income tax on citizens. City Hall wants a numerical summary of its citizens’ income to estimate the total tax base. Mean: Although income is likely to be right-skewed, the city government wants to know about the total tax base. In a study of standard of living of typical families in Middletown, a sociologist makes a numerical summary of family income in that city. Median: The sociologist is interested in a “typical” family and wants to lessen the impact of extreme incomes. Obtaining data Observation versus Experiment and association versus causation Population versus sample Sampling methods Simple random samples Stratified samples Probelms in sampling Data Available data are data that were produced in the past for some other purpose but that may help answer a present question inexpensively. The library and the Internet are sources of available data. Government statistical offices are the primary source for demographic, economic, and social data (visit the Fed-Stats site at www.fedstats.gov). When data are not available conduct a statistical survey or experiment Observation and experiment Observational study: Record data on individuals without attempting to influence the responses. Example: Watch the behavior of consumers looking at store displays, or the interaction between managers and employees. Experimental study: Deliberately impose a treatment on individuals and record their responses. Influential factors can be controlled. Example: To answer the question “Which TV ad will sell more toothpaste?” show each ad to a separate group of consumers and note whether they buy toothpaste. Observational studies are essential sources of data on a variety of topics. However, when our goal is to understand cause and effect, experiments are the only source of fully convincing data. Association or cause–effect If ”early retirement” and ”early death” are found to be related (associative or causative or both)? That is, Does ”early retirement” causes ”early death”? Does ”personal health” or similar thing causes the both simultaneously? Are both happening? Collecting data: statistical surveys Sampling plans Population size=N; 1. 2. 3. 4. sample size=n; k=N/n Simple random sampling: subjects are selected with equal chances (n/N) Systematic random sampling: selected the first one from first k subjects in the population with equal chances and then select every kth subject until the sample is completed Stratified random sampling: divide population into several groups (strata) and then select a random sample from each of them (Can be either proportional or disproportional) Cluster random sampling: divide the population into a large numebr of clusters and then select a random sample of clusters. Use all the subjects in them Sampling variability Each time we take a random sample from a population, we are likely to get a different set of individuals and a calculate a different statistic. This is called sampling variability. The good news is that, if we take lots of random samples of the same size from a given population, the variation from sample to sample—the sampling distribution—will follow a predictable pattern. All of statistical inference is based on this knowledge. The variability of a statistic is described by the spread of its sampling distribution. This spread depends on the sampling design and the sample size n, with larger sample sizes leading to lower variability. Large random samples are almost always close estimates of the true population parameter. However, this only applies to random samples. Remember the “QuickVote” online surveys. They are worthless no matter how many people participate because they use a voluntary sampling design and not random sampling. Bias The study design is biased if it systematically favors certain outcomes There are many types of biases: sampling bias, response bias, etc. Managing bias and variability To reduce bias, use random sampling. The values of a statistic computed from an SRS neither consistently overestimate or underestimate the value of a population parameter. To reduce the variability of a statistic from an SRS, use a larger sample. You can make the variability as small as you want by taking a large enough sample. Population size doesn’t matter: The variability of a statistic from a random sample does not depend on the size of the population, as long as the population is at least 100 times larger than the sample. Practical note Large samples are not always attainable. Sometimes the cost, difficulty, or preciousness of what is studied limits drastically any possible sample size Blood samples/biopsies: No more than a handful of repetitions acceptable. We often even make do with just one. Opinion polls have a limited sample size due to time and cost of operation. During election times though, sample sizes are increased for better accuracy. Caution about sampling surveys Nonresponse: People who feel they have something to hide or who don’t like their privacy being invaded probably won’t answer, yet they are part of the population. Response bias: Fancy term for lying when you think you should not tell the truth, or forgetting. This is particularly important when the questions are very personal (e.g., “How much do you drink?”) or related to the past. Wording effects: Questions worded like “Do you agree that it is awful that…” are prompting you to give a particular response.