Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Lesson 2: Descriptive Statistics Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-1 2004 Regional Nominal GDP per Capita Summary Statistics (yuan per person) Mean Standard Error Median Mode Standard Deviation Sample Variance Kurtosis Skewness Range Minimum Maximum Sum Count Ka-fu Wong © 2007 14079.39 1912.88 9608.00 #N/A 10650.44 113431828.11 7.14 2.50 51092.00 4215.00 55307.00 436461.00 31 ECON1003: Analysis of Economic Data Guizhou Shanghai Lesson2-2 Outline Measures of Central Tendency Measures of Variability Measures for two variables Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-3 Population Parameters and Sample Statistics A population parameter is number calculated from all the population measurements that describes some aspect of the population. The population mean, denoted , is a population parameter and is the average of the population measurements. A point estimate is a one-number estimate of the value of a population parameter. A sample statistic is number calculated using sample measurements that describes some aspect of the sample. Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-4 Measures of Central Tendency Overview Central Tendency Mean Median Mode Midpoint of ranked values Most frequently observed value n x x i1 i n Arithmetic average Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-5 The Mean Imagine an economy with N individuals (that is, a population) with income X1, X2, …, XN Sample x1, x2, …, xn Income per capita () Population Mean N Ka-fu Wong © 2007 X x Sample Mean n i i =1 N ECON1003: Analysis of Economic Data x x i i =1 n Lesson2-6 Arithmetic Mean The arithmetic mean (mean) is the most common measure of central tendency For a population of N values: N x x1 x 2 ... x N μ N N i 1 i Population values Population size For a sample of size n: n x x i 1 n i x1 x 2 ... x n n Observed values Sample size Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-7 Effect of extreme value on Arithmetic Mean The most common measure of central tendency Mean = sum of values divided by the number of values Affected by extreme values (outliers) 0 1 2 3 4 5 6 7 8 9 10 Mean = 3 1 2 3 4 5 15 3 5 5 Ka-fu Wong © 2007 0 1 2 3 4 5 6 7 8 9 10 Mean = 4 1 2 3 4 10 20 4 5 5 ECON1003: Analysis of Economic Data Lesson2-8 Median In an ordered list, the median is the “middle” number (50% above, 50% below) 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Median = 3 Median = 3 Not affected by extreme values Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-9 Finding the Median The location of the median: Median position n 1 position in the ordered data 2 If the number of values is odd, the median is the middle number If the number of values is even, the median is the average of the two middle numbers n 1 Note that 2 is not the value of the median, only the position of the median in the ranked data Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-10 Mode A measure of central tendency Value that occurs most often Not affected by extreme values Used for either numerical or categorical data There may may be no mode There may be several modes 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mode = 9 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data 0 1 2 3 4 5 6 No Mode Lesson2-11 Review Example Five houses on a hill by the beach $2,000 K House Prices: $2,000,000 500,000 300,000 100,000 100,000 $500 K $300 K Mean: = ($3,000,000/5) = $600,000 Median: middle value of ranked data = $300,000 Mode: most frequent value = $100,000 $100 K $100 K Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-12 Which measure of location is the “best”? Mean is generally used, unless extreme values (outliers) exist Then median is often used, since the median is not sensitive to extreme values. Example: Median home prices may be reported for a region – less sensitive to outliers In census of income, it is often that several digits are reserved to record income, say, 6 digits. The income will be recorded as 000000 to 999998. 999999 will be reserved for “missing value”. That is, an individual with income 1234567 will be recorded as 999998. When we have substantial proportion of individual with income larger than 999998, median will be a better measure of central tendency. Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-13 Shape of a Distribution Describes how data are distributed Measures of shape Symmetric or skewed Left-Skewed Symmetric Right-Skewed Mean < Median Mean = Median Median < Mean Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-14 Relations between mean and median in skewed distributions -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 4 5 6 Symmetric: mean = median = 0 -6 -5 -4 -3 -2 -1 0 1 2 3 Symmetric: mean = median = 0 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-15 Relations between mean and median in skewed distributions -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 4 5 6 Symmetric distributions: mean = median = 0 -6 -5 -4 -3 -2 -1 0 1 2 3 Left skewed distributions: mean (= -1) < median (= 0) Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-16 Relations between mean and median in skewed distributions -6 -5 -4 -6 -5 -4 Ka-fu Wong © 2007 -3 -2 -1 0 1 4 5 6 -3 -2 -1 0 1 4 5 6 2 3 Symmetric distributions: mean = median = 0 2 3 Right skewed distributions: mean > median ECON1003: Analysis of Economic Data Lesson2-17 Measures of Variability Variation Range Interquartile Range Variance Standard Deviation Coefficient of Variation Measures of variation give information on the spread or variability of the data values. Same center, different variation Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-18 Range Simplest measure of variation Difference between the largest and the smallest observations: Range = Xlargest – Xsmallest Example: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Range = 14 - 1 = 13 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-19 Disadvantages of the Range Ignores the way in which data are distributed 7 8 9 10 11 Range = 12 - 7 = 5 12 7 8 9 10 11 12 Range = 12 - 7 = 5 Sensitive to outliers 1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,3,3,3,3,4,5 Range = 5 - 1 = 4 1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,3,3,3,3,4,120 Range = 120 - 1 = 119 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-20 Interquartile Range interquartile range eliminates some outlier problems Eliminate high- and low-valued observations and Calculate the range of the middle 50% of the data Interquartile range = 3rd quartile – 1st quartile IQR = Q3 – Q1 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-21 Interquartile Range Example: X minimum Q1 25% 12 Median (Q2) 45 X maximum 25% 25% 25% 30 Q3 57 70 Interquartile range = 57 – 30 = 27 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-22 Quartiles Quartiles split the ranked data into 4 segments with an equal number of values per segment 25% 25% Q1 25% Q2 25% Q3 The first quartile, Q1, is the value for which 25% of the observations are smaller and 75% are larger Q2 is the same as the median (50% are smaller, 50% are larger) Only 25% of the observations are greater than the third quartile Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-23 Quartile Formulas Find a quartile by determining the value in the appropriate position in the ranked data, where First quartile position: Q1 = 0.25(n+1) Second quartile position: (the median position) Q2 = 0.50(n+1) Third quartile position: Q3 = 0.75(n+1) where n is the number of observed values Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-24 Quartiles Example: Find the first quartile Sample Ranked Data: 11 12 13 16 16 17 18 21 22 (n = 9) Q1 = is in the 0.25(9+1) = 2.5 position of the ranked data so use the value half way between the 2nd and 3rd values, so Ka-fu Wong © 2007 Q1 = 12.5 ECON1003: Analysis of Economic Data Lesson2-25 Population Variance Average of squared deviations of values from the mean Population variance: N σ2 Where (x i 1 2 μ) i N μ = population mean N = population size xi = ith value of the variable x Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-26 Sample Variance Average (approximately) of squared deviations of values from the mean Sample variance: n s2 Where 2 (x x ) i i1 n -1 X = arithmetic mean n = sample size Xi = ith value of the variable X Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-27 Population Standard Deviation Most commonly used measure of variation Shows variation about the mean Has the same units as the original data Population standard deviation: N σ Ka-fu Wong © 2007 (x i 1 i μ)2 N ECON1003: Analysis of Economic Data Lesson2-28 Sample Standard Deviation Most commonly used measure of variation Shows variation about the mean Has the same units as the original data Sample standard deviation: n S Ka-fu Wong © 2007 2 (x x ) i i1 n -1 ECON1003: Analysis of Economic Data Lesson2-29 Calculation Example: Sample Standard Deviation Sample Data (xi) : 10 12 n=8 s 14 15 17 18 18 24 Mean = x = 16 (10 X ) 2 (12 x) 2 (14 x) 2 ... (24 x) 2 n 1 (10 16) 2 (12 16) 2 (14 16) 2 ... (24 16) 2 8 1 130 7 Ka-fu Wong © 2007 4.309 ECON1003: Analysis of Economic Data Lesson2-30 Measuring variation Small standard deviation Large standard deviation Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-31 Comparing Standard Deviations Data A 11 12 13 14 15 16 17 18 19 20 21 Mean = 15.5 s = 3.338 20 21 Mean = 15.5 s = 0.926 20 21 Mean = 15.5 s = 4.570 Data B 11 12 13 14 15 16 17 18 19 Data C 11 12 Ka-fu Wong © 2007 13 14 15 16 17 18 19 ECON1003: Analysis of Economic Data Lesson2-32 Advantages of Variance and Standard Deviation Each value in the data set is used in the calculation Values far from the mean are given extra weight (because deviations from the mean are squared) Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-33 Interpretation and Uses of the Standard Deviation Chebyshev’s theorem: For any set of observations, the minimum proportion of the values that lie within k standard deviations of the mean is at least: 1 1- 2 k where k is any constant greater than 1. Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Chebyshev’s theorem Chebyshev’s theorem: For any set of observations, the minimum proportion of the values that lie within k standard deviations of the mean is at least 1- 1/k2 Ka-fu Wong © 2007 K Coverage 1 0% 2 75.00% 3 88.89% 4 93.75% 5 96.00% 6 97.22% ECON1003: Analysis of Economic Data Lesson2-35 The Empirical Rule If the data distribution is bell-shaped, then the interval: μ 1σ contains about 68% of the values in the population or the sample 68% μ μ 1σ Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-36 The Empirical Rule μ 2σ contains about 95% of the values in the population or the sample μ 3σ contains about 99.7% of the values in the population or the sample Ka-fu Wong © 2007 95% 99.7% μ 2σ μ 3σ ECON1003: Analysis of Economic Data Lesson2-37 Why are we concern about dispersion? Dispersion is used as a measure of risk. Consider two assets of the same expected (mean) returns. -2%, 0%,+2% -4%, 0%,+4% The dispersion of returns of the second asset is larger then the first. Thus, the second asset is more risky. Thus, the knowledge of dispersion is essential for investment decision. And so is the knowledge of expected (mean) returns. Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-38 Coefficient of Variation Measures relative variation Always in percentage (%) Shows variation relative to mean Can be used to compare two or more sets of data measured in different units s CV x 100% Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-39 Comparing Coefficient of Variation Stock A: Average price last year = $50 Standard deviation = $5 Stock B: s CVA x $5 100% 100% 10% $50 Average price last year = $100 Standard deviation = $5 s $5 CVB 100% 100% 5% $100 x Both stocks have the same standard deviation, but stock B is less variable relative to its price. Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-40 Sharpe Ratio and Relative Dispersion Sharpe Ratio is often used to measure the performance of investment strategies, with an adjustment for risk. If X is the return of an investment strategy in excess of the market portfolio, the inverse of the CV is the Sharpe Ratio. An investment strategy of a higher Sharpe Ratio is preferred. http://www.stanford.edu/~wfsharpe/art/sr/sr.htm Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-41 Using Microsoft Excel Descriptive Statistics can be obtained from Microsoft® Excel Use menu choice: tools / data analysis / descriptive statistics Enter details in dialog box Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-42 Using Excel Use menu choice: tools / data analysis / descriptive statistics Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-43 Using Excel (continued) Enter dialog box details Check box for summary statistics Click OK Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-44 Excel output Microsoft Excel descriptive statistics output, using the house price data: House Prices: $2,000,000 500,000 300,000 100,000 100,000 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-45 2004 Regional Nominal GDP per Capita Summary Statistics (yuan per person) Mean Standard Error Median Mode Standard Deviation Sample Variance Kurtosis Skewness Range Minimum Maximum Sum Count Ka-fu Wong © 2007 14079.39 1912.88 9608.00 #N/A 10650.44 113431828.11 7.14 2.50 51092.00 4215.00 55307.00 436461.00 31 Why is it different from the national per capita GDP (10561 yuan)? ECON1003: Analysis of Economic Data Lesson2-46 Weighted Mean The weighted mean of a set of data is n w x x w i 1 i i w1x1 w 2 x 2 ... w n x n wi Where wi is the weight of the ith observation Use when data is already grouped into n classes, with wi values in the ith class Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-47 Approximations for Grouped Data Suppose a data set contains values m1, m2, . . ., mk, occurring with frequencies f1, f2, . . . fK For a population of N observations the mean is K μ fimi K where i1 N fi i1 N For a sample of n observations, the mean is K x Ka-fu Wong © 2007 fm i 1 i n i K where n fi i1 ECON1003: Analysis of Economic Data Lesson2-48 2004 Regional Nominal GDP per Capita Summary Statistics (yuan per person) Mean Standard Error Median Mode Standard Deviation Sample Variance Kurtosis Skewness Range Minimum Maximum Sum Count Ka-fu Wong © 2007 14079.39 1912.88 9608.00 #N/A 10650.44 113431828.11 7.14 2.50 51092.00 4215.00 55307.00 436461.00 31 Why is it different from the national per capita GDP (10561 yuan)? National per capita GDP based on weighted mean (with population size as weights): 13107.66 yuan ECON1003: Analysis of Economic Data Lesson2-49 Approximations for Grouped Data Suppose a data set contains values m1, m2, . . ., mk, occurring with frequencies f1, f2, . . . fK For a population of N observations the variance is K σ2 2 f (m μ) i i i 1 N For a sample of n observations, the variance is K s2 Ka-fu Wong © 2007 2 f (m x ) i i i1 n 1 ECON1003: Analysis of Economic Data Lesson2-50 The Sample Covariance The covariance measures the strength of the linear relationship between two variables The population covariance: N Cov (x , y) xy (x The sample covariance: i1 i x )(y i y ) N n Cov (x , y) s xy (x x)(y i1 i i y) n 1 Only concerned with the strength of the relationship No causal effect is implied Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-51 Interpreting Covariance Covariance between two variables: Cov(x,y) > 0 x and y tend to move in the same direction Cov(x,y) < 0 x and y tend to move in opposite directions Cov(x,y) = 0 x and y are independent Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-52 Coefficient of Correlation Measures the relative strength of the linear relationship between two variables Population correlation coefficient: Sample correlation coefficient: Ka-fu Wong © 2007 Cov (x , y) ρ σXσY r Cov (x , y) sX sY ECON1003: Analysis of Economic Data Lesson2-53 Features of Correlation Coefficient, r Unit free Ranges between –1 and 1 The closer to –1, the stronger the negative linear relationship The closer to 1, the stronger the positive linear relationship The closer to 0, the weaker any linear relationship Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-54 Scatter Plots of Data with Various Correlation Coefficients Y Y Y X X r = -1 r = -.6 Y r=0 Y Y r = +1 Ka-fu Wong © 2007 X X X r = +.3 ECON1003: Analysis of Economic Data X r=0 Lesson2-55 Using Excel to Find the Correlation Coefficient Select Tools/Data Analysis Choose Correlation from the selection menu Click OK . . . Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-56 Using Excel to Find the Correlation Coefficient (continued) Input data range and select appropriate options Click OK to get output Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-57 Interpreting the Result 100 y = 0.6666x + 30.241 R2 = 0.5376 95 Test #2 Score 90 85 80 75 70 70 75 80 85 90 95 100 Test #1 Score r = .733 There is a relatively strong positive linear relationship between test score #1 and test score #2. Students who scored high on the first test tended to score high on second test. Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-58 Obtaining Linear Relationships An equation can be fit to show the best linear relationship between two variables: Y = β0 + β1X where Y is the dependent variable and X is the independent variable Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-59 Least Squares Regression Estimates for coefficients β0 and β1 are found to minimize the sum of the squared residuals The least-squares regression line, based on sample data, is ˆ b0 b1 x y Where b1 is the slope of the line and b0 is the y-intercept: sy Cov(x, y) b1 r 2 sx sx Ka-fu Wong © 2007 b0 y b1x ECON1003: Analysis of Economic Data Lesson2-60 Lesson 2: Descriptive Statistics - END - Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson2-61