Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Degrees of freedom (statistics) wikipedia , lookup
Psychometrics wikipedia , lookup
Sufficient statistic wikipedia , lookup
Bootstrapping (statistics) wikipedia , lookup
History of statistics wikipedia , lookup
Confidence interval wikipedia , lookup
Taylor's law wikipedia , lookup
Statistical hypothesis testing wikipedia , lookup
Foundations of statistics wikipedia , lookup
Resampling (statistics) wikipedia , lookup
Mathematical Statistics Motoya Machida April 12, 2006 This material is designed to introduces basic concepts and fundamental theory of mathematical statistics. A review of basic concepts will include likelihood functions, sufficient statistics, and exponential family of distributions. Then point estimation will be discussed, including minimum variance unbiased estimates, Cramér-Rao inequality, maximum likelihood estimates and asymptotic theory. Topics in general theory of statistical tests include Neyman–Pearson theorem, uniformly most powerful tests, and likelihood ratio tests. 1 Point estimates A random sample X1 , . . . , X n is regarded as independent and identically distributed (iid) random variables governed by an underlying probability density function f (x; θ). A value θ represents the characteristics of this underlying distribution, and is called a parameter. Suppose, for example, that the underlying distribution is the normal distribution with (µ, σ 2 ). Then the values µ and σ 2 are the parameters. Since X1 , . . . , Xn are random variables, the sample mean X̄ also becomes a random variable. In general, a random variable u(X) constructed from the random vector X = (X1 , . . . , Xn ) is called a statistic. For example, the sample mean X̄ is a statistic. A point estimate is a statistic u(X) which is a “best guess” for the true value θ. Suppose that the underlying distribution is the normal distribution with (µ, σ 2 ). Then the sample mean X̄ is in some sense a best guess of the parameter µ. Mean square-error. Let u(X) be a point estimate for θ. Then the functional R(θ, u) = E[(u(X) − θ)2 ] of u is called the mean square-error risk function. We can immediately observe that R(θ, u) = Var(u(X)) + [E(u(X)) − θ]2 = Var(u(X)) + [b(θ, u)]2 , where b(θ, u) = E(u(X)) − θ is called the bias of u(X). 2 Maximum likelihood estimate Having observed a random sample (X1 , . . . , Xn ) = (x1 , . . . , xn ) from an underlying pdf f (x; θ), we can construct the likelihood function n Y L(θ, x) = f (xi ; θ), i=1 1 Mathematical Statistics and consider it as a function of θ. Then the maximum likelihood estimate (MLE) θ̂ is the value of θ which “maximizes” the likelihood function L(θ, x). It is usually easier to maximize the log likelihood ln L(θ, x) = n X ln f (xi ; θ). i=1 2.1 Bernoulli trials Let f (x; θ) = θx (1−θ)1−x be the Bernoulli frequency function with success probability θ. By solving the equation Pn Pn 1 1 ∂ ln L(θ, x) = ( i=1 xi ) − (n − i=1 xi ) = 0, ∂θ θ 1−θ we obtain θ∗ = 2.2 Pn i=1 xi n which minimizes ln L(θ, x). Therefore, θ̂ = X̄ is the MLE of θ. Normal distribution Let X1 , . . . , Xn be a random sample from a normal distribution with parameter (µ, σ 2 ). Then the log likelihood function of parameter (µ, σ 2 ) is given by ln L(µ, σ 2 ) = − n 1 X n n ln σ 2 − 2 (xi − µ)2 − ln 2π. 2 2σ i=1 2 By solving n 1 X ∂ ln L(µ, σ 2 ) (xi − µ) = 0; = 2 ∂µ σ i=1 n ∂ ln L(µ, σ 2 ) n 1 X = − + (xi − µ)2 = 0, ∂σ 2 2σ 2 2(σ 2 )2 i=1 we can obtain the MLE’s µ̂ and σ̂ 2 as follows. n µ̂ = 1X Xi = X̄; n i=1 n σ̂ 2 = 1X (Xi − µ̂)2 . n i=1 Although the MLE σ̂ 2 of σ 2 is consistent, it should be noted that this point estimate σ̂ 2 is not an unbiased one. 2.3 Poisson distribution Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. Then the log likelihood function of parameter λ is given by ln L(λ) = ln λ n X xi − nλ − i=1 n X ln xi !. i=1 By solving n d ln L(λ) 1X = xi − n = 0, dλ λ i=1 Lecture Note Page 2 Mathematical Statistics we can obtain the MLE n λ̂ = 2.4 1X Xi = X̄. n i=1 Exercise Let X1 , . . . , Xn be a random sample from each of the following density function f (x; θ). Find the MLE θ̂ of θ. 1. f (x; θ) = (1 + θ)xθ , 0 ≤ x ≤ 1, where θ > −1. 2. f (x; θ) = θe−θx , x ≥ 0, where θ > 0. 3. f (x; θ) = θLθ x−θ−1 , x ≥ L, where L > 0 is given and θ > 1 (Pareto distribution). 2.5 Solutions to exercise 1. ln L(θ) = − Pn ! ln xi θ + n ln(1 + θ). By solving ln xi d dθ ln L(θ) = n X i=1 i=1 n i=1 n X ! ln xi + n = 0, we obtain θ̂ = 1+θ − 1. ! n X n d n 2. ln L(θ) = − xi θ + n ln θ. By solving ln L(θ) = − xi + = 0, we obtain θ̂ = Pn . dθ θ i=1 xi i=1 i=1 ! ! n n X X d n ln xi (θ + 1) + n ln θ + nθ ln L. By solving 3. ln L(θ) = − ln xi + + n ln L = 0, ln L(θ) = − dθ θ i=1 i=1 n we obtain θ̂ = Pn . i=1 ln xi − n ln L n X 3 3.1 ! Properties of MLE Consistency One of the important attributes of point estimate is unbiasedness. Since a statistic θ̂ is a random variable, we can consider the expectation E(θ̂) of θ̂. Then the point estimate θ̂ of θ is unbiased if it satisfies E(θ̂) = θ. In the case of normal distribution, the MLE X̄ for µ is unbiased since E(X̄) = µ. However, the MLE σ̂ 2 for σ 2 is not unbiased, since # " n n−1 2 1X 2 (Xi − X̄) = σ E n i=1 n Note that the point estimate θ̂ of θ is also dependent on the sample size n. We say that θ̂ is consistent if θ̂ converges in probability to θ̂ as n → ∞. For example, the above MLE’s X̄ and σ̂ 2 are both consistent by the weak law of large number. In general, the MLE is consistent under appropriate conditions. 3.2 Invariance Suppose that h(θ) is a one-to-one function of θ. Then it is clearly seen that θ̂ is the MLE for θ if and only if h(θ̂) is the MLE for h(θ). Even if h(θ) is not one-to-one, h(θ̂) will be viewed as the MLE which corresponds to the maximum likelihood. Lecture Note Page 3 Mathematical Statistics 3.3 Asymptotic normality ∂ ln L(θ̂, X) Suppose that θ̂ is the MLE and consistent, and that it satisfies = 0. By the Taylor expansion we ∂θ have ∂ ln L(θ̂, X) ∂ ln L(θ, X) ∂ 2 ln L(θ, X) ≈ + (θ̂ − θ) = 0. ∂θ ∂θ2 ∂θ Since θ̂ is close to θ by consistency, the approximation is valid. Furthermore, we can make the following observations: (Xi ;θ) ’s are iid with mean 0 and variance I1 (θ) = Var 1. The random variables ∂ ln f∂θ the central limit theorem, n X ∂ ln f (Xi ; θ) ∂ ln L(θ, X) ∂θ = i=1 p Zn = p ∂θ nI1 (θ) nI1 (θ) ∂ ∂θ ln f (X1 ; θ) . By converges to N (0, 1) in distribution as n → ∞. 2 f (Xi ;θ) ’s are iid with mean (−I1 (θ)) and finite variance. By the weak law of 2. The random variables ∂ ln∂θ 2 large number, n 1 ∂ 2 ln L(θ, X) 1 X ∂ 2 ln f (Xi ; θ) Wn = = n ∂θ2 n i=1 ∂θ2 converges to −I1 (θ) in probability as n → ∞. Together with Slutsky’s theorem we can find that p nI1 (θ)(θ̂ − θ) ≈ Zn Wn /(−I1 (θ)) converges to N (0, 1) in distribution as n → ∞. Hence, the MLE θ̂ has “approximately” a normal distribution 1 with mean θ and variance nI11(θ) = I(θ) if n is large, where I(θ) is the Fisher information for the random sample X. This suggests that n × Var(θ̂) → 1/I1 (θ) as n → ∞. Then we call θ̂ asymptotically efficient (cf. Bickel and Doksum, “Mathematical Statistics,” Chapter 4). 4 Confidence interval Let X = (X1 , . . . , Xn ) be a random sample from f (x; θ). Let u1 (X) and u2 (X) be statistics satisfying u1 (X) ≤ u2 (X). If P (u1 (X) < θ < u2 (X)) = 1 − α for every θ, then the random interval (u1 (X), u2 (X)) is called a confidence interval of level (1 − α). 4.1 Population mean Let X1 , . . . , Xn be iid random variables from N (µ, σ). The sample mean X̄ is an unbiased estimate of the X̄ − µ √ has the t-distribution with (n − 1) degrees of freedom. Thus, by parameter µ. Then the random variable S/ n Lecture Note Page 4 Mathematical Statistics using the critical point tα/2,n−1 we obtain X̄ − µ tα/2,n−1 S tα/2,n−1 S √ √ P √ < tα/2,n−1 = P X̄ − < µ < X̄ + = 1 − α. S/ n n n This implies that the parameter µ is in the interval tα/2,n−1 S tα/2,n−1 S √ √ X̄ − , X̄ + n n with probability (1 − α). The interval is also known as the t-interval. Example. A random sample of n milk containers is selected, and their milk contents are weighed. The data X1 , . . . , X n (1) can be used to investigate the unknown population mean of the milk container weights. The random selection of sample should ensure that the above data can be assumed to be iid. Suppose that we have calculated X̄ = 2.073 and S = 0.071 from the actual data with n = 30. Then by choosing α = 0.05, we have the critical point t0.025,29 = 2.045, and therefore, obtain the confidence interval 2.045 × 0.071 2.045 × 0.071 √ √ 2.073 − , 2.073 + = (2.046, 2.100) 30 30 of level 0.95 (or, of level 95%). Even if the data (1) are not normally distributed, the central limit theorem says that the estimate X̄ is approximately distributed as N (µ, σ 2 /n). In either case it is sensible to use critical points from the t-distribution. 4.2 Population proportion Let X1 , . . . , Xn be iid Bernoulli random variables with success probability p. The sample mean X̄ is the MLE X̄ − p of the parameter p, and unbiased. By the central limit theorem, the random variable p has approxp(1 − p)/n imately N (0, 1) as n gets larger (at least np > 5 and n(1 − p) > 5 by rule of thumb). Here we define the critical point zα for standard normal distribution by P (X > zα ) = α with standard normal random variable X. Thus, we have ! ! r r X̄ − p p(1 − p) p(1 − p) P p < p < X̄ + zα/2 ≈ 1 − α. < zα/2 = P X̄ − zα/2 p(1 − p)/n n n q q X̄) p(1−p) Here we can use X̄(1− as an estimate for . (We will see later that it is the MLE via the invariance n n property since X̄ is the MLE of p.) Together we obtain the confidence interval ! r r X̄(1 − X̄) X̄(1 − X̄) X̄ − zα/2 , X̄ + zα/2 n n of level (1 − α). There is an alternative and possibly more accurate method to derive a confidence interval. Here we observe that ! X̄ − p 2 2 P p n + zα/2 p2 − 2 nX̄ + zα/2 /2 p + nX̄ 2 ≤ 0 ≈ 1 − α. < zα/2 = P p(1 − p)/n This implies that the parameter p is in the interval (p̂− , p̂+ ) with probability (1 − α), where q 2 2 /4 nX̄ + zα/2 /2 ± zα/2 nX̄(1 − X̄) + zα/2 p̂± = . 2 n + zα/2 Lecture Note Page 5 Mathematical Statistics 5 Exploratory Data Analysis The data values recorded x1 , . . . , xn are typically considered as the observed values of random variables X1 , . . . , Xn having a common probability distribution f (x). To judge the quality of data, it is useful to envisage a population from which the sample should be drawn. A random sample is chosen at random from the population to ensure that the sample is representative of the population. Once a data set has been collected, it is useful to find an informative way of presenting it. Graphical representations of data in various forms can be quite informative. 5.1 Relative frequency histogram Given the number of observations fi , called frequency, in the i-th interval, the height hi of the i-th rectangle above the i-th interval is represented by hi = fi . n × (width of the i-th interval) When the width of each interval is equally chosen, the width w is called bandwidth and the height hi becomes hi = Lecture Note fi . n×w Page 6 Mathematical Statistics 5.2 Stem and leaf plot This is much like a histogram except it portrays a data set itself. Data 20.5 21.5 22.7 23.4 24.1 24.9 25.8 set 20.7 22.0 22.7 23.5 24.3 24.9 25.9 20.8 22.1 22.9 23.6 24.5 25.1 26.1 21.0 22.5 22.9 23.6 24.5 25.1 26.7 21.0 22.6 23.1 23.6 24.8 25.2 21.4 22.6 23.3 23.9 24.8 25.6 Stem-and-leaf 20 | 578 21 | 0045 22 | 015667799 23 | 13456669 24 | 13558899 25 | 112689 26 | 17 5.3 Boxplot The sample median is the value of the “middle” data point. When there is an odd number of numbers, the median is simply the middle number. For example, the median of 2, 4, and 7 is 4. When there is an even number of numbers, the median is the mean of the two middle numbers. Thus, the median of the numbers 2, 4, 7, 12 is (4+7)/2 = 5.5. The 25-sample percentile is the value indicating that 25% of the observations takes values smaller than the value. Similarly, we can define 50-percentile, 75-percentile, and so on. Note that 50-percentile is the median. We call 25-percentile the lower sample quartile and 75-percentile the upper sample quartile. A box is drawn stretching from the lower sample quartile (the 25-percentile) to the upper quartile (the 75percentile). The median is shown as a line across the box. Therefore 1/4 of the distribution is between this line and the right of the box and 1/4 of the distribution is between this line and the left of the box. Vertical lines (dotted), called “whiskers,” stretch out from the ends of the box to the largest and smallest data. Lecture Note Page 7 Mathematical Statistics 5.4 Outliers Graphical presentations can be used to identify “odd-looking” value which does not fit in with the rest of the data. Such a value is called an outlier. In many cases an outlier is discovered to be a misrecorded data value, or represents some special condition that was not in effect when the data were collected. Lecture Note Page 8 Mathematical Statistics Lecture Note Page 9 Mathematical Statistics In the above histogram and boxplot, the value in far right appears to be quite separate from the rest of the data, and can be considered to be an outlier. 6 Tests of statistical hypotheses Suppose that a researcher is interested in whether the new drug works. The process of determining whether the outcome of the experiment points to “yes” or “no” is called hypothesis testing. A widely used formalization of this process is due to Neyman and Pearson. Our hypothesis is then the null hypothesis that the new drug has no effect —the null hypothesis is often the reverse of what we actually believe, why? Because the researcher hopes to reject the hypothesis and announce that the new drug leads to significant improvements. If the hypothesis is not rejected, the researcher announces nothing and goes on to a new experiment. 6.1 Hypothesis testing of population mean Hospital workers are subject to a radiation exposure emanating from the skin of the patient. A researcher is interested in the plausibility of the statement that the population mean µ of radiation level is µ0 —the researcher’s hypothesis. Then the null hypothesis is H0 : µ = µ0 . Lecture Note Page 10 Mathematical Statistics The “opposite” of the null hypothesis, called an alternative hypothesis, becomes HA : µ 6= µ0 . Thus, the hypothesis testing problem “H0 versus HA ” is formed. The problem here is to whether or not to reject “H0 in favor of HA .” To assess this hypothesis, the radiation levels X1 , . . . , Xn are measured from n patients who had been injected with a radioactive tracer, and assumed to be independent and normally distributed with the mean µ. Under the null hypothesis, the random variable X̄ − µ0 √ T = S/ n has the t-distribution with (n − 1) degrees of freedom. Thus, we obtain the exact probability P |T | ≥ tα/2,n−1 = α. When α is chosen to be a small value (0.05 or 0.01, for example), it is unlikely that the absolute value |T | is larger than the critical point tα/2,n−1 . Then we say that the null hypothesis H0 is rejected with significance level α (or, size α) when the observed value t of T satisfies |t| > tα/2,n−1 . Example. We have µ0 = 5.4 for the hypothesis, and decided to give a test with significance level α = 0.05. Suppose that we have obtained X̄ = 5.145 and S = 0.7524 from the actual data with n = 28. Then we can compute 5.145 − 5.4 √ ≈ −1.79. T = 0.7524/ 28 Since |T | = 1.79 ≤ t0.025,27 = 2.052, the null hypothesis cannot be rejected. Thus, the evidence against the null hypothesis is not persuasive. 6.2 p-value The above random variable T is called the t-statistic. Having observed “T = t,” we can calculate the p-value p∗ = P (|Y | ≥ |t|) = 2 × P (Y ≥ |t|), where the random variable Y has a t-distribution with (n − 1) degrees of freedom. Then we have the relation “p∗ < α ⇔ |t| > tα/2,n−1 .” Thus, we reject H0 with significance level α when p∗ < α. In the above example, we can compute the p-value p∗ = 2 × P (Y ≥ 1.79) ≈ 0.0847 ≥ 0.05; thus, we cannot reject H0 . 6.3 One-sided hypothesis testing In the same case of hospital workers subject to a radiation exposure, this time the researcher is interested in the plausibility of the statement that the population mean µ is greater than µ0 . Then the hypothesis testing problem is H0 : µ ≥ µ0 versus HA : µ < µ0 . X̄ − µ0 √ is used as a test statistic. And we reject H0 with significant level α when S/ n you find that t < −tα,n−1 for the observed value t of T . 1. The same t-statistic T = 2. Alternatively we can construct the p-value p∗ = P (Y ≤ t), where the random variable Y has a t-distribution with (n − 1) degrees of freedom. Because of the relation “p∗ < α ⇔ t < −tα,n−1 ,” we can reject H0 with significant level α when p∗ < α. Lecture Note Page 11 Mathematical Statistics Example. We use the same µ0 = 5.4 for the hypothesis and the same significance level α = 0.05, but use the one-sided test. Recall that X̄ = 5.145 and S = 0.7524 were obtained from the data with n = 28. 1. Then we compute T = 5.145 − 5.4 √ ≈ −1.79. 0.7524/ 28 Since T = −1.79 < −t0.05,27 = −1.703, the null hypothesis H0 is rejected. Thus, the outcome is statistically significant so that the population mean µ is smaller than 5.4. 2. Alternatively, we can find the p-value p∗ = P (Y ≤ −1.79) ≈ 0.0423 < 0.05; thus, the null hypothesis should be rejected. We can also consider the hypothesis testing problem H0 : µ ≤ µ0 1. Using the t-statistics T = versus HA : µ > µ0 . X̄ − µ0 √ , we can reject H0 with significant level α when the observed value t of S/ n T satisfies t > tα,n−1 . 2. Alternatively we can construct the p-value p∗ = P (Y ≥ t) with the random variable Y has a t-distribution with (n − 1) degrees of freedom. Because of the relation “p∗ < α ⇔ t > tα,n−1 ,” we can reject H0 when p∗ < α. 6.4 Summary When the null hypothesis H0 is rejected, it is reasonable to find out the confidence interval of the population mean µ. The following table shows the confidence interval we can construct when your null hypothesis is rejected. √ 0 is the test statistic, and α is the significance level of your choice. Here T = X̄−µ S/ n Hypothesis testing 6.5 When can we reject H0 ? H0 : µ = µ0 versus HA : µ 6= µ0 . |T | > tα/2,n−1 H0 : µ ≤ µ0 versus HA : µ > µ0 . T > tα,n−1 H0 : µ ≥ µ0 versus HA : µ < µ0 . T < −tα,n−1 (1 − α)-level confidence interval S S X̄ − tα/2,n−1 √ , X̄ + tα/2,n−1 √ n n S X̄ − tα,n−1 √ , ∞ n S −∞, X̄ + tα,n−1 √ n Exercises 1. An experimenter is interested in the hypothesis testing problem H0 : µ = 3.0mm versus HA : µ 6= 3.0mm, where µ is the population mean of thickness of glass sheets. Suppose that a sample of n = 21 glass sheets is obtained and their thicknesses are measured. (a) For what values of the t-statistic does the experimenter accept the null hypothesis with a size α = 0.10? (b) For what values of the t-statistic does the experimenter reject the null hypothesis with a size α = 0.01? Lecture Note Page 12 Mathematical Statistics Suppose that the sample mean X̄ = 3.04mm and the sample standard deviation is S = 0.124mm. Is the null hypothesis accepted or rejected with α = 0.10? With α = 0.01? 2. A machine is set to cut metal plates to a length of 44.350mm. The length of a random sample of 24 metal plates have a sample mean of X̄ = 44.364mm and a sample standard deviation of S = 0.019mm. Is there any evidence that the machine is miscalibrated? 3. An experimenter is interested in the hypothesis testing problem H0 : µ ≤ 0.065 versus HA : µ > 0.065 where µ is the population mean of the density of a chemical solution. Suppose that a sample of n = 31 bottles of the chemical solution is obtained and their densities are measured. (a) For what values of the t-statistics does the experimenter accept the null hypothesis with a size α = 0.10? (b) For what values of the t-statistics does the experimenter reject the null hypothesis with a size α = 0.01? Suppose that the sample mean X̄ = 0.0768 and the sample standard deviation is S = 0.0231. Is the null hypothesis accepted or rejected with α = 0.10? With α = 0.01? 4. A chocolate bar manufacturer claims that at the time of purchase by a consumer the average age of its product is no more than 120 days. In an experiment to test this claim a random sample of 26 chocolate bars are found to have ages at the time of purchase with a sample mean of X̄ = 122.5 days and a sample standard deviation of S = 13.4 days. With this information how do you feel about the manufacturer’s claim? 7 Power of test We define a function K(θ) of parameter θ by the probability that H0 is rejected given µ = θ. K(θ) = P (“Reject H0 ” | µ = θ) Then K(θ) is called the power function. 7.1 Type I error What is the probability that we incorrectly reject H0 when it is actually true? Such an error is called type I error, and the probability of type I error is exactly the significant level α, as explained in the following: 1. The probability of type I error for the two-sided hypothesis test is given by K(µ0 ). Then we have K(µ0 ) = P |T | ≥ tα/2,n−1 = α. 2. In one-sided hypothesis test, the probability of type I error is the worst (that is, largest possible) probability max K(θ) of type I error. Given µ = θ, the random variable θ≥µ0 X̄ − θ θ − µ0 √ =T − √ =T −δ S/ n S/ n has the t-distribution with (n − 1) degrees of freedom, where δ = θ − µ0 √ . By observing that δ ≥ 0 if S/ n θ ≥ µ0 , we obtain K(θ) = P (T ≤ −tα,n−1 ) = P (T − δ ≤ −tα,n−1 − δ) ≤ P (T − δ ≤ −tα,n−1 ) = α. Thus, we obtain max K(θ) = α. θ≥µ0 Lecture Note Page 13 Mathematical Statistics 7.2 Power of test What is the probability that we incorrectly accept H0 when it is actually false? Such probability β is called the probability of type II error. Then the value (1 − β) is known as the power of the test, indicating how correctly we can reject H0 when it is actually false. Again, consider the case of hospital workers subject to a radiation exposure. Given the current estimate S = s of standard deviation and the current sample size n = n1 , the X̄ − µ0 µ−µ √ can be approximated by N (δ, 1) with δ = √ 0 . t-statistic T = s/ n1 S/ n Example. Suppose that the true population mean is µ = 5.1 (versus the value µ0 = 5.4 in our hypotheses). Then we can calculate the power of the test with δ ≈ −2.11 as follows. 1. In the two-sided hypothesis testing, we reject H0 when |T | > t0.025,27 = 2.052. Therefore, the power of the test is K(5.1) = P (|T | > 2.052 | µ = 5.1) ≈ 0.523 2. In the one-sided hypothesis testing, we reject H0 when T < −t0.05,27 = −1.703. Therefore, the power of the test is K(5.1) = P (T < −1.703 | µ = 5.1) ≈ 0.658. This explains why we could not reject H0 in the two-sided hypothesis testing. Our chance to detect the falsehood of H0 is only 52%, while we have 66% of the chance in the one-sided hypothesis testing. 7.3 Effect of sample size For a fixed significance level α of your choice, the power of the test increases as the sample size n increases. In the two-sided hypothesis testing discussed above, we could recommend to collect additional data to increase the power of the test. But how many additional data do we need? Here is one possible way to calculate a desirable sample size n: In the two-sided hypothesis testing, the power (1 − β) of the test is approximated by P (|T | > tα/2,n−1 ) ≈ P (Y < −tα/2,n−1 − δ) + P (Y > tα/2,n−1 − δ) ≥ P (Y > tα/2,n−1 − |δ|) with a random variable Y having the t-distribution with (n − 1) degrees of freedom. Given the current estimate S = s of standard deviation and the current sample size n1 , we can achieve the power (1 − α/2) of the test by increasing a total sample size n and consequently satisfying |δ| ≥ 2tα/2,n1 −1 . In the above example of radiation exposure of hospital workers, such size n can be calculated as n≥ 8 2tα/2,n1 −1 s |µ − µ0 | 2 = 2t0.025,27 × 0.7524 |5.1 − 5.4| 2 = 105.9. Comparison of two populations We often want to compare two populations on the basis of experiment. For example, a researcher wants to test the effect of his drug on blood pressure. In any treatment, an improvement could have been due to the placebo effect when the subject believes that he or she has been given an effective treatment. To protect against such biases, the study should consider (i) the use of a control group in which the subjects are given a placebo, and an experimental group in which the subjects are treated with the new drug, (ii) the randomization by assigning the subjects between the control and the exprimental groups randomly, and (iii) a double-blind experiment by concealing the nature of treatment from the subjects and the person taking measurements. Then it becomes the hypothesis testing problem H0 : µ1 = µ2 versus HA : µ1 6= µ2 . where µ1 and µ2 are the respective population means of the control and the experimental groups Lecture Note Page 14 Mathematical Statistics As a result of experiment, we typically obtain the measurements X1 , . . . , X n of the subjects from the control group, and the measurements Y1 , . . . , Ym of the subjects from the experimental group. Then it is usually assumed that X1 , . . . , Xn and Y1 , . . . , Ym are independent and normally distributed with (µ1 , σ12 ) and (µ2 , σ22 ), respectively. Even when they are not normally distributed, large sample sizes (n, m ≥ 30) ensure that the tests are appropriate via the central limit theorem. 8.1 Pooled variance procedure Let Sx and Sy be the sample standard deviations constructed from X1 , . . . , Xn and Y1 , . . . , Ym , respectively. When it is reasonable to assume “σ12 = σ22 ,” we can construct the pooled sample variance Sp2 = (n − 1)Sx2 + (m − 1)Sy2 n+m−2 The test statistic T = X̄ − Ȳ q Sp n1 + 1 m has the t-distribution with (n + m − 2) degrees of freedom under the null hypothesis H0 . Thus, we reject the null hypothesis H0 with significant level α when the observed value t of T satisfies |t| > tα/2,n+m−2 . Or, equivalently we can compute the p-value p∗ = 2 × P (Y ≥ |t|) with Y having a t-distribution with (n + m − 2) degrees of freedom, and reject H0 when p∗ < α. Confidence interval. The following table shows the corresponding confidence interval of the population mean difference µ1 − µ2 , when your null hypothesis H0 is rejected. Hypothesis testing H0 : µ1 = µ2 vs. HA : µ1 6= µ2 . (1 − α)-level confidence interval q q 1 X̄ − Ȳ − tα/2,n+m−2 Sp n1 + m , X̄ − Ȳ + tα/2,n+m−2 Sp n1 + q H0 : µ1 ≤ µ2 vs. HA : µ1 > µ2 . X̄ − Ȳ − tα,n+m−2 Sp H0 : µ1 ≥ µ2 vs. HA : µ1 < µ2 . −∞, X̄ − Ȳ + tα,n+m−2 Sp 1 n + 1 m, q 1 n 1 m ∞ + 1 m Example. Suppose that we consider the significant level α = 0.01, and that we have obtained X̄ = 80.02 and Sx = 0.024 from the control group of size n = 13, and Ȳ = 79.98 and Sy = 0.031 from the experimental group of size m = 8. Here we have assumed that σ12 = σ22 . Then we can compute the square root Sp = 0.027 of the pooled sample variance Sp2 , and the test statistic T = 80.02 − 79.98 q ≈ 3.33. 1 0.027 13 + 18 Thus, we can obtain p∗ = 2 × P (Y ≥ 3.33) ≈ 0.0035 < 0.01, and reject H0 . We conclude that the two population means are significantly different. And the 99% confidence interval for the mean difference is (0.006, 0.074). Lecture Note Page 15 Mathematical Statistics 8.2 General procedure When “σ12 6= σ22 ,” under the null hypothesis H0 the test statistic X̄ − Ȳ T =q 2 Sy2 Sx n + m has approximately the t-distribution with ν degree of freedom, where ν is the nearest integer to 2 2 Sy2 Sx n + m . 4 Sy4 Sx n2 (n−1) + m2 (m−1) Thus, we reject the null hypothesis H0 with significant level α when the observed value t of T satisfies |t| > tα/2,ν . Or, equivalently we can compute the p-value p∗ = 2 × P (Y ≥ |t|) with Y having a t-distribution with ν degrees of freedom, and reject H0 when p∗ < α. Confidence interval. The following table shows the corresponding confidence interval of the population mean difference µ1 − µ2 , when your null hypothesis H0 is rejected. Hypothesis testing H0 : µ1 = µ2 versus HA : µ1 6= µ2 . H0 : µ1 ≤ µ2 versus HA : µ1 > µ2 . H0 : µ1 ≥ µ2 versus HA : µ1 < µ2 . (1 − α)-level confidence interval q q S2 S2 S2 X̄ − Ȳ − tα/2,ν nx + my , X̄ − Ȳ + tα/2,ν nx + q S2 S2 X̄ − Ȳ − tα/2,ν nx + my , ∞ q S2 S2 −∞, X̄ − Ȳ + tα/2,ν nx + my Sy2 m Example. Suppose that we consider the significant level α = 0.01, and that we have obtained X̄ = 80.02 and Sx = 0.024 from the control group of size n = 13, and Ȳ = 79.98 and Sy = 0.031 from the experimental group of size m = 8 as before. Then the test statistic T ≈ 3.12, and ν = 12. Thus, we can obtain p∗ = 2 × P (Y ≥ 3.12) ≈ 0.0089 < 0.01, and still reject H0 . 9 Inference on proportions In experiments on pea breeding, Mendel observed the different kinds of seeds obtained by crosses from plants with round yellow seeds and plants with wrinkled green seeds. Possible types of progeny were: “round yellow”, “wrinkled yellow”, “round green”, and “wrinkled green.” When the data values recorded x1 , . . . , xn takes several types, or categories, we call them the categorical data. 9.1 Point estimate Let X be the number of observations for a particular type in categorical data of size n, and let p be the population proportion of this type (that is, the probability of occurrence of this type). Then the random variable X has the binomial distribution with parameter (n, p). And the point estimate of the population proportion p is p̂ = Lecture Note X . n Page 16 Mathematical Statistics We can easily see that 1 X = E(X) = p E(p̂) = E n n Thus, p̂ is an unbiased estimate of p. Furthermore, recall by the central limit theorem that we have approximately X ∼ N (np, np(1 − p)) when n is large. Then the point estimate p̂ is approximately distributed as the normal distribution with parameter (p, p(1−p) ). n 9.2 Hypothesis test Suppose that the vaccine can be approved for widespread use if it can be established that the probability p of serious adverse reaction is less than p0 . Then the hypothesis testing problem becomes H0 : p ≥ p0 versus HA : p < p 0 . (2) Let X be the number of participants who suffer an adverse reaction among n participants. Then, the random variable X has the binomial distribution with parameter (n, p) and is approximated by the normal distribution with parameter (np, np(1 − p)) when n is large [that is, to satisfy np > 5 and n(1 − p) > 5]. Critical point. The critical point of the standard normal distribution, denoted by zα , is defined as the value satisfying P (Z > zα ) = α where Z is a standard normal random variable. Since the normal distribution is symmetric, it implies that P (Z < −zα ) = α. Testing procedure. When np0 > 5 and n(1 − p0 ) > 5, X − np0 Z=p (3) np0 (1 − p0 ) is used for the test statistic. Then we can reject H0 in (2) with significance level α if the value z of the test statistic Z satisfies z < −zα . Equivalently, we can proceed to construct the p-value p∗ = P (X < z) = Φ(z), and reject H0 when p∗ < α. Since the consideration of continuity correction improves the accuracy, the alternative test statistic X − np0 + 0.5 Z= p np0 (1 − p0 ) may be also used. Confidence interval. When H0 is rejected, we want to further investigate the confidence interval for the population proportion p which corresponds to the result of hypothesis test. We have the point estimate p̂ = X/n. Then the two different formulas ! p X + zα2 /2 + zα X(n − X)/n + zα2 /4 0, (4) n + zα2 ! r p̂(1 − p̂) (5) 0, p̂ + zα n can be used for the confidence interval of level α. Although Formula (4) is known to be more accurate, Formula (5) may be used in most of our problems since it is easier to calculate. Example. Suppose that p0 = 0.05 is required, and that the significance level α = 0.5 is chosen. And the study shows that X = 4 adverse reactions are found out of n = 155 participants. Note that (0.05)(155) = 7.75 > 5 and (0.95)(155) = 147.25 > 5. Thus, we have 4 − (155)(0.05) + 0.5 Z= p ≈ −1.20 (155)(0.05)(0.95) Lecture Note and p∗ = Φ(−1.20) ≈ 0.115 Page 17 Mathematical Statistics We can also obtain the point estimate p̂ ≈ 0.0258 and the 95% confidence interval (0, 0.0562) by using (4) [we get (0, 0.0467) if we use (5)]. Since p∗ ≥ 0.05, we cannot reject the null hypothesis. Thus, it is not advisable that the vaccine be approved as the result of this study. 9.3 Sample size calculations We always guarantee the possibility of incorrectly rejecting H0 when H0 is true—Type I error, say, to be less than 5% of the chance. But, at the same time we sacrifice the power of detecting the falsehood of H0 when H0 is false—power of the test. In order for the hypothesis testing problem H 0 : p ≥ p0 versus HA : p < p 0 , to achive the power (1 − β) of the test, we need a sample of size n≥ zα p p0 (1 − p0 ) + zβ p − p0 !2 p p(1 − p) . (6) In the above vaccine experiment, if the true population mean p is 0.025, then the power of the test is only 0.18. To increase the power of the test at least 0.8, we need the sample size at least n = 464. 9.4 Summary Possible null hypotheses for the inference on population proportion are “H0 : p = p0 ”, “H0 : p ≤ p0 ”, and “H0 : p ≥ p0 ”. In either case we can use the test statistic Z in (3) if we do not make a “continuity correction.” Then the corresponding testing procedures are summarized in the following table. Null hypothesis When we reject it H 0 : p = p0 |Z| > zα/2 H 0 : p ≤ p0 Z > zα (1 − α)-level confidence interval q q p̂(1−p̂) p̂(1−p̂) p̂ − zα/2 , p̂ + zα/2 n n p̂ − zα q H 0 : p ≥ p0 Z < −zα 0, p̂ + zα p̂(1−p̂) , n q 1 p̂(1−p̂) n For the sample size calculation, use Formula (6) if the null hypothesis is either “H0 : p ≤ p0 ” or “H0 : p ≥ p0 ”. When the null hypothesis is “H0 : p = p0 ”, the sample size n can be computed as !2 p p zα/2 p0 (1 − p0 ) + zβ p(1 − p) n≥ . p − p0 9.5 Comparison of two proportions A researcher is interested in whether there is discrimination against women in a university. In terms of statistics this is the hypothesis testing problem H 0 : pA ≤ p B versus HA : pA > pB where pA and pB are the respective population proportions of men and women who are admitted to the university. The researcher decided to collect the data for graduate program in the university. Let X and Y be the respective Lecture Note Page 18 Mathematical Statistics numbers of men and women who are admitted to the graduate school, which are summarized in the following table: Men X n−X n Admit Deny Total Women Y m−Y m The test statistic is given by p̂A − p̂B Z=q p̂(1 − p̂) 1 n + 1 m where p̂A = X/n and p̂B = Y /m are the point estimates of pA and pB , and p̂ = X +Y n+m is called a pooled estimate of the common population proportion. Under the null hypothesis, the probability that Z > zα becomes approximately less than α. Thus, we reject H0 when the observed value z of Z satisfies z > zα . Or, equivalently we can reject p∗ = 1 − Φ(z) < α. Confidence interval. We may want to further investigate the confidence interval for the difference pA − pB . Having constructed the hypothesis testing problems “H0 : pA = pB ”, “H0 : pA ≤ pB ”, or “H0 : pA ≥ pB ”, the following table shows the corresponding testing procedure and the confidence interval. Null hypothesis When we reject it H 0 : p A = pB |z| > zα/2 [That is, p∗ = 2 × (1 − Φ(|z|)) < α] H 0 : p A ≤ pB z > zα [That is, p∗ = 1 − Φ(z) < α] H 0 : p A ≥ pB z < −zα [That is, p∗ = Φ(z) < α] (1 − α)-level confidence interval for pA − pB q p̂A ) p̂B ) + p̂B (1− , p̂A − p̂B − zα/2 p̂A (1− n m q p̂A (1−p̂A ) p̂B (1−p̂B ) p̂A − p̂B + zα/2 + , n m q p̂A ) p̂B ) + p̂B (1− ,1 p̂A − p̂B − zα p̂A (1− n m −1, p̂A − p̂B + zα q p̂A (1−p̂A ) n + p̂B (1−p̂B ) m Example. The following table classifies the applications for the graduate school according to admission status and sex. Admit Deny Total Men 97 263 360 Women 40 42 82 Total 137 305 442 Then we have p̂A = 97/360 ≈ 0.269, p̂B = 40/82 ≈ 0.488, and p̂ = 137/442 ≈ 0.310. And we can obtain 0.269 − 0.488 Z=p (0.31)(0.69)(1/360 + 1/82) ≈ −3.87 and p∗ = 1 − Φ(−3.87) ≈ 0.9999 Thus, we cannot reject H0 , indicating that there is no discrimination against women in this particular graduate program. In fact, the null hypothesis “H0 : pA ≥ pB ” will be rejected in this example. Lecture Note Page 19 Mathematical Statistics 10 Chi-square test In the experiment on pea breeding, Mendel observed the different kinds of seeds obtained by crosses from plants with round yellow seeds and plants with wrinkled green seeds. Possible types of progeny were: “round yellow”, “wrinkled yellow”, “round green”, and “wrinkled green.” And Mendel’s theory predicted the associated probabilities of occurrence as follows. Probabilities Round yellow 9/16 Wrinkled yellow 3/16 Round green 3/16 Wrinkled green 1/16 We want to test whether the data from n observation is consistent with his theory—goodness of fit test, in which the statement of null hypothesis becomes “the model is valid.” 10.1 Chi-square test In general, each observation is classified into one of k categories or “cells,” which results in the cell frequencies X1 , . . . , Xk . The goodness of fit to a particular model can be assessed by comparing the observed cell frequencies X1 , . . . , Xk with the expected cell frequencies E1 , . . . , E k , which are predicted from the model. The discrepancy between the data and the model can be measured by the Pearson’s chi-square statistic k X (Xi − Ei )2 . χ2 = Ei i=1 Under the null hypothesis (that is, assuming that the model is correct), the distribution of Pearson’s chi-square χ2 is approximated by the chi-square distribution with df = (number of cells) − 1 − (number of parameters in the model) degrees of freedom. Therefore, if you observe that χ2 = x and x > χ2α,df , then we can reject the null hypothesis, casting doubt on the validity of the model. Or, by computing the p-value p∗ = P (X > x) with a random variable X having the chi-square distribution with df degrees of freedom, equivalently we can reject the null hypothesis when p∗ < α. Example. In the experiment of pea breeding, we have obtained the data as in the following table. Frequencies Round yellow 315 Wrinkled yellow 101 Round green 108 Wrinkled green 32 With the total number of observations n = 556, the expected cell frequencies from the Mendel’s theory can be calculated as Expected frequencies Round yellow 312.75 Wrinkled yellow 104.25 Round green 104.25 Wrinkled green 34.75 We can compute the Pearson’s chi-square χ2 = 0.47. Since the Mendel’s model has no parameter, the chi-square distribution has 3 = (4 − 1) degrees of freedom and we get the p-value p∗ = 0.925. Thus, there is little reason to doubt the Mendel’s theory on the basis of Pearson’s chi-square test. Lecture Note Page 20 Mathematical Statistics 10.2 Test of independence Consider again the study of discrimination against women in university admission. In the study, there are two characteristics: men or women; admitted or denied. The researcher wanted to know whether such characteristics are linked or independent. For such a study, we take a random sample of size n from the population, which is summarized in the contingency table Admit Deny Total Men X11 X21 X·1 Women X12 X22 X·2 Total X1· X2· n = X·· The statement of null hypothesis becomes “the two characteristics are independent.” Under the null hypothesis, the expected frequencies for the contingency table can be given by Admit Deny Total Men np1 q1 np2 q1 nq1 Women np1 q2 np2 q2 nq2 Total np1 np2 n The point estimates of p1 , p2 , q1 , and q2 are p̂1 = X1· /n, p̂2 = X2· /n, q̂1 = X·1 /n, and q̂2 = X·2 /n. With these point estimates, the chi-square statistic is χ2 = 2 2 X X (Xij − Xi· X·j /n)2 , Xi· X·j /n i=1 j=1 and the degree of freedom is (4 − 1 − 2) = 1. Example. By using the same data as before, we can obtain the chi-square statistic [40 − (137)(82)/(442)]2 [97 − (137)(360)/(442)]2 + (137)(360)/(442) (137)(82)/(442) 2 [263 − (305)(360)/(442)] [42 − (305)(82)/(442)]2 + + ≈ 14.89, (305)(360)/(442) (305)(82)/(442) χ2 = and the p-value p∗ = 0.0001. Thus, the null hypothesis is rejected at any reasonable level, indicating that the two characteristics are somewhat dependent. 11 Minimum variance unbiased estimator. One of the important attributes of point estimate is unbiasedness. Since a statistic u(X) is a random variable, we can consider the expectation E[u(X)]. Then the point estimate u(X) of θ is called an unbiased estimator if it satisfies E[u(X)] = θ. For example, the sample mean X̄ is an unbiased estimate of the mean µ, since E(X̄) = µ. Furthermore, u(X) is called an minimum variance unbiased estimator if (i) it is unbiased, and (ii) for every other unbiased estimator s(X) for θ we have R(θ, u) = Var(u(X)) ≤ Var(s(X)) = R(θ, s) 11.1 for all θ. Sufficient statistics Let f (x; θ) be a joint density function of x = (x1 , . . . , xn ) for a random vector X = (X1 , . . . , Xn ). Note that f (x1 , . . . , xn ; θ) is of the form f (x1 ; θ) · · · f (xn ; θ) if (X1 , . . . , Xn ) is a random sample. Suppose that s(X) is a Lecture Note Page 21 Mathematical Statistics statistic having the pdf g(s; θ). Then s(X) is called a sufficient statistic if f (x; θ) g(s(x); θ) is a function of x and does not depend on θ. Factorization theorem. s(X) is a sufficient statistic if and only if the joint density f (x; θ) can be expressed in a form of f (x; θ) = k(s(x); θ) · h(x). Furthermore, the pdf g(s; θ) for the statistic s(X) is proportional to k(s; θ), or possibly a multiple c(s) × k(s; θ) with a function c(s) of s. 11.2 Rao-Blackwell’s theorem Let X = (X1 , . . . , Xn ) be a random sample from f (x; θ). Suppose that s(X) is a sufficient statistic, and that u(X) is an unbiased estimator for θ. Then we can construct a new statistic ϕ(s(X)) = E(u(X) | s(X)), which is a function of the sufficient statistic s(X). Furthermore, we obtain E[ϕ(s(X))] = θ by the law of total probability, and Var(ϕ(s(X))) ≤ Var(u(X)) via the conditional variance formula. 11.3 Lehmann-Scheffé’s theorem Let s(X) be a statistic having the pdf g(s; θ). Then s(X) is called a complete statistic if for any function φ, E[φ(s(X))] = 0 for all θ suffices to imply that φ(s(X)) ≡ 0. Lehmann-Scheffé’s theorem. Suppose that s(X) is a complete and sufficient statistic. Then the statistic ϕ∗ (s(X)) = E(u(X) | s(X)) is well-defined with any choice of unbiased statistic u(X). Moreover, ϕ∗ (s(X)) is the minimum variance unbiased estimator for θ, which is unique among functions of s(X). 11.4 Exponential families Let A be an interval (or a subset) on R, and let f (x; θ) = exp [c(θ)k(x) + h(x) + d(θ)] , x ∈ A; otherwise, f (x; θ) = 0 for all x 6∈ A. (7) be a probability density function with parameter θ. Here an interval A can be (−∞, ∞), [0, ∞), or [0, 1], or a subset A can be {0, 1, . . .} or {0, . . . , n}, for example; but A should not depend on the parameter θ. Then we say that the pdf f (x; θ) is of one-parameter exponential family. For example, 1. an exponential density, Lecture Note Page 22 Mathematical Statistics 2. a normal density with known σ 2 , 3. a Poisson frequency function are of the one-parameter exponential family. Natural sufficient statistics and completeness. Let X1 , . . . , Xn be a random sample from (??). Then the joint density " # n n X X f (x; θ) = exp c(θ) k(xi ) + h(xi ) + nd(θ) , x ∈ An i=1 i=1 Pn is of the exponential family. By the factorization theorem, the statistic s(X) = i=1 k(Xi ) is sufficient, which we call a natural sufficient statistic. Furthermore, the statistic s(X) has the pdf of the exponential family f (s; θ) = exp [c(θ)s + h∗ (s) + nd(θ)] , s ∈ A∗ , and is complete. 12 Efficient estimator We call a statistic u(X) efficient, if u(X) achieves the Cramér-Rao lower bound. 12.1 Fisher information Let f (x; θ) be a joint density function for a random sample X = (X1 , . . . , Xn ). Furthermore, we assume that ∂ ln f (x; θ) exists; we will call the conditions in (i)–(ii) (i) A = {x : f (x; θ) > 0} does not depend on θ, and (ii) ∂θ the regularity assumptions. For example, a pdf of exponential family satisfies the regularity assumptions. By ∂ observing that E ∂θ ln f (X; θ) = 0, we can define the Fisher information I(θ) by " 2 # ∂ ∂ I(θ) = E ln f (X; θ) = Var ln f (X; θ) . ∂θ ∂θ Now suppose that f (x; θ) is of the form f (x1 ; θ) · · · f (xn ; θ). By setting I1 (θ) = Var ∂ ln f (X1 ; θ) = nI1 (θ). I(θ) = n × Var ∂θ Exercise. Show that I(θ) = −E 12.2 h ∂2 ∂θ 2 ∂ ∂θ ln f (X1 ; θ) , we obtain i ln f (X; θ) . Cramér-Rao lower bound. Let X and Y be random variables, and let a be a real number. Then we have Var(aX − Y ) = a2 Var(X) − 2aCov(X, Y )+Var(Y ) ≥ 0. By substituting a = Cov(X, Y ) /Var(X) , we can find the Cauchy-Schwarz inequality [Cov(X, Y )]2 ≤ Var(X) · Var(Y ). Let u(X) be a statistic. Then E[u(X)] is a function of θ, say ψ(θ) = E[u(X)]. Here we can show that ∂ ψ 0 (θ) = Cov u(X), ln f (X; θ) . ∂θ Lecture Note Page 23 Mathematical Statistics By applying the Cauchy-Schwarz inequality we obtain the Cramér-Rao inequality Var(u(X)) ≥ [ψ 0 (θ)]2 . I(θ) (8) When u(X) is an unbiased statistic, the right-hand side of (8) becomes 1 /I(θ) , and is called the Cramér-Rao lower bound. If a statistic u(X) achieves the Cramér-Rao lower bound, we call u(X) an efficient estimator. Clearly an efficient and unbiased statistic is a minimum variance unbiased estimator. Now let f (x; θ) be a joint density which satisfies the regularity assumptions, and let u(X) be an unbiased statistic for θ. If the joint density is of the exponential family f (x; θ) = exp [c(θ)u(x) + h(x) + d(θ)] , x ∈ A∗ ; otherwise, f (x; θ) = 0 for all x 6∈ A∗ , (9) then the natural sufficient statistic u(X) is an efficient estimator. Conversely, if u(X) is an efficient estimator, then the joint density f (x; θ) is given in the form (9). Remark. The notion of efficient statistic is somewhat stronger than that of minimum variance. Even worse, there exist minimum variance unbiased estimators which do not achieve their respective Cramér-Rao lower bound. 13 Hypothesis testing Let θ be a parameter of an underlying probability density function f (x; θ) for a certain population. The hypothesis “H0 : θ = θ0 ” is called a simple hypothesis, since it completely specifies the underlying distribution. Whereas, the hypothesis “H0 : θ ∈ Θ0 ” with a set Θ0 of parameters is called a composite hypothesis if the set Θ0 contains more than one element. The “opposite” of the null hypothesis is called an alternative hypothesis, and is similarly expressed as “HA : θ ∈ Θ1 ” where Θ1 is another set of parameters such that Θ0 ∩ Θ1 = ∅. The set Θ1 is typically (but not necessarily) chosen to be the complement of Θ0 . Thus, the hypothesis testing problem “H0 versus HA ” can be formed as H0 : θ ∈ Θ0 versus HA : θ ∈ Θ1 . (10) The problem stated above is to whether or not to reject “H0 in favor of HA .” 13.1 Test statistic Given a random sample X = (X1 , . . . , Xn ), a function ( 1 if H0 is rejected; δ(X) = 0 otherwise. (11) is called a test function. Given the test (11), we can define the power function by K(θ0 ) = P (“Reject H0 ” | θ = θ0 ) = E (δ(X) | θ = θ0 ) . A typical test, however, is presented in the form “H0 is rejected if T (X) ≥ c.” Here T (X) is called a test statistic, and c is called a critical value. Then the test function can be expressed as ( 1 T (X) ≥ c; δ(X) = (12) 0 otherwise. Thus, we obtain K(θ0 ) = P (T (X) ≥ c | θ = θ0 ). The probability of type I error (i.e., “H0 is incorrectly rejected when H0 is true”) is defined by α = sup K(θ0 ), θ0 ∈Θ0 which is also known as the size of the test. Having calculated the size α of the test, (11) or (12) is said to be a level α test, or a test with significant level α. Lecture Note Page 24 Mathematical Statistics 13.2 Uniformly most powerful test What is the probability that we incorrectly accept H0 when it is actually false? Such probability β is called the probability of type II error. Then the value (1 − β) is known as the power of the test, indicating how correctly we can reject H0 when it is actually false. Suppose that H0 is in fact false, say θ = θ1 for some θ1 ∈ Θ1 . Then the power of the test is calculated by K(θ1 ). Suppose that the test (11) has the size α. This test is said to be uniformly most powerful if it satisfies K(θ1 ) ≥ K 0 (θ1 ) for all θ1 ∈ Θ1 0 for the power function K of every other level α test. Furthermore, if this is given in the form (12) with test statistic T (X), then the test statistic T (X) is said to be optimal. 14 Likelihood ratio test Consider the testing problem with simple (null and alternative) hypotheses: H0 : θ = θ 0 Let L(θ; x) = Qn i=1 versus HA : θ = θ 1 . f (xi ; θ) be the likelihood function, and let L(θ0 , θ1 ; x) = be the likelihood ratio. Then ( 1 δ(X) = 0 L(θ1 , x) L(θ0 , x) L(θ0 , θ1 ; X) ≥ c; otherwise. (13) becomes a uniformly most powerful test, and is called the Neyman–Pearson test. The test function (13) has the following property: For any function ψ(x) satisfying 0 ≤ ψ(x) ≤ 1, E(ψ(X) | θ = θ1 ) − E(δ(X) | θ = θ1 ) ≤ c[E(ψ(X) | θ = θ0 ) − E(δ(X) | θ = θ0 )]. 14.1 Monotone likelihood ratio family Let f (x; θ) be a joint density function with parameter θ, and let L(θ0 , θ1 ; x) be the likelihood ratio. Suppose that T (X) is a statistic and does not depend on the parameter θ. Then f (x; θ) is called a monotone likelihood ratio family in T (X) if 1. f (x; θ0 ) and f (x; θ1 ) are distinct for θ0 6= θ1 ; 2. L(θ0 , θ1 ; x) is a non-decreasing function of T (x) whenever θ0 < θ1 . Now consider the following test problem: H0 : θ ≤ θ0 (or H0 : θ = θ0 ) versus HA : θ > θ0 . (14) If f (x; θ) is a monotone likelihood ratio family in T (X), then the test functions (12) and (13) is equivalent whenever θ0 < θ1 , and the power function K(θ) for these tests becomes an increasing function. Furthermore, T (X) is an optimal test statistic, and the size of the test is simply given by α = K(θ0 ). Suppose that f (x; θ) is of the exponential family f (x; θ) = exp [c(θ)u(x) + h(x) + d(θ)], x ∈ A∗ , and that c(θ) is a strictly increasing function. Then f (x; θ) is a monotone likelihood ratio family in u(X). And the natural sufficient statistic u(X) becomes an optimal test statistic. Remark. Essentially, uniformly most powerful tests exist only for the test problem (14). Lecture Note Page 25 Mathematical Statistics 14.2 Test procedure The Neyman–Pearson test (13) can be generalized for the composite hypotheses in (10): (i) obtain the maximum likelihood estimate (MLE) θ̂ of θ, (ii) calculate also the MLE θ̂0 restricted for θ ∈ Θ0 , and (iii) construct the likelihood ratio sup L(θ; X) sup L(θ; X) L(θ̂; X) θ∈Θ1 θ λ(X) = = = max ,1 . sup L(θ; X) sup L(θ; X) L(θ̂0 ; X) θ∈Θ0 θ∈Θ0 The test statistic λ(X) yields an excellent test procedure in many practical applications, though it is not an optimal test in general. 15 Bayesian theory Let f (x; θ) be a density function with parameter θ ∈ Ω. In a Bayesian model the paramter space Ω has a distribution π(θ), called a prior distribution. Furthermore, f (x; θ) is viewed as the conditional distribution of X given θ. By the Bayes’ rule the conditional density π(θ | x) can be derived from , X π(θ)f (x; θ) π(θ)f (x; θ) if Ω is discrete; θ∈Ω π(θ | x) = Z π(θ)f (x; θ) π(θ)f (x; θ) dθ if Ω is continuous. Ω The distribution π(θ | x) is called the posterior distribution. Whether Ω is discrete or continuous, the posterior distribution π(θ | x) is “proportional” to π(θ)f (x; θ) up to the constant. Thus, we write π(θ | x) ∝ π(θ)f (x; θ). 15.1 Conjugate family It is often the case that both the prior density function π(θ) and the posterior density function π(θ | x) belong to the same family of density function π(θ; η) with parameter η. Then π(θ; η) is called conjugate to f (x; θ). Let f (x; θ) = exp nc0 (θ) + m X cj (θ)kj (x) + h(x) ; j=1 π(θ; η0 , η1 , . . . , ηm ) = exp c0 (θ)η0 + m X cj (θ)ηj + w(η0 , η1 , . . . , ηm ) . j=1 Suppose that a prior distribution is given by π(θ) = π(θ; η0 , η1 , . . . , ηm ). Then we obtain the posterior density π(θ | x) = π(θ; η0 + n, η1 + k1 (x), . . . , ηm + km (x)). Thus, the family of π(θ; η0 , η1 , . . . , ηm ) is conjugate to f (x; θ). Lecture Note Page 26 Mathematical Statistics 15.2 Decision model Given a random sample X from f (x; θ), we can introduce a decision function δ(X), and incur a loss l(θ, δ(X)) associated with the state of θ. Together we calculate the risk by Z R(θ, δ) = l(θ, δ(x))f (x; θ) dx. If the decision function δ is strictly dominated by no other decision function δ 0 , that is, if no decision function δ 0 satisfies R(θ, δ 0 ) ≤ R(θ, δ) for all θ ∈ Ω with strict inequality for some θ ∈ Ω, then δ is called admissible. In the Bayesian model where the parameter θ has the prior distribution π(θ), we can define the Bayes risk by Z r(δ) = R(θ, δ)π(θ) dθ. Ω ∗ Then the decision function δ is called a Bayes solution if δ ∗ minimizes the Bayes risk r(δ). When the parameter space Ω is an interval on R and π(θ) > 0 and R(θ, δ) is continuous at every point θ and every decision function δ, the Bayes solution δ ∗ is admissible. Having observed X = x, we can compute the posterior density π(θ | x), and construct the posterior risk by Z r(δ(x) | x) = l(θ, δ(x))π(θ | x) dθ. Ω If there is a decision function δ0 such that δ0 (x) minimizes the posterior risk r(δ(x) | x) for every x, then δ0 is a Bayes solution. Lecture Note Page 27