Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Probability and Statistical Inference (9th Edition) Chapter 5 (Part 2/2) Distributions of Functions of Random Variables November 25, 2015 1 Outline 5.5 Random Functions Associated with Normal Distributions 5.6 The Central Limit Theorem 5.7 Approximations for Discrete Distributions 5.8 Chebyshev’s Inequality and Convergence in Probability 2 Random Functions Associated with Normal Distributions Theorem: Assume that X1, X2,…, Xn are independent random variables with distributions N(μ1,σ12), N(μ2,σ22),…, N(μn,σn2), respectively. Then, n n 2 2 Y ci X i is N ci i , ci i i 1 i 1 i 1 n 3 Random Functions Associated with Normal Distributions Proof: Recall the mgf of N(,2) is 2 2 2 i ci t M Y t M xi ci t exp i ci t 2 i 1 i 1 n n n t2 exp t i ci 2 i 1 ci i i 1 n 2 2 n n i 1 i 1 Therefore, Y is N ( ci i , ci i ) 2 2 4 Random Functions Associated with Normal Distributions Example 1: If X1 and X2 are independent normal random variables N(µ1,σ12) and N(µ2,σ22), respectively, then X1 + X2 is N(µ1+µ2, σ12+σ22), and X1 - X2 is N(µ1-µ2, σ12+σ22) 5 Random Functions Associated with Normal Distributions Example 2: If X1,X2,…,Xn correspond to random samples from a normal distribution N(μ,σ2), then the sample mean 1 n is N(μ,σ2/n) X n i 1 X i Proof: t 2 t 2 t M t M Exp 2 n n 2 n t , since M t Exp t n n x 2 2 Therefore, X is N , n 2 2 6 Random Functions Associated with Normal Distributions One important implication of the distribution of X is that it has a greater probability of falling in an interval containing μ than does a single sample Xk The larger the sample size n, the smaller the variance of the sample mean “Mean” is a constant, but “sample mean” is a random variable 7 Random Functions Associated with Normal Distributions For example, assume that X1, X2,…, Xn are random samples from N(50,16) distribution. Then, X is N(50,16/n). The following figure shows the pdf of X with different values of n 8 Random Functions Associated with Normal Distributions 9 Random Functions Associated with Normal Distributions Recall: Let Z1, Z2, …, Zn be i.i.d. N(0,1). Then, w = Z12+ Z22+ …+ Zn2 is χ2(n) Let X1, X2,…, Xn be independent chisquare random variables with k1, k2,…, kn degrees of freedom, i.e., χ2(k1), χ2(k2),…, χ2(kn), respectively. Then, Y=X1+X2+…+Xn is χ2(k1+k2+…+kn) 10 Random Functions Associated with Normal Distributions Theorem: Let X1, X2,…, Xn be random samples from the N(μ,σ2) distribution. The sample mean and sample variance are given by Then, (a) (b) and are independent is χ2(n-1) 11 Random Functions Associated with Normal Distributions We will accept (a) without proving it Proof of (b): 12 Random Functions Associated with Normal Distributions 13 Random Functions Associated with Normal Distributions 14 Random Functions Associated with Normal Distributions It is interesting to observe that n U X i i 1 n W i 1 2 X 2 is 2 n and 2 i X 2 n 1 is 2 That is, when the actual mean is replaced by the sample mean, one degree of freedom is lost 15 Central Limit Theorem It is useful to first review some related theorems Theorem (Sample Mean): Let X1, X2, …, Xn be a sequence of i.i.d. random variables with mean and variance 2. Then, the sample mean is a random variable with mean and variance 2/n 16 Central Limit Theorem Theorem (Strong Law of Large Numbers): Let X1, X2, …, Xn be a sequence of i.i.d. random variables with mean . Then, with probability 1, That is, (The sample mean converges almost surely, or converges with probability 1, to the expected value) 17 Central Limit Theorem Theorem (Strong Law of Large Numbers)(Cont.): This theorem holds for any distribution of the Xi’s This is one of the most well-known results in probability theory 18 Central Limit Theorem Theorem (Central Limit Theorem): Let X1, X2, …, Xn be a sequence of i.i.d. random variables with mean and variance 2. Then the distribution of is N(0,1) as That is, (convergence in distribution) 19 Central Limit Theorem Theorem (Central Limit Theorem)(Cont.): While tends to “degenerate” to zero (Strong Law of Large Numbers), the factor in “spreads out” the probability enough to prevent this degeneration 20 Central Limit Theorem Theorem (Central Limit Theorem)(Cont.): One observation that helps make sense of this result is that, in the case of normal distribution (i.e., X1, X2, …, Xn are i.i.d. normal), is N(,2/n) Hence, value of n is (exactly) N(0,1) for each positive Thus, in the limit, the distribution must also be N(0,1) 21 Central Limit Theorem Theorem (Central Limit Theorem)(Cont.): The powerful fact is that this theorem holds for any distribution of the Xi’s It explains the remarkable fact that the empirical frequencies of so many natural “populations” exhibit a bell-shaped (i.e., normal) curve The term “central limit theorem” traces back to George Polya who first used the term in 1920 in the title of a paper. Polya referred to the theorem as “central” due to its importance in probability theory 22 Central Limit Theorem The Central Limit Theorem and the Strong Law of Large Numbers are the two fundamental theorems of probability 23 Central Limit Theorem Example 1 (Normal Approximation to the Uniform Sum Distribution (a.k.a. the Irwin-Hall Distribution)): Let Xi, i=1,2,… be i.i.d. U(0,1). Compare the graph of the pdf of Y=X1+X2+…+Xn, with the graph of the N(n(1/2), n(1/12)) pdf n=2 24 Central Limit Theorem n=4 25 Central Limit Theorem Example 2 (Normal Approximation to the Uniform Sum Distribution (a.k.a. the Irwin-Hall Distribution)): Let Xi, i=1,2,…,10 be i.i.d. U(0,1). Estimate P(X1+X2+…+X10 > 7) Solution: With and by the central limit theorem, 26 Central Limit Theorem Example 3 (Normal Approximation to the Chi-Square Distribution): Let X1,X2,…,Xn be i.i.d. N(0,1). Then, is chi-square with n degrees of freedom, with E(Y)=n and Var(Y)=2n Recall the pdf of Y is Let 27 Central Limit Theorem The pdf of W is given by Compare the pdf of W and the pdf of N(0,1): n=20 n=100 28 Approximations for Discrete Distributions The beauty of the central limit theorem is that it holds regardless of the underlying distribution (even discrete) 29 Approximations for Discrete Distributions Example 4 (Normal Approximation to the Binomial Distribution): X1,X2,… Xn are random samples from a Bernoulli distribution with μ=p and σ2 = p(1-p). Then, Y=X1+X2+…+Xn is binomial b(n,p). The central limit theorem states that is N(0,1) as n approaches infinity 30 Approximations for Discrete Distributions Thus, if n is sufficiently large, the distribution of Y is approximately N(np,np(1-p)), and the probabilities for the binomial distribution b(n,p) can be approximated with this normal distribution, i.e., for sufficiently large n 31 Approximations for Discrete Distributions Consider n=10, p=1/2, i.e., Y~b(10,1/2). Then, by CLT, Y can be approximated by the normal distribution with mean 10(1/2)=5 and variance 10(1/2)(1/2)=5/2. Compare the pmf of Y and the pdf of N(5,5/2): 32 Approximations for Discrete Distributions Example 5 (Normal Approximation to the Poisson Distribution): Recall the Poisson pmf where parameter the distribution is both the mean and variance of Poisson random variable counts the number of discrete occurrences (sometimes called “events” or “arrivals”) that take place during a time-interval of given length 33 Approximations for Discrete Distributions A random variable having a Poisson distribution with mean 20 can be thought of as the sum Y of the observations of a random sample of size 20 from a Poisson distribution with mean 1. Thus, has a distribution that is approximately N(0,1), and the distribution of Y is approximately N(20,20) 34 Approximations for Discrete Distributions Compare the pmf of Y and the pdf of N(20,20): 35 Markov’s Inequality Theorem (Markov’s Inequality): If X is a continuous random variable that takes only nonnegative values, then for any a>0, The inequality is valid for all distributions (discrete or continuous) 36 Markov’s Inequality Proof: 37 Markov’s Inequality Intuition behind Markov’s Inequality, using a fair dice (discrete) example: Then, … 38 Chebyshev’s Inequality Theorem (Chebyshev’s Inequality): If X is a continuous random variable with mean and variance 2, then for any k>0, The inequality is valid for all distributions (discrete or continuous) for which the standard deviation exists 39 Chebyshev’s Inequality Proof: Since (X-)2 is a nonnegative random variable, we can apply Markov’s inequality (with a=k2) to obtain Thus, 40 Chebyshev’s Inequality (Another Form) Chebyshev’s Inequality (another form): Chebyshev’s inequality states that the probability that X differs from its mean by at least k standard deviations is less than or equal to 1/k2 It follows that the probability that X differs from its mean by less than k standard deviations is at least 1-1/k2 41 Chebyshev’s Inequality The importance of Markov’s and Chebyshev’s inequalities is that they enable us to derive (sometimes loose but still useful) bounds on probabilities when only the mean, or both the mean and the variance, of the probability distribution are known 42 Chebyshev’s Inequality Example 1: If it is known that X has a mean of 25 and a variance of 16, then, a lower bound for P(17<X<33) is given by and an upper bound for P(|X-25|>=12) is The results hold for any distribution with mean 25 and variance 16 43