Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
9 Expectation and Variance Two numbers are often used to summarize a probability distribution for a random variable X. The mean is a measure of the center or middle of the probability distribution, and the variance is a measure of the dispersion, or variability in the distribution. These two measures do not uniquely identify a probability distribution. That is, two different distributions can have the same mean and variance. Still, these measures are simple, useful summaries of the probability distribution of X. 9.1 Expectation of Discrete Random Variable The most important characteristic of a random variable is its expectation. Synonyms for expectation are expected value, mean, and first moment. The definition of expectation is motivated by the conventional idea of numerical average. Recall that the numerical average of n numbers, say a1 , a2 , . . . , an is n 1X ak . n k=1 We use the average to summarize or characterize the entire collection of numbers a1 , . . . , an with a single value. Example 9.1. Consider 10 numbers: 5, 2, 3, 2, 5, -2, 3, 2, 5, 2. The average is 5 + 2 + 3 + 2 + 5 + (−2) + 3 + 2 + 5 + 2 27 = = 2.7. 10 10 We can rewrite the above calculation as −2 × 1 4 2 3 +2× +3× +5× 10 10 10 10 102 Definition 9.2. Suppose X is a discrete random variable, we define the expectation (or mean or expected value) of X by X X x × pX (x). (15) x × P [X = x] = EX = x x In other words, The expected value of a discrete random variable is a weighted mean of the values the random variable can take on where the weights come from the pmf of the random variable. • Some references use mX or µX to represent EX. • For conciseness, we simply write x under the summation symbol in (15); this means that the sum runs over all x values in the support of X. (Of course, for x outside of the support, pX (x) is 0 anyway.) 9.3. Analogy: In mechanics, think of point masses on a line with a mass of pX (x) kg. at a distance x meters from the origin. In this model, EX is the center of mass (the balance point). This is why pX (x) is called probability mass function. Example 9.4. When X ∼ Bernoulli(p) with p ∈ (0, 1), Note that, since X takes only the values 0 and 1, its expected value p is “never seen”. 9.5. Interpretation: The expected value is in general not a typical value that the random variable can take on. It is often helpful to interpret the expected value of a random variable as the long-run average value of the variable over many independent repetitions of an experiment 1/4, x = 0 Example 9.6. pX (x) = 3/4, x = 2 0, otherwise 103 Example 9.7. For X ∼ P(α), EX = ∞ X ie i −α (α) i! i=0 = e−α α ∞ X k=0 = ∞ X e i −α (α) i! i=1 i+0=e αk = e−α αeα = α. k! −α ∞ X (α)i−1 (α) (i − 1)! i=1 Example 9.8. For X ∼ B(n, p), n n X X n i n! EX = i p (1 − p)n−i = pi (1 − p)n−i i i! (n − i)! i i=0 i=1 n n X X (n − 1)! n−1 i n−i i =n p (1 − p) =n p (1 − p)n−i (i − 1)! (n − i)! i−1 i=1 i=1 Let k = i − 1. Then, EX = n n−1 X k=0 n−1 X n − 1 n − 1 k+1 n−(k+1) p (1 − p) = np pk (1 − p)n−1−k k k k=0 We now have the expression in the form that we can apply the binomial theorem which finally gives EX = np(p + (1 − p))n−1 = np. We shall revisit this example again using another approach in Example 11.48. Example 9.9. Pascal’s wager : Suppose you concede that you don’t know whether or not God exists and therefore assign a 50 percent chance to either proposition. How should you weigh these odds when deciding whether to lead a pious life? If you act piously and God exists, Pascal argued, your gain–eternal happiness–is infinite. If, on the other hand, God does not exist, your loss, or negative return, is small–the sacrifices of piety. To weigh these possible gains and losses, Pascal proposed, you multiply the probability of each possible outcome by its payoff and add them all up, forming a kind of average or expected payoff. In other words, the mathematical expectation of your return on piety is one-half infinity (your gain if God exists) minus one-half a small number (your loss if he does not exist). Pascal knew enough about infinity to 104 know that the answer to this calculation is infinite, and thus the expected return on piety is infinitely positive. Every reasonable person, Pascal concluded, should therefore follow the laws of God. [14, p 76] • Pascals wager is often considered the founding of the mathematical discipline of game theory, the quantitative study of optimal decision strategies in games. 9.10. Technical issue: Definition (15) is only meaningful if the sum is well defined. The sum of infinitely many nonnegative terms is always welldefined, with +∞ as a possible value for the sum. • Infinite Expectation: Consider a random variable X whose pmf is defined by 1 , x = 1, 2, 3, . . . pX (x) = cx2 0, otherwise P 1 2 Then, c = ∞ n=1 n2 which is a finite positive number (π /6). However, EX = ∞ X k=1 ∞ ∞ X 1X1 11 = +∞. kpX (k) = k 2= ck c k k=1 k=1 Some care is necessary when computing expectations of signed random variables that take infinitely many values. • The sum over countably infinite many terms is not always well defined when both positive and negative terms are involved. • For example, the infinite series 1 − 1 + 1 − 1 + . . . has the sum 0 when you sum the terms according to (1 − 1) + (1 − 1) + · · · , whereas you get the sum 1 when you sum the terms according to 1 + (−1 + 1) + (−1 + 1) + (−1 + 1) + · · · . • Such abnormalities cannot happen when all terms in the infinite summation are nonnegative. 105 It is the convention in probability theory that EX should be evaluated as X X EX = xpX (x) − (−x)pX (x), x≥0 x<0 • If at least one of these sums is finite, then it is clear what value should be assigned as EX. • If both sums are +∞, then no value is assigned to EX, and we say that EX is undefined. Example 9.11. Undefined Expectation: Let 1 , x = ±1, ±2, ±3, . . . pX (x) = 2cx2 0, otherwise Then, EX = ∞ X k=1 kpX (k) − −1 X (−k) pX (k). k=−∞ The first sum gives ∞ X kpX (k) = k=1 ∞ X k=1 ∞ 1 1 X1 ∞ k = = . 2ck 2 2c k 2c k=1 The second sum gives −1 X k=−∞ (−k) pX (k) = ∞ X kpX (−k) = k=1 ∞ X k=1 ∞ 1 X1 1 ∞ = k = . 2 2ck 2c k 2c k=1 Because both sums are infinite, we conclude that EX is undefined. 9.12. More rigorously, to define EX, we let X + = max {X, 0} and X − = − min {X, 0}. Then observe that X = X + − X − and that both X + and X − are nonnegative r.v.’s. We say that a random variable X admits an expectation if EX + and EX − are not both equal to +∞. In which case, EX = EX + − EX − . 106 9.2 Function of a Discrete Random Variable Given a random variable X, we will often have occasion to define a new random variable by Y ≡ g(X), where g(x) is a real-valued function of the real-valued variable x. More precisely, recall that a random variable X is actually a function taking points of the sample space, ω ∈ Ω, into real numbers X(ω). Hence, we have the following definition Definition 9.13. The notation Y = g(X) is actually shorthand for Y (ω) := g(X(ω)). • The random variable Y = g(X) is sometimes called derived random variable. Example 9.14. Let pX (x) = 1 2 cx , 0, x = ±1, ±2 otherwise and Y = X 4. Find pY (y) and then calculate EY . 9.15. For discrete random variable X, the pmf of a derived random variable Y = g(X) is given by X pY (y) = pX (x). x:g(x)=y 107 Note that the sum is over all x in the support of X which satisfy g(x) = y. Example 9.16. A “binary” random variable X takes only two values a and b with P [X = b] = 1 − P [X = a] = p. X can be expressed as X = (b − a)I + a, where I is a Bernoulli random variable with parameter p. 9.3 Expectation of a Function of a Discrete Random Variable Recall that for discrete random variable X, the pmf of a derived random variable Y = g(X) is given by X pX (x). pY (y) = x:g(x)=y If we want to compute EY , it might seem that we first have to find the pmf of Y . Typically, this requires a detailed analysis of g which can be complicated, and it is avoided by the following result. 9.17. Suppose X is a discrete random variable. X g(x)pX (x). E [g(X)] = x This is referred to as the law/rule of the lazy/unconcious statistician (LOTUS) [22, Thm 3.6 p 48],[9, p. 149],[8, p. 50] because it is so much easier to use the above formula than to first find the pmf of Y . It is also called substitution rule [21, p 271]. Example 9.18. Back to Example 9.14. Recall that 1 2 x , x = ±1, ±2 pX (x) = c 0, otherwise (a) When Y = X 4 , EY = 108 (b) E [2X − 1] 9.19. Caution: A frequently made mistake of beginning students is to set E [g(X)] equal to g (EX). In general, E [g(X)] 6= g (EX). 1 (a) In particular, E X1 is not the same as EX . (b) An exception is the case of an affine function g(x) = ax + b. See also (9.23). Example 9.20. Continue from Example 9.4. For X ∼ Bernoulli(p), (a) EX = p (b) E X 2 = 02 × (1 − p) + 12 × p = p 6= (EX)2 . Example 9.21. Continue from Example 9.7. Suppose X ∼ P(α). ∞ ∞ i X 2 X αi−1 −α 2 −α α =e α i E X = ie i! (i − 1)! i=1 i=0 (16) We can evaluate the infinite sum in (16) by rewriting i as i−1+1: ∞ X i=1 ∞ i ∞ ∞ X X X αi−1 αi−1 αi−1 αi−1 = (i − 1 + 1) = (i − 1) + (i − 1)! (i − 1)! (i − 1)! (i − 1)! i=1 ∞ X =α i=2 αi−2 (i − 2)! + ∞ X i=1 i=1 i−1 α (i − 1)! i=1 = αeα + eα = eα (α + 1). Plugging this back into (16), we get E X 2 = α (α + 1) = α2 + α. 9.22. 9.8. For X ∼ B(n, p), one can find 2 Continue from Example E X = np(1 − p) + (np)2 . 109 9.23. Some Basic Properties of Expectations (a) For c ∈ R, E [c] = c (b) For c ∈ R, E [X + c] = EX + c and E [cX] = cEX (c) For constants a, b, we have E [aX + b] = aEX + b. (d) For constants c1 and c2 , E [c1 g1 (X) + c2 g2 (X)] = c1 E [g1 (X)] + c2 E [g2 (X)] . (e) For constants c1 , c2 , . . . , cn , " n # n X X E ck gk (X) = ck E [gk (X)] . k=1 k=1 Definition 9.24. Some definitions involving expectation of a function of a random variable: i i h h 0 k (a) Absolute moment: E |X| , where we define E |X| = 1 (b) Moment: mk = E X k = the k th moment of X, k ∈ N. • The first moment of X is its expectation EX. • The second moment of X is E X 2 . 110 9.4 Variance and Standard Deviation An average (expectation) can be regarded as one number that summarizes an entire probability model. After finding an average, someone who wants to look further into the probability model might ask, “How typical is the average?” or, “What are the chances of observing an event far from the average?” A measure of dispersion/deviation/spread is an answer to these questions wrapped up in a single number. (The opposite of this measure is the peakedness.) If this measure is small, observations are likely to be near the average. A high measure of dispersion suggests that it is not unusual to observe events that are far from the average. Example 9.25. Consider your score on the midterm exam. After you find out your score is 7 points above average, you are likely to ask, “How good is that? Is it near the top of the class or somewhere near the middle?”. Example 9.26. In the case that the random variable X is the random payoff in a game that can be repeated many times under identical conditions, the expected value of X is an informative measure on the grounds of the law of large numbers. However, the information provided by EX is usually not sufficient when X is the random payoff in a nonrepeatable game. Suppose your investment has yielded a profit of $3,000 and you must choose between the following two options: • the first option is to take the sure profit of $3,000 and • the second option is to reinvest the profit of $3,000 under the scenario that this profit increases to $4,000 with probability 0.8 and is lost with probability 0.2. The expected profit of the second option is 0.8 × $4, 000 + 0.2 × $0 = $3, 200 and is larger than the $3,000 from the first option. Nevertheless, most people would prefer the first option. The downside risk is too big for them. A measure that takes into account the aspect of risk is the variance of a random variable. [21, p 35] 111 9.27. The most important measures of dispersion are the standard deviation and its close relative, the variance. Definition 9.28. Variance: h 2 Var X = E (X − EX) i . (17) • Read “the variance of X” 2 , or VX [22, p. 51] • Notation: DX , or σ 2 (X), or σX • In some references, to avoid confusion from the two expectation symbols, they first define m = EX and then define the variance of X by Var X = E (X − m)2 . • We can also calculate the variance via another identity: Var X = E X 2 − (EX)2 • The units of the variance are squares of the units of the random variable. 9.29. Basic properties of variance: • Var X ≥ 0. • Var X ≤ E X 2 . • Var[cX] = c2 Var X. • Var[X + c] = Var X. • Var[aX + b] = a2 Var X. 112 Definition 9.30. Standard Deviation: p σX = Var[X]. • It is useful to work with the standard deviation since it has the same units as EX. • Informally we think of outcomes within ±σX of EX as being in the center of the distribution. Some references would informally interpret sample values within ±σX of the expected value, x ∈ [EX − σX , EX + σX ], as “typical” values of X and other values as “unusual”. • σaX+b = |a| σX . √ √ 9.31. σX and Var X: Note that the √ · function is a strictly increasing function. Because σX = Var X, if one of them is large, another one is also large. Therefore, both values quantify the amount of spread/dispersion in RV X (which can be observed from the spread or dispersion of the pmf or the histogram or the relative frequency graph). However, Var X does not have the same unit as the RV X. 9.32. In finance, standard deviation is a key concept and is used to measure the volatility (risk) of investment returns and stock returns. It is common wisdom in finance that diversification of a portfolio of stocks generally reduces the total risk exposure of the investment. We shall return to this point in Example 11.67. Example 9.33. Continue from Example 9.25. If the standard deviation of exam scores is 12 points, the student with a score of +7 with respect to the mean can think of herself in the middle of the class. If the standard deviation is 3 points, she is likely to be near the top. Example 9.34. Suppose X ∼ Bernoulli(p). (a) E X 2 = 02 × (1 − p) + 12 × p = p. 113 (b) Var X = EX 2 − (EX)2 = p − p2 = p(1 − p). Alternatively, if we directly use (17), we have Var X = E (X − EX)2 = (0 − p)2 × (1 − p) + (1 − p)2 × p = p(1 − p)(p + (1 − p)) = p(1 − p). Example 9.35. Continue from Example 9.7 and Example 9.21. Suppose X ∼ P(α). We have Var X = E X 2 − (EX)2 = α2 + α − α2 = α. Therefore, for Poisson random variable, the expected value is the same as the variance. Example 9.36. Consider two pmfs shown in Figure 11. The 2.4the Expectation 85 random variable X with pmf at the left has a smaller variance than the random variable Y withofpmf atitsthe right because more The variance is the average squared deviation X about mean. The variance characterizes how likely itmass is to observe values of the random variable far (their from its mean. For example, probability is concentrated near zero mean) in the consider the two pmfs shown in Figure 2.9. More probability mass is concentrated near zero graph at atthe than ingraph the atgraph at the right. [9, p. 85] in the graph the left left than in the the right. p ( i) p ( i) Y X −2 −1 1/3 1/3 1/6 1/6 0 1 2 i −2 −1 0 1 2 i Figure Example 2.27 shows the random X with pmf at the left has a smaller variance than the Figure2.9.11: Example 9.36thatshows thatvariable a random variable whose probability mass random variable Y with pmf at the right. is concentrated near the mean has smaller variance. [9, Fig. 2.9] Example 2.27. Let X and Y be the random variables with respective pmfs shown in Figure 2.9.We Compute and var(Y ). 9.37. havevar(X) already talked about variance and standard de- viation as By a number of=the ] and Solution. symmetry, that both Xindicates and Y have spread/dispersion zero mean, and so var(X) E[X 2pmf. 2 ]. Write var(Y ) = E[Y More specifically, let’s imagine a pmf that shapes like a bell curve. 2 ] = (−2)2 1 + (−1)2 1 + (1)2 1 + (2)2 1 = 2, E[Xσ As the value of X gets smaller, 6 3 the spread 3 6 of the pmf will be smaller and hence the pmf would “look sharper”. Therefore, the and 2 ] = (−2)2 1 + (−1)2 1 + (1)2 1 + (2)2 1 = 3. E[Ythe probability that random X 6would3 take a value that is 3 variable 6 Thus, X andthe Y aremean both zero-mean random variables taking the values ±1 and ±2. But Y far from would be smaller. is more likely to take values far from its mean. This is reflected by the fact that var(Y ) > var(X). When a random variable does not have114 zero mean, it is often convenient to use the variance formula, var(X) = E[X 2 ] − (E[X])2 , (2.17) The next property involves the use of σX to bound “the tail probability” of a random variable. 9.38. Chebyshev’s Inequality : 2 σX P [|X − EX| ≥ α] ≤ 2 α or equivalently P [|X − EX| ≥ nσX ] ≤ 1 n2 • Useful only when α > σX Example 9.39. If X has mean m and variance σ 2 , it is sometimes convenient to introduce the normalized random variable Y = X −m . σ Definition 9.40. Central Moments: A generalization of the variance is the nth central moment which is defined to be µn = E [(X − EX)n ] . (a) µ1 = E [X − EX] = 0. 2 (b) µ2 = σX = Var X: the second central moment is the variance. 115 Sirindhorn International Institute of Technology Thammasat University School of Information, Computer and Communication Technology ECS315 2014/1 Part IV.1 Dr.Prapun 10 10.1 Continuous Random Variables From Discrete to Continuous Random Variables In many practical applications of probability, physical situations are better described by random variables that can take on a continuum of possible values rather than a discrete number of values. For this type of random variable, the interesting fact is that • any individual value has probability zero: P [X = x] = 0 for all x (18) and that • the support is always uncountable. These random variables are called continuous random variables. 10.1. We can see from (18) that the pmf is going to be useless for this type of random variable. It turns out that the cdf FX is still useful and we shall introduce another useful function called probability density function (pdf) to replace the role of pmf. However, integral calculus36 is required to formulate this continuous analog of a pmf. 10.2. In some cases, the random variable X is actually discrete but, because the range of possible values is so large, it might be more convenient to analyze X as a continuous random variable. 36 This is always a difficult concept for the beginning student. 117 Example 10.3. Suppose that current measurements are read from a digital instrument that displays the current to the nearest onehundredth of a mA. Because the possible measurements are limited, the random variable is discrete. However, it might be a more convenient, simple approximation to assume that the current measurements are values of a continuous random variable. Example 10.4. If you can measure the heights of people with infinite precision, the height of a randomly chosen person is a continuous random variable. In reality, heights cannot be measured with infinite precision, but the mathematical analysis of the distribution of heights of people is greatly simplified when using a mathematical model in which the height of a randomly chosen person is modeled as a continuous random variable. [17, p 284] Example 10.5. Continuous random variables are important models for (a) voltages in communication receivers (b) file download times on the Internet (c) velocity and position of an airliner on radar (d) lifetime of a battery (e) decay time of a radioactive particle (f) time until the occurrence of the next earthquake in a certain region Example 10.6. The simplest example of a continuous random variable is the “random choice” of a number from the interval (0, 1). • In MATLAB, this can be generated by the command rand. In Excel, use rand(). • The generation is “unbiased” in the sense that “any number in the range is as likely to occur as another number.” • Histogram is flat over (0, 1). • Formally, this is called a uniform RV on the interval (0, 1). 118 Definition 10.7. We say that X is a continuous random variable37 if we can find a (real-valued) function38 f such that, for any set B, P [X ∈ B] has the form Z P [X ∈ B] = f (x)dx. (19) B • In particular, P [a ≤ X ≤ b] = Z b f( x)dx. (20) a In other words, the area under the graph of f (x) between the points a and b gives the probability P [a ≤ X ≤ b]. • The function f is called the probability density function (pdf) or simply density. • When we want to emphasize that the function f is a density of a particular random variable X, we write fX instead of f . 37 To be more rigorous, this is the definition for absolutely continuous random variable. At this level, we will not distinguish between the continuous random variable and absolutely continuous random variable. When the distinction between them is considered, a random variable X is said to be continuous (not necessarily absolutely continuous) when condition (18) is satisfied. Alternatively, condition (18) is equivalent to requiring the cdf FX to be continuous. Another fact worth mentioning is that if a random variable is absolutely continuous, then it is continuous. So, absolute continuity is a stronger condition. 38 Strictly speaking, δ-“function” is not a function; so, can’t use δ-function here. 119 2. The range of possible x values is along the horizontal axis. 3. The probability that x will take on a value between a and b will be the area under the curve between points a and b, as shown in Figure 7.1. The FIGURE 7.1 Area = P(a ≤ x ≤ b) f (x) For a continuous random variable, the probability distribution is described by a curve called the probability density function, f(x). The total area beneath the curve is 1.0, and the probability that x will take on some value between a and b is the area beneath the curve between points a and b. a x b Figure 13: For a continuous random variable, the probability distribution is described by a curve called the probability density function, f (x). The total area beneath the curve is 1.0, and the probability that X will take on some value between a and b is the area beneath the curve between points a and b. Example 10.8. For the random variable generated by the rand command in MATLAB39 or the rand() command in Excel, Definition 10.9. Recall that the support SX of a random variable X is any set S such that P [X ∈ S] = 1. For continuous random variable, SX is usually set to be {x : fX (x) > 0}. 39 The rand command in MATLAB is an approximation for two reasons: (a) It produces pseudorandom numbers; the numbers seem random but are actually the output of a deterministic algorithm. (b) It produces a double precision floating point number, represented in the computer by 64 bits. Thus MATLAB distinguishes no more than 264 unique double precision floating point numbers. By comparison, there are uncountably infinite real numbers in the interval from 0 to 1. 120 10.2 Properties of PDF and CDF for Continuous Random Variables 10.10. fX is determined only almost everywhere40 . That is, given a pdf f for a random variable X, if we construct a function g by changing the function f at a countable number of points41 , then g can also serve as a pdf for X. 10.11. The cdf of any kind of random variable X is defined as FX (x) = P [X ≤ x] . Note that even though there are more than one valid pdfs for any given random variable, the cdf is unique. There is only one cdf for each random variable. 10.12. For continuous random variable, given the pdf fX (x), we can find the cdf of X by Z x FX (x) = P [X ≤ x] = fX (t)dt. −∞ 10.13. Given the cdf FX (x), we can find the pdf fX (x) by • If FX is differentiable at x, we will set d FX (x) = fX (x). dx • If FX is not differentiable at x, we can set the values of fX (x) to be any value. Usually, the values are selected to give simple expression. (In many cases, they are simply set to 0.) 40 41 Lebesgue-a.e, to be exact More specifically, if g = f Lebesgue-a.e., then g is also a pdf for X. 121 Example 10.14. For the random variable generated by the rand command in MATLAB or the rand() command in Excel, Example 10.15. Suppose that the lifetime X of a device has the cdf x<0 0, 1 2 x , 0≤x≤2 FX (x) = 4 1, x>2 Observe that it is differentiable at each point x except at x = 2. The probability density function is obtained by differentiation of the cdf which gives 1 x, 0 < x < 2 fX (x) = 2 0, otherwise. At x = 2 where FX has no derivative, it does not matter what values we give to fX . Here, we set it to be 0. 10.16. In many situations when you are asked to find pdf, it may be easier to find cdf first and then differentiate it to get pdf. Exercise 10.17. A point is “picked at random” in the inside of a circular disk with radius r. Let the random variable X denote the distance from the center of the disk to this point. Find fX (x). 10.18. Unlike the cdf of a discrete random variable, the cdf of a continuous random variable has no jump and is continuous everywhere. Rx 10.19. pX (x) = P [X = x] = P [x ≤ X ≤ x] = x fX (t)dt = 0. Again, it makes no sense to speak of the probability that X will take on a pre-specified value. This probability is always zero. 10.20. P [X = a] = P [X = b] = 0. Hence, P [a < X < b] = P [a ≤ X < b] = P [a < X ≤ b] = P [a ≤ X ≤ b] 122 • The corresponding integrals over an interval are not affected by whether or not the endpoints are included or excluded. • When we work with continuous random variables, it is usually not necessary to be precise about specifying whether or not a range of numbers includes the endpoints. This is quite different from the situation we encounter with discrete random variables where it is critical to carefully examine the type of inequality. R 10.21. fX is nonnegative and R fX (x)dx = 1. Example 10.22. Random variable X has pdf −2x ce , x > 0 fX (x) = 0, otherwise Find the constant c and sketch the pdf. Definition 10.23. A continuous random variable is called exponential if its pdf is given by −λx λe , x > 0, fX (x) = 0, x≤0 for some λ > 0 Theorem 10.24. Any nonnegative42 function that integrates to one is a probability density function (pdf) of some random variable [8, p.139]. 42 or nonnegative a.e. 123 −∞ B for some integrable function f .a Since P(X ∈ IR) = 1, the function f must integrate to one; ∞ i.e., −∞ f (t) dt = 1. Further, since P(X ∈ B) ≥ 0 for all B, it can be shown that f must be nonnegative.1 A nonnegative function that integrates to one is called a probability density function (pdf). 10.25. Intuition/Interpretation: Usually, the set B is an interval such as B = [a, b]. In this case, The use of the word “density” originated with the analogy to the distribution of matterinb space. In physics, any finite volume, f (t) dt. P(a ≤ X ≤ b) = no matter how small, has aa positive mass, but there is no mass at single point. similar description applies to continuous See Figure 4.1(a).a Computing such A probabilities is analogous to determining the mass of random a variables. piece of wire stretching from a to b by integrating its mass density per unit length from a to b. Since most probability densities we work are continuous, for a small interval, say Approximately, for awith small ∆x, [x, x + ∆x], we have Z x+∆x x+∆x P [X ∈ [x, x + ∆x]] = fX (t)dt ≈ fX (x)∆x. P(x ≤ X ≤ x + ∆x) = x f (t) dtx ≈ f (x) ∆x. See Figure 4.1(b).This is why we call fX the density function. a x x+ x b Figure 14: (a) P [x ≤ X ≤ x + ∆x] is (b) the area of the shaded vertical strip. Figure 4.1. (a) P(a ≤ X ≤ b) = ab f (t) dt is the area of the shaded region under the density f (t). (b) P(x ≤ X ≤ In other words, the probability of random variable X taking x + ∆x) = xx+∆x f (t) dt is the area of the shaded vertical strip. on a value in a small interval around point c is approximately equal Note that for to random variables with f (c)∆c when ∆ca density, is the length of the interval. [x<X≤x+∆x] P(a ≤ X ≤ b) = P(a < X ≤ b) = P(aP ≤ X < b) = P(a < X < b) • In fact, fX (x) = lim ∆x→0 ∆x since the corresponding integrals over an interval are not affected by whether or not the • The number fX (x) itself is not a probability. In particular, endpoints are included or excluded. it does not have to be between 0 and 1. Some common densities • fX (c)of is a relative measure forA summary the likelihood that random Here are some examples continuous random variables. of the more comX will take a value in the immediate neighborhood mon ones can be foundvariable on the inside of the backon cover. a Later, of random pointvariable c. is involved, we write fX (x) instead of f (x). when more than one Stated differently, the pdf fX (x) expresses how densely the probability mass of random variable X is smeared out in the neighborhood of point x. Hence, the name of density function. 124 10.26. Histogram and pdf [17, 143approximation and 145]: From Histogram to ppdf Number of samples = 5000 2 4 6 Histogram 8 10 12 5000 Samples 14 16 18 Number of occurrences 1000 0.25 Vertical axis scaling 500 0 2 4 6 8 10 12 x Frequency (%) of occurrences 14 0.216 pdf Estimated pdf 18 20 0.15 10 0 2 4 6 8 10 x 12 14 16 0.1 18 0.05 6 0 2 4 6 8 10 x 12 14 16 18 Figure 15: From histogram to pdf. (a) A probability histogram is a bar chart that divides the range of values covered by the samples/measurements into intervals of the same width, and shows the proportion (relative frequency) of the samples in each interval. • To make a histogram, you break up the range of values covered by the samples into a number of disjoint adjacent intervals each having the same width, say width ∆. The height of the bar on each interval [j∆, (j + 1)∆) is taken such that the area of the bar is equal to the proportion of the measurements falling in that interval (the proportion of measurements within the interval is divided by the width of the interval to obtain the height of the bar). • The total area under the probability histogram is thus standardized/normalized to one. (b) If you take sufficiently many independent samples from a continuous random variable and make the width ∆ of the base intervals of the probability histogram smaller and smaller, the graph of the probability histogram will begin to look more and more like the pdf. 125 (c) Conclusion: A probability density function can be seen as a “smoothed out” version of a probability histogram 10.3 Expectation and Variance 10.27. Expectation: Suppose X is a continuous random variable with probability density function fX (x). Z ∞ EX = xfX (x)dx (21) Z−∞ ∞ E [g(X)] = g(x)fX (x)dx (22) −∞ In particular, E X2 = Var X = Z ∞ Z−∞ ∞ −∞ x2 fX (x)dx (x − EX)2 fX (x)dx = E X 2 − (EX)2 . Example 10.28. For the random variable generated by the rand command in MATLAB or the rand() command in Excel, Example 10.29. For the exponential random variable introduced in Definition 10.23, 126 10.30. If we compare other characteristics of discrete and continuous random variables, we find that with discrete random variables, many facts are expressed as sums. With continuous random variables, the corresponding facts are expressed as integrals. 10.31. All of the properties for the expectation and variance of discrete random variables also work for continuous random variables as well: (a) Intuition/interpretation of the expected value: As n → ∞, the average of n independent samples of X will approach EX. This observation is known as the “Law of Large Numbers”. (b) For c ∈ R, E [c] = c (c) For constants a, b, we have E [aX + b] = aEX + b. P P (d) E [ ni=1 ci gi (X] = ni=1 ci E [gi (X)]. (e) Var X = E X 2 − (EX)2 (f) Var X ≥ 0. (g) Var X ≤ E X 2 . (h) Var[aX + b] = a2 Var X. (i) σaX+b = |a| σX . 10.32. Chebyshev’s Inequality : P [|X − EX| ≥ α] ≤ 2 σX α2 or equivalently 1 n2 • This inequality use variance to bound the “tail probability” of a random variable. P [|X − EX| ≥ nσX ] ≤ • Useful only when α > σX 127 Example 10.33. A circuit is designed to handle a current of 20 mA plus or minus a deviation of less than 5 mA. If the applied current has mean 20 mA and variance 4 mA2 , use the Chebyshev inequality to bound the probability that the applied current violates the design parameters. Let X denote the applied current. Then X is within the design parameters if and only if |X − 20| < 5. To bound the probability that this does not happen, write P [|X − 20| ≥ 5] ≤ Var X 4 = = 0.16. 52 25 Hence, the probability of violating the design parameters is at most 16%. 10.34. Interesting applications of expectation: (a) fX (x) = E [δ (X − x)] (b) P [X ∈ B] = E [1B (X)] 128 Sirindhorn International Institute of Technology Thammasat University School of Information, Computer and Communication Technology ECS315 2014/1 Part IV.2 Dr.Prapun 10.4 Families of Continuous Random Variables Theorem 10.24 states that any nonnegative function f (x) whose integral over the interval (−∞, +∞) equals 1 can be regarded as a probability density function of a random variable. In real-world applications, however, special mathematical forms naturally show up. In this section, we introduce a couple families of continuous random variables that frequently appear in practical applications. The probability densities of the members of each family all have the same mathematical form but differ only in one or more parameters. 10.4.1 Uniform Distribution Definition 10.35. For a uniform random variable on an interval [a, b], we denote its family by uniform([a, b]) or U([a, b]) or simply U(a, b). Expressions that are synonymous with “X is a uniform random variable” are “X is uniformly distributed”, “X has a uniform distribution”, and “X has a uniform density”. This family is characterized by 0, x < a, x > b fX (x) = 1 b−a , a ≤ x ≤ b • The random variable X is just as likely to be near any value in [a, b] as any other value. 129 • In MATLAB, (a) use X = a+(b-a)*rand or X = random(’Uniform’,a,b) to generate the RV, 84 (b) use pdf(’Uniform’,x,a,b) and cdf(’Uniform’,x,a,b) to calculate the pdf and cdf, respectively. 0, x < a, x > b Exercise 10.36. Show that FX (x) = x−a b−a , a ≤ x ≤ b Probability theory, random variables and random processes Fx(x) fx(x) 1 1 b–a Fig. 3.5 a x b 0 a Fig. 3.6 x The pdf and cdf for the uniform random variable. Figure 16: The pdf and cdf for the uniform random variable. [16, Fig. 3.5] Fx(x) fx(x) 1 Example 10.37 (F2011). Suppose X is uniformly distributed on 2πσ 2 1 the interval (1, 2). (X ∼ U(1, 2).) (a) Plot the pdf fX (x) of X. b 0 0 1 2 x μ 0 μ x The pdf and cdf of a Gaussian (b) Plot the cdf Frandom of X. X (x) variable. G a u ss i a n ( o r n o r m a l ) ra n d o m va r i a b l e This is a continuous random variable that is described by the following pdf: 10.38. The uniform distribution provides a probability model for ' 1from the interval (x − μ)2 [a, b]. selecting a point at frandom (x) = √ exp − , (3.16) x 2π σ 2 2σ 2 • Use with caution to model a quantity that is known to vary parameters whose meaning is described later. Itelse is usually denoted where μrandomly and σ 2 are two between a and b but about which little is known. 2 as N (μ, σ ). Figure 3.6 shows sketches of the pdf and cdf of a Gaussian random variable. The Gaussian random variable is the most important and frequently encountered random variable in communications. This is because 130 thermal noise, which is the major source of noise in communication systems, has a Gaussian distribution. Gaussian noise and the Gaussian pdf are discussed in more depth at the end of this chapter. The problems explore other pdf models. Some of these arise when a random variable Example 10.39. [9, Ex. 4.1 p. 140-141] In coherent radio communications, the phase difference between the transmitter and the receiver, denoted by Θ, is modeled as having a uniform density on [−π, π]. (a) P [Θ ≤ 0] = 1 2 (b) P Θ ≤ π2 = 3 4 Exercise 10.40. Show that when X ∼ U([a, b]), EX = 2 2 1 2 2 = . Var X = (b−a) , and E X b + ab + a 12 3 10.4.2 a+b 2 , Gaussian Distribution 10.41. This is the most widely used model for the distribution of a random variable. When you have many independent random variables, a fundamental result called the central limit theorem (CLT) (informally) says that the sum (technically, the average) of them can often be approximated by normal distribution. Definition 10.42. Gaussian random variables: (a) Often called normal random variables because they occur so frequently in practice. (b) In MATLAB, use X = random(’Normal’,m,σ) or X = σ*randn + m. (c) fX (x) = √ 1 e− 2 ( 2πσ 1 x−m 2 σ ). • In Excel, use NORMDIST(x,m,σ,FALSE). In MATLAB, use normpdf(x,m,σ) or pdf(’Normal’,x,m,σ). • Figure 17 displays the famous bell-shaped graph of the Gaussian pdf. This curve is also called the normal curve. 131 84 Probability theory, random variables and random processes Fx(x) fx(x) (d) FX (x) has no closed-form expression. However, see 10.48. 1 1 Fig. 3.5 • Inb –MATLAB, use normcdf(x,m,σ) or cdf(’Normal’,x,m,σ). a • In Excel, use NORMDIST(x,m,σ,TRUE). x x a b 0 b σ2 . 0 N m, (e) We write aX ∼ The pdf and cdf for the uniform random variable. Fx(x) fx(x) 1 2πσ 2 1 1 2 0 Fig. 3.6 x μ 0 μ x The pdf and cdf of a Gaussian random variable. Figure 17: The pdf and cdf of N (µ, σ 2 ). [16, Fig. 3.6] G a u ss i a n ( o r n o r m a l ) ra n d o m va r i a b l e 2 10.43. EX by=themfollowing and Var is described pdf: X = σ . This is a continuous random variable that ' (x − μ)2 exp − , probabilities: 2σ 2 2π σ 2 fx (x) = √ 1 (3.16) 10.44. Important P [|X − µ| < σ] = 0.6827; where μ and σ 2 are two parameters whose meaning is described later. It is usually denoted P [|X −(μ,µ|σ 2> σ] =3.60.3173; ). Figure shows sketches of the pdf and cdf of a Gaussian random variable. as N Gaussian random variable is the most important and frequently encountered ranP [|X The − µ| > 2σ] = 0.0455; dom variable in communications. This is because thermal noise, which is the major source P [|X − µ| < 2σ] = 0.9545 of noise in communication systems, has a Gaussian distribution. Gaussian noise and the These are illustrated Figure 20. Gaussianvalues pdf are discussed in more depth in at the end of this chapter. The problems explore other pdf models. Some of these arise when a random variable is passed through a nonlinearity. How to determine the pdf of the random variable in this Example 10.45. case is discussed next.Figure 21 compares several deviation scores and the normal distribution: n c t i o n s of a ra n d o m va r i a b l e A function of a random variable y = g(x) is itself a (a) FuStandard scores have a mean of zero and a standard deviation random variable. From the definition, the cdf of y can be written as of 1.0. Fy (y) = P(ω ∈ : g(x(ω)) ≤ y). (3.17) (b) Scholastic Aptitude Test scores have a mean of 500 and a standard deviation of 100. 132 109 3.5 The Gaussian random variable and process (a) 0.6 0.4 Signal amplitude (V) 0.2 0 −0.2 −0.4 −0.6 −0.8 0 0.2 0.4 0.6 0.8 1 t (s) (b) 4 Histogram Gaussian fit Laplacian fit 3.5 3 fx(x) (1/V) 2.5 2 1.5 1 0.5 Fig. 3.14 0 −1 −0.5 0 x (V) 0.5 1 (a) A sample skeletal muscle (emg) signal, and (b) its histogram and pdf fits. Figure 18: Electrical activity of a skeletal muscle: (a) A sample skeletal muscle (emg) signal, and (b) its histogram and pdf fits. [16, Fig. 3.14] 1= = = 2 ∞ fx (x)dx = ∞ K1 e−ax dx 2 −∞ −∞ ∞ ∞ 2 2 K12 e−ax dx e−ay dy x=−∞ y=−∞ ∞ ∞ 2 2 K12 e−a(x +y ) dxdy. x=−∞ y=−∞ 133 2 (3.103) standard score. 3.5 The Gaussian random variable andhelp processus to determine The normal distribution can probabilities. 0.4 6.4 Notation σx = 1 σ =2 0.35 The z notation is critical in the use ofσx =normal x 5 0.3 distributions. 6.5 Normal Approximation of the Binomial 0.2 0.25 fx(x) 6.3 Applications of Normal Distributions 111 Binomial probabilities can be estimated by using a normal distribution. 0.15 0.1 23) Fourier transform: F ( fX ) = 0.05 ∞ ∫ f ( x)e X − jω x dt = e 1 − jω m − ω 2σ 2 2 mal Probability Distributions ence Scores . −∞ ∞ 24) Note that ∫e 0 x2 −α−15 dx = −∞ π−10 . α −5 0 x 5 10 15 Plots of the zero-meanxGaussian pdf for different values −m x −ofmstandard deviation, x−m σ . ⎞ x ⎛ ⎞ ⎛ ⎞ ⎛ 1− Q ⎜ = Q⎜− . [ X > x ] =ofQ the ⎜ ⎟ ; P [ X < x ] =Gaussian ⎟pdf Figure 25) 19:P Plots zero-mean for different values of standard σ ⎟⎠ ⎝ σ ⎠ ⎝ σ ⎠ ⎝ deviation,• σXP.⎡ X[16, − μ Fig. < σ ⎤ =3.15] 0.6827, P ⎡ X − μ > σ ⎤ = 0.3173 Fig. 3.15 ⎣ ⎦ ⎣ ⎦ μ > 2σ ⎤⎦Table = 0.0455, P ⎡⎣ X −single μ σ<x2on σ ⎤⎦different =most 0.9545 mal probability distributionP ⎡⎣isX −considered the important proba3.1 Influence of quantities stribution. An unlimited number a normal Range (±kσx )of continuous random k = 1 variables k = 2 have either k=3 k=4 roximately normal distribution. f ( x) ) x) P(mx − kσx < x ≤ mxf +( xkσ 0.683 0.955 0.997 0.999 −6 −8 e all familiar with IQ (intelligence quotient) scores and/or Test) 10−4 10Aptitude 10 Error probability 10−3 SAT (Scholastic the mean deviation 3.09 3.72scores have 4.75 a mean 5.61 of scores have a mean of 100Distance and afrom standard of 16. SAT 95% 68% a standard deviation of 100. But did you know that these continuous random variables w a normal distribution? of the pdf are ignorable. Indeed when communication systems are considered later it is the X X μ −σ μ μ +σ μ − 2σ μ μ + 2σ presence of these tails that results2 in bit errors. The probabilities are on the order of 10−3 – ∞ x − 1 10−12 , very small, but still significant in terms oftosystem It N is of Q-function : corresponds 26) Q z = e P [ X >performance. zof ~ ( ) ( 0,1interest ] where ∫z 2π 2 dx density Figure 20: Probability function X ∼X N (µ, σ)2; ) . to see how far, in terms of σx , one must be from the mean value to have the different levels of A, pictures the comparison of sevthat is Q ( z ) is the probability of the “tail” of N ( 0,1) . error probabilities. shall F IAs GU R EbeAseen in later chapters this translates to the required SNR to iation scores and the normal distriachieve a specified bit error probability. This N ( 0,1 ) is also shown in Table 3.1. Standard scores have a mean of Having considered the single (or univariate) Gaussian random variable, we turn our d a standard deviationattention of 1.0. to the case of two jointly Gaussian random variables (or the bivariate case). Again tic Aptitude Test scoresthey have a are described by their joint pdf which, in general, is an exponential whose exponent ( z(ax ) 2 +bx+cxy+dy+ey2 +f ) , where the conis a quadratic f 500 and a standard deviation of in the two variables, i.e., fx,y (x, y) =QKe stants K, a, b, c, d, e, and f are chosen to satisfy the basic properties of a valid joint pdf, 0.5 namely being always nonnegative (≥ 0), having unit volume, and also that the marginal 0 z t Intelligence Scale scores have a ∞ ∞ pdfs, fx (x) = −∞ fx,y2% (x, y)dy and = −∞34% fx,y (x,14% y)dx, are 2% valid. Written in standard 14%fy (y)34% 100 and a standard deviation of 16. 1 form joint pdf is a) the Q is a decreasing function with Q ( 0 ) = . case there are 34 percent of the –3.0 –2.0 –1.0 0 2 1.0 2.0 3.0 Standard Scores b) Q ( − z ) = 1 − Q ( z ) etween the mean and one standard −1 n, 14 percent between one and c) Qtwo ) ) = − z 300 400 500 600 700 800 (1 − Q ( z200 SAT π Scores d deviations, and 2 percent beyond π x x 1 2 − 2 sin θ 1 4 − 2 sin θ 2 dard deviations. d) Q ( x ) = ∫ e52 dθ68 . Q ( x84 e d116 θ. ) = ∫100 132 148 π π 2 2 2 0 2 0 Binet Intelligence Scale Scores ( f ( x )) 2 2 d 1 − x2 d 1 − 2 d ck, Applying Psychology: Critical and Creative of Deviation QThinking, e ;6.2Q“Pictures − Comparison f ( x ) ) =the e f (Several x) . e) ( x ) = − Figure ( dx dxby permission 2π 2πPearson the Normal Distribution,” ©Figure 1992 Prentice-Hall, Inc. Reproduced of Education, Inc. 21: Comparison of Several Deviation Scoresdx and the Normal Distribution 134 (c) Binet Intelligence Scale43 scores have a mean of 100 and a standard deviation of 16. In each case there are 34 percent of the scores between the mean and one standard deviation, 14 percent between one and two standard deviations, and 2 percent beyond two standard deviations. [Source: Beck, Applying Psychology: Critical and Creative Thinking.] 10.46. N (0, 1) is the standard Gaussian (normal) distribution. • In Excel, use NORMSINV(RAND()). In MATLAB, use randn. • The standard normal cdf is denoted by Φ(z). ◦ It inherits all properties of cdf. ◦ Moreover, note that Φ(−z) = 1 − Φ(z). 10.47. Relationship between N (0, 1) and N (m, σ 2 ). (a) An arbitrary Gaussian random variable with mean m and variance σ 2 can be represented as σZ +m, where Z ∼ N (0, 1). 43 Alfred Binet, who devised the first general aptitude test at the beginning of the 20th century, defined intelligence as the ability to make adaptations. The general purpose of the test was to determine which children in Paris could benefit from school. Binets test, like its subsequent revisions, consists of a series of progressively more difficult tasks that children of different ages can successfully complete. A child who can solve problems typically solved by children at a particular age level is said to have that mental age. For example, if a child can successfully do the same tasks that an average 8-year-old can do, he or she is said to have a mental age of 8. The intelligence quotient, or IQ, is defined by the formula: IQ = 100 × (Mental Age/Chronological Age) There has been a great deal of controversy in recent years over what intelligence tests measure. Many of the test items depend on either language or other specific cultural experiences for correct answers. Nevertheless, such tests can rather effectively predict school success. If school requires language and the tests measure language ability at a particular point of time in a childs life, then the test is a better-than-chance predictor of school performance. 135 3.6 Iz 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.30 0.31 0.32 0.33 0.34 035 0.36 0.37 0.38 0.39 0.40 0.41 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 <I>(z) 0.5000 0.5040 0.5080 0.5120 0.5160 0.5199 0.5239 0.5279 0.5319 0.5359 0.5398 0.5438 0.5478 0.5517 0.5557 0.5596 0.5636 0.5675 0.5714 0.5753 0.5793 0.5832 0.5871 0.5910 0.5948 0.5987 0.6026 0.6064 0.6103 0.6141 0.6179 0.6217 0.6255 0.6293 0.6331 0.6368 0.6406 0.6443 0.6480 0.6517 0.6554 0.6591 0.6628 0.6664 0.6700 0.6736 0.6772 0.6808 0.6844 0.6879 Iz 0.50 0.51 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.60 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.70 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.80 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.88 0.89 0.90 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 ~(z) 0.6915 0.6950 0.6985 0.7019 0.7054 0.7088 0.7123 0.7157 0.7190 0.7224 0.7257 0.7291 0.7324 0.7357 0.7389 0.7422 0.7454 0.7486 0.7517 0.7549 0.7580 0.7611 0.7642 0.7673 0.7704 0.7734 0.7764 0.7794 0.7823 0.7852 0.7881 0.7910 0.7939 0.7967 0.7995 0.8023 0.8051 0.8078 0.8106 0.8133 0.8159 0.8186 0.8212 0.8238 0.8264 0.8289 0.8315 0.8340 0.8365 0.8389 Iz 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 1.20 1.21 1.22 1.23 1.24 1.25 1.26 1.27 1.28 1.29 1.30 1.31 1.32 1.33 1.34 1.35 1.36 1.37 1.38 1.39 1.40 1.41 1.42 1.43 1.44 1.45 1.46 1.47 1.48 1.49 DELTA FUNCTIONS, MIXED RANDOM VARIABLES <I>(z) 0.8413 0.8438 0.8461 0.8485 0.8508 0.8531 0.8554 0.8577 0.8599 0.8621 0.8643 0.8665 0.8686 0.8708 0.8729 0.8749 0.8770 0.8790 0.8810 0.8830 0.8849 0.8869 0.8888 0.8907 0.8925 0.8944 0.8962 0.8980 0.8997 0.9015 0.9032 0.9049 0.9066 0.9082 0.9099 0.9115 0.9131 0.9147 0.9162 0.9177 0.9192 0.9207 0.9222 0.9236 0.9251 0.9265 0.9279 0.9292 0.9306 0.9319 Iz 1.50 1.51 1.52 1.53 1.54 1.55 1.56 1.57 1.58 1.59 1.60 1.61 1.62 1.63 1.64 1.65 1.66 1.67 1.68 1.69 1.70 1.71 1.72 1.73 1.74 1.75 1.76 1.77 1.78 1.79 1.80 1.81 1.82 1.83 1.84 1.85 1.86 1.87 1.88 1.89 1.90 1.91 1.92 1.93 1.94 1.95 1.96 1.97 1.98 1.99 <I>(z) 0.9332 0.9345 0.9357 0.9370 0.9382 0.9394 0.9406 0.9418 0.9429 0.9441 0.9452 0.9463 0.9474 0.9484 0.9495 0.9505 0.9515 0.9525 0.9535 0.9545 0.9554 0.9564 0.9573 0.9582 0.9591 0.9599 0.9608 0.9616 0.9625 0.9633 0.9641 0.9649 0.9656 0.9664 0.9671 0.9678 0.9686 0.9693 0.9699 0.9706 0.9713 0.9719 0.9726 0.9732 0.9738 0.9744 0.9750 0.9756 0.9761 0.9767 Iz 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25 2.26 2.27 2.28 2.29 2.30 2.31 2.32 2.33 2.34 2.35 2.36 2.37 2.38 2.39 2.40 2.41 2.42 2.43 2.44 2.45 2.46 2.47 2.48 2.49 <I>(z) 0.97725 0.97778 0.97831 0.97882 0.97932 0.97982 0.98030 0.98077 0.98124 0.98169 0.98214 0.98257 0.98300 0.98341 0.98382 0.98422 0.98461 0.98500 0.98537 0.98574 0.98610 0.98645 0.98679 0.98713 0.98745 0.98778 0.98809 0.98840 0.98870 0.98899 0.98928 0.98956 0.98983 0.99010 0.99036 0.99061 0.99086 0.99111 0.99134 0.99158 0.99180 0.99202 0.99224 0.99245 0.99266 0.99286 0.99305 0.99324 0.99343 0.99361 Iz 2.50 2.51 2.52 2.53 2.54 2.55 2.56 2.57 2.58 2.59 2.60 2.61 2.62 2.63 2.64 2.65 2.66 2.67 2.68 2.69 2.70 2.71 2.72 2.73 2.74 2.75 2.76 2.77 2.78 2.79 2.80 2.81 2.82 2.83 2.84 2.85 2.86 2.87 2.88 2.89 2.90 2.91 2.92 2.93 2.94 2.95 2.96 2.97 2.98 2.99 <I> (z) 123 I 0.99379 0.99396 0.99413 0.99430 0.99446 0.99461 0.99477 0.99492 0.99506 0.99520 0.99534 0.99547 0.99560 0.99573 0.99585 0.99598 0.99609 0.99621 0.99632 0.99643 0.99653 0.99664 0.99674 0.99683 0.99693 0.99702 0.99711 0.99720 0.99728 0.99736 0.99744 0.99752 0.99760 0.99767 0.99774 0.99781 0.99788 0.99795 0.99801 0.99807 0.99813 0.99819 0.99825 0.99831 0.99836 0.99841 0.99846 0.99851 0.99856 0.99861 Table 3.1 The standard nonnal CDF <t>(y). - This relationship can be used to generate general Gaussian RV from standard Gaussian RV. (b) If X ∼ N m, σ 2 , the random variable X −m σ is a standard normal random variable. That is, Z ∼ N (0, 1). Z= • Creating a new random variable by this transformation is referred to as standardizing. • The standardized variable is called “standard score” or “z-score”. 10.48. It is impossible to express the integral of a Gaussian PDF between non-infinite limits (e.g., (20)) as a function that appears on most scientific calculators. • An old but still popular technique to find integrals of the Gaussian PDF is to refer to tables that have been obtained by numerical integration. ◦ One such table is the table that lists Φ(z) for many values of positive z. ◦ For X ∼ N m, σ 2 , we can show that the CDF of X can be calculated by x−m FX (x) = Φ . σ Example 10.49. Suppose Z ∼ N (0, 1). Evaluate the following probabilities. (a) P [−1 ≤ Z ≤ 1] 136 (b) P [−2 ≤ Z ≤ 2] Example 10.50. Suppose X ∼ N (1, 2). Find P [1 ≤ X ≤ 2]. ⎛ 1⎞ N ⎜ 0, ⎟ ⎝ 2⎠ erf ( z ) Q 10.51. Q-function: Q (z) = R∞ 0 z 2 − x2 √1 e z 2π ( 2z ) dx corresponds to P [X > z] where X ∼ N (0, 1); that is Q (z) is the probability of the “tail” of N (0, 1). The Q function is then a complementary cdf (ccdf). N ( 0,1) 1 0.9 0.8 Q(z) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 -3 z -2 -1 0 1 2 3 z Figure 22: Q-function (a) Q is a decreasing function with Q (0) = 21 . (b) Q (−z) = 1 − Q (z) = Φ(z) 10.52. Error function (MATLAB): erf (z) = √ 1 − 2Q 2z √2 π Rz 2 e−x dx = 0 (a) It is an odd function of z. (b) For z ≥ 0, it corresponds to P [|X| < z] where X ∼ N 0, 21 . (c) lim erf (z) = 1 z→∞ 137 124 CHAPTER 3 CONTINUOUS RANDOM VARIABLES Iz I z' Q(z) Q(z) Iz Q(z) Iz Q(z) Iz Q(z) 3.00 1.35.10- 3 3.40 3.37-10- 4 3.80 7.23.10- 5 4.20 1.33.10- 5 4.60 2.11.10- 6 3.01 1.31.10- 3 3.41 3.25.10- 4 3.81 6.95.10- 5 4.21 1.28.10- 5 4.61 2.01.10- 6 3.02 1.26.10- 3 3.42 3.13.10- 4 3.82 6.67.10- 5 4.22 1.22.10- 5 4.62 1.92-10- 6 3.03 1.22.10-3 3.43 3.02.10- 4 3.83 6.41.10- 5 4.23 1.17.10- 5 4.63 1.83.10- 6 3.04 1.18.10- 3 3.44 2.91.10- 4 3.84 6.15.10- 5 4.24 1.12.10- 5 4.64 1.74.10- 6 3.05 1.14.10- 3 3.45 2.80.10- 4 3.85 5.91.10- 5 4.25 ,1.07.10- 5 4.65 1.66.10- 6 3.06 1.11.10- 3 3.46 2.70.10- 4 3.86 5.67.10- 5 4.26 1.02.10- 5 4.66 1.58.10- 6 3.07 1.07.10- 3 3.47 2.60.10-4 3.87 5.44.10- 5 4.27 9.77-10- 6 4.67 1.51.10- 6 3.08 1.04.10- 3 3.48 2.51.10-4 3.88 5.22.10- 5 4.28 9.34.10- 6 4.68 1.43.10- 6 3.09 1.00.10- 3 3.49 2.42.10- 4 3.89 5.01.10- 5 4.29 8.93.10- 6 4.69 1.37.10- 6 3.10 9.68.10- 4 3.50 2.33-10- 4 3.90 4.81.10- 5 4.30 8.54.10- 6 4.70 1.30.10- 6 3.11 9.35.10- 4 3.51 2.24.10- 4 3.91 4.61.10- 5 4.31 8.16.10- 6 4.71 1.24.10- 6 3.12 9.04.10- 4 3.52 2.16.10- 4 3.92 4.43.10- 5 4.32 7.80.10- 6 4.72 1.18.10- 6 3.13 8.74.10- 4 3.53 2.08.10- 4 3.93 4.25.10- 5 4.33 7.46.10- 6 4.73 1.12.10- 6 3.14 8.45.10-4 3.54 2.00.10- 4 3.94 4.07-10- 5 4.34 7.12.10- 6 4.74 1.07.10- 6 3.15 8.16.10- 4 3.55 1.93.10-4 3.95 3.91.10- 5 4.35 6.81.10- 6 4.75 1.02.10- 6 3.16 7.89.10- 4 3.56 1.85.10-4 3.96 3.75.10-5 4.36 6.50.10- 6 4.76 9.68.10- 7 3.17 7.62.10- 4 3.57 1.78.10-4 3.97 3.59.10-5 4.37 6.21.10- 6 4.77 9.21.10- 7 3.18 7.36.10- 4 3.58 1.72-10-4 3.98 3.45.10- 5 4.38 5.93.10- 6 4.78 8.76.10- 7 3.19 7.11.10- 4 3.59 1.65.10-4 3.99 3.30.10- 5 4.39 5.67.10- 6 4.79 8.34.10- 7 3.20 6.87-10- 4 3.60 1.59.10-4 4.00 3.17.10- 5 4.40 5.41.10- 6 4.80 7.93.10- 7 3.21 6.64.10- 4 3.61 1.53.10- 4 4.01 3.04.10- 5 4.41 5.17.10- 6 4.81 7.55.10- 7 3.22 6.41.10- 4 3.62 1.47.10-4 4.02 2.91.10- 5 4.42 4.94.10- 6 4.82 7.18.10- 7 3.23 6.19.10- 4 3.63 1.42.10-4 4.03 2.79.10- 5 4.43 4.71.10- 6 4.83 6.83-10- 7 3.24 5.98.10- 4 3.64 1.36.10-4 4.04 2.67.10- 5 4.44 4.50.10- 6 4.84 6.49.10- 7 3.25 5.77-10- 4 3.65 1.31.10-4 4.05 2.56.10- 5 4.45 4.29.10- 6 4.85 6.17.10- 7 3.26 5.57.10- 4 3.66 1.26.10-4 4.06 2.45.10- 5 4.46 4.10.10- 6 4.86 5.87-10- 7 3.27 5.38.10- 4 3.67 1.21.10-4 4.07 2.35.10- 5 4.47 3.91.10- 6 4.87 5.58.10- 7 3.28 5.19.10-4 3.68 1.17.10-4 4.08 2.25.10- 5 4.48 3.73-10- 6 4.88 5.30.10- 7 3.29 5.Ql.1O- 4 3.69 1.12.10-4 4.09 2.16.10- 5 4.49 3.56.10- 6 4.89 5.04.10- 7 3.30 4.83.10-4 3.70 1.08.10-4 4.10 2.07.10- 5 4.50 3.40.10- 6 4.90 4.79.10- 7 3.31 4.66.10- 4 3.71 1.04.10-4 4.11 1.98.10- 5 4.51 3.24.10- 6 4.91 4.55.10- 7 3.32 4.50.10- 4 3.72 9.96.10- 5 4.12 1.89.10- 5 4.52 3.09.10- 6 4.92 4.33.10- 7 3.33 4.34·10-4 3.73 9.57.10- 5 4.13 1.81.10-5 4.53 2.95.10- 6 4.93 4.11.10- 7 3.34 4.19·10-4 3.74 9.20.10- 5 4.14 1.74.10- 5 4.54 2.81.10- 6 4.94 3.91.10- 7 3.35 4.04·10-4 3.75 8.84.10- 5 4.15 1.66.10- 5 3,36 4.55 2.68.10- 6 4.95 3.71-10- 7 3.90·10-4 3.76 8.50.10- 5 4.16 1.59.10- 5 4.56 2.56.10- 6 4.96 3.52.10- 7 3.37 3.76.10--;4 3.77 8.16.10- 5 4.17 1.52.10- 5 3.62·10-4 4.97 3.35.10- 7 3.38 4.57 2.44.10- 6 3.78 7.84.10- 5 4.18 1.46.10- 5 3.39 4.98 3.18.10- 7 3.49·10-4 4.58 2.32.10- 6 3.79 7.53.10- 5 4.19 1.39.10- 5 4.59 2.22.10- 6 4.99 3.02.10-7 Table 3.2 The standard normal complementary CDF Q(z) . _------ ..... . '- .. ... --~-------- • ⎧0, k k −2 E ⎡( X − μ ) ⎤ = ( k − 1) E ⎡( X − μ ) ⎤ = ⎨ ⎣ ⎦ ⎣ ⎦ 1⋅ 3 ⋅ 5 ⋅ ⎩ • ⎧ 2 k , k odd k ⎪ 2 ⋅ 4 ⋅ 6 ⋅ ⋅ ( k − 1) σ ⎡ ⎤ [Papoulis p 111]. E X −μ =⎨ π ⎣ ⎦ ⎪1 ⋅ 3 ⋅ 5 ⋅ ⋅ ( k − 1) σ k , k even ⎩ k odd ⋅ ( k − 1)σ , k even k (d) erf (−z) = −erf (z) ⎤⎦ = 4 μ 2σ 2 + 2 σ4. • Var ⎡⎣ X 2 1 x 1 x √ (e) Φ(x) = 2 1 + erf ⎡ √ =⎡ 2 kerfc − 2 ⎧0, k −2 ⎤ ⎤ 28) For N ( 0,1) and k ≥ 1 , E ⎣ X ⎦(2) = ( k − 1) E ⎣ X ⎦ = ⎨1 ⋅ 3 ⋅ 5 ⋅ ⎩ k odd ⋅ ( k − 1) , k even (f)29)The complementary error 2function: √e− x dx= 1 − 22Q ( R 2∞z ) corresponds Error function (Matlab): erf ( z ) = to 2 ∫ erfc (z) = 1 − erf (z) = 2Qπ 0 2z = √π z e−x dx z 2 ⎛ 1⎞ P ⎡⎣ X < z ⎤⎦ where X ~ N ⎜ 0, ⎟ . ⎝ 2⎠ ⎛ 1⎞ N ⎜ 0, ⎟ ⎝ 2⎠ erf ( z ) Q 0 ( 2z ) z a) lim erf ( z ) = 1 z →∞ Figure 23: erf-function and Q-function b) erf ( − z ) = −erf ( z ) 10.4.3 Exponential Distribution Definition 10.53. The exponential distribution is denoted by E (λ). (a) λ > 0 is a parameter of the distribution, often called the rate parameter. (b) Characterized by −λx λe , x > 0, • fX (x) = 0, x≤0 1 − e−λx , x > 0, • FX (x) = 0, x≤0 138 • Survival-, survivor-, or reliability-function: (c) MATLAB: • X = exprnd(1/λ) or random(’exp’,1/λ) • fX (x) = exppdf(x,1/λ) or pdf(’exp’,x,1/λ) • FX (x) = expcdf(x,1/λ) or cdf(’exp’,x,1/λ) Example 10.54. Suppose X ∼ E(λ), find P [1 < X < 2]. Exercise 10.55. Exponential random variable as a continuous version of geometric random variable: Suppose X ∼ E (λ). Show that bXc ∼ G0 (e−λ ) and dXe ∼ G1 (e−λ ) Example 10.56. The exponential distribution is intimately related to the Poisson process. It is often used as a probability model for the (waiting) time until a “rare” event occurs. • time elapsed until the next earthquake in a certain region • decay time of a radioactive particle • time between independent events such as arrivals at a service facility or arrivals of customers in a shop. • duration of a cell-phone call • time it takes a computer network to transmit a message from one node to another. 139 10.57. EX = 1 λ Example 10.58. Phone Company A charges $0.15 per minute for telephone calls. For any fraction of a minute at the end of a call, they charge for a full minute. Phone Company B also charges $0.15 per minute. However, Phone Company B calculates its charge based on the exact duration of a call. If T , the duration of a call in minutes, is exponential with parameter λ = 1/3, what are the expected revenues per call E [RA ] and E [RB ] for companies A and B? Solution: First, note that ET = λ1 = 3. Hence, E [RB ] = E [0.15 × T ] = 0.15ET = $0.45. and E [RA ] = E [0.15 × dT e] = 0.15E dT e . Now, recall, from Exercise 10.55, that dT e ∼ G1 e−λ . Hence, E dT e = 1−e1 −λ ≈ 3.53. Therefore, E [RA ] = 0.15E dT e ≈ 0.5292. 10.59. Memoryless property : The exponential r.v. is the only continuous44 r.v. on [0, ∞) that satisfies the memoryless property: P [X > s + x |X > s] = P [X > x] for all x > 0 and all s > 0 [18, p. 157–159]. In words, the future is independent of the past. The fact that it hasn’t happened yet, tells us nothing about how much longer it will take before it does happen. • Imagining that the exponentially distributed random variable X represents the lifetime of an item, the residual life of an item has the same exponential distribution as the original lifetime, 44 For discrete random variable, geometric random variables satisfy the memoryless property. 140 regardless of how long the item has been already in use. In other words, there is no deterioration/degradation over time. If it is still currently working after 20 years of use, then today, its condition is “just like new”. • In particular, suppose we define the set B+x to be {x + b : b ∈ B}. For any x > 0 and set B ⊂ [0, ∞), we have P [X ∈ B + x|X > x] = P [X ∈ B] because P [X ∈ B + x] = P [X > x] −λt dt τ =t−x B+x λe = e−λx R R B λe−λ(τ +x) dτ . e−λx 10.60. Summary: X∼ Support SX Uniform U(a, b) (a, b) Normal (Gaussian) N (m, σ 2 ) R Exponential E(λ) (0, ∞) fX (x) = 1 , a < x < b, b−a 0, otherwise. 1 x−m 2 − √ 1 e 2( σ ) 2πσ −λx λe , x > 0, 0, x≤0 Table 4: Examples of probability density functions. Here, λ, σ > 0. 141 10.5 Function of Continuous Random Variables: SISO Reconsider the derived random variable Y = g(X). Recall that we can find EY easily by (22): Z EY = E [g(X)] = g(x)fX (x)dx. R However, there are cases when we have to evaluate probability directly involving the random variable Y or find fY (y) directly. Recall that for discrete random variables, it is easy to find pY (y) by adding all pX (x) over all x such that g(x) = y: X pX (x). (23) pY (y) = x:g(x)=y For continuous random variables, it turns out that we can’t45 simply integrate the pdf of X to get the pdf of Y . 10.61. For Y = g(X), if you want to find fY (y), the following two-step procedure will always work and is easy to remember: (a) Find the cdf FY (y) = P [Y ≤ y]. (b) Compute the pdf from the cdf by “finding the derivative” d fY (y) = dy FY (y) (as described in 10.13). 10.62. Linear Transformation: Suppose Y = aX + b. Then, the cdf of Y is given by h i y−b P X≤ a i , a > 0, h FY (y) = P [Y ≤ y] = P [aX + b ≤ y] = P X ≥ y−b , a < 0. a Now, by definition, we know that y−b y−b = FX , P X≤ a a 45 When you applied Equation (23) to continuous random variables, what you would get is 0 = 0, which is true but not interesting nor useful. 142 and y−b y−b y−b P X≥ =P X> +P X = a a a y−b y−b = 1 − FX +P X = . a a h i y−b For continuous random variable, P X = a = 0. Hence, FX y−b , a > 0, a FY (y) = 1 − FX y−b , a < 0. a Finally, fundamental theorem of calculus and chain rule gives y−b 1 , a > 0, d a fX a fY (y) = FY (y) = − 1 fX y−b , a < 0. dy a a Note that we can further simplify the final formula by using the | · | function: 1 y−b , a 6= 0. fY (y) = fX (24) |a| a Graphically, to get the plots of fY , we compress fX horizontally by a factor of a, scale it vertically by a factor of 1/|a|, and shift it to the right by b. Of course, if a = 0, then we get the uninteresting degenerated random variable Y ≡ b. Example 10.63. Suppose X ∼ E(λ). Let Y = 5X. Find fY (y). 143 10.64. Suppose X ∼ N (m, σ 2 ) and Y = aX+b for some constants a and b. Then, we can use (24) to show that Y ∼ N (am + b, a2 σ 2 ). Example 10.65. Amplitude modulation in certain communication systems can be accomplished using various nonlinear devices such as a semiconductor diode. Suppose we model the nonlinear device by the function Y = X 2 . If the input X is a continuous random variable, find the density of the output Y = X 2 . Example 10.66. Suppose X ∼ E(λ). Let Y = 144 1 X2 . Find fY (y). Exercise 10.67 (F2011). Suppose X is uniformly distributed on the interval (1, 2). (X ∼ U(1, 2).) Let Y = X12 . (a) Find fY (y). (b) Find EY . Exercise 10.68 (F2011). Consider the function x, x ≥ 0 g(x) = −x, x < 0. Suppose Y = g(X), where X ∼ U(−2, 2). Remark: The function g operates like a full-wave rectifier in that if a positive input voltage X is applied, the output is Y = X, while if a negative input voltage X is applied, the output is Y = −X. (a) Find EY . (b) Plot the cdf of Y . (c) Find the pdf of Y 145 P [X ∈ B] = P [X = x] = Discrete P pX (x) Continuous R fX (x)dx x∈B B pX (x) = F (x) − F (x− ) P X ((a, b]) = F (b) − F (a) Interval prob. P X ([a, b]) = F (b) − F a− P X ([a, b)) = F b− − F a− P X ((a, b)) = F b− − F (a) P EX = 0 P X ((a, b]) = P X ([a, b]) = P X ([a, b)) = P X ((a, b)) Zb = fX (x)dx = F (b) − F (a) a +∞ R xpX (x) x xfX (x)dx −∞ d P [g(X) ≤ y] . dy fY (y) = Alternatively, For Y = g(X), P pY (y) = pX (x) x: g(x)=y For Y = g(X), P [Y ∈ B] = P pX (x) x:g(x)∈B P E [g(X)] = g(x)pX (x) fY (y) = xk are the real-valued roots of the equation y = g(x). R fX (x)dx {x:g(x)∈B} +∞ R g(x)fX (x)dx x E [X 2 ] = P −∞ +∞ R x2 pX (x) x Var X = P x X fX (xk ) , |g 0 (xk )| k x2 fX (x)dx −∞ (x − EX)2 pX (x) +∞ R −∞ (x − EX)2 fX (x)dx Table 5: Important Formulas for Discrete and Continuous Random Variables 146 Sirindhorn International Institute of Technology Thammasat University School of Information, Computer and Communication Technology ECS315 2014/1 Part V.1 Dr.Prapun 11 Multiple Random Variables One is often interested not only in individual random variables, but also in relationships between two or more random variables. Furthermore, one often wishes to make inferences about one random variable on the basis of observations of other random variables. Example 11.1. If the experiment is the testing of a new medicine, the researcher might be interested in cholesterol level, blood pressure, and the glucose level of a test person. 11.1 A Pair of Discrete Random Variables In this section, we consider two discrete random variables, say X and Y , simultaneously. 11.2. The analysis are different from Section 9.2 in two main aspects. First, there may be no deterministic relationship (such as Y = g(X)) between the two random variables. Second, we want to look at both random variables as a whole, not just X alone or Y alone. Example 11.3. Communication engineers may be interested in the input X and output Y of a communication channel. 149 Example 11.4. Of course, to rigorously define (any) random variables, we need to go back to the sample space Ω. Recall Example 7.4 where we considered several random variables defined on the sample space Ω = {1, 2, 3, 4, 5, 6} where the outcomes are equally likely. In that example, we define X(ω) = ω and Y (ω) = (ω − 3)2 . Example 11.5. Consider the scores of 20 students below: 10, 9, 10, 9, 9, 10, 9, 10, 10, 9, 1, 3, 4, 6, 5, 5, 3, 3, 1, 3. | {z } | {z } Room #1 Room #2 The first ten scores are from (ten) students in room #1. The last 10 scores are from (ten) students in room #2. Suppose we have the a score report card for each student. Then, in total, we have 20 report cards. Figure 24: In Example 11.5, we pick a report card randomly from a pile of cards. I pick one report card up randomly. Let X be the score on that card. • What is the chance that X > 5? (Ans: P [X > 5] = 11/20.) 150 • What is the chance that X = 10? (Ans: pX (10) = P [X = 10] = 5/20 = 1/4.) Now, let the random variable Y denote the room# of the student whose report card is picked up. • What is the probability that X = 10 and Y = 2? • What is the probability that X = 10 and Y = 1? • What is the probability that X > 5 and Y = 1? • What is the probability that X > 5 and Y = 2? Now suppose someone informs me that the report card which I picked up is from a student in room #1. (He may be able to tell this by the color of the report card of which I have no knowledge.) I now have an extra information that Y = 1. • What is the probability that X > 5 given that Y = 1? • What is the probability that X = 10 given that Y = 1? 151 11.6. Recall that, in probability, “,” means “and”. For example, P [X = x, Y = y] = P [X = x and Y = y] and P [3 ≤ X < 4, Y < 1] = P [3 ≤ X < 4 and Y < 1] = P [X ∈ [3, 4) and Y ∈ (−∞, 1)] . In general, the event [“Some condition(s) on X”,“Some condition(s) on Y ”] is the same as the intersection of two events: [“Some condition(s) on X”] ∩ [“Some condition(s) on Y ”] which simply means both statements happen. More technically, [X ∈ B, Y ∈ C] = [X ∈ B and Y ∈ C] = [X ∈ B] ∩ [Y ∈ C] and P [X ∈ B, Y ∈ C] = P [X ∈ B and Y ∈ C] = P ([X ∈ B] ∩ [Y ∈ C]) . Remark: Linking back to the original sample space, this shorthand actually says [X ∈ B, Y ∈ C] = [X = {ω = {ω = [X ∈ B and Y ∈ C] ∈ Ω : X(ω) ∈ B and Y (ω) ∈ C} ∈ Ω : X(ω) ∈ B } ∩ {ω ∈ Ω : Y (ω) ∈ C} ∈ B] ∩ [Y ∈ C] . 152 11.7. The concept of conditional probability can be straightforwardly applied to discrete random variables. For example, P [“Some condition(s) on X” | “Some condition(s) on Y ”] (25) is the conditional probability P (A|B) where A = [“Some condition(s) on X”] and B = [“Some condition(s) on Y ”]. Recall that P (A|B) = P (A ∩ B)/P (B). Therefore, P [X = x and Y = y] , P [Y = y] P [X = x| Y = y] = and P [3 ≤ X < 4| Y < 1] = P [3 ≤ X < 4 and Y < 1] P [Y < 1] More generally, (25) is P ([“Some condition(s) on X”] ∩ [“Some condition(s) on Y ”]) P ([“Some condition(s) on Y ”]) P ([“Some condition(s) on X”,“Some condition(s) on Y ”]) = P ([“Some condition(s) on Y ”]) P [“Some condition(s) on X”,“Some condition(s) on Y ”] = P [“Some condition(s) on Y ”] = More technically, P [X ∈ B| Y ∈ C] = P ([X ∈ B] |[Y ∈ C]) = = P [X ∈ B, Y ∈ C] . P [Y ∈ C] 153 P ([X ∈ B] ∩ [Y ∈ C]) P ([Y ∈ C]) Definition 11.8. Joint pmf : If X and Y are two discrete random variables (defined on a same sample space with probability measure P ), the function pX,Y (x, y) defined by pX,Y (x, y) = P [X = x, Y = y] is called the joint probability mass function of X and Y . (a) We can visualize the joint pmf via stem plot. See Figure 25. (b) To evaluate the probability for a statement that involves both X and Y random variables: We first find all pairs (x, y) that satisfy the condition(s) in the statement, and then add up all the corresponding values from the joint pmf . More technically, we can then evaluate P [(X, Y ) ∈ R] by X P [(X, Y ) ∈ R] = pX,Y (x, y). (x,y):(x,y)∈R Example 11.9 (F2011). Consider random variables X and Y whose joint pmf is given by c (x + y) , x ∈ {1, 3} and y ∈ {2, 4} , pX,Y (x, y) = 0, otherwise. (a) Check that c = 1/20. (b) Find P X 2 + Y 2 = 13 . In most situation, it is much more convenient to focus on the “important” part of the joint pmf. To do this, we usually present the joint pmf (and the conditional pmf) in their matrix forms: 154 2.3 Multiple random variables 75 Example11.10. 2.13. In the precedingboth example, is the that the first cache valDefinition When Xwhat and Y probability take finitely many miss occurs after the third memory access? ues (both have finite supports), say SX = {x1 , . . . , xm } and SY = Solution. We need to find {y1 , . . . , yn }, respectively, we can arrange the probabilities pX,Y (xi , yj ) ∞ P(T > 3) = ∑ P(T = k). in an m × n matrix k=4 However, since P(T = k) = 0 for k ≤ 0, a finite series is obtained by writing pX,Y (x1 , y1 ) pX,Y (x−,P(T y2 )≤ 3). . . pX,Y (x1 , yn ) 3) = 1 1 p (x , y )P(T > X,Y 2 1 pX,Y (x2 , y3 2 ) . . . pX,Y (x2 , yn ) = .1 − ∑ P(T = (26) . .. .. .. k=1 . .k). . . 2 ]. pX,Y (xm , y1 ) pX,Y=(x1m−,(1y−2 )p)[1.+. .p +ppX,Y (xm , yn ) • We callmass thisfunctions matrix the joint pmf matrix. Joint shall probability The joint probability mass function of X and Y is defined by • The sum of all the entries in the matrix is one. pXY (xi , y j ) := P(X = xi ,Y = y j ). (2.7) An example for integer-valued random variables is sketched in Figure 2.8. 0.06 0.04 0.02 0 8 7 6 5 4 3 2 1 i 0 6 5 3 4 2 1 0 j 2.8. Sketch of bivariate probability mass function pXY (i, j). Figure 25:Figure Example of the plot of a joint pmf. [9, Fig. 2.8] • It turns out that we can extract the marginal probability mass functions pX (xi ) and pY (y j ) from the joint pmf p46 j ) using the formulas pX,Y (x, y) = 0 if XY (xxi , y ∈ / SX or y ∈ / SY . In other words, we don’t have to consider the the supports pXY (xiy , y joutside ) pX (x (2.8)of X i) =x ∑and j and Y , respectively. 46 To see this, note that pX,Y (x, y) can not exceed pX (x) because P (A ∩ B) ≤ P (A). Now, suppose at x = a, we have pX (a) = 0. Then pX,Y (a, y) must also = 0 for any y because it can not exceed pX (a) = 0. Similarly, suppose at y = a, we have pY (a) = 0. Then pX,Y (x, a) = 0 for any x. 155 11.11. From the joint pmf, we can find pX (x) and pY (y) by X pX,Y (x, y) (27) pX (x) = y pY (y) = X pX,Y (x, y) (28) x In this setting, pX (x) and pY (y) are call the marginal pmfs (to distinguish them from the joint one). (a) Suppose we have the joint pmf matrix in (26). Then, the sum of the entries in the ith row is47 pX (xi ), and the sum of the entries in the jth column is pY (yj ): pX (xi ) = n X pX,Y (xi , yj ) and pY (yj ) = m X pX,Y (xi , yj ) i=1 j=1 (b) In MATLAB, suppose we save the joint pmf matrix as P XY, then the marginal pmf (row) vectors p X and p Y can be found by p_X = (sum(P_XY,2))’ p_Y = (sum(P_XY,1)) Example 11.12. Consider the following joint pmf matrix 47 To see this, we consider A = [X = xi ] and a collection defined by Bj = [Y = yj ] and / SY ]. Note that the collection B0 , B1 , . . . , Bn partitions Ω. So, P (A) = Pn B0 = [Y ∈ P (A ∩ B j ). Of course, because the support of Y is SY , we have P (A ∩ B0 ) = 0. Hence, j=0 the sum can start at j = 1 instead of j = 0. 156 Definition 11.13. The conditional pmf of X given Y is defined as pX|Y (x|y) = P [X = x|Y = y] which gives pX,Y (x, y) = pX|Y (x|y)pY (y) = pY |X (y|x)pX (x). (29) 11.14. Equation (29) is quite important in practice. In most cases, systems are naturally defined/given/studied in terms of their conditional probabilities, say pY |X (y|x). Therefore, it is important the we know how to construct the joint pmf from the conditional pmf. Example 11.15. Consider a binary symmetric channel. Suppose the input X to the channel is Bernoulli(0.3). At the output Y of this channel, the crossover (bit-flipped) probability is 0.1. Find the joint pmf pX,Y (x, y) of X and Y . Exercise 11.16. Toss-and-Roll Game: Step 1 Toss a fair coin. Define X by 1, if result = H, X= 0, if result = T. Step 2 You have two dice, Dice 1 and Dice 2. Dice 1 is fair. Dice 2 is unfair with p(1) = p(2) = p(3) = 92 and p(4) = p(5) = p(6) = 1 9. (i) If X = 0, roll Dice 1. (ii) If X = 1, roll Dice 2. 157 Record the result as Y . Find the joint pmf pX,Y (x, y) of X and Y . Exercise 11.17 (F2011). Continue from Example 11.9. Random variables X and Y have the following joint pmf c (x + y) , x ∈ {1, 3} and y ∈ {2, 4} , pX,Y (x, y) = 0, otherwise. (a) Find pX (x). (b) Find EX. (c) Find pY |X (y|1). Note that your answer should be of the form ?, y = 2, ?, y = 4, pY |X (y|1) = 0, otherwise. (d) Find pY |X (y|3). Definition 11.18. The joint cdf of X and Y is defined by FX,Y (x, y) = P [X ≤ x, Y ≤ y] . 158 Definition 11.19. Two random variables X and Y are said to be identically distributed if, for every B, P [X ∈ B] = P [Y ∈ B]. Example 11.20. Let X ∼ Bernoulli(1/2). Let Y = X and Z = 1 − X. Then, all of these random variables are identically distributed. 11.21. The following statements are equivalent: (a) Random variables X and Y are identically distributed . (b) For every B, P [X ∈ B] = P [Y ∈ B] (c) pX (c) = pY (c) for all c (d) FX (c) = FY (c) for all c Definition 11.22. Two random variables X and Y are said to be independent if the events [X ∈ B] and [Y ∈ C] are independent for all sets B and C. 11.23. The following statements are equivalent: (a) Random variables X and Y are independent. [Y ∈ C] for all B, C. |= (b) [X ∈ B] (c) P [X ∈ B, Y ∈ C] = P [X ∈ B] × P [Y ∈ C] for all B, C. (d) pX,Y (x, y) = pX (x) × pY (y) for all x, y. (e) FX,Y (x, y) = FX (x) × FY (y) for all x, y. Definition 11.24. Two random variables X and Y are said to be independent and identically distributed (i.i.d.) if X and Y are both independent and identically distributed. 11.25. Being identically distributed does not imply independence. Similarly, being independent, does not imply being identically distributed. 159 Example 11.26. Roll a dice. Let X be the result. Set Y = X. Example 11.27. Suppose the pmf of a random variable X is given by 1/4, x = 3, α, x = 4, pX (x) = 0, otherwise. Let Y be another random variable. Assume that X and Y are i.i.d. Find (a) α, (b) the pmf of Y , and (c) the joint pmf of X and Y . 160 Example 11.28. Consider a pair of random variables X and Y whose joint pmf is given by 1/15, x = 3, y = 1, 2/15, x = 4, y = 1, 4/15, x = 3, y = 3, pX,Y (x, y) = β, x = 4, y = 3, 0, otherwise. (a) Are X and Y identically distributed? (b) Are X and Y independent? 161 11.2 Extending the Definitions to Multiple RVs Definition 11.29. Joint pmf: pX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = P [X1 = x1 , X2 = x2 , . . . , Xn = xn ] . Joint cdf: FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = P [X1 ≤ x1 , X2 ≤ x2 , . . . , Xn ≤ xn ] . 11.30. Marginal pmf: Definition 11.31. Identically distributed random variables: The following statements are equivalent. (a) Random variables X1 , X2 , . . . are identically distributed (b) For every B, P [Xj ∈ B] does not depend on j. (c) pXi (c) = pXj (c) for all c, i, j. (d) FXi (c) = FXj (c) for all c, i, j. Definition 11.32. Independence among finite number of random variables: The following statements are equivalent. (a) X1 , X2 , . . . , Xn are independent (b) [X1 ∈ B1 ], [X2 ∈ B2 ], . . . , [Xn ∈ Bn ] are independent, for all B1 , B2 , . . . , Bn . Q (c) P [Xi ∈ Bi , ∀i] = ni=1 P [Xi ∈ Bi ], for all B1 , B2 , . . . , Bn . Q (d) pX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = ni=1 pXi (xi ) for all x1 , x2 , . . . , xn . Q (e) FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = ni=1 FXi (xi ) for all x1 , x2 , . . . , xn . Example 11.33. Toss a coin n times. For the ith toss, let 1, if H happens on the ith toss, Xi = 0, if T happens on the ith toss. We then have a collection of i.i.d. random variables X1 , X2 , X3 , . . . , Xn . 162 Example 11.34. Roll a dice n times. Let Ni be the result of the ith roll. We then have another collection of i.i.d. random variables N1 , N2 , N3 , . . . , Nn . Example 11.35. Let X1 be the result of tossing a coin. Set X2 = X3 = · · · = Xn = X1 . 11.36. If X1 , X2 , . . . , Xn are independent, then so is any subcollection of them. 11.37. For i.i.d. Xi ∼ Bernoulli(p), Y = X1 + X2 + · · · + Xn is B(n, p). Definition 11.38. A pairwise independent collection of random variables is a collection of random variables any two of which are independent. (a) Any collection of (mutually) independent random variables is pairwise independent (b) Some pairwise independent collections are not independent. See Example (11.39). Example 11.39. Let suppose X, Y , and Z have the following joint probability distribution: pX,Y,Z (x, y, z) = 14 for (x, y, z) ∈ {(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)}. This, for example, can be constructed by starting with independent X and Y that are Bernoulli1 2 . Then set Z = X ⊕ Y = X + Y mod 2. (a) X, Y, Z are pairwise independent. (b) X, Y, Z are not independent. 163 Sirindhorn International Institute of Technology Thammasat University School of Information, Computer and Communication Technology ECS315 2014/1 11.3 Part V.2 Dr.Prapun Function of Discrete Random Variables 11.40. Recall that for discrete random variable X, the pmf of a derived random variable Y = g(X) is given by X pY (y) = pX (x). x:g(x)=y Similarly, for discrete random variables X and Y , the pmf of a derived random variable Z = g(X, Y ) is given by X pX,Y (x, y). pZ (z) = (x,y):g(x,y)=z Example 11.41. Suppose the joint 1/15, 2/15, pX,Y (x, y) = 4/15, 8/15, 0, Let Z = X + Y . Find the pmf of Z. 164 pmf of X and Y is given by x = 0, y = 0, x = 1, y = 0, x = 0, y = 1, x = 1, y = 1, otherwise. Exercise 11.42 (F2011). Continue from Exercise 11.9. Let Z = X +Y. (a) Find the pmf of Z. (b) Find EZ. 11.43. In general, when Z = X + Y , X pX,Y (x, y) pZ (z) = (x,y):x+y=z = X y pX,Y (z − y, y) = X x pX,Y (x, z − x). Furthermore, if X and Y are independent, X pX (x) pY (y) pZ (z) = (30) (x,y):x+y=z = X y pX (z − y) pY (y) = X x pX (x) pY (z − x). (31) Example 11.44. Suppose Λ1 ∼ P(α1 ) and Λ2 ∼ P(α2 ) are independent. Let Λ = Λ1 +Λ2 . Use (31) to show48 that Λ ∼ P(α1 +α2 ). First, note that pΛ (`) would be positive only on nonnegative integers because a sum of nonnegative integers (Λ1 and Λ2 ) is still a nonnegative integer. So, the support of Λ is the same as the support for Λ1 and Λ2 . Now, we know, from (31), that X P [Λ = `] = P [Λ1 + Λ2 = `] = P [Λ1 = i] P [Λ2 = ` − i] i Of course, we are interested in ` that is a nonnegative integer. The summation runs over i = 0, 1, 2, . . .. Other values of i would make P [Λ1 = i] = 0. Note also that if i > `, then ` − i < 0 and P [Λ2 = ` − i] = 0. Hence, we conclude that the index i can only 48 Remark: You may feel that simplifying the sum in this example (and in Exercise 11.45 is difficult and tedious, in Section 13, we will introduce another technique which will make the answer obvious. The idea is to realize that (31) is a convolution and hence we can use Fourier transform to work with a product in another domain. 165 be integers from 0 to k: P [Λ = `] = ` X i −α1 α1 −α2 e i! i=0 e ` α2`−i (` − i)! 1X `! α1i α2`−i `! i=0 i! (` − i)! =e −(α1 +α2 ) =e −(α1 +α2 ) (α1 + α2 )` , `! where the last equality is from the binomial theorem. Hence, the sum of two independent Poisson random variables is still Poisson! ( ` −(α1 +α2 ) (α1 +α2 ) e , ` ∈ {0, 1, 2, . . .} `! pΛ (`) = 0, otherwise Exercise 11.45. Suppose B1 ∼ B(n1 , p) and B2 ∼ B(n2 , p) are independent. Let B = B1 + B2 . Use (31) to show that B ∼ B(n1 + n2 , p). 11.4 Expectation of Function of Discrete Random Variables 11.46. Recall that the expected value of “any” function g of a discrete random variable X can be calculated from X E [g(X)] = g(x)pX (x). x Similarly49 , the expected value of “any” function g of two discrete random variables X and Y can be calculated from XX E [g(X, Y )] = g(x, y)pX,Y (x, y). x y 49 Again, these are called the law/rule of the lazy statistician (LOTUS) [22, Thm 3.6 p 48],[9, p. 149] because it is so much easier to use the above formula than to first find the pmf of g(X) or g(X, Y ). It is also called substitution rule [21, p 271]. 166 P [X = Y ] Discrete P pX (x) x∈B P pX,Y (x, y) (x,y):(x,y)∈R P pX (x) = pX,Y (x, y) y P pY (y) = pX,Y (x, y) P P x pX,Y (x, y) xP y: y<x P = pX,Y (x, y) y x: x>y P pX,Y (x, x) X Y Conditional E [g(X, Y )] pX,Y (x, y) = pX (x)pY (y) p (x,y) pX|Y (x|y) = X,Y p (y) Y PP g(x, y)pX,Y (x, y) P [X ∈ B] P [(X, Y ) ∈ R] Joint to Marginal: (Law of Total Prob.) P [X > Y ] |= x x y Table 6: Joint pmf: A Summary 11.47. E [·] is a linear operator: E [aX + bY ] = aEX + bEY . (a) Homogeneous: E [cX] = cEX (b) Additive: E [X + Y ] = EX + EY P P (c) Extension: E [ ni=1 ci gi (Xi )] = ni=1 ci E [gi (Xi )]. Example 11.48. Recall from 11.37 that when i.i.d. Xi ∼ Bernoulli(p), Y = X1 + X2 + · · · Xn is B(n, p). Also, from Example 9.4, we have EXi = p. Hence, " n # n n X X X EY = E Xi = E [Xi ] = p = np. i=1 i=1 i=1 Therefore, the expectation of a binomial random variable with parameters n and p is np. 167 Example 11.49. A binary communication link has bit-error probability p. What is the expected number of bit errors in a transmission of n bits? Theorem 11.50 (Expectation and Independence). Two random variables X and Y are independent if and only if E [h(X)g(Y )] = E [h(X)] E [g(Y )] for “all” functions h and g. • In other words, X and Y are independent if and only if for every pair of functions h and g, the expectation of the product h(X)g(Y ) is equal to the product of the individual expectations. • One special case is that Y |= X implies E [XY ] = EX × EY. (32) |= However, independence means more than this property. In other words, having E [XY ] = (EX)(EY ) does not necessarily imply X Y . See Example 11.61. 11.51. Let’s combined what we have just learned about independence into the definition/equivalent statements that we already have in 11.32. The following statements are equivalent: (a) Random variables X and Y are independent. [Y ∈ C] for all B, C. |= (b) [X ∈ B] (c) P [X ∈ B, Y ∈ C] = P [X ∈ B] × P [Y ∈ C] for all B, C. (d) pX,Y (x, y) = pX (x) × pY (y) for all x, y. (e) FX,Y (x, y) = FX (x) × FY (y) for all x, y. (f) 168 Exercise 11.52 (F2011). Suppose X and Y are i.i.d. with EX = EY = 1 and Var X = Var Y = 2. Find Var[XY ]. 11.53. To quantify the amount of dependence between two random variables, we may calculate their mutual information. This quantity is crucial in the study of digital communications and information theory. However, in introductory probability class (and introductory communication class), it is traditionally omitted. 11.5 Linear Dependence Definition 11.54. Given two random variables X and Y , we may calculate the following quantities: (a) Correlation: E [XY ]. (b) Covariance: Cov [X, Y ] = E [(X − EX)(Y − EY )]. (c) Correlation coefficient: ρX,Y = Cov[X,Y ] σX σY Exercise 11.55 (F2011). Continue from Exercise 11.9. (a) Find E [XY ]. 1 (b) Check that Cov [X, Y ] = − 25 . 11.56. Cov [X, Y ] = E [(X − EX)(Y − EY )] = E [XY ] − EXEY • Note that Var X = Cov [X, X]. 11.57. Var [X + Y ] = Var X + Var Y + 2Cov [X, Y ] 169 Definition 11.58. X and Y are said to be uncorrelated if and only if Cov [X, Y ] = 0. 11.59. The following statements are equivalent: (a) X and Y are uncorrelated. (b) Cov [X, Y ] = 0. (c) E [XY ] = EXEY . (d) |= 11.60. Independence implies uncorrelatedness; that is if X Y , then Cov [X, Y ] = 0. The converse is not true. Uncorrelatedness does not imply independence. See Example 11.61. Example 11.61. Let X be uniform on {±1, ±2} and Y = |X|. 11.62. The variance of the sum of uncorrelated (or independent) random variables is the sum of their variances. 170 Exercise 11.63. Suppose two fair dice are tossed. Denote by the random variable V1 the number appearing on the first dice and by the random variable V2 the number appearing on the second dice. Let X = V1 + V2 and Y = V1 − V2 . (a) Show that X and Y are not independent. (b) Show that E [XY ] = EXEY . 11.64. Cov [aX + b, cY + d] = acCov [X, Y ] Cov [aX + b, cY + d] = E [((aX + b) − E [aX + b]) ((cY + d) − E [cY + d])] = E [((aX + b) − (aEX + b)) ((cY + d) − (cEY + d))] = E [(aX − aEX) (cY − cEY )] = acE [(X − EX) (Y − EY )] = acCov [X, Y ] . Definition 11.65. Correlation coefficient: Cov [X, Y ] σX σY E [XY ] − EXEY Y − EY X − EX = . =E σX σY σX σY ρX,Y = • ρX,Y is dimensionless • ρX,X = 1 • ρX,Y = 0 if and only if X and Y are uncorrelated. • Cauchy-Schwartz Inequality 50 : |ρX,Y | ≤ 1. In other words, ρXY ∈ [−1, 1]. 50 Cauchy-Schwartz inequality shows up in many areas of Mathematics. A general form of this inequality can be stated in any inner product space: | ha, bi |2 ≤ ha, ai hb, bi . Here, the inner product is defined by hX, Y i = E [XY ]. The Cauchy-Schwartz inequality then gives |E [XY ] |2 ≤ E X 2 E Y 2 . 171 11.66. Linear Dependence and Cauchy-Schwartz Inequality 1, a > 0 (a) If Y = aX + b, then ρX,Y = sign(a) = −1, a < 0. • To be rigorous, we should also require that σX > 0 and a 6= 0. (b) When σY , σX > 0, equality occurs if and only if the following conditions holds ≡ ≡ ≡ ≡ ∃a 6= 0 such that (X − EX) = a(Y − EY ) ∃a 6= 0 and b ∈ R such that X = aY + b ∃c 6= 0 and d ∈ R such that Y = cX + d |ρXY | = 1 a In which case, |a| = σσXY and ρXY = |a| = sgn a. Hence, ρXY is used to quantify linear dependence between X and Y . The closer |ρXY | to 1, the higher degree of linear dependence between X and Y . Example 11.67. [21, Section 5.2.3] Consider an important fact that investment experience supports: spreading investments over a variety of funds (diversification) diminishes risk. To illustrate, imagine that the random variable X is the return on every invested dollar in a local fund, and random variable Y is the return on every invested dollar in a foreign fund. Assume that random variables X and Y are i.i.d. with expected value 0.15 and standard deviation 0.12. If you invest all of your money, say c, in either the local or the foreign fund, your return R would be cX or cY . • The expected return is ER = cEX = cEY = 0.15c. • The standard deviation is cσX = cσY = 0.12c Now imagine that your money is equally distributed over the two funds. Then, the return R is 21 cX + 12 cY . The expected return 172 is ER = 12 cEX + 12 cEY = 0.15c. Hence, the expected return remains at 15%. However, i c2 hc c2 c2 Var R = Var (X + Y ) = Var X + Var Y = × 0.122 . 2 4 4 2 √ c ≈ 0.0849c. So, the standard deviation is 0.12 2 In comparison with the distributions of X and Y , the pmf of 1 2 (X + Y ) is concentrated more around the expected value. The centralization of the distribution as random variables are averaged together is a manifestation of the central limit theorem. 11.68. [21, Section 5.2.3] Example 11.67 is based on the assumption that return rates X and Y are independent from each other. In the world of investment, however, risks are more commonly reduced by combining negatively correlated funds (two funds are negatively correlated when one tends to go up as the other falls). This becomes clear when one considers the following hypothetical situation. Suppose that two stock market outcomes ω1 and ω2 are possible, and that each outcome will occur with a probability of 1 2 Assume that domestic and foreign fund returns X and Y are determined by X(ω1 ) = Y (ω2 ) = 0.25 and X(ω2 ) = Y (ω1 ) = −0.10. Each of the two funds then has an expected return of 7.5%, with equal probability for actual returns of 25% and 10%. The random variable Z = 21 (X + Y ) satisfies Z(ω1 ) = Z(ω2 ) = 0.075. In other words, Z is equal to 0.075 with certainty. This means that an investment that is equally divided between the domestic and foreign funds has a guaranteed return of 7.5%. 173 Exercise 11.69. The input X and output Y of a system subject to random perturbations are described probabilistically by the following joint pmf matrix: x 1 3 y 2 4 5 0.02 0.10 0.08 0.08 0.32 0.40 (a) Evaluate the following quantities. (i) EX (ii) P [X = Y ] (iii) P [XY < 6] (iv) E [(X − 3)(Y − 2)] (v) E X(Y 3 − 11Y 2 + 38Y ) (vi) Cov [X, Y ] (vii) ρX,Y (b) Calculate the following quantities using what you got from part (a). (i) Cov [3X + 4, 6Y − 7] (ii) ρ3X+4,6Y −7 (iii) Cov [X, 6X − 7] (iv) ρX,6X−7 174 Answers: (a) (i) EX = 2.6 (ii) P [X = Y ] = 0 (iii) P [XY < 6] = 0.2 (iv) E [(X − 3)(Y − 2)] = −0.88 (v) E X(Y 3 − 11Y 2 + 38Y ) = 104 (vi) Cov [X, Y ] = 0.032 (vii) ρX,Y = 0.0447 (b) (i) Hence, Cov [3X + 4, 6Y − 7] = 3 × 6 × Cov [X, Y ] ≈ 3 × 6 × 0.032 ≈ 0.576 . (ii) Note that Cov [aX + b, cY + d] σaX+b σcY +d ac acCov [X, Y ] = = ρX,Y = sign(ac) × ρX,Y . |a|σX |c|σY |ac| ρaX+b,cY +d = Hence, ρ3X+4,6Y −7 = sign(3 × 4)ρX,Y = ρX,Y = 0.0447 . (iii) Cov [X, 6X − 7] = 1 × 6 × Cov [X, X] = 6 × Var[X] ≈ 3.84 . (iv) ρX,6X−7 = sign(1 × 6) × ρX,X = 1 . 175 11.6 Multiple Continuous Random Variables Discrete P pX (x) x∈B P pX,Y (x, y) P [X ∈ B] P [(X, Y ) ∈ R] Joint to Marginal: (x,y):(x,y)∈R pX (x) = P pX,Y (x, y) Continuous R fX (x)dx B RR fX,Y (x, y)dxdy {(x,y):(x,y)∈R} +∞ R fX (x) = y (Law of Total Prob.) pY (y) = P pX,Y (x, y) fY (y) = x P [X > Y ] P P pX,Y (x, y) P P pX,Y (x, y) y x: x>y P [X = Y ] P fX,Y (x, y)dx −∞ x y: y<x = fX,Y (x, y)dy −∞ +∞ R +∞ R Rx fX,Y (x, y)dydx −∞ −∞ +∞ R R∞ = fX,Y (x, y)dxdy −∞ y pX,Y (x, x) 0 x |= X Y Conditional E [g(X, Y )] P [g(X, Y ) ∈ B] Z =X +Y pX,Y (x, y) = pX (x)pY (y) p (x,y) pX|Y (x|y) = X,Y pY (y) PP g(x, y)pX,Y (x, y) x y P pX,Y (x, y) (x,y): g(x,y)∈B P pZ (z) = pX,Y (x, z − x) P x = pX,Y (z − y, y) y Table 7: pmf vs. pdf 176 fX,Y (x, y) = fX (x)fY (y) f (x,y) fX|Y (x|y) = X,Y fY (y) +∞ R +∞ R g(x, y)fX,Y (x, y)dxdy −∞ −∞ RR fX,Y (x, y)dxdy {(x,y): g(x,y)∈B} R +∞ fZ (z) = −∞ fX,Y (x, z − x)dx R +∞ = −∞ fX,Y (z − y, y) Sirindhorn International Institute of Technology Thammasat University School of Information, Computer and Communication Technology ECS315 2014/1 12 12.1 Part VI Dr.Prapun Limiting Theorems Law of Large Numbers (LLN) Definition 12.1. Let X1 , X2 , . . . , Xn be a collection of random variables with a common mean E [Xi ] = m for all i. In practice, since we do not know m, we use the numerical average, or sample mean, n 1X Mn = Xi n i=1 in place of the true, but unknown value, m. Q: Can this procedure of using Mn as an estimate of m be justified in some sense? A: This can be done via the law of large number. 12.2. The law of large number basically says that if you have a sequence of i.i.dPrandom variables X1 , X2 , . . .. Then the sample means Mn = n1 ni=1 Xi will converge to the actual mean as n → ∞. 12.3. LLN is easy to see via the property of variance. Note that " n # n X 1 1X E [Mn ] = E Xi = EXi = m n i=1 n i=1 and " Var[Mn ] = Var n 1X n i=1 # Xi n 1 X 1 = 2 Var Xi = σ 2 , n i=1 n 176 (33) Remarks: (a) For (33) to hold, it is sufficient to have uncorrelated Xi ’s. (b) From (33), we also have 1 σMn = √ σ. n (34) In words, “when uncorrelated (or independent) random variables each having the same distribution are averaged together, the standard deviation is reduced according to the square root law.” [21, p 142]. Exercise 12.4 (F2011). Consider i.i.d. random variables X1 , X2 , . . . , X10 . Define the sample mean M by 10 1 X M= Xk . 10 k=1 Let 10 1 X V1 = (Xk − E [Xk ])2 . 10 k=1 and 10 1 X V2 = (Xj − M )2 . 10 j=1 Suppose E [Xk ] = 1 and Var[Xk ] = 2. (a) Find E [M ]. (b) Find Var[M ]. (c) Find E [V1 ]. (d) Find E [V2 ]. 177 12.2 Central Limit Theorem (CLT) In practice, there are many random variables that arise as a sum of many other random variables. In this section, we consider the sum n X Sn = Xi (35) i=1 where the Xi are i.i.d. with common mean m and common variance σ2. • Note that when we talk about Xi being i.i.d., the definition is that they are independent and identically distributed. It is then convenient to talk about a random variable X which shares the same distribution (pdf/pmf) with these Xi . This allow us to write i.i.d. Xi ∼ X, (36) which is much more compact than saying that the Xi are i.i.d. with the same distribution (pdf/pmf) as X. Moreover, 2 we can also use EX and σX for the common expected value and variance of the Xi . Q: How does Sn behave? In the previous section, we consider the sample mean of identically distributed random variables. More specifically, we consider the random variable Mn = n1 Sn . We found that Mn will converge to m as n increases to ∞. Here, we don’t want to rescale the sum Sn by the factor n1 . 12.5 (Approximation of densities and pmfs using the CLT). The actual statement of the CLT is a bit difficult to state. So, we first give you the interpretation/insight from CLT which is very easy to remember and use: For n large enough, we can approximate Sn by a Gaussian random variable with the same mean and variance as Sn . 178 Note that the mean and variance of Sn is nm and nσ 2 , respectively. Hence, for n large enough we can approximate Sn by 2 N nm, nσ . In particular, s−nm (a) FSn (s) ≈ Φ σ√n . (b) If the Xi are continuous random variable, then fSn (s) ≈ √ 2 1 √ ) − 21 ( x−nm σ n . √ e 2πσ n (c) If the Xi are integer-valued, then 2 1 1 1 √ − 12 ( k−nm σ n ) . P [Sn = k] = P k − < Sn ≤ k + ≈√ e √ 2 2 2πσ n [9, eq (5.14), p. 213]. The approximation is best for k near nm [9, p. 211]. Example 12.6. Approximation for Binomial Distribution: For X ∼ B(n, p), when n is large, binomial distribution becomes difficult to compute directly because of the need to calculate factorial terms. (a) When p is not close to either 0 or 1 so that the variance is also large, we can use CLT to approxmiate (k−EX)2 1 e− 2 Var X 2π Var X (k−np)2 1 =p e− 2np(1−p) . 2πnp (1 − p) P [X = k] ≈ √ (37) (38) This is called Laplace approximation to the Binomial distribution [25, p. 282]. (b) When p is small, the binomial distribution can be approximated by P(np) as discussed in 8.45. (c) If p is very close to 1, then n − X will behave approximately Poisson. 179 i =1 nλ0 = λ . Hence X is approximately normal N ( λ , λ ) for λ large. Some says that the normal approximation is good when λ > 5 . p := 0.05 −λ e ⋅ λ −λ −1 1 p := 0.05 x e Γ ( x+ 1) 2⋅ π λ n := 100 λ := 5 ⋅e 2 ⋅λ ⋅ 0.15 ( x− λ ) −1 1 2⋅ π λ 0.1 − x λ−1 ⋅e 2 ⋅λ 0.06 ( x− λ ) 2 0.04 − x λ−1 e ⋅x Γ ( λ) Γ ( n− x+ 1) ⋅ Γ ( x+ 1) x Γ ( x+ 1) 2 e ⋅x Γ ( n+ 1) λ n := 800 λ := 40 Γ ( λ) x ⋅ p ⋅ ( 1− p ) Γ ( n+ 1) n− x 0.05 Γ ( n− x+ 1) ⋅ Γ ( x+ 1) 0 0 5 x ⋅ p ⋅ ( 1− p ) n− x 0.02 0 10 x 0 20 40 60 x The above Figure figure compare 1) Poisson when x istointeger, 2) Gaussian, 3) Gamma,and 4) 26: Gaussian approximation Binomial, Poisson distribution, Binomial. Gamma distribution. • • • If g : Z + → R is any bounded function and Λ ~ P ( λ ) , then E ⎡⎣λ g ( Λ + 1) − Λg ( Λ ) ⎤⎦ = 0 . Exercise 12.7 (F2011). Continue from Exercise 6.53. The stronger i ∞ ∞ λ i +1 if∞ n is very λ i ⎞ large. (By person (Kakashi) should win −λ λ − λ ⎛ the competition Proof. ∑ ( λ g ( i + 1) − ig ( i ) ) e = e ⎜ ∑ g ( i + 1) − ∑ ig ( i ) ⎟ of wins i ! the⎝ proportion i ! fights !⎠ i = 0 the law of large numbers, i =0 i =1 that iKakashi should be close to 55%.) However, because the results aremrandom ∞ λ i +1 ∞ λ +1 ⎞ −λ ⎛ e ⎜∑ i + 1)not guarantee 1) Kakashi − ∑ g ( m +that and n can not be very=large, weg (can ⎟ i ! m! ⎠ i 0 m 0 = = ⎝ will win. However, it may be good enough if the probability that =0 Kakashi wins the competition is greater than 0.85. + to for findwhich the minimal such that theinprobability E ⎡⎣ f ( Λ value Any function We expressed the form →R f : Zwant )⎤⎦ = 0 ofcann be that Kakashi wins the competition is greater than 0.85. f ( j ) = λ g ( j + 1) − jg ( j ) for a bounded function g. Let N be the number of fights that Kakashi wins among the n E ⎡⎣λ gwe + 1) − Λg ( Λ ) ⎤⎦ = 0 for all bounded g, then Λ has the Poisson Thus, conversely, fights. ifThen, ( Λneed h ni distribution P ( λ ) . ≥ 0.85. (39) P N> 2 Poisson distribution can be obtained as a limit from negative binomial distributions. Use the binomial central distribution limit theorem Table r3.1 Table 3.2 from (Thus, the negative withand parameters andor p can be approximated by the [Yates and Goodman] to approximate the minimal value of n such rq Poisson distribution with parameter λ = (maen-matching), provided that p is that (39) is satisfied. p “sufficiently” close to 1 and r is “sufficiently” large. Let X be Poisson with mean λ . Suppose, that the mean λ is chosen in accord with a probability distribution FΛ ( λ ) . Hence, 180 Sirindhorn International Institute of Technology Thammasat University School of Information, Computer and Communication Technology ECS315 2014/1 13 Part VII Dr.Prapun Three Types of Random Variables 13.1. Review: You may recall51 the following properties for cdf of discrete random variables. These properties hold for any kind of random variables. (a) The cdf is defined as FX (x) = P [X ≤ x]. This is valid for any type of random variables. (b) Moreover, the cdf for any kind of random variable must satisfies three properties which we have discussed earlier: CDF1 FX is non-decreasing CDF2 FX is right-continuous CDF3 lim FX (x) = 0 and lim FX (x) = 1. x→−∞ x→∞ (c) P [X = x] = FX (x) − FX (x− ) = the jump or saltus in F at x. Theorem 13.2. If you find a function F that satisfies CDF1, CDF2, and CDF3 above, then F is a cdf of some random variable. 51 If you don’t know these properties by now, you should review them as soon as possible. 181 Example 13.3. Consider an input X to a device whose output Y will be the same as the input if the input level does not exceed 5. For input level that exceeds 5, the output will be saturated at 5. Suppose X ∼ U(0, 6). Find FY (y). 13.4. We can categorize random variables into three types according to its cdf: (a) If FX (x) is piecewise flat with discontinuous jumps, then X is discrete. (b) If FX (x) is a continuous function, then X is continuous. (c) If FX (x) is a piecewise continuous function with discontinuities, then X is mixed. 182 81 3.1 Random variables (a) Fx(x) 1.0 ¥ –¥ 0 (b) Fx(x) x 1.0 –¥ 0 (c) Fx(x) ¥ x ¥ x 1.0 Fig. 3.2 –¥ 0 TypicalFigure cdfs: (a) variable, (b) a continuous random variable, and (c) a mixed random 27:a discrete Typicalrandom cdfs: (a) a discrete random variable, (b) a continuous random variable. variable, and (c) a mixed random variable [16, Fig. 3.2]. For a discrete random variable, Fx (x) is a staircase function, whereas a random variable is called continuous if Fx (x) is a continuous function. A random variable is called mixed if it is neither discrete nor continuous. Typical cdfs for discrete, continuous, and mixed random variables are shown in Figures 3.2(a), 3.2(b), and 3.2(c), respectively. Rather than dealing with the cdf, it is more common to deal with the probability density 183 of F (x), i.e., function (pdf), which is defined as the derivative x fx (x) = dFx (x) . dx (3.11) We have seen in Example 13.3 that some function can turn a continuous random variable into a mixed random variable. Next, we will work on an example where a continuous random variable is turned into a discrete random variable. Example 13.5. Let X ∼ U(0, 1) and Y = g(X) where 1, x < 0.6 g(x) = 0, x ≥ 0.6. Before going deeply into the math, it is helpful to think about the nature of the derived random variable Y . The definition of g(x) tells us that Y has only two possible values, Y = 0 and Y = 1. Thus, Y is a discrete random variable. Example 13.6. In MATLAB, we have the rand command to generate U(0, 1). If we want to generate a Bernoulli random variable with success probability p, what can we do? Exercise 13.7. In MATLAB, how can we generate X ∼ binomial(2, 1/4) from the rand command? 184