Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
EE603 Class Notes 12/05/13 John Stensby Chapter 11: Sequences of Finite-Second-Moment Random Variables The theory of sequences of finite-second-moment random variables is the topic of this chapter. We study their application to system theory, where they serve as the system’s input and output. It is natural to ask if a given random sequence has a limit, and in what sense is the limit is approached. Convergence of random variable sequences is discussed in this chapter. This chapter deals with discrete phenomenon and mathematics. Random sequences occur in applications where analog signals are sampled. They have applications in the fields of signal and image processing, digital control and digital communications. They have many applications outside of electrical engineering (for example, in the world of games, stocks, money and finance). Sequence of Random Variables – A Basic Definition Let (S, F, P) be a probability space (see Chapter 1 of these notes). A random variable X() maps S into the extended real line R+. (See Chapter 2 for the definition of a random variable.) A mapping from a sample space into a set of discrete-time sample functions is called a random, or stochastic, sequence X(n;), also known as a discrete-time random process. Often, we suppress the argument and write X(n). For each fixed in some sample space S, the function of n denoted by X(n;) is an “ordinary” deterministic sequence of numbers known as a sample function. Alternatively, also true is the fact that X(n;) is a sequence of random variables that is indexed on n. That is, for a fixed index n0, X(n0;) is a random variable. Example 11-1: X(n;) X()f(n), where X() is a random variable, and f(n) is a deterministic sequence of real numbers, is a simple random sequence. Example 11-2: X(n;) A()sin(n/10 + ()), where A() and () are random variables, is a random sequence. These two elementary examples have the feature that their future values are predictable from their present and past values. Updates at http://www.ece.uah.edu/courses/ee385/ 11-1 EE603 Class Notes 12/05/13 John Stensby Repeated Bernoulli Trials - the Quintessential Example of an Infinite Random Sequence Consider the tossing of a fair coin. Here, we have the sample space S = [H, T], the set of events (i.e., -algebra) F = {[H], [T], [H,T], Ø}, and the probability measure P that is usually associated with the tossing of a fair coin (i.e., P[H] = P[T] = 1/2, etc.). (S, F, P) is the probability space for the coin tossing experiment. We define a random variable X: S R as X (H) 1 (11-1) X (T) 0 . We know that {X < 1/2} = [T], {X > 1/2} = [H], etc. In what follows, probability space (S, F, P) and random variable X will be used “to build” the Bernoulli trials random sequence. This random sequence X(n) is defined easily. On the nth toss, assign X(n) = 1 (alternatively, X(n) = 0) if a heads (alternatively, tails) is obtained. We call X(n) the Bernoulli trials random sequence. This simple sequence must be described by using the methodology outlined above, a task that introduces some complexity. We need a probability space (S , F, P) so that X(n;) can be defined as a mapping from S into a set of binary functions. The space (S , F, P), the development of which is outlined below, is a product space. Our product space (S , F, P) is developed by using ideas from Chapter 1 of these notes (also see Chapter 8 of Stark and Woods, Probability and Random Processes, 4rd ed.). Instead of considering individual heads and tails as elementary outcomes of separate experiments, our product space has elementary outcomes that are infinite head/tail sequences. We define the Bernoulli trials random sequence X(n;) as a mapping from sample space S into a set of binary functions. Our product space will be built as an infinite Cartesian product of (S, F, P) with itself (recall that (S, F, P) describes the coin tossing experiment). But first, by (Sn, Fn, Pn), we denote the “nth repetition” of (S, F, P); that is, Sn = {[Hn], [Tn]} and Fn = {[Hn], [Tn], [Hn, Tn], Ø}, where Hn and Tn denotes “heads on the nth toss” and “tails on the nth toss”, respectively. Pn is the usual Updates at http://www.ece.uah.edu/courses/ee385/ 11-2 EE603 Class Notes 12/05/13 John Stensby probably measure that is used for the tossing of a fair coin (i.e., Pn[Hn] = Pn[Tn] = 1/2, etc.). Now, denoted as (S, F, P), our infinite-dimensional product space is determined from the (Sk, Fk, Pk), k 1, as outlined in what follows. The sample space S is the infinite Cartesian product S S k S1 S2 S3 S n . (11-2) k 1 Elements of S consist of infinite sequences of heads and tales. Element S has the form 1, , , , (11-3) where k Sk is either heads or tails, 1 k < ( is a sequence of heads/tails outcomes, not the outcome of a specific trial). F denotes the set of events (i.e., the -algebra) for the product space. F includes all sets of the form A k A1 A2 k 1 An , (11-4) where Ak Fk, 1 k < (set (11-4) is called a generalized rectangle). Also, all countable intersections and unions of such sets are included in F. For example, consider the event [the first two tosses produce different outcomes] F . This event in F is represented as {H1} {T2} S3 S4 {T1} {H 2 } S3 S 4 , (11-5) the union of two generalized rectangles each of the form (11-4). Also, the intersection of events Updates at http://www.ece.uah.edu/courses/ee385/ 11-3 EE603 Class Notes 12/05/13 John Stensby [{H1}{S2}{S3} … ] [{S1}{T2}{S3} … ] must mean the event [{H1}{ T2}{S3} … ]. As it turns out, F is the -algebra generated by the collection (i.e., set) of all generalized rectangles of the form (11-4) (see Chapter 1 for details on how a -algebra can be generated by a collection of sets). To finish our product space, we must define P , a probability measure on the product space. To accomplish this, we use the fact that the successive trials are independent, and probabilities can be multiplied (without this assumption, it would not be possible to define P without knowing the interdependence of each trial on the other trials). We start with events of the form given by (11-4), and we define P A n Pn (An ) P1(A1)P2 (A 2 )P3 (A3 ) Pn (A n ) n 1 n 1 Algebraic Product Cartesian Product (11-6) (note the different interpretation/usage of the symbol). We realize that every event in F can be represented as countable unions and/or intersections of events of the form (11-4). And, we use the Axioms of Probability (specifically, the Countable Additivity property - possessed by all valid probability measures) to extend definition (11-6) to all of F. For example, consider the event [the first two tosses are different] F given by (11-5). The probability of this event is P the first two tosses are different {T1} {H 2 } S3 S 4 P {H1} {T2 } S3 S 4 P {H1} {T2 } S3 S 4 P[H1 ] P [T2 ] P[S3 ] P [S 4 ] P {T1} {H 2 } S3 S 4 P[T1 ] P [H 2 ] P[S3 ] P [S 4 ] (11-7) 1/ 4 1/ 4 1/ 2 , Updates at http://www.ece.uah.edu/courses/ee385/ 11-4 EE603 Class Notes 12/05/13 John Stensby where we have used the fact that the event [the first two tosses are different] can be represented as the union of two events of the form (11-4). This finishes the definitions of P and our product space (S, F, P). Note that we have developed the same product space that is discussed in Chapter 8 of Stark and Woods, 4rd edition (also in the 3rd edition, Ch. 6). Finally, using our infinite-dimensional product space (S, F, P), we are in a position to define the Bernoulli trials random sequence. Denote an elementary outcome in S as . That is, = (1, 2, ... ) S , where each k Sk, k 1, is either a head or tail (so that is an infinite indexed sequence of heads and tails). We define the Bernoulli trials random sequence as X (n ; ) 1, n H n 0, (11-8) n Tn a mapping from S into the set of binary functions (remember that k is the kth component of ). The Limit of Nested Event Sequences When dealing with an infinite sequence of random variables, we need to be able to define the notion of a limit of an event sequence. In general, the limit of an event sequence is somewhat complicated and abstract. Before considering the general case, we first consider the important simple special case of nested sequences. A nested decreasing sequence of events is a simple concept. The event sequence Ak, k 1, is nested and decreasing if for each integer n 1 we have A1 A2 … An . (11-9) A convenient feature of such a sequence is that Updates at http://www.ece.uah.edu/courses/ee385/ 11-5 EE603 Class Notes AN 12/05/13 John Stensby N Ak . (11-10) k 1 Like a bounded and monotone sequence of real numbers, all of which have a real-number limit, a nested decreasing sequence of events has a well-defined limit event. As N in (11-10), we obtain A a countable intersection of events. And, a countable intersection of events is an event (recall that the set of events, a -algebra, is closed under countable unions and intersections). So, the limit of (11-10) is well defined. Often, we write AN A , where A is the limit. In some applications, an event can be expressed as the limit of a nested decreasing sequence of events, a sometimes-valuable representation. For example, let X(n), n 1, be a sequence of random variables and consider A X (n) 5, n 1 X (1) 5 X (2) 5 X (3) 5 X (N) 5 limit AN , (11-11) N where AN N {X(n) 5} . (11-12) n=1 Note that A1 A2 … AN so that AN is a nested decreasing sequence that has the limit A {X(n) < 5, n 1}. Similar results and statements can be made for a nested increasing sequence of events. The sequence BN is a nested increasing event sequence if B1 B2 … BN for all N. Furthermore, we can write Updates at http://www.ece.uah.edu/courses/ee385/ 11-6 EE603 Class Notes BN 12/05/13 John Stensby N Bn . (11-13) n 1 A nested increasing event sequence always has a limit B limit Bn (11-14) n since B can be written as a countable union of events. Often, we write BN B . Nested sequences of events are special cases of general event sequences. In Appendix 11b, we define the limit, when it exists, of an arbitrary sequence of events (unlike the case of nested sequences, the limit of an arbitrary event sequence may not exist!). Concerning infinite intersections and unions, some standard notation needs to be reviewed. For Bn, n 1, an arbitrary sequence of events, we utilize the standard notation N n=1 n=1 N n=1 n=1 Bn Nlimit Bn (11-15) Bn Nlimit Bn . (11-16) Of course, the limits (11-15) and (11-16) may, or may not, exist when the Bn are non-nested. Computing P[A], where A is the Limit of a Nested Event Sequence We need to be able to compute probabilities like P[X(n) < 5, n 1]. This probability can, we will argue, be computed as the limit of P[AN], where AN is represented by (11-12). That is, we need to show the second equality in (the first equality is a definition) N N P {X (n) < 5} P limit {X (n) < 5} limit P {X (n) < 5} . n=1 N n=1 N n=1 Updates at http://www.ece.uah.edu/courses/ee385/ (11-17) 11-7 EE603 Class Notes 12/05/13 John Stensby To any specified accuracy, this limit can be approximated by using sufficiently large N. The second equality in Equation (11-17) follows from the continuity of the probability measure P, a fact that we will argue in what follows. The events AN N {X(n) < 5} (11-18) n=1 form an indexed set of nested, decreasing events. The limit of the nested sequence is A limit AN limit N N {X(n) < 5} N n=1 {X(n) < 5} . (11-19) n=1 As will be shown in a section that follows, for the nested sequence of decreasing events, we have P A P limit AN limit P AN . N N (11-20) That is, we can interchange P and the limit operations. A similar statement will be made for a nested sequence of increasing events, BN B. Nested sequences are just special cases. In Appendix 11-B, we define what is meant by the limit of an event sequence where the events are not generally nested. Also, we argue that (11-20) is true for arbitrary convergent sequences of events. Continuity of a Probability Measure On a general probability space (S, F, P), the probability measure P has a continuity property. This is satisfying from an intuitive sense; it allows us to use P as a metric, or “gauge”, to “measure” the “size” of an event. Also, the continuity of P is used when we approximate the probability of an event that is represented as the limit of an infinite sequence of nested events. There is an analog here to the theory of continuous functions. Let f(x) be any function Updates at http://www.ece.uah.edu/courses/ee385/ 11-8 EE603 Class Notes 12/05/13 B2 B1 B3 John Stensby B4 Figure 11-1: An increasing sequence of events. with domain that includes x0. Then f(x) is continuous at x0 if and only if limit f ( xn ) f (limit xn ) f ( x0 ) n n (11-21) for all sequences {xn} that converge to x0. In words, Equation (11-21) states that one can interchange limit and function computation. In the sense described by Theorem 11-1 (and the more inclusive results given in Appendix 11B), this basic idea carries over to probability measures. Theorem 11-1: Consider an increasing sequence of events as shown by Figure 11-1. That is, the events are such that Bn Bn+1 for all n 1. Define the infinite union of these events as B limit BN limit N N Bn N n=1 Bn , (11-22) n=1 a well-defined event (since a -algebra is closed under countable unions). Then, to any degree of accuracy that is required, P[B] can be approximated by P[Bn] for sufficiently large n. That is, we have limit P[ Bn ] P limit Bn P[ B ] . n n Updates at http://www.ece.uah.edu/courses/ee385/ (11-23) 11-9 EE603 Class Notes 12/05/13 John Stensby In words, (11-23) says that we can move the limit operation from “outside” to “inside” the probability measure (interchange the limit and probability operations). Proof: We define the sequence of events A1 B1 A 2 B2 B (11-24) An Bn Bn 1 where the over-bar denotes set complement. The Bn are nested, and the disjoint An, 1 n N, “union up” to BN so we can write BN N N n 1 n 1 Bn An , 1 N (i.e., including N ) . (11-25) As a result of this, we have N N N P BN P Bn P An P[ An ] n 1 n 1 n 1 (11-26) for all finite N. Now take the limit of (11-26) to obtain limit P BN limit N N P[ An ] N n 1 P[ An ] . (11-27) n 1 Now, the most crucial step in the proof answers the question: does the sum on the right-hand Updates at http://www.ece.uah.edu/courses/ee385/ 11-10 EE603 Class Notes 12/05/13 John Stensby side of (11-27) converge? If yes, what does it converge to? Since the An are disjoint, we can use the Countable Additivity Property of P (see Chapter 1) to write limit P BN N P[ An ] P An 1 . n 1 n =1 (11-28) In (11-27), the middle Nth partial sum is an increasing sequence of real numbers that is bounded above by unity, as can be seen by (11-28). Hence, the limits in (11-27) and (11-28) converge. To find out what they converge to, simply use n =1 n =1 An Bn B , (11-29) in (11-28) to obtain the desired result limit P Bn P B P limit Bn . n n (11-30) Corollary 11-1: A version of Theorem 11-1 holds for a decreasing nested sequence of events. That is, suppose Bn Bn+1 for n 1. Then we can write P B P limit Bn limit P[ Bn ] , n n (11-31) where BN N Bn , B n 1 Bn . (11-32) n 1 Updates at http://www.ece.uah.edu/courses/ee385/ 11-11 EE603 Class Notes 12/05/13 John Stensby Proof: Similar to the proof given for Theorem 11-1. Appendix 11B extends Theorem 11-1 to more general, non-nested sequences of events. In the appendix, we define the limit, if it exists, of an event sequence, not necessarily nested. If event A is the limit of an infinite event sequence An , we show that P[A] is the limit of P[An] as index n approaches infinity. So, the probability measure P is continuous!! The analogy, drawn in the paragraph preceding Theorem 11-1, to continuous function is valid! Example 11-3: Theorem 11-1 and its corollary are used to approximate the probability of an event that is represented as the limit of an infinite sequence of events. For example, for each n 0, let Bn = {X[k] < 2 for 0 k n}. This is a decreasing and nested sequence of events: Bn+1 Bn, n 0. Suppose we wanted to calculate P[B], where B = {X[k] < 2 for 0 k}. We know that B Bn . (11-33) n=0 We use the corollary to approximate (as closely as desired) P[B] as the probability of a finite intersection. That is, based on our accuracy requirements, we select N and approximate N P[ B ] P[ Bn ] P[ Bn ] P X(0) 2, X(1) 2, , X(N) 2 . n=0 (11-34) n=0 Example 11-4: Back in Chapter 2 of these class notes, we were told that probability distribution functions are right continuous. We were told that F( x ) limit F(x +1/n) n (11-35) for any distribution function F(x) and all x. However, Equation (11-35) follows directly from Updates at http://www.ece.uah.edu/courses/ee385/ 11-12 EE603 Class Notes 12/05/13 John Stensby Theorem 11.1 since limit F(x +1/n) = limit P X x 1/ n n n P limit{X x 1/ n} n (11-36) P {X x} F(x). Statistical Specification of a Random Sequence In this chapter, we assume that all random variables are real-valued. This assumption greatly simplifies the notation, definitions and theory. From a conceptual standpoint, little is lost by assuming that everything is real valued (however, complex-valued random sequences are important - and often used - in many applications where band-pass signals are represented by their complex-valued, low-pass equivalents). A random sequence is statistically specified by its distribution functions, all orders are required in general. That is, for each positive integer n, and for all positive integer sequences k1 k2 … kn, we need knowledge of the nth-order distribution function F( xk1 , xk 2 ,, xk n ; k1, k 2 ,..., k n ) (11-37) P X[k1 ] xk1 , X[k 2 ] xk 2 , , X[k n ] xk n . Note that a complete statistical specification requires an infinite set of distribution functions. In (11-37), the algebraic variables xk1 , xk 2 , , xk n are called realization variables. The subscripts on these variables serve only to distinguish one variable from another; F(, , ; k1, k2, k3) is just as meaningful as F( xk1 , xk 2 , xk3 ; k1, k 2 , k 3 ) . Updates at http://www.ece.uah.edu/courses/ee385/ 11-13 EE603 Class Notes 12/05/13 John Stensby The probability density functions are obtained by differentiating distribution functions. That is, the nth-order probability density function is defined as f ( xk1 , xk 2 , , xk n ; k1, k 2 ,..., k n ) n F( xk1 , xk 2 , , xk n ; k1, k 2 ,..., k n ) xk1 xk 2 xk n . (11-38) The moments of a random sequence are important in applications. The mean (sometimes called the first-order average) is defined as [n] E X(n) (11-39) x f (x, n)dx for a sequence of continuous random variables. Second-order statistical averages appear often in practice. For example, the autocorrelation function is defined as R X (n, m) E X(n)X(m) x x f ( xn ,xm ; n, m)dxn dxm . n m (11-40) In a similar manner, the autocovariance function is defined as CX (n, m) E {X(n) (n)}{X(m) (m)} . (11-41) {xn - (n)}{xm (m)} f ( xn ,xm ; n, m)dxn dxm Note that both RX and CX are symmetric Updates at http://www.ece.uah.edu/courses/ee385/ 11-14 EE603 Class Notes 12/05/13 R X (n, m) E X(n)X(m) E X(m)X(n) John Stensby (11-42) R X (m, n) CX (n, m) E {X(n) (n )}{X(m) (m )} E {X(m) (m )}{X(n) (n )} (11-43) CX (m, n). Also, we can write CX (n, m) R X (n, m) (n)(m) . (11-44) The sequence X(n) is said to have uncorrelated elements (or to be uncorrelated) if R X (n, m) E X(n)X(m) E X(n)] E[X(m) (n)(m), n m . For such a sequence, (11-44) leads to the conclusion that CX (n, m) R X (n, m) (n)(m) 2 (n) 0 nm (11-45) nm where 2(n) denotes the sequence variance. Example 11-5: Many applications involve the arrival of objects. For example, we may be interested in the arrival of cars at an intersection, the arrival of electrons at the plate of a vacuum tube, etc. A commonly-used simplifying assumption is that the objects arrive independently of one another. Let (n) denote the interval of time (in seconds) between the arrival of the (n-1)th and nth objects (relative to a given initial time 0, (1) is the arrival time for the first object). The Updates at http://www.ece.uah.edu/courses/ee385/ 11-15 EE603 Class Notes 12/05/13 John Stensby time line is depicted by Fig. 11-2 below. For n 1, we assume that (n) is a sequence of identical, independent random variables each with the exponential density f (t ; n) exp[ t]U(t) . (11-46) The mean of (n) is (n) E[ (n)] x e x dx 1/ , (11-47) 0 and its variance is 2(n) E[ (n)2 ] (1/ )2 x2 e x dx (1/ )2 2 / 2 1/ 2 0 . (11-48) 1/ 2 Relative to a given initial time 0, the running sum of these intervals is the arrival times of the objects . That is, the arrival time of the nth object is T(n) n (k) , (11-49) k 1 (1) 0 (2) (3) (4) T(1) T(2) T(3) T(4) Fig. 11-2: Random arrival times. (n) is the time between arrivals, and T(n) is the actual arrival time (relative to origin 0). Updates at http://www.ece.uah.edu/courses/ee385/ 11-16 EE603 Class Notes 12/05/13 John Stensby a sequence of random variables indexed on n. Since the time intervals are independent, the density function fT(t;n) for T(n) is an n-1 fold convolution of (11-46) with itself. We claim that this first-order density function is fT (t; n) ( t) n 1 exp(- t)U(t) . (n 1)! (11-50) This result can be established by induction (by using a different approach, this same result was derived in Appendix 9B). Clearly, the result is correct for n = 1; assume it is true for n-1. Now, we convolve again to obtain f T (t ; n) f T (t ; n -1) exp( t)U(t) t ({t }) n 2 exp( ) exp( {t })d U(t) (n 2)! 0 t n-2 d U(t) nexp( t) 0 (n 2)! (11-51) t n 1 exp( t) U(t) n (n 1)! as claimed. Equation (11-51) is the Erlang density, and T(n) is an Erlang distributed random variable (this same result was obtained in Appendix 9B). The expected value of random variable T(n) is T (n) n (n) n / . (11-52) Since the interval random variables are independent, the variance of T(n) is Updates at http://www.ece.uah.edu/courses/ee385/ 11-17 EE603 Class Notes 12/05/13 T2 (n) n2 (n) n / 2 . John Stensby (11-53) Gaussian Random Sequences A random sequence X(n) is called a Gaussian random sequence if all its nth-order probability density functions are Gaussian. Such sequences are very popular. Because of the Central Limit Theorem, Gaussian sequences occur in many applications. Also, they are completely described by only first- and second-order statistical averages (i.e., means and covariances). Finally, use of Gaussian statistics simplifies many technical developments and makes mathematically tractable many problems in the areas of filtering, estimation, detection and control. Example 11-6: Let X(n) be a zero-mean Gaussian sequence; that is, E[X(n)] = 0 for all n. Also, let X be delta correlated; that is, RX(n,m) = E[X(n)X(m)] = 2(n-m), where 2 is the variance and 1, k 0 (k) . 0, k 0 (11-54) Often, delta-correlated sequences are said to be white; in many applications, delta-correlated Gaussian sequences are called white Gaussian noise. For n m, X(n) and X(m) are uncorrelated and, since they are Gaussian, independent. As a result, an Nth-order density function factors into a product of N first-order density functions. Most computer-based math packages (such as MatLab, Matcad, etc.) generate periodic sequences that, for many purposes, can be used to approximate white Gaussian noise. For these sequences, the correlation between elements can be very low, and the sequence period is very long relative to the number of sequence values that are needed. Independent Increments Random sequence X(n) is said to have independent increments if for all N > 1 and n1 Updates at http://www.ece.uah.edu/courses/ee385/ 11-18 EE603 Class Notes 12/05/13 John Stensby n2 ... nN the process increments X(n1), X(n2) - X(n1), X(n3) - X(n2), ... , X(nN) - X(nN-1) are jointly independent. Such processes have the nice feature that Nth-order density and distribution functions can be “built up” as products of the densities of the individual increments. For example, the second order distribution, for the case n2 > n1, can be written as F(x1, x 2 ; n1, n 2 ) P X(n1 ) x1, X(n 2 ) x 2 P X(n1 ) x1, X(n 2 ) X(n1 ) x 2 x1 (11-55) P X(n1 ) x1 P X(n 2 ) X(n1 ) x 2 x1 . We have seen independent increment processes in previous chapters. For example, the Random Walk process, introduced in Chapter 6, has independent increments. Stationarity Often, random sequences are generated by a mechanism that is not changing with time. In these cases, the sequence moments are constant. More precisely, a random sequence is said to be stationary if, for all positive integers N, the Nth-order density function of the sequence is invariant to any shift of the index. That is, stationarity requires f ( xn1 , xn 2 , , xn N ; n1, n 2 , ..., n N ) (11-56) f ( xn1 , xn 2 , , xn N ; n1 n0 , n 2 n0 , ..., n N n 0 ) for all orders N and index shift values n0. Example 11-5 introduces a random sequence (n) of interval times. Since the interval times are independent, Nth-order densities can be built up as products of first-order densities, each of a form given by (11-46). Clearly, the random sequence (n) of interval times is described by an Nth-order density that satisfies (11-56); the sequence (n) is stationary. On the other hand, the total waiting time to the nth arrival, the sum T(n) given by (11-49), is not Updates at http://www.ece.uah.edu/courses/ee385/ 11-19 EE603 Class Notes 12/05/13 John Stensby stationary as is obvious by inspection of (11-52) and (11-53). Wide-Sense Stationarity (WSS) A weaker form of stationarity is adequate in some applications. Sometimes, all that is required is “stationarity in all second-order statistics”. We say that a random sequence is wide sense stationary (WSS) if its mean function is constant and its covariance depends only on the time difference. That is, the sequence is WSS if (n) (0) (11-57) R X (n, m) R X (n - m) R X (k) , (11-58) where k n - m is the time difference between the two sequence values. Clearly, all stationary sequences are WSS. However, the converse is not true. Gaussian sequences provide an interesting example for which there is no difference between the two forms of stationarity. Two distinct sequences can have “mutual stationarity” properties. Wide sense stationary sequences X(n) and Y(m) are said to be jointly wide sense stationary if R xy [n, m] E X(n)Y(m) R xy [n m] . (11-59) That is, the cross correlation depends only on the time difference n-m. Suppose X(n) and Y(m) are jointly WSS so that Rxy[n-m] = E[X(n)Y(m)]. We define k n-m and write R xy [k] E X(m k)Y(m) . (11-60) That is, for Rxy[k], k denotes the shift applied to the first indexed sequence (i.e., X(m)). Note that Rxy[k] Ryx[k], in general. However, note that Rxy[k] = E[X(m+k)Y(m)] = E[X(m)Y(m-k)] Updates at http://www.ece.uah.edu/courses/ee385/ 11-20 EE603 Class Notes 12/05/13 John Stensby = Ryx[-k]. Power Spectral Density Let X(n) be a real-valued, wide-sense-stationary sequence with finite average power E[X(n)2] < . Denote the autocorrelation of X as Rx(k) = E[X(n+k)X(n)]. The power spectrum (or power spectral density) is denoted as Sx(). The celebrated Wiener-Khinchine theorem states that the power spectrum and autocorrelation comprise a discrete-time Fourier transform (DTFT) pair. That is, we write S x () F R x (k) R x (k) k R x (k) e jk , (11-61) 1 S x () e jk d . 2 Actually, Sx is 2 periodic in and only need be specified on – < . The average power in X(n) can be expressed as 1 S ()d Avg. Pwr E X 2 (n) 2 x (11-62) watts. White Noise Sequence A zero-mean X(n) is said to be a white noise sequence if R x (k) E[X(n k)X(n)] 2(k) . (11-63) Note that 2 is the finite variance of X(n). The power spectral density of X is S x () F R x (k) 2 (k) e jk 2 . (11-64) k Updates at http://www.ece.uah.edu/courses/ee385/ 11-21 EE603 Class Notes 12/05/13 John Stensby The average power in X is 1 2 Avg. Pwr E X 2 (n) d 2 watts. 2 (11-65) Note that a discrete-time white sequence has a finite average power (contrast this with the continuous-time case discussed in Chapter 8). Systems We are interested in systems with random sequence inputs. First, we review some basic definitions involving systems. Then, we focus on determining the mean and autocorrelation of the output of a linear system given descriptions of the input process and system impulse response. Given input sequence X(n,), we denote the system output as Y(n, ) L X(n, ) , (11-66) where operator L[·] maps input X into output Y. Often, we do not explicitly write the variable in the notation; we write Y(n) = L[X(n)] with the implied. The system is said to be linear if L X1(n) + X 2 (n) L X1(n) L X 2 (n) (11-67) for all inputs X1, X2 and all constants , . Linear system can be described by an unit sample response, denoted as h(n,m), assumed to be real-valued in what follows. This function is the response at time n to an unit sample function applied at time m. In general, impulse response h(n,m) may, or may not, depend on the absolute values of indices n and m, and h(n,m) may, or may not, be nonzero for values of n less than m. Given input X(n) and impulse response Updates at http://www.ece.uah.edu/courses/ee385/ 11-22 EE603 Class Notes 12/05/13 John Stensby h(n,m), we can express the output as Y(n) = L X(n) = h(n, )X( ) . (11-68) = A linear system is said to be shift invariant (or time invariant) if a simple delay in the input sequence produces a corresponding delay in the output sequence. More formally, we say that linear system L[·] is shift invariant if Y(n) = L X(n) Y(n - n 0 ) = L X(n - n 0 ) (11-69) for all input/output pairs (X, Y) and all index shifts n0. Shift invariant systems depend only on the difference of n and , not their absolute values. In this case, we can write h(n,) = h(n - ). Also, for shift invariant systems, Equation (11-68) becomes Y(n) = L X(n) = h(n - )X( ) = h * X , (11-70) = the convolution of input X with impulse response h. A system is said to be bounded input - bounded output (BIBO) stable if bounded input sequences produce bounded output sequences. A linear, shift-invariant system is BIBO stable if, and only if, its impulse response is absolutely summable; that is, BIBO stability is equivalent to h(n) . (11-71) n= A linear, shift-invariant system is said to be causal if it does not respond before it is Updates at http://www.ece.uah.edu/courses/ee385/ 11-23 EE603 Class Notes 12/05/13 John Stensby excited. More explicitly, for a causal system, if two inputs X1(n) and X2(n) are equal up to some index n0, then the corresponding outputs Y1(n) = L[X1(n)] and Y2(n) = L[X2(n)] must be equal up to index n0; what happens to the inputs after index n0 in no way influences the outputs before index n0. One can show that a linear, shift-invariant system is causal if, and only if, h(n) = 0 for n < 0. For a linear, shift-invariant and causal system, the input-output relationship becomes Y(n) = L X(n) = n h(n )X( ) . (11-72) = One should consider the differences between (11-68), the most general I/O formula, (11-70) for the shift-invariant case and (11-72) which describes the most restrictive case. Linear, shift-invariant systems can be analyzed in the frequency domain. For this purpose, we describe the Fourier Transform of signal X(k) as X F (e j ) X(k) e jk (11-73) k=- (we will use a subscript of F to denote a Fourier transform). If (11-73) converges, XF is a continuous, 2-periodic function of frequency variable . The inverse Fourier transform is X(n) = 1 X F (e j ) e jnd . 2 (11-74) The Fourier transform of a linear, shift-invariant system's output can be found easily. Simply use the convolution theorem with (11-70) to obtain YF (e j ) F [h(n) X(n)] H F (e j )X F (e j ) . Updates at http://www.ece.uah.edu/courses/ee385/ (11-75) 11-24 EE603 Class Notes 12/05/13 John Stensby Systems With Random Inputs Given a system with a random input, we determine below the mean and autocorrelation of the output. A more general, difficult problem is to find the Nth-order density function that describes the system output. A linear system with a Gaussian input will have a Gaussian output. Unfortunately, a general statement of this scope cannot be made for nonlinear systems or systems driven by non-Gaussian inputs. Theorem 11-2: Consider the linear system with input X(n) and output Y(n) = L[X(n)] (we do not require shift-invariance or causality). Suppose that both X(n) = E[X(n)] and Y(n) = E[Y(n)] exist. For this case, we can write Y (n) = E Y(n) E L[X(k)] L E[X(k)] L X (n) . (11-76) That is, it is possible to interchange the operations of L[·] and E[·]. We write E Y(n) = E h(n, m)X(m) . m (11-77) Then, we formally interchange the summation and expectation to obtain Y (n) = E Y(n) = E h(n, m)X(m) h(n, m)E[X(m)] m m= m , (11-78) h(n, m) X (m) and (11-76) is established. Note that our derivation of (11-76) is not rigorous. A potential problem with (11-78) is the formal interchange of expectation and summation. In cases where the mean of Y(n) does not Updates at http://www.ece.uah.edu/courses/ee385/ 11-25 EE603 Class Notes 12/05/13 John Stensby exist, this interchange is not valid (can you construct a simple example where the mean of Y(n) does not exist, i.e., the interchange in (11-78) is not valid?). We will consider this “interchange problem” again once we have studied some convergence concepts. Let's consider a special case of (11-78); suppose that input X(n) is wide-sense stationary and the system is shift invariant. Then, h(n,m) = h(n - m) and X(n) = X is a constant so that Y (n) h(n - m) X h(m) X H(e j ) X , m m= (11-79) so Y(n) = Y is a constant as well. The bracketed quantity on the right-hand side of (11-79) is the DC gain of the system (which we assume to be finite in the development of (11-79)). Example 11-7: Consider a low-pass filter with impulse response h(n) = nU(n), where 0 < < 1 to insure stability. The Fourier transform of h is H(e j ) 1 1 e j (11-80) According to (11-79), the mean of the filter output is X/(1 - ). Next, we determine the cross correlation between a system input X(n) and its output Y(n), both input and output assumed to be real valued. This quantity is defined as R XY (n, m) E X(n)Y(m) (11-81) Then, we use this result to find the autocorrelation RY of the system output. Theorem 11-3: Let X(n) and Y(n) denote the input and output, respectively, of a linear operator L[·]; that is, Y(n) = L[X(n)]. The cross-correlation between the input X and output Y can be calculated by the formula Updates at http://www.ece.uah.edu/courses/ee385/ 11-26 EE603 Class Notes 12/05/13 R XY (n, m) = L2 R X (n, m) , John Stensby (11-82) where L2 signifies that L operate with respect to the second variable (i.e., “m” is the independent variable in the operation), treating the first variable (i.e., “n”) as a constant. In a similar manner, the autocorrelation of the output can be calculated by the formula R Y (n, m) = L1 R XY (n, m) , (11-83) where L1 signifies that L operate with respect to the first variable only (i.e., “n” is the independent variable in the operation). Proof (see Theorems 7-1 and 7-2 for continuous-time version of this result): First, we write X(n)Y(m) = X(n)L[X(m)] = L2 [X(n)X(m)] , (11-84) where L2 operates on X(m). Now, take the expected value of this result to obtain E[X(n)Y(m)] = E L2 [X(n)X(m)] = L2 E[X(n)X(m)] = L2 R X (n, m) , (11-85) and this establishes (11-82). The formula for the autocorrelation of the output can be developed by taking the expectation of the product Y(n)Y(m) to obtain R Y (n, m) = E Y(n)Y(m) = E L[X(n)]Y(m) = E L1[X(n)Y(m)] = L1 E[X(n)Y(m)] (11-86) = L1 R XY (n, m) , (L1 operates on functions of n) and this establishes (11-83) so that the theorem is established. Updates at http://www.ece.uah.edu/courses/ee385/ 11-27 EE603 Class Notes 12/05/13 John Stensby Note that Theorem 11-3 does not require that operator L (i.e., linear system) be time invariant or that the input be wide-sense stationary. Let us consider Theorem 11-3 specialized to the case of a WSS input sequence X(n) and a shift-invariant, linear system described by unit sample function h(n). For this case, formula (11-82) yields R XY (n, m) = =- R X (n, m - )h() (11-87) = =- R X ([n - m] + )h() = =- R X ([n - m] - )h(-) . Observe that the right-hand side of (11-87) depends on n, m only through the difference k n-m. Hence, X and Y are jointly wide sense stationary, and we can write R XY (k) = R X (k) * h(-k) . (11-88) For the WSS case, the output correlation formula (11-86) becomes R Y (n, m) = R XY ({n - }- m)h() = R XY ({n - m}- )h() , (11-89) a formula depending on k n-m. Hence, we write R Y (k) = R XY (k) h(k) . (11-90) Finally, combining (11-88) and (11-90) yields Updates at http://www.ece.uah.edu/courses/ee385/ 11-28 EE603 Class Notes 12/05/13 R Y (k) = R X (k) h(-k) h(k) = R X (k) {h(-k) h(k)} , John Stensby (11-91) and we see that a WSS input produces an WSS output. Example 11-8: Suppose output Y is related to input X by the simple relationship Y(n) = L[X(n)] = X(n) - X(n -1) , (11-92) the first-order, backwards difference. For example, sequence Y(n) might be subjected to a threshold to implement a “pulse detector” function. The mean of the output is E[Y(n)] = E[X(n)] - E[X(n -1)] = X (n) - X (n -1) . (11-93) The cross-correlation between input and output is R XY (n, m) = L2 R X (n, m) = R X (n, m) - R X (n, m -1) . (11-94) Finally, the autocorrelation of the output is R Y (n, m) = L1 R XY (n, m) = R XY (n, m) - R XY (n -1, m) = R X (n, m) - R X (n, m -1) - { R X (n -1, m) - R X (n -1, m -1) } (11-95) = R X (n, m) - R X (n -1, m) - R X (n, m -1) + R X (n -1, m -1) . Suppose the input is WSS with autocorrelation R X (n,m) = a n-m , 0 < a < 1. Updates at http://www.ece.uah.edu/courses/ee385/ (11-96) 11-29 EE603 Class Notes 12/05/13 John Stensby Then Equations (11-93) and (11-94) yield Y = 0 RXY(n,m) = an-m - an-m+1 , (11-97) and (11-95) yields RY(n,m) = 2an-m - an-1-m -an-m+1. (11-98) The output sequence Y(n) is WSS; if k n – m, then (11-96) and (11-98) become RX(k) = ak (11-99) RY(k) = 2ak - ak-1 -ak+1, (11-100) respectively. A comparison of Fig. 11-3 and Fig. 11-4 (both correlations were computed and plotted for a = .6) reveals that the “pulse detector” (11-92) “decorrelates” the input data X(n), at Rx(k) Fig. 11-3: Eqn. (11-99) with a = .6. Updates at http://www.ece.uah.edu/courses/ee385/ Ry(k) Fig. 11-4: Eqn. (11-100) with a = .6. 11-30 EE603 Class Notes 12/05/13 John Stensby least to some extent. Vector Space of Random Variables All real-valued random variables with finite second moments (i.e., finite average power) comprise a vector space over the field of real numbers. We define vector space L2 as L2 X : E[ X 2 ] , (11-101) all real-valued, finite-second-moment random variables. We take the real number field, denoted here by R, as our scalar field. To show that L2 is a valid vector space, we must show, among other things, that L2 is closed under vector addition (i.e., X L2 and Y L2 implies that X + Y L2) and scalar multiplication (i.e., X L2 and c R implies that cX L2). The fact that L2 is closed under scalar multiplication follows easily. Clearly if X L2 and c R then E[cX2] = c2E[X2] < so cX L2. The fact that L2 is closed under vector addition follows from use of the Schwarz inequality (sometimes called the Cauchy-Schwarz inequality). Theorem 11-4 (Schwarz): Let X L2 and Y L2. Then E[XY] 2 2 2 E[ X ] E[ Y ] . (11-102) Proof: Let be any real-valued number and consider 2 2 2 2 E[ X Y ] E[ Y ] E[XY] E[XY] E[ X ] 0 . (11-103) Now, Equation (11-103) is a quadratic equation in , and the roots are either complex-valued or real and equal (see Fig. 5). Hence, in the quadratic equation, the discriminant must be nonpositive, or Updates at http://www.ece.uah.edu/courses/ee385/ 11-31 EE603 Class Notes 12/05/13 John Stensby E[ Y ] 2 2E[XY] E[ X ] 2 2 -axis Fig. 5: Graph of quadratic equation 2 2 2 4 E[XY] 4E[ X ]E[ Y ] 0 . (11-104) The Schwarz inequality follows directly from (11-104). In (11-102), equality results when Y is a scalar multiple of X. Now, we show that L2 is closed under vector addition. Let X L2 and Y L2 and consider the sum X + Y. The second moment of the sum satisfies 2 2 2 E[ X Y ] E[ X ] 2E[XY] E[ Y ] . 2 2 2 (11-105) 2 E[ X ] 2 E[ X ] E[ Y ] E[ Y ] However, all quantities on the right-hand-side of (11-105) are finite since X L2 and Y L2. Hence, the sum X+Y L2, and L2 is closed under vector addition. Closure under vector addition and scalar multiplication is necessary for L2 to be a valid vector space. The remaining requirements (found in any elementary text on linear algebra) that L2 must satisfy are shown easily. Hence, we can consider the set of all real-valued random variables with finite second moments to be a valid vector space. Updates at http://www.ece.uah.edu/courses/ee385/ 11-32 EE603 Class Notes 12/05/13 John Stensby Equality of Random Variables Let X and Y be random variables. The statement X = Y can be interpreted in different ways. Everything said about statement X = Y can be said about the equivalent statement X - Y = 0, and vice-versa. Hence, without loss of generality, we discuss the meaning of the statement random variable X = 0. X 0 Identically The statement X 0 identically means that the numerical value of X() = 0 for all S. This is a very restrictive form of equality, one that is rarely needed in applications. Hence, we seek a “looser” interpretation of statement X = 0. X = 0 Almost Surely (a.s.) Means P[X = 0] =1 The statement X = 0 almost surely (a.s.) means P[X = 0] = P[{ S : X = 0}] = 1. Often, this condition is stated as 1) X = 0 almost everywhere (a.e.) 2) X = 0 with probability one, both equivalent phrases (used by different authors). It should be noted that X = 0 (a.s.) is NOT equivalent to X 0 (i.e., X = 0 for all S, or everywhere). If X = 0 (a.s.), the event B = { S : X 0} has probability zero, however it can be nonempty. E[X2] = 0 is Equivalent to P[X = 0] = 1 (Same as X = 0 (a.s.)) In words, E[X ] = 0 is stated as X = 0 in mean square, or more simply, X = 0 (m.s.). E[X ] = 0 is equivalent to P[X 0] = 0. To prove this, we show E[X = 0 if, and only if, P[X 0] = 0. First, we show the “if” part: assume P[X 0] = 0. Then, X is a discrete random variable with all probably concentrated at the origin; that is, its distribution function is F(x) = U(x), a unit step. Observe that 2 dF x 2 dx x 2(x) dx 0 dx E[ X ] Updates at http://www.ece.uah.edu/courses/ee385/ (11-106) 11-33 EE603 Class Notes 12/05/13 John Stensby Second, we show the “only if” part: assume E[X = 0. With this, use the Generalized Tchebycheff Inequality (see Chapter 2 of these class notes) to write 2 E[ X ] 2 P X 2 1/ N N E[ X ] 0 1/ N (11-107) for each integer N > 0. Now, note that N 2 2 P X 0 P X 0 = P {X 1/ n} P limit {X 2 1/ n} . n 1 N n 1 (11-108) But, as indexed on n, sequence of events {X 2 1/ n} is nested increasing. Use continuity of probability and (11-107) to write N N P X 0 P limit {X 2 1/ n} limit P {X 2 1/ n} limit P X 2 1/ N N n 1 N n 1 N (11-109) 0. Equations (11-106) and (11-109) lead to the conclusion that 2 E[ X ] 0 if, and only if, P[X 0] = 0 (same as X = 0 (a.s.)). (11-110) It is worth repeating that P[X = 0] = 1 is not the equivalent to the statement X 0 for all S. Subspace of L2 M is said to be a subspace of L2 if it is a valid vector space (i.e., closed under scalar multiplication and vector addition in addition to the other requirements given in any linear algebra text) and M L2. Subspaces play a crucial role in many applications that involve Updates at http://www.ece.uah.edu/courses/ee385/ 11-34 EE603 Class Notes 12/05/13 John Stensby optimization problems. Inner Product and Norm It is natural to define an inner product on L2 as the expected value of a product. That is, for any X L2 and Y L2, we denote the inner product (dot product or scalar product) as X,Y, and we define X, Y E[XY] . (11-111) The Cauchy-Schwarz inequality (11-102) implies that X, Y X, X Y, Y . (11-112) That is, the inner product exists as a real number for every vector X and Y in L2. It can be shown that inner product X,Y = E[XY] satisfies the properties 1. X, X 0, and X, X 0 if and only if X = 0 almost surely (i.e., P[X = 0] = 1), 2. X,Y Y,X and (11-113) 3. cX,Y c X,Y , where c R. If E[X] = 0, then second moment X, X E[X 2 ] is the variance of random variable X. Random variables X and Y are said to be orthogonal if X, Y E[XY] 0 . In Part 1) of (11-113), the statement “X = 0 almost surely” is not equivalent to X (i.e., X identically zero); so, X, X 0 is not equivalent to X However, the equivalence of X, X 0 and X 0 is a general requirement of an inner product, as defined in almost all linear algebra books. However, in the applications literature, this subtle “issue” is overlooked, and (11-111) is declared a valid inner product. Updates at http://www.ece.uah.edu/courses/ee385/ 11-35 EE603 Class Notes 12/05/13 John Stensby Some authors change how random variables are defined/interpreted in an attempt to remove the phrase “almost surely” from Part 1) of (11-113) and “fix” the above-mentioned “issue”. They interpret a given X as a class of all random variables that are equal to X almost surely. Two members of the same class can differ on some set B as long as P[B] = 0. All class members will have the same expected value; sets of probability zero do not influence expectations. When computing X, Y E XY , X represents any member from its class, as does Y; the expectation will be the same regardless of which class members are used. Interpreting a random variable as a class of equivalent random variables allows us to “fix” Part 1 of (11-113), removing the phrase “almost surely”. In terms of equivalent classes, the statement “X = 0” refers to a class of random variables, all of which are zero almost surely. On a vector space, a vector norm maps vectors into real numbers in a manner that adapts the concept of length to vectors. Almost universally, the norm of vector X is denoted as X . On vector space L2, we define the norm of X as X, X E[X 2 ] X (11-114) (we say that the inner product induces the norm). From (11-113) it follows directly that (11-114) satisfies 1. X 0. X 0 if, and only if, X = 0 almost surely (i.e., P[X = 0] = 1), 2. cX c X , for any c R and (11-115) 3. X+Y X Y (the triangle inequality). If E[X] = 0, then X is the standard deviation of X. In Part 1) of (11-115), P[X = 0] = 1 is not equivalent to X (i.e., X identically zero); so, X 0 is not equivalent to X However, the equivalence of X 0 and X 0 is a Updates at http://www.ece.uah.edu/courses/ee385/ 11-36 EE603 Class Notes 12/05/13 John Stensby general requirement of a vector norm (see any text on linear analysis). However, in the applications literature, this subtle “issue” is overlooked, and (11-114) is declared to be a valid vector norm. This “problem” can be “fixed” by interpreting each random variable as a class, as discussed above. Often, norm (11-114) is called the mean-square norm since it involves the mean of the square of a random variable. In terms of (11-114), we can restate the Schwarz inequality as X, Y X Y . (11-116) Equation (11-116) is how the Schwarz inequality is usually stated in the analysis literature where the notions of inner product and norm play central roles. The triangle inequality (Part 3 of (11-115)) has a form similar to the well-known triangle inequality for real numbers (which states that r1 r2 r1 r2 for any real numbers r1 and r2 This inequality follows from the observation X+Y 2 2 X + Y, X + Y X, X 2 X, Y Y, Y X 2 X Y Y 2 (11-117) X Y 2 which leads to the triangular inequality X + Y X Y . The norm (11-114) allows us a way to define the equality of two vectors (random variables). If X and Y are random variables for which X-Y 0 (11-118) we say that X = Y in the mean-square sense, or we say X = Y (m.s.). From (11-110), we see that (11-118) is equivalent to P[X = Y] = 1 and P[X Y] = 0. Updates at http://www.ece.uah.edu/courses/ee385/ 11-37 EE603 Class Notes 12/05/13 John Stensby Convergence of Random Sequences Often, one has to deal with sequences of random variables that converge to a random variable. We say that the random sequence X(n;) converges to random variable X0() if for every fixed 0 S the sequence of numbers X(n; ) converges to the number X0(). This is "ordinary", sometimes called point-wise, sequence convergence (a topic that is usually covered in a Calculus course) that has nothing to do with the fact that we are dealing with random variables. Also, it is very restrictive. In applications, we can get by with much "weaker" modes of convergence; we discuss three alternative convergence modes. In what follows, we discuss almost sure (a.s.) convergence, convergence in probability (i.p.) and mean-square (m.s.) convergence. Mean square convergence is convergence in the mean-square norm (11-114). We discuss m.s. convergence first. Mean-Square Convergence (m.s. Convergence) As n goes to infinity, a sequence of random variables X(n) L2 converges in m.s. to a random variable X0 L2 if ½ limit X 0 - X(n) 0 same as limit E[{X0 - X(n)}2 ] = 0 . n n (11-119) The norm used in (11-119) is the mean-square norm given by (11-114). Often, this type of convergence is denoted as m.s (11-120) l.i.m X(n) = X 0 , (11-121) X(n) X0 , or n Updates at http://www.ece.uah.edu/courses/ee385/ 11-38 EE603 Class Notes 12/05/13 John Stensby where l.i.m denotes limit in the mean. Example 11-9: Let Z be a random variable with E Z2 (i.e., Z L2 ). .Let cn, n 0, be a sequence of deterministic real numbers converging to real number c. Then, cnZ, n 0, is a sequence of random variables. We show that l.i.m c n Z cZ . (11-122) n To see this, consider 2 2 2 2 2 E c n Z cZ E cn c Z cn c E Z . Now, cn c and EZ2] < implies EcnZ - cZ2] 0, and this proves (11-122). Example 11-10: Consider the probability space (S,B,P), where S = [0, 1], B the Borel sets (B is the -algebra generated by the open intervals on S. See Chapter 1 of class notes), and P B d , B B (11-123) B (if B is an interval, then P[B] is the interval length. P can be thought of as a “generalized length” of event B). Consider the sequence of random variable defined by Fig. 6: Sequence of random variables Updates at http://www.ece.uah.edu/courses/ee385/ 11-39 EE603 Class Notes 12/05/13 John Stensby 0 1 X(n; ) 1, n 0, 1 n (11-124) 1, as illustrated by Fig. 6. This sequence has a point-wise limit given by limit X(n; ) 1, 0 n (11-125) 0, 0 This sequence has zero as its mean-square limit since limit n X(n; ) 0 2 limit E[X 2 (n; )] limit [1 1 ] 0 n n n (11-126) Theorem 11-5: Mean-square convergence is additive. That is, if X 0 l.i.m X(n) n (11-127) Y0 l.i.m Y(n) , n then for any real-valued constants a and b we have aX 0 bY0 l.i.m aX(n) bY(n) . n (11-128) Proof: Note that Updates at http://www.ece.uah.edu/courses/ee385/ 11-40 EE603 Class Notes 12/05/13 John Stensby {aX(n) bY(n)} {aX 0 bY0 } a{X(n) X 0 } b{Y(n) Y0 } a{X(n) X0 } b{Y(n) Y0 } (11-129) = a X(n) X 0 b Y(n) Y0 . However, Equation (11-127) ensures that the right-hand-side of (11-129) approaches zero as n approaches infinity, and this proves (11-128). Not every sequence of random variables has a mean square limit. We need tools and techniques for determining if a sequence has a mean-square limit. Fortunately, our intuition is helpful in this regard. Also helpful is some knowledge of real number sequences. Recall that real number sequences have the Cauchy property. This property states that a real number sequence {rn} converges if, and only if, rn - rm 0 as both n and m approach infinity. When equipped with the Euclidean norm, the set of real numbers is complete, we say. Similarly, sequences in L2 have the Cauchy property. This property states that a sequence X(n) L2 converges (in the mean square norm) if, and only if, X(n) X(m) 0 as both n and m approach infinity. When equipped with the mean square norm, the set of L2 random variables is complete, we say. Stated again, a random sequence X(n) L2 has a mean-square limit X0 if, and only if, it is Cauchy (that is, X(n) X(m) 0 as both n and m approach infinity). Mean-Square Cauchy Sequences and Completeness Let X(n), n 0, be a sequence in L2. The sequence is said to be a mean-square Cauchy sequence if limit X(n) - X(m) = 0 . n,m (11-130) More tersely, we say that the sequence is m.s. Cauchy if (11-130) is true. For a m.s. Cauchy sequence, the quantity X(n) X(m) approaches zero as n and m approach infinity, in any manner whatever. Basically, the further you “go out” in a mean-square Cauchy sequence the Updates at http://www.ece.uah.edu/courses/ee385/ 11-41 EE603 Class Notes 12/05/13 John Stensby "closer" (in the mean-square sense) the elements become. It is easy to show that mean-square convergence implies the mean-square Cauchy property (i.e., (11-120) implies (11-130)). Actually, this is true for arbitrary normed vector spaces (i.e., all convergent sequences are Cauchy sequences, regardless of the normed vector space under consideration). However, for the general normed vector space, Cauchy sequences are not necessarily convergence. But, for L2 space equipped with the mean-square norm, the mean-square Cauchy property implies mean square convergence. This is stated by the following theorem. Theorem 11-6 (Special Case of Riesz-Fischer Theorem) Vector space L2 is complete in the sense that a mean-square Cauchy sequence has a unique limit in L2. That is, for sequence X(n) in L2, there exists a unique element X0 L2 such that limit X 0 - X(n) 0 n denoted symbolically as l.i.m X(n) = X 0 n (11-131) denoted symbolically as l.i.m [X(n) - X(m)] 0 . n,m (11-132) if limit X(n) - X(m) = 0 n,m Since the converse is true (see paragraph before the theorem statement), (11-131) and (11-132) are equivalent for vector space L2. In (11-132), one must remember that the double limit is zero regardless of how n and m approach infinity. The value of Theorem 11.6 is this: we do not have to know/find the m.s. limit of a sequence to know that the sequence is m.s. convergent. To show that L2 sequence X(n) converges to some m.s. limit X0, we need not know/find X0. Instead, to show convergence, it is Updates at http://www.ece.uah.edu/courses/ee385/ 11-42 EE603 Class Notes 12/05/13 John Stensby sufficient to show that X(n) has elements that come arbitrarily close to one another as you “go out” in the sequence. In some cases, establishing (11-132) is much easier than finding X0 described by (11-131). With the introduction of Theorem 11.6, we have established L2 as a complete vector space with norm (11-114) that is induced by inner product (11-111). In the literature, such vector spaces are referred to as Hilbert Spaces. They are the natural setting for many significant problems in Fourier series, communication theory, optimal filtering, etc. Mean-square convergence has a number of useful properties. We discuss the ability to interchange l.i.m and expectation. Also, we show that a mean-square limit is unique (with equality in the mean-square sense). To develop these results, we must mention some (almost) obvious, facts. Note that l.i.m X(n) (11-133) n is a random variable, but limit E[X(n)] (11-134) n is an "ordinary" limit of an "ordinary" sequence. Also, for any random variable X in L2, we have 2 E[X] E X E X 1 E X 12 X . (11-135) The first inequality results from the fact that the absolute value of an integral is less than, or equal to, the integral of the absolute value. The second inequality comes from the CauchySchwarz inequality (11-102) with Y = 1. Now, we show that we can interchange expectation and l.i.m. Updates at http://www.ece.uah.edu/courses/ee385/ 11-43 EE603 Class Notes 12/05/13 John Stensby Theorem 11-7: Let X(n) be a sequence in L2. Suppose X(n) has a m.s. limit X0 L2 that is, m.s X(n) X0 limit X(n) - X 0 0 . n (11-136) Then it follows that E[X 0 ] E l.i.m X n limit E[X(n)] . n n (11-137) That is, expectation and l.i.m are interchangeable. Proof: Since L2 is complete, mean-square limit X0 is in L2 (X0 has a finite second moment), so E[X0] exists (i.e., the mean is finite). Now, from (11-135), we have E[X(n)] E[X 0 ] E[X(n) X 0 ] E X(n) X 0 X(n) X 0 . (11-138) However, from (11-136) we know that the norm on the right-hand side of (11-138) goes to zero as n approaches infinity. Hence, we have the desired result (11-137). An important use of Theorem 11-7 deals with interchanging expectations and summations. For k = 1, 2, … , let Xk L2 be a sequence of random variables with finite second moments. Define the nth partial sum Yn n Xk . (11-139) k 1 Suppose that Updates at http://www.ece.uah.edu/courses/ee385/ 11-44 EE603 Class Notes Y l.i.m Yn l.i.m n 12/05/13 John Stensby n Xk . (11-140) n k 1 We say that partial sum (11-139) converges in mean square to Y. By Theorem 11-7, we can write n n E Y E l.i.m Yn E l.i.m X k limit E[X k ] E[X k ] . n k 1 n k 1 n k 1 (11-141) Theorem 11-8: The mean-square limit of a sequence is unique. That is, if X 0 l.i.m X(n) n Y0 l.i.m X(m) m limit X 0 - X(n) 0 n , (11-142) limit Y0 - X(m) 0 m then X 0 Y0 0 and P[X0 = Y0] = 1. Proof: Observe that X 0 - Y0 = {X 0 - X(n)} + {X(n) - Y0} < X 0 - X(n) + X(n) - Y0 (11-143) from the triangle inequality. Now, on the right-hand side of (11-143), both norms go to zero as a consequence of (11-142). Hence, we have X 0 Y0 0 as claimed. The fact that P[X0 = Y0] = 1 follows immediately from (11-110). Example 11-11: We are trying to sample a DC voltage (for example, the output of a strain gauge, water tank level detector, etc.). However, our samples contain additive noise; the kth sample is Y(k) = mdc + (k), where mdc is the DC voltage we are trying to measure, and (k) is a real-valued sample of stationary, zero mean noise with variance 2. We assume that (k) is Updates at http://www.ece.uah.edu/courses/ee385/ 11-45 EE603 Class Notes 12/05/13 John Stensby uncorrelated from sample to sample (any two different-indexed samples are uncorrelated). We try the “time-honored” technique of averaging out the noise. That is, we form the average X(n) = 1 n Y(k) . n k=1 (11-144) Note that X(n) has mdc as its mean and 2/n as its variance (indeed, with increasing n, we are “averaging out” the noise). However, the question remains: As n , does the random sequence X(n) L2 converge in mean square to a random variable? Let’s see if the sequence is mean-square Cauchy; consider X(m) - X(n) 2 = E [{X(m) - mdc }-{X(n) - mdc }]2 E {X(m) - mdc }2 - 2{X(m) - mdc }{X(n) - mdc }+ {X(n) - mdc }2 (11-145) 2 2 . 2E {X(m) mdc }{X(n) - mdc } m n Consider the case n > m and use the fact that the noise is uncorrelated from sample to sample to evaluate the middle term E {X(m) mdc }{X(n) mdc } E {X(m) mdc } {X(m) mdc }+ {X(n) - X(m)} E {X(m) mdc }2 E X(m) mdc E X(n) X(m) (11-146) 2 00 m Similarly, note that E[{X(m) - mdc}{X(n) - mdc}] = 2/n for the case m > n. Therefore, we can write (11-145) as Updates at http://www.ece.uah.edu/courses/ee385/ 11-46 EE603 Class Notes X(m) - X(n) 2 12/05/13 2 1 1 = 2 + . m min{n, m} n John Stensby (11-147) As m and n approach infinity (in any order), (11-147) approaches zero, so the sequence is meansquare Cauchy. By Theorem 11-6, the sequence is mean-square convergent. But what is its limit? The obvious “candidate” is mdc. To see that this is the limit, consider 1 n (k) limit 0. n n k 1 n n limit X(n) mdc limit n (11-148) So, we see that X(n) converges in mean square to mdc (and we can expect to get “better” results the more samples are included in the average). With Example 11-10, we have established a Mean-Square Law of Large Numbers for sequences of uncorrelated random variables. More general, let Yk, k = 1, 2, … , be a sequence of uncorrelated random variables with common mean E[Yk] = m and common variance VAR[Yk] = 2. Then the sample mean X(n) = 1 n Y(k) n k=1 (11-149) converges in mean square to m. In a subsequent section, we will show that (11-149) converges to m in probability, a yetto-be-defined mode of convergence that is weaker than mean-square convergence. That sample mean (11-149) converges in probability to m is just the well-known and popular Law of Large Numbers, (weak version) that is cited often in the popular press. Example 11-12: Let X(k), k 1, be a sequence of independent random variables each of which is either 1 or 0. Furthermore, suppose that Updates at http://www.ece.uah.edu/courses/ee385/ 11-47 EE603 Class Notes 12/05/13 John Stensby P[X(k) = 1] = 1/k . (11-150) P[X(k) = 0] = 1-1/k As k , does X(k) converge in mean square? Let’s check the obvious candidate X = 0; consider limit X(n) - 0 limit 1/ n 0 . n (11-151) n So, we see that X(n) converges in mean square to the random variable X = 0. However, in Example 11-16, we will see that X(n) does not converge (to zero) in a point-wise manner. Example 11-13: Let X(k), k 1, be a sequence of independent random variables similar to the previous example. However, suppose that X(k) is either k or 0 with P[X(k) = k] = 1/k 2 . (11-152) P[X(k) = 0] = 1-1/k 2 So, as k becomes large, we see that X is getting larger with a smaller probability. Is X(k) mean square convergent? To find out, consider X(m) - X(n) 2 = E X(m)2 - 2X(m)X(n) + X(n)2 , (11-153) = 2[1-1/nm] a result that converges to 2 as m, n approach infinity. Hence, X(n) is not mean-square Cauchy; hence, it is not mean square convergent. The last two examples illustrate the fact that meansquare convergence depends on both the numerical values a sequence takes on and the probabilities of taking on those values. Updates at http://www.ece.uah.edu/courses/ee385/ 11-48 EE603 Class Notes 12/05/13 John Stensby Theorem 11-7 tells us that expectation and l.i.m. are interchangeable for m.s. convergent sequences. A similar result holds for the inner product operation defined by (11-111). Theorem 11-9 (Continuity of the Inner Product): Let X(n) and Y(m) be m.s. convergent sequences with m.s. limits X0 and Y0, respectively, so that X 0 l.i.m X(n) n Y0 l.i.m Y(m) m limit X 0 - X(n) 0 n . (11-154) limit Y0 - Y(m) 0 m Under these conditions, we claim that X 0 , Y0 l.i.m X(n), l.i.m Y(m) limit X(n), Y(m) . n m n,m (11-155) Proof: First, consider the simple algebra X(n), Y(m) - X 0 , Y0 = X(n), Y(m) - X(n), Y0 + X(n), Y0 - X 0 , Y0 = X(n), Y(m) - Y0 + X(n) - X 0 , Y0 (11-156) X(n), Y(m) - Y0 + X(n) - X 0 , Y0 X(n) Y(m) - Y0 + X(n) - X 0 Y0 . m.s. X0 Now, since X(n) as n , the sequence X(n) is bounded (can you show this??), say X(n) < M. Use this fact, (11-154) and (11-156) to conclude limit n,m X(n), Y(m) - X0 , Y0 limit M Y(m) - Y0 + X(n) - X0 Y0 0 , n,m Updates at http://www.ece.uah.edu/courses/ee385/ (11-157) 11-49 EE603 Class Notes 12/05/13 John Stensby a result that proves (11-155) and the continuity of the inner product. Theorem 11-9 establishes continuity of the inner product X,Y E[XY]. What we mean by this is simple. Suppose we are given sequences X(n) and Y(m) with m.s. limits X0 and Y0, respectively, as described by (11-154). For “large” n and m, X(n) and Y(m) “get close” to X0 and Y0, respectively, and X(n),Y(m) E[X(n)Y(m)] “gets close” to X0,Y0 E[X0,Y0]. This intuitive idea is known as continuity of the inner product. Convergence in Probability (i.p. Convergence) Some results that involve mean square convergence of random sequences can be generalized to a "weaker" convergence mode. This new mode is called convergence in probability. It is "weaker" (i.e., more general) than m.s. convergence; m.s. convergent sequences also converge in probability, but the converse is not true. As n , a random sequence X(n) converges in probability (i.p.) to a random variable X0 if, for every > 0, we have limit P X(n) - X 0 0 . n (11-158) Often, this type of convergence is denoted by either of i.p. X(n) X0 (11-159) l.i.p X(n) X . (11-160) n For convergence in probability, many of the results parallel those given above for m.s. convergence. First, as we “go out” in a series (i.e., as the index becomes large), it may be more likely that the terms are closer together (this does not mean that the terms must be closer together in the m.s. sense). We say that a random sequence X(n) is Cauchy in probability if, for every > Updates at http://www.ece.uah.edu/courses/ee385/ 11-50 EE603 Class Notes 12/05/13 John Stensby 0, we have limit P X(m) - X(n) 0 . (11-161) n,m Cauchy in probability is a “weaker” condition than Cauchy in the mean square sense. A sequence that is mean square Cauchy is also Cauchy in probability, but the converse is not true. Condition (11-130) implies condition (11-161); however, the converse is not true. Next, we provide a theorem that does for convergence in probability what Theorem 11-6 did for convergence in mean square. Theorem 11-10: As n , a sequence X(n) converges in probability to a random variable X0 if, and only if, the sequence is Cauchy in probability. Proof: First, we show that if X(n) converges in probability to X0 then it is Cauchy in probability. Suppose that the sequence converges in probability. Then note the event (i.e., set) relationship X(m) - X(n) > ε X(m) - X0 > ε/2 X(n) - X 0 > ε/2 , (11-162) as depicted by Figure 11-7. From (11-162), we see that P X(m) - X(n) > ε P X(m) - X 0 > ε/2 P X(n) - X 0 > ε/2 . (11-163) Longer Than X0 X(n) X(m) Longer Than Figure 11-7: If X(n) - X(m) then either X(n) - X0 or X(m) - X0 Updates at http://www.ece.uah.edu/courses/ee385/ 11-51 EE603 Class Notes 12/05/13 John Stensby Now, since X(n) converges to X0 in probability, both terms on the right hand side of (11-163) approach zero as n and m approach infinity. Hence, the sequence is Cauchy in probability as claimed. The converse (if X(n) is Cauchy in probability then it converges in probability) is harder to prove and is not given here (see M. Loève, Probability Theory I, 4th Edition, pp. 117118). Theorem 11-11: If a sequence converges in probability, then the limit is unique. That is, suppose X(n) converges in probability to both X0 and Y0. Then it necessarily follows that P[X0 Y0] = 0. Proof: Using the same reasoning that led to (11-163), we can write X0 Y0 X0 - X(n) / 2 Y0 - X(n) / 2 (11-164) P X 0 Y0 P X 0 - X(n) / 2 P Y0 - X(n) / 2 . (11-165) However, both terms on the right-hand side of (11-165) approach zero as n approaches infinity. Hence, for every > 0 we have P X0 Y0 0 , (11-166) so that limit P X 0 Y0 0 . 0+ (11-167) Continuity of the probability measure (see Appendix 11B) and (11-167) lead to the conclusion P X 0 Y0 0 0 , Updates at http://www.ece.uah.edu/courses/ee385/ (11-168) 11-52 EE603 Class Notes 12/05/13 John Stensby and this establishes the claim that P[X0 Y0] = 0. As claimed previously, convergence in mean square implies convergence in probability. This claim is substantiated by the following theorem (which is a nice application of the Tchebycheff inequality). Theorem 11-12: Convergence in mean square implies convergence in probability. Proof: Let X(n) be a sequence that converges in mean square to the random variable X0. For each n, apply the generalized Tchebycheff inequality (see Chapter 2 of these notes) to X(n) - X0 and obtain 2 E X(n) - X 0 X(n) - X 2 0 P X(n) - X 0 ε = 2 2 ε ε (11-169) m.s. X 0 , so that X(n) - X 0 0 as n . for every > 0. However, we know that X(n) Hence, with (11-169), we have limit P X(n) - X 0 0 , n (11-170) i.p. so that X(n) X 0 as claimed. Let’s reconsider Examples 11-11 and 11-12, both of which provided sequences that converged in the mean square sense. Now, we know that these sequences converge in probability, as implied by Theorem 11-12. Actually, that the sequence in Example 11-11 converges in probability is just a statement of the Law of Large Numbers (weak version). Theorem 11-13 (The Weak Law of Large Numbers): Let X(n) be a sequence of independent, identically distributed (i.i.d) random variables with mean X and variance 2X . Then, the sample mean Updates at http://www.ece.uah.edu/courses/ee385/ 11-53 EE603 Class Notes ˆ n 12/05/13 John Stensby 1 n X(k) n k=1 (11-171) converges in probability to the “real” mean X as n approaches infinity. Proof: The proof of this theorem follows from Example 11-11 and Theorem 11-12. The law of large numbers is the basis for estimating X from measurements. In applications, it is common to take the sample mean (11-171) as an estimate of the “real” mean X. The basis for doing this is the Law of Large Numbers. Example 11-14: In Example 11-13, we considered a sequence of independent random variables X(k), k 1, with P[X(k) = k] = 1/k2 P[X(k) = 0] = 1 - 1/k2. We found out that this sequence does not converge in the mean square sense (a “sufficient number” of the sample function sequences contain a “sufficient number” of instances where X(k) = k so that m.s. convergence is not achieved). Now, we show that it does converge in probability to X0 = 0. For every > 0, we have limit P X(k) - X 0 limit P X(k) > limit P X(k) = k limit 1/ k 2 0 , k k k k (11-172) and we see that the sequence converges in probability to zero (in (11-172), only probabilities that X(k) = k are involved; for k 1, the actual numerical values of X(k) do not enter into the computation). The converse of Theorem 11-12 is not true (convergence in probability does not imply convergence in mean square), and Example 11-14 is a counter example that establishes this fact. Updates at http://www.ece.uah.edu/courses/ee385/ 11-54 EE603 Class Notes 12/05/13 John Stensby Basically, convergence in mean square is dependent upon both the numerical values of the sequence elements and the probabilities associated with the values. On the other hand, convergence in probability is only concerned with the probabilities. Example 11-15: For convergence in probability, this example shows that one cannot interchange the limit and expectation operations. For n 1, consider the sequence X(n), where X(n) is either = -1 or n. Also, suppose that P X(n) = n = 1/n P X(n) = -1 = 1-1/n . (11-173) The sequence converges in probability to X0 = -1 since limit P X(k) - X 0 limit P X(k) - (-1) k k limit P X(k) = k limit 1/k 0 k . (11-174) k Now, we look at mean values. Clearly, E[X0] = -1, and 1 1 1 E X(n) = n + (-1) 1- = , n n n (11-175) which has a zero limit as n approaches infinity. Hence, we have shown that limit E X(n) 0 E[ l.i.p X(n) ] E[X 0 ] 1 . n n (11-176) Example 11-15 serves as a counter example that shows that you cannot, in general, interchange the operations of limit in probability and expectation. That is, it is not generally true Updates at http://www.ece.uah.edu/courses/ee385/ 11-55 EE603 Class Notes 12/05/13 John Stensby that E l.i.p X(n) n and limit E X(n) n produce the same value (This differs from mean square convergence; recall that Theorem 11-7 proved that expectation and l.i.m are interchangeable). So, while convergence in probability is very general (and weak), there are limitations on what you can do with it. Convergence Almost Surely (a.s. Convergence) The last form of convergence we will study is called almost surely (a.s.) convergence. The random sequence X(n) converges almost surely to the random variable X0 if the sequence of functions X(n;) converges to X0() for all S except possibly on a set of probability zero (recall that S denotes the sample space). Almost surely convergence requires that P limit X(n) X 0 P S : limit X(n; ) X 0 ( ) 1 . n n (11-177) In other words, X(n) converges almost surely to random variable X0 if there exists an event A, _ with P(A) = 1 (and P(A) = 0), for which X(n;) X0() for all A. Often we write a.s. X(n) X0 . (11-178) Obviously, this type of convergence is “weaker” than pointwise (p.w.) convergence (p.w. Updates at http://www.ece.uah.edu/courses/ee385/ 11-56 EE603 Class Notes 12/05/13 John Stensby convergence requires that X(n;) X0() for all S). However, as shown below, almost surely (a.s.) convergence implies convergence in probability (i.p.). And, it doesn’t imply, nor is it implied by, convergence in mean square (m.s.). In the literature, a.s. convergence goes by the names convergence almost everywhere and convergence with probability one (other names are used as well). Like convergence in mean square and probability, in the context of almost sure convergence, it is possible to examine the separation, or distance, between sequence elements as we “go farther out” in a sequence. We say that X(n) is an almost surely Cauchy sequence if P limit X(n) X(m) 0 P S : limit X(n; ) X(m; ) 0 1 . n,m n,m (11-179) In other words, there exists an event A, P(A) = 1, for which limit X(n; ) X(m; ) 0 n,m (11-180) for all A. To establish that X(n) is an almost surely Cauchy sequence, we do not require knowledge of a sequence limit. With regard to necessary and sufficient conditions for the Cauchy criteria, almost surely convergence parallels m.s. and i.p. convergence. To show almost surely convergence of a sequence, it is not necessary to come up with a limit (in the almost surely convergent sense) for the sequence. Instead, as shown by the following theorem, we can use the Cauchy criteria. Theorem 11-14: A sequence X(n) is almost surely convergent if, and only if, it is an almost surely Cauchy sequence. Proof: This theorem follows from the fact that, in the real number system, sequences of real numbers converge if, and only if, they are Cauchy sequences. A practical and useful test for almost surely convergence is given by the following Updates at http://www.ece.uah.edu/courses/ee385/ 11-57 EE603 Class Notes 12/05/13 John Stensby theorem. Theorem 11-15: Let X(n) denote a sequence of random variables. Suppose that X(n) converges to random variable X0 almost surely; that is, we suppose that a.s. X(n) X0 . (11-181) Then, for every > 0 we have limit P X(n) - X 0 for all n m limit P { X(n) - X 0 } 1 , m m n=m (11-182) which we write as limit P A m 1 , (11-183) m where Am is defined as A m S : X(n; ) - X 0 () for all n m { S : X(n; ) X0 () } , (11-184) n m an event that depends on m and . The converse is true as well; hence, (11-182) and (11-181) are equivalent (i.e., one implies the other). Note: The sequence Am , m 0, is nested increasing with m; that is, Am Am+1 for all m and all > 0. Also, the complement of (11-184) is (DeMorgan’s Laws come in handy here) Updates at http://www.ece.uah.edu/courses/ee385/ 11-58 EE603 Class Notes 12/05/13 John Stensby A m S : X(n; ) - X 0 ( ) for some n m (11-185) { S : X(n; ) X0 () }. nm a.s. So, Theorem 11-15 is sometimes stated as: X(n) X 0 iff for all > 0 we have limit P X(n) X 0 for some n m limit P A m 0 . m (11-186) m a.s. Proof: First, suppose that X(n) X 0 . Then, there exists an event 1 for which P 1 1 P {S 1} P 1 0 (11-187) limit X(n; ) X0 () for each 1. n Now, show that 1 k 1 A k . Take any 0 . As shown by (11-187), X(n;0) converges in an “ordinary” sense to X0(0); this means that, given any > 0, there exists an integer m(0,) (integer m depends on 0 and with the property X(n, 0 ) X 0 ( 0 ) (11-188) for n m(0,). Hence, we see that 0 A k , for all k m(0,); that is, we can write 0 1 0 A k { S : X(n;) X0 () }, k m(0 ,) . (11-189) n=k Since the Ak are nested increasing, we have Updates at http://www.ece.uah.edu/courses/ee385/ 11-59 EE603 Class Notes 1 12/05/13 John Stensby Ak . (11-190) k=1 Since P(1) = 1, Equation (11-190) yields P Ak 1 . k=1 (11-191) This leads to the conclusion n n 1 P A k P limit A k limit P A k limit P(A n ) , n n k =1 k =1 k=1 n (11-192) a.s. and we have proven that (11-181), which states X(n) X 0 , implies (11-182), which states n P[A n ] 1 . Now, we show the converse; we show that (11-182) implies (11-181). We do this by showing that a false (11-181) implies a false (11-182) (this is the contrapositive of the statement “(11-182) implies (11-181)”). Hence, assume that (11-181) is false and show that limit P[A m ] 1 (i.e., (11-182) is false). If (11-181) is false there exists an event , P() > 0, m such that X(n,) / X0() for ( i.e., convergence does not occur for ) . Now, consider the random variable Z( ) lim sup X(n, ) X 0 ( ) , S . n (11-193) The event { S : Z() > 0} can be expressed as { S : Z( ) 0} { S : Z() 1/ n} . (11-194) n 1 Updates at http://www.ece.uah.edu/courses/ee385/ 11-60 EE603 Class Notes 12/05/13 John Stensby For each 0 , we have Z(0) > 0, so 0 { S : Z() > 0}; this fact implies that S : Z() 0. (11-195) Now, P() > 0 implies P ({ S : Z() 0 }) 0 and the existence of some integer n0 for which the event { S : Z() > 1/n0} has a strictly positive probability (to see this, equate the probability of both sides of (11-194) and use the continuity of P). That is, we have P[ { S : Z() > 1/n0 } ] > 0. (11-196) But, this positive probability event is contained in the complement of Am, m 1, defined using = 1/n0. This observation is written as { S : Z() > 1/ n0} A m { S : X(n, ) X0 () 1/ n0} , (11-197) n=m for every integer m (apply DeMorgan’s Law to (11-184) to get this complement). Hence, for every integer m, we have P(A m ) P( { S : Z() ) 0 , (11-198) so that P(A m ) is bounded away from zero, and P(Am) is bounded away from unity, as m . Hence, Equation (11-183) (equivalently, Equation (11-182)) cannot be true; we have shown that a false (11-181) implies a false (11-182) (equivalently, we have shown that (11-182) implies (11-181)). Updates at http://www.ece.uah.edu/courses/ee385/ 11-61 EE603 Class Notes 12/05/13 John Stensby Theorem 11-16: Almost surely (a.s.) convergence implies convergence in probability (i.p.). Proof: This is easy to show. Suppose that X(n) X0 almost surely (a.s.) so that P(A m ) 0 as m for any fixed (but arbitrary) > 0 used in the definition of Am. Note that { S : X(m, ) X 0 () } A m { S : X(n, ) X0 () } . (11-199) n=m Hence, P(A m ) 0 as m implies that P({ X(m) X 0 }) 0 as m , and we have X(m) X0 in probability (i.p.). Theorem 11-16 shows that a.s. convergence implies convergence in probability; however, the converse is not true, as shown by the next example. Example 11-16: This example shows that convergence in mean square (m.s.) does not imply convergence almost surely (a.s.). Recall that Example 11-12 discussed a binary random sequence X(k), all independent random variables, with P[X(k) 1] 1/ k . (11-200) P[X(k) 0] 1 1/ k In Example 11-12, we saw that X(k) converges in mean square (m.s.) to X0 0 (hence, it also converges in probability (i.p.) to X0 0). Now, we show that this sequence does not converge almost surely (a.s.). In terms of Am given by (11-184), observe that Updates at http://www.ece.uah.edu/courses/ee385/ 11-62 EE603 Class Notes 12/05/13 John Stensby limit P[A n ] limit P { X(m) - X 0 } limit P {X(m) = 0} n m n n n m n limit( 1 1 )( 1 1 ) n n limit n 1 ( 1 n 1m ) n m=0 (11-201) limit exp 1 n m n m=0 0. Since this limit is not unity, X(m) cannot converge almost surely to X0 = 0 (study again Equation (11-182)). What we have provided here is a counter example that shows that mean square (m.s.) convergence does not imply almost surely (a.s.) convergence. Also, the example shows that convergence in probability (i.p.) does not imply convergence almost surely (a.s.). Also, see Stark and Woods (3rd Edition), Example 6.7-3, p. 381 for a similar example. Example 11-17: This example shows that convergence almost surely (a.s.) does not imply convergence in mean square (m.s.). Recall that Example 11-13 presented a binary random sequence X(k) of independent random variables with P[X(k) k] 1/k 2 . (11-202) P[X(k) 0] 1-1/k 2 As shown by Example 11-13, this sequence is not mean square (m.s.) convergent. We show that X(k) converges almost surely (a.s.) to X0 = 0. In terms of A m defined by (11-199), observe that Updates at http://www.ece.uah.edu/courses/ee385/ 11-63 EE603 Class Notes 12/05/13 John Stensby limit P[A n ] limit P { X(m) - X 0 } limit P {X(m) m} n n n m=n mn (11-203) 1/ m2 0. n limit m=n Equivalently, in terms of An given by (11-184), this last result implies that limit P[A n ] limit P { X(m) - X 0 } 1 . n n m=n (11-204) From Theorem 11-15 (see Equation (11-182)), we can conclude that X(n) converges almost surely (a.s.) to X0 = 0. Together with Example 11-13, this example shows that convergence almost surely (a.s.) does not imply convergence in mean square (m.s.). Also, this example shows that convergence in probability (i.p.) does not imply convergence in mean square (m.s.). The next example is somewhat counter intuitive. It demonstrates that convergence point wise does not imply convergence in mean square, in general. Even though X (n; ) X 0 () for all S (i.e., the random variable converges point wise), the integral in the computation of E[X[n;] - X02] may diverge so that X(n) does not converge to X0 in mean square. Example 11-18: Consider the probability space (S,B,P), where S = [0, 1], B the Borel sets (B is the -algebra generated by the open intervals on S. See Chapter 1 of class notes), and P B d , B B (11-205) B (if B is an interval, then P[B] is the interval length. P can be thought of as a “generalized length” of event B). For S, define the random variable sequence Updates at http://www.ece.uah.edu/courses/ee385/ 11-64 EE603 Class Notes 12/05/13 John Stensby Figure 11-8: Relationship between modes of convergence. n, 1 2 n n X(n, ) n I[ 1 , 2 ] () n n 0, otherwise (11-206) (note that IB() is called the Indicator Function). On S, X(n) converges to zero in a pointwise p.w. manner. We say that X(n) 0 . Sometimes, we say that X(n) converges everywhere or surely. However, sequence X(n) does not converge to zero in the mean square sense since X(n) 0 2 2 2 1 E X(n) n 2 n . n n (11-207) Venn Diagram Describing Convergence Modes Figure 11-8 shows a Venn diagram that depicts the interrelationships between i.p., m.s., a.s., and p.w. convergence. The diagram follows directly from the definitions, theorems and counter examples given in this chapter. Mean square convergence neither implies, nor is it implied by, a.s. convergence; see Examples 11-16 and 11-17 for relevant counterexamples. The fact that p.w. convergence does not imply m.s. convergence is established by Example 11-18. Theorem 11-12 (alternatively, Theorem 11-16) establishes that m.s. (alternatively, a.s.) convergence implies i.p. convergence. Updates at http://www.ece.uah.edu/courses/ee385/ 11-65