Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
ME(EE) 550 Foundations of Engineering Systems Analysis Chapter Four: Inner Product Spaces The concept of normed vector spaces, presented in Chapter 3, synergistically combines the topological structure of metric spaces and the algebraic structure of vector spaces. Inner product spaces form a special case of normed spaces, where every inner product induces a unique norm; and the resulting geometry is largely similar to that in the familiar Euclidean spaces. Many familiar notions (e.g., root mean square value, standard deviation, least squares estimation, and orthogonal functions) in engineering analysis can be explained from the perspectives of inner product spaces. For example, the inner product of two vectors is a generalization of the familiar dot product in vector calculus. This chapter should be read along with Chapter 5 Parts A and B of Naylor & Sell. Specifically, both solved examples and exercises in Naylor & Sell are very useful. 1 Basic Concepts Definition 1.1. (Inner Product) Let (V, ⊕) be a vector space over a (complete) field F, where we choose F to be R or C. Then, a function h•, •i : V × V → F is defined to be an inner product if the following conditions hold ∀x, y, z ∈ V ∀α ∈ F: • Additivity: h(x ⊕ y), zi = hx, zi + hy, zi. • Homogeneity: hαx, yi = αhx, yi. • Hermiticity: hy, xi = hx, yi. • Positive Definiteness: hx, xi > 0 if x 6= 0V . A vector space with an inner product is called an inner product space. Example 1.1. Examples of Inner Product Spaces: P 1. Let V = Fn and x, y ∈ V . Then, nk=1 ȳk xk is an inner product, where the vector x ≡ [x1 , x2 , · · · , xn ]. 2. Let V = Fn×m and A, B ∈ V . Then, trace(B H A) is an inner product. P∞ 3. Let V = ℓ2 (F) and x, y ∈ V . Then, k=1 ȳk xk is an inner product. R 4. Let V = L2 (µ) and x, y ∈ V . Then, dµ(t) ȳ(t) x(t) is an inner product. 1 Remark 1.1. Let V be a vector space over a (complete) field F and x ∈ V . Then hx, xi ∈ R, because it follows from the homogeneity property in Definition 1.1 that hx, xi = hx, xi. Furthermore, it follows from the additivity and positive definiteness properties in Definition 1.1 that hx, xi ∈ [0, ∞) ∀x ∈ V . Remark 1.2. The following results are derived from Definition 1.1. • hy, αxi = ᾱ hy, xi. • hx, (y ⊕ z)i = hx, yi + hx, zi. • If hx, xi = 0, then then x = 0V . • If hx, yi = 0 ∀y ∈ V , then x = 0V . Lemma 1.1. (Cauchy-Schwarz Inequality) Let V be a vector space over a (complete) field F. Then, the following inequality holds: |hx, yi| ≤ kxkkyk ∀x, y ∈ V , p where kxk , hx, xi. The equality holds if and only if either x = αy or y = αx for some α ∈ F. Proof. Let x, y ∈ V and α = hx,yi hy,yi . Then, it follows that 0 ≤ h(x − αy), (x − αy)i = kxk2 − which completes the proof. |hx, yi|2 kyk2 Corollary 1.1. The equality part in Lemma 1.1 holds if and only if x and y are collinear, i.e., if either x = αy or y = αx for some α ∈ F. Proof. The proof follows by substituting x or y in the inequality in Lemma 1.1. Remark 1.3. Note that kx ⊕ yk = kxk + kyk or kx ⊕ yk = |kxk − kyk| if and only if x and y are collinear. p Lemma 1.2. Let V be a vector space and x ∈ V . Then kxk , hx, xi is a valid norm on the vector space V . Proof. It directly follows from Definition 1.1 that kxk satisfies the strict positivity and homogeneity properties of a norm. What remains to be shown is the triangular inequality property. By choosing arbitrary vectors x, y ∈ V and applying the Cauchy-Schwarz inequality in Lemma 1.1, it follows that kx ⊕ yk2 = h(x ⊕ y), (x ⊕ y)i ≤ kxk2 + 2kxkkyk + kyk2 = kxk + kyk 2 Lemma 1.3. (Continuity of inner products) Let (V, h•, •i) be an inner product space over a (complete) field F. Let {xk } and {y k } be two Cauchy sequences of vectors in V, converging to x ∈ V and y ∈ V , respectively. Then, the sequence of inner products {hxk , y k i} converge to the inner product hx, yi. 2 Proof. Subtracting and adding a term, using triangle inequality on scalars, and using the Cauchy-Schwarz inequality and the continuity property of a norm, it follows that |hxk , y k i − hx, yi| = |hxk , y k i − hxk , yi + hxk , yi − hx, yi| ≤ |hxk , (y k ⊕ (−y))i| + |h(xk ⊕ (−x)), yi| ≤ kxk kk(y k ⊕ (−y))k + k(xk ⊕ (−x))kky k k →0 because kxk k < ∞, ky k k < ∞, k(y k ⊕ (−y))k → 0, and k(xk ⊕ (−x))k → 0 as k → ∞. Remark 1.4. Every inner product space is a normed space but every normed space is not an inner product space. For example, the normed space ℓ2 is an inner product space but the normed space ℓp for p 6= 2 is not an inner product space. Given a normed space, how do we find out whether it qualifies as an inner product space? This issue is addressed in the following theorem. Theorem 1.1. (Parallelogram Equality) Let (V, h•, •i) be an inner product space. Let k • k be the norm induced by the inner product h•, •i. Then, for all x, y ∈ V , the following equality, known as the parallelogram equality holds: kx ⊕ yk2 + kx ⊕ (−y)k2 = 2kxk2 + 2kyk2 Proof. The proof follows by substituting hx, xi for kxk2 and performing algebraic operations. It is also a fact that the converse of Theorem 1.1 is true. That is, if V is a normed vector space and satisfies the parallelogram equality, then there is a unique inner product defined on the V , which generates the norm. In other words, V is an inner product space if and only if the norm satisfies the parallelogram equality. The issue is how to find the inner product from the knowledge of the norm. This issue is addressed in the following theorem. Theorem 1.2. (Polarization Identity) Let (V, k • k) be a normed vector space that satisfies the parallelogram equality. Then, the (unique) inner product h•, •i on V that satisfies the the relation hx, xi = kxk2 ∀x ∈ V is given as follows: 1. If the scalar field F over which the vector space is constructed is the real field R, then the inner product is constructed from the given norm as follows: 1 kx ⊕ yk2 − kx ⊕ (−y)k2 hx, yi = 4 2. If the scalar field F over which the vector space is constructed is the complex field C, then the inner product is constructed from the given norm as follows: 1 kx ⊕ yk2 − kx ⊕ (−y)k2 + ikx ⊕ iyk2 − ikx ⊕ i(−y)k2 hx, yi = 4 √ where i , −1. Proof. The proof follows by substituting hx, xi for kxk2 and performing algebraic operations. 3 Definition 1.2. A complete inner product space is called a Hilbert space. Remark 1.5. Every inner product space (V, h•, •i induces a unique norm in the vector space V defined over the field F. Since a norm is a valid metric in a vector space, every inner product space is a metric space. Therefore, the topology of an inner product space is the one generated by the metric: d(x, y) , kx ⊕ (−y)k = p h(x ⊕ (−y)), (x ⊕ (−y))i; for example, this is the usual topology in the Euclidean space Rn or the unitary space Cn . It is also seen in Chapter 1 that every metric space has an (essentially) unique completion; therefore, every inner product space has an (essentially) unique completion that leads to the construction of a unique Hilbert space from the given inner product space. In the next lemma, the inner product h•, •i is viewed as a (continuous) mapping of the product space (V, h•, •i)× (V, h•, •i) into the scalar field F. 1.1 Orthogonality in Inner Product Spaces The Hilbert space is a close cousin of the Euclidean space in the sense that the geometrical notions in these spaces are similar. The rationale for this similarity is largely due to the concept of orthogonality that is admissible in inner product spaces. Definition 1.3. (Orthogonality) Let (V, h•, •i) be an inner product space. Then, two vectors x, y ∈ V are said to orthogonal (to each other) if hx, yi = 0. Orthogonality of x and y is denoted as x ⊥ y. Two subsets (not necessarily subspaces) A, B ⊆ V are said to be orthogonal (to each other) if hx, yi = 0 for all x ∈ A and all y ∈ B; this is denoted as A ⊥ B. Lemma 1.4. (Pythagorean Theorem) Let (V, h•, •i) be an inner product space and let x, y ∈ V . If x ⊥ y, then kx ⊕ yk2 = kxk2 + kyk2 . Proof. kx ⊕ yk2 = h(x ⊕ y), (x ⊕ y)i = kxk2 + hx, yi + hy, xi + kyk2 = kxk2 + kyk2 because hx, yi = hy, xi = 0. Definition 1.4. (Direct Sum) Let V be a vector space and let U and W be two subspaces of V . Then, a subspace Y is the direct sum of U and W , denoted as Y = L U W , if ∀ y ∈ Y ∃ u ∈ U and w ∈ W such that there is a unique representation y = u ⊕ w; and U is called the algebraic complement of W (alternatively, W is called the algebraic complement of U ) in Y . Definition 1.5. (Orthogonal Complement) Let (V, h•, •i) be an inner product space and let ∅ ⊂ M ⊆ V . (Note that M may or may not be a vector space.) Then, the orthogonal complement of M in (V, h•, •i) is defined as M⊥ = x ∈ V : x ⊥ y ∀ y ∈ M i.e., M ⊥ is made of all vectors in V , which are orthogonal to each vector in M . An x ∈ M ⊥ is denoted as x ⊥ M . Remark 1.6. If M = {0V }, then M ⊥ = V and if M = V , then M ⊥ == {0V }. Also note that 4 M \ M ⊥ = ( {0V } ∅ if 0V ∈ M if 0V ∈ /M Theorem 1.3. (Closure Complement) Let (V, h•, •i) be an inner product space and let ∅ ⊂ M ⊆ V . Then, M ⊥ is a closed subspace of (V, h•, •i). Proof. It follows from Definition 1.5 that M ⊥ satisfies all conditions for a vector space because any linear combination of vectors orthogonal to M is also orthogonal to M . Therefore, M ⊥ is a subspace of V . What remains to show that M ⊥ is closed in (V, h•, •i). Let {xk } be a convergent sequence in M ⊥ and let xk → x̃. Since hxk , yi = 0 ∀ xk for any y ∈ M , it follows from continuity of inner products (see Lemma 1.3) that hx̃, yi = limk→∞ hxk , yi = 0. Therefore, the limit x̃ ∈ M ⊥ and so M ⊥ is a closed subspace of (V, h•, •i). Theorem 1.4. : Let (V, h•, •i) be an inner product space and let ∅ ⊂ S ⊆ V and ∅ ⊂ T ⊆ V . Then, the following statements are valid. 1. If S ⊆ T , then T ⊥ ⊆ S ⊥ and S ⊥⊥ ⊆ T ⊥⊥ . 2. S ⊆ S ⊥⊥ and S ⊥ = S ⊥⊥⊥ . T 3. If x ∈ S S ⊥ , then x = 0V . Hence, it follows taht ( \ {0V } if 0V ∈ S ⊥ S S = ∅ if 0V ∈ /S 4. {0V }⊥ = V and V ⊥ = {0V }. 5. If S is a dense subset of V , then S ⊥ = {0V }. Note: These results do not need completion of the inner product space (V, h•, •i). Proof. The proof is as follows. 1. Let x ∈ S ⊆ T and y ∈ T ⊥ , where both choices of x and y are arbitrary. Therefore, x⊥y and hence y ∈ S ⊥ . So, T ⊥ ⊆ S ⊥ . Applying this procedure once more on the derived condition: T ⊥ ⊆ S ⊥ , it follows that S ⊥⊥ ⊆ T ⊥⊥ . 2. Let x ∈ S, where the choice of x is arbitrary. So x⊥S ⊥ , which implies that x ∈ S ⊥⊥ . Hence, S ⊆ S ⊥⊥ . Then, by (1) it follows that S ⊥⊥⊥ ⊆ S ⊥ and also S ⊥ ⊆ (S ⊥ )⊥⊥ = S ⊥⊥⊥ . Hence, S ⊥ = S ⊥⊥⊥ . T 3. If x ∈ S S ⊥ , then x⊥x implying thar hx, xi = 0. That is, kxk = 0 or x = 0V . The remaining part follows in the same way from Definition 1.5. 4. The proof follows directly from Definition 1.5. 5. If x ∈ S ⊥ , then ∀ y ∈ S it follows from Pythagorean theorem that kx ⊕ (−y)k2 = kxk2 + kyk2 ≥ kxk2 Since S is dense in V , y can be chosen such that kx ⊕ (−y)k becomes arbitrarily small. Hence, kxk = 0. (Note that if S ⊥ = {0V }, then S is dense in V only if (V, h•, •i) is a Hilbert space.) 5 Theorem 1.5. Let (H, h•, •i) be a Hilbert space and let G be a subspace of H. Then, the following statements are valid. 1. G = G⊥⊥ , where G is the closure of G. 2. If G is closed in (H, h•, •i), then G⊥⊥ = G. 3. If G⊥ = {0V } if and only if G is dense in H. 4. If G is closed in (H, h•, •i) and if G⊥ = {0V }, then G = H. Note: These results do need completion of the inner product space (H, h•, •i). Proof. The proof is as follows. 1. It is known that G ⊆ G. It follows from Theorem 1.4 that G ⊆ G⊥⊥ , It also follows from Theorem 1.3 that G ⊆ G⊥⊥ . Now, if G 6= G⊥⊥ , then there is a nonzero vector z ∈ G⊥⊥ such that z⊥G.since G ⊆ G, it follows that z⊥G. T Therefore, z ∈ G G⊥ z = 0V . This is a contradiction. Hence, G = G⊥⊥ . 2. The proof follows directly from part (1) above. 3. The “if” part follows from Theorem 1.4 part (5). Now let G⊥ = {0V }. Then, G⊥⊥ = V and it follows from part (1) that G = V . Hence, G is dense in V . 4. This part follows from part (3) above. Theorem 1.6. Let (H, h•, •i) be a Hilbert space and let F and G be two closed L subspaces of H. If F ⊥ G, the direct sum F G is a closed subspace of H. Note: This result does need completion of the inner product space (H, h•, •i) L Proof. Let {zk } be a convergent sequence in F G with zk → z. we want to show L that z ∈ F G. That is, zk = xk ⊕ yk , where xk ∈ F and yk ∈ G. Since F ⊥ G, it follows from Pythagorean Theorem that kzk − zℓ k2 = kxk − xℓ k2 + kyk − yℓ k2 hence, {xk } and {yk } are Cauchy sequences in F and G , respectively. It follows L from continuity of addition that z = x ⊕ y. Therefore, F G is closed in V . Theorem 1.7. (Projection Theorem: Version 1) Let (H, h•, •i) be a Hilbert space and let G be a closed subspace of H. Then, L 1. H = G G⊥ . 2. Each x ∈ H can be uniquely expressed as x = y ⊕z, where y ∈ G and z ∈ G⊥ , and kxk2 = kyk2 + kzk2. Note: This result does need completion of the inner product space (H, h•, •i)] and closedness of G in H. L Proof. It follows from Theorem 1.6 that G G⊥ is a closed subspace of H. Since L L G ⊆ G G⊥ and G⊥ ⊆ G G⊥ , it follows from Theorem 1.4 and Theorem 1.5 L ⊥ that H = G G . See Naylor & Sell: Proof of Theorem 5.15.6 in Page 298 for more details. 6 1.2 Orthogonal Projection Definition 1.6. (Orthogonal Projection) A projection P on an inner product space is called an orthogonal projection if the range space and null space of P are orthogonal, i.e., R(P )⊥N (P ). (See Naylor & Sell (1982), p. 201 and p. 300.) Remark 1.7. Note that all projections in an inner product space need not be orthogonal. For example, in R2 with the inner product hx, yi , x1 y1 + x2 y2 , the projection matrix P = " 0 0 −1 1 # is linear and idempotent, i.e., yields P 2 = P , but its range space and null space are not mutually orthogonal. Remark 1.8. It follows from Definition 1.6 that if P is an orthogonal projection, then so is I − P . Theorem 1.8. (Continuity of Orthogonal Projection) An orthogonal projection in an inner product space is continuous. Proof. See Naylor & Sell: Proof of Theorem 5.16.2 in Page 300. Theorem 1.9. Let P be an orthogonal projection on an inner product space V . Then, 1. R(P ) and N (P ) are closed subspaces of V . 2. N (P ) = R(P )⊥ and R(P ) = N (P )⊥ . 3. Every vector x ∈ V can be uniquely expressed as x = r ⊕ n, where r ∈ R(P ) and n ∈ N (P ). 4. kxk2 = krk2 + knk2 . Proof. See Naylor & Sell: Proof of Theorem 5.16.3 in Page 301. Theorem 1.10. (Projection Theorem: Version 2) Let (H, h•, •i) be a Hilbert space and let G be a closed subspace of H. Then, 1. R(P ) and N (P ) are closed subspaces of V . 2. N (P ) = R(P )⊥ and R(P ) = N (P )⊥ . 3. Every vector x ∈ V can be uniquely expressed as x = r ⊕ n, where r ∈ R(P ) and n ∈ N (P ). 4. kxk2 = krk2 + knk2 . Proof. See Naylor & Sell: Proof of Theorem 5.16.2 in Page 300. 7 1.3 Optimization in a Hilbert Space Let us digress slightly from the main theme of this chapter and focus our attention on optimization in a Hilbert space, where the projection theorem characterizes the optimal solution. We present two slightly different versions of the theorem – one dealing with an arbitrary inner product space and the other providing a stronger solution in a Hilbert space. In this context, the problem is stated as follows. Let V be an inner product space and let W be a subspace of V . Given an arbitrary vector y ∈ V , the task is to identify a vector x̂ ∈ W closest to y in the sense that the error ky − x̂k is minimized. If y ∈ W , the the obvious solution is x̂ = y. Therefore, we deal with the situation y ∈ / W , where the following questions must be answered for a complete solution of the optimization problem. (i) (Existence) Does there always exist a vector x̂ ∈ W that minimizes the error k(y − x̂)k, or there is no x̂ ∈ W that is as least as good as all others? (ii) (Uniqueness) Is the solution unique? (iii) (Characterization) How the solution is characterized if it exists? Theorem 1.11. (Uniqueness of the minimizing vector) Let V be an inner product space and let W be a subspace of V . Given an arbitrary vector y ∈ V , if there exists a vector x̂ ∈ W such that k(y − x̂)k ≤ k(y − x)k ∀y ∈ V , then x̂ ∈ W is unique. A necessary and sufficient condition that x̂ be a unique minimizing vector in W is that the error vector (y − x̂) is orthogonal to W . Proof. First we apply a contradiction to show that if x̂ ∈ W is a minimizing vector, then the error vector (y − x̂) is orthogonal to W . Let us assume that there exists an x ∈ W \ {0H } that is not orthogonal to (y − x̂). It is noted that kxk > 0 and let ε , h(y−x̂),xi ; obviously, ε 6= 0. Let x̃ , x̂ + εx. Then, kxk ky − x̃k2 = ky − x̂ − εxk2 = ky − x̂k2 − h(y − x̂), εxi − hεx, (y − x̂)i + |ε|2 kxk2 ⇒ ky − x̃k2 = ky − x̂k2 − |ε|2 |xk2 < ky − x̂k2 Thus, if (y − x̂) is not orthogonal to W , then x̂ is not a minimizing vector. Next we show that if (y − x̂) is orthogonal to W , then x̂ is a unique minimizing vector. For any x ∈ W , the Pythagorean Theorem yields ky − xk2 = k(y − x̂) + (x̂ − x)k2 = ky − x̂k2 + kx̂ − xk2 Thus, ky − xk > ky − x̂k ∀x 6= x̂. Next we address the conditions for existence of the minimizing vector by strengthening the above theorem on uniqueness. Thus, we will satisfy the criteria for both existence and uniqueness of the minimizing vector. Theorem 1.12. (Existence and uniqueness of the minimizing vector) Let H be a Hilbert space and G be a closed subspace of H. Then, corresponding to any vector y ∈ H, there exists a unique vector x̂ ∈ G such that ky − x̂k ≤ ky − xk ∀x ∈ G. Furthermore, a necessary and sufficient condition that x̂ ∈ G be the unique minimizing vector is that (y − x̂) be orthogonal to G. 8 Proof. The uniqueness and orthogonality have been established in Theorem 1.11. What remains to be establish is existence of the minimizing vector x̂ ∈ G. If y ∈ G, then trivially x̂ = y; let y ∈ / G and δ , inf x∈G ky − xk. In order to identify an x̂ ∈ G such that ky − x̂k = δ, we proceed as follows. Let {xk } be a sequence of vectors in G such that limk→∞ ky − xk k = δ. Then, it follows by using the parallelogram law that k(xk − y) + (y − xℓ )k2 + k(xk − y) + (y − xℓ )k2 = 2 kxk − yk2 + ky − xℓ k2 A rearrangement of the above expression yields 2 (xk + xℓ ) k(xk − xℓ )k2 = 2kxk − yk2 + 2ky − xℓ k2 − 4 y − 2 (xk +xℓ ) (xk +xℓ ) Since the vector ∈ G ∀k, ℓ ∈ N, it follows that y − 2 ≥ δ. Therefore, 2 k(xk − xℓ )k2 ≤ 2kxk − yk2 + 2ky − xℓ k2 − 4δ 2 As ky − xk k2 → δ 2 and ky − xℓ k2 → δ 2 , it follows that limk.ℓ→∞ kxk − xℓ k2 = 0. Therefore, {xk } is a Cauchy sequence in G that is a closed (and hence a complete) subspace of the Hilbert space H. Therefore, {xk } converges to the limit point in G. Then, by continuity of the norm, it follows that the limit is x̂ ∈ G. Remark 1.9. Although the proof of the above theorem does not make any explicit reference to the inner product, it does make use of the parallelogram law that, in turn, makes use of the inner product for its proof. 2 Orthonormal Sets and Bases We have presented the notion of Hamel basis in Chapter 02 and Schauder basis in Chapter 03, which involve only algebraic notions. Now we incorporate both algebraic and topological notions in the formulation of a basis. Definition 2.1. (Orthogonal Set) Let I be an arbitrary nonempty index set (i.e., finite or countable or uncountable). A set {xα : α ∈ I} in an inner product space is called orthogonal if ∀α 6= β, xα ⊥xβ , i.e., hxα , xβ i = 0. In addition, if hxα , xα i = 1 ∀α ∈ I, i.e., hxα , xβ i = δαβ α, β ∈ I, then the set {xα } is called orthonormal. Note that δαβ , ( 1 if α = β 0 if α 6= β is called the Kronecker delta. Also note that any orthogonal set of non-zero vectors α can be converted into an orthonormal set by replacing xα with kxxα k ∀α. Lemma 2.1. Every orthonormal set of vectors in an inner product space is linearly independent. Proof. Let S = {xα } be an orthonormal set in an inner product space V . It suffices to show that, for every subset {xk ∈ S : k ∈ {1, 2, · · · , n} and n ∈ N, Pn k the only solution of the algebraic equation k=1 αk x = 0V is αk = 0 for all 9 k ∈ {1, 2, · · · , n}. Now, taking inner product on both sides with xj for all j ∈ {1, 2, · · · , n}, n DX E αk xk , xj = 0V , xj = 0 ⇒ αj = 0 ∀j. k=1 Definition 2.2. (Complete Orthonormal Set) Let I be an arbitrary nonempty index set. An orthonormal set B = {xα } in an inner product space V is called complete (or maximal) if the following condition holds x⊥xα ∀α ∈ I ⇒ x = 0V S In other words, no unit vector x ∈ V exists such that Big(B {x} is an orthonormal set. A complete orthonormal set in a Hilbert space V is called an orthonormal basis for V . Definition 2.3. (Orthonormal Basis) Let I be an arbitrary nonempty index set. An orthonormal set B = {xα } in an inner product space V is called complete (or maximal) if the following condition holds x⊥xα ∀α ∈ I ⇒ x = 0V S Equivalently, no unit vector x ∈ V exists such that B {x} is an orthonormal set. Remark 2.1. Every orthonormal set of vectors in in a Hilbert space can be extended to form an orthonormal basis. Furthermore, two orthonormal bases of a Hilbert space must have the same cardinality. 2.1 The Fourier Series Theory This subsection presents the fundamental properties of orthogonal bases by combining the topological and algebraic structures of a Hilbert space. Lemma 2.2. (Bessel Inequality) Let {xk } be an orthnormal set in an inner product space V . Then, for every x ∈ V , the following condition holds; X | hx, xk i |2 ≤ kxk2 k∈N 1 n Proof. Let {x , · · · , x } be a finite subset from {xk }. Then, by making use of the orthonarmality hxj , xk i = δjk , it follows that n 2 X hx, xj ixj 0 ≤ x − j=1 = = hx, xi − n X j=1 hx, xj ihxj , xi − n X k=1 D x− n X n E X hx, xk ixj hx, xj ixj , x − j=1 n X n X hx, xk ihxk , xi + j=1 k=1 k=1 hx, xj ihxj , xihx, xk ihxk , xi 2 = kxk − n X k=1 | hx, xk i |2 Since the above inequality is true for all n ∈ N, it must also hold true for a (countable) infinite sum. Hence, the proof is established. 10 Theorem 2.1. (Fourier Series Theorem) Let {xk } be an orthonormal set in a Hilbert space H. Then, the following statements are equivalent. (a) (Orthonormal basis) {xk } be an orthonormal basis of H. (b) (Fourier series expansion) Every x ∈ H can be expanded as k k k hx, x ix . P (c) (Parseval equality) ∀x, y ∈ H, the following relation holds; hx, yi = P k k hx, x i hy, xk i. (d) (Norm decomposition) The norm of every x ∈ H can be decomposed as: X kxk2 = |hx, xk i|2 < ∞. k (e) (Dense subspace of H) Let V be a subspace of H such that {xk } ⊂ V . Then, V is dense in H, i.e. closure(V )=H. Proof. Following Naylor & Sell (pp. 307-312), the proof is completed by showing the equivalence of individual parts in the following order: (a) ⇒ (b) ⇒ (c) ⇒ (d) ⇒ (a) and (b) ⇒ (e) ⇒ (b). Corollary 2.1. Let V be a closed subspace of a Hilbert space H, where V 6= {0H }; and let {xk } be an orthonormal basis of V . Then, the orthogonal projection P of P H onto V is given by P x = k hx, xk ixk ∀x ∈ H. Proof. The proof follows from Theorem 2.1. 2.1.1 Gram-Schmidt Procedure Let V be an inner product space and let {y k } be a linearly independent set with countable cardinality in V . The objective here is to construct an orthonormal sequence from the By making use of Bessel inequality in Lemma 2.2. Now we present the following theorem. Theorem 2.2. (Gram-Schmidt Orthonormalization) Let {xk } be an at most countable (i.e., finite or countably infinite) set of linearly independent vectors in an inner product space V . Then, there exists an orthonormal sequence {ek } ⊂ V such that, for every n ∈ N, Span[e1 , · · · , en ] = Span[x1 , · · · , xn ]. 1 Proof. First let us normalize the first vector x1 as e1 , kxx1 k that obviously spans the same space as x1 does. Next we construct z 2 , x2 − hx2 , e1 ie1 . Also note that z 2 6= 0V because x2 and x1 are linearly independent and Span[e1 , e2 ] = 2 Span[x1 , x2 ]. Now, let e2 , kzz2 k , which assures he2 , e1 i = 0 and he2 , e2 i = 1. Proceeding in this way, the remaining ek ’s are generated by induction as follows. P k Given z k , xk − j=1 k − 1hxk , ej iej and ek , kzzk k for all k ∈ {3, · · · , n}. Since hej , ek i = δjk ∀j, k ∈ {1, · · · , n}, we have hz n+1 , ek i = 0, i.e., z n+1 is orthogonal n+1 to Span[e1 , · · · , en ] = Span[x1 , · · · , xn ]. Next, define en+1 , kzzn+1 k . The proof by induction is thus completed. Example 2.1. (Orthogonal polynomials) Let T = (−1, 1) and let us consider the (linearly independent) sequence of vectors {tk : k ∈ N ∪ {0} and t ∈ T } on the inner product space L2 (T ) over the real field R, where the inner product is defined 11 R as hx, yi , T dt x(t)y(t) ∀x, y ∈ L2 (T ). Then, Gram-Schmidt orthonormalization of the sequence {tk } generates a set of orthonormal polynomials, called Legendre polynomials as seen below. √ R 0 Noting that h1, 1i = −1 1 dt = 2 ⇒ k1k = 2, we have e1 = ktt0 k = √12 . q R 2 z 2 , t − ht, e1 ie1 = t − 12 −1 1 t dt = t ⇒ e2 = kzz2 k = 23 t. q 3 1 2 z 3 , t2 − ht2 , √12 i √12 − ht2 , √32t i √32t = t2 − 13 ⇒ e3 = kzz3 k = 45 8 (t − 3 ). .. . There are many other families of orthonormal polynomials that can by gramSchmidt orthonormalization of the (linearly independent) sequence of vectors {tk : k ∈ N ∪ {0} and t ∈ T } on different inner product spaces. A few examples follow. • Legendre Polynomials on the space where the inner product is defined as: R hx, yi , T dt x(t) y(t), where T = (−1, 1). • Laugure Polynomials on the space where the inner product is defined as: R hx, yi , T dt exp(−t) x(t) y(t), where T = (0, ∞). • Hermite Polynomials on the space where the inner product is defined as: R hx, yi , T dt exp(−t2 ) x(t) y(t), where T = (−∞, ∞). • Chebyshev Polynomials on the space where the inner product is defined as: p R hx, yi , T dt (1 − t2 ) x(t) y(t), where T = (−1, 1). 2.1.2 Fourier Transform Now we introduce the concept of Fourier transform of (absolute) integrable functions, i.e., the functions belonging to the L1 (µ) space, where µ is the Lebesgue measure on the real line R. We note that, in the space Lp (µ) for p ∈ [1, ∞], Lebesgue measure µ is defined on any measurable subset of the real line R. In other words, R we consider functions f ∈ L1 (µ), where R dµ(t)|f (t)| < ∞, for Fourier transform. Definition 2.4. (Fourier transform) For f ∈ L1 (µ), the Fourier transform of f is: fˆ(ξ) , Z R dµ(t) exp (−i2πξt)f (t) ∀ξ ∈ R and for fˆ ∈ L1 (µ), i.e., f (t) , Z R R R Fourier Analysis Formula dµ(ξ)|fˆ(ξ)| < ∞, inverse Fourier transform of fˆ is: dµ(ξ) exp (i2πξt)fˆ(ξ) ∀t ∈ R Fourier Synthesis Formula Remark 2.2. The Fourier transform converts time translation into a phase shift and vice versa, and convolution into products as seen below. To see this, let f ∈ L1 (µ) and α, λ ∈ R. Then, 1. If g(t) = f (t) exp(i2παt), then ĝ(ξ) = fˆ(ξ − α) and if g(t) = f (t − α), then ĝ(ξ) = fˆ(ξ) exp(−i2παξ). 2. If g ∈ L1 (µ) and h = f ⋆ g, then ĥ(ξ) = fˆ(ξ)ĝ(ξ). Similarly, if ĝ ∈ L1 (µ) and ĥ = fˆ ⋆ ĝ, then h(t) = f (t)g(t). 3. If g(t) = f (−t), then ĝ(ξ) = fˆ(ξ). 12 4. If λ 6= 0 and g(t) = f ( λt ), then ĝ(ξ) = |λ|fˆ(λξ). 5. Let n ∈ N. If f is a C ∞ -continuous function, then Fourier transform of n ˆ ∂ n f (t) is (i2πξ)n fˆ(ξ); similarly, Fourier transform of (−i2πt)n f (t) is ∂ ∂ξf n(ξ) , ∂tn provided that the Fourier transforms exist. Rt fˆ(ξ) provided that fˆ(0) = 0. 6. Fourier transform of −∞ dµ(τ )f (τ ) is i2πξ R∞ n ˆ 7. Let n ∈ N ∪ {0}. Then, (−i2π)n −∞ dµ(τ )τ n f (τ ) = ∂∂ξnf , provided that ξ=0 the integral and the derivatives exist. Also note that if f ∈ L1 (µ) and fˆ(ξ) = 0 ∀ξ ∈ R, then f (t) = 0 almost everywhere ( µ-a.e.) on R. Next we introduce the notion of Plancherel Theorem. Since the Lebesgue measure µ of R is not finite (it is σ-finite), we cannot claim L2 (µ) to be a subspace of L1 (µ). Fourier transform in Definition 2.4 is not directly applicable to every T f ∈ L2 (µ). However, the definition does apply if f ∈ L1 (µ) L2 (µ) and it turns T out that fˆ ∈ L2 (µ). In fact, kfˆkL2 = kf kL2 . This isometry of L1 (µ) L2 (µ) into L2 (µ) extends to an isometry of L2 (µ) onto L2 (µ), and this defines an extension of the Fourier transform, which is often called as the Plancherel transform and is applicable to every f ∈ L2 (µ). The resulting L2 -theory has more symmetry than that in the L1 -theory. In other words, f and fˆ play the same role in L2 (µ). Theorem 2.3. (Plancherel Theorem) Each function f ∈ L2 (µ) can be associated to a function fˆ ∈ L2 (µ) such that the following properties hold: T 1. If f ∈ L1 (µ) L2 (µ), then fˆ is the the Fourier transform of f , i.e., R fˆ(ξ) = R dt exp (−i2πξt)f (t) ∀ξ ∈ R. 2. kfˆkL2 = kf kL2 for every f ∈ L2 (µ). 3. The mapping f 7→ fˆ is a Hilbert space isomorphism of L2 (µ) onto L2 (µ). 4. The following symmetric relation exists between f and fˆ: RA RA If ϕA (ξ) , −A dµ(t) exp(−i2πξt)f (t) and ψA (t) , −A dµ(ξ) exp(i2πξt)fˆ(ξ), then kϕA − fˆkL2 → 0 and kψA − f kL2 → 0 as A → ∞. Proof. See Real and Complex Analysis by Walter Rudin (pp.187-189). 2.1.3 Hilbert Dimension and Separable Hilbert Spaces Let us classify Hilbert spaces (over the same field) by cardinality of the orthonormal basis set. Let B1 and B2 be two orthonormal bases of a Hilbert space H. Since B1 and B2 must have the same cardinal numbers, this cardinal number is called the Hilbert dimension of H. For finite-dimension spaces H, the Hilbert dimension of H is the same as the dimension related to a Hamel base. However, for infinite-dimensional Hilbert spaces, the Hibert dimension could be countable or uncountable. An example of countable Hilbert dimension is the Hilbert space of periodic functions L2 [−π, π]. 13 Example 2.2. The Hilbert space ℓ2 is isometrically isomorphic to the Hilbert space L2 [−π, π]. That is, there exists a linear bijective mapping f : ℓ2 → L2 [−π, π] such that the inner products satisfy the following equality hx, x̃iℓ2 = hf (x), f (x̃)iL2 [−π,π] ∀x, x̃ ∈ ℓ2 To see this, we proceed as follows. Let {xk } be an orthonormal basis for P L2 [−π, π] and let f : ℓ2 → L2 [−π, π] be defined as: f (a) = k∈N ak xk We cite below an example of a Hilbert space having uncountable Hilbert dimension. Example 2.3. (Uncountable Hilbert dimension) Let us consider a trigonometric Pm polynomial p : R → C such that ∀t ∈ R p(t) , k=1 ck exp(irk t) for some m ∈ N, where r1 , · · · , rm ∈ R and c1 , · · · , cm ∈ C. Let p̃ : R → C be another Pm̃ trigonometric polynomial defined as ∀t ∈ R p̃(t) , k=1 c̃k exp(ir̃k t) for some m̃ ∈ N, where r̃1 , · · · , r̃m̃ ∈ R and c̃1 , · · · , c̃m̃ ∈ C. Then, it follows that p(t) p̃(t) = m X m̃ X j=1 k=1 cj c̃k exp i(rj − r̃k )t We define an inner product in the space trigonometric polynomials as 1 hp, p̃i , lim T →∞ 2T Z T dt p(t) p̃(t) = −T Z T −T cj c̃k j=1 k=1 because 1 lim T →∞ 2T m X m̃ X dt exp irt = ( 1 if r = 0 0 if r ∈ R \ {0} The completion of the inner product space of trigonometric polynomials is a Hilbert space called the space of almost periodic functions. Let us call this Hilbert space as HtrigP . For all r ∈ R let us consider the function ur (t) , exp(irt), t ∈ R. Then, {ur : r ∈ R} is an uncountable orthonormal set in HtrigP . Since the set of all trigonometric polynomials is dense in HtrigP , it follows that if p ∈ H and hp, ur i = 0 ∀r ∈ R, then kpk2 = hp, pi = 0 ⇒ p = 0H . Hence, {ur : r ∈ R} is an uncountable orthonormal basis for HtrigP . If {zn : n ∈ N} is a countable subset of HtrigP , then the open ball B √1 zn 2 √ contain at most one ur , because kur − us k = 2 for ∀ r 6= s. Hence, the collection of all open balls B √1 zn can cover only a countable number of ur ’s. Since the set 2 {ur : r ∈ R} has uncountable cardinality, no countable set can be dense in HtrigP . Hence, HtrigP is a non-separable Hilbert space. Theorem 2.4. (Separable Hilbert Spaces) Let H be a Hilbert space. Then, H is a separable Hilbert space (i.e., H has a countable subset that dense in H) if and only if every orthonormal basis of H is countable. Proof. To prove the only if part, let the Hilbert space H be separable, U ⊆ H be countable and dense in H, and M be an orthonormal basis of H. Then, for every x, y ∈ M , ( 0 if x = y √ kx − yk = 2 if x 6= y 14 Then, for ∀x, y ∈ U and x 6= y, the open neighborhoods N √1 (x) and N √1 (y) in H 2 2 are disjoint. Therefore, every N √1 (x) in H can have at most one element of M . 2 Since U is countable, M must be countable; else U cannot be dense in H. To prove the if part, let {ek } be a countable orthonormal basis of H and let A be a set of all (finite) linear combinations γ1n en + · · · + γnn en for all n ∈ N, where γjn , αnj + iβjn and αnj , βjn ∈ Q, the set of rational numbers. Note that the set A is countable and if H is defined over R, then set all βjn = 0. We will prove that A is dense in H by showing ∀x ∈ H ∀ε > 0 ∃ x̃ ∈ A such that kx − x̃k < ε Given any x ∈ H and ε > 0, let Yn be the space spanned by a selected subset of vectors {e1 , · · · en } from the orthonormal basis {ek } such that kx−yk < ε2 ∀y ∈ Yn . This is possible because {ek } is a complete ortonormal set in H. It follows, by orthogonal projection of x onto Yn , that n n ε X X y= hx, ek iek ⇒ x − hx, ek iek < 2 k=1 k=1 Pn Let x̃ = k=1 γkn ek be the orthogonal decomposition of x̃ ∈ A. Then, n X γkn ek kx − x̃k = x − k=1 n n n ε ε X X X γkn ek < + = ε hx, ek iek − ≤ x − hx, ek iek + 2 2 k=1 k=1 k=1 which proves that A is dense in H; and since A is countable, H is separable. Example 2.4. The spaces ℓ2 and L2 ([−π, π]), defined on the same field F, have the same Hilbert dimension. It is also true that ℓ2 and L2 ([−π, π]) are isometrically isomorphic as explained below. Let {xk } be an orthonormal basis for the Hilbert space L2 ([−π, π]) and let f : ℓ2 → L2 ([−π, π]) be defined as follows: f (a) , ∞ X k=1 ak xk , where a , {a1 a2 · · · } ∈ ℓ2 . Now we show that f is linear and bijective. Linearity is established as follows. f (αa + b) = ∞ X αa + b)k xk = α k=1 ∞ X ak xk + ∞ X bk xk = αf (a) + f (b) k=1 k=1 To establish injectivity of f , let f (a) = f (b). Since f and {xk } is orthonormal, it follows that ∞ ∞ ∞ 2 X X X 0 = kf (a) − f (b)k2 = (a − b)k xk = |(a − b)k |2 kxk k2 = |(a − b)k |2 k=1 k=1 k=1 which implies that a = b. To establish surjectivity of f and isometry of the spaces ℓ2 and L2 ([−π, π]), let P∞ x ∈ L2 ([−π, π]). Then, it follows from Fourier series theorem that x = k=1 hx, xk i. Setting ak = hx, xk i, it follows from orthonormality of {xk } that ∞ ∞ ∞ 2 X X X kxk2 = ak xk = |ak |2 kxk k2 = |ak |2 = kak2 k=1 k=1 k=1 15 which implies that, for each x ∈ L2 ([−π, π]), there exists an a ∈ ℓ2 such that f (a) = x and kakℓ2 = kxkL2 ([−π,π]) . 3 Applications of Hilbert Spaces Hilbert spaces have numerous applications in Science and Engineering. In modern Physics (that started in early 120’s), the mathematics of Quantum Mechanics is built upon the concept of Hilbert spaces over the complex field C. While there is abundance of literature in mathematics of Quantum Mechanics, Chapter 11 of Introductory Functional Analysis with Applications by E. Kreyszig provides a brief explanation of the underlying concepts. In this section, we provide a few examples of applications of Hilbert spaces in Signal Processing and Quantum Mechanics. 3.1 Shannon Sampling: Bandlimited Signals Sampling of continuous-time signal is a key step to discrete-time signal processing. In engineering practice, sampling is accomplished by analog-to-digital (A/D) conversion of continuously varying signals that are generated by various sensors. It is apparent that the sampling frequency should be selected depending on the bandwidth of the signal and the sensor that is used to measure the signal. For example, if the sampling is performed infrequently, the vital information content of the signal might be lost. On the other hand, a very high sampling frequency may generate excessively redundant data that would require additional memory and computation time. We present the analytical relationship between the signal bandwidth and the smallest sampling frequency for no loss of information. Theorem 3.1. (The Shannon Sampling Theorem) Let a (real-valued) signal f ∈ L2 (R) be band-limited by Ω, i.e., fˆ(ξ) = 0 ∀ ξ ∈ / [−Ω, Ω]. Then, f (t) can be perfectly k reconstructed from its samples f (tk ), collected at time instants tk , 2Ω , k ∈ Z, by the following interpolation formula: X sin 2πΩ (t − tk ) f (tk ) f (t) = 2πΩ (t − tk ) k∈Z Proof. By Plancherel theorem, the L2 -norm of Fourier transform is expressed as: Z Ω 2 ˆ k f kL2 (−Ω,Ω) = dξ | fˆ(ξ) |2 −Ω Z ∞ dξ | fˆ(ξ) |2 =k fˆ k2L2 (R) = −∞ =k f k2L2 (R) Expansion of fˆ ∈ L2 (−Ω, Ω) in a Fourier series yields 1 X ck exp − i 2πξtk ) fˆ(ξ) = 2Ω k∈Z where the Fourier coefficients ck are identically equal to the samples of the signal k , i.e., f (t) taken the time instants tk , 2Ω Z Ω ck = dξ exp i 2πξtk fˆ(ξ) = f (tk ) −Ω 16 It follows by equality in the L2 -sense and by exchange (due to uniform convergence of the summand) of the integral and the infinite sum that Z f (t) = dξ exp i 2πξtk fˆ(ξ) = Z R Ω −Ω dξ exp i 2πξtk fˆ(ξ) Z Ω 1 = dξ exp i 2πξtk 2Ω −Ω X × f (tk ) exp − i 2πξtk k∈Z 1 X = 2Ω k∈Z Since RΩ −Ω Z Ω −Ω dξ exp i 2πξ(t − tk f (tk ) dξ exp i 2πξ(t − tk = sin(2πΩ (t−tk )) 2πΩ (t−tk ) , it follows that X sin 2πΩ (t − tk ) f (t) = f (tk ) 2πΩ (t − tk ) k∈Z in the L2 -sense. As band-limited signals are analytic in the entire complex plane, it suffices that f (•) is continuous. Hence, the above equality holds pointwise everywhere in the domain of f (•). 3.2 Fourier Transform in Quantum Mechanics Let x be the position of a particle and the quantum wave function ψ(x) be a complex-valued function of x, which belongs to a Hilbert space over the complex field C. The state of a classical particle in one dimension is specified by both position x and momentum p while, in quantum physics, the wave function depends on only one of the two variables. The information on momentum p is contained in the wave function ψ(x). A convenient way to extract the information on p as a Fourier transform of ψ(x) is to construct the momentum wave function Z ∞ Z ∞ −ipx −i2πpx 1 1 ψ̂(p) , √ ψ(x) = √ ψ(x) dx exp dx exp ~ h 2π~ −∞ h −∞ where the Planck constant ~ = 1.054572 × 10−34 Joule-sec and the original Plank constant h = 2π~. The above Fourier transform can be inverted to generate the information on position x from ψ̂(p) as Z ∞ Z ∞ +ipx +i2πpx 1 1 ψ(x) , √ ψ̂(p) = √ ψ̂(p) dp exp dp exp ~ h 2π~ −∞ h −∞ Therefore, the position waveform ψ(x) and the momentum waveform ψ̂(p) contain the precisely the same information about a quantum state. By Plancherel Theorem, the inner product is obtained in either position domain or momentum domain as ! Z ∞ Z ∞ dp ψ̂(p)ϕ̂(p) hϕ, ψi = hϕ̂, ψ̂i ⇔ dx ψ(x)ϕ(x) = −∞ −∞ 17 In essence, the information about the momentum of a quantum particle can be obtained from the momentum wave function in the same way that information about its position can be obtained from the position wave function as explained below. The position wave function has a compact support, i.e., ψ(x) vanishes outside in a finite interval x1 ≤ x ≤ x2 ; similarly, momentum wave function has a compact support, i.e., ψ̂(p) vanishes outside a finite interval p1 ≤ p ≤ p2 . Therefore, it is safe to say that the quantum particle does not lie outside a position interval and a momentum interval. However, we may not be able to specify a deterministic value of the position and momentum within these intervals. This issue is further examined in the next section from the perspectives of both Quantum Mechanics and Signal Processing. 3.3 Heisenberg Uncertainty Principle and Signal Analysis Let us start with the statement that the use of ”uncertainty” in the so-called Uncertainty Principle is perhaps a misnomer, because there is nothing uncertain about the uncertainty principle. In the discipline of signal analysis, it is a well-known mathematical fact that A narrow-band waveform yields a wide spectrum and a wide-band waveform yields a narrow spectrum. It is possible that both time and frequency can be narrow, but the probability densities of time and frequency localization cannot be made simultaneously arbitrarily narrow, i.e., their variances cannot be made simultaneously arbitrarily small. In both physics and signal analysis, the uncertainty principle applies to a pair of variables whose associated operators do not commute. In the physics literature, Position Operator: χ≡ x ∂ h ∂ Momentum Operator: p≡ − i~ ∂x or − i 2π ∂x , The commutator of χ and p on an arbitrary (differentiable) function f (x) is defined as: ∂ ∂ f (x) + i~ xf (x) [χ, p] , χp − pχ ⇒ [χ, p]f (x) = x − i~ ∂x ∂x ∂f ∂f −x − f (x) = i~f (x) = −i~ x ∂x ∂x h ⇒ [χ, p] = i~ or [χ, p] = i 2π Similarly, in the signal analysis literature, i d Time Operator: ℑ≡ t , 2π dξ −i d Frequency Operator: Ξ ≡ ξ , 2π dt 18 The commutator of ℑ and Ξ on an arbitrary (differentiable) signal s(t) is defined as: −i d −i d s(t) − ts(t) 2π dt 2π dt ds i −i ds t − t − s(t) = s(t) = 2π dt dt 2π i ⇒ [ℑ, Ξ] = 2π [ℑ, Ξ] , ℑ Ξ − Ξ ℑ ⇒ [ℑ, Ξ] s(t) = t Alternatively, [ℑ, Ξ] , ℑ Ξ − Ξ ℑ ⇒ [ℑ, Ξ] ŝ(ξ) = 3.4 i d i d ξ ŝ(ξ)) − ξ ŝ(ξ) 2π dξ 2π dξ dŝ i dŝ i dŝ ŝ(ξ) + ξ = = −ξ ξ 2π dξ dξ 2π dξ i ⇒ [ℑ, Ξ] = 2π Time-frequency Localization R∞ Let us consider a unit energy signal s(t) ∈ L2 (R), i.e., −∞ dt | s(t) |2 = 1 and R∞ 2 2 −∞ dξ | ŝ(ξ) | = 1. In line with the concepts in Quantum Mechanics, | s(t) | and 2 | ŝ(ξ) | are the probability densities of time and frequency localization, respectively. By appropriate time translation and frequency modulation, it follows that Z ∞ Z ∞ 2 dt | s(t) | t = 0 and dξ | ŝ(ξ) |2 ξ = 0 −∞ −∞ without loss of generality. Let us define the so-called variances of time localization and frequency localization as: Z ∞ Z σt2 , dt | s(t) |2 t2 and σξ2 , −∞ ∞ −∞ dξ | ŝ(ξ) |2 ξ 2 Theorem 3.2. (Heisenberg Uncertainty for Time-frequency Localization) If the 1 1 signal s(t) ∈ L2 (R) decays to zero faster than |t|− 2 as t → ± ∞, then σt σξ ≥ 4π pα and the equality holds for Gaussian signals s(t) = π exp(−αt2 ). Proof. By Cauchy-Schwarz inequality Z ∞ Z Z ∞ 2 ∞ ds 2 dt t s(t) dt t s(t) dt ≤ dt −∞ −∞ −∞ ds 2 . dt = The first integral on the right is equal to σt2 .Taking Fourier transform of ds 2 R ∞ 2dt R∞ i2πξŝ(ξ) and by using Plancherel Theorem, i.e., −∞ dts(t) = −∞ dξ ŝ(ξ) , the second integral on the right is expressed as: Z ∞ ds 2 Z ∞ 2 dt = dξ i2πξŝ(ξ) = (2π)2 σξ2 dt −∞ −∞ R ∞ Therefore, −∞ dt t s(t) ds dt ≤ 2πσt σξ . Now, integration by parts yields ∞ ds 1 dt t s(t) = dt 2 −∞ Z 19 Z ∞ d |s(t)|2 dt t dt −∞ = Z ∞ 1 1 1 ∞ 1 t |s(t)|2 − dt |s(t)|2 = 0 − = − 2 2 −∞ 2 2 −∞ ⇒ σt σξ ≥ The Cauchy-Schwarz inequality becomes an equality if ts(t) and For Gaussian signals, r ds d α = exp(−αt2 ) dt dt π r α exp(−αt2 ) = −2αts(t) = −2αt π p 1 2 Hence, σt σξ = 4π for s(t) = α π exp(−αt ). 20 ds dt 1 4π are collinear.