Download Inner Product Spaces - Penn State Mechanical Engineering

ME(EE) 550
Foundations of Engineering Systems Analysis
Chapter Four: Inner Product Spaces
The concept of normed vector spaces, presented in Chapter 3, synergistically
combines the topological structure of metric spaces and the algebraic structure of
vector spaces. Inner product spaces form a special case of normed spaces, where
every inner product induces a unique norm; and the resulting geometry is largely
similar to that in the familiar Euclidean spaces. Many familiar notions (e.g., root
mean square value, standard deviation, least squares estimation, and orthogonal
functions) in engineering analysis can be explained from the perspectives of inner
product spaces. For example, the inner product of two vectors is a generalization
of the familiar dot product in vector calculus. This chapter should be read along
with Chapter 5 Parts A and B of Naylor & Sell. Specifically, both solved examples
and exercises in Naylor & Sell are very useful.
1
Basic Concepts
Definition 1.1. (Inner Product) Let (V, ⊕) be a vector space over a (complete)
field F, where we choose F to be R or C. Then, a function h•, •i : V × V → F is
defined to be an inner product if the following conditions hold ∀x, y, z ∈ V ∀α ∈ F:
• Additivity: h(x ⊕ y), zi = hx, zi + hy, zi.
• Homogeneity: hαx, yi = αhx, yi.
• Hermiticity: hy, xi = hx, yi.
• Positive Definiteness: hx, xi > 0 if x 6= 0V .
A vector space with an inner product is called an inner product space.
Example 1.1. Examples of Inner Product Spaces:
P
1. Let V = Fn and x, y ∈ V . Then, nk=1 ȳk xk is an inner product, where the
vector x ≡ [x1 , x2 , · · · , xn ].
2. Let V = Fn×m and A, B ∈ V . Then, trace(B H A) is an inner product.
P∞
3. Let V = ℓ2 (F) and x, y ∈ V . Then, k=1 ȳk xk is an inner product.
R
4. Let V = L2 (µ) and x, y ∈ V . Then, dµ(t) ȳ(t) x(t) is an inner product.
1
Remark 1.1. Let V be a vector space over a (complete) field F and x ∈ V . Then
hx, xi ∈ R, because it follows from the homogeneity property in Definition 1.1 that
hx, xi = hx, xi. Furthermore, it follows from the additivity and positive definiteness
properties in Definition 1.1 that hx, xi ∈ [0, ∞) ∀x ∈ V .
Remark 1.2. The following results are derived from Definition 1.1.
• hy, αxi = ᾱ hy, xi.
• hx, (y ⊕ z)i = hx, yi + hx, zi.
• If hx, xi = 0, then then x = 0V .
• If hx, yi = 0 ∀y ∈ V , then x = 0V .
Lemma 1.1. (Cauchy-Schwarz Inequality) Let V be a vector space over a (complete) field F. Then, the following inequality holds: |hx, yi| ≤ kxkkyk ∀x, y ∈ V ,
p
where kxk , hx, xi. The equality holds if and only if either x = αy or y = αx for
some α ∈ F.
Proof. Let x, y ∈ V and α =
hx,yi
hy,yi .
Then, it follows that
0 ≤ h(x − αy), (x − αy)i = kxk2 −
which completes the proof.
|hx, yi|2
kyk2
Corollary 1.1. The equality part in Lemma 1.1 holds if and only if x and y are
collinear, i.e., if either x = αy or y = αx for some α ∈ F.
Proof. The proof follows by substituting x or y in the inequality in Lemma 1.1.
Remark 1.3. Note that kx ⊕ yk = kxk + kyk or kx ⊕ yk = |kxk − kyk| if and only
if x and y are collinear.
p
Lemma 1.2. Let V be a vector space and x ∈ V . Then kxk , hx, xi is a valid
norm on the vector space V .
Proof. It directly follows from Definition 1.1 that kxk satisfies the strict positivity
and homogeneity properties of a norm. What remains to be shown is the triangular
inequality property. By choosing arbitrary vectors x, y ∈ V and applying the
Cauchy-Schwarz inequality in Lemma 1.1, it follows that
kx ⊕ yk2 = h(x ⊕ y), (x ⊕ y)i ≤ kxk2 + 2kxkkyk + kyk2 = kxk + kyk
2
Lemma 1.3. (Continuity of inner products) Let (V, h•, •i) be an inner product
space over a (complete) field F. Let {xk } and {y k } be two Cauchy sequences of
vectors in V, converging to x ∈ V and y ∈ V , respectively. Then, the sequence of
inner products {hxk , y k i} converge to the inner product hx, yi.
2
Proof. Subtracting and adding a term, using triangle inequality on scalars, and
using the Cauchy-Schwarz inequality and the continuity property of a norm, it
follows that
|hxk , y k i − hx, yi|
= |hxk , y k i − hxk , yi + hxk , yi − hx, yi|
≤ |hxk , (y k ⊕ (−y))i| + |h(xk ⊕ (−x)), yi|
≤ kxk kk(y k ⊕ (−y))k + k(xk ⊕ (−x))kky k k
→0
because kxk k < ∞, ky k k < ∞, k(y k ⊕ (−y))k → 0, and k(xk ⊕ (−x))k → 0 as
k → ∞.
Remark 1.4. Every inner product space is a normed space but every normed
space is not an inner product space. For example, the normed space ℓ2 is an inner
product space but the normed space ℓp for p 6= 2 is not an inner product space.
Given a normed space, how do we find out whether it qualifies as an inner product
space? This issue is addressed in the following theorem.
Theorem 1.1. (Parallelogram Equality) Let (V, h•, •i) be an inner product space.
Let k • k be the norm induced by the inner product h•, •i. Then, for all x, y ∈ V ,
the following equality, known as the parallelogram equality holds:
kx ⊕ yk2 + kx ⊕ (−y)k2 = 2kxk2 + 2kyk2
Proof. The proof follows by substituting hx, xi for kxk2 and performing algebraic
operations.
It is also a fact that the converse of Theorem 1.1 is true. That is, if V is a
normed vector space and satisfies the parallelogram equality, then there is a unique
inner product defined on the V , which generates the norm. In other words, V is
an inner product space if and only if the norm satisfies the parallelogram equality.
The issue is how to find the inner product from the knowledge of the norm. This
issue is addressed in the following theorem.
Theorem 1.2. (Polarization Identity) Let (V, k • k) be a normed vector space that
satisfies the parallelogram equality. Then, the (unique) inner product h•, •i on V
that satisfies the the relation hx, xi = kxk2 ∀x ∈ V is given as follows:
1. If the scalar field F over which the vector space is constructed is the real field
R, then the inner product is constructed from the given norm as follows:
1
kx ⊕ yk2 − kx ⊕ (−y)k2
hx, yi =
4
2. If the scalar field F over which the vector space is constructed is the complex
field C, then the inner product is constructed from the given norm as follows:
1
kx ⊕ yk2 − kx ⊕ (−y)k2 + ikx ⊕ iyk2 − ikx ⊕ i(−y)k2
hx, yi =
4
√
where i , −1.
Proof. The proof follows by substituting hx, xi for kxk2 and performing algebraic
operations.
3
Definition 1.2. A complete inner product space is called a Hilbert space.
Remark 1.5. Every inner product space (V, h•, •i induces a unique norm in the
vector space V defined over the field F. Since a norm is a valid metric in a vector
space, every inner product space is a metric space. Therefore, the topology of an
inner product space is the one generated by the metric: d(x, y) , kx ⊕ (−y)k =
p
h(x ⊕ (−y)), (x ⊕ (−y))i; for example, this is the usual topology in the Euclidean
space Rn or the unitary space Cn . It is also seen in Chapter 1 that every metric
space has an (essentially) unique completion; therefore, every inner product space
has an (essentially) unique completion that leads to the construction of a unique
Hilbert space from the given inner product space. In the next lemma, the inner
product h•, •i is viewed as a (continuous) mapping of the product space (V, h•, •i)×
(V, h•, •i) into the scalar field F.
1.1
Orthogonality in Inner Product Spaces
The Hilbert space is a close cousin of the Euclidean space in the sense that the
geometrical notions in these spaces are similar. The rationale for this similarity
is largely due to the concept of orthogonality that is admissible in inner product
spaces.
Definition 1.3. (Orthogonality) Let (V, h•, •i) be an inner product space. Then,
two vectors x, y ∈ V are said to orthogonal (to each other) if hx, yi = 0. Orthogonality of x and y is denoted as x ⊥ y. Two subsets (not necessarily subspaces)
A, B ⊆ V are said to be orthogonal (to each other) if hx, yi = 0 for all x ∈ A and
all y ∈ B; this is denoted as A ⊥ B.
Lemma 1.4. (Pythagorean Theorem) Let (V, h•, •i) be an inner product space and
let x, y ∈ V . If x ⊥ y, then kx ⊕ yk2 = kxk2 + kyk2 .
Proof. kx ⊕ yk2 = h(x ⊕ y), (x ⊕ y)i = kxk2 + hx, yi + hy, xi + kyk2 = kxk2 + kyk2
because hx, yi = hy, xi = 0.
Definition 1.4. (Direct Sum) Let V be a vector space and let U and W be two
subspaces of V . Then, a subspace Y is the direct sum of U and W , denoted as Y =
L
U
W , if ∀ y ∈ Y ∃ u ∈ U and w ∈ W such that there is a unique representation
y = u ⊕ w; and U is called the algebraic complement of W (alternatively, W is
called the algebraic complement of U ) in Y .
Definition 1.5. (Orthogonal Complement) Let (V, h•, •i) be an inner product
space and let ∅ ⊂ M ⊆ V . (Note that M may or may not be a vector space.)
Then, the orthogonal complement of M in (V, h•, •i) is defined as
M⊥ = x ∈ V : x ⊥ y ∀ y ∈ M
i.e., M ⊥ is made of all vectors in V , which are orthogonal to each vector in M . An
x ∈ M ⊥ is denoted as x ⊥ M .
Remark 1.6. If M = {0V }, then M ⊥ = V and if M = V , then M ⊥ == {0V }.
Also note that
4
M
\
M
⊥
=
(
{0V }
∅
if 0V ∈ M
if 0V ∈
/M
Theorem 1.3. (Closure Complement) Let (V, h•, •i) be an inner product space and
let ∅ ⊂ M ⊆ V . Then, M ⊥ is a closed subspace of (V, h•, •i).
Proof. It follows from Definition 1.5 that M ⊥ satisfies all conditions for a vector
space because any linear combination of vectors orthogonal to M is also orthogonal
to M . Therefore, M ⊥ is a subspace of V . What remains to show that M ⊥ is closed
in (V, h•, •i).
Let {xk } be a convergent sequence in M ⊥ and let xk → x̃. Since hxk , yi = 0 ∀ xk
for any y ∈ M , it follows from continuity of inner products (see Lemma 1.3) that
hx̃, yi = limk→∞ hxk , yi = 0. Therefore, the limit x̃ ∈ M ⊥ and so M ⊥ is a closed
subspace of (V, h•, •i).
Theorem 1.4. : Let (V, h•, •i) be an inner product space and let ∅ ⊂ S ⊆ V and
∅ ⊂ T ⊆ V . Then, the following statements are valid.
1. If S ⊆ T , then T ⊥ ⊆ S ⊥ and S ⊥⊥ ⊆ T ⊥⊥ .
2. S ⊆ S ⊥⊥ and S ⊥ = S ⊥⊥⊥ .
T
3. If x ∈ S S ⊥ , then x = 0V . Hence, it follows taht
(
\
{0V } if 0V ∈ S
⊥
S
S =
∅
if 0V ∈
/S
4. {0V }⊥ = V and V ⊥ = {0V }.
5. If S is a dense subset of V , then S ⊥ = {0V }.
Note: These results do not need completion of the inner product space (V, h•, •i).
Proof. The proof is as follows.
1. Let x ∈ S ⊆ T and y ∈ T ⊥ , where both choices of x and y are arbitrary.
Therefore, x⊥y and hence y ∈ S ⊥ . So, T ⊥ ⊆ S ⊥ . Applying this procedure
once more on the derived condition: T ⊥ ⊆ S ⊥ , it follows that S ⊥⊥ ⊆ T ⊥⊥ .
2. Let x ∈ S, where the choice of x is arbitrary. So x⊥S ⊥ , which implies that
x ∈ S ⊥⊥ . Hence, S ⊆ S ⊥⊥ . Then, by (1) it follows that S ⊥⊥⊥ ⊆ S ⊥ and
also S ⊥ ⊆ (S ⊥ )⊥⊥ = S ⊥⊥⊥ . Hence, S ⊥ = S ⊥⊥⊥ .
T
3. If x ∈ S S ⊥ , then x⊥x implying thar hx, xi = 0. That is, kxk = 0 or
x = 0V . The remaining part follows in the same way from Definition 1.5.
4. The proof follows directly from Definition 1.5.
5. If x ∈ S ⊥ , then ∀ y ∈ S it follows from Pythagorean theorem that
kx ⊕ (−y)k2 = kxk2 + kyk2 ≥ kxk2
Since S is dense in V , y can be chosen such that kx ⊕ (−y)k becomes arbitrarily small. Hence, kxk = 0. (Note that if S ⊥ = {0V }, then S is dense in
V only if (V, h•, •i) is a Hilbert space.)
5
Theorem 1.5. Let (H, h•, •i) be a Hilbert space and let G be a subspace of H.
Then, the following statements are valid.
1. G = G⊥⊥ , where G is the closure of G.
2. If G is closed in (H, h•, •i), then G⊥⊥ = G.
3. If G⊥ = {0V } if and only if G is dense in H.
4. If G is closed in (H, h•, •i) and if G⊥ = {0V }, then G = H.
Note: These results do need completion of the inner product space (H, h•, •i).
Proof. The proof is as follows.
1. It is known that G ⊆ G. It follows from Theorem 1.4 that G ⊆ G⊥⊥ , It also
follows from Theorem 1.3 that G ⊆ G⊥⊥ . Now, if G 6= G⊥⊥ , then there is
a nonzero vector z ∈ G⊥⊥ such that z⊥G.since G ⊆ G, it follows that z⊥G.
T
Therefore, z ∈ G G⊥ z = 0V . This is a contradiction. Hence, G = G⊥⊥ .
2. The proof follows directly from part (1) above.
3. The “if” part follows from Theorem 1.4 part (5). Now let G⊥ = {0V }. Then,
G⊥⊥ = V and it follows from part (1) that G = V . Hence, G is dense in V .
4. This part follows from part (3) above.
Theorem 1.6. Let (H, h•, •i) be a Hilbert space and let F and G be two closed
L
subspaces of H. If F ⊥ G, the direct sum F
G is a closed subspace of H.
Note: This result does need completion of the inner product space (H, h•, •i)
L
Proof. Let {zk } be a convergent sequence in F
G with zk → z. we want to show
L
that z ∈ F
G. That is, zk = xk ⊕ yk , where xk ∈ F and yk ∈ G. Since F ⊥ G,
it follows from Pythagorean Theorem that
kzk − zℓ k2 = kxk − xℓ k2 + kyk − yℓ k2
hence, {xk } and {yk } are Cauchy sequences in F and G , respectively. It follows
L
from continuity of addition that z = x ⊕ y. Therefore, F
G is closed in V .
Theorem 1.7. (Projection Theorem: Version 1) Let (H, h•, •i) be a Hilbert space
and let G be a closed subspace of H. Then,
L
1. H = G G⊥ .
2. Each x ∈ H can be uniquely expressed as x = y ⊕z, where y ∈ G and z ∈ G⊥ ,
and kxk2 = kyk2 + kzk2.
Note: This result does need completion of the inner product space (H, h•, •i)] and
closedness of G in H.
L
Proof. It follows from Theorem 1.6 that G G⊥ is a closed subspace of H. Since
L
L
G ⊆ G G⊥ and G⊥ ⊆ G G⊥ , it follows from Theorem 1.4 and Theorem 1.5
L ⊥
that H = G G .
See Naylor & Sell: Proof of Theorem 5.15.6 in Page 298 for more details.
6
1.2
Orthogonal Projection
Definition 1.6. (Orthogonal Projection) A projection P on an inner product space
is called an orthogonal projection if the range space and null space of P are orthogonal, i.e., R(P )⊥N (P ). (See Naylor & Sell (1982), p. 201 and p. 300.)
Remark 1.7. Note that all projections in an inner product space need not be
orthogonal. For example, in R2 with the inner product hx, yi , x1 y1 + x2 y2 , the
projection matrix
P =
"
0 0
−1 1
#
is linear and idempotent, i.e., yields P 2 = P , but its range space and null space
are not mutually orthogonal.
Remark 1.8. It follows from Definition 1.6 that if P is an orthogonal projection,
then so is I − P .
Theorem 1.8. (Continuity of Orthogonal Projection) An orthogonal projection in
an inner product space is continuous.
Proof. See Naylor & Sell: Proof of Theorem 5.16.2 in Page 300.
Theorem 1.9. Let P be an orthogonal projection on an inner product space V .
Then,
1. R(P ) and N (P ) are closed subspaces of V .
2. N (P ) = R(P )⊥ and R(P ) = N (P )⊥ .
3. Every vector x ∈ V can be uniquely expressed as x = r ⊕ n, where r ∈ R(P )
and n ∈ N (P ).
4. kxk2 = krk2 + knk2 .
Proof. See Naylor & Sell: Proof of Theorem 5.16.3 in Page 301.
Theorem 1.10. (Projection Theorem: Version 2) Let (H, h•, •i) be a Hilbert space
and let G be a closed subspace of H. Then,
1. R(P ) and N (P ) are closed subspaces of V .
2. N (P ) = R(P )⊥ and R(P ) = N (P )⊥ .
3. Every vector x ∈ V can be uniquely expressed as x = r ⊕ n, where r ∈ R(P )
and n ∈ N (P ).
4. kxk2 = krk2 + knk2 .
Proof. See Naylor & Sell: Proof of Theorem 5.16.2 in Page 300.
7
1.3
Optimization in a Hilbert Space
Let us digress slightly from the main theme of this chapter and focus our attention
on optimization in a Hilbert space, where the projection theorem characterizes the
optimal solution. We present two slightly different versions of the theorem – one
dealing with an arbitrary inner product space and the other providing a stronger
solution in a Hilbert space. In this context, the problem is stated as follows.
Let V be an inner product space and let W be a subspace of V . Given an
arbitrary vector y ∈ V , the task is to identify a vector x̂ ∈ W closest to y in the
sense that the error ky − x̂k is minimized. If y ∈ W , the the obvious solution is
x̂ = y. Therefore, we deal with the situation y ∈
/ W , where the following questions
must be answered for a complete solution of the optimization problem.
(i) (Existence) Does there always exist a vector x̂ ∈ W that minimizes the error
k(y − x̂)k, or there is no x̂ ∈ W that is as least as good as all others?
(ii) (Uniqueness) Is the solution unique?
(iii) (Characterization) How the solution is characterized if it exists?
Theorem 1.11. (Uniqueness of the minimizing vector) Let V be an inner product
space and let W be a subspace of V . Given an arbitrary vector y ∈ V , if there exists
a vector x̂ ∈ W such that k(y − x̂)k ≤ k(y − x)k ∀y ∈ V , then x̂ ∈ W is unique.
A necessary and sufficient condition that x̂ be a unique minimizing vector in W is
that the error vector (y − x̂) is orthogonal to W .
Proof. First we apply a contradiction to show that if x̂ ∈ W is a minimizing vector,
then the error vector (y − x̂) is orthogonal to W . Let us assume that there exists
an x ∈ W \ {0H } that is not orthogonal to (y − x̂). It is noted that kxk > 0 and
let ε , h(y−x̂),xi
; obviously, ε 6= 0. Let x̃ , x̂ + εx. Then,
kxk
ky − x̃k2 = ky − x̂ − εxk2 = ky − x̂k2 − h(y − x̂), εxi − hεx, (y − x̂)i + |ε|2 kxk2
⇒ ky − x̃k2 = ky − x̂k2 − |ε|2 |xk2 < ky − x̂k2
Thus, if (y − x̂) is not orthogonal to W , then x̂ is not a minimizing vector.
Next we show that if (y − x̂) is orthogonal to W , then x̂ is a unique minimizing
vector. For any x ∈ W , the Pythagorean Theorem yields
ky − xk2 = k(y − x̂) + (x̂ − x)k2 = ky − x̂k2 + kx̂ − xk2
Thus, ky − xk > ky − x̂k ∀x 6= x̂.
Next we address the conditions for existence of the minimizing vector by strengthening the above theorem on uniqueness. Thus, we will satisfy the criteria for both
existence and uniqueness of the minimizing vector.
Theorem 1.12. (Existence and uniqueness of the minimizing vector) Let H be a
Hilbert space and G be a closed subspace of H. Then, corresponding to any vector
y ∈ H, there exists a unique vector x̂ ∈ G such that ky − x̂k ≤ ky − xk ∀x ∈
G. Furthermore, a necessary and sufficient condition that x̂ ∈ G be the unique
minimizing vector is that (y − x̂) be orthogonal to G.
8
Proof. The uniqueness and orthogonality have been established in Theorem 1.11.
What remains to be establish is existence of the minimizing vector x̂ ∈ G.
If y ∈ G, then trivially x̂ = y; let y ∈
/ G and δ , inf x∈G ky − xk. In order to
identify an x̂ ∈ G such that ky − x̂k = δ, we proceed as follows.
Let {xk } be a sequence of vectors in G such that limk→∞ ky − xk k = δ. Then,
it follows by using the parallelogram law that
k(xk − y) + (y − xℓ )k2 + k(xk − y) + (y − xℓ )k2 = 2 kxk − yk2 + ky − xℓ k2
A rearrangement of the above expression yields
2
(xk + xℓ ) k(xk − xℓ )k2 = 2kxk − yk2 + 2ky − xℓ k2 − 4
y
−
2
(xk +xℓ )
(xk +xℓ ) Since the vector
∈ G ∀k, ℓ ∈ N, it follows that y − 2 ≥ δ. Therefore,
2
k(xk − xℓ )k2 ≤ 2kxk − yk2 + 2ky − xℓ k2 − 4δ 2
As ky − xk k2 → δ 2 and ky − xℓ k2 → δ 2 , it follows that limk.ℓ→∞ kxk − xℓ k2 = 0.
Therefore, {xk } is a Cauchy sequence in G that is a closed (and hence a complete)
subspace of the Hilbert space H. Therefore, {xk } converges to the limit point in
G. Then, by continuity of the norm, it follows that the limit is x̂ ∈ G.
Remark 1.9. Although the proof of the above theorem does not make any explicit
reference to the inner product, it does make use of the parallelogram law that, in
turn, makes use of the inner product for its proof.
2
Orthonormal Sets and Bases
We have presented the notion of Hamel basis in Chapter 02 and Schauder basis
in Chapter 03, which involve only algebraic notions. Now we incorporate both
algebraic and topological notions in the formulation of a basis.
Definition 2.1. (Orthogonal Set) Let I be an arbitrary nonempty index set (i.e.,
finite or countable or uncountable). A set {xα : α ∈ I} in an inner product
space is called orthogonal if ∀α 6= β, xα ⊥xβ , i.e., hxα , xβ i = 0. In addition, if
hxα , xα i = 1 ∀α ∈ I, i.e., hxα , xβ i = δαβ α, β ∈ I, then the set {xα } is called
orthonormal. Note that
δαβ ,
(
1 if α = β
0 if α 6= β
is called the Kronecker delta. Also note that any orthogonal set of non-zero vectors
α
can be converted into an orthonormal set by replacing xα with kxxα k ∀α.
Lemma 2.1. Every orthonormal set of vectors in an inner product space is linearly
independent.
Proof. Let S = {xα } be an orthonormal set in an inner product space V . It
suffices to show that, for every subset {xk ∈ S : k ∈ {1, 2, · · · , n} and n ∈ N,
Pn
k
the only solution of the algebraic equation
k=1 αk x = 0V is αk = 0 for all
9
k ∈ {1, 2, · · · , n}. Now, taking inner product on both sides with xj for all j ∈
{1, 2, · · · , n},
n
DX
E αk xk , xj = 0V , xj = 0 ⇒ αj = 0 ∀j.
k=1
Definition 2.2. (Complete Orthonormal Set) Let I be an arbitrary nonempty
index set. An orthonormal set B = {xα } in an inner product space V is called
complete (or maximal) if the following condition holds
x⊥xα ∀α ∈ I ⇒ x = 0V
S
In other words, no unit vector x ∈ V exists such that Big(B {x} is an orthonormal
set. A complete orthonormal set in a Hilbert space V is called an orthonormal basis
for V .
Definition 2.3. (Orthonormal Basis) Let I be an arbitrary nonempty index set.
An orthonormal set B = {xα } in an inner product space V is called complete (or
maximal) if the following condition holds
x⊥xα ∀α ∈ I ⇒ x = 0V
S
Equivalently, no unit vector x ∈ V exists such that B {x} is an orthonormal set.
Remark 2.1. Every orthonormal set of vectors in in a Hilbert space can be extended to form an orthonormal basis. Furthermore, two orthonormal bases of a
Hilbert space must have the same cardinality.
2.1
The Fourier Series Theory
This subsection presents the fundamental properties of orthogonal bases by combining the topological and algebraic structures of a Hilbert space.
Lemma 2.2. (Bessel Inequality) Let {xk } be an orthnormal set in an inner product
space V . Then, for every x ∈ V , the following condition holds;
X
| hx, xk i |2 ≤ kxk2
k∈N
1
n
Proof. Let {x , · · · , x } be a finite subset from {xk }. Then, by making use of the
orthonarmality hxj , xk i = δjk , it follows that
n
2
X
hx, xj ixj 0 ≤ x −
j=1
=
= hx, xi −
n
X
j=1
hx, xj ihxj , xi −
n
X
k=1
D
x−
n
X
n
E
X
hx, xk ixj
hx, xj ixj , x −
j=1
n X
n
X
hx, xk ihxk , xi +
j=1 k=1
k=1
hx, xj ihxj , xihx, xk ihxk , xi
2
= kxk −
n
X
k=1
| hx, xk i |2
Since the above inequality is true for all n ∈ N, it must also hold true for a (countable) infinite sum. Hence, the proof is established.
10
Theorem 2.1. (Fourier Series Theorem) Let {xk } be an orthonormal set in a
Hilbert space H. Then, the following statements are equivalent.
(a) (Orthonormal basis) {xk } be an orthonormal basis of H.
(b) (Fourier series expansion) Every x ∈ H can be expanded as
k k
k hx, x ix .
P
(c) (Parseval equality) ∀x, y ∈ H, the following relation holds; hx, yi =
P
k
k hx, x i
hy, xk i.
(d) (Norm decomposition) The norm of every x ∈ H can be decomposed as:
X
kxk2 =
|hx, xk i|2 < ∞.
k
(e) (Dense subspace of H) Let V be a subspace of H such that {xk } ⊂ V . Then,
V is dense in H, i.e. closure(V )=H.
Proof. Following Naylor & Sell (pp. 307-312), the proof is completed by showing the
equivalence of individual parts in the following order: (a) ⇒ (b) ⇒ (c) ⇒ (d) ⇒ (a)
and (b) ⇒ (e) ⇒ (b).
Corollary 2.1. Let V be a closed subspace of a Hilbert space H, where V 6= {0H };
and let {xk } be an orthonormal basis of V . Then, the orthogonal projection P of
P
H onto V is given by P x = k hx, xk ixk ∀x ∈ H.
Proof. The proof follows from Theorem 2.1.
2.1.1
Gram-Schmidt Procedure
Let V be an inner product space and let {y k } be a linearly independent set with
countable cardinality in V . The objective here is to construct an orthonormal
sequence from the By making use of Bessel inequality in Lemma 2.2. Now we
present the following theorem.
Theorem 2.2. (Gram-Schmidt Orthonormalization) Let {xk } be an at most countable (i.e., finite or countably infinite) set of linearly independent vectors in an inner
product space V . Then, there exists an orthonormal sequence {ek } ⊂ V such that,
for every n ∈ N,
Span[e1 , · · · , en ] = Span[x1 , · · · , xn ].
1
Proof. First let us normalize the first vector x1 as e1 , kxx1 k that obviously spans
the same space as x1 does. Next we construct z 2 , x2 − hx2 , e1 ie1 . Also note
that z 2 6= 0V because x2 and x1 are linearly independent and Span[e1 , e2 ] =
2
Span[x1 , x2 ]. Now, let e2 , kzz2 k , which assures he2 , e1 i = 0 and he2 , e2 i = 1.
Proceeding in this way, the remaining ek ’s are generated by induction as follows.
P
k
Given z k , xk − j=1 k − 1hxk , ej iej and ek , kzzk k for all k ∈ {3, · · · , n}. Since
hej , ek i = δjk ∀j, k ∈ {1, · · · , n}, we have hz n+1 , ek i = 0, i.e., z n+1 is orthogonal
n+1
to Span[e1 , · · · , en ] = Span[x1 , · · · , xn ]. Next, define en+1 , kzzn+1 k . The proof by
induction is thus completed.
Example 2.1. (Orthogonal polynomials) Let T = (−1, 1) and let us consider the
(linearly independent) sequence of vectors {tk : k ∈ N ∪ {0} and t ∈ T } on the
inner product space L2 (T ) over the real field R, where the inner product is defined
11
R
as hx, yi , T dt x(t)y(t) ∀x, y ∈ L2 (T ). Then, Gram-Schmidt orthonormalization
of the sequence {tk } generates a set of orthonormal polynomials, called Legendre
polynomials as seen below.
√
R
0
Noting that h1, 1i = −1 1 dt = 2 ⇒ k1k = 2, we have e1 = ktt0 k = √12 .
q
R
2
z 2 , t − ht, e1 ie1 = t − 12 −1 1 t dt = t ⇒ e2 = kzz2 k = 23 t.
q
3
1
2
z 3 , t2 − ht2 , √12 i √12 − ht2 , √32t i √32t = t2 − 13 ⇒ e3 = kzz3 k = 45
8 (t − 3 ).
..
.
There are many other families of orthonormal polynomials that can by gramSchmidt orthonormalization of the (linearly independent) sequence of vectors {tk :
k ∈ N ∪ {0} and t ∈ T } on different inner product spaces. A few examples follow.
• Legendre Polynomials on the space where the inner product is defined as:
R
hx, yi , T dt x(t) y(t), where T = (−1, 1).
• Laugure Polynomials on the space where the inner product is defined as:
R
hx, yi , T dt exp(−t) x(t) y(t), where T = (0, ∞).
• Hermite Polynomials on the space where the inner product is defined as:
R
hx, yi , T dt exp(−t2 ) x(t) y(t), where T = (−∞, ∞).
• Chebyshev Polynomials on the space where the inner product is defined as:
p
R
hx, yi , T dt (1 − t2 ) x(t) y(t), where T = (−1, 1).
2.1.2
Fourier Transform
Now we introduce the concept of Fourier transform of (absolute) integrable functions, i.e., the functions belonging to the L1 (µ) space, where µ is the Lebesgue measure on the real line R. We note that, in the space Lp (µ) for p ∈ [1, ∞], Lebesgue
measure µ is defined on any measurable subset of the real line R. In other words,
R
we consider functions f ∈ L1 (µ), where R dµ(t)|f (t)| < ∞, for Fourier transform.
Definition 2.4. (Fourier transform) For f ∈ L1 (µ), the Fourier transform of f is:
fˆ(ξ) ,
Z
R
dµ(t) exp (−i2πξt)f (t) ∀ξ ∈ R
and for fˆ ∈ L1 (µ), i.e.,
f (t) ,
Z
R
R
R
Fourier Analysis Formula
dµ(ξ)|fˆ(ξ)| < ∞, inverse Fourier transform of fˆ is:
dµ(ξ) exp (i2πξt)fˆ(ξ) ∀t ∈ R
Fourier Synthesis Formula
Remark 2.2. The Fourier transform converts time translation into a phase shift
and vice versa, and convolution into products as seen below. To see this, let f ∈
L1 (µ) and α, λ ∈ R. Then,
1. If g(t) = f (t) exp(i2παt), then ĝ(ξ) = fˆ(ξ − α) and if g(t) = f (t − α), then
ĝ(ξ) = fˆ(ξ) exp(−i2παξ).
2. If g ∈ L1 (µ) and h = f ⋆ g, then ĥ(ξ) = fˆ(ξ)ĝ(ξ). Similarly, if ĝ ∈ L1 (µ)
and ĥ = fˆ ⋆ ĝ, then h(t) = f (t)g(t).
3. If g(t) = f (−t), then ĝ(ξ) = fˆ(ξ).
12
4. If λ 6= 0 and g(t) = f ( λt ), then ĝ(ξ) = |λ|fˆ(λξ).
5. Let n ∈ N. If f is a C ∞ -continuous function, then Fourier transform of
n ˆ
∂ n f (t)
is (i2πξ)n fˆ(ξ); similarly, Fourier transform of (−i2πt)n f (t) is ∂ ∂ξf n(ξ) ,
∂tn
provided that the Fourier transforms exist.
Rt
fˆ(ξ)
provided that fˆ(0) = 0.
6. Fourier transform of −∞ dµ(τ )f (τ ) is i2πξ
R∞
n ˆ
7. Let n ∈ N ∪ {0}. Then, (−i2π)n −∞ dµ(τ )τ n f (τ ) = ∂∂ξnf , provided that
ξ=0
the integral and the derivatives exist.
Also note that if f ∈ L1 (µ) and fˆ(ξ) = 0 ∀ξ ∈ R, then f (t) = 0 almost everywhere
( µ-a.e.) on R.
Next we introduce the notion of Plancherel Theorem. Since the Lebesgue measure µ of R is not finite (it is σ-finite), we cannot claim L2 (µ) to be a subspace
of L1 (µ). Fourier transform in Definition 2.4 is not directly applicable to every
T
f ∈ L2 (µ). However, the definition does apply if f ∈ L1 (µ) L2 (µ) and it turns
T
out that fˆ ∈ L2 (µ). In fact, kfˆkL2 = kf kL2 . This isometry of L1 (µ) L2 (µ) into
L2 (µ) extends to an isometry of L2 (µ) onto L2 (µ), and this defines an extension
of the Fourier transform, which is often called as the Plancherel transform and is
applicable to every f ∈ L2 (µ). The resulting L2 -theory has more symmetry than
that in the L1 -theory. In other words, f and fˆ play the same role in L2 (µ).
Theorem 2.3. (Plancherel Theorem) Each function f ∈ L2 (µ) can be associated
to a function fˆ ∈ L2 (µ) such that the following properties hold:
T
1. If f ∈ L1 (µ) L2 (µ), then fˆ is the the Fourier transform of f , i.e.,
R
fˆ(ξ) = R dt exp (−i2πξt)f (t) ∀ξ ∈ R.
2. kfˆkL2 = kf kL2 for every f ∈ L2 (µ).
3. The mapping f 7→ fˆ is a Hilbert space isomorphism of L2 (µ) onto L2 (µ).
4. The following symmetric relation exists between f and fˆ:
RA
RA
If ϕA (ξ) , −A dµ(t) exp(−i2πξt)f (t) and ψA (t) , −A dµ(ξ) exp(i2πξt)fˆ(ξ),
then kϕA − fˆkL2 → 0 and kψA − f kL2 → 0 as A → ∞.
Proof. See Real and Complex Analysis by Walter Rudin (pp.187-189).
2.1.3
Hilbert Dimension and Separable Hilbert Spaces
Let us classify Hilbert spaces (over the same field) by cardinality of the orthonormal basis set. Let B1 and B2 be two orthonormal bases of a Hilbert space H.
Since B1 and B2 must have the same cardinal numbers, this cardinal number is
called the Hilbert dimension of H. For finite-dimension spaces H, the Hilbert dimension of H is the same as the dimension related to a Hamel base. However,
for infinite-dimensional Hilbert spaces, the Hibert dimension could be countable or
uncountable. An example of countable Hilbert dimension is the Hilbert space of
periodic functions L2 [−π, π].
13
Example 2.2. The Hilbert space ℓ2 is isometrically isomorphic to the Hilbert space
L2 [−π, π]. That is, there exists a linear bijective mapping f : ℓ2 → L2 [−π, π] such
that the inner products satisfy the following equality
hx, x̃iℓ2 = hf (x), f (x̃)iL2 [−π,π] ∀x, x̃ ∈ ℓ2
To see this, we proceed as follows. Let {xk } be an orthonormal basis for
P
L2 [−π, π] and let f : ℓ2 → L2 [−π, π] be defined as: f (a) = k∈N ak xk
We cite below an example of a Hilbert space having uncountable Hilbert dimension.
Example 2.3. (Uncountable Hilbert dimension) Let us consider a trigonometric
Pm
polynomial p : R → C such that ∀t ∈ R p(t) ,
k=1 ck exp(irk t) for some
m ∈ N, where r1 , · · · , rm ∈ R and c1 , · · · , cm ∈ C. Let p̃ : R → C be another
Pm̃
trigonometric polynomial defined as ∀t ∈ R p̃(t) ,
k=1 c̃k exp(ir̃k t) for some
m̃ ∈ N, where r̃1 , · · · , r̃m̃ ∈ R and c̃1 , · · · , c̃m̃ ∈ C. Then, it follows that
p(t) p̃(t) =
m X
m̃
X
j=1 k=1
cj c̃k exp i(rj − r̃k )t
We define an inner product in the space trigonometric polynomials as
1
hp, p̃i , lim
T →∞ 2T
Z
T
dt p(t) p̃(t) =
−T
Z
T
−T
cj c̃k
j=1 k=1
because
1
lim
T →∞ 2T
m X
m̃
X
dt exp irt =
(
1 if r = 0
0 if r ∈ R \ {0}
The completion of the inner product space of trigonometric polynomials is a
Hilbert space called the space of almost periodic functions. Let us call this Hilbert
space as HtrigP . For all r ∈ R let us consider the function ur (t) , exp(irt), t ∈ R.
Then, {ur : r ∈ R} is an uncountable orthonormal set in HtrigP . Since the set
of all trigonometric polynomials is dense in HtrigP , it follows that if p ∈ H and
hp, ur i = 0 ∀r ∈ R, then kpk2 = hp, pi = 0 ⇒ p = 0H . Hence, {ur : r ∈ R} is an
uncountable orthonormal basis for HtrigP .
If {zn : n ∈ N} is a countable subset of HtrigP , then the open ball B √1 zn
2
√
contain at most one ur , because kur − us k = 2 for ∀ r 6= s. Hence, the collection
of all open balls B √1 zn can cover only a countable number of ur ’s. Since the set
2
{ur : r ∈ R} has uncountable cardinality, no countable set can be dense in HtrigP .
Hence, HtrigP is a non-separable Hilbert space.
Theorem 2.4. (Separable Hilbert Spaces) Let H be a Hilbert space. Then, H is a
separable Hilbert space (i.e., H has a countable subset that dense in H) if and only
if every orthonormal basis of H is countable.
Proof. To prove the only if part, let the Hilbert space H be separable, U ⊆ H be
countable and dense in H, and M be an orthonormal basis of H. Then, for every
x, y ∈ M ,
(
0
if x = y
√
kx − yk =
2 if x 6= y
14
Then, for ∀x, y ∈ U and x 6= y, the open neighborhoods N √1 (x) and N √1 (y) in H
2
2
are disjoint. Therefore, every N √1 (x) in H can have at most one element of M .
2
Since U is countable, M must be countable; else U cannot be dense in H.
To prove the if part, let {ek } be a countable orthonormal basis of H and let A
be a set of all (finite) linear combinations γ1n en + · · · + γnn en for all n ∈ N, where
γjn , αnj + iβjn and αnj , βjn ∈ Q, the set of rational numbers. Note that the set A is
countable and if H is defined over R, then set all βjn = 0. We will prove that A is
dense in H by showing
∀x ∈ H ∀ε > 0 ∃ x̃ ∈ A such that kx − x̃k < ε
Given any x ∈ H and ε > 0, let Yn be the space spanned by a selected subset of
vectors {e1 , · · · en } from the orthonormal basis {ek } such that kx−yk < ε2 ∀y ∈ Yn .
This is possible because {ek } is a complete ortonormal set in H. It follows, by
orthogonal projection of x onto Yn , that
n
n
ε
X
X
y=
hx, ek iek ⇒ x −
hx, ek iek <
2
k=1
k=1
Pn
Let x̃ = k=1 γkn ek be the orthogonal decomposition of x̃ ∈ A. Then,
n
X
γkn ek kx − x̃k = x −
k=1
n
n
n
ε ε
X
X
X
γkn ek < + = ε
hx, ek iek −
≤ x −
hx, ek iek + 2 2
k=1
k=1
k=1
which proves that A is dense in H; and since A is countable, H is separable.
Example 2.4. The spaces ℓ2 and L2 ([−π, π]), defined on the same field F, have
the same Hilbert dimension. It is also true that ℓ2 and L2 ([−π, π]) are isometrically
isomorphic as explained below.
Let {xk } be an orthonormal basis for the Hilbert space L2 ([−π, π]) and let
f : ℓ2 → L2 ([−π, π]) be defined as follows:
f (a) ,
∞
X
k=1
ak xk , where a , {a1 a2 · · · } ∈ ℓ2 .
Now we show that f is linear and bijective. Linearity is established as follows.
f (αa + b) =
∞
X
αa + b)k xk = α
k=1
∞
X
ak xk +
∞
X
bk xk = αf (a) + f (b)
k=1
k=1
To establish injectivity of f , let f (a) = f (b). Since f and {xk } is orthonormal,
it follows that
∞
∞
∞
2 X
X
X
0 = kf (a) − f (b)k2 = (a − b)k xk =
|(a − b)k |2 kxk k2 =
|(a − b)k |2
k=1
k=1
k=1
which implies that a = b.
To establish surjectivity of f and isometry of the spaces ℓ2 and L2 ([−π, π]), let
P∞
x ∈ L2 ([−π, π]). Then, it follows from Fourier series theorem that x = k=1 hx, xk i.
Setting ak = hx, xk i, it follows from orthonormality of {xk } that
∞
∞
∞
2 X
X
X
kxk2 = ak xk =
|ak |2 kxk k2 =
|ak |2 = kak2
k=1
k=1
k=1
15
which implies that, for each x ∈ L2 ([−π, π]), there exists an a ∈ ℓ2 such that
f (a) = x and kakℓ2 = kxkL2 ([−π,π]) .
3
Applications of Hilbert Spaces
Hilbert spaces have numerous applications in Science and Engineering. In modern
Physics (that started in early 120’s), the mathematics of Quantum Mechanics is
built upon the concept of Hilbert spaces over the complex field C. While there
is abundance of literature in mathematics of Quantum Mechanics, Chapter 11 of
Introductory Functional Analysis with Applications by E. Kreyszig provides a brief
explanation of the underlying concepts. In this section, we provide a few examples
of applications of Hilbert spaces in Signal Processing and Quantum Mechanics.
3.1
Shannon Sampling: Bandlimited Signals
Sampling of continuous-time signal is a key step to discrete-time signal processing.
In engineering practice, sampling is accomplished by analog-to-digital (A/D) conversion of continuously varying signals that are generated by various sensors. It is
apparent that the sampling frequency should be selected depending on the bandwidth of the signal and the sensor that is used to measure the signal. For example,
if the sampling is performed infrequently, the vital information content of the signal
might be lost. On the other hand, a very high sampling frequency may generate
excessively redundant data that would require additional memory and computation
time. We present the analytical relationship between the signal bandwidth and the
smallest sampling frequency for no loss of information.
Theorem 3.1. (The Shannon Sampling Theorem) Let a (real-valued) signal f ∈
L2 (R) be band-limited by Ω, i.e., fˆ(ξ) = 0 ∀ ξ ∈
/ [−Ω, Ω]. Then, f (t) can be perfectly
k
reconstructed from its samples f (tk ), collected at time instants tk , 2Ω
, k ∈ Z, by
the following interpolation formula:
X sin 2πΩ (t − tk )
f (tk )
f (t) =
2πΩ (t − tk )
k∈Z
Proof. By Plancherel theorem, the L2 -norm of Fourier transform is expressed as:
Z Ω
2
ˆ
k f kL2 (−Ω,Ω) =
dξ | fˆ(ξ) |2
−Ω
Z ∞
dξ | fˆ(ξ) |2 =k fˆ k2L2 (R)
=
−∞
=k f k2L2 (R)
Expansion of fˆ ∈ L2 (−Ω, Ω) in a Fourier series yields
1 X
ck exp − i 2πξtk )
fˆ(ξ) =
2Ω
k∈Z
where the Fourier coefficients ck are identically equal to the samples of the signal
k
, i.e.,
f (t) taken the time instants tk , 2Ω
Z Ω
ck =
dξ exp i 2πξtk fˆ(ξ) = f (tk )
−Ω
16
It follows by equality in the L2 -sense and by exchange (due to uniform convergence
of the summand) of the integral and the infinite sum that
Z
f (t) =
dξ exp i 2πξtk fˆ(ξ)
=
Z
R
Ω
−Ω
dξ exp i 2πξtk fˆ(ξ)
Z Ω
1
=
dξ exp i 2πξtk
2Ω −Ω
X
×
f (tk ) exp − i 2πξtk
k∈Z
1 X
=
2Ω
k∈Z
Since
RΩ
−Ω
Z
Ω
−Ω
dξ exp i 2πξ(t − tk f (tk )
dξ exp i 2πξ(t − tk =
sin(2πΩ (t−tk ))
2πΩ (t−tk ) ,
it follows that
X sin 2πΩ (t − tk )
f (t) =
f (tk )
2πΩ (t − tk )
k∈Z
in the L2 -sense. As band-limited signals are analytic in the entire complex plane,
it suffices that f (•) is continuous. Hence, the above equality holds pointwise everywhere in the domain of f (•).
3.2
Fourier Transform in Quantum Mechanics
Let x be the position of a particle and the quantum wave function ψ(x) be a
complex-valued function of x, which belongs to a Hilbert space over the complex
field C. The state of a classical particle in one dimension is specified by both
position x and momentum p while, in quantum physics, the wave function depends
on only one of the two variables. The information on momentum p is contained
in the wave function ψ(x). A convenient way to extract the information on p as a
Fourier transform of ψ(x) is to construct the momentum wave function
Z ∞
Z ∞
−ipx −i2πpx 1
1
ψ̂(p) , √
ψ(x) = √
ψ(x)
dx exp
dx exp
~
h
2π~ −∞
h −∞
where the Planck constant ~ = 1.054572 × 10−34 Joule-sec and the original Plank
constant h = 2π~.
The above Fourier transform can be inverted to generate the information on
position x from ψ̂(p) as
Z ∞
Z ∞
+ipx +i2πpx 1
1
ψ(x) , √
ψ̂(p) = √
ψ̂(p)
dp exp
dp exp
~
h
2π~ −∞
h −∞
Therefore, the position waveform ψ(x) and the momentum waveform ψ̂(p) contain
the precisely the same information about a quantum state. By Plancherel Theorem,
the inner product is obtained in either position domain or momentum domain as
!
Z ∞
Z ∞
dp ψ̂(p)ϕ̂(p)
hϕ, ψi = hϕ̂, ψ̂i ⇔
dx ψ(x)ϕ(x) =
−∞
−∞
17
In essence, the information about the momentum of a quantum particle can be
obtained from the momentum wave function in the same way that information
about its position can be obtained from the position wave function as explained
below.
The position wave function has a compact support, i.e., ψ(x) vanishes outside
in a finite interval x1 ≤ x ≤ x2 ; similarly, momentum wave function has a compact
support, i.e., ψ̂(p) vanishes outside a finite interval p1 ≤ p ≤ p2 . Therefore, it is
safe to say that the quantum particle does not lie outside a position interval and
a momentum interval. However, we may not be able to specify a deterministic
value of the position and momentum within these intervals. This issue is further
examined in the next section from the perspectives of both Quantum Mechanics
and Signal Processing.
3.3
Heisenberg Uncertainty Principle and Signal Analysis
Let us start with the statement that the use of ”uncertainty” in the so-called Uncertainty Principle is perhaps a misnomer, because there is nothing uncertain about
the uncertainty principle. In the discipline of signal analysis, it is a well-known
mathematical fact that
A narrow-band waveform yields a wide spectrum and a wide-band waveform yields a narrow spectrum.
It is possible that both time and frequency can be narrow, but the probability densities of time and frequency localization cannot be made simultaneously
arbitrarily narrow, i.e., their variances cannot be made simultaneously arbitrarily
small.
In both physics and signal analysis, the uncertainty principle applies to a pair
of variables whose associated operators do not commute.
In the physics literature,
Position Operator:
χ≡ x
∂
h ∂
Momentum Operator:
p≡
− i~ ∂x
or − i 2π
∂x ,
The commutator of χ and p on an arbitrary (differentiable) function f (x) is
defined as:
∂ ∂ f (x) + i~
xf (x)
[χ, p] , χp − pχ ⇒ [χ, p]f (x) = x − i~
∂x
∂x
∂f
∂f
−x
− f (x) = i~f (x)
= −i~ x
∂x
∂x
h
⇒ [χ, p] = i~ or [χ, p] = i
2π
Similarly, in the signal analysis literature,
i d
Time Operator:
ℑ≡ t ,
2π
dξ
−i d
Frequency Operator: Ξ ≡ ξ ,
2π dt
18
The commutator of ℑ and Ξ on an arbitrary (differentiable) signal s(t) is defined
as:
−i d −i d s(t) −
ts(t)
2π dt
2π dt
ds
i
−i ds
t − t − s(t) =
s(t)
=
2π dt
dt
2π
i
⇒ [ℑ, Ξ] =
2π
[ℑ, Ξ] , ℑ Ξ − Ξ ℑ ⇒ [ℑ, Ξ] s(t) = t
Alternatively,
[ℑ, Ξ] , ℑ Ξ − Ξ ℑ ⇒ [ℑ, Ξ] ŝ(ξ) =
3.4
i d i d
ξ ŝ(ξ)) − ξ
ŝ(ξ)
2π dξ
2π dξ
dŝ
i dŝ
i
dŝ
ŝ(ξ) + ξ
=
=
−ξ
ξ
2π
dξ
dξ
2π dξ
i
⇒ [ℑ, Ξ] =
2π
Time-frequency Localization
R∞
Let us consider a unit energy signal s(t) ∈ L2 (R), i.e., −∞ dt | s(t) |2 = 1 and
R∞
2
2
−∞ dξ | ŝ(ξ) | = 1. In line with the concepts in Quantum Mechanics, | s(t) | and
2
| ŝ(ξ) | are the probability densities of time and frequency localization, respectively.
By appropriate time translation and frequency modulation, it follows that
Z ∞
Z ∞
2
dt | s(t) | t = 0 and
dξ | ŝ(ξ) |2 ξ = 0
−∞
−∞
without loss of generality. Let us define the so-called variances of time localization
and frequency localization as:
Z ∞
Z
σt2 ,
dt | s(t) |2 t2 and σξ2 ,
−∞
∞
−∞
dξ | ŝ(ξ) |2 ξ 2
Theorem 3.2. (Heisenberg Uncertainty for Time-frequency Localization) If the
1
1
signal s(t) ∈ L2 (R) decays to zero faster than |t|− 2 as t → ± ∞, then σt σξ ≥ 4π
pα
and the equality holds for Gaussian signals s(t) = π exp(−αt2 ).
Proof. By Cauchy-Schwarz inequality
Z ∞
Z
Z ∞
2 ∞
ds 2
dt t s(t)
dt t s(t)
dt
≤
dt
−∞
−∞
−∞
ds 2
.
dt
=
The first integral on the right is equal to σt2 .Taking Fourier transform of ds
2 R ∞
2dt
R∞
i2πξŝ(ξ) and by using Plancherel Theorem, i.e., −∞ dts(t) = −∞ dξ ŝ(ξ) , the
second integral on the right is expressed as:
Z ∞
ds 2 Z ∞
2
dt =
dξ i2πξŝ(ξ) = (2π)2 σξ2
dt
−∞
−∞
R
∞
Therefore, −∞ dt t s(t) ds
dt ≤ 2πσt σξ . Now, integration by parts yields
∞
ds 1
dt t s(t)
=
dt
2
−∞
Z
19
Z
∞
d |s(t)|2 dt t
dt
−∞
=
Z
∞
1
1
1 ∞
1
t |s(t)|2 −
dt |s(t)|2 = 0 − = −
2
2 −∞
2
2
−∞
⇒ σt σξ ≥
The Cauchy-Schwarz inequality becomes an equality if ts(t) and
For Gaussian signals,
r
ds
d α
=
exp(−αt2 )
dt
dt
π
r
α
exp(−αt2 ) = −2αts(t)
= −2αt
π
p
1
2
Hence, σt σξ = 4π
for s(t) = α
π exp(−αt ).
20
ds
dt
1
4π
are collinear.