Download Orthogonal Diagonalization of Symmetric Matrices

MATH10212 • Linear Algebra • Brief lecture notes
57
Gram–Schmidt Process enables us to find an orthogonal basis of a subspace. Let ~u1 , . . . , ~uk be a basis of a subspace V of Rn . We begin the process
of finding a new, orthogonal, basis ~v1 , . . . , ~vk by setting ~v1 = ~u1 .
Then we seek ~v2 = ~u2 + c1 ~u1 with unknown coeff. c1 ; we need ~v2 · ~v1 = 0,
u1
that is, ~u2 · ~u1 + c1 (~u1 · ~u1 ) = 0, whence c1 = − ~~uu21 ·~
u1 · ~u1 6= 0 as
·~
u1 (note that ~
~u1 6= ~0 being a vector in a basis).
Next we seek ~v3 = ~u3 + d1~v1 + d2~v2 with unknown coeff. d1 , d2 ; we need
~v3 · ~v1 = 0, that is, ~u3 · ~v1 + c1 (~v1 · ~v1 ) = 0 (the term with ~v2 · ~v1 = 0 vanishes),
·~
v1
whence d1 = − ~u~v13·~
v3 · ~v2 = 0, that is, ~u3 · ~v2 + c1 (~v2 · ~v2 ) = 0
v1 ; also we need ~
·~
v2
(the term with ~v1 · ~v2 = 0 vanishes), whence d2 = − ~u~v23·~
v2 .
...And so on, at each step we seek ~vj = ~uj + linear comb. of preceding
~v1 , . . . , ~vj−1 with unknown coeff., which are easily found.
Orthogonal Diagonalization of Symmetric Matrices
Definition. A square matrix A is orthogonally diagonalizable if there exists an orthogonal matrix Q such that QT AQ = D is a diagonal matrix.
Remarks. Since QT = Q−1 for orthogonal Q, the equality QT AQ = D is
the same as Q−1 AQ = D, so A ∼ D, so this a special case of diagonalization: the diagonal entries of D are eigenvalues of A, and the columns of
Q are corresponding eigenvectors. The only difference is the additional requirement that Q be orthogonal, which is equivalent to the fact that those
eigenvectors – columns of Q – form an orthonormal basis of Rn .
Theorem 5.17.
metric.
If a matrix is orthogonally diagonalizable, then it is sym-
Proof. We have QT AQ = D; times Q on the left, and QT on the right
gives A = QDQT (since QT = Q−1 ). Then AT = (QDQT )T = (QT )T DT QT =
QDQT = A, so A is symmetric.
Theorem 5.18. All eigenvalues (all roots of the characteristic polynomial)
of a symmetric matrix are real.
Theorem 5.19. Eigenvectors of a symmetric matrix corresponding to different eigenvalues are orthogonal.
Proof. Let AT = A have eigenvectors ~v1 and ~v2 for eigenvalues λ1 6= λ2 .
We compute the dot product (A~v1 ) ·~v2 = (λ1~v1 ) ·~v2 = λ1 (~v1 ·~v2 ). On the other
hand, the left-hand side can be written as a matrix product: (A~v1 ) · ~v2 =
MATH10212 • Linear Algebra • Brief lecture notes
58
(A~v1 )T ~v2 = ~v1T AT ~v2 = ~v1T (A~v2 ) = ~v1 · (λ2~v2 ) = λ2 (~v1 · ~v2 ). Thus, λ1 (~v1 · ~v2 ) =
λ2 (~v1 · ~v2 ). Since λ1 6= λ2 by hypothesis, we must have ~v1 · ~v2 = 0.
Theorem 5.20. Every symmetric matrix is orthogonally diagonalizable.
Method for orthogonal diagonalization of a symmetric matrix. Find
eigenvalues of A. Find the eigenspace for each eigenvalue. For repeated
eigenvalues (when the dimension of the eigenspace is greater than 1) apply Gram–Schmidt orthogonalization to find an orthogonal basis. Together,
these orthogonal bases of eigenspaces form an orthogonal basis of Rn . Normalize, dividing each vector of the basis by its length. The resulting orthonormal basis can be taken as columns of Q such that QT AQ = Q−1 AQ =
D, where D is diagonal with the eigenvalues of A on the diagonal, in the
order corresponding to the order of their eigenvectors as columns of Q.


1 2 2
Example. For A = 2 1 2 the characteristic polynomial is
2 2 1
¯
¯
¯1 − λ
2
2 ¯¯
¯
1−λ
2 ¯¯ =
det(A − λI) = ¯¯ 2
¯ 2
2
1 − λ¯
(1 − λ)3 + 8 + 8 − 4(1 − λ) − 4(1 − λ) − 4(1 − λ) = · · · = −(λ − 5)(λ + 1)2 . Thus,
eigenvalues are 5 and −1.

   
x1
0
2 2 2
Eigenspace E−1 : (A−(−1)I)~x = ~0; 2 2 2 x2  = 0; x1 = −x2 −x3 ,
x3  0
 2 2  2
−s
−
t


where x2 , x3 are free var.; E−1 =  s  | s, t ∈ R ;



 
 t 
−1
−1 

a basis of E−1 : ~u1 =  1  , ~u2 =  0  .


0
1
Apply Gram–Schmidt to orthogonalize: ~v1 = ~u1 ; seek ~v2 = ~u2 + c~
v1 ; 
−1/2
·~
v1
1

−1/2.
for ~v2 · ~v1 = 0 obtain c = − ~u~v12·~
=
−
,
thus,
~
v
=
~
u
−
(1/2)~
v
=
2
2
1
v1
2

 
  1
−4 2
2
x1
0
Eigenspace E5 : (A − 5I)~x = ~0;  2 −4 2  x2  = 0; solve this
2
2 −4 x3 
0 

 t

system....: x1 = x2 = x3 , where x3 is a free var.; E5 = t | t ∈ R ;


t
 
 1 
1 .
a basis of E5 :


1
(Note, E5 is automatically orthogonal to E−1 .)
MATH10212 • Linear Algebra • Brief lecture notes
59
Together, we have an orthogonal basis of R3 consisting of eigenvectors:
  
  
1
−1
−1/2
 1  , −1/2 , 1 .
0
1
1
Normalize:
√   √ 
√  

−1/√6
1/√3
−1/√ 2
 1/ 2  , −1/ 6 , 1/ 3 .
p
√
0
1/ 3
2/3
√
√
√ 

−1/√ 2 −1/√6 1/√3

Let Q =  1/ 2 −1/
p 6 1/√3 , which is an orthogonal matrix;
0
2/3 1/ 3
−1 0 0
then QT AQ =  0 −1 0.
0
0 5
Orthogonal Complements and Orthogonal Projections
Definition Let W be a subspace of Rn . We say that a vector ~v in Rn
is orthogonal to W if ~v is orthogonal to every vector in W . The set of all
vectors that are orthogonal to W is called the orthogonal complement of W ,
denoted W ⊥ . That is,
~ = 0 for all w
W ⊥ = {~v in Rn | ~v · w
~ in W }
Theorem 5.9
Let W be a subspace of Rn .
a. W ⊥ is a subspace of Rn .
b. (W ⊥ )⊥ = W .
c. W ∩ W ⊥ = {~0}.
d. If W = span(w
~ 1, . . . , w
~ k ), then ~v is in W ⊥ if and only if ~v · w
~ i = 0 for all
i = 1, . . . , k.
Theorem 5.10 Let A be an m × n matrix. Then the orthogonal complement of the row space of A is the null space of A, and the orthogonal complement of the column space of A is the null space of AT :
(row(A))⊥ =null(A) and (col(A))⊥ =null(AT )
60
MATH10212 • Linear Algebra • Brief lecture notes
Orthogonal Projections
Theorem 5.11
The Orthogonal Decomposition Theorem
Let V be a subspace of Rn and let ~u be a vector in Rn . Then there are
unique vectors ~v ∈ V and w
~ ∈ V ⊥ such that ~u = ~v + w.
~
In fact: if ~v1 , . . . , ~vr is an orthogonal basis of V , then an orthogonal basis
~vr+1 , . . . , ~vn of V ⊥ must have n−r vectors, and together these n vectors form
an orthogonal basis ~v1 , . . . , ~vr , ~vr+1 , . . . , ~vn of Rn
Definition In Theorem 5.11, the vector ~v is called the orthogonal projection of ~u onto V , and the length of w
~ is called the distance from ~u to V .
Method for finding the orthogonal projection and the distance (given
a subspace V of Rn and some vector ~u ∈ Rn ).
Choose an orthogonal basis ~v1 , . . . , ~vr of V (if V is given as a span of some
non-orthogonal vectors, apply Gram–Schmidt first to obtain an orthogonal
basis of V ); we know that there is an orthogonal basis ~vr+1 , . . . , ~vn of V ⊥
such that
~v1 , . . . , ~vr , ~vr+1 , . . . , ~vn
is an orthogonal basis of Rn (but we do not really need these ~vr+1 , . . . , ~vn !).
Then
~u =
r
X
ai~vi +
i=1
n
X
bj ~vj .
j=r+1
We now find the coefficients ai . For that, take dot product with ~vi0 for each
i0 = 1, . . . , r: on the right only one term does not vanish, since the ~vk are
orthogonal to each other:
~u · ~vi0 = ai0 (~vi0 · ~vi0 ),
whence
~u · ~vi0
.
~vi0 · ~vi0
a i0 =
Having found these ai0 for each i0 = 1, . . . , r, we now have the orthogonal
projection of ~u onto V :
r
X
~v =
ai~vi .
i=1
The orthogonal component is found by subtracting:
w
~ = ~u − ~v ∈ V ⊥ ,
and the distance from ~u to V is kwk.
~
MATH10212 • Linear Algebra • Brief lecture notes
Corollary 5.12
61
If W is a subspace of Rn , then
(W ⊥ )⊥ = W
Theorem 5.13
If W is a subspace of Rn , then
dim W + dim W ⊥ = n
Corollary 5.14
The Rank Theorem
If A is an m × n matrix, then
rank (A) + nullity (A) = n
Vector Spaces and Subspaces
Definition Let V be a set on which two operations, called addition and
scalar multiplication, have been defined. If ~u and ~v are defined in V , the
sum of ~u and ~v is denoted by ~u + ~v , and if c is a scalar, the scalar multiple of
~u is denoted by c~u. If the following axioms hold for all ~u, ~v and w
~ in V and
for all scalars c and d, then V is called a vector space and its elements are
called vectors.
1. ~u + ~v is in V .
Closure under addition
2. ~u + ~v = ~v + ~u
Commutativity
3. (~u + ~v ) + w
~ = ~u + (~v + w)
~
Associativity
4. There exists an element ~0 in V , called a zero vector such that ~u +~0 = ~u.
5. For each ~u in V , there is an element −~u in V such that ~u + (−~u) = ~0.
6. c~u is in V .
Closure under scalar multiplication
7. c(~u + ~v ) = c~u + c~v
Distributivity
8. (c + d)~u = c~u + d~u
Distributivity
9. c(d~u) = (cd)~u
10. 1~u = ~u
62
MATH10212 • Linear Algebra • Brief lecture notes
Theorem 6.1
Let V be a vector space, ~u a vector in V , and c a scalar.
a. 0~u = ~0
b. c~0 = ~0
c. (−1)~u = −~u
d. If c~u = ~0, then c = 0 or ~u = ~0.
Subspaces
Definition A subset W of a vector space V is called a subspace of V if W
is itself a vector space with the same scalars, addition and scalar multiplication as V .
Theorem 6.2 Let V be a vector space and let W be a nonempty subset of
V . Then W is a subspace of V if and only if the following conditions hold:
a. If ~u and ~v are in W , then ~u + ~v is in W .
b. If ~u is in W and c is a scalar, then c~u is in W .
Example 6.14 If V is a vector space, then V is clearly a subspace of itself. The set {~0}, consisting of only the zero vector, is also a subspace of V ,
called the zero subspace. To show this, we simply note that the two closure
conditions of Theorem 6.2 are satisfied:
~0 + ~0 = ~0
and
c~0 = ~0
for any scalar c
The subspaces {~0} and V are called the trivial subspaces of V .
Spanning Sets
Definition If
S = {~v1 , ~v2 , . . . , ~vk }
is a set of vectors in a vector space V , then the set of all linear combinations
of
~v1 , ~v2 , . . . , ~vk
is called the span of
~v1 , ~v2 , . . . , ~vk
and is denoted by
span(~v1 , ~v2 , . . . , ~vk )
or span(S). If V =span(S), then S is called a spanning set for V and V is
said to be spanned by S.
MATH10212 • Linear Algebra • Brief lecture notes
Theorem 6.3
63
Let ~v1 , ~v2 , . . . , ~vk be vectors in a vector space V .
a. span(~v1 , ~v2 , . . . , ~vk ) is a subspace of V .
b. span(~v1 , ~v2 , . . . , ~vk ) is a smallest subspace of V that contains ~v1 , ~v2 , . . . , ~vk .
Linear Independence, Basis and Dimension
Linear Independence
Definition A set of vectors S = {~v1 , ~v2 , . . . , ~vk } in a vector space V is linearly dependent if there are scalars c1 , c2 , . . . , ck at least one of which is not
zero, such that
c1~v1 + c2~v2 + · · · + ck~vk = ~0
A set of vectors that is not linearly dependent is said to be linearly independent.
Theorem 6.4 A set of vectors S = {~v1 , ~v2 , . . . , ~vk } in a vector space V is
linearly dependent if and only if at least one of the vectors can be expressed
as a linear combination of the others.
Bases
Definition A subset B of a vector space V is a basis for V if
1. B spans V and
2. B is linearly independent.