Download Section 6.5

6 Orthogonality and Least
Squares
6.5
LEAST-SQUARES PROBLEMS
© 2012 Pearson Education, Inc.
LEAST-SQUARES PROBLEMS
 Definition: If A is m  n and b is in
, a leastn
A
x

b
x̂
squares solution of
is an in
such that
m
b  Axˆ  b  Ax
for all x in
n
.
 The most important aspect of the least-squares
problem is that no matter what x we select, the vector
Ax will necessarily be in the column space, Col A.
 So we seek an x that makes Ax the closest point in
Col A to b. See the figure on the next slide.
© 2012 Pearson Education, Inc.
Slide 6.5- 2
LEAST-SQUARES PROBLEMS
 Solution of the General Least-Squares Problem
 Given A and b, apply the Best Approximation
Theorem to the subspace Col A.
 Let
© 2012 Pearson Education, Inc.
b̂  projCol Ab
Slide 6.5- 3
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Because b̂ is in the column space A, the equation Ax  bˆ
n
x̂
is consistent, and there is an in such that
Ax̂  bˆ
----(1)
 Since b̂ is the closest point in Col A to b, a vector x̂ is a
least-squares solution of Ax  b if and only if x̂
satisfies (1).
n
 Such an x̂ in
is a list of weights that will build b̂ out
of the columns of A. See the figure on the next slide.
© 2012 Pearson Education, Inc.
Slide 6.5- 4
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Suppose x̂ satisfies Ax̂  bˆ .
 By the Orthogonal Decomposition Theorem, the
projection b̂ has the property that b  bˆ is orthogonal
to Col A, so b  Axˆ is orthogonal to each column of A.
ˆ  0,
 If aj is any column of A, then a j (b  Ax)
T
ˆ .
and a j (b  Ax)
© 2012 Pearson Education, Inc.
Slide 6.5- 5
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Since each a Tj is a row of AT,
ˆ 0
AT (b  Ax)
 Thus
----(2)
AT b  AT Axˆ  0
A Ax̂  A b
T
T
 These calculations show that each least-squares
solution of Ax  b satisfies the equation
T
T
----(3)
A Ax  A b
 The matrix equation (3) represents a system of
equations called the normal equations for Ax  b .
 A solution of (3) is often denoted by x̂.
© 2012 Pearson Education, Inc.
Slide 6.5- 6
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Theorem 13: The set of least-squares solutions of Ax  b
coincides with the nonempty set of solutions of the
T
T
normal equation A Ax  A b.
 Proof: The set of least-squares solutions is nonempty and
each least-squares solution x̂ satisfies the normal
equations.
 Conversely, suppose x̂ satisfies AT Ax̂  AT b.
 Then x̂ satisfies (2), which shows that b  Axˆ is
orthogonal to the rows of AT and hence is orthogonal to
the columns of A.
 Since the columns of A span Col A, the vector b  Axˆ is
orthogonal to all of Col A.
© 2012 Pearson Education, Inc.
Slide 6.5- 7
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Hence the equation
ˆ
b  Axˆ  (b  Ax)
is a decomposition of b into the sum of a vector in Col
A and a vector orthogonal to Col A.
 By the uniqueness of the orthogonal decomposition, Ax̂
must be the orthogonal projection of b onto Col A.
 That is, Ax̂  bˆ and x̂ is a least-squares solution.
© 2012 Pearson Education, Inc.
Slide 6.5- 8
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Example 1: Find a least-squares solution of the
inconsistent system Ax  b for
4 0
 2




A  0 2 ,b  0


 
 1 1
11
 Solution: To use normal equations (3), compute:
4 0
 4 0 1 
17 1
T

A A
0 2 




0
2
1
1
5

 1 1 



© 2012 Pearson Education, Inc.
Slide 6.5- 9
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 2
 4 0 1   19 
T
A b
0  



0
2
1

 11 11
 
 Then the equation A Ax  A b becomes
T
T
17 1  x1  19 
 1 5  x   11

 2  
© 2012 Pearson Education, Inc.
Slide 6.5- 10
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Row operations can be used to solve the system on the
previous slide, but since ATA is invertible and 2  2, it
is probably faster to compute
1  5 1
( A A)  
84  1 17 
1
T
and then solve A Ax  A b as
T
T
x̂  ( A A) A b
T
1
T
1  5 1 19 1  84  1
= 
   



84  1 17  11 84 168  2
© 2012 Pearson Education, Inc.
Slide 6.5- 11
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Theorem 14: Let A be an m  n matrix. The following

statements are logically equivalent:
a. The equation Ax  b has a unique least-squares
m
solution for each b in .
b. The columns of A are linearly independent.
c. The matrix ATA is invertible.
When these statements are true, the least-squares
solution x̂ is given by
T
1 T
----(4)
x̂  ( A A) A b
When a least-squares solution x̂ is used to produce Ax̂
as an approximation to b, the distance from b to Ax̂ is
called the least-squares error of this approximation.
© 2012 Pearson Education, Inc.
Slide 6.5- 12
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 Example 2: Find a least-squares solution of Ax  b for
1 6 
 1
1 2 
 2
,b   
A
1 1
 1
1 7 
 6


 
 Solution: Because the columns a1 and a2 of A are
orthogonal, the orthogonal projection of b onto Col A is
given by
b a1
b a2
8
45
b̂ 
a1 
a 2  a1  a 2 ----(5)
a1 a1
© 2012 Pearson Education, Inc.
a2 a2
4
90
Slide 6.5- 13
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 2   3  1
 2   1 

1


 
 2   1/ 2   5 / 2 
 2  7 / 2  11/ 2 
  
 

 Now that b̂ is known, we can solve Ax̂  bˆ .
 But this is trivial, since we already know weights to
place on the columns of A to produce b̂.
 It is clear from (5) that
© 2012 Pearson Education, Inc.
 8/ 4   2 
xˆ  



45
/
90
1/
2

 

Slide 6.5- 14
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 Theorem 15: Given an m  n matrix A with linearly
independent columns, let A  QR be a QR factorization
m
of A. Then, for each b in
, the equation Ax  b
has a unique least-squares solution, given by
1 T
----(6)
x̂  R Q b
 Proof: Let x̂  R Q b.
1
 Then
T
Axˆ  QRxˆ  QRR 1QT b  QQT b
© 2012 Pearson Education, Inc.
Slide 6.5- 15
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 The columns of Q form an orthonormal basis for Col A.
 Hence, by Theorem 10, QQTb is the orthogonal
projection b̂ of b onto Col A.
 Then Ax̂  bˆ , which shows that x̂ is a least-squares
solution of Ax  b.
 The uniqueness of x̂ follows from Theorem 14.
© 2012 Pearson Education, Inc.
Slide 6.5- 16