Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
6 Orthogonality and Least Squares 6.5 LEAST-SQUARES PROBLEMS © 2012 Pearson Education, Inc. LEAST-SQUARES PROBLEMS Definition: If A is m n and b is in , a leastn A x b x̂ squares solution of is an in such that m b Axˆ b Ax for all x in n . The most important aspect of the least-squares problem is that no matter what x we select, the vector Ax will necessarily be in the column space, Col A. So we seek an x that makes Ax the closest point in Col A to b. See the figure on the next slide. © 2012 Pearson Education, Inc. Slide 6.5- 2 LEAST-SQUARES PROBLEMS Solution of the General Least-Squares Problem Given A and b, apply the Best Approximation Theorem to the subspace Col A. Let © 2012 Pearson Education, Inc. b̂ projCol Ab Slide 6.5- 3 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Because b̂ is in the column space A, the equation Ax bˆ n x̂ is consistent, and there is an in such that Ax̂ bˆ ----(1) Since b̂ is the closest point in Col A to b, a vector x̂ is a least-squares solution of Ax b if and only if x̂ satisfies (1). n Such an x̂ in is a list of weights that will build b̂ out of the columns of A. See the figure on the next slide. © 2012 Pearson Education, Inc. Slide 6.5- 4 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Suppose x̂ satisfies Ax̂ bˆ . By the Orthogonal Decomposition Theorem, the projection b̂ has the property that b bˆ is orthogonal to Col A, so b Axˆ is orthogonal to each column of A. ˆ 0, If aj is any column of A, then a j (b Ax) T ˆ . and a j (b Ax) © 2012 Pearson Education, Inc. Slide 6.5- 5 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Since each a Tj is a row of AT, ˆ 0 AT (b Ax) Thus ----(2) AT b AT Axˆ 0 A Ax̂ A b T T These calculations show that each least-squares solution of Ax b satisfies the equation T T ----(3) A Ax A b The matrix equation (3) represents a system of equations called the normal equations for Ax b . A solution of (3) is often denoted by x̂ . © 2012 Pearson Education, Inc. Slide 6.5- 6 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Theorem 13: The set of least-squares solutions of Ax b coincides with the nonempty set of solutions of the T T normal equation A Ax A b. Proof: The set of least-squares solutions is nonempty and each least-squares solution x̂ satisfies the normal equations. Conversely, suppose x̂ satisfies AT Ax̂ AT b. Then x̂ satisfies (2), which shows that b Axˆ is orthogonal to the rows of AT and hence is orthogonal to the columns of A. Since the columns of A span Col A, the vector b Axˆ is orthogonal to all of Col A. © 2012 Pearson Education, Inc. Slide 6.5- 7 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Hence the equation ˆ b Axˆ (b Ax) is a decomposition of b into the sum of a vector in Col A and a vector orthogonal to Col A. By the uniqueness of the orthogonal decomposition, Ax̂ must be the orthogonal projection of b onto Col A. That is, Ax̂ bˆ and x̂ is a least-squares solution. © 2012 Pearson Education, Inc. Slide 6.5- 8 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Example 1: Find a least-squares solution of the inconsistent system Ax b for 4 0 2 A 0 2 ,b 0 1 1 11 Solution: To use normal equations (3), compute: 4 0 4 0 1 17 1 T A A 0 2 0 2 1 1 5 1 1 © 2012 Pearson Education, Inc. Slide 6.5- 9 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM 2 4 0 1 19 T A b 0 0 2 1 11 11 Then the equation A Ax A b becomes T T 17 1 x1 19 1 5 x 11 2 © 2012 Pearson Education, Inc. Slide 6.5- 10 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Row operations can be used to solve the system on the previous slide, but since ATA is invertible and 2 2 , it is probably faster to compute 1 5 1 ( A A) 84 1 17 1 T and then solve A Ax A b as T T x̂ ( A A) A b T 1 T 1 5 1 19 1 84 1 = 84 1 17 11 84 168 2 © 2012 Pearson Education, Inc. Slide 6.5- 11 SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Theorem 14: Let A be an m n matrix. The following statements are logically equivalent: a. The equation Ax b has a unique least-squares m solution for each b in . b. The columns of A are linearly independent. c. The matrix ATA is invertible. When these statements are true, the least-squares solution x̂ is given by T 1 T ----(4) x̂ ( A A) A b When a least-squares solution x̂ is used to produce Ax̂ as an approximation to b, the distance from b to Ax̂ is called the least-squares error of this approximation. © 2012 Pearson Education, Inc. Slide 6.5- 12 ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS Example 2: Find a least-squares solution of Ax b for 1 6 1 1 2 2 ,b A 1 1 1 1 7 6 Solution: Because the columns a1 and a2 of A are orthogonal, the orthogonal projection of b onto Col A is given by b a1 b a2 8 45 b̂ a1 a 2 a1 a 2 ----(5) a1 a1 © 2012 Pearson Education, Inc. a2 a2 4 90 Slide 6.5- 13 ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS 2 3 1 2 1 1 2 1/ 2 5 / 2 2 7 / 2 11/ 2 Now that b̂ is known, we can solve Ax̂ bˆ . But this is trivial, since we already know weights to place on the columns of A to produce b̂. It is clear from (5) that © 2012 Pearson Education, Inc. 8/ 4 2 xˆ 45 / 90 1/ 2 Slide 6.5- 14 ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS Theorem 15: Given an m n matrix A with linearly independent columns, let A QR be a QR factorization m of A. Then, for each b in , the equation Ax b has a unique least-squares solution, given by 1 T ----(6) x̂ R Q b Proof: Let x̂ R Q b. 1 Then T Axˆ QRxˆ QRR 1QT b QQT b © 2012 Pearson Education, Inc. Slide 6.5- 15 ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS The columns of Q form an orthonormal basis for Col A. Hence, by Theorem 10, QQTb is the orthogonal projection b̂ of b onto Col A. Then Ax̂ bˆ , which shows that x̂ is a least-squares solution of Ax b. The uniqueness of x̂ follows from Theorem 14. © 2012 Pearson Education, Inc. Slide 6.5- 16