* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download s06.pdf
Survey
Document related concepts
Capelli's identity wikipedia , lookup
Rotation matrix wikipedia , lookup
Linear least squares (mathematics) wikipedia , lookup
Principal component analysis wikipedia , lookup
Jordan normal form wikipedia , lookup
Eigenvalues and eigenvectors wikipedia , lookup
Four-vector wikipedia , lookup
Singular-value decomposition wikipedia , lookup
Determinant wikipedia , lookup
Matrix (mathematics) wikipedia , lookup
Perron–Frobenius theorem wikipedia , lookup
Non-negative matrix factorization wikipedia , lookup
Orthogonal matrix wikipedia , lookup
Matrix calculus wikipedia , lookup
System of linear equations wikipedia , lookup
Cayley–Hamilton theorem wikipedia , lookup
Transcript
Session 6 Direct Solution Methods for Systems of Linear Algebraic Equations Ernesto Gutierrez-Miravete Spring 2003 1 Introduction Systems of linear algebraic equations occur when several unknown quantities are interliked by means of linear relationships. Such systems appear during the analysis of very many engineering and scientic problems. Well known engineering examples include kinematics, statics and dynamics of mechanical systems, the disturbed motion of bodies, the exural vibrations of structural members and the nodal equations obtained form applying nite dierence, nite volume or nite element methodologies for the solution of initial-boundary value problems in continuum mechanics. The goal of analysis is to determine the values of the unknowns in the system. Solution methods are classied as direct or iterative. Direct methods compute a solution in a predetermined number of steps while iterative methods proceed by computing a sequence of iincreasingly improved approximations. This session describes the most important direct methods used for the solution of systems of linear algebraic equations. 2 Systems of Linear Algebraic Equations A linear system of n simultaneous algebraic equations for n variables x ; x ; :::; x is a set of equations labelled E ; E ; :::; E of the form a x + a x + ::: + a x = b a x + a x + ::: + a x = b 1 1 2 2 n n 11 1 12 2 1n n 1 21 1 22 2 2n n 2 : : ::: : : + a x + ::: + a x = b The key to facilitate the solution of the system is to transform it into an equivalent system by means of permissible operations. An equivalent system can be obtained from the above system by performing some or all of the operations below. an1 x1 n2 2 nn 1 n n 1) Equation E can be multiplied by a constant . The resulting equation can the be used instead of E . I.e. E ! E 2) Equation E can be multiplied by a constant and the result added to equation E . The resulting equation can then be used instead of E . I.e. E + E ! E 3) Equations E and E can be exchanged. I.e. E $ E Exercise 1. Example 6.1.1 in B-F. Transform the system x + x + 3x = 4 2x + x x + x = 1 3x x x + 2x = 3 x + 2x + 3x x =4 into the equivalent triangular system i i i i j i i i j j i 1 1 1 2 2 2 1 i i i 4 3 4 3 4 2 3 4 + x + 3x = 4 x x 5x = 7 3x + 13x = 13 13x = 13 simply by performing the following sequence of rearrangements (E 2E ) ! E , (E 3E ) ! E , (E + E ) ! E , (E 4E ) ! E , and (E + 3E ) ! E (check!). Note that the appropriate coeÆcients used in each rearrangement are determined in part from the values of a . For example, in (E 2E ) the coeÆcient 2 = a =a and in (E 3E ) the coeÆcient 3 = a =a . The number a is the pivot element in this process. The idea is to proceed to eliminate rst the terms containing x from all equations but the rst one. Then, to eliminate all terms containing x from all equations except the rst two. Then, to eliminate all terms containing x from all equations except the rst three and so on. For this process to work, it is essential that all the pivot elements be non-zero. Row exchanges can be used at any stage to try to obtain non-zero pivots. However, if no non-zero pivot can be found at some stage of the elimination process, it can be shown that the system has no unique solution. Once the system is in triangular form, it can be easily solved by a backward substitution process starting with the last equation and moving upwards (check!). Here x = 1, x = 0, x = 2 and x = 1. The process of rst triangularizing the original system of equations and then solving the resulting triangular system by backsubstitution is the essence of the Gauss elimination method. Exercise 2. Use Gauss elimination to nd the solution to the system 2x + 3x x = 5 x1 2 2 4 3 4 3 4 4 2 1 3 4 1 4 3 2 2 ij 31 3 4 1 11 2 21 1 2 3 4 11 3 1 11 1 2 3 4 2 1 1 2 3 2 3 4x + 4x 3x = 3 2x + 3x x = 1 Denition of n by m Matrix. A rectangular array of elements with n rows and m columns where value and location of elements are meaningful. An n m matrix A is written as 1 2 1 A = (a ij 3 2 3 2 a11 6 a21 6 6 6 ) = 66 :: 6 6 4 : a12 a22 : : : an2 an1 3 ::: a1m 7 ::: a2m 7 7 ::: : 7 7 ::: : 7 7 7 ::: : 5 ::: anm 1 n matrices are called n-dimensional row vectors y = [ y y ::: y ] while n 1 matrices are called n-dimensional column vectors. 1 2 n 2 3 x1 6 7 x2 7 6 6 7 6 : 7 6 x = 6 : 77 6 7 6 4 : 7 5 xn In the sequel we will focus exclusively on the case where n = m (i.e the number of unknowns is equal to the numebnr of equations). Therefore, the n n system a x + a x + ::: + a x = b a x + a x + ::: + a x = b 11 1 12 2 1n n 1 21 1 22 2 2n n 2 : an1 x1 +a : ::: n2 + ::: + a x2 nn : xn can be represented as Ax where 2 a11 6 a21 6 6 6 : A=6 6 : 6 6 4 : an1 =b a12 a22 : : : an2 3 3 ::: a1m 7 ::: a2m 7 7 ::: : 7 7 ::: : 7 7 7 ::: : 5 ::: anm =b : n 2 3 x1 6 7 x2 7 6 6 7 6 : 7 6 x = 6 : 77 6 7 6 4 : 7 5 xn and 2 3 b1 6 7 b2 7 6 6 7 6 : 7 6 b = 6 : 77 6 7 6 4 : 7 5 bn The augmented matrix A aug Aaug = [A; b] is the n n + 1 rectangular matrix 2 a11 6 a21 6 6 6 = [A; b] = 66 :: 6 6 4 : a12 a22 : : : an2 an 2 1 a11 a12 6 a21 a22 6 6 6 = 66 :: :: 6 6 4 : : an1 an2 x1 ::: a1n ::: a2n ::: : ::: : ::: : ::: ann 3 b1 7 b1 7 7 : 7 7 : 7 7 7 : 5 b1 = 3 ::: a1n a1n+1 7 ::: a2n a2n+1 7 7 7 ::: : : 7 7 ::: : : 7 7 5 ::: : : ::: ann a1n+1 In terms of the augmented matrix Gaussian elimination works as follows. As we eliminate from all equations but the rst one, new coeÆcientes a develop according to the formulae (1) ij (1) a11 (1) a1j 2 = aa (j > 1) 1j 11 =a a a (i; j 2) from all equations except the rst two, the corresponding formulae (1) aij If now, we eliminate x are =1 ij i1 (1) 1j (2) a22 =1 4 (2) a2j = (1) a2j (j > 2) (1) a22 =a a a (i; j 3) and so on. Backsubstitution is trivial. The result obtained by Gauss elimination can be improved if the necessary corrections are relatively small. Say the (approximate) solution to the system Ax = b has been found and it is x0 . Since the real solution is x = x0 + Æ and the residual = b Ax0 then AÆ = Gaussian elimination can now be performed in this system to nd the correction Æ. Maple has capabilities for matrix work. The matrix command creates matrices. Operations on rows are performed with the addrow and swaprow commands. The command gausselim produces the reduced triangular matrix and the command backsub solves it. Gaussian Elimination with Backward Substitution Algorithm. Give the matrix A and the vector b Transform the orignal system into an equivalent triangular system. Solve the triangular system by backsubstitution. Concerning the number of operations performed in the elimination stage one has (2n + 3n 5n)=6 multiplications/divisions and (n n)=3 additions/substractions. The backward substitution requires (n +n)=2 multiplications/divisions and (n n)=3 additions/substractions. Thus, in total we have n =3+n n=3 n multiplications/divisions and n =3+n =2 5n=6 n additions/substractions. This total number of operations increases rapidly with the size of the problem and shows that roundo error can become a signicant problem. (2) (1) aij (1) i2 ij (2) 2;j 3 2 3 2 2 3 2 3 3 2 3 3 Pivoting Even if all pivots are non-zero, when pivots are small in relative magnitude, roundo errors can be dramatically magnied. If the candidate pivot a is small relative to the entries a for k i n and k j n the partial pivoting or maximal column pivoting strategy can be used. This consists of selecting a larger element (usually the largest) a in the column where the candidate pivot lies as pivot instead and then swapping rows k and p. Exercise 3. Example 6.2.2 in B-F. For the system 0:00300x + 59:14x = 59:17 5:291x 6:130x = 46:78 kk ij pq 1 2 1 2 5 the rst pivot a is relatively small in magnitude so we look for the maximum coeÆcient in column 1 (5.291), swap the rows and triangularize to get 11 5:291x 1 6:130x = 46:78 59:14x 59:14 2 2 Gaussian Elimination with Partial Pivoting Algorithm. Give the matrix A and the vector b. Start triangularization process. If a pivot is small, exchange row with another one containing a larger number. Solve the triangular system by backsubstitution. Partial pivoting can still fail even if the pivots are large in absolute size. In scaled partial pivoting the maximum coeÆcient in each row i is identied and called the i-th row scaling factor, s is rst computed for each row as the maximum coeÆcient in the row. Then, scaled pivots are calculated with reference to the scaling factors as ja j=s . The largest scaled pivot is identied and the associated rows are swapped.. Exercise 4. Example 6.2.3 in B-F. For the system i ij i 30x + 591400x = 591700 5:291x 6:130x = 46:78 1 2 1 2 the value of rst scaled pivot a =s = 30=591400 5 10 is relatively small in magnitude compared to a =s = 5:291=6:13 0:86 so swap the rows to get 11 21 5 1 2 5:291x 6:130x = 46:78 30x + 591400x = 591700 1 1 2 2 which now can readily (and accurately) be reduced. Gaussian Elimination with Scaled Partial Pivoting Algorithm. Give the matrix A and the vector b. Start triangularization process. Determine the scaling factors. If a scaled pivot is small, exchange row with another one containing a larger number. Solve the triangular system by backsubstitution. 6 For the highest possible accuracy, complete pivoting is possible. In complete pivoting the largest matrix element is selected and its row (principal row) and column (principal column) are identied. Operations are then performed to make all coeÆcients in the principal column = 0 so it can be eliminated producing a reduced matrix. The process is repeated on the reduced matrix to generate a further reduced matrix and so on. Finally, a simple rearrangement of the matrix formed by all the principal rows obtained produces a triangular system which is easily solved 4 Review of Foundations of Linear Algebra This section contains a review of foundations of linear algebra which are essential for a full understanding of numerical linear algebra. Two matrices A and B are equal if they have the same size (i.e. n m) and a = b for each i = 1; 2; :::; n and j = 1; 2; :::; m. If A and B are n m, their sum C = A + B is a matrix with elements c = a + b for each i = 1; 2; :::; n and j = 1; 2; :::; m. If is a real number, the scalar multiplication of A with is A = (a ) for each i = 1; 2; :::; n and j = 1; 2; :::; m. Theorem of Matrix Properties Involving Sum. Matrix summation has commutative and associative properties. There are also a zero matrix, a unit matrix and the negative matrix of A. Multiplication with scalars is distributive. Let A be n m and B be m p the matrix product C = AB = (c ) = (P a b ) for each i = 1; 2; :::; n and j = 1; 2; :::; m. I.e. entries on the i-th row of A, a are multiplied with corresponding entries in the j -th column of B , b and all the products are added together to form the entry c in the product matrix C . A is square if n = m. A square matrix A is diagonal if the only non-zero elements are a . The identity matrix I is a square diagonal matrix in which each a = 1. A is upper triangular if all elements below the diagonal are zero. A is lower triangular if all elements above the diagonal are zero. A square n n matrix is strictly diagonally dominant if ja j > P 6 ja j, for each i = 1; 2; :::; n. Theorem of Matrix Properties Involving Product. Matrix multiplication has associative and distributive properties but not commutative. Pre-multiplication of a matrix product with a scalar is the same when performed before or after matrix multiplication. A square matrix A is non-singular if there exists another matrix, called its inverse A such that AA = A A = I . Theorem about the Properties of Non-singular Matrices. If A is non-singular n n, then i;j i;j ij ij ij ij m k =1 ij ik kj ik kj ij ii ii n ii j =1;j =i ij 1 1 A 1 1 is unique. 7 A is non-singular and (A ) = A. If B is non-singular n n, then (AB ) = B A . If A = (a ) is a n n square matrix its transpose A = (a ). If A = A , the matrix is 1 1 1 1 1 1 t t ij ji called symmetric. Theorem of Properties Involving Transpose. The transpose of the transpose is the original matrix. The transpose of a matrix product is the product of the transposes and that of a sum is the sum of the transposes. The transpose of an inverse is the inverse of the transpose. A is positive denite if it is symmetric and if x Ax > 0 for every x 6= 0. A square n n matrix is a band matrix if integers p > 1 and q < n exist such that a = 0 whenever i + p j or j + q i. The bandwidth w = p + q 1. If w = 3 the matrix is tridiagonal. The determinant of a n n matrix A is detA = jAj. The following is a list of characteristics of determinants. If A = (a) is 1 1 then detA = a. If A is n n the minor M is the determinant of the reduced n 1 n 1 submatrix obtained by deleting the i-th row and the j -th column from A. The cofactor A associated with M is A = ( 1) M . The determinant of A, detA is computed as t ij ij ij ij detA = jAj = X i+j ij n aij Aij j =1 = X n ij ( 1) i+j aij Mij ( 1) i+j aij Mij j =1 for any i = 1; 2; :::; n,or as detA = jAj = X n aij Aij i=1 = X n i=1 for any j = 1; 2; :::; n. Theorems Involving Determinants. jAj = 0 if any row or column of A has only zeros or if two rows or two columns are the same. Row swapping changes the sign of detA. If a row of A is changed by scalar multiplication with , the new determinant is jAj. Substitution of a row of A by the sum of itself with the scalar multiplication of another the determinant does not change. The determinant of a product is the product of the determinants. The determinant of A is the same as that of A. The determinant of A is the reciprocal of the jAj. For triangular or diagonal matrices jAj = a . Theorem of Fundamental Properties of Linear Systems. If A is a n n square matrix the following are all equivalent 1 t n i=1 8 ii 5 = 0 has a unique solution x = 0. Ax = b has a unique solution for any b. A is non-singular. detA 6= 0 Gaussian elimination can be performed on Ax = b for any b. Ax Matrix Factorization for the Solution of Linear Systems Matrix factorization is of great importance in numerical linear algebra. It consists of representing the original matrix by a product of matrices with special form. It is often useful to factorize a matrix A so that A = LU where L and U are, respectively, lower and upper triangular matrices. Note that if A = LU , in the system Ax = b = LU x = b = Ly = b where y = U x. Therefore, using factorization one can solve a linear system in two easily performed steps. First solve Ly = b for y then solve U x = y for x. Since L and U are triangular, solution is easy and requires much fewer operations (O(n ))! Theorem of LU Factorization. If Gaussian elimination can be performend on Ax = b without row interchanges then A can be factored such that A = LU . LU Factorization Algorithm. Give the matrix A and the vector b. Select l and u such that l u = a For j = 2; 3; ::; n set u = a =l and l = a =u . For i = 2; 3; ::; n 1 select l and u such that l u = a P l u . For j = i + 1; :::; n set u = [a P l u ]=l and l = [a P l u ]=u . Select l and u such that l u = a P l u In the sequence above, if l u or l u equal zero factorization is impossible and if l u = 0 , A is singular. 2 11 11 11 1j 1j 11 11 11 j1 j1 11 i ii ii ii i ij nn nn 11 nn 11 nn nn ii 1 k =1 ik i ik kj ii i nn ii 1 k =1 ij ii ii nn 9 ji 1 k =1 nk kn ji ki 1 k =1 jk ki ii 6 Special Matrices Often, matrices encountered in applications have special forms and one takes advantage of the properties of such matrices in order to develop eective numerical methods for the solution of linear systems. Diagonally dominant positive denite and banded matrices are special matrices of great importance in applications. Theorem of Properties of a Diagonally Dominant Matrix. If A is strictly diagonally dominant then it is non-singular. The associated system Ax = b can be solved by Gaussian elimination without row or column exchanges to give its unique solution and the computations are stable with respect to roundo error. Theorems of Positive Denite Matrices. If A is n n positive denite then A is non-singular. a > 0 for each i = 1; 2; :::; n. max ja j max ja j. (a ) < a a , for each i 6= j . The symmetric matrix A is pd if and only if Gaussian elimination without row interchanges can be performed on Ax = b with all pivots > 0. A is pd if the factorizations A = LDL and if A = LL exist. The factorizations of A into LDL and into LL have important application for the solution of linear systems involving positive denite matrices. LDL Factorization Algorithm. Give the matrix A. For i = 1; 2; ::; n and for j = P1; :::; 1, v = l d ; d = a P l v and while for j = i + 1; :::; n set l = [a l v ]=d Concerning the LL factorization with L = U (Choleski factorization) A = (a ) = LU , one proceeds as follows in terms of the augmented matrix A = [A; b] = (a ) with i = 1; 2; :::; n rows and j = 1; 2; :::; n +1columns. Let L = (l ) be lower triangular and U = (u ) upper triangular with unit diagonal. Determine the coeÆcients l and u according to the following column-row, colum-row scheme. Column: l =a (i 1) ii 1 ij 2 k;j n ii 1 kj i n ii jj t t t t t i j 1 k =1 ij j i 1 j =1 ii ij j i ji ji t jk k i t ij aug ij ij ij ij i1 lij i1 X1 j =a ij (i j > 1) lik ukj k =1 10 ij Row: u1j uij = al (j > 1) 1j 11 = l1 (a X1 i ij ii lik ukj k =1 ) (j > i > 1) and so on. Once the factorization has been accomplished, the lower traingular system Ly = b is solved for y abd then, the upper triangular system U x = y is solved for x. LU Factorization (Choleski) Algorithm Give the matrix A. Set l = pa . For j = 2; :::; n set l = a =l . P For i = 1; 2; ::; n set l = [a ] and for j = i + 1; :::; n set l = [a P l l ]=l Set l = [a P l ] Exercise 5. Example 6.6.4 in B-F. Produce the LDL and LL factorizations of the following matrix 2 3 4 1 1 6 7 A = 4 1 4:25 2:75 5 1 2:75 3:5 Tridiagonal systems of equations are easily solved using Crout factorization A = LU where L = (l ) is lower triangular and U = (u ) upper triangular with unit diagonal (Cholesky factorization for tridiagonal matrix). Consider for example the tridiagonal system represented by the following augmented matrix 2 2 1 0 0 13 6 7 A = 664 01 21 21 01 00 775 0 0 1 2 1 The LU factorization of A yields 2 0 03 2 0 0 0 321 6 0 0 777 666 0 1 0 777 6 1 A = LU = 6 5 40 1 0540 0 1 0 0 1 0 0 0 1 The solution of Lz = b is z = ( ; ; ; 1) and the solution of U x = z is x = (1; 1; 1; 1) . Solution of a Triadiagonal System by Crout Factorization Algorithm. 11 11 j1 j1 11 1 1=2 i 1 k =1 ii k =1 ii i j k ik ji ii n nn ji nn 1 2 k =1 1=2 nk t ij t ij aug 3 2 1 2 1 3 1 4 1 2 4 3 5 4 t 2 3 3 4 t 11 Give the matrix A and vector b. Set l = a ; u = a =l ; z = a For i = 2; :::; n 1 set l = a 11 11 12 12 11 1;n+1 =l11 1 . = ;l a l u ;u =a [a l z ]=l . Set l = a ; l = a l u ; z = [a l z ]=l . Set x = z . For i = n 1; ::; 1 set x = z u x . Exercise 6. Find the solution to the following system of equations 1 i;i i;n+1 n;n n i;i 1 1 i n;n 1 1 i;i ii ii i;i 1 i 1;i i;i+1 i;i+1 ii 1 nn nn n;n 1 n 1;n n n;n+1 n;n n i i 2x xi 1 + 2x xn 1 i 1 i;i+1 x2 =1 xi+1 =0 i+1 i + 2x = 0 n 12 = 2; 3; :::; n 1 1 n 1 nn ; =lii zi =