* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download The alogorithm
Survey
Document related concepts
Determinant wikipedia , lookup
Quadratic equation wikipedia , lookup
Factorization of polynomials over finite fields wikipedia , lookup
Quartic function wikipedia , lookup
Linear algebra wikipedia , lookup
Non-negative matrix factorization wikipedia , lookup
Singular-value decomposition wikipedia , lookup
Eigenvalues and eigenvectors wikipedia , lookup
Signal-flow graph wikipedia , lookup
Matrix calculus wikipedia , lookup
Elementary algebra wikipedia , lookup
Cayley–Hamilton theorem wikipedia , lookup
System of polynomial equations wikipedia , lookup
History of algebra wikipedia , lookup
Transcript
3.2 Gaussian Elimination Without Pivoting. Gaussian elimination is one popular procedure to solve linear equations. As we shall see, it leads to a decomposition of the coefficient matrix A as the product A = LU of a lower triangular matrix L and an upper triangular matrix U. L contains the multipliers used in eliminating variables and U corresponds to the system of equations that we get when we have eliminated variables. We first consider LU decompositions without pivoting and then with pivoting. 3.2.1 The Algorithm. We illustrate this method by means of an example. Example 1. 4x1 - 2x2 + (1) - 3x1 x1 - x3 = 15 x2 + 4x3 = 8 4 -2 1 x1 15 -3 -1 4 x2 = 8 1 -1 3 x3 13 or x2 + 3x3 = 13 In general, we are solving a system of equations of the form a11x1 + a12x2 + + a1nxn = b1 a21x1 + a22x2 + + a2nxn = b2 a a a or (2) an1x1 + an2x2 + + annxn = bn aa Thus the equations can be written as Ax = b where A = a 11 21 n1 a12 a1n a22 a2n an2 ann 21 a12 a1n a22 a2n n1 ann 11 an2 xx x 1 2 n b1 b2 = bn bb xx , x = and b = . b x 1 1 2 2 n n We eliminate x1 from equations 2, 3, …, n. To do this we subtract multiples of equation 1 from each of the a21 other equations. To eliminate x1 from equation 2 we subtract m = times equation 1 from equation 2. In a11 aj1 general, to eliminate x1 from equation j we subtract m = times equation 1 from equation j. If terms of the a11 aj1 matrix A and vector b we are subtracting m = times row 1 from row j. a11 a21 3 a31 1 = - times equation 1 from equation 2 and = times a11 4 a11 4 3 1 equation 1 from equation 3. In terms of the matrix A and vector b we subtract - times row 1 from row 2 and 4 4 Example 1 (continued). In Example 1 we subtract times row 1 from row 3. 3.2.1 - 1 4 -2 1 15 Recall A = -3 -1 4 and b = 8 . 1 -1 3 13 A Row 2: A2,● 3 4 3 4 ( ) (Row 1): 3 Row 2 - (- 4) (Row 1): ( )A 3 A - (- 4) A Row 3: A3,● 1,● 2,● 1 4 1 4 ( ) (Row 1): 1 Row 3 - (4) (Row 1): ( )A 1 A - (4) A 1,● 1,● 3,● 1,● b = (-3, -1, = (-3, 3 , 2 3 - ) 4 5 , 2 19 ) 4 = ( 0, - = (1, -1, = (1, 1 - , 2 = (0, - 4) b2 8 ( )b = 3 b - (- 4)b = 45 4 1 2 3) 1 ) 4 1 , 2 = 3 4 77 4 b3 = 13 ( )b 1 b - (4)b = 4 1 4 11 ) 4 1 15 1 3 37 1 = 4 After doing this the equations have become 4x1 - 2x2 + (3) 0 0 x3 = 15 5 19 77 4 1 11 37 4 - 2 x2 + 4 x 3 = - 2 x2 + 4 x 3 = or A(1)x = b(1) where A(1) 0 = 0 or 4 -2 1 5 2 1 2 19 4 11 4 and b 4 -2 1 5 2 1 2 19 4 11 4 x xx 2 = 3 1 77 374 4 15 77 = 4 . 374 15 (1) There is a way of viewing what we have done in terms of matrix multiplication. It is easiest to see this if we use the following way of looking at matrix multiplication. Suppose we are forming the product BC of matrices B and C. Then the rows of BC are linear combinations of the rows of C using the entries of the rows of B as the multipliers. More precisely, (4) Row i of BC is a linear combination of the rows of C using the entries of row i of B as coefficients, i.e. (BC)i,● = Bi,1C1,● + Bi,2C2,● + + Bi,nCn,● Example 2. Let B = 1 2 and C = 5 6 . If one computes BC the traditional way then 3 4 7 8 1 2 5 6 BC = 3 4 7 8 = (1, 2) (3, 4) ( 57 ) (1, 2) ( 68 ) = 5+14 6+16 = 19 22 ( 15+28 18+32 ) ( 43 50 ) ( 57 ) (3, 4) ( 68 ) On the other hand, if one computes BC using (3) then one has 3.2.1 - 2 BC = 1 2 5 6 = 1(5, 6) + 2(7, 8) = (5, 6) + (14, 16) = 3 4 7 8 3(5, 6) + 4(7, 8) (15, 18) + (28, 32) ( 1943 2250 ) 3 Example 1 (continued). With this way of doing matrix multiplication in mind, let E = 4 - 14 1 0 1 0 be the 1 0 0 matrix obtained by taking the identity matrix and replacing the entries in the first column below the diagonal by the negatives of the multipliers. Using (4) it is not hard to see that EA = A(1) and Eb = b(1). Thus the process of eliminating x1 from rows 2 and 3 above can be viewed as taking the equation Ax = b and multipling by E giving 1 0 0 EAx = Eb. Using EA = A(1) and Eb = b(1) one obtains A(1)x = b(1). Let M(1) = - 3 14 4 1 0 be the matrix 1 0 obtained by taking the identity matrix and replacing the entries in the first column below the diagonal by the the multipliers themselves. Using (4) it is not hard to see that EM(1) = I, so E -1 = M(1). Multiplying EA = A(1) by E -1 = M(1) one obtains M(1)A(1) = A (5) Let's take a look an algorithm for doing the above. In order to subtract m = aj1 times row 1 from row j we do a11 m = aj1/a11 for p = 2 to n ajp = ajp – ma1p bj = bj – mb1 The algorithm overwrites the original A and b with A(1) and b(1). If the original A and b are needed later in the aj1 program then they should be saved in some other variables. To subtract m = times row 1 from row j for rows a11 j = 2, 3, …, n we enclose the above in a for loop, i.e. for j = 2 to n aj1 | m= a11 (6) | for p = 2 to n | bj = bj – mb1 ajp = ajp – ma1p One thing we will be interested in as we go along is comparing different algorithms as to the amount of time they take. This is actually quite difficult since the time depends not only on the algorithm, but the computer on which the algorithm is run and the programming language the algorithm is translated into. However, we can get some idea by counting the number of floating point operations (or flops, for short) the algorithm takes. With this im mind, lets look at the algorithm (6). However, let's not count the statement "bj = bj – mb1" for reasons which will be discussed shortly. 3.2.1 - 3 The statement "for j = 2 to n" has no flops. The statement "m = aj1/a11" has one flop and it is done n-1 times so there are n-1 flops altogether The statement "for p = 2 to n" has no flops. The statement "ajp = ajp – ma1p" has two flops. As p runs from 2 to n it is done n-1 times so there are 2(n-1) flops for each value of j. There are n-1 values of j as j runs from 2 to n. So altogether we get 2(n-1)2 flops from the statement "ajp = ajp – ma1p". This gives a count of (n-1) + 2(n-1)2 for the number of flops in the algorithm (6) if we don't include the statement "bj = bj – mb1". We shall pursue this further as we continue with algorithm. After eliminating x1, the equations 2, …, n, only contain the variables x2, …, xn. Next we eliminate x2 from equations 3, …, n by subtracting multiples of equation 2 from each of the other equations. To eliminate x2 from (1) a j2 equation j we subtract m = (1) times equation 2 from equation j. If terms of the matrix A and vector b we are a 22 (1) subtracting m = a j2 (1) times row 2 from row j. Here are the details for Example 1. a 22 Example 1 (continued). In Example 1 we are now at the set of equations (3). We subtract equation 2 from equation 3. In terms of the matrix A(1) and vector b(1) we subtract Recall A(1) 0 = 0 4 -2 1 5 2 1 2 19 4 11 4 and b a32 - 1/2 1 = = times a22 - 5/2 5 1 times row 2 from row 3. 5 77 = 4 . 374 15 (1) A(1) (1) Row 3: A3,● = (0, (15) (Row 2): 1 Row 3 - (5) (Row 2): (15) A 1 A - (5) A (1) 2,● (1) 3,● = (0 (1) 2,● 1 , 2 1 - , 2 11 ) 4 19 ) 20 - = (0, b(1) (1) b3 (15)b 1 b - (5)b (1) 2 9 ) 5 0, = (1) 3 = (1) 2 After doing this the equations have become 4x1 - 2x2 + (7) - (2) or A x = b (2) 5 x 2 2 + where A 0 0 x3 = 15 19 x 4 3 = 77 4 9 x 5 3 = 27 5 (2) 0 = 0 or 4 -2 1 5 2 19 4 9 5 0 and b 4 -2 1 5 2 19 4 9 5 0 x xx 77 = 4 . If we let M 275 15 (2) (2) 3.2.1 - 4 2 = 3 1 77 274 5 15 1 0 0 0 1 0 = then 0 1 1 5 37 4 77 20 27 = 5 M(2)A(2) = A(1) (8) Since M(1)A(1) = A, it follows that M(1)M(2)A(2) = A. Let's write this as L(2)A(2) = A (9) where - 3 = 4 14 1 L(2) = M(1)M(2) (10) 0 1 0 1 0 0 1 0 0 0 1 1 5 - 3 = 4 1 14 0 0 1 0 1 1 5 1 0 0 L(2) is an example of a lower triangular matrix. This is a matrix all of whose entries above the main diagonal are zero. A(2) is an example of a upper triangular matrix, i.e. a matrix all of whose entries below the main diagonal are zero. An algorithm for eliminating x2 from equations 3, …, n would be the following. for j = 3 to n aj2 | m= a22 | for p = 3 to n | bj = bj – mb2 ajp = ajp – ma2p An analysis similar to the one we gave for the algorithm (6) would show that there are (n-2) + 2(n-2)2 flops in this algorithm if we don't include the statement "bj = bj – mb2". In general, after eliminating x2 from equations 3, …, n, we would eliminating x3 from equations 4, …, n, and so on. Finally we reach the point where each equation j only contains the variables xj, …, xn, i.e. the system of equations has become a11x1 + a12x2 + a13x3 + + a1nxn = b1 (1) (1) (1) (1) a 22 x2 + a 23 x3 + + a 2n xn = b 2 (2) (2) (2) a 33 x3 + + a 3n xn = b 3 a (n-1) nn xn = b (n-1) n or Ux = b(n-1) where 3.2.1 - 5 U = A (n-1) = a 0 0 11 a12 a1n (1) (1) a 22 a 2n 0 a (n-1) nn (k-1) a jk Note that U is upper triangular. The multipliers Ljk = (k-1) used in the elimination process are stored in lower a kk triangular matrix L and one has LU = A. This formula which writes A as a product of a lower triangular matrix and an upper triangular matrix is called a LU decompostion of A. In Example 1 we have already reached that point. An algortihm for finding the LU decompostion of A would be the following. L=I for k = 1 to n-1 | (11) | for j = k+1 to n ajk | m= akk | | Ljk = m | | for p = k+1 to n ajp = ajp – makp Note that we have held off in working with the right hand side of the equations b. If we anticipate solving a series of equations all with the same coefficient matrix A but having different right hand sides b, it is more economical time wise to do the LU decompostion of A once and save it in a disk file. Then for each new right hand side, we use L to do the same steps on b that were used in the elimination process that we used to get the LU decomposition of A. That would be the following. for k = 1 to n-1 | for j = k+1 to n bj = bj – Ljkbk Let's count the number of flops in the algorithm (11). An analysis similar to the one we gave for the algorithm (6) would show that for a fixed value of k, the statements for j = k+1 to n ajk | m= akk | Lkj = m | for p = k+1 to n ajp = ajp – makp have (n-k) + 2(n-k)2 flops. We have to add this up as k runs from 1 to n-1. This is [(n-1) + 2(n-1)2] + [(n-2) + 2(n-2)2] + [(n-3) + 2(n-3)2] + + [2 + 222] + [1 + 212] 3.2.1 - 6 = [1 + 2 + + + + (n-2) + (n-1)] + 2[12 + 22 + + + + (n-2)2 + (n-1)2] = (n-1)n 2(n-1)n(2(n-1)+1) (n-1)n(4n+1) + = 2 6 6 2 1 1 = 3n3 - 2n2 - 6n Here we have used the formulas n(n+1) 2 1 + 2 + + + + (n-1) + n = n(n+1)(2n+1) 6 12 + 22 + + + + (n-1)2 + n2 = 2 1 1 Thus the number of flops in the algorithm (11) is 3n3 - 2n2 - 6n. We are particularly interested in this for large 2 values of n. If n is large then n is small compared ot n3. So the number of flops is approximately 3n3. Now that we reached the point where each equation j only contains the variables xj, …, xn, we can solve the last equation for xn. Substituting this into equation n – 1, we can then solve for xn-1 and so on. This process is called back substitution. Example 1 (continued). Recall the equations (7) are 4x1 - 2x2 + - 5 x 2 2 + x3 = 15 19 x 4 3 = 77 4 9 x 5 3 = 27 5 Solving the last equation for x3 one obtains 27/5 x3 = 9/5 = 3 Substituting this into the previous equation and solving for x2 gives x2 = 77 19 - (3) 4 4 5 2 ( ) = 77 57 4 4 5 2 = 20 4 5 2 = -2 Substituting this into the first equation and solving for x2 gives x1 = 15 + (2)(-2) - 3 4 = 2 3.2.1 - 7 In general xj = bj - aj,j+1xj+1 - aj,j+2xj+2 - aj,nxn aj,j An algorithm to do the complete back substitution process including the same steps on b that were used in the elimination process that we used to get the LU decomposition of A be the following. for k = 1 to n-1 | for j = k+1 to n bj = bj – Ljkbk for j = n down to 1 | for p = j+1 to n | bj = bj – ajpxp xj = bj ajj Again we are interested in the number of flops. The statement "bj = bj – Ljkbk" has two flops. As j runs from k+1 to n this gives 2(n-k) flops. As k runs from 1 to n-1 this gives 2[1 + 2 + + + + (n-2) + (n-1)] = n(n-1) flops. The statement "bj = bj – ajpxp" has two flops. As p runs from j+1 to n this gives 2(n-j) flops. As j runs from n down to 1 this gives 2[1 + 2 + + + + (n-2) + (n-1)] = n(n-1) flops. The statement "xj = bj/ajj" has one flop. As j runs from n down to 1 this gives n flops. So the total number of flops is n(n-1) + n(n-1) + n = 2(n-1)n + n = 2n2 - n. If n is large this is approximately 2n2. Recall that the LU 2 2 decomposition of A has approximately 3n3 flops. If n is large then 3n3 is a lot larger than 2n2. Thus if we are solving a bunch of sets of linear equations all with the same coefficient matrix A then it makes sense to do the LU decomposition of A just once and only do the back substitution part for each set. Here is an example illustrating solving a set of linear equations by first doing a LU decomposition of the coefficient matrix and then using it to solve the equations. Example 2. 2x1 + x2 + x3 = 4 x4 = 6 8x1 + 7x2 + 9x3 + 5x4 = 8 4x1 + 3x2 + 3x3 + 2 4 8 6 or 1 3 7 7 1 3 9 9 0 x1 1 x2 5 x3 = 8 x4 4 6 8 - 2 6x1 + 7x2 + 9x3 + 8x4 = - 2 2 4 or Ax = b where A = 8 6 1 3 7 7 1 3 9 9 0 x1 4 1 x2 6 5, , x = x3 and b = 8 . Start with L equal to the identity matrix. 8 x4 - 2 3.2.1 - 8 Part 1. Find the LU decomposition of A. Step 1. Subtract multiples of row 1 of A from rows 2, 3 and 4 to get zeros in column 1 in rows 2 – 4 below the diagonal. Subtract L21 = 4/2 = 2 times row 1 from row 2 Subtract L31 = 8/2 = 4 times row 1 from row 3 Subtract L41 = 6/2 = 3 times row 1 from row 4 2 0 After doing this A has been tranformed to A(1) = 0 0 1 0 0 0 2 1 0 0 (1) L = 4 0 1 0 and we have L(1)A(1) = A. 3 0 0 1 1 1 3 4 1 1 5 6 0 1 5 . Store the multipliers in L. It is now 8 Step 2. Subtract multiples of row 2 of A(1) from rows 3 and 4 to get zeros in column 2 in rows 3 and 4 below the diagonal. Subtract L32 = 3/1 = 3 times row 2 from row 3 Subtract L42 = 4/1 = 4 times row 2 from row 4 2 0 After doing this A has been tranformed to A(2) = 0 0 1 0 0 0 2 1 0 0 (2) L = 4 3 1 0 and we have L(2)A(2) = A. 3 4 0 1 1 1 0 0 1 1 2 2 0 1 2 . Store the multipliers in L. It is now 4 Step 3. Subtract multiples of row 3 of A(2) from row 4 to get zeros in column 3 in row 4 below the diagonal. Subtract L43 = 2/2 = 1 times row 3 from row 4 2 1 1 0 0 1 1 1 After doing this A has been tranformed to A(3) = U = 0 0 2 2 . Store the multipliers in L. It is now 0 0 0 2 1 0 0 0 2 1 0 0 L(3) = L = 4 3 1 0 and we have LU = A and this is the LU decompostion of A. 3 4 1 1 Part 2. Use the LU decomposition of A to solve the original equations. 3.2.1 - 9 Step 1. Use L to do the same row operations on b as was done on A to transform it to U. Subtract L21 = 2 times b1 from b2 Subtract L31 = 4 times b1 from b3 Subtract L41 = 3 times b1 from b4 4 -2 After doing this b has been tranformed to b(1) = - 8 . - 4 Subtract L32 = 3 times b2 from b3 Subtract L42 = 4 times b2 from b4 4 -2 After doing this b has been tranformed to b(2) = - 2 . - 6 Subtract L43 = 1 times b3 from b4 4 -2 After doing this b has been tranformed to b(3) = - 2 . The original equations have been transformed to - 4 Ux = b(3), i.e. 2x1 + x2 + x3 x2 + x3 + = 4 2 4 8 6 x4 = - 2 2x3 + 2x4 = - 2 1 3 7 7 1 3 9 9 0 x1 1 x2 5 x3 = 8 x4 2x4 = - 4 Step 2 – Back Substitution. Solve the transformed equations by back substitution. -4 x4 = 2 = -2 x3 = - 2 - (2)(-2) 2 = 1 x2 = -2 - 1 - (-2) 1 = -1 x1 = 4 - (-1) - 1 2 = 2 3.2.1 - 10 4 6 8 - 2