Download The alogorithm

3.2 Gaussian Elimination Without Pivoting.
Gaussian elimination is one popular procedure to solve linear equations. As we shall see, it leads to a
decomposition of the coefficient matrix A as the product A = LU of a lower triangular matrix L and an upper
triangular matrix U. L contains the multipliers used in eliminating variables and U corresponds to the system of
equations that we get when we have eliminated variables. We first consider LU decompositions without
pivoting and then with pivoting.
3.2.1 The Algorithm.
We illustrate this method by means of an example.
Example 1.
4x1 - 2x2 +
(1)
- 3x1 x1 -
x3 = 15
x2 + 4x3 =
8
 4 -2 1  x1 
 15 
-3 -1 4  x2  =  8 
 1 -1 3  x3 
 13 
or
x2 + 3x3 = 13
In general, we are solving a system of equations of the form
a11x1 + a12x2 +  + a1nxn = b1
a21x1 + a22x2 +  + a2nxn = b2
a
a

a
or

(2)
an1x1 + an2x2 +  + annxn = bn
 aa
Thus the equations can be written as Ax = b where A = 
a
11
21
n1
a12  a1n
a22  a2n

an2  ann
21
a12  a1n
a22  a2n
n1

 ann
11
an2
  xx 
 
  x 

1
2
n
 b1 
b2
= 
 
 bn 
 bb 
  xx 
, x =    and b =   .
b 
 x 
1
1
2
2
n
n
We eliminate x1 from equations 2, 3, …, n. To do this we subtract multiples of equation 1 from each of the
a21
other equations. To eliminate x1 from equation 2 we subtract m =
times equation 1 from equation 2. In
a11
aj1
general, to eliminate x1 from equation j we subtract m =
times equation 1 from equation j. If terms of the
a11
aj1
matrix A and vector b we are subtracting m =
times row 1 from row j.
a11
a21
3
a31 1
= - times equation 1 from equation 2 and
= times
a11
4
a11 4
3
1
equation 1 from equation 3. In terms of the matrix A and vector b we subtract - times row 1 from row 2 and
4
4
Example 1 (continued). In Example 1 we subtract
times row 1 from row 3.
3.2.1 - 1
 4 -2 1
 15 
Recall A = -3 -1 4 and b =  8 .
 1 -1 3
 13 
A
Row 2:
A2,●
3
4
3
4
( ) (Row 1):
3
Row 2 - (- 4) (Row 1):
( )A
3
A - (- 4) A
Row 3:
A3,●
1,●
2,●
1
4
1
4
( ) (Row 1):
1
Row 3 - (4) (Row 1):
( )A
1
A - (4) A
1,●
1,●
3,●
1,●
b
= (-3,
-1,
= (-3,
3
,
2
3
- )
4
5
,
2
19
)
4
= ( 0,
-
= (1,
-1,
= (1,
1
- ,
2
= (0,
-
4)
b2
8
( )b =
3
b - (- 4)b =
45
4
1
2
3)
1
)
4
1
,
2
=
3
4
77
4
b3
= 13
( )b
1
b - (4)b
= 4
1
4
11
)
4
1
15
1
3
37
1
= 4
After doing this the equations have become
4x1 - 2x2 +
(3)
0

0
x3 = 15
5
19
77
4
1
11
37
4
- 2 x2 + 4 x 3 =
- 2 x2 + 4 x 3 =
or A(1)x = b(1) where A(1)
0
=
0
or
4 -2
1
5
2
1
2
19
4
11
4

 and b

4 -2
1
5
2
1
2
19
4
11
4
 x
  xx


2 =
3
1
 77 
 374 
 4
15
 77 
=  4 .
 374 
15
(1)
There is a way of viewing what we have done in terms of matrix multiplication. It is easiest to see this if we use
the following way of looking at matrix multiplication. Suppose we are forming the product BC of matrices B
and C. Then the rows of BC are linear combinations of the rows of C using the entries of the rows of B as the
multipliers. More precisely,
(4)
Row i of BC is a linear combination of the rows of C using the entries of row i of B as coefficients, i.e.
(BC)i,● = Bi,1C1,● + Bi,2C2,● +  + Bi,nCn,●
Example 2. Let B =
 1 2  and C =  5 6 . If one computes BC the traditional way then
3 4
7 8
1 2
5 6
BC =  3 4   7 8  =
 (1, 2)

 (3, 4)

( 57 ) (1, 2) ( 68 )  = 5+14 6+16 = 19 22
( 15+28 18+32 ) ( 43 50 )
( 57 ) (3, 4) ( 68 ) 
On the other hand, if one computes BC using (3) then one has
3.2.1 - 2
BC =
 1 2   5 6  =  1(5, 6) + 2(7, 8)  =  (5, 6) + (14, 16)  =
 3 4   7 8   3(5, 6) + 4(7, 8) 
 (15, 18) + (28, 32) 
( 1943 2250 )
3
Example 1 (continued). With this way of doing matrix multiplication in mind, let E =  4
- 14
1
0
1
0

 be the
1
0
0
matrix obtained by taking the identity matrix and replacing the entries in the first column below the diagonal by
the negatives of the multipliers. Using (4) it is not hard to see that EA = A(1) and Eb = b(1). Thus the process of
eliminating x1 from rows 2 and 3 above can be viewed as taking the equation Ax = b and multipling by E giving
1 0 0
EAx = Eb. Using EA = A(1) and Eb = b(1) one obtains A(1)x = b(1). Let M(1) =
- 3
 14
4
1
0

 be the matrix
1 
0
obtained by taking the identity matrix and replacing the entries in the first column below the diagonal by the the
multipliers themselves. Using (4) it is not hard to see that EM(1) = I, so E -1 = M(1). Multiplying EA = A(1) by
E -1 = M(1) one obtains
M(1)A(1) = A
(5)
Let's take a look an algorithm for doing the above. In order to subtract m =
aj1
times row 1 from row j we do
a11
m = aj1/a11
for p = 2 to n

ajp = ajp – ma1p
bj = bj – mb1
The algorithm overwrites the original A and b with A(1) and b(1). If the original A and b are needed later in the
aj1
program then they should be saved in some other variables. To subtract m =
times row 1 from row j for rows
a11
j = 2, 3, …, n we enclose the above in a for loop, i.e.
for j = 2 to n
aj1
|
m=
a11
(6)
|
for p = 2 to n
|


bj = bj – mb1
ajp = ajp – ma1p
One thing we will be interested in as we go along is comparing different algorithms as to the amount of time
they take. This is actually quite difficult since the time depends not only on the algorithm, but the computer on
which the algorithm is run and the programming language the algorithm is translated into. However, we can get
some idea by counting the number of floating point operations (or flops, for short) the algorithm takes. With
this im mind, lets look at the algorithm (6). However, let's not count the statement "bj = bj – mb1" for reasons
which will be discussed shortly.
3.2.1 - 3
The statement "for j = 2 to n" has no flops.
The statement "m = aj1/a11" has one flop and it is done n-1 times so there are n-1 flops altogether
The statement "for p = 2 to n" has no flops.
The statement "ajp = ajp – ma1p" has two flops. As p runs from 2 to n it is done n-1 times so there are
2(n-1) flops for each value of j. There are n-1 values of j as j runs from 2 to n. So altogether we
get 2(n-1)2 flops from the statement "ajp = ajp – ma1p".
This gives a count of (n-1) + 2(n-1)2 for the number of flops in the algorithm (6) if we don't include the
statement "bj = bj – mb1". We shall pursue this further as we continue with algorithm.
After eliminating x1, the equations 2, …, n, only contain the variables x2, …, xn. Next we eliminate x2 from
equations 3, …, n by subtracting multiples of equation 2 from each of the other equations. To eliminate x2 from
(1)
a j2
equation j we subtract m = (1) times equation 2 from equation j. If terms of the matrix A and vector b we are
a 22
(1)
subtracting m =
a j2
(1) times row 2 from row j. Here are the details for Example 1.
a 22
Example 1 (continued). In Example 1 we are now at the set of equations (3). We subtract
equation 2 from equation 3. In terms of the matrix A(1) and vector b(1) we subtract
Recall A(1)
0
=
0
4 -2
1
5
2
1
2
19
4
11
4

 and b

a32 - 1/2 1
=
= times
a22 - 5/2 5
1
times row 2 from row 3.
5
 77 
=  4 .
 374 
15
(1)
A(1)
(1)
Row 3:
A3,●
= (0,
(15) (Row 2):
1
Row 3 - (5) (Row 2):
(15) A
1
A - (5) A
(1)
2,●
(1)
3,●
= (0
(1)
2,●
1
,
2
1
- ,
2
11
)
4
19
)
20
-
= (0,
b(1)
(1)
b3
(15)b
1
b - (5)b
(1)
2
9
)
5
0,
=
(1)
3
=
(1)
2
After doing this the equations have become
4x1 - 2x2 +
(7)
-
(2)
or A x = b
(2)
5
x
2 2
+
where A
0

0
x3 = 15
19
x
4 3
=
77
4
9
x
5 3
=
27
5
(2)
0
=
0
or
4 -2
1
5
2
19
4
9
5
0

 and b

4 -2
1
5
2
19
4
9
5
0
 x
  xx

 77 
=  4 . If we let M
 275 
15
(2)
(2)
3.2.1 - 4

2 =
3
1
 77 
 274 
 5
15
1 0 0 
0 1 0
=
then
0 1 1 
 5

37
4
77
20
27
= 5
M(2)A(2) = A(1)
(8)
Since M(1)A(1) = A, it follows that M(1)M(2)A(2) = A. Let's write this as
L(2)A(2) = A
(9)
where
- 3
=  4
 14
1
L(2) = M(1)M(2)
(10)
0
1
0
 1
0 0

1  0
0
0
1
1
5
- 3
=  4
1 
 14
0
0
1
0
1
1
5


1 
0
0
L(2) is an example of a lower triangular matrix. This is a matrix all of whose entries above the main diagonal are
zero. A(2) is an example of a upper triangular matrix, i.e. a matrix all of whose entries below the main diagonal
are zero.
An algorithm for eliminating x2 from equations 3, …, n would be the following.
for j = 3 to n
aj2
|
m=
a22
|
for p = 3 to n
|


bj = bj – mb2
ajp = ajp – ma2p
An analysis similar to the one we gave for the algorithm (6) would show that there are (n-2) + 2(n-2)2 flops in
this algorithm if we don't include the statement "bj = bj – mb2".
In general, after eliminating x2 from equations 3, …, n, we would eliminating x3 from equations 4, …, n, and so
on. Finally we reach the point where each equation j only contains the variables xj, …, xn, i.e. the system of
equations has become
a11x1 + a12x2 + a13x3 +  + a1nxn = b1
(1)
(1)
(1)
(1)
a 22 x2 + a 23 x3 +  + a 2n xn = b 2
(2)
(2)
(2)
a 33 x3 +  + a 3n xn = b 3

a
(n-1)
nn xn
= b
(n-1)
n
or Ux = b(n-1) where
3.2.1 - 5
U = A
(n-1)
=
a
0

0
11

a12  a1n
(1)
(1)
a 22  a 2n
0 
a
(n-1)
nn




(k-1)
a jk
Note that U is upper triangular. The multipliers Ljk = (k-1) used in the elimination process are stored in lower
a kk
triangular matrix L and one has LU = A. This formula which writes A as a product of a lower triangular matrix
and an upper triangular matrix is called a LU decompostion of A.
In Example 1 we have already reached that
point.
An algortihm for finding the LU decompostion of A would be the following.
L=I
for k = 1 to n-1
|
(11)
|
for j = k+1 to n
ajk
|
m=
akk
|
|
Ljk = m
|
|
for p = k+1 to n



ajp = ajp – makp
Note that we have held off in working with the right hand side of the equations b. If we anticipate solving a
series of equations all with the same coefficient matrix A but having different right hand sides b, it is more
economical time wise to do the LU decompostion of A once and save it in a disk file. Then for each new right
hand side, we use L to do the same steps on b that were used in the elimination process that we used to get the
LU decomposition of A. That would be the following.
for k = 1 to n-1
|
for j = k+1 to n


bj = bj – Ljkbk
Let's count the number of flops in the algorithm (11). An analysis similar to the one we gave for the algorithm
(6) would show that for a fixed value of k, the statements
for j = k+1 to n
ajk
|
m=
akk
|
Lkj = m
|
for p = k+1 to n


ajp = ajp – makp
have (n-k) + 2(n-k)2 flops. We have to add this up as k runs from 1 to n-1. This is
[(n-1) + 2(n-1)2] + [(n-2) + 2(n-2)2] + [(n-3) + 2(n-3)2] +  + [2 + 222] + [1 + 212]
3.2.1 - 6
= [1 + 2 + +  + + (n-2) + (n-1)] + 2[12 + 22 + +  + + (n-2)2 + (n-1)2]
=
(n-1)n
2(n-1)n(2(n-1)+1)
(n-1)n(4n+1)
+
=
2
6
6
2
1
1
= 3n3 - 2n2 - 6n
Here we have used the formulas
n(n+1)
2
1 + 2 + +  + + (n-1) + n =
n(n+1)(2n+1)
6
12 + 22 + +  + + (n-1)2 + n2 =
2
1
1
Thus the number of flops in the algorithm (11) is 3n3 - 2n2 - 6n. We are particularly interested in this for large
2
values of n. If n is large then n is small compared ot n3. So the number of flops is approximately 3n3.
Now that we reached the point where each equation j only contains the variables xj, …, xn, we can solve the last
equation for xn. Substituting this into equation n – 1, we can then solve for xn-1 and so on. This process is called
back substitution.
Example 1 (continued). Recall the equations (7) are
4x1 - 2x2 +
-
5
x
2 2
+
x3 = 15
19
x
4 3
=
77
4
9
x
5 3
=
27
5
Solving the last equation for x3 one obtains
27/5
x3 = 9/5 = 3
Substituting this into the previous equation and solving for x2 gives
x2 =
77
19
- (3)
4
4
5
2
( )
=
77 57
4 4
5
2
=
20
4
5
2
= -2
Substituting this into the first equation and solving for x2 gives
x1 =
15 + (2)(-2) - 3
4
= 2
3.2.1 - 7
In general
xj =
bj - aj,j+1xj+1 - aj,j+2xj+2 -  aj,nxn
aj,j
An algorithm to do the complete back substitution process including the same steps on b that were used in the
elimination process that we used to get the LU decomposition of A be the following.
for k = 1 to n-1
|
for j = k+1 to n


bj = bj – Ljkbk
for j = n down to 1
|
for p = j+1 to n
|

bj = bj – ajpxp

xj =
bj
ajj
Again we are interested in the number of flops. The statement "bj = bj – Ljkbk" has two flops. As j runs from
k+1 to n this gives 2(n-k) flops. As k runs from 1 to n-1 this gives 2[1 + 2 + +  + + (n-2) + (n-1)] = n(n-1)
flops. The statement "bj = bj – ajpxp" has two flops. As p runs from j+1 to n this gives 2(n-j) flops. As j runs
from n down to 1 this gives 2[1 + 2 + +  + + (n-2) + (n-1)] = n(n-1) flops. The statement "xj = bj/ajj" has one
flop. As j runs from n down to 1 this gives n flops. So the total number of flops is
n(n-1) + n(n-1) + n = 2(n-1)n + n = 2n2 - n. If n is large this is approximately 2n2. Recall that the LU
2
2
decomposition of A has approximately 3n3 flops. If n is large then 3n3 is a lot larger than 2n2. Thus if we are
solving a bunch of sets of linear equations all with the same coefficient matrix A then it makes sense to do the
LU decomposition of A just once and only do the back substitution part for each set.
Here is an example illustrating solving a set of linear equations by first doing a LU decomposition of the
coefficient matrix and then using it to solve the equations.
Example 2.
2x1 +
x2 +
x3
=
4
x4 =
6
8x1 + 7x2 + 9x3 + 5x4 =
8
4x1 + 3x2 + 3x3 +
2
4
8
6
or
1
3
7
7
1
3
9
9
0   x1 
1   x2 
5   x3  =
8   x4 
 4
 6
 8
- 2
6x1 + 7x2 + 9x3 + 8x4 = - 2
2
4
or Ax = b where A = 8

6
1
3
7
7
1
3
9
9
0
 x1 
 4
1
x2 

 6
5, , x =  x3  and b =  8 . Start with L equal to the identity matrix.
8
 x4 
- 2
3.2.1 - 8
Part 1. Find the LU decomposition of A.
Step 1. Subtract multiples of row 1 of A from rows 2, 3 and 4 to get zeros in column 1 in rows 2 – 4 below the
diagonal.
Subtract L21 = 4/2 = 2 times row 1 from row 2
Subtract L31 = 8/2 = 4 times row 1 from row 3
Subtract L41 = 6/2 = 3 times row 1 from row 4
2
0
After doing this A has been tranformed to A(1) = 0

0
1
0
0
0


2 1 0 0

(1)
L = 4 0 1 0 and we have L(1)A(1) = A.


3 0 0 1
1
1
3
4
1
1
5
6
0
1
5 . Store the multipliers in L. It is now
8
Step 2. Subtract multiples of row 2 of A(1) from rows 3 and 4 to get zeros in column 2 in rows 3 and 4 below the
diagonal.
Subtract L32 = 3/1 = 3 times row 2 from row 3
Subtract L42 = 4/1 = 4 times row 2 from row 4
2
0
After doing this A has been tranformed to A(2) = 0

0
1
0
0
0


2 1 0 0

(2)
L = 4 3 1 0 and we have L(2)A(2) = A.


3 4 0 1
1
1
0
0
1
1
2
2
0
1
2 . Store the multipliers in L. It is now
4
Step 3. Subtract multiples of row 3 of A(2) from row 4 to get zeros in column 3 in row 4 below the diagonal.
Subtract L43 = 2/2 = 1 times row 3 from row 4
2 1 1 0
0 1 1 1
After doing this A has been tranformed to A(3) = U = 0 0 2 2 . Store the multipliers in L. It is now


0 0 0 2
1 0 0 0
2 1 0 0
L(3) = L = 4 3 1 0 and we have LU = A and this is the LU decompostion of A.


3 4 1 1
Part 2. Use the LU decomposition of A to solve the original equations.
3.2.1 - 9
Step 1. Use L to do the same row operations on b as was done on A to transform it to U.
Subtract L21 = 2 times b1 from b2
Subtract L31 = 4 times b1 from b3
Subtract L41 = 3 times b1 from b4
 4
-2
After doing this b has been tranformed to b(1) = - 8 .
 
- 4
Subtract L32 = 3 times b2 from b3
Subtract L42 = 4 times b2 from b4
 4
-2
After doing this b has been tranformed to b(2) = - 2 .
 
- 6
Subtract L43 = 1 times b3 from b4
 4
-2
After doing this b has been tranformed to b(3) = - 2 . The original equations have been transformed to
 
- 4
Ux = b(3), i.e.
2x1 +
x2 +
x3
x2 +
x3 +
=
4
2
4
8
6
x4 = - 2
2x3 + 2x4 = - 2
1
3
7
7
1
3
9
9
0   x1 
1   x2 
5   x3  =
8   x4 
2x4 = - 4
Step 2 – Back Substitution. Solve the transformed equations by back substitution.
-4
x4 = 2
= -2
x3 =
- 2 - (2)(-2)
2
= 1
x2 =
-2 - 1 - (-2)
1
= -1
x1 =
4 - (-1) - 1
2
= 2
3.2.1 - 10
 4
 6
 8
- 2