Download s06.pdf

Session 6
Direct Solution Methods for Systems of
Linear Algebraic Equations
Ernesto Gutierrez-Miravete
Spring 2003
1
Introduction
Systems of linear algebraic equations occur when several unknown quantities are interliked
by means of linear relationships. Such systems appear during the analysis of very many
engineering and scientic problems. Well known engineering examples include kinematics,
statics and dynamics of mechanical systems, the disturbed motion of bodies, the exural
vibrations of structural members and the nodal equations obtained form applying nite
dierence, nite volume or nite element methodologies for the solution of initial-boundary
value problems in continuum mechanics. The goal of analysis is to determine the values of the
unknowns in the system. Solution methods are classied as direct or iterative. Direct methods
compute a solution in a predetermined number of steps while iterative methods proceed by
computing a sequence of iincreasingly improved approximations. This session describes the
most important direct methods used for the solution of systems of linear algebraic equations.
2
Systems of Linear Algebraic Equations
A linear system of n simultaneous algebraic equations for n variables x ; x ; :::; x is a set of
equations labelled E ; E ; :::; E of the form
a x + a x + ::: + a x = b
a x + a x + ::: + a x = b
1
1
2
2
n
n
11
1
12
2
1n
n
1
21
1
22
2
2n
n
2
:
:
:::
:
:
+ a x + ::: + a x = b
The key to facilitate the solution of the system is to transform it into an equivalent system
by means of permissible operations. An equivalent system can be obtained from the above
system by performing some or all of the operations below.
an1 x1
n2
2
nn
1
n
n
1) Equation E can be multiplied by a constant . The resulting equation can the be
used instead of E . I.e. E ! E
2) Equation E can be multiplied by a constant and the result added to equation E .
The resulting equation can then be used instead of E . I.e. E + E ! E
3) Equations E and E can be exchanged. I.e. E $ E
Exercise 1. Example 6.1.1 in B-F. Transform the system
x + x + 3x = 4
2x + x x + x = 1
3x x x + 2x = 3
x + 2x + 3x
x =4
into the equivalent triangular system
i
i
i
i
j
i
i
i
j
j
i
1
1
1
2
2
2
1
i
i
i
4
3
4
3
4
2
3
4
+ x + 3x = 4
x
x
5x = 7
3x + 13x = 13
13x = 13
simply by performing the following sequence of rearrangements (E 2E ) ! E , (E
3E ) ! E , (E + E ) ! E , (E 4E ) ! E , and (E + 3E ) ! E (check!). Note that
the appropriate coeÆcients used in each rearrangement are determined in part from the
values of a . For example, in (E 2E ) the coeÆcient 2 = a =a and in (E 3E ) the
coeÆcient 3 = a =a . The number a is the pivot element in this process. The idea is
to proceed to eliminate rst the terms containing x from all equations but the rst one.
Then, to eliminate all terms containing x from all equations except the rst two. Then, to
eliminate all terms containing x from all equations except the rst three and so on. For
this process to work, it is essential that all the pivot elements be non-zero. Row exchanges
can be used at any stage to try to obtain non-zero pivots. However, if no non-zero pivot can
be found at some stage of the elimination process, it can be shown that the system has no
unique solution.
Once the system is in triangular form, it can be easily solved by a backward substitution
process starting with the last equation and moving upwards (check!). Here x = 1, x = 0,
x = 2 and x = 1.
The process of rst triangularizing the original system of equations and then solving
the resulting triangular system by backsubstitution is the essence of the Gauss elimination
method.
Exercise 2. Use Gauss elimination to nd the solution to the system
2x + 3x x = 5
x1
2
2
4
3
4
3
4
4
2
1
3
4
1
4
3
2
2
ij
31
3
4
1
11
2
21
1
2
3
4
11
3
1
11
1
2
3
4
2
1
1
2
3
2
3
4x + 4x 3x = 3
2x + 3x x = 1
Denition of n by m Matrix. A rectangular array of elements with n rows and m
columns where value and location of elements are meaningful. An n m matrix A is written
as
1
2
1
A
= (a
ij
3
2
3
2
a11
6
a21
6
6
6
) = 66 ::
6
6
4 :
a12
a22
:
:
:
an2
an1
3
::: a1m
7
::: a2m 7
7
::: : 7
7
::: : 7
7
7
::: : 5
::: anm
1 n matrices are called n-dimensional row vectors
y = [ y y ::: y ]
while n 1 matrices are called n-dimensional column vectors.
1
2
n
2 3
x1
6
7
x2 7
6
6
7
6
: 7
6
x = 6 : 77
6
7
6
4 : 7
5
xn
In the sequel we will focus exclusively on the case where n = m (i.e the number of
unknowns is equal to the numebnr of equations). Therefore, the n n system
a x + a x + ::: + a x = b
a x + a x + ::: + a x = b
11
1
12
2
1n
n
1
21
1
22
2
2n
n
2
:
an1 x1
+a
:
:::
n2
+ ::: + a
x2
nn
:
xn
can be represented as
Ax
where
2
a11
6
a21
6
6
6
:
A=6
6
:
6
6
4 :
an1
=b
a12
a22
:
:
:
an2
3
3
::: a1m
7
::: a2m 7
7
::: : 7
7
::: : 7
7
7
::: : 5
::: anm
=b
:
n
2 3
x1
6
7
x2 7
6
6
7
6
: 7
6
x = 6 : 77
6
7
6
4 : 7
5
xn
and
2 3
b1
6
7
b2 7
6
6
7
6
: 7
6
b = 6 : 77
6
7
6
4 : 7
5
bn
The augmented matrix A
aug
Aaug
= [A; b] is the n n + 1 rectangular matrix
2
a11
6
a21
6
6
6
= [A; b] = 66 ::
6
6
4 :
a12
a22
:
:
:
an2
an
2 1
a11 a12
6
a21 a22
6
6
6
= 66 :: ::
6
6
4 : :
an1 an2
x1
::: a1n
::: a2n
::: :
::: :
::: :
::: ann
3
b1
7
b1 7
7
: 7
7
: 7
7
7
: 5
b1
=
3
::: a1n a1n+1
7
::: a2n a2n+1 7
7
7
::: : :
7
7
::: : :
7
7
5
::: : :
::: ann a1n+1
In terms of the augmented matrix Gaussian elimination works as follows. As we eliminate
from all equations but the rst one, new coeÆcientes a develop according to the formulae
(1)
ij
(1)
a11
(1)
a1j
2
= aa
(j > 1)
1j
11
=a a a
(i; j 2)
from all equations except the rst two, the corresponding formulae
(1)
aij
If now, we eliminate x
are
=1
ij
i1
(1)
1j
(2)
a22
=1
4
(2)
a2j
=
(1)
a2j
(j > 2)
(1)
a22
=a a a
(i; j 3)
and so on. Backsubstitution is trivial.
The result obtained by Gauss elimination can be improved if the necessary corrections
are relatively small. Say the (approximate) solution to the system Ax = b has been found
and it is x0 . Since the real solution is x = x0 + Æ and the residual = b Ax0 then
AÆ = Gaussian elimination can now be performed in this system to nd the correction Æ.
Maple has capabilities for matrix work. The matrix command creates matrices. Operations on rows are performed with the addrow and swaprow commands. The command
gausselim produces the reduced triangular matrix and the command backsub solves it.
Gaussian Elimination with Backward Substitution Algorithm.
Give the matrix A and the vector b
Transform the orignal system into an equivalent triangular system.
Solve the triangular system by backsubstitution.
Concerning the number of operations performed in the elimination stage one has (2n +
3n 5n)=6 multiplications/divisions and (n n)=3 additions/substractions. The backward
substitution requires (n +n)=2 multiplications/divisions and (n n)=3 additions/substractions.
Thus, in total we have n =3+n n=3 n multiplications/divisions and n =3+n =2 5n=6 n additions/substractions. This total number of operations increases rapidly with the size
of the problem and shows that roundo error can become a signicant problem.
(2)
(1)
aij
(1)
i2
ij
(2)
2;j
3
2
3
2
2
3
2
3
3
2
3
3
Pivoting
Even if all pivots are non-zero, when pivots are small in relative magnitude, roundo errors
can be dramatically magnied.
If the candidate pivot a is small relative to the entries a for k i n and k j n
the partial pivoting or maximal column pivoting strategy can be used. This consists of
selecting a larger element (usually the largest) a in the column where the candidate pivot
lies as pivot instead and then swapping rows k and p.
Exercise 3. Example 6.2.2 in B-F. For the system
0:00300x + 59:14x = 59:17
5:291x 6:130x = 46:78
kk
ij
pq
1
2
1
2
5
the rst pivot a is relatively small in magnitude so we look for the maximum coeÆcient in
column 1 (5.291), swap the rows and triangularize to get
11
5:291x
1
6:130x = 46:78
59:14x 59:14
2
2
Gaussian Elimination with Partial Pivoting Algorithm.
Give the matrix A and the vector b.
Start triangularization process.
If a pivot is small, exchange row with another one containing a larger number.
Solve the triangular system by backsubstitution.
Partial pivoting can still fail even if the pivots are large in absolute size. In scaled partial
pivoting the maximum coeÆcient in each row i is identied and called the i-th row scaling
factor, s is rst computed for each row as the maximum coeÆcient in the row. Then, scaled
pivots are calculated with reference to the scaling factors as ja j=s . The largest scaled pivot
is identied and the associated rows are swapped..
Exercise 4. Example 6.2.3 in B-F. For the system
i
ij
i
30x + 591400x = 591700
5:291x 6:130x = 46:78
1
2
1
2
the value of rst scaled pivot a =s = 30=591400 5 10 is relatively small in magnitude
compared to a =s = 5:291=6:13 0:86 so swap the rows to get
11
21
5
1
2
5:291x 6:130x = 46:78
30x + 591400x = 591700
1
1
2
2
which now can readily (and accurately) be reduced.
Gaussian Elimination with Scaled Partial Pivoting Algorithm.
Give the matrix A and the vector b.
Start triangularization process.
Determine the scaling factors.
If a scaled pivot is small, exchange row with another one containing a larger number.
Solve the triangular system by backsubstitution.
6
For the highest possible accuracy, complete pivoting is possible. In complete pivoting
the largest matrix element is selected and its row (principal row) and column (principal
column) are identied. Operations are then performed to make all coeÆcients in the principal
column = 0 so it can be eliminated producing a reduced matrix. The process is repeated
on the reduced matrix to generate a further reduced matrix and so on. Finally, a simple
rearrangement of the matrix formed by all the principal rows obtained produces a triangular
system which is easily solved
4
Review of Foundations of Linear Algebra
This section contains a review of foundations of linear algebra which are essential for a full
understanding of numerical linear algebra.
Two matrices A and B are equal if they have the same size (i.e. n m) and a = b
for each i = 1; 2; :::; n and j = 1; 2; :::; m.
If A and B are n m, their sum C = A + B is a matrix with elements c = a + b for
each i = 1; 2; :::; n and j = 1; 2; :::; m.
If is a real number, the scalar multiplication of A with is A = (a ) for each
i = 1; 2; :::; n and j = 1; 2; :::; m.
Theorem of Matrix Properties Involving Sum. Matrix summation has commutative and associative properties. There are also a zero matrix, a unit matrix and the negative
matrix of A. Multiplication with scalars is distributive.
Let A be n m and B be m p the matrix product C = AB = (c ) = (P a b ) for
each i = 1; 2; :::; n and j = 1; 2; :::; m. I.e. entries on the i-th row of A, a are multiplied
with corresponding entries in the j -th column of B , b and all the products are added
together to form the entry c in the product matrix C .
A is square if n = m. A square matrix A is diagonal if the only non-zero elements are
a . The identity matrix I is a square diagonal matrix in which each a = 1. A is upper
triangular if all elements below the diagonal are zero. A is lower triangular if all elements
above the diagonal are zero.
A square n n matrix is strictly diagonally dominant if ja j > P 6 ja j, for each
i = 1; 2; :::; n.
Theorem of Matrix Properties Involving Product. Matrix multiplication has
associative and distributive properties but not commutative. Pre-multiplication of a matrix
product with a scalar is the same when performed before or after matrix multiplication.
A square matrix A is non-singular if there exists another matrix, called its inverse A
such that AA = A A = I .
Theorem about the Properties of Non-singular Matrices. If A is non-singular
n n, then
i;j
i;j
ij
ij
ij
ij
m
k =1
ij
ik
kj
ik
kj
ij
ii
ii
n
ii
j =1;j =i
ij
1
1
A
1
1
is unique.
7
A is non-singular and (A ) = A.
If B is non-singular n n, then (AB ) = B A .
If A = (a ) is a n n square matrix its transpose A = (a ). If A = A , the matrix is
1
1
1
1
1
1
t
t
ij
ji
called symmetric.
Theorem of Properties Involving Transpose. The transpose of the transpose is the
original matrix. The transpose of a matrix product is the product of the transposes and
that of a sum is the sum of the transposes. The transpose of an inverse is the inverse of the
transpose.
A is positive denite if it is symmetric and if x Ax > 0 for every x 6= 0.
A square n n matrix is a band matrix if integers p > 1 and q < n exist such that a = 0
whenever i + p j or j + q i. The bandwidth w = p + q 1. If w = 3 the matrix is
tridiagonal.
The determinant of a n n matrix A is detA = jAj.
The following is a list of characteristics of determinants.
If A = (a) is 1 1 then detA = a.
If A is n n the minor M is the determinant of the reduced n 1 n 1 submatrix
obtained by deleting the i-th row and the j -th column from A.
The cofactor A associated with M is A = ( 1) M .
The determinant of A, detA is computed as
t
ij
ij
ij
ij
detA
= jAj =
X
i+j
ij
n
aij Aij
j =1
=
X
n
ij
( 1)
i+j
aij Mij
( 1)
i+j
aij Mij
j =1
for any i = 1; 2; :::; n,or as
detA
= jAj =
X
n
aij Aij
i=1
=
X
n
i=1
for any j = 1; 2; :::; n.
Theorems Involving Determinants. jAj = 0 if any row or column of A has only zeros
or if two rows or two columns are the same. Row swapping changes the sign of detA. If a row
of A is changed by scalar multiplication with , the new determinant is jAj. Substitution
of a row of A by the sum of itself with the scalar multiplication of another the determinant
does not change. The determinant of a product is the product of the determinants. The
determinant of A is the same as that of A. The determinant of A is the reciprocal of the
jAj. For triangular or diagonal matrices jAj = a .
Theorem of Fundamental Properties of Linear Systems. If A is a n n square
matrix the following are all equivalent
1
t
n
i=1
8
ii
5
= 0 has a unique solution x = 0.
Ax = b has a unique solution for any b.
A is non-singular.
detA 6= 0
Gaussian elimination can be performed on Ax = b for any b.
Ax
Matrix Factorization for the Solution of Linear Systems
Matrix factorization is of great importance in numerical linear algebra. It consists of representing the original matrix by a product of matrices with special form. It is often useful
to factorize a matrix A so that A = LU where L and U are, respectively, lower and upper
triangular matrices. Note that if A = LU , in the system Ax = b = LU x = b = Ly = b
where y = U x. Therefore, using factorization one can solve a linear system in two easily
performed steps. First solve Ly = b for y then solve U x = y for x. Since L and U are
triangular, solution is easy and requires much fewer operations (O(n ))!
Theorem of LU Factorization. If Gaussian elimination can be performend on Ax = b
without row interchanges then A can be factored such that A = LU .
LU Factorization Algorithm.
Give the matrix A and the vector b.
Select l and u such that l u = a
For j = 2; 3; ::; n set u = a =l and l = a =u .
For i = 2; 3; ::; n 1 select l and u such that l u = a P l u .
For j = i + 1; :::; n set u = [a P l u ]=l and l = [a P l u ]=u .
Select l and u such that l u = a P l u
In the sequence above, if l u or l u equal zero factorization is impossible and if
l u = 0 , A is singular.
2
11
11
11
1j
1j
11
11
11
j1
j1
11
i
ii
ii
ii
i
ij
nn
nn
11
nn
11
nn
nn
ii
1
k =1
ik
i
ik
kj
ii
i
nn
ii
1
k =1
ij
ii
ii
nn
9
ji
1
k =1
nk
kn
ji
ki
1
k =1
jk
ki
ii
6
Special Matrices
Often, matrices encountered in applications have special forms and one takes advantage
of the properties of such matrices in order to develop eective numerical methods for the
solution of linear systems. Diagonally dominant positive denite and banded matrices are
special matrices of great importance in applications.
Theorem of Properties of a Diagonally Dominant Matrix. If A is strictly diagonally dominant then it is non-singular. The associated system Ax = b can be solved by
Gaussian elimination without row or column exchanges to give its unique solution and the
computations are stable with respect to roundo error.
Theorems of Positive Denite Matrices. If A is n n positive denite then
A is non-singular.
a > 0 for each i = 1; 2; :::; n.
max ja j max ja j.
(a ) < a a , for each i 6= j .
The symmetric matrix A is pd if and only if Gaussian elimination without row interchanges can be performed on Ax = b with all pivots > 0.
A is pd if the factorizations A = LDL and if A = LL exist.
The factorizations of A into LDL and into LL have important application for the
solution of linear systems involving positive denite matrices.
LDL Factorization Algorithm.
Give the matrix A.
For i = 1; 2; ::; n and for j = P1; :::; 1, v = l d ; d = a P l v and while for
j = i + 1; :::; n set l = [a
l v ]=d
Concerning the LL factorization with L = U (Choleski factorization) A = (a ) = LU ,
one proceeds as follows in terms of the augmented matrix A = [A; b] = (a ) with i =
1; 2; :::; n rows and j = 1; 2; :::; n +1columns. Let L = (l ) be lower triangular and U = (u )
upper triangular with unit diagonal. Determine the coeÆcients l and u according to the
following column-row, colum-row scheme.
Column:
l =a
(i 1)
ii
1
ij
2
k;j
n
ii
1
kj
i
n
ii
jj
t
t
t
t
t
i
j
1
k =1
ij
j
i
1
j =1
ii
ij
j
i
ji
ji
t
jk
k
i
t
ij
aug
ij
ij
ij
ij
i1
lij
i1
X1
j
=a
ij
(i j > 1)
lik ukj
k =1
10
ij
Row:
u1j
uij
= al
(j > 1)
1j
11
= l1 (a
X1
i
ij
ii
lik ukj
k =1
)
(j > i > 1)
and so on. Once the factorization has been accomplished, the lower traingular system Ly = b
is solved for y abd then, the upper triangular system U x = y is solved for x.
LU
Factorization (Choleski) Algorithm
Give the matrix A.
Set l = pa .
For j = 2; :::; n set l = a =l .
P
For
i = 1; 2; ::; n set l = [a
] and for j = i + 1; :::; n set l = [a
P
l l ]=l
Set l = [a P l ]
Exercise 5. Example 6.6.4 in B-F. Produce the LDL and LL factorizations of the
following matrix
2
3
4
1
1
6
7
A = 4 1 4:25 2:75 5
1 2:75 3:5
Tridiagonal systems of equations are easily solved using Crout factorization A = LU
where L = (l ) is lower triangular and U = (u ) upper triangular with unit diagonal
(Cholesky factorization for tridiagonal matrix).
Consider for example the tridiagonal system represented by the following augmented
matrix
2
2 1 0 0 13
6
7
A
= 664 01 21 21 01 00 775
0 0 1 2 1
The LU factorization of A yields
2
0 03
2 0 0 0 321
6
0 0 777 666 0 1
0 777
6 1
A = LU = 6
5
40
1 0540 0 1
0 0 1
0 0 0 1
The solution of Lz = b is z = ( ; ; ; 1) and the solution of U x = z is x = (1; 1; 1; 1) .
Solution of a Triadiagonal System by Crout Factorization Algorithm.
11
11
j1
j1
11
1 1=2
i
1
k =1
ii
k =1
ii
i
j k ik
ji
ii
n
nn
ji
nn
1 2
k =1
1=2
nk
t
ij
t
ij
aug
3
2
1
2
1
3
1
4
1
2
4
3
5
4
t
2
3
3
4
t
11
Give the matrix A and vector b.
Set l = a ; u = a =l ; z = a
For i = 2; :::; n 1 set l = a
11
11
12
12
11
1;n+1 =l11
1
.
=
;l a l u ;u =a
[a
l
z ]=l .
Set l = a ; l = a l u ; z = [a
l
z
]=l .
Set x = z .
For i = n 1; ::; 1 set x = z u x .
Exercise 6. Find the solution to the following system of equations
1
i;i
i;n+1
n;n
n
i;i
1
1
i
n;n
1
1
i;i
ii
ii
i;i
1
i
1;i
i;i+1
i;i+1
ii
1
nn
nn
n;n
1
n
1;n
n
n;n+1
n;n
n
i
i
2x
xi
1
+ 2x
xn
1
i
1
i;i+1
x2
=1
xi+1
=0
i+1
i
+ 2x = 0
n
12
= 2; 3; :::; n 1
1
n
1
nn
;
=lii zi
=