Download Linear Algebraic Equations System

Lecture 3
Numerical Methods of Linear Algebraic Equations Systems (NMLAES)
Direct Methods: Gauss, Square Roots, Cholesky Factorization.
We consider Linear Algebraic Equations System
а11 x1 
 a1k xk  b1
(1)
аn1 x1 
where the scalar
 x1,
a 
ij
 ank xk  bk ,
and
b  are
j
interpreted as given coefficients and
xk  are the unknowns that we need to determine. It is then clear that if we
a1k 
 a11
 x1 
b1 




  then the system (1) can be
write A  
,
,
x

b


 
 
 an1
 xk 
bn 
ank 
expressed as
(2)
Ax  b
Then, associate the set of all possible coefficients b1 , , bn appearing on the
right-hand side of system (1) with the vector space n and the set of possible
values for the unknowns x1 , , xk with the vector space k .
In matrix form , the problem of finding solutions for linear systems can be
rephrased as follows:
Problem 1 (Linear Systems) Given a vector b n and a matrix A  M nk ,
find a vector x  k such that Ax  b .
For the case of square systems, that is, corresponding to n  n matrices, the
solution of Problem 1 is summarized in the following theorem. The case of
under- and overdetermined systems, where the number of equations differs from
the number of unknowns.
Theorem 1 Consider a system of n linear equations with n unknowns
x1 , , xn , written in the form Ax  b , where the n  n matrix A and the vector
b n are given.
1. If A is invertible, then the system has a unique solution given by
x  A1b , regardless of the vector b .
2. If A is singular and b  range  A , then the system has no solution.
3. If A is singular and b  range  A , then the system has infinitely many
solutions.
Direct Methods
There are two types of methods for solving linear systems: direct and
iterative. If we disregard the effects of finite-precision arithmetics, direct
1
methods, treated in this section, produce an exact solution in a finite number of
steps.
We start with a very special type of linear system defined by a lower
triangular matrix L , which is an n  n matrix such that lij  0 for all i  j . In
other words, all entries above its diagonal are equal to zero. That is, the matrix
equation Lx  b has the form
0   x1 
l11 0
b1 
l l



b 
0
x
21
22
2

     2 .

  
 

  
 
lnn   xn 
ln1 ln 2
bn 
Therefore, if we start from the first equation of the system, corresponding to the
b
first row of the matrix L , we have l11 x1  b1  x1  1 . We can then store this
l11
value for the variable x1 and proceed directly to the second equation of the
b l x
system, which reads l21 x1  l22 x2  b2  x2  2 21 1 , because x1 is known
l22
for this step. Just to make sure that you understand the pattern, let us store the
values of x1 and x2 and proceed with the third equation of the system:
b   l31x1  l32 x2 
. It is now clear that at the i th
l31x1  l32 x2  l33 x3  b3  x3  3
l33
step we would have already calculated the values for x1 , , xi 1 and could use
these values to obtain
i 1

1
xi   bi   lij x j  .
(3)
lii 
j 1

For obvious reasons, this algorithm for solving a lower-triangular system is
called forward substitution.
The source of trouble is the vanishing diagonal entry appearing on the second
row. As you can see from (3), this leads to a division by zero. The good news is
that the same formula also shows that this is the only way in which forward
substitution might fail. That is, provided all the diagonal entries are different
from zero, forward substitution leads to the unique solution for the system. This
is rephrased in the following theorem:
Theorem 2 A lower-triangular matrix L is invertible if and only if all of its
diagonal entries are different from zero.
The pictorial opposite of a lower-triangular matrix is an upper-triangular
matrix, that is, a matrix U with the property that all the entries below its
diagonal are equal to zero, that is, with uij  0 for all i  j . Therefore, if U is
upper triangular, the linear system Ux  b has the from
2
u11 u12
0 u
22



0 0
u1n 
u2 n 


unn 
 x1 
b1 
x 
 
 2   b2  .
 
 
 
 
 xn 
bn 
It is now hopeless to start from the first equation because it might contain
terms in all variables. A more profitable approach is to start from the last
b
equation, obtaining unn xn  bn  xn  n . We can then this value for xn and
unn
proceed backward to the equation before the last, which reads
b  un1,n xn
, because xn is already known
un1,n1 xn1  un1,n xn  bn  xn1  n
unn
at this point. It is now easy to identify the general form for the step to calculated
xi . In it, we would have already calculated the values of xi 1 , , xn and could use
n

1
these values to obtain xi   bi   uij x j  . This algorithm for solving an
uii 
j i 1

upper-triangular linear system is called backward substitution.
Most linear systems encountered in practice are neither lower nor upper
triangular, and we cannot apply the convenient forward and backward
substitution methods directly. However, we can try to transform a given system
into lower- or upper-triangular systems having the same solutions. To do that ,
we must know what kinds of transformations of linear systems have the property
of preserving their solutions.
One such transformation is obtained by multiplying both sides of the matrix
equation Ax  b by a nonsingular matrix M . To see this, observe that if x0
satisfies MAx0  Mb , then we have Ax0  M 1  MAx0   M 1MAb  b , which
shows that x0 is a solution to the original system Ax  b .
The method known as Gaussian elimination consists of successive
transformations of a system through multiplication by nonsingular matrices M k
unit the resulting system is upper triangular, which can then be solved by
backward substitution. Start with the following example:
3
3 
1 2 2   x1 
Ax   4 4 2   x2   6   b . If you multiply both sides of this matrix
 


10
 2 6 4   x3 
 1 0 0
equation
by
you
obtain
M 1   4 1 0  ,


 2 0 1
2 2
 1 0 0  1 2 2 
1
M 1 A   4 1 0  4 4 2   0  4  6  and

 



 2 0 1  2 6 4 
0 2 0 
 1 0 0  3 
 3
M 1b   4 1 0   6    6

  

 2 0 1 10 
 4

 . Observe what was achieved in this step: all


the elements below the diagonal on the first column of the matrix M 1 A are zero.
Next, multiply both sides of the new equation M 1 Ax  M 1b by
0 0
1
M 2  0
1 0  , obtaining


0 0.5 1
0 0  1
2 2
1
M 2 M 1 A  0
1 0  0  4  6  

 

0 0.5 1 0 2 0 
2 2
1
0  4  6  and


0 0  3 
0 0   3
1
 3
M 2 M 1b  0
1 0  6    6 .

  
 
0 0.5 1  4
 1 
Observe now that the modified matrix M 2 M 1 A is upper triangular, so that
the system M 2 M 1 Ax  M 2 M 1b can be solved by backward substitution.
The general strategy for Gaussian elimination is to look at a matrix of the
4
1


0
form M k  
0


0
vector v 
n
1


0

0


0
0
0
1
0
0


0
 . The effect of this matrix on a general
0


1 
 mk 1 1
 mn
0
is
0
0
1
0
 mk 1 1
 mn
0
0


0

0


1 
v1
v1 


 


 


vk 

 .
vk
   

vk 1 
vk 1  mk 1vk 
 


 


vn 
vn  mnvk 
In particular, if the matrix A has already been made upper triangular up to its
a
first k  1 columns and we choose mi  ik , i  k  1, , n , then the matrix M k
akk
leaves the first k  1 columns of A unchanged while transforming the k th
 a1k 
 
 
a 
column into  kk  . For this reason, the matrices M k are called elementary
0 
 
 
 0 
elimination matrices. These elimination matrices are lower triangular with all of
their diagonal entries being equal to 1. Hence, we conclude that each of them is
invertible, and so is their product. Therefore, as long as none of the diagonal
elements of A is zero, successive multiplication by exactly n  1 of such
matrices leads to an upper triangular system of the form
M n1 M 2 M 1 Ax  M n1 M 2 M 1b having the same solution as the original
system Ax  b .
Square Roots Method
We consider linear algebraic equations system
Ах  b .
(1)
The special class of matrices for which this method is applicable is that of
symmetric positive definite matrices. We say that an n  n matrix A is
5
symmetric if A  A . For matrices with real entries, this simply means that
aij  a ji for all i, j , while for matrices with complex numbers as entries, the
symmetry condition means that aij  a ji for all i, j .
Positivity is a more delicate topic. In analogy with real numbers, we would
like to say that a matrix is positive if it is, in some sense, “greater than zero.”
But because it is difficult to introduce an appropriate order relation for
multidimensional objects such as matrices, the definition of positive requires
more work.
If we consider matrices not just as arrays of real (or complex) numbers, but
as linear transformations between vector spaces, then it turns out that positivity
can be defined according to their effect on vectors. More precisely, because x
and Ax are, respectively, a row and a column vector, we know that the product
x   Ax  , which can be denoted simply by xAx , is a real (or complex) number.
Let matrix A is symmetric and the scalar xAx is always a real number. We then
say that a symmetric matrix A is positive definite if xAx  0 for all nonzero
x  n (or n ).
The matrix A we can be written as (symmetric and positive definite real
matrix)
(2)
A  T T ,
t11 t12 ... t1n 
t11 0 ... 0 


where T  0 t22 ... t2 n  and T   t12 t22 ... 0  .


0 0 ... tnn 
t1n t2 n ... t nn 
It follows from equation (2) that the tij elements of matrices T and T  defined:
t1it1 j  t2it2 j  ...  tiitij  aij ,
t12i  t22i  ...  tii2  aii
(i  j ) 






a
t11  a11 , t1 j  1 j
( j  1)

t11


i 1

(3)
tii  aii   tki2
(1  i  n)

k 1

i 1

aij   tkitkj

k 1
tij 
, (i  j ),
tij  0, i  j 

tii
If tii  0 (i  1,2,..., n) , then the system (1) has unique solution given and
det A  det T   det T  (det T )2  (t11t 12 ...tnn )2  0. T The coefficients of matrix T
is real if tii2  0 .
It follows from equation (1) with condition (2) that T y  b,
6
Тх  у or
t11 y1  b1


t12 y1  t22 y2  b2



t1n y1  t2 n y2   tnn yn  bп 
t11 x1  t12 x2 
t22 x2  t23 x3 
tnn xn  yn
 t1n xn  y1 
 t2 n xn  y2 



(4)
(5)
it follows that
y1 
b1
,
t11
i 1
yi 
хn 
xi 
bi   tki yk
k 1
tii
yn
,
tnn
yi 
n
 tik xk
k i 1
tii
Remark 1 If tss2  0 then t sj is complex.



,

(i  1) 




 .

(i  n) 

(6)
(7)
Cholesky Factorization
We consider linear algebraic equations system
Ax  b
(1)
The matrix A is symmetric and positive definite real matrix. The matrix A we
can be written as
A  B C ,
(2)
b11 0
b b
where B   21 22


bn1 bn 2
0 
0 
,


bnn 
1 c12
0 1
C


0 0
7
c1n 
c2 n 
.


1 
It follows from equation (2) that bij and cij elements of matrices B and C defined:
bi1  ai1 ,
j 1
bij  aij   bik ckj
k 1
c1 j 
a1 j
b11
1
cij 
bii
,
i 1


  aij   bik ckj 
k 1




,
 i  j  1

(3)




1  i  j 

(4)
it follows that
Ву  b ,
Сх  y .
(5)
From here
a
y1  1,n1 ,
b11
1
уi 
bii
i 1


  ai ,n1   bik yk  ,
k 1


i  1
(6)
and
 xn  yn

n
,

x

y

c
x

i
i
ik
k

k i 1

i  n .
Remark 2 If A is symmetric matrix, then cij 
8
(7)
b ji
bii
i  j  .