Download Teo

The Order Of The Gauss-Seidel Iterative Method
For Solving Linear Systems
Teo L.H. and Lin P.
Department of Mathematics, Faculty of Science,
National University of Singapore
3 Science Drive 2, Singapore 117543
ABSTRACT
The main purpose of this paper is to discuss how the implementation order of the GaussSeidel method affects its convergence rate, from a number of examples of linear systems.
Systems of equations are used to analytically represent physical problems that involve the
interaction of various properties. Although not all physical problems can be represented using a
linear system, the solution to many problems either have this form or can be approximated by
such a system. There are two types of methods in solving linear system. One is called direct
method. The other is call iterative method. Iterative methods are popular because they are more
effective particularly when the matrix is sparse, that is, the matrix has a high percentage of zero
entries. Another reason is that they require far less computational and reduce round-off error. In
many instances, however, methods converge slowly. Therefore, the study of convergence rate is
very important and very helpful in application of the method. The Gauss-Seidel method is a
popular classical iterative method.
INTRODUCTION
In this paper, we shall discuss an iterative method, the Gauss-Seidel method. This iterative
algorithm is used to approximate the solution of a linear equation
Ax=b,
(1)
where A is large and sparse. The matrix A can be written in many different ways, decomposing
it as a sum of upper and lower triangular matrices. Consider A = N – P. The system can be
rewritten as
Nx = Px+b.
(2)
The iterative method starts with an initial guess vector x(0) and compares a sequence of vectors
x(k), k = 1,2,…, using
Nx(k) = Px(k-1)+b,
k = 1,2,….
(3)
The matrix N should be chosen such that it is invertible and it’s associated linear systems easier
to solve.
GAUSS-SEIDEL ITERATIVE METHOD
The matrix form of the Gauss-Seidel method is given by
x
(k)
 Ms x
(k 1)
1
 (D  L) b,
(4)
where D = diag(a11, a22, …, ann), and L as the strictly lower triangular part of A minus the
diagonal and U as the strictly upper triangular part of A minus the diagonal, we chose N = D+L
and Ms := -(D+L)-1U is called the Gauss-Seidel matrix.
Definition 1 A matrix A is positive definite if it is symmetric and if xtAx > 0 for every ndimensional column vector x  0.
Theorem 1 Let A be a positive definite matrix. The vector sequence x(k), k = 1,2,…, generated
by the Gauss-Seidel method converges to the true solution of Ax = b for any starting vector x(0).
Definition 2 An n x n matrix A is strictly diagonally dominant if
| aii |
n
| a
j 1. j  i
ij
| , i = 1,2 ,…,n.
(5)
where aij are the entries of the matrix A.
Theorem 2 Let A = N – P, N invertible, xt be the true solution of Ax = b and
Nx ( k )  Px ( k 1)  b , k = 1, 2, …, x(0) be given.
Then the rate of convergence c is given by c || N 1 P || and || x ( k )  x t || c k || x ( 0)  x t || .
THE IMPLEMENTATION ORDER OF THE GAUSS-SEIDEL METHOD
The following section redefines the problem mathematically and discusses the convergence
of the newly defined system.
Redefining the problem
When the implementation order of the Gauss-Seidel method is
changed, the matrix M is changed as well. What we are investigating is how c (=||N-1P||) is
affected by the change in the matrix M (=N-1P). In other words, find the number to iterations
taken for each ordering.
Now, the problem is to use the ith equation to find xi first. This is equivalent to using the
(i)th equation to find x(i) in the order {(1), …, (n)}. The Gauss-Seidel method is used, but
the matrices A and b have to be redefined accordingly to preserve the original system. As such,
the following theorem and proposition discuss how the system is redefined and A and b are
modified accordingly.
Definition 3 Q is a permutation matrix if it is a composition of elementary matrices of the first
type only, i.e. matrices obtained from the identity matrix by swapping 2 columns or rows.
Note: Q is a multiplication of elementary matrices of the 1st kind, which are called transposition
matrices (row swapping). i.e. Q = E1… En.
Theorem 3 Elementary matrices are projection matrices, i.e. E = E-1.
Proposition 1 Permutation matrices are orthogonal matrices.
Defining A’ = QAQT, x’ = Qx and b’ = Qb, the new system A’x’ = b’ is equivalent to Ax = b
Thus, the original problem is reduced to a matter of finding the matrix A’, which also involves
finding the permutation matrix Q.
Proposition 2 If A is symmetric positive definite, then A’ = QAQT is symmetric positive
definite, where Q is a permutation matrix.
Proposition 3 If A is diagonally dominant, A’ is diagonally dominant.
From the above propositions, theoretically the new system should converge if the old does.
The results
In this paper, we have restricted A to be an even square matrix. The argument when A is an
odd matrix is similar. A is also restricted to be a relatively sparse matrix as iterative methods are
particularly effective for large and sparse matrices. The tolerance level is fixed at 10-8.
The orderings used are as follows:
(1) Natural ordering;
(2) Inverse ordering;
(3) Odd-even ordering;
(4) Even-odd ordering.
The cases for matrix A are as follows:
(1) Small, sparse and diagonally dominant.
(2) Large, sparse and diagonally dominant.
(3) Large and tridiagonal.
Case 1 Small, sparse and diagonally dominate
The outputs were as follows:
For the natural ordering, the number of iterations was 14.
For the inverse ordering, the number of iterations was 12.
For the odd-even ordering, the number of iterations was 11.
For the even-odd ordering, the number of iterations was 11.
Case 2 Large, sparse and diagonally dominant
The output for all four orderings was 7 iterations for A,50 x 50, and 100 x 100.
Case 3 Large and tridiagonal
The outputs were as follows:
For the natural ordering, the number of iterations was 15.
For the inverse ordering, the number of iterations was 15.
For the odd-even ordering, the number of iterations was 13.
For the even-odd ordering, the number of iterations was 13.
CONCLUSION AND IMPLICATIONS
From the above results and from the theoretical analysis of the Gauss-Seidel method, we can
conclude:
(i)
The implementation order does not affect the convergence rate for large and
sparse matrices, which are diagonally dominant.
(ii)
For small systems, the convergence is slightly different with each
implementation.
(iii) For tridiagonal matrices, the convergence rate differs slightly as well.
Remark: A generalization of the applying permutation matrices could be extended to the nonsymmetric permutations of the rows and columns.
Instead of applying Q and QT , a matrix P is used as the permutation matrix for unknowns. Thus
the new system is as follows.
QAPT Px  Qb is equivalent to Ax = b, and A’ = QAPT, x’ = Px and b’=Qb.
With suitable permutation matrices, any system will converge using the Gauss-Seidel
method. But, it is computationally expensive to find such matrices. Thus, the natural ordering
has been used as a convention in the wake of new numerical methods and the Gauss-Seidel has
been taken a standard iterative method.
To discuss the convergence rate of the implementation order of the Gauss-Seidel method, as
the results have shown, is the same for large and sparse matrices, and slightly different for
certain classes of matrices (e.g. Tridiagonal matrices). To exhaust all classes of matrices is
impossible.
In conclusion, the implementation order does affect the convergence rate of the Gauss-Seidel
method, in varying degrees according to which permutation matrices are used.
REFERENCES
Burden, Richard L. and Faires, J. Douglas (1997), Numerical analysis (6th edition),
Brooks/Cole Pub. Co., Pacific Grove, Calif.
Hageman, Louis A. and Young, David M. (1981), Applied iterative methods, New York:
Academic Press.
Varga, Richard (1962), Matrix iterative analysis, Prentice Hall, Englewood Cliffs, N.J.