Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Linear least squares (mathematics) wikipedia , lookup
Resampling (statistics) wikipedia , lookup
Horner's method wikipedia , lookup
Dynamic substructuring wikipedia , lookup
Compressed sensing wikipedia , lookup
System of linear equations wikipedia , lookup
Determinant wikipedia , lookup
Singular-value decomposition wikipedia , lookup
Newton's method wikipedia , lookup
Root-finding algorithm wikipedia , lookup
Gaussian elimination wikipedia , lookup
The Order Of The Gauss-Seidel Iterative Method For Solving Linear Systems Teo L.H. and Lin P. Department of Mathematics, Faculty of Science, National University of Singapore 3 Science Drive 2, Singapore 117543 ABSTRACT The main purpose of this paper is to discuss how the implementation order of the GaussSeidel method affects its convergence rate, from a number of examples of linear systems. Systems of equations are used to analytically represent physical problems that involve the interaction of various properties. Although not all physical problems can be represented using a linear system, the solution to many problems either have this form or can be approximated by such a system. There are two types of methods in solving linear system. One is called direct method. The other is call iterative method. Iterative methods are popular because they are more effective particularly when the matrix is sparse, that is, the matrix has a high percentage of zero entries. Another reason is that they require far less computational and reduce round-off error. In many instances, however, methods converge slowly. Therefore, the study of convergence rate is very important and very helpful in application of the method. The Gauss-Seidel method is a popular classical iterative method. INTRODUCTION In this paper, we shall discuss an iterative method, the Gauss-Seidel method. This iterative algorithm is used to approximate the solution of a linear equation Ax=b, (1) where A is large and sparse. The matrix A can be written in many different ways, decomposing it as a sum of upper and lower triangular matrices. Consider A = N – P. The system can be rewritten as Nx = Px+b. (2) The iterative method starts with an initial guess vector x(0) and compares a sequence of vectors x(k), k = 1,2,…, using Nx(k) = Px(k-1)+b, k = 1,2,…. (3) The matrix N should be chosen such that it is invertible and it’s associated linear systems easier to solve. GAUSS-SEIDEL ITERATIVE METHOD The matrix form of the Gauss-Seidel method is given by x (k) Ms x (k 1) 1 (D L) b, (4) where D = diag(a11, a22, …, ann), and L as the strictly lower triangular part of A minus the diagonal and U as the strictly upper triangular part of A minus the diagonal, we chose N = D+L and Ms := -(D+L)-1U is called the Gauss-Seidel matrix. Definition 1 A matrix A is positive definite if it is symmetric and if xtAx > 0 for every ndimensional column vector x 0. Theorem 1 Let A be a positive definite matrix. The vector sequence x(k), k = 1,2,…, generated by the Gauss-Seidel method converges to the true solution of Ax = b for any starting vector x(0). Definition 2 An n x n matrix A is strictly diagonally dominant if | aii | n | a j 1. j i ij | , i = 1,2 ,…,n. (5) where aij are the entries of the matrix A. Theorem 2 Let A = N – P, N invertible, xt be the true solution of Ax = b and Nx ( k ) Px ( k 1) b , k = 1, 2, …, x(0) be given. Then the rate of convergence c is given by c || N 1 P || and || x ( k ) x t || c k || x ( 0) x t || . THE IMPLEMENTATION ORDER OF THE GAUSS-SEIDEL METHOD The following section redefines the problem mathematically and discusses the convergence of the newly defined system. Redefining the problem When the implementation order of the Gauss-Seidel method is changed, the matrix M is changed as well. What we are investigating is how c (=||N-1P||) is affected by the change in the matrix M (=N-1P). In other words, find the number to iterations taken for each ordering. Now, the problem is to use the ith equation to find xi first. This is equivalent to using the (i)th equation to find x(i) in the order {(1), …, (n)}. The Gauss-Seidel method is used, but the matrices A and b have to be redefined accordingly to preserve the original system. As such, the following theorem and proposition discuss how the system is redefined and A and b are modified accordingly. Definition 3 Q is a permutation matrix if it is a composition of elementary matrices of the first type only, i.e. matrices obtained from the identity matrix by swapping 2 columns or rows. Note: Q is a multiplication of elementary matrices of the 1st kind, which are called transposition matrices (row swapping). i.e. Q = E1… En. Theorem 3 Elementary matrices are projection matrices, i.e. E = E-1. Proposition 1 Permutation matrices are orthogonal matrices. Defining A’ = QAQT, x’ = Qx and b’ = Qb, the new system A’x’ = b’ is equivalent to Ax = b Thus, the original problem is reduced to a matter of finding the matrix A’, which also involves finding the permutation matrix Q. Proposition 2 If A is symmetric positive definite, then A’ = QAQT is symmetric positive definite, where Q is a permutation matrix. Proposition 3 If A is diagonally dominant, A’ is diagonally dominant. From the above propositions, theoretically the new system should converge if the old does. The results In this paper, we have restricted A to be an even square matrix. The argument when A is an odd matrix is similar. A is also restricted to be a relatively sparse matrix as iterative methods are particularly effective for large and sparse matrices. The tolerance level is fixed at 10-8. The orderings used are as follows: (1) Natural ordering; (2) Inverse ordering; (3) Odd-even ordering; (4) Even-odd ordering. The cases for matrix A are as follows: (1) Small, sparse and diagonally dominant. (2) Large, sparse and diagonally dominant. (3) Large and tridiagonal. Case 1 Small, sparse and diagonally dominate The outputs were as follows: For the natural ordering, the number of iterations was 14. For the inverse ordering, the number of iterations was 12. For the odd-even ordering, the number of iterations was 11. For the even-odd ordering, the number of iterations was 11. Case 2 Large, sparse and diagonally dominant The output for all four orderings was 7 iterations for A,50 x 50, and 100 x 100. Case 3 Large and tridiagonal The outputs were as follows: For the natural ordering, the number of iterations was 15. For the inverse ordering, the number of iterations was 15. For the odd-even ordering, the number of iterations was 13. For the even-odd ordering, the number of iterations was 13. CONCLUSION AND IMPLICATIONS From the above results and from the theoretical analysis of the Gauss-Seidel method, we can conclude: (i) The implementation order does not affect the convergence rate for large and sparse matrices, which are diagonally dominant. (ii) For small systems, the convergence is slightly different with each implementation. (iii) For tridiagonal matrices, the convergence rate differs slightly as well. Remark: A generalization of the applying permutation matrices could be extended to the nonsymmetric permutations of the rows and columns. Instead of applying Q and QT , a matrix P is used as the permutation matrix for unknowns. Thus the new system is as follows. QAPT Px Qb is equivalent to Ax = b, and A’ = QAPT, x’ = Px and b’=Qb. With suitable permutation matrices, any system will converge using the Gauss-Seidel method. But, it is computationally expensive to find such matrices. Thus, the natural ordering has been used as a convention in the wake of new numerical methods and the Gauss-Seidel has been taken a standard iterative method. To discuss the convergence rate of the implementation order of the Gauss-Seidel method, as the results have shown, is the same for large and sparse matrices, and slightly different for certain classes of matrices (e.g. Tridiagonal matrices). To exhaust all classes of matrices is impossible. In conclusion, the implementation order does affect the convergence rate of the Gauss-Seidel method, in varying degrees according to which permutation matrices are used. REFERENCES Burden, Richard L. and Faires, J. Douglas (1997), Numerical analysis (6th edition), Brooks/Cole Pub. Co., Pacific Grove, Calif. Hageman, Louis A. and Young, David M. (1981), Applied iterative methods, New York: Academic Press. Varga, Richard (1962), Matrix iterative analysis, Prentice Hall, Englewood Cliffs, N.J.