Download Handout #5

Handout #5
Matrix and Von Neumann Stability Analyses (Continued…)
In the previous handout, we showed that the error in the solution at the end of an iteration
(n+1) is related to the error at the start of the iteration by the relation:
en1  M n1 e0
(1)
where the matrix {M} is given by:
M   I  B 1 A
(2)
Recall that [A] is the original matrix for solving the system
AT   b
(3)
while [B] is a more easily inverted matrix, which allows us to solve the system:
BT   b  AT n  R
(4)
Here {R} is called the residual.
If [B] is identical to [A], then [M] will be a null matrix( i.e. a matrix with only zero
elements). In that case, the error (as given by equation 1) will reduce to zero in just one
iteration. This is not surprising, because if [B] equals [A], we are solving equation (3)
exactly. We can also conclude that the closer is the matrix [B] to the original matrix [A],
the faster it will converge. This is why an SLOR scheme which retains 4 of the original 5
diagonals of the matrix [A] in your computer project works so well compared to the Point
Jacobi scheme, which retains only the main diagonal of [A].
Using equation (1) to study the stability of iterative schemes is not practical, because it
requires computing the matrix [M], which is relatively easily done, and multiplying [M]
by itself several times (not so easily done). This effort is necessary to make sure
that[M]n+1 turns into a null matrix after sufficiently large number of ‘n’, i.e. after lots of
iterations.
Fortunately, there is an easier way. From matrix algebra, any matrix [M] may be written
as
M   T  T 1
(5)
where {] is a diagonal matrix which contains the eigenvalues of [M], while T contains
the eigenvectors of the matrix [M] as its columns. If you have forgotten what
eigenvalues and eigenvectors are, join the club. Eigenvalues are simply solutions to the
equation:
detM  I   0
(6)
If [M] is an NxN matrix, there will be N roots. These will, in general, be complex
numbers. A program such as Mathematica or MATLAB will spit out these eigenvalues in
a matter of seconds (unless you use a Mac) for sufficiently small values of N.
An eigenvector is an array {x} that is a solution to:
M x  x
(7)
while MATLAB will give the eigenvectors as well, these are not needed.
Using equation (5), we can write [M]n+1 as:
M n1  [M ][ M ][ M ]....  [T ][][T ]1[T ][][T ]1[T ][][T ]1 ...  [T ][]n1[T ]1
(8)
If the absolute magnitude of each of the eigenvalues of [] were less than one (unity for
you mathematicians), then []n+1 will go to zero. After all, multiplying a number less
than unity (real or complex) by itself enough times will yield zero.
In summary,
“If the absolute value of the largest eigenvalue of the matrix [M]= [I-B-1A] is less than
unity, the iterative process will converge. “
(Note: The absolute value of the largest eigenvalue of a matrix is called the spectral
radius of the matrix by mathematicians).
Application of the matrix stability analysis is easy, although it consumes a lot of memory
and can only be done on a computer. It requires
(a) computing the matrix [A],
(b) computing the matrix [B],
(c) computing the matrix [M],
and,
(d) finally computing the spectral radius of [M].
Matrix stability analysis is usually done for a small size matrix (one corresponding to
N=5 or N=10), say. If it works for such small size problems, it is assumed (with the
fingers crossed) that the spectral radius will be less than unity for large size problems as
well.
Of course, one can arbitrarily try different [B] matrices in an iterative scheme to see
which one works. This kind of programming seldom works (although there have been a
few brilliant discoveries totally attributable to pure chance- e.g. penicillin, radium). As
Robert Heinlein says in his sci-fi classic Strangers in a Strange Land, “Experience is a
hard teacher. But fools will learn no other way…”
Von Neumann Stability Analysis
The Von Neumann stability analysis is another way of finding which choices for
the matrix [B] work best. We will look at the stability of Point Jacobi scheme as
predicted by the Von Neumann stability analysis. Other schemes (SOR, Gauss-Seidel
etc.) can be analyzed in a similar fashion.
For the Point Jacobi scheme, from equation (12) of handout #4, at a typical node i, we
get:


 2 Ti n 1  Ti n
Ti n1  2Ti n  Ti n1

q

x 2
x 2
(9)
The exact solution to our discretized form of the equation is:


 2 Ti exact  Ti exact
Ti exact
 2Ti exact  Ti exact
1
1

q

x 2
x 2
(10)
Subtract (10) from equation (9). The difference between the computed solution at any
iteration‘n’ (or ‘n+1’) and the exact solution is given the symbol ‘e’. In other words,
T n  T exact  e n
T n 1  T exact  e n 1
(11)
With this definition of error, equations (10) subtracted from equation (9) gives:


 2 ein 1  ein
ein1  2ein  ein1


x 2
x 2
(12)
Simplifying equation (12), we get for the Point Jacobi scheme:
ein 1 
ein1  ein1
2
(13)
Von Neumann assumed that the error in a solution at the end of any iteration ‘n’ may be
written as a Fourier series:
e n x    Apn exp( jpx)
e n1 x    Apn 1 exp( jpx)
(14)
This may not look like the Fourier series that we learned (and forgot) in Math
4582. The symbol j is simply the square root of –1. This quantity is usually called ‘i’ in
complex number theory, but we have already used the symbol ‘i’ to denote the nodes.
The quantity exp(jpx) is simply sin(px) + j cos(px). The coefficient Apn is a complex
number of the type C+jD where C and D are real numbers. If we perform the
multiplication of Apn and exp(jpx), and retain only the real part, we will obtain a more
traditional representation of the Fourier series.
The symbol ‘p’ is called the wave number. In the Fourier series above, the terms
for which ‘p’ is small represents slowly varying terms (sin2x, for example), where as
large values of p (sin10x for example) represents rapidly varying components. The
components corresponding to small values of p are called, therefore, the low frequency
error components, and vice versa.
The quantity Apn is called the amplitude of the component exp(jpx) at the iteration
n. Obviously, the smaller this amplitude, the smaller the error. We hope that as the
iterations increase Apn will go to zero for each and every component in the above series.
In other words, we hope that the ratio Apn+1 / Apn is less than 1 in magnitude for all ‘n’
(i.e. for every iteration) for all ‘p’.
Because equation (13) is linear, Von Neumann suggested that we can test one
component of the Fourier series at a time to see how if that particular component varies.
Taking a typical term in the Fourier series given in equation (14) and plugging it into
equation (13), we get:
A pn exp( jp[ x  x])  A pn exp( jp[ x  x])
A pn 1 exp( jpx) 
2
Cancel _ common _ terms :
A
n 1
p
A
n 1
p
A pn

A pn exp( j[ px])  A pn exp( j[ px])
2
 A pn cos( px)
 cos( px)
(15)
The Amplitude at the end of the n+1 iteration of a component of the error will
always be less than the amplitude at the end of iteration n for that component. In other
words, Point Jacobi method will always converge, since cos(px) is always less than
unity.
Of course, if p is small, or x is small (fine grid), the convergence will be
extremely slow. This is why your computer code converges more slowly on a fine grid
than on a coarse grid.
We will next apply the Von Neumann stability analysis to the Gauss Seidel
scheme. From handout #4, for our model ODE, Gauss-Seidel scheme becomes:
Ti 1
Ti n1  2Ti n  Ti n1
2


T

q

i
x 2 x 2
x 2
(16)
The exact solution satisfies:
Ti exact
 2Ti exact  Ti exact
1
1
0q
2
x
(17)
Subtract (17) from (16). Use:
e n  T n  T exact
e n 1  T n 1  T exact
e n 1  e n  T n 1  T n  T
We get:

 
ein11  ein1  2 ein1  ein   ein1  2ein  ein1
Or ,

   
ein11  2 ein1   ein1
The error e is made of a Fourier series, given by equation (14). We test a single
component of error at a time. A single component of en is Apn exp(jpx), while a single
component of en+1 is given by Apn exp(jpx). Plugging these definitions in, and canceling
out the common term exp(jpx), we get:
Apn1 exp(  px)  2 Apn1   Apn exp( px)
Simplifying, we get :
A pn 1
A
n
p

exp( px)
2  exp(  px))
Or ,
A pn 1
A
n
p

1
5  4 cos px 
Since the denominator is always greater than the numerator, the amplitudes steadily
decrease. Thus the Gauss-Seidel scheme also always converges.
We can similarly test other schemes (SOR, SLOR) and show they are all stable.