Download MT5241 Computational Fluid Dynamics I

MT5241 Computational Fluid Dynamics I
J.S.B. Gajjar
September 2004
1
Introduction
In this course we will be studying computational fluid dynamics or CFD. CFD
is concerned with the the study of fluid flow problems using computational techniques, as opposed to analytical or experimental methods. The modelling of fluid
flow leads to the solution of unsteady, nonlinear, partial differential equations,
and there are only a handful of known ‘exact’ analytical solution. Even in the
simplest of geometries the solution of these equations is quite challenging. In this
CFD course we will aim to study the different techniques which are used to solve
the different types of equations which arise. As we will see, to be good at CFD,
one has to be fluent in many different subject areas such as numerical analysis,
programming, fluid mechanics. This introductory course will concentrate mainly
on finite-difference methods. In CFD II you will meet the popular finite-volume
method. In the finite-difference method, the solution is sought on a grid of points,
and we deal with function values at these nodal points. In a finite-element environment on the other hand, the regions is split into various sub-regions, and the
solution in these sub-regions is approximated by basis functions.
1.1
Errors
To start with we need to discuss errors. Typically numerical computation of any
sort will generate errors and it is one of the tasks of the CFD practitioner to be
able to assess and be aware of the errors arising from numerical computation.
The different types of errors can be categorised into the following main types:
• Roundoff errors.
• Errors in modelling.
• Programming errors.
• Truncation and discretization errors.
Roundoff errors.
1
These arise when a computer is used for doing numerical calculations. Some
√
typical examples include the inexact representation of numbers such as π, 2.
Roundoff and chopping errors arise from the way numbers are stored on the
computer, and in the way arithmetic operations are carried out. Whereas most
of the time the way numbers are stored on a computer is not under our control,
the way certain expressions are computed definitely is. A simple example will
illustrate the point. Consider the computation of the roots of a quadratic equation
ax2 + bx + c = 0 with the expressions
√
√
−b + b2 − 4ac
−b − b2 − 4ac
x1 =
, x2 =
.
2a
2a
Let us take a = c = 1, b = −28. Then
√
√
x1 = 14 + 195, x2 = 14 − 195.
√
Now to 5 significant figures we have 195 = 13.964. Hence x1 = 27.964 x2 =
0.036. So what can we say about the error. We need some way to quantify errors.
There are two useful measures for this.
Absolute error
Suppose that φ∗ is an approximation to a quantity φ. Then the absolute error
is defined by |φ∗ − φ|.
Relative error
Another measure is the relative error and this is defined by |φ∗ − φ|/|φ| if
φ 6= 0.
For our example above we see that |x∗1 − x1 | = 2.4 ∗ 10−4 and |x∗2 − x2 | =
|x∗ −x |
2.4 ∗ 10−4 which look small. On the other hand the relative errors are 1|x1 | 1 =
|x∗ −x |
8.6 ∗ 10−6 and 2|x2 | 2 = 6.7 ∗ 10−3 . Thus the accuracy in computing x2 is far less
than in computing x1 . On the other hand if we compute x2 via
x2 = 14 −
√
195 =
14 +
1
√
195
we obtain x2 = 0.03576 with a much smaller absolute error of 3.4 ∗ 10−7 and a
relative error of 9.6 ∗ 10−6 .
Errors in numerical modelling.
These arise for example when the equations being solved are not the proper
equations for the flow problem in question. An example is using the Euler equations for calculating the solution of flows where viscosity effects are important.
No matter how accurately the solution has been computed, it may not be close
to the real physical solution because viscous effects have been neglected throught
the computation.
Programming errors, and bugs.
These are errors entirely under the control of the programmer. To eliminate
these requires careful testing of the code and logic, as well as comparison with
2
previous work. Even then, for your problem for which there may not be previous work to compare with, one has to do numerous self-consistency checks with
further analysis as necessary.
Truncation and discretization errors.
These errors arise when we take the continuum model and replace it with a
discrete approximation. For example suppose we wish to solve
d2 U
= f (x).
dx2
Using Taylor series we can apoproximate the second derivative term by
u(xi+1 ) − 2u(xi ) + u(xi−1 )
h2
where we have taken a uniform grid with spacing h say and node points xi . As far
the approximation of the equation is concerned we will have a truncation error
given by
d2 U (x1 )
h2 d4 U (xi )
τ (xi ) =
−
f
(x
)
=
+ ....
i
dx2
12 dx4
Even though the discrete equations may be solved to high accuracy, there will be
still an error of O(h2 ) arising from the discretization of the equations. Of course
with more points, we would expect the error to diminish.
3
2
Initial value problems
Here we will look at the solution of ordinary differential equations of the type,
say
dy
= f (x, y), a ≤ x ≤ b,
dx
subject to an initial condition
y(a) = α
(1)
(2)
The methods that we will use generalise readily to systems of differential equations of the type
dY
= F(x, Y), a ≤ x ≤ b,
(3)
dx
where
Y = (y1 (x), y2 (x), ..., yN (x))T ,
F = (f1 (x, Y), f2 (x, Y), ..., fN (x, Y))T ,
with initial data
Y(a) = α,
(4)
say, where α = (α1 , α2 , ..., αN )T .
One question which arises immediately is, what about second and third order
differential equations? Consider for example
1
g 000 + gg 00 + (1 − g 02 ) = 0,
2
g(0) = g 0 (0) = g 0 (∞) − 1 = 0,
(5)
which arises in fluid applications. The answer to this is that if we take any
differential equation, we can always write it as a system of first order equations.
Example
In (5) let
Y1 = g(x), Y2 = g 0 (x), Y3 = g 00 (x),
then (5) becomes


Y2
dY 

Y3
=
 = F.
dx
1
2
−Y1 Y3 − 2 (1 − Y1 )
(6)
The boundary conditions in (5) are not of the form as in (4) because we have
conditions given at two points x = 0 and x = ∞. We will see how to deal with
this later.
Consider equation (1). There are various mathematical results that we should
be aware of, although we will not go too deeply into the implications or theory
behind some of these.
Suppose we define D to be the domain
D = {(x, y) | a ≤ x ≤ b,
4
−∞ < y < ∞}
and f (x, y) is continuous on D. If f (x, y) satisfies a Lipschitz condition on D
then (1) has a unique solution for a ≤ x ≤ b. Recall f (x, y) satisfies a Lipschitz
condition on D means that there exists a constant L > 0 (called the Lipschitz
constant) such that
|f (x1 , y1 ) − f (x2 , y2 )| ≤ L|y1 − y2 |
whenever (x1 , y1 ), (x2 , y2 ) belong to D.
2.1
Euler’s Method
This is the simplest of techniques for the numerical solution of (1). It is good for
proving various results, but it is rarely used in practice because of far superior
methods.
For simplicity define an equally spaced mesh
Sfrag replacements
xj = a + jh,
j = 0, .., N
where h = (b − a)/N is called the step size.
h
a = x0
x1
x2
xi
xi+1
xN = b
We can derive Euler’s method as follows. Suppose y(x) is the unique solution
to (1), and twice differentiable. Then by Taylor’s theorem we have
h2 00
y(xi+1 ) = y(xi + h) = y(xi ) + y (xi )h + y (ξ)
2
0
(7)
where xi ≤ ξ ≤ xi+1 . But from the differential equation y 0 (xi ) = f (xi ), and
yi = y(xi ). This suggests the scheme
w0 = α
wi+1 = wi + hf (xi , wi ),
i = 1, 2, .., N − 1,
(8)
for calculating the wi which will be our approximate solution to the equation.
This is called Euler’s method.
2.1.1
Truncation error for Euler’s method
Suppose that yi = y(xi ) is the exact solution (1) at x = xi . Then the truncation
error is defined by
yi+1 − (yi + hf (xi , yi ))
h
yi+1 − yi
=
− f (xi , yi ),
h
τi+1 (h) =
5
(9)
for i = 0, 1, ..., N − 1. From earlier (7) we find that
τi+1 (h) =
h 00
y (ξi )
2
for some ξi in (xi , xi+1 ). So if y 00 (x) is bounded by a constant M in (a, b) then
|τi+1 (h)| ≤
h
M
2
and thus we see that the truncation error for Euler’s method is O(h). In general
if τi+1 = hp we say that the method is of order hp . In principle if h decreases, we
should be able to achieve greater accuracy, although in practice roundoff error
limits the smallest size of h that we can take.
2.2
Higher order methods
2.2.1
Modified Euler
The modified Euler method is given by
w0 = α
k1 = hf (xi , wi ),
h
wi+1 = wi + [f (xi , wi ) + f (xi+1 , wi+1 )],
2
i = 1, 2, .., N − 1,
This has truncation error O(h2 ). Sometimes this is also called a Runge-Kutta
method of order 2.
2.2.2
Runge-Kutta method of order 4
One of the most common Runge-Kutta methods of order 4 is given by
w0 = α,
k1 = hf (xi , wi ),
1
h
k2 = hf (xi + , wi + k1 )
2
2
h
1
k3 = hf (xi + , wi + k2 )
2
2
k4 = hf (xi+1 , wi + k3 )
1
wi+1 = wi + (k1 + 2k2 + 2k3 + k4 )
6
(10)
for i = 0, 1, .., N − 1.
Note that this requires 4 function evaluations per step.
All these methods generalise to a system of first order equations of the form
(3). Thus for instance the RK(4) method (10) above becomes
6
w0 = α,
k1 = hf (xi , wi ),
1
h
k2 = hf (xi + , wi + k1 )
2
2
h
1
k3 = hf (xi + , wi + k2 )
2
2
k4 = hf (xi+1 , wi + k3 )
1
wi+1 = wi + (k1 + 2k2 + 2k3 + k4 )
6
for i = 0, 1, .., N − 1.
In the above we have taken the step size to be constant. But there is no
reason to do this and the step size can be adjusted dyanmically. This forms the
basis of the so called Kutta-Fehlberg methods.
2.3
m-step multi-step method
The methods discussed above are called one-step methods. Methods that use the
approximate values at more than one previous mesh point are called multi-step
methods. There are two distinct types worth mentioning.
These are of the form
wi+1 = cm−1 wi + cm−2 wi−1 + ... + c0 wi+1−m
+h[bm f (xi+1 , wi+1 ) + bm−1 f (xi , wi ) + ... + b0 f (xi+1−m , wi+1−m )] (11)
aIf bm = 0 (so there is no term with wi+1 on the right hand side of (11), the
method is explicit.
If bm 6= 0 we have an implicit method.
2.3.1
Adams-Bashforth 4th order method (explicit)
Here we have
w0 = α,
w 1 = α1 ,
w2 = α 2 ,
w3 = α 3 ,
where these values are obtained using other methods such as RK(4) for instance.
Then for i = 3, 4, ..., N − 1 we use
wi+1 = wi +
h
[55f (xi , wi ) − 59f (xi−1 , wi−1 ) + 37f (xi−2 , wi−2 ) − 9f (xi−3 , wi−3 )].
24
aa
7
2.3.2
Adams-Moulton 4th order method (implicit)
Here we have
w0 = α,
w 1 = α1 ,
w2 = α 2 ,
where these values are obtained using by other methods such as RK(4) for instance. Then
h
[9f (xi+1,wi +1 ) + 19f (xi , wi )
24
−5f (xi−1 , wi−1 ) + f (xi−2 , wi−2 )], for i = 2, 4, ..., N − 1. (12)
wi+1 = wi +
Notice the presence of the term containing wi+1 on the right hand side of (12).
The Adams-Bashforth, Adams-Moulton methods can be derived by integrating the differential equations as
dy
= f (x, y),
dx
y(xi+1 ) − y(xi ) =
Z
xi+1
xi
f (s, y) ds,
(13)
and replacing f (x, y) in the integral by an interpolating polynomial passing
through previous points.
The implicit methods are generally more stable, and have less roundoff errors
than the explicit methods. But here we need to bring in the ideas of stability of
a method.
2.3.3
Predictor-corrector methods
In practice the multi-step methods are rarely used as described above. Instead
they are frequently used as predictor-corrector methods. First we predict a value
using an explicit method, and correct it using an implicit method.
4th order predictor-corrector
h
[55f (xi , wi ) − 59f (xi−1 , wi−1 ) + 37f (xi−2 , wi−2 ) − 9f (xi−3 , wi−3 )],
24
(14)
h
(k)
(k−1)
wi+1 = wi + [9f (xi+1 , wi+1 +19f (xi , wi )−9f (xi−1 , wi−1 )+f (xi−2 , wi−2 )]. (15)
24
Here (14) is used to predict a value for wi+1 and (15) used to iteratively correct.
(0)
wi+1 = wi +
2.4
Improving accuracy with extrapolation
Suppose that we use a method with truncation error of O(hm ) to compute an
approximation wi . We can use Richardson extrapolation to get an approximation
(1)
with greater accuracy. First generate an approximation wi say with step size h
(2)
and an approximation wi with step size 2h. Then we can write
(1)
wi
= yi + Ehm + E1 hm+1 + . . . ,
8
and
(2)
wi
= yi + E(2h)m + E1 (2h)m+1 + . . . .
Then we can eliminate the E term to get
(1)
(2)
2 m wi − w i
Thus
= (2m − 1)yi + O(hm+1 ).
(1)
wi∗
(2)
2 m wi − w i
=
2m − 1
(1)
(2)
is a more accurate approximationto the solution than wi or wi .
For a 4th order Runge-Kutta method the above gives
(1)
wi∗ =
(2)
16wi − wi
.
15
9
3
Stability
We have introduced a number of different methods. How do we select which
one to use? In practice most methods should work reasonably well on standard
problems. However certain types of problems (stiff problems) can cause difficulty
and care needs to be exercised in the choice of the method. Typically boundary
layer type problems (say with a small parameter multiplying a highest derivative)
are examples of stiff problems. This is where one needs to be aware of the stability
properties of a particular method.
3.1
Consistent
A method is said to be consistent if the local truncation error tends to zero as
the step size → 0, i.e
lim max |τi (h)| = 0.
h→0
3.2
i
Convergence
A method is said to be convergent with respect to the equation it approximates
if
lim max |wi − y(xi )| = 0,
h→0
i
where y(x) is the exact solution and wi an approximation produced by the
method.
3.3
Stability
If we consider an m-step method
w0 = α0 , w1 = α1 , . . . , wm−1 = αm−1 ,
wi+1 = am−1 wi + am−2 wi−1 + ... + a0 wi+1−m
+h[F (xi , wi+1 , wi , ..., wi+1−m )]
(16)
then ignoring the F term the homogenous part is just a difference equation. The
stability is thus connected with the stability of this difference equation and hence
the roots of the characteristic polynomial
λm − am−1 λm−1 − ... − a1 λ − a0 = 0.
Why? Well consider (1) with f (x, y) = 0. This has the solution y(x) = α.
Thus the difference equation has to produce the same solution, ie wn = α. Now
consider
wi+1 = am−1 wi + am−2 wi−1 + ... + a0 wi+1−m .
(17)
10
If we look for solutions of the form wn = λn then this gives the characteristic
polynomial equation
λm − am−1 λm−1 − ... − a1 λ − a0 = 0.
(18)
Suppose λ1 , ...λm are distinct roots of (18). Then we can write
wn =
m
X
c i λm
i .
(19)
i=1
Since wn = α is a solution, the difference equation (17) gives
α − am−1 α − ... − a0 α = 0,
or
α(1 − am−1 − ... − a0 ) = 0.
This shows that λ = 1 is a root, and we may take c1 = α in (19). Thus (19) may
be written as
m
wn = α +
X
ci λni .
(20)
i=2
In the absence of round-off error all the ci in (20) would be zero. If |λi | ≤ 1 then
the error due to roundoff will not grow. The argument used above generalises
to multiple roots λi of the characteristic polynomial. Thus if the roots λi , (i =
1, .., m) of the characteristic polynomial satisfy |λi | ≤ 1, then the method is
stable.
Example
Consider the 4th order Adams-Bashforth method
wi+1 = wi +
h
[55f (xi , wi ) − 59f (xi−1 , wi−1 ) + 37f (xi−2 , wi−2 ) − 9f (xi−3 , wi−3 )].
24
This has the characteristic equation
λ4 − λ3 = 0,
giving the roots as λ = 1, 0, 0, 0. Thus this is stable according to our definition
of stability above.
It can be proven that if the difference method is consistent with
the differential equation, then the method is stable if and only if the
method is convergent.
Is it enough just to have stability as defined above? Consider the solution of
dy
= −30y,
dx
y(0) = 1/3.
The RK(4) method, although stable, has difficulty in computing the accurate
solution of this problem. This means that we need something more than just the
idea of stability defined above.
11
3.4
Absolute stability
Consider
dy
= ky, y(0) = α, k < 0.
(21)
dx
The exact solution of this is y(x) = αekx . If we take our one-step method and
apply it to this equation we obtain
wi+1 = Q(hk)wi .
Similarly for a multi-step of the type used earlier (11), when applied to the test
equation (21) gives rise to the difference equation
wi+1 = am−1 wi + am−2 wi−1 + ... + a0 wi+1−m
+h[bm kwi+1 + bm−1 kwi + ... + b0 kwi+1−m ].
(22)
Thus if we seek solutions of the form wi = z i this will give rise to the characteristic
polynomial equation
Q(z, hk) = 0,
where
Q(z, hk) = (1 − hkbm )z m − (am−1 + hkbm−1 )z m−1 − ... − (a0 + hkb0 ).
The region R of absolute stability for a one-step method is defined as the
region in the complex plane R= {hk ∈ C, |Q(hk)| < 1}, and for a multi-step
method R = {hk ∈ C, |βj | < 1}, where βj is a root of Q(z, hk) = 0.
A numerical method is A-stable if R contains the entire left half plane.
In practice the above conditions place a limit on the size of the step size which
we can use.
Consider the modified Euler method
w0 = α
k1 = hf (xi , wi ),
h
wi+1 = wi + [f (xi , wi ) + f (xi+1 , wi+1 )],
2
This is an A-stable method.
12
i = 1, 2, .., N − 1.
4
4.1
Boundary Value Problems
Shooting Methods
Consider the differential equation
d2 y
dy
+k
+ xy = 0,
2
dx
dx
y(0) = 0,
y(1) = 1.
(23)
This is an example of a boundary value problem because conditions have to
satisfied at both ends. If we write this as a system of first order equations we
have
Y1 = y,
dy
,
Y2 =
dx
dY1
= Y2 ,
dx
dY2
= −kY2 − xY1 .
dx
(24)
The boundary conditions give
Y1 (0) = 0,
Y1 (1) = 1.
We do not know the value of Y2 (0). If we knew this value then we could use our
standard integrator to obtain a solution. So how do we handle this situation.
Suppose we guess the value of Y2 (0) = g, say. Then we can integrate (24) with
the initial condition
!
!
Y1 (0)
0
Y(0) =
=
,
Y2 (0)
g
using any integrator eg RK(4). This will give us
Y(1) =
Y1 (1)
Y2 (1)
!
=
β1
β2
!
,
where, because we guessed the value g = Y2 (0), β1 will not necessarily satisfy the
required condition Y1 (1) = 1, in (23). So now we need to go through some kind
of iterative process to try and get the correct value of g such that the required
condition at x = 1 is satisfied.
To do this define
φ(g) = Y1 (1; g) − 1,
(25)
where the semi-colon notation is used to indicate an additional parametric dependence on g. We want to find the value of g such that φ(g) = 0. This gives rise
to the idea of a shooting method. We adjust the value of g to force φ(g) = 0. We
can also think of this as finding the root of the equation φ(g) = 0.
13
4.2
How do we adjust the value of g?
To solve φ(g) = 0 we can use any of the root finding methods we know, eg secant
method, Newton’s method, bisection, etc.
4.3
Newton’s Method, Secant Method
Suppose that we have a guess g̃ and we seek a correction dg such that φ(g̃ + dg) =
0. By Taylor expansion we have
φ(g̃ + dg) = φ(g̃) +
dφ
(g̃)dg + O(dg 2).
dg
This suggests that we take
dg = −
φ(g̃)
,
φ0 (g̃)
and hence a new value for g is g + dg. Thus this gives rise to a sequence of
iterations for n = 0, 1, ...,
φ(gn )
,
(26)
gn+1 = gn − 0
φ (gn )
where we start the iteration with a suitable guess g0 . Provided that g0 is close to to
the true value, Newton’s method converges very fast, ie quadratically. Difficulties
with convergence arise when φ0 (g) is close to zero.
Now in (26) we are required to find φ0 (gn ). How can we do this? One way is
to estimate φ0 (gn ) by
φ(gn ) − φ(gn−1 )
.
φ0 (gn ) =
gn − gn−1
This gives
gn+1 = gn −
φ(gn )(gn − gn−1 )
,
φ(gn ) − φ(gn−1 )
(27)
which is known as the secant method. We need two starting values for g and
then we can use (27) to generate the remaining values.
Consider (24) again
dY1
= Y2 ,
dx
dY2
= −kY2 − xY1 ,
dx
with Y(0) = (0, g)T . Now from (25)
φ0 (g) =
∂Y1
(1; g),
∂g
14
so this suggests differentiating the original system of equations and boundary
conditions with respect to g to get
d
dx
∂Y
∂g
!
∂Y2
∂g
=
1
2
− x ∂Y
−k ∂Y
∂g
∂g
!
,
!
∂Y
(x = 0) =
∂g
0
1
!
.
(28)
The system (28) defines another initial value problem with given initial conditions.
Thus the procedure is as follows. First integrate (24) with initial conditions
Y(0) = (0, g)T and with a suitable initial guess g. Then, either simultaneously
at each step, or after a step, solve (28). After integrating over all the steps we
(x = 1) from (28). Note
will have at x = 1 the values Y(x = 1) from (24) and ∂Y
∂g
that from (25)
∂Y1
(1; g).
φ0 (g) =
∂g
1
From the solution of (28) we can extract ∂Y
(x = 1) and hence substitute into (26)
∂g
to update g. This forms the basis of Newton’s method combined with shooting,
to solve boundary value problems.
It is clear how one can solve (28) after solving (24) at each step. This can also
be done simultaneously by working with an augmented system where we define
Y1 = y,
Y2 =
dy
,
dx
Y3 =
∂y
∂Y1
=
,
∂g
∂g
Y4 =
∂Y2
,
∂g
and then

d 


dx 
4.4
Y1
Y2
Y3
Y4









=
Y2
−kY2 − xY1
Y4
−kY4 − xY3




,




Y(0) = 
Y1 (0)
Y2 (0)
Y3 (0)
Y4 (0)









=
0
g
0
1



.

(29)
Multiple end conditions
What we have discussed so far works fine if we just have one condition to satisfy.
The same procedure works also for cases where we have two or more conditions
to satisfy. For example with
d4 y
dy
= y 3 − ( )2 ,
4
dx
dx
y(0) = 1,
dy
(0) = 0,
dx
y(1) = 2,
dy
(1) = 1,
dx
we would need two starting values at x = 0 and then we will have two conditions
to satisfy at x = 1. The corrections can be calculated in a similar way. Thus in
the example above, suppose we define
Y1 = y, Y2 = y 0 , Y3 = y 00 , y4 = y 000 ,
15
then we will need to use guesses for Y3 (0) = e, say, and Y4 (0) = g. The conditions
we need to satisfy can be written as
φ1 (e, g) = φ2 (e, g) = 0,
where
φ1 (e, g) = Y1 (1; e; g) − 2,
φ2 (e, g) = Y2 (1; e; g) − 1.
To obtain the corrections to guessed values ẽ, g̃ we have
∂φ1
∂φ1
(ẽ, g̃) + dg
(ẽ, g̃) + O(de2 , dg 2 ),
∂e
∂g
∂φ2
∂φ2
φ2 (ẽ + de, g̃ + dg) = 0 = φ2 (ẽ, g̃) + de
(ẽ, g̃) + dg
(ẽ, g̃) + O(de2 , dg 2 ).
∂e
∂g
φ1 (ẽ + de, g̃ + dg) = 0 = φ1 (ẽ, g̃) + de
This can be solved to obtain the corrections (de, dg).
If we generalise to a multidimensional case where we have a vector of guesses
g̃ we can find the corrections dg as
dg = −J−1 (g̃)φ(g̃),
∂φi
and φ is the vector of conditions.
where J is the Jacobian ∂g
k
There are of course many variations on the above theme and different strategies when dealing with linear equations. For stiff systems one can also integrate
from the two ends and match in the middle. Many of these techniques are discussed extensively in the book by Keller (1976).
16
4.5
Solution of boundary value problems using finite-differences
Boundary value problems can also be tackled directly using finite-differences or
some other technique such as spectral approximation. We will look at one specific
example to illustrate the finite-difference technique.
Consider
d2 y
1
dy
= (32 + 2x3 − y ),
2
dx
8
dx
1 ≤ x ≤ 3,
y(1) = 17,
y(3) =
43
.
3
(30)
The exact solution of (30) is y(x) = x2 + (16/x).
Sfrag replacements
Let us first define a uniform grid (x0 , x1 , ..., xN ) with N + 1 points and grid
spacing h = (xN − x0 )/N, so that xj = x0 + jh, for (j = 0, 1, .., N ).
h
a = x0
x2
x3
xi
xi+1
xN = b
Let us approximate y by wi at each of the nodes x = xi . The derivatives of y
in (30) are approximated in finite-difference form as
dy
dx
d2 y
dx2
!
!
wi+1 − wi−1
+ O(h2 ),
2h
(31)
wi+1 − 2wi + wi−1
+ O(h2 ).
h2
(32)
=
x=xi
=
x=xi
These can be derived by making use of a Taylor expansion about the point x = xi .
Thus for example
dy
h2 d 2 y
h3 d 3 y
h4 d 4 y
(xi ) +
(x
)
+
(x
)
+
(xi ) + O(h5 ),
i
i
dx
2 dx2
6 dx3
24 dx4
(33)
2 2
3 3
4 4
h dy
h dy
h dy
dy
(x
)
−
(x
)
+
(xi ) + O(h5 ).
y(xi−1 ) = y(xi ) − h (xi ) +
i
i
dx
2 dx2
6 dx3
24 dx4
(34)
y(xi+1 ) = y(xi ) + h
By adding and subtracting (33),(34) and replacing y(xi ) by wi we obtain (31),(32).
Next replace y and its derivatives in (30) by the above approximations to get
wi+1 − wi−1
wi+1 − 2wi + wi−1
x3i
=
4
+
−
w
,
i
h2
4
16h
for (i = 1, 2, ...N − 1)
17
(35)
and
43
.
(36)
3
The system of equations (35),(36) is a set of nonlinear difference equations. We
have N + 1 equations for N + 1 unknowns w0 , ..., wN . At this stage one has
recourse to a number of different techniques for handling the nonlinearity. One
(k)
is to set up an iteration scheme replacing wi by wi say, and starting with a
(0)
suitable guess for wi . The nonlinear term in (35) can now be tackled in many
different ways. Thus for example we can replace it by
w0 = 17,
(k−1)
wi
or

wN =
(k)
(k)

w − wi−1 
 i+1
,
16h

(k−1)
(k−1)

w
− wi−1 
(k)
wi  i+1
,
16h
(k)
and we are then left with a linear system of equations to find the wi and the
iterations can be continued until a suitable convergence condition is satisfied.
What is a suitable convergence criterion? One condition is that the nonlinear
difference equations (35) need to be satisfied to sufficient accuracy.
4.5.1
Newton linearization
Where practicable Newton’s method offers the best means for handling nonlinear
equations because of its superior convergence properties. To use this technique
for nonlinear difference equations of the form (35), suppose that we have a guess
for the solutions wi = Wi . We seek corrections δwi such that the wi = Wi + δwi
satisfies the system (35). Substituting wi = Wi + δwi into (35) and linearizing,
ie ignoring terms of O(δwi2 ), gives
!
δwi+1 − 2δwi + δwi−1
Wi+1 − Wi−1
δwi+1 − δwi−1
− Wi
,
= Fi − δwi
2
h
16h
16h
for (i = 1, 2, ...N − 1) (37)
and
δw0 = F0 ,
δwN = FN .
(38)
where
Wi+1 − Wi−1
x3
Wi+1 − 2Wi + Wi−1
,
Fi = 4+ i −Wi
−
4
16h
h2
and
F0 = 17 − W0 ,
FN =
18
43
− WN .
3
for i = 1, ..., N −1,
The system (37),(38) is now linear and can be solved to obtain the corrections
δwi and hence we can update the Wi by Wi + δwi . This is continued until the δwi
are small enough and the difference equations are satisfied to sufficient accuracy.
Instead of solving for the corrections δwi one can also solve for the total
quantities wi by replacing δwi in (37), (38) by δwi = wi − Wi , and solving for the
wi .
The choice of the iteration scheme is dependent on a number of factors including ease of implementation, convergence properties and so on. The techniques
described above lead to the solution of a tridiagonal systems of linear equations
of the form
αi wi−1 + βi wi + γi wi+1 = δi , i = 0, 1, .., N,
(39)
where the αi , βi , γi are coefficients obtainable from the linear system and α0 =
γN = 0. For instance from (37) we have
αi =
1
Wi
−
,
2
h
16h
βi = −
γi =
Wi
1
+
,
2
h
16h
δi = F i ,
2
Wi+1 − Wi−1
+
,
2
h
16h
i = 1, 2, .., N − 1,
and
β0 = 1, γ0 = 0, δ0 = F0 ,
αN = 0, βN = 1, δN = FN .
This can be solved using any standard library routine, or Thomas’s algorithm.
4.6
Thomas’s tridiagonal algorithm
This version of a tridiagonal solver is based on Gaussian elimination. First we
create zeros below the diagonal and then once we have a triangular matrix, we
solve for the wi using back substitution. Thus the algorithm takes the form
γj−1 αj
j = 2, 3, ..., N,
βj−1
δj−1 αj
= δj −
,
j = 2, 3, ..., N,
βj−1
(δj − γj wj+1 )
δN
, wj =
,
j = N − 1, ..., 1.
=
βN
βj
βj = β j −
δj
wN
19
5
Numerical solution of pde’s
In this part of the course we will be concerned with the numerical solution of partial differential equations (pde’s), using finite-difference methods. This involves
covering the domain with a mesh of points and calculating the approximate solution of the pde at these mesh points. The techniques that we will use will depend
on the type of pde that we are dealing with, ie whether it is elliptic, parabolic,
or hyperbolic, although there are common features. We will need to learn how to
construct approximations to the given equation, how to solve the equation using
iterative methods, investigate stability and so on.
5.1
Classification
Partial differential equations can be classified as being of type elliptic, parabolic or
hyperbolic. In some cases equations can be of mixed type. There are systematic
methods for classifying an equation but for our purposes it is enough to summarize
the main results. Consider the second order pde
A
∂2φ
∂2φ
∂2φ
∂φ
∂φ
+
B
+
C
+D
+E
+ F φ + G = 0,
2
2
∂x
∂x∂y
∂y
∂x
∂y
(40)
where, in general, A, B, C, D, E, F, and G are functions of the independent variables x and y and of the dependent variables φ. The equation (40) is said to
be
• elliptic if B 2 − 4AC < 0,
• parabolic if B 2 − 4AC = 0, or
• hyperbolic if B 2 − 4AC > 0.
An example of an elliptic equation is Poisson’s equation
∂2φ ∂2φ
+ 2 = f (x, y).
∂x2
∂y
The heat equation
∂2φ
∂φ
=k 2
∂t
∂x
is of parabolic type, and the wave equation
∂2φ ∂2φ
− 2 =0
∂x2
∂y
is a hyperbolic pde. An example of a mixed type equation is the transonic small
disturbance equation given by
(K −
∂φ ∂ 2 φ ∂ 2 φ
)
+ 2 = 0.
∂x ∂x2
∂y
20
A more formal approach to classification is to consider a system of first order
partial differential equations for the unknowns U = (u1 , u2 , ..., un )T and independent variables x = (x1 , x2 , ..., xm )T . Suppose that the equations can be written
in quasi-linear form
m
X
∂U
=Q
(41)
Ak
∂xk
k=1
where the Ak are (n × n) matrices and Q is an (n × 1) column vector, and both
can depend on xk and U but not on the derivatives of U. If we seek plane wave
solutions of the homogeneous part of (41) in the form
U = Uo eix.s ,
where s = (s1 , s2 , .., sm )T , then (41) leads to the system of equations
i
"
m
X
#
Ak sk U = 0.
k=1
(42)
This will have a non-trivial solution only if the characteristic equation
det |
m
X
k=1
Ak sk | = 0,
holds. This will have at most n solutions and n vectors s, which can be regarded
as normals to the characteristic surfaces (which are the surfaces of constant phase
of the wave).
The system is hyperbolic if n real characteristics exist. If all the charcteristics
are complex, the system is elliptic. If some are real and some complex, the system
is of mixed type. If the system (42) is of rank less than n, then we have a parabolic
system.
Example
Consider the steady two-dimensional Euler equations for incompressible flow
∂u ∂v
+
= 0,
∂x ∂y
∂u
∂u
∂p
u
+v
=− ,
∂x
∂y
∂x
∂v
∂p
∂v
+v
=− .
u
∂x
∂y
∂y
Introducing the vector U = (u, v, p)T and x = (x, y)T we can express this in
matrix vector form (41) with Q = 0 and



1 0 0


A1 =  u 0 1  ,
0 u 0

0 1 0


A2 =  v 0 0  .
0 v 1
21
Introducing λ = s1 /s2 the characteristic equation gives


λ
1
0

0
λ 
det  uλ + v
 = 0,
0
uλ + v 1
which leads to λ = −v/u or λ = ±i. Thus the Euler equations are hybrid.
5.2
Consistency, convergence and Lax equivalence theorem
These ideas were introduced earlier for ordinary differntial equations but apply
equally to pdes.
Consistent A discrete approximation to a partial differential equation is said
to be consistent if in the limit of the step-size(s) going to zero, the original pde
system is recovered, ie the truncation error approaches zero.
Stability If we define the error to be the difference between the computed
solutions and the exact solution of the discrete approximation, then the scheme
is stable if the error remains uniformly bounded for successive iterations.
Convergence A scheme is stable if the solution of the discrete equations
approaches the solution of the pde in the limit that the step-sizes approach zero.
Lax’s Equivalence Theorem For a well posed initial-value problem and
a consistent discretization, stability is the necessary and sufficient condition for
convergence.
5.3
Difference formulae
Let us first consider the various approximations to derivatives that we can construct. The usual tool for this is the Taylor expansion. Suppose that we have a
grid of points with equal mesh spacing ∆x in the x− direction and equal spacing
∆y in the y− direction. Thus we can define points xi , yj by
xi = x0 + i∆x ,
yj = y0 + j∆y .
Suppose that we are trying to approximate a derivative of a function φ(x, y) at
the points xi , yj . Denote the approximate value of φ(x, y) at the point xi , yj by
wi,j say.
5.3.1
Central Differences
The first and second derivatives in x or y may be approximated as before by
∂2φ
∂x2
!
=
ij
wi+1,j − 2wi,j + wi−1,j
+ O((∆x )2 ),
2
(∆x )
22
(43)
∂2φ
∂y 2
!
=
ij
wi,j+1 − 2wi,j + wi,j−1
+ O((∆y )2 ),
2
(∆y )
∂φ
∂x
!
∂φ
∂y
=
wi+1,j − wi−1,j
+ O((∆x )2 ),
2∆x
(45)
=
wi,j+1 − wi,j−1
+ O((∆y )2 ).
2∆y
(46)
ij
!
(44)
ij
∂2φ
Note that replacing the derivative
by the approximation in (43) gives
∂x2 ij
rise to a truncation error O((∆x )2 ). The approximations listed above are centered
at the points (xi , yj ), and are called central-difference approximations.
!
•
i − 1, j
×
i, j
•
i + 1, j
Consider the approximation
∂φ
∂x
!
wi+1,j − wi,j
.
∆x
=
ij
By Taylor expansion we see that this gives rise to a truncation error of O(∆x ).
In addition this approximation is centered at the point xi+ 1 ,j .
2
•
i, j
5.3.2
×
•
i + 1, j
One-sided approximations
We can also construct one-sided approximations to derivatives. Thus for example
a second-order forward approximation to ∂φ
at the point (xi , yj ) is given by
∂x
∂φ
∂x
!
=
ij
−3wi,j + 4wi+1,j − wi+2,j
.
2∆x
Tables 1 and 2 list some of the more commonly used central and one-sided
approximations. In these tables the numbers denote the weights at the nodes,
and the whole expression is multiplied by (∆1x )j where j denote the order of the
derivative, ie j = 1 for a first derivative and j = 2 for a second derivative.
5.3.3
Mixed derivatives
The techniques for finding suitable discrete approximations for mixed derivatives
is exactly the same, except this time we would make use a multidimensional
23
Table 1: Weights for central differences
Order of
Accuracy
1st derivative
(∆x )2
(∆x )4
Node Points
i−2 i−1 i
1
12
− 21
0
1
2
− 23
0
2
3
1
-2
1
4
3
− 25
4
3
2nd derivative
(∆x )2
(∆x )4
i+1 i+2
1
− 12
1
− 12
1
− 12
Table 2: Weights for one-sided differences
Order of
Accuracy
1st derivative
(∆x )
Node Points
i
i+1 i+2
i+3 i+4
-1
1
(∆x )2
− 23
2
− 12
(∆x )3
− 11
6
3
− 32
1
3
(∆x )4
25
− 12
4
-3
4
3
2nd derivative
(∆x )
1
-2
1
(∆x )2
2
-5
4
-1
(∆x )3
35
12
− 26
3
19
2
− 14
3
24
− 14
11
12
Taylor expansion. Thus for example second order approximations to ∂ 2 φ/∂x∂y
at the point i, j are given
wi+1,j+1 − wi−1,j+1 + wi−1,j−1 − wi+1,j−1
∂2φ
=
+ O((∆x )2 , (∆y )2 ),
∂x∂y
4∆x ∆y
or
wi+1,j+1 − wi+1,j − wi,j+1 + wi−1,j−1 − wi−1,j − wi,j−1 + 2wi,j
∂2φ
=
+O((∆x )2 , (∆y )2 ).
∂x∂y
2∆x ∆y
5.4
Solution of elliptic pde’s
A prototype elliptic pde is Poisson’s equation given by
∂2φ ∂2φ
+ 2 = f (x, y),
∂x2
∂y
(47)
where f (x, y) is a known/given function. The equation (47) has to be solved in a
domain D, say, with boundary conditions given on the boundary δD of D. These
can be of three types:
• Dirichlet
φ = g(x, y) on δD.
• Neumann
∂φ
= g(x, y) on δD.
∂n
∂φ
• Robin/Mixed B(φ,
) = 0 on δD. Robin boundary conditions involve a
∂n
linear combination of φ and its normal derivative on the boundary. Mixed
boundary conditions involve different conditions for one part of the boundary, and another type for other parts of the boundary.
In the above g(x, y) is a given function, and derivatives with respect to n denote
the normal derivative, ie the derivative in the direction n which is normal to the
boundary.
Let us consider a model problem with
∂2φ ∂2φ
+ 2 = f (x, y), 0 < x, y < 1
∂x2
∂y
φ = 0 on δD.
(48)
(49)
Here the domain D is the square region 0 < x < 1 and 0 < y < 1. Construct a
finite difference mesh with points (xi , yj ), say where
xi = i∆x ,
i = 0, 1, ..., N,
yj = j∆y ,
j = 0, 1, ..., M,
where ∆x = 1/N, and ∆y = 1/M are the step sizes in the x and y directions.
25
Next replace the derivatives in (48) by the approximations from (43),(44) to
get
wi+1,j − 2wi,j + wi−1,j wi,j+1 − 2wi,j + wi,j−1
+
= fi,j ,
(∆x )2
(∆y )2
1 ≤ i ≤ N − 1, 1 ≤ j ≤ M − 1
(50)
and
wi,j = 0,
if
i = 1, N,
1 ≤ j ≤ M,
(51)
wi,j = 0,
if
j = 1, M,
1 ≤ i ≤ N.
(52)
Thus we have (N − 1) × (M − 1) unknown values wi,j to find at the interior points
of the domain. If we write wi = (wi,1 , wi,2 , ..., wi,M −1 )T , fi = (fi,1 , fi,2 , ..., fi,M −1 )T ,
we can write the above system of equations (50) in matrix form as
B I

 I B I


I B I







I B









w1
w2
w3
wN −1











=



2
(∆x ) 




f1
f2
f3
fN −1





.




In the above I is the (M − 1) × (M − 1) identity matrix and B is a similar sized
matrix given by

B=













b c
c b c
0 c b c
.
.
.
c b






,





with b = −2(
1
1
+
),
(∆x )2 +(∆y )2
c = 1.
What we observe is that the matrix is very sparse and can be very large even
with a modest number of grid points. For instance with N = M = 101 the linear
system is of size 104 × 104 and we have 104 unknowns to find.
To solve the system of equations arising from the above discretization, one
has recourse to a number of methods. Direct methods are not feasible because of
the size of the matrices involved and in addition they are expensive and require a
lot of storage, unless one exploits the sparsity pattern of the matrices. Iterative
methods offer an alternative and are easy to use, but depending on the method,
convergence can be slow with an increasing number of mesh points.
26
5.5
Iterative methods for the solution of linear systems
5.5.1
Jacobi method
Consider (50) which we can rewrite as
wi,j =
1
2
2
w
+
w
+
β
(w
+
w
)
−
(∆
)
f
,
i+1,j
i−1,j
i,j+1
i,j−1
x
i,j
2(1 + β 2 )
(53)
with β = ∆x /∆y . This suggests the iterative scheme
new
wi,j
=
1
old
old
2
old
old
2
w
+
w
+
β
(w
+
w
i−1,j
i,j+1
i,j−1 − (∆x ) fi,j ).
2(1 + β 2 ) i+1,j
(54)
old
We start with a guess for wi,j
and compute a new value from (54) and continue
iterating until suitable convergence criteria are satisfied. What are suitable convergence criteria? Ideally we wish to stop when the corrections are small, or when
the equations (53) are satisfied to sufficient accuracy.
Suppose we write the linear system as
Av = f
where v is the exact solution of the linear system. Then if w is an approximate
solution, the error e is defined by e = v − w. But this is not much use unless we
know the exact solution. On the other hand we can write
Ae = A(v − w) = f − Aw.
Then the residual is defined by r = f − Aw which can be computed. For the
Jacobi scheme the residual is given by
ri,j = (∆x )2 fi,j + 2(1 + β 2 )wi,j − (wi+1,j + wi−1,j + β 2 (wi,j+1 + wi,j−1 )).
Therefore a suitable stopping condition might be
max |ri,j | < 1 ,
i,j
5.5.2
or
sX
2
ri,j
< 2 .
i,j
Gauss-Siedel iteration
new
old
The Jacobi scheme involves two levels of storage, wi,j
and wi,j
which can be
wasteful. It is more economical to overwrite values as and when they are computed and use the new values when ready. This gives rise to the Gauss-Seidel
scheme, which, if we sweep with i increasing followed by j, is given by
new
wi,j
=
1
old
new
2
old
new
2
w
+
w
+
β
(w
+
w
i−1,j
i,j+1
i,j−1 − (∆x ) fi,j ),
2(1 + β 2 ) i+1,j
where the new values overwrite existing values.
27
(55)
5.5.3
Relaxation and the SOR method
Instead of updating the new values as indicated above, it is better to use relaxation. Here we compute
old
∗
wi,j = (1 − ω)wi,j
+ ωwi,j
,
∗
where ω is called the relaxation factor, and wi,j
denotes the value as computed by
the Jacobi, or Gauss-Seidel scheme. ω = 1 reduces to the Jacobi or Gauss-Seidel
scheme. The Gauss-Seidel scheme with ω 6= 1 is called the method of successive
overrelaxation or SOR scheme.
5.5.4
Line relaxation
The Jacobi, Gauss-Seidel and SOR schemes are called point relaxation methods.
On the other hand we may compute a whole line of new values at a time, leading
to the line-relaxation methods. For instance suppose we write the equations as
new
new
new
old
wi+1,j
− 2(1 + β 2 )wi,j
+ wi−1,j
= −β 2 (wi,j+1 + wi,j−1
)) + (∆x )2 fi,j ,
(56)
then starting from j = 1 we may compute the values wi,j , for i = 1, ..., N − 1 in
one go. The left hand side of (56) can be seen to lead to a tridiagonal system
of equations and we may use a tridiagonal solver to compute the solution, and
incrementing j until all the points have been computed. The above scheme can
be combined with relaxation to give us a line-SOR method. In general the more
implicit the scheme the better the convergence properties. A variation on the
above scheme is to alternate the direction of sweep, by doing i lines first and on
the next iteration doing the j lines first. This leads to the ADI, or alternating
direction implicit, scheme.
28
5.6
Convergence properties of basic iteration schemes
Consider the linear system
Ax = b
(57)
where A = (ai,j ) is an n × n matrix, and x, b are n × 1 column vectors. Suppose
we write the matrix A in the form A = D−L−U where D, L, U are the diagonal
matrix, lower and upper triangular parts of A, ie

D=













a1,1 0
0 a2,2 0
0
0 a3,3 0
.
.
.
0 an,n

−U =


















,






−L =













0
0
a2,1 0 0
a3,1 a3,2 0 0
.
.
.
an,1 an,2
0 a1,2 a1,3 . . . a1,n
0 0 a2,3 . . . a2,n
0 0
0 a3,4 . . a3,n
.
.
0 an−1,n
0 0
0
Then (57) can be written as
an,n−1 0







.





Dx = (L + U)x + b.
The Jacobi iteration is then defined as
x(k+1) = D−1 (L + U)x(k) + D−1 b.
(58)
The Gauss-Seidel iteration is defined by
(D − L)x(k+1) = Ux(k) + b,
(59)
x(k+1) = (D − L)−1 Ux(k) + (D − L)−1 b.
(60)
or
In general an iteration scheme for (57) may be written as
x(k+1) = Px(k) + Pb
(61)
where P is called the iteration matrix. For the Jacobi scheme scheme we have
P = PJ = D−1 (L + U)
29






,





and for the Gauss-Seidel scheme
P = PG = (D − L)−1 U.
The exact solution to the linear system (57) satisfies
x = Px + Pb.
(62)
Hence if we define the error at the kth iteration by e(k) = x − x(k) , then by
subtracting (61) from (62) we see that the error satisfies the equation
e(k+1) = Pe(k) = P2 e(k−1) = Pk+1 e(0) .
(63)
In order that the error diminishes as k → ∞ we must have
||Pk || → 0 as k → ∞,
(64)
Since
||Pk || = ||P||k
we see that (64) is true provided ||P|| < 1. From linear algebra it can be proven
that the condition that ||P|| < 1 is equivalent to the requirement that
ρ(P) = max |λi | < 1
i
where λi are the eigenvalues of the matrix P. Note that ρ(P) is called the spectral
radius of P. Thus for the iteration to converge we require that the spectral matrix
of the iteration matrix to be less than unity. One further result which is useful
is that for large k
||e(k+1) || = ρ||e(k) ||.
Suppose we want to estimate how many iterations it takes to reduce the initial
error by a factor . Then we see that we need q iterations where q is the smallest
value for which
ρq < giving
q ≥ qd =
ln .
ln ρ
The quantity − ln ρ is called the asymptotic rate of convergence of the iteration.
It gives a measure of how much the error decreases at each iteration. Thus
iteration matrices where the spectral radius is close to 1 will converge slowly.
For the model problem it can be shown that for Jacobi iteration
π
π
1
cos + cos
,
ρ = ρ(PJ ) =
2
N
M
30
and for Gauss-Seidel
ρ = ρ(PG ) = [ρ(PJ )]2 .
If we take N = M and N >> 1 then for Jacobi iteration we have
qd =
2N 2
ln =
−
ln .
2
π
π2
ln(1 − 2N
2)
Gauss-Seidel converges twice as fast as Jacobi, and requires less storage.
For point SOR the spectral radius depends on the relaxation factor ω, but for
the model problem with optimum values and N = M it can be shown that
1 − sin Nπ
ρ=
1 + sin Nπ
giving
N
ln .
2π
Thus SOR with the optimum requires an order of magnitude less number of
iterations to converge as compared to Gauss-Seidel. Finally similar results can
also be derived for line SOR, and it can be shown that using optimum values
qd = −
ρ=
π
1 − sin 2N
π
1 + sin 2N
!2
,
31
N
qd = − √ ln .
2 2π
6
Solution of differential equations using Chebychev collocation
In this section we will briefly look at the solution of equation using Chebychev
collocation. Spectral methods offer a completely different means for obtaining a
solution of a differential equation (ode or pde) and they have a number of major
advantages over finite difference and other similar methods. One is the concept
of spectral accuracy. With finite difference methods increasing the number of
points for a second order method means truncation error reducing by a factor 4.
On the other hand with spectral methods the accuracy of the approximation (for
smooth functions) increases exponentially with increased number of points. This
is referred to as spectral accuracy. There are many different types of spectral
methods (for example the tau method, Galerkin method, Fourier approximation)
and a good place to start studying the basic ideas and theory is in the book
by Canuto et al. (1988). Here we will briefly look at one namely, Chebychev
collocation.
With any spectral method the idea is to represent the function as an expansion
in terms of some basis functions, usually orthogonal polynomials or in the case of
Fourier approximation, trigonometric functions or complex exponentials. With
Chebychev collocation a function is represented in terms of expansions using
Chebychev polynomials.
6.1
Basic properties of Chebychev polynomials
A Chebychev polynomial of degree n is defined by
Tn (x) = cos(n cos−1 x).
(65)
The Chebychev polynomials are eigenfunctions of the Sturm-Liouville equation
d √
dTk (x)
k2
1 − x2
+√
Tk (x) = 0.
dx
dx
1 − x2
!
The following properties are easy to prove
Tk+1 (x) = 2xTk (x) − Tk−1 (x),
with T0 (x) = 1 and T1 (x) = x. Also
|Tk (x)| ≤ 1,
Tk (±1) = (±1)k ,
2Tk (x) =
−1 ≤ x ≤ 1,
Tk0 (±1) = (±1)k k 2 ,
1 dTk−1 (x)
1 dTk+1 (x)
−
,
(k + 1)
dx
(k − 1)
dx
32
k ≥ 1.
Suppose we expand a (smooth) function u(x) in terms of Chebychev polynomials, then
u(x) =
∞
X
2
ûk =
πck
ûk Tk (x),
k=0
where
ck =
(
Z
1
−1
u(x)Tk (x)
√
dx,
1 − x2
2
k = 0, N
.
1 1≤k ≤N −1
If we define ũ(θ) = u(cos θ) then
ũ(θ) =
∞
X
ûk cos(kθ),
k=0
so that the Chebychev series for u corresponds to a cosine series for ũ. This
implies that if u is infinitely differentiable then the Chebychev coefficients of the
expansion will decay faster than algebraically.
In a collocation method we may represent a smooth function in terms of its
values at set of discrete points. Derivatives of the function are approximated by
analytic derivatives of the interpolating polynomial. It is common to work with
the Gauss-Lobatto points defined by
xj = cos(
jπ
) j = 0, 1, ..., N.
N
Suppose the given function is approximated at these points and we represent the
approxmiate values of the function u(x) at these points x = xj by uj . Here we
are assuming that we are working with N + 1 points in the interval −1 ≤ x ≤ 1.
In a given differential equation we need to approximate the derivatives at these
node points. The theory says that we can write
du
dx
!
=
j
N
X
Dj,k uk ,
k=0
where Dj,k are the elements of the Chebychev collocation differentiation matrix
D say. The elements of the matrix D are given by
Dp,j =



























cp (−1)p+j
cj (xp −xj )
p 6= j
x
j
− 2(1−x
1≤p=j ≤N −1
2)
j
2N 2 +1
6
2
− 2N 6 +1
p=j=1
p=j=N
33
Thus in matrix form








dw0
dx
dw1
dx
.
.
dwN
dx








=




D



w0
w1
.
.
wN




,



and
34








d2 w 0
dx2
d2 w 1
dx2
.
.
d2 w N
dx2








=




D2 



w0
w1
.
.
wN








7
Parabolic Equations
In this section we will look at the solution of parabolic partial differential equations. The techniques introduced earlier apply equally to parabolic pdes.
One of the simplest parabolic pde is the diffusion equation which in one space
dimensions is
∂2u
∂u
= κ 2.
(66)
∂t
∂x
For two or more space dimensions we have
∂u
= κ∇2 u.
∂t
In the above κ is some given constant.
Another familiar set of parabolic pdes is the boundary layer equations
ux + vy = 0,
ut + uux + vuy = −px + uyy ,
0 = −py .
With a parabolic pde we expect, in addition to boundary conditions, an initial
condition at say, t = 0.
Let us consider (66) in the region a ≤ x ≤ b. We will take a uniform mesh in
x with xj = a + j∆x , j = 0, 1, .., N and ∆x = (b − a)/N . For the differencing
in time we assume a constant step size ∆t so that t = tk = k∆t .
7.1
First order central difference approximation
We may approximate (66) by
k
k
wjk+1 − wjk
wj+1
− 2wjk + wj−1
=κ
.
∆t
∆2x
"
#
(67)
Here wjk denotes an approximation to the exact solution u(x, t) of the pde at
x = xj , t = tk . The scheme given in (67) is first order in time O(∆t ) and
second order in space O(∆x )2 . Let us assume that we are given a suitable initial
condition, and boundary conditions of the form
u(a, t) = f (t),
u(b, t) = g(t).
The scheme (67) is explicit because the unknowns at level k + 1 can be computed
directly. Notice that there is a time lag before the effect of the boundary data is
felt on the solution. As we will see later this scheme is conditionally stable for
κ∆t
β ≤ 1/2, where β = 2 . Note that β is sometimes called the Peclet or diffusion
∆x
number.
35
7.2
Fully implicit, first order
A better approximation is one which makes use of an implicit scheme. Then
instead of (67) we have
k+1
wjk+1 − wjk
w k+1 − 2wjk+1 + wj−1
.
= κ j+1
∆t
∆2x
#
"
(68)
The unknowns at level k + 1 are coupled together and we have a set of implicit
equations to solve. If we rearrange (68) then
k+1
k+1
βwj−1
− (1 + 2β)wjk+1 + βwj−1
= −wjk ,
1 ≤ j ≤ N − 1,
(69)
In addition approximation of the boundary conditions gives
w0k+1 = f (tk+1 ),
k+1
wN
= g(tk+1 ).
(70)
The discrete equations (69),(70) are of tridiagonal form and thus easily solved.
The scheme is unconditionally stable . The accuracy of the above fully implicit
scheme is only first order in time. We can try and improve on this with a second
order scheme.
7.3
Richardson method
Consider
k
k
wj+1
− 2wjk + wj−1
wjk+1 − wjk−1
.
=κ
2∆t
∆2x
#
"
(71)
This uses three time levels and has accuracy O(∆2t , ∆2x ). The scheme was devised
by a meteorologist and is unconditionally unstable!
7.4
Du-Fort Frankel
This uses the approximation
k
k
wj+1
− wjk+1 − wjk−1 + wj−1
wjk+1 − wjk−1
=κ
.
2∆t
∆2x
"
#
(72)
∆t 2
) ), and is an explicit scheme. The
∆x
scheme is unconditionally stable, but is inconsistent if ∆t → 0, ∆x → 0 but with
∆t /∆x remaining fixed.
This has truncation error O(∆2t , ∆2x , (
36
7.5
Crank-Nicolson
A popular scheme is the Crank-Nicolson scheme given by
k+1
k+1
k
k
wj+1
− 2wjk + wj−1
wjk+1 − wjk
− 2wjk+1 + wj−1
κ wj+1
=
+
.
∆t
2
∆2x
∆2x
"
#
(73)
This is second order accurate O(∆2t , ∆2x ) and is unconditionally stable. (Taking
very large time steps can however cause problems). As can be seen it is also an
implicit scheme.
7.6
Multi-space dimensions
The schemes outlined above are easily extended to multi-dimensions. Thus in
two space dimensions a first order explict approximation to
∂u
= κ∇2 u,
∂t
is
k+1
k
k
k
k
k
k
wi,j+1
− 2wi,j
+ wi,j−1
wk
− 2wi,j
+ wi−1,j
wi,j
− wi,j
+
.
= κ i+1,j
∆t
∆2x
∆2y
#
"
(74)
This is first order in ∆t and second order in space. It is conditionally stable for
κ∆t
κ∆t
1
+
≤
.
(∆x )2 (∆y )2
2
If we use a fully implicit scheme we would obtain
k+1
k+1
k+1
k+1
k+1
k+1
k
− wi,j
wi,j
w k+1 − 2wi,j
+ wi−1,j
wi,j+1
− 2wi,j
+ wi,j−1
= κ i+1,j
+
.
∆t
∆2x
∆2y
"
#
(75)
This leads to an implicit system of equations of the form
k+1
k+1
k+1
k+1
k+1
k
αwi+1,j
+ αwi−1,j
− (2α + 2β + 1)wi,j
+ βwi,j−1
+ βwi,j+1
= −wi,j
,
where α = κ∆t /∆2x , β = κ∆t /∆2y . The form of the discrete equations is very
much like the system of equations arising in elliptic pdes.
From the computational point of view a better scheme is
k+ 1
wi,j 2
k
wi,j
−
∆t /2
k+ 21
k+1
− wi,j
wi,j
∆t /2

k+ 21
w
 i+1,j
= κ

k+ 1
−
k+ 1
2wi,j 2
∆2x
+
k+ 21
wi−1,j
k+ 1
k+ 1
+
k
wi,j+1
−
k
2wi,j
∆2y
+

k
wi,j−1

,

k+1
k+1
k+1
2
2
2
+ wi,j−1
− 2wi,j
wi,j+1

 wi+1,j − 2wi,j + wi−1,j
+
= κ
 , (76)
2
2
∆x
∆y
which leads to a tridiagonal system of equations similar to the ADI scheme. The
above scheme is second order in time and space and also unconditionally stable.
37
7.7
Consistency revisited
Let us consider the truncation error for the first order central (explicit) scheme
(67), and also the Du-Fort Frankel scheme (72).
If u(x, t) is the exact solution then we may write ukj = u(xj , tk ) and thus from
a Taylor series expansion
uk+1
= u(xj , tk + ∆t ) =
j
∂u
∂t
ukj +∆t
∆2
+ t
2
j,k
!
∂2u
∂t2
!
+ O(∆t )3 ,
(77)
j,k
and
ukj+1 = u(xj + ∆x , tk ) =
ukj + ∆x
∂u
∂x
!
∂2u
∂x2
∆2
+ x
2
j,k
!
∆3
+ x
6
j,k
∂3u
∂x3
∆4
+ x
24
j,k
!
∂4u
∂x4
!
+ O(∆x )5 ,
j,k
(78)
and
ukj−1 = u(xj − ∆x , tk ) =
ukj
− ∆x
∂u
∂x
∆2
+ x
2
j,k
∂2u
∂x2
!
!
∆3
− x
6
j,k
∂3u
∂x3
!
∆4
+ x
24
j,k
∂4u
∂x4
!
+ O(∆x )5 .
j,k
(79)
Using (77, 78,79) and substituting into (67) gives

1  k
∂u
uj + ∆ t
∆t
∂t

κ  k
∂u
uj + ∆ x
2
∆x
∂x
+
ukj − ∆x
∂u
∂x
∆2
+ t
2
j,k
!
!
∆2
+ x
2
j,k
∆2
+ x
2
j,k
∂2u
∂x2
!
∂2u
∂t2
!
∂2u
∂x2
!
j,k
!

− ukj + O(∆t )3  =
∆3
+ x
6
j,k
∆3
− x
6
j,k
∂3u
∂x3
∂3u
∂x3
!
∆4
+ x
24
j,k
!
∆4
+ x
24
j,k
∂4u
∂x4
∂4u
∂x4
!
j,k
!
j,k
− 2ukj

+ O(∆x )5 
from which we obtain
"
∂2u
∂u
−κ 2
∂t
∂x
#
j,k
=−
∆t ∂ 2 u
κ∆2x ∂ 4 u
( 2 )j,k +
(
)j,k .
2 ∂t
12 ∂x4
(80)
Thus (80) shows that as ∆t → 0 and ∆x → 0 the original pde is satisfied, and
the right hand side of (80) implies a truncation error O(∆t , ∆2x ).
38
If we do the same for the Du-Fort Frankel scheme we find that
#
" k
uj+1 − uk+1
− ujk−1 + ukj−1
uk+1
− ujk−1
j
j
−κ
2∆t
∆2x
"
#
∂u
∂2u
∆2 ∂ 2 u
∆2
=
− κ 2 + κ 2t 2
+ O(∆2t , ∆2x , 2t ).
∂t
∂x
∆x ∂t j,k
∆x
(81)
This shows that the Du-Fort scheme is only consistent if the step sizes approach
∆t
zero and also ∆
→ 0 simultaneously. Otherwise if we take step sizes such that
x
∆t
remains constant as both step sizes approach zero, then (81) shows that we
∆x
are solving the wrong equation.
7.8
Stability
Consider the first order explicit scheme which can be written as
k
k
wjk+1 = βwj−1
+ (1 − 2β)wjk + βwj+1
,
1 ≤ j ≤ N − 1,
k
with w0k , wN
given. We can write the above in matrix form as

wk+1 =












(1 − 2β) β
β
(1 − 2β) β
0
β
(1 − 2β) β
.
.
β (1 − 2β)
k
T
where wk = (w1k , w2k , ..., wN
−1 ) . Thus (82) can be written as






 k
w ,





(82)
wk+1 = Awk .
If we recall the results from the discussion of convergence of iterative methods,
the above scheme is stable if and only if ||A|| ≤ 1. Now the infinity norm ||A||∞
is defined by
||A||∞ = max
i
N
X
j
|ai,j |
for an N × N matrix A. Thus from (82) we have
||A||∞ = β + |1 − 2β| + β = 2β + |1 − 2β|.
So if (1 − 2β) ≥ 0 ie β ≤ 1/2 then
||A||∞ = 2β + 1 − 2β = 1.
If (1 − 2β) < 0 then
||A||∞ = 2β + 2β − 1 = 4β − 1 > 1.
Thus we have proved that the explicit scheme is unstable if β > 1/2.
39
7.8.1
Stability of Crank-Nicolson scheme
The Crank-Nicolson (73) scheme may be written as
−βwj−1,k+1 + (2 + 2β)wj,k+1 − βwj+1,k+1 = βwj−1,k + (2 − 2β)wj,k + βwj+1,k ,
j = 1, 2, ..., N − 1
(83)
where β =













∆t κ
.
(∆x )2
In matrix form this can be written as

(2 + 2β) −β
−β
(2 + 2β) −β
0
−β
(2 + 2β) −β
.













.
−β (2 + 2β)
(2 − 2β) β
β
(2 − 2β) β
0
β
(2 − 2β) β





 k+1
w





.
.
β (2 − 2β)
k
T
where wk = (w1k , w2k , ..., wN
−1 ) . Thus (84) is of the form
=






 k
w ,





(84)
Bwk+1 = Awk ,
where B = 2IN −1 − βSN −1 , and A = 2IN −1 + βSN −1 and IN is the N × N identity
matrix, and SN −1 is the (N − 1) × (N − 1) matrix

SN −1 =












−2 1
1 −2 1
0
1 −2 1
.
.
1 −2







.





Thus the Crank-Nicolson scheme will be stable if the spectral radius of the matrix
B−1 A is less than unity, ie ρ(B−1 A) < 1. We therefore need the eigenvalues of
the matrix B−1 A. To find these we need to invoke some theorems from linear
algebra. Recall that λ is an eigenvalue of the matrix S, and x a corresponding
eigenvector if
Sx = λx.
40
Thus for any integer p
Sp x = Sp−1 Sx = Sp−1 λx = ... = λp x.
Hence the eigenvalues of Sp are λp with eigenvector x. Extending this result, if
P (S) is the matrix polynomial
P (S) = a0 Sn + a1 Sn−1 + ... + an I
then
1
x.
P (λ)
Finally if Q(S) is any other polynomial in S then we see that
and P −1 (S)x =
P (S)x = P (λ)x,
P −1 (S)Q(S)x =
Q(λ)
x.
P (λ)
If we let P = B(SN −1 ) = 2IN −1 − βSN −1 , and Q = A(SN −1 ) = 2IN −1 + βSN −1
then the eigenvalues of the matrix B−1 A are given by
µ=
2 + βλ
2 − βλ
where λ is an eigenvalue of the matrix SN −1 .
Now the eigenvalues of the N × N matrix

T=
can be shown to be given by









a b
c a b
0 c a b
. . .
. . .
c a
√
nπ
,
λ = λn = a + 2 bc cos
N +1










n = 1, 2, .., N.
Hence the eigenvalues of SN −1 are
λn = −4 sin2
nπ
,
2N
n = 1, 2, ..., N − 1
and so the eigenvalues of B−1 A are
2 − 4β sin2
µn =
2 + 4β sin2
nπ
N
nπ
N
n = 1, 2, .., N − 1.
Clearly
ρ(B−1 A) = max |µn | < 1 ∀β > 0.
n
This proves that the Crank-Nicolson scheme is unconditionally stable.
41
7.9
Stability condition allowing exponential growth
In the above discussion of stability we have said that the solution of
wk+1 = Awk
is stable if ||A|| ≤ 1. This condition does not make allowance for solutions of
the pde which may be growing exponentially in time. A necessary and sufficient
condition for stability when the solution of the pde is increasing exponentially in
time is that
||A|| ≤ 1 + M ∆t = 1 + O(∆t )
where M is a constant independent of ∆x and ∆t .
7.10
von-Neumann stability analysis
The matrix method for analysing the stability of a scheme can become difficult for
more complicated schemes and in multi-dmensional situations. A very versatile
tool in this regard is the Fourier method developed by von Neumann. Here initial
values at mesh points are expressed in terms of a finite Fourier series, and we
consider the growth of individual Fourier components. A finite sine or cosine
series expansion in the interval a ≤ x ≤ b takes the form
X
X
nπx
nπx
an sin(
bn cos(
), or
),
L
L
n
n
where L = b − a. Now consider an individual component written in complex
exponential form at a mesh point x = xj = a + j∆x
An e
inxπ
L
= An e
inaπ
L
e
injπ∆x
L
= Ãn eiαn j∆x
where αn = nπ/L.
Given initial data we can express the initial values as
wp0 =
N
X
Ãn eiαn p∆x
p = 0, 1, ..., N,
n=0
and we have N + 1 equations to determine the N + 1 unknowns Ã. To find how
each Fourier mode develops in time, assume a simple separable solution of the
form
wpk = eiαn p∆x eΩtk = eiαn p∆x eΩk∆t = eiαn p∆x ξ k ,
where ξ = eΩ∆t . Here ξ is called the amplification factor. For stability we thus
require |ξ| ≤ 1. If the exact solution of the pde grows exponentially, then the
difference scheme will allow such solutions if
|ξ| ≤ 1 + M ∆t
where M does not depend on ∆x or ∆t .
42
7.10.1
Fully implicit scheme
Consider the fully implicit scheme
k+1
wjk+1 − wjk
w k+1 − 2wjk+1 + wj−1
.
= κ j+1
∆t
∆2x
#
"
(85)
Let
wjk = ξ k eiαn j∆x .
Then substituting into (85) gives
1 k
κξ k+1 −iαn ∆x
ξ (ξ − 1)eiαn j∆x =
(e
− 2 + eiαn ∆x )eiαn j∆x .
∆t
∆2x
Thus with β = ∆t κ/∆2x
ξ − 1 = βξ(2 cos(αn ∆x ) − 2) = −4βξ sin2 (
This gives
α n ∆x
).
2
1
,
1 + 4β sin2 ( αn2∆x )
ξ=
and clearly 0 < ξ ≤ 1 for all β > 0 and for all αn . Thus the fully implicit scheme
is unconditionally stable.
7.10.2
Richardson’s scheme
The Richardson scheme is given by
k
k
wj+1
− 2wjk + wj−1
wjk+1 − wjk−1
=κ
.
2∆t
∆2x
"
#
Using a von-Neumann analysis and writing
wpk = ξ k eiαn p∆x ,
gives after substitution into (86)
eiαn p∆x ξ k−1 (ξ 2 − 1) = βξ k (e−iαn ∆x − 2 + eiαn ∆x )eiαn p∆x .
This gives
ξ 2 − 1 = −4ξβ sin2 (
α n ∆x
),
2
where β = 2∆t κ/∆2x . Thus
ξ 2 + 4ξβ sin2 (
α n ∆x
) − 1 = 0.
2
43
(86)
This quadratic has two roots ξ1 , ξ2 . The sum and product of the roots is given
by
α n ∆x
ξ1 + ξ2 = −4ξβ sin2 (
), ξ1 ξ2 = −1.
(87)
2
For stability we require |ξ1 | ≤ 1 and |ξ2 | ≤ 1 and (87) shows that if |ξ1 | < 1 then
|ξ2 | > 1, and vice-versa. Also if ξ1 = 1 and ξ2 = −1 then again from (87) we
must have β = 0. Thus the Richardson scheme is unconditionally unstable.
7.10.3
von Neumann analysis in multi-dimensions
For two or more dimensions, the Fourier method extends easily. For instance for
two space dimensions we would seek solutions of the form
k
wp,q
= eiαn p∆x eiγn q∆y ξ k .
44
7.11
Advanced iterative methods
We have looked at a number of simple iterative techniques such as point SOR,
line SOR and ADI. In this section we will consider more advanced techniques
such as the conjugate gradient method, and minimimal residual methods.
7.11.1
Conjugate Gradient Method
This method was first put forward by Hestenes & Stiefel (1952) and was reinvented again and popularised in the 1970’s. The method has one important
property that it converges to the exact solution of the linear system in a finite
number of iterations, in the absence of roundoff error. Used on its own the
method has a number of drawbacks, but combined with a good preconditioner it
is an effective tool for solving linear equations iteratively.
Consider
Ax = b,
(88)
where A is an n×n symmetric and positive definite matrix. Recall A is symmetric
if AT = A and positive definite if xT Ax > 0 for any non zero vector x. Consider
the quadratic form
1
φ(x) = xT Ax − xT b.
(89)
2
We see that
1
1
φ(x) = xT (Ax − b) − xT b,
2
2
and from the postive definiteness of A the quadratic form has a unique minimum
- 21 bT A−1 b when x = A−1 b.
We will try and find a solution to (88) iteratively. Let xk be the kth iterate.
Now we know that φ decreases most rapidly in the direction ∇φ, and from (89)
we have
∇φ(xk ) = b − Axk = rk .
If the residual rk 6= 0 then we can find an α such that φ(xk + αxk ) < φ(xk ). In
the method of steepest descent we choose α to minimize
1 T
(x + αrTk )A(xk + αrk ) − (xTk + αrTk )b
2 k
1
1 T
xk Axk + αrTk Axk + α2 rTk Ark − xTk b − αrTk b,
=
2
2
φ(xk + αrk ) =
(90)
where we have used xTk Ark = (xTk Ark )T = rTk Axk . Differentiating (90) with
respect to α and equating to zero to find a minimum gives
0 = rTk Axk + αrTk Ark − rT b,
which gives
αk =
rTk rk
.
rTk Ark
45
(91)
The problem with the steepest descent algorithm as constructed above is that
convergence can be slow especially if A is ill conditioned. The condition number
of a matrix A is defined by
max λi
κ(A) =
min λi
where λi is an eigenvalue of the matrix A. The matrix is said to be ill-conditioned
if κ is large. The steepest descent algorithm works well if κ is small, but for the
system of equations arising from a second-order finite difference approximation
to Poisson’s equation, κ is very large. In a sense the steepest descent method
is slow because the search directions are too similar. We can try and search for
the new iterate in another direction, say pk . To minimize φ(xk + αpk ) we must
choose as in (91)
pT pk
αk = T k
,
pk Apk
where now the search direction is different from the direction of the gradient.
Then
1
φ(xTk−1 + αk pTk ) = (xTk−1 + αk pTk )A(xk−1 + αk pk ) − (xTk−1 + αk pTk )b,
2
= φ(xk−1 ) −
1 (pTk rk−1 )2
,
2 pTk Apk
and φ is reduced provided that pk is not orthogonal to rk−1 . Thus the algorithm
loops over
rk−1 = b − Axk−1
pTk pk
αk =
pTk Apk
xk = xk−1 + αk pk
(92)
Thus xk is a linear combination of the vectors p1 , p2 , ..., pk . How can we choose
the pi ? It turns out that we can choose them so that each xk solves the problem
min φ(x),
x ∈ span(p1 , p2 , ..., pk ),
when we choose α to solve
min φ(xk−1 + αpk )
α
provided that we make the pi A conjugate.
Define an n×k matrix Pk by Pk = [p1 , p2 , ..., pk ] . Then if x ∈ span(p1 , p2 , ..., pk )
there exists a (k − 1) vector y such that
x = Pk−1 y + αpk ,
46
and
φ(x) = φ(Pk−1 y) + αyT PTk−1 Apk −
Choosing the {pi } so that
α2 T
p Apk − αpk b.
2 k
PTk−1 Apk = 0
decouples the globalpart of the minimization from the part in the pk direction.
Then the required global minimization of φ(x) is obtained by choosing
Pk−1 y = xk−1 ,
and αk =
pTk b
pTk rk−1
=
,
pTk Apk
pTk Apk
as before.
From (??)
pTi Apk = 0,
i = 1, 2, ..., k − 1,
ie pk is A to p1 , p2 , ..., pk−1 . Thus the {pi } are linearly independent and hence
xn is a linear combination of p1 , p2 , ..., pn−1 is the exact solution of
Ax = b.
Condition ?? does not determine pk uniquely. To do this we choose the pk
to be closest to rk−1 because rk−1 gives the most rapid reduction of φ(x).
It can be shown that, (eg, see Golub & van Loan) that
• the residuals r1 , r2 , ..., rk−1 are orthogonal,
• pTi rj = 0,
i = 1, 2, .., j,
• rj = rj−1 − αj Apj ,
• pk = rk−1 + βk pk−1 where
βk = −
pTk−1 Ark−1
.
pTk−1 Apk−1
47
7.11.2
Conjugate Gradient Algorithm
Using these results we obtain the following algorithm
r0 = b − Ax0
p0 = r 0
FOR i = 0, ..., n DO
BEGIN
(ri , ri )
(pi , Api )
xi+1 = xi + αi pi
ri+1 = ri − αi Api
(ri+1 , ri+1 )
βi =
(ri , ri )
pi+1 = ri+1 + βi pi
αi =
END
In practice xn 6= x because of rounding errors and the residuals are not
orthogonal. It can be shown that
"
√ #2k
1− κ
√
||x − xk ||A ≤ ||x − x0 ||A
1+ κ
so that convergence can be slow for very large κ. Preconditioning of A is required
for the CG method to be useful.
7.11.3
Implementation issues
Note that the algorithm requires the matrix vector product Ap to be computed.
The matrix A is usually very large and sparse. As far as implementation of the
algorithm is concerned we do not explicitly construct the matrix A, only the effect
of A on a vector. Consider, for example, a second-order discretization applied to
say Poisson’s equation, ∇2 φ = f (x, y) with Dirichlet boundary conditions. The
discretization gives the set of equations
wi+1,j − 2wi,j + wi+1,j wi,j+1 − 2wi,j + wi,j−1
+
= fi,j ,
∆2x
∆2y
2≤i≤N −1
,
2≤j ≤M −1
and
w1,j = f l(j),
w(N, j) = f r(j),
48
1 ≤ j ≤ M,
wi,1 = gb(i),
wi,M = gt(i),
2 ≤ i ≤ N − 1,
and f l(y), f r(y), gb(x), gt(x) are known functions given on the boundaries. Here
we may define x to the N M × 1 vector of unknowns
x = (w1,1 , w2,1 .., wN,1 , w1,2 , ..., wi,j , ..., wN,M )T .
In fact we may identify the qth component xq of x by
xq = wi,j
where q = i + N (j − 1).
Then the matrix vector product Ax will have elements dq with q = i + N (j − 1)
given by
wi+1,j − 2wi,j + wi+1,j wi,j+1 − 2wi,j + wi,j−1
dq =
+
,
∆2x
∆2y
if i, j identifies an interior point of the domain, and
d1+N (j−1) = w1,j ,
di = wi,1 ,
dN +N (j−1) = wN,j ,
di+(M −1)N = wi,M ,
1 ≤ j ≤ M,
2 ≤ i ≤ N − 1,
for points on the boundary.
One other point to re-emphazise is that in the above analysis/discussion the
matrix A is assumed to be symmetric and positive definite. Usually the system
of equations that we come across are not symmetric. Here one needs to consider
alternative algorithms, such as the GMRES method discussed later, or one could
try and use the conjugate gradient algorithm with the system
AT Ax = AT b.
The latter is not recommended as the condition number of the matrix AT A
deteriorates if the original matrix A is ill-conditioned to start with.
7.11.4
CG, alternative derivation
Let A be a symmetric positive definite matrix ie
A = AT ,
xT Ax > 0
for any non-zero vector x.
Also define an inner product
(x, y) = xT y.
We saw that the functional
1
F (x) = (x, Ax) − (b, x),
2
49
is minimized when x is a solution of
Ax = b
The steepest descent method is defined by looking at iterations of the form
x(k+1) = x(k) + αk r(k)
where
r(k) = b − Ax(k)
is the residual, and in the direction of the gradient of F (x).
The α’s are chosen to minimize F (x(k+1) ), and come out to be
αk =
(r(k) , r(k) )
.
(r(k) , Ax(k) )
The convergence of this technique can be slow, especially if the matrix is
ill-conditioned. By choosing the directions differently we obtain the conjugate
gradient algorithm.
This works as follows. Let
x(k+1) = x(k) + αk p(k)
where p(k) is a direction vector. For the Conjugate Gradient method we let
p(k) = r(k) + βk p(k−1)
for n ≥ 1, and βk is chosen so that p(k) is A conjugate to x, ie
(p(k) , Ap(k−1) ) = 0.
This gives
(r(k) , Ap(k−1) )
.
(p(k−1) , Ap(k−1) )
As before α is chosen to minimize F (x(k+1) ), and come out to be
βk = −
(p(k) , r(k) )
αk = (k)
.
(p , Ap(k) )
Collecting all the results together gives the CG algorithm
x(0)
x
(k+1)
p(k)
arbitrary
= x + αk p(k) , k = 0, 1, ..
(
r(k) , k = 0
=
r(k) + βk p(k−1) , k = 1, 2,
(k)
(r(k) , Ap(k−1) )
, k = 1, 2, ..
(p(k−1) , Ap(k−1) )
= b − Ax(k) , k = 0, 1, ..
(p(k) , r(k) )
=
, k = 0, 1, ..
(p(k) , Ap(k) )
βk = −
r(k)
αk
50
It can be shown that the αk , βk and r(k) are equivalent to
(r(k) , r(k) )
, k = 1, 2, ..
(r(k−1) , r(k−1) )
(r(k) , r(k) )
= − (k)
, k = 0, 1, ..
(p , Ap(k) )
= r(k−1) − αk−1 Ap(k−1) , k = 1, 2, ..
βk = −
αk
r(k)
Also the following relations hold
(r(i) , r(j) ) = 0 i 6= j
(p(i) , Ap(j) ) = 0 i 6= j
(r(i) , Ap(j) ) = 0 i 6= j
and i 6= j + 1
The residual vectors r(k) are mutually orthogonal and the direction vectors p(k)
are mutually A conjugate. Hence the name conjugate gradient.
7.11.5
Preconditioning
The conjugate gradient method combined with preconditioning is a very effective
tool for solving linear equations iteratively. Preconditioning helps to redistribute
the eigenvalues of the iteration matrix and this can significantly enhance the
convergence properties. Suppose that we are dealing with the linear system
Ax = b.
(93)
A preconditioning matrix M is such that M is close to A in some sense, and
easily invertible. For instance if we were dealing with the matrix A arising from
a discretization of the Laplace operator, then one could take M to be any one
matrices stemming from one of the classical iteration schemes such as Jacobi,
SOR, or ADI, say. The system (93) can be replaced by
M−1 Ax = M−1 b.
The conjugate algorithm is then applied to (94) leading to the algorithm
x0
guess r0
solve Mz0
p0
FOR k
αk
=
=
=
=
Ax0 − b
r0 and put
z0
1, n
(zk−1 , rk−1 )
=
(pk−1 , Apk−1 )
51
(94)
xk = xk−1 + αk pk−1
rk = rk−1 − αk Apk−1
Solve Mzk = rk
(zk , rk )
βk =
(zk−1 , rk−1 )
pk = zk + βk pk−1
END
52