Download Optimization Iterative Methods for Smooth Objective Functions

Optimization
Iterative Methods for Smooth Objective Functions
 Quadratic Objective Functions
 Stationary Iterative Methods (first/second order)
 Steepest Descent Method
 Landweber/Projected Landweber Methods
 Conjugate Gradient Method
 Non-Quadratic Smooth Objective Functions
 Conjugate Gradient Method
 Newton’s Method
 Trust Region Globalization of Newton’s Method
 BFGS Method
IPIM, IST, José Bioucas, 2015
1
References
[1] O. Axelsson, Iterative Solution Methods. New York: Cambridge
Univ. Press, 1996.
[2] Golub, G.H. and Van Loan, C.F., Matrix Computations, Johns Hopkins
University Press, Baltimore, Maryland, 1983.
[3] C. Byrne, A unified treatment of some iterative algorithms in
signal processing and image reconstruction, Inverse Problems, vol.
20, pp. 103–120, 2004.
IPIM, IST, José Bioucas, 2015
2
Rates of convergence
Suppose that
as
Linear convergence rate: there exits a constant
for which
Superlinear convergence rate: there exits a sequence
of real numbers such that
and
Quadratic convergence rate: there exits a constant
for which
IPIM, IST, José Bioucas, 2015
3
Rates of convergence: example
0
10
linear
superlinear
quadratic
-20
10
-40
10
-60
10
-80
10
-100
10
-120
10
-140
10
0
IPIM, IST, José Bioucas, 2015
5
10
15
20
25
30
35
40
45
50
4
Comparing linear convergence rates
 Many iterative methods for large scale inverse problems
have linear converge rate:
– convergence factor
r - log10 – convergence rate
– number of iterations to reduce the error by a factor of 10
IPIM, IST, José Bioucas, 2015
5
Induced norms and spectral radius
 Given a vector norm
vector norm is
, the matrix norm
induced by the
 When the vector norm is the Euclidian norm, the induced norm is termed
the spectral norm and is given by
 If is Hermitian
spectral radius of A,
IPIM, IST, José Bioucas, 2015
, the matrix norm
is given by the
6
Key results involving the spectral radius
IPIM, IST, José Bioucas, 2015
7
Tikhonov regularization/Gaussian priors
Assume that
is non-singular. Then
The solution is obtained by solving the system
IPIM, IST, José Bioucas, 2015
8
Stationary iterative methods
Consider the system
, where
is non-singular
First Order Stationary Iterative Methods
Let
is nonsingular
be a splitting of
for
must be ease to invert
Jacobi
Gauss-Seidel
IPIM, IST, José Bioucas, 2015
9
Stationary iterative methods
Frequently, we can not access to the elements of A or D, but only
apply these operators. Thus C should depend only on these operators
Example 1: Landweber iterations
Example 2:
Easy to compute when D is
diagonal or a convolution
IPIM, IST, José Bioucas, 2015
10
First order stationary iterative methods: convergence
Consider the system
Let
be a splitting of
is nonsingular
and
Then
iff
IPIM, IST, José Bioucas, 2015
11
First order stationary iterative methods: convergence
Consider the system
Let
be a splitting of
for
Convergence
iff
IPIM, IST, José Bioucas, 2015
12
First order stationary iterative methods (cont.)
Ill-conditioned systems
Number of iterations to attenuate the error norm by 10
 Landweber C = I

Under what conditions?
The eigenvalues of
IPIM, IST, José Bioucas, 2015
tend to be less spread than those of
13
Second order stationary iterative methods: convergence
Consider the system
Let
be a splitting of
Convergence [1]
iff
IPIM, IST, José Bioucas, 2015
14
First/second order stationary iterative methods: comparison
Ill-conditioned systems
First order
Second order
Example
Second order is 100 times faster
IPIM, IST, José Bioucas, 2015
15
Steepest descent method
non-stationary first order iterative
method
Optimal  (line search)
IPIM, IST, José Bioucas, 2015
Convergence
16
Conjugate gradient method
Consider the system
Are conjugate with respect to
if
Equivalently
Let
be a sequence of n mutualy conjugate directions and
Since
Then
and
IPIM, IST, José Bioucas, 2015
17
Conjugate gradient method as an iterative method
Computing the solution of
2- Define
is equivalent to minimize
to as the projection error of
onto the direction
1- minimize along conjugate directions directions
IPIM, IST, José Bioucas, 2015
18
Conjugate gradient and steepest descent paths
steepest descent
conjugate gradient
IPIM, IST, José Bioucas, 2015
19
The resulting algorithm
(
IPIM, IST, José Bioucas, 2015
denotes the negative of the gradient)
20
Some remarks about the CG method
Convergenge [2]
IPIM, IST, José Bioucas, 2015
21
Comparison: CG and First/Second Order Stationary Iterative Methods
4
10
1st order
3
10
2
10
1
10
2nd order
0
10
CG
-1
10
0
50
100
IPIM, IST, José Bioucas, 2015
150
200
250
300
350
400
450
500
22
Preconditioned conjugate gradient (PCG) method
Let
be a s.p.d matrix such that
The eigenvalues of
CG solves the system
system
are more clustered than those of
faster than the
Note: PCG can be written as a small modification of CG: The complexity
of each PCG iteration is that of CG plus the computation of
IPIM, IST, José Bioucas, 2015
23
Constrained Tikhonov regularization/Gaussian priors
where
is a closed convex set
Projection onto a convex set
is non-expansive
IPIM, IST, José Bioucas, 2015
24
Projected iterations
is a closed convex set
Let
be a contraction mapping
Assume that sequence generated by
solution of the unconstrained problem
converges to the
Define the operator:

is a contraction mapping
 for any starting element
, the sequence of sucessive approximations
is convergent and its limit is the unique fixed point of
 the unique fixed point of
optimization problem
IPIM, IST, José Bioucas, 2015
is the solution of the constrained
25