Download 05 Image restoration.pdf

Today
Image restoration
• Modeling image degradation: degradation
function and additive noise
Computational Photography
• Restoration under noise only
• Estimating the degradation function
• Deconvolution
Matthias Zwicker
University of Bern
Fall 2009
Image restoration
Image restoration
• Given: degraded image
• Assumption: degradation function is linear
and shift invariant
• Assume degradation model
• Degradation model
– Blur modeled by degradation function (lens
imperfections, motion blur)
– Additive noise (sensor noise)
– Spatial domain
g(x,, y) = h(x,
g(
( , y) ? f ((x,, y) + η(
η(x,, y)
• Goal: remove degradations
f (x, y)
g(x, y)
Degradation
function H
+
Noise
Degradation
– Frequency domain
Restoration
filters
ˆ y)
f(x,
η (x,y)
Restoration
Noise models
• Model noise as random variables
G(u, v) = H(u, v)F (u, v) + N (u, v)
• Non-linear, spatially varying degradation
functions common, but much harder to
handle
Noise PDFs
Gaussian
Uniform
Impulse
– Noise values determined by a probability
density function (PDF)
• Common assumptions: noise PDF is
– IIndependent
d
d t off spatial
ti l coordinates,
di t
or periodic
i di
– Uncorrelated to image itself
2
p(z) =
(z−z̄)
√ 1 e− 2σ 2
2πσ
⎧
⎨
p(z) = ⎩
1
b−a
0
if a ≤ z ≤ b
otherwise
⎧
⎪
⎪
P for z = a
⎪
⎨ a
p(z) = ⎪ Pb for z = b
⎪
⎪
⎩ 0 otherwise
1
Examples
Examples
• Image without noise
Gaussian
Uniform
Estimating PDFs
Today
• One possibility: Fit PDF parameters to histogram of
uniform image region
Image restoration
• Example: Gaussian noise
– Probability for image values
– Estimated from histogram
Estimated mean
z̄ =
X
zi pS (zi )
i
pS (zi )
Estimated variance
σ2 =
X
i
(zi − z̄)2 pS (zi)
Impulse
• Modeling image degradation: degradation
function and additive noise
• Restoration under noise only
• Estimating
i
i the
h d
degradation
d i ffunction
i
• Inverse filtering
• Wiener filtering
• Constrained least squares filtering
Uniform patch
Histogram
• Richardson-Lucy non-blind deconvolution
Restoration under noise only
Restoration under noise only
• „Denoising“
• General idea: replace each pixel with
some sort of local average of pixel values
• Could use linear smoothing (low-pass)
filters
– Problem: sharp edges are blurred
• Many sophisticated, non-linear edge
preserving smoothing filters exist
• Today: some simple examples, more later
• Various options to estimate a local average
– Neighborhood Sxy with mn pixels
– Arithmetic mean (corresponds to box filter)
X
1
g(s, t)
fˆ(x, y) =
mn (s,t)∈Sxy
– Geometric mean (nonlinear!)
⎡
⎤
fˆ(x, y) = ⎢⎣
Y
(s,t)∈Sxy
1/mn
g(s, t)⎥⎦
– Harmonic mean (nonlinear!)
fˆ(x, y) =
mn
P
1
(s,t)∈Sxy g(s,t)
2
Filtering using order statistics
• Idea
– Order pixels in a neighborhood according to their
values
– Replace central pixel by certain value found from
the ordered list
Median filter
• Advantage over smoothing with linear filter
– More effective removal of outliers
– Less blurring
– Suitable for „salt and pepper noise“
• Examples for order statistics filters
– Minimum
– Maximum
– Median: „value in the middle of ordered list“,
50th percentile, 50% of values are smaller, 50%
larger
• Filters are nonlinear!
Input
3x3 box
3x3 median
Today
Estimating the degradation fct.
Image restoration
• Remember: assume linear shift invariant
degradation
• Modeling image degradation: degradation
function and additive noise
• Restoration under noise only
• Estimating
i
i the
h d
degradation
d i ffunction
i
• Inverse filtering
• Wiener filtering
• Constrained least squares filtering
• Richardson-Lucy non-blind deconvolution
g(x, y) = h(x, y) ? f (x, y) + η(x, y)
G(u, v) = H(u, v)F (u, v) + N (u, v)
•G
Goal:
l estimate
i
iimpulse
l response h (or
(
frequency response H) of degradation fct.
– Degradation function also known as blur kernel
• Approaches
– Using mathematical modeling
– From observation
Using mathematical modeling
From observation
• Use knowledge about capturing process
• Assume you have two images of the same
scene, one degraded, one undegraded
• Example: camera moves relative to scene
at known speed, known exposure time
– Degradation function for motion blur can be
d i d ffrom thi
derived
this iinformation
f
ti
• Disadvantage: applicable only for specific
cases where capturing process is precisely
known
– Rarely the case in consumer photography
– Degraded (in freq. domain) Gs (u, v)
– Undegraded (in freq. domain) F̂s (u, v)
• Recover degradation function from this
image pair
– Assume no noise for now
Hs (u, v) =
Gs (u, v)
F̂s(u, v)
3
From observation
Practical problems
• How to obtain a pair of degraded and
undegraded images?
• Often impossible to capture pair of
degraded/undegraded image
• Capture image of known, simple objects
– Undegraded image is known a priori
– Example: Image of an impulse directly shows
impulse
p
response!
p
• Example: blur due to camera shake of
hand-held camera
– Shaking is different for every shot
• Solution: use other known properties of
captured image and degradation function
Undegraded image of
impulse (known a priori)
Degraded, captured
image of impulse
– E.g., most edges in images are sharp,
degradation function is smooth, statistical
properties of image
– Active research area
Example from recent research
Example from recent research
• „PSF Estimation using Sharp Edge Prediction”,
Joshi et al. 2008
• “Removing Camera Shake from a Single
Photograph”, Fergus et al. 2006
http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/psf_estimation.pdf
– Includes technique to estimate undegraded image
(sharp edges) from degraded image
Undegraded Degraded, captured
image (cropped)
image of a
priori known
scene
Recovered degradation
function
(spatially varying)
http://www.cs.nyu.edu/~fergus/papers/deblur_fergus.pdf
• Use statistical assumption about undegraded image
to recover blur kernel
– Sparse derivative prior: image has few, but sharp
edges
– Leads to pretty hairy optimization problem
Degraded, captured image
Recovered blur kernel
(not to scale)
Today
Deconvolution
Image restoration
• Given: degraded image, blur kernel, noise
properties
• Modeling image degradation: degradation
function and additive noise
• Restoration under noise only
• Goal: restore image
• Known as non-blind deconvolution
• Estimating the degradation function
• Deconvolution
Degraded image and blur kernel
Restored image
4
Deconvolution
Inverse filtering
• Today
• Degradation model (frequency domain)
G(u, v) = H(u, v)F (u, v) + N (u, v)
– Inverse filtering
– Wiener filtering
– Constrained least square filtering
• Restored image using inverse filtering
• Not covered
– Blind deconvolution: recover blur kernel and
restore image simultaneously
– Assuming degradation function H known
G(u, v)
N(u, v)
F̂ (u, v) =
= F (u, v) +
H(u, v)
H(u, v)
• Problems
– Noise N(u,v) is never known
– Cannot restore F(u,v) exactly
– Often H(u,v) has (some) very small values,
therefore noise dominates restored image!
Inverse filtering
Wiener filtering
• Observation
• Goal: solid mathematical problem
statement for image (or any other signal)
restoration in the presence of noise
– Blur kernel has small values only for high frequencies
– Only high frequencies in result of inverse filtering are
corrupted by noise
• Idea: after inverse filtering, remove high
frequencies using a low-pass filter
• Degraded image modeled as random field
– 2D array off random
d
variables
i bl
• Find restored image, such that expected
error to undegraded image is minimized
min E{(f − fˆ)2 }
fˆ
Degraded image
Inv. filter & low-pass
No low-pass
– Uncorrupted image f
– Estimated image fˆ
Wiener filtering
Wiener filtering
• Assumptions
• Usually power spectrum of undegraded
image not known
– Noise and image are uncorrelated
– Noise or image has zero mean (usually noise)
– Estimate is linear function of degraded image
• Solution
⎡
F̂ (u, v) = ⎣
⎤
|H(u, v)|2
1
⎦ G(u, v)
2
H(u, v) |H(u, v)| + Sη (u, v)/Sf (u, v)
• Simplification
⎡
⎤
1
|H(u, v)|2 ⎦
F̂ ((u, v)) =
G(u,
( v))
H( v)) |H(u,
H(u,
|H( v)|
)|2 + K
⎣
– Manually adjusted constant K
– Degradation function H(u, v)
– Power sepctrum of noise Sη (u, v)
– Power spectrum of undegraded image Sf (u, v)
5
Wiener filtering
Wiener filtering
• Parameters tweaked for best results
Disadvantages
• Need to know ratio of power spectra of
noise and undegraded image
• Approximation as a constant not always
suitable
⎡
F̂ (u, v) = ⎣
Degraded image
Wiener filter
Inverse filter
with low-pass
Terminology
⎡
⎤
|H(u, v)|2 ⎦
1
F̂ (u, v) =
G(u, v)
H(u, v) |H(u, v)|2 + K
⎣
Constrained least squares filtering
• Signal-to-noise ratio
SNR =
• Mean square error
MSE =
⎤
|H(u, v)|2
1
⎦ G(u, v)
2
H(u, v) |H(u, v)| + Sη (u, v)/Sf (u, v)
1
MN
P P
2
u v |F (u, v)|
P P
2
u v |N (u, v)|
XX
u
v
[f (x, y) − fˆ(x, y)]2
– Image resolution MN
• Often use square root of SNR and MSE
– Root mean square signal to noise ratio
– Root mean square (RMS) error
• Requires knowledge only of mean and
variance of noise
– Easier to estimate than power spectrum, as
required for Wiener filtering
• Problem statement: find image that is
– As smooth as possible
– Difference between restored image and
degraded image matches known noise
parameters
Matrix formulation
Matrix formulation
• Image of size M x N as a column vector of
size (M x N) x 1
• Minimize Laplacian („result is as smooth as
possible“)
XX
[∇2 fˆ(x, y)]2
– Euclidean vector norm denoted by
k · k of
• Convolution kernel applied to image
size M x N as a matrix of size
(M x N) x (M x N)
• Image degradation model in matrix form
g = Hf + η
– Equivalent to
g(x, y) = h(x, y) ? f (x, y) + η(x, y)
x
y
• Under the constraint („estimate matches
known noise level“)
kg − Hf̂k2 = kηk2
• Solution
⎡
F̂ (u, v) = ⎣
⎤
H ∗ (u, v)
⎦ G(u, v)
|H(u, v)|2 + γ|P (u, v)|2
– Fourier transform of Laplacian P (u, v)
– Parameter that needs to be adjusted γ
6
Finding gamma
Obtaining noise estimate
• Adjust gamma such that
• Definition
2
2
kηk2 =
2
krk = kg − Hf̂k = kηk ± ²
• Iterative procedure
1. Specify
p
y initial value for γ
2. Obtain estimate f̂ and compute krk2
3. If above equation not satisfied, adjust γ,
go back to 2.
• Variance
ση2 =
1
MN
• Mean
mη =
M
−1 NX
−1
X
η(x, y)2
x=0 y=0
MX
−1 NX
−1
x=0 y=0
1
MN
[η(x,, y) − mη ]2
[η(
M
−1 NX
−1
X
η(x, y)
x=0 y=0
• Therefore: need only variance and mean!
kηk2 = MN [ση2 + m2η ]
Summary
Richardson-Lucy deconvolution
• Wiener and constrained least squares filter
lead to very similar equations and results
• Think of image as 2D probability
distribution
• Wiener filter designed to optimize
restoration in statistical sense
• Constrained least squares filter formulated
as problem to find optimal restoration of
one specific input
• Detailed derivation, e.g., in „Image
Processing: The Fundamentals“, Petrou &
Bosdogianni, Wiley & Sons
Bayes‘ theorem (discrete events)
– Undegraded image represented by probability
f(ξi) for photon to fall on pixel ξi
– 1D Notation for convenience
• Image degradation
d
d
– Probability that we measure photon at pixel xj,
although it really was at ξi, given by P (xj |ξi )
X
– Degraded image
g(xj ) = f (ξi )P (xj |ξi )
i
– Conditional probability corresponds to blur
kernel P (xj |ξi ) = h(xj − ξi )
Derivation
• Probability of event A given B
• P(A) prior probability of A
• Similar
– "prior“:
p
does not use information about B
• P(A|B) conditional probability of A, given B
• Together
– Also “posterior probability”: depends upon the
specified value of B
• P(B|A) conditional probability of B given A
• P(B) is the prior B
7
Alternative form
Richardson-Lucy deconvolution
• Given partition of the event space {Ai}
• Undegraded image expressed using
unknown „inverse filter“
f (ξi ) =
X
j
P (ξi |xj )g(xj )
(#)
• Using Bayes theorem
• Bayes
Bayes‘ theorem
P (ξi |xj ) =
P
P (x
( j |ξi )f (ξi )
=
P (xj |ξk )f (ξk )
k
• Iterative procedure
P
h(xj − ξi )f (ξi )
h(
(##)
h(xj − ξk )f (ξk )
k
– Estimate P (ξi |xj ) using (##) and current f (ξi )
– Update f (ξi ) using (#)
Richardson-Lucy deconvolution
Maximum likelihood
• Compactly, iteration index r
• Given: probability distribution with a set
of parameters
f r+1(ξi ) = f (ξi )r
X
j
P
h(xj − ξi )g(xj )
r
k h(xj − ξk )f (ξk )
• Why does this make sense, i.e., converge
to a desired solution?
– Can show that converges to maximum
likelihood solution for Poisson noise model
– Derivation next
• Likelihood function: given observation,
tells us for each parameter setting how
likely it is that these parameters lead to
observation
b
i
• Maximum likelihood solution: given
observation, the one parameter setting
that maximizes the likelihood function
– Unknown parameters that most likely
generated the observation
In our context
Poisson distribution
• Assumption
• Expected number of occurences (i.e.,
photons) λ
– Photons, i.e. pixels, are independent
– Measurement error of pixels obeys Poisson
distribution
• Observed number of occurences k
• Likelihood function
– Product of Poisson distributions
– As a function of undegraded pixel values
• Maximum likelhood
– Most likely undegraded pixel values that led to
observed image
k
8
In equations
Maximizing log-likelihood
• Likelihood for observed image y given
undegraded image x, blur kernel h, Poisson noise
• Take derivative with respect to x
• Maximize likelihood, or equivalently loglikelihood, with respect to undegraded image x
• Using matrix notation
• Nonlinear in unknown x, iterative solution
Richardson-Lucy deconvolution
Richardson-Lucy Summary
• Non-blind deconvolution based on
probabilistic model
• Yields maximum likelihood solution under
Poisson noise model
• Standard method used widely
– Astronomy, Hubble space telescope
• Suffers from noise amplification
Degraded image
Richardson-Lucy, 10 iterations
Literature
Recent advances
• Original papers
• Progressive Inter-scale and Intra-scale Nonblind Image Deconvolution, Yuan et al.,
SIGGRAPH 2008
– “Bayesian-Based Iterative Method of Image
Restoration”, Richardson, 1972
http://www.opticsinfobase.org/abstract.cfm?URI=josa-62-1-55
– “An iterative technique for the rectification of
observed distributions”,, Lucy,
y, 1974
http://articles.adsabs.harvard.edu/full/1974AJ.....79..745L
• Adaptive method, derived by maximizing
likelihood using Poisson noise model
http://research.microsoft.com/en-us/um/people/jiansun/papers/ImageDeconv_Siggraph08.pdf
• Extension of Richardson-Lucy
– Edge preserving filter
– Multiresolution approach
– “An adaptively accelerated Lucy-Richardson
method for image deblurring”, Singh et al.
2008
http://portal.acm.org/citation.cfm?id=1387867
9
Results
Input with blur
kernel (not to scale)
Results
Richardson-Lucy
Yuan et al.
Input with blur
kernel (not to scale)
Richardson-Lucy
Yuan et al.
Next time
• Gradient based image manipulation
10