Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Today Image restoration • Modeling image degradation: degradation function and additive noise Computational Photography • Restoration under noise only • Estimating the degradation function • Deconvolution Matthias Zwicker University of Bern Fall 2009 Image restoration Image restoration • Given: degraded image • Assumption: degradation function is linear and shift invariant • Assume degradation model • Degradation model – Blur modeled by degradation function (lens imperfections, motion blur) – Additive noise (sensor noise) – Spatial domain g(x,, y) = h(x, g( ( , y) ? f ((x,, y) + η( η(x,, y) • Goal: remove degradations f (x, y) g(x, y) Degradation function H + Noise Degradation – Frequency domain Restoration filters ˆ y) f(x, η (x,y) Restoration Noise models • Model noise as random variables G(u, v) = H(u, v)F (u, v) + N (u, v) • Non-linear, spatially varying degradation functions common, but much harder to handle Noise PDFs Gaussian Uniform Impulse – Noise values determined by a probability density function (PDF) • Common assumptions: noise PDF is – IIndependent d d t off spatial ti l coordinates, di t or periodic i di – Uncorrelated to image itself 2 p(z) = (z−z̄) √ 1 e− 2σ 2 2πσ ⎧ ⎨ p(z) = ⎩ 1 b−a 0 if a ≤ z ≤ b otherwise ⎧ ⎪ ⎪ P for z = a ⎪ ⎨ a p(z) = ⎪ Pb for z = b ⎪ ⎪ ⎩ 0 otherwise 1 Examples Examples • Image without noise Gaussian Uniform Estimating PDFs Today • One possibility: Fit PDF parameters to histogram of uniform image region Image restoration • Example: Gaussian noise – Probability for image values – Estimated from histogram Estimated mean z̄ = X zi pS (zi ) i pS (zi ) Estimated variance σ2 = X i (zi − z̄)2 pS (zi) Impulse • Modeling image degradation: degradation function and additive noise • Restoration under noise only • Estimating i i the h d degradation d i ffunction i • Inverse filtering • Wiener filtering • Constrained least squares filtering Uniform patch Histogram • Richardson-Lucy non-blind deconvolution Restoration under noise only Restoration under noise only • „Denoising“ • General idea: replace each pixel with some sort of local average of pixel values • Could use linear smoothing (low-pass) filters – Problem: sharp edges are blurred • Many sophisticated, non-linear edge preserving smoothing filters exist • Today: some simple examples, more later • Various options to estimate a local average – Neighborhood Sxy with mn pixels – Arithmetic mean (corresponds to box filter) X 1 g(s, t) fˆ(x, y) = mn (s,t)∈Sxy – Geometric mean (nonlinear!) ⎡ ⎤ fˆ(x, y) = ⎢⎣ Y (s,t)∈Sxy 1/mn g(s, t)⎥⎦ – Harmonic mean (nonlinear!) fˆ(x, y) = mn P 1 (s,t)∈Sxy g(s,t) 2 Filtering using order statistics • Idea – Order pixels in a neighborhood according to their values – Replace central pixel by certain value found from the ordered list Median filter • Advantage over smoothing with linear filter – More effective removal of outliers – Less blurring – Suitable for „salt and pepper noise“ • Examples for order statistics filters – Minimum – Maximum – Median: „value in the middle of ordered list“, 50th percentile, 50% of values are smaller, 50% larger • Filters are nonlinear! Input 3x3 box 3x3 median Today Estimating the degradation fct. Image restoration • Remember: assume linear shift invariant degradation • Modeling image degradation: degradation function and additive noise • Restoration under noise only • Estimating i i the h d degradation d i ffunction i • Inverse filtering • Wiener filtering • Constrained least squares filtering • Richardson-Lucy non-blind deconvolution g(x, y) = h(x, y) ? f (x, y) + η(x, y) G(u, v) = H(u, v)F (u, v) + N (u, v) •G Goal: l estimate i iimpulse l response h (or ( frequency response H) of degradation fct. – Degradation function also known as blur kernel • Approaches – Using mathematical modeling – From observation Using mathematical modeling From observation • Use knowledge about capturing process • Assume you have two images of the same scene, one degraded, one undegraded • Example: camera moves relative to scene at known speed, known exposure time – Degradation function for motion blur can be d i d ffrom thi derived this iinformation f ti • Disadvantage: applicable only for specific cases where capturing process is precisely known – Rarely the case in consumer photography – Degraded (in freq. domain) Gs (u, v) – Undegraded (in freq. domain) F̂s (u, v) • Recover degradation function from this image pair – Assume no noise for now Hs (u, v) = Gs (u, v) F̂s(u, v) 3 From observation Practical problems • How to obtain a pair of degraded and undegraded images? • Often impossible to capture pair of degraded/undegraded image • Capture image of known, simple objects – Undegraded image is known a priori – Example: Image of an impulse directly shows impulse p response! p • Example: blur due to camera shake of hand-held camera – Shaking is different for every shot • Solution: use other known properties of captured image and degradation function Undegraded image of impulse (known a priori) Degraded, captured image of impulse – E.g., most edges in images are sharp, degradation function is smooth, statistical properties of image – Active research area Example from recent research Example from recent research • „PSF Estimation using Sharp Edge Prediction”, Joshi et al. 2008 • “Removing Camera Shake from a Single Photograph”, Fergus et al. 2006 http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/psf_estimation.pdf – Includes technique to estimate undegraded image (sharp edges) from degraded image Undegraded Degraded, captured image (cropped) image of a priori known scene Recovered degradation function (spatially varying) http://www.cs.nyu.edu/~fergus/papers/deblur_fergus.pdf • Use statistical assumption about undegraded image to recover blur kernel – Sparse derivative prior: image has few, but sharp edges – Leads to pretty hairy optimization problem Degraded, captured image Recovered blur kernel (not to scale) Today Deconvolution Image restoration • Given: degraded image, blur kernel, noise properties • Modeling image degradation: degradation function and additive noise • Restoration under noise only • Goal: restore image • Known as non-blind deconvolution • Estimating the degradation function • Deconvolution Degraded image and blur kernel Restored image 4 Deconvolution Inverse filtering • Today • Degradation model (frequency domain) G(u, v) = H(u, v)F (u, v) + N (u, v) – Inverse filtering – Wiener filtering – Constrained least square filtering • Restored image using inverse filtering • Not covered – Blind deconvolution: recover blur kernel and restore image simultaneously – Assuming degradation function H known G(u, v) N(u, v) F̂ (u, v) = = F (u, v) + H(u, v) H(u, v) • Problems – Noise N(u,v) is never known – Cannot restore F(u,v) exactly – Often H(u,v) has (some) very small values, therefore noise dominates restored image! Inverse filtering Wiener filtering • Observation • Goal: solid mathematical problem statement for image (or any other signal) restoration in the presence of noise – Blur kernel has small values only for high frequencies – Only high frequencies in result of inverse filtering are corrupted by noise • Idea: after inverse filtering, remove high frequencies using a low-pass filter • Degraded image modeled as random field – 2D array off random d variables i bl • Find restored image, such that expected error to undegraded image is minimized min E{(f − fˆ)2 } fˆ Degraded image Inv. filter & low-pass No low-pass – Uncorrupted image f – Estimated image fˆ Wiener filtering Wiener filtering • Assumptions • Usually power spectrum of undegraded image not known – Noise and image are uncorrelated – Noise or image has zero mean (usually noise) – Estimate is linear function of degraded image • Solution ⎡ F̂ (u, v) = ⎣ ⎤ |H(u, v)|2 1 ⎦ G(u, v) 2 H(u, v) |H(u, v)| + Sη (u, v)/Sf (u, v) • Simplification ⎡ ⎤ 1 |H(u, v)|2 ⎦ F̂ ((u, v)) = G(u, ( v)) H( v)) |H(u, H(u, |H( v)| )|2 + K ⎣ – Manually adjusted constant K – Degradation function H(u, v) – Power sepctrum of noise Sη (u, v) – Power spectrum of undegraded image Sf (u, v) 5 Wiener filtering Wiener filtering • Parameters tweaked for best results Disadvantages • Need to know ratio of power spectra of noise and undegraded image • Approximation as a constant not always suitable ⎡ F̂ (u, v) = ⎣ Degraded image Wiener filter Inverse filter with low-pass Terminology ⎡ ⎤ |H(u, v)|2 ⎦ 1 F̂ (u, v) = G(u, v) H(u, v) |H(u, v)|2 + K ⎣ Constrained least squares filtering • Signal-to-noise ratio SNR = • Mean square error MSE = ⎤ |H(u, v)|2 1 ⎦ G(u, v) 2 H(u, v) |H(u, v)| + Sη (u, v)/Sf (u, v) 1 MN P P 2 u v |F (u, v)| P P 2 u v |N (u, v)| XX u v [f (x, y) − fˆ(x, y)]2 – Image resolution MN • Often use square root of SNR and MSE – Root mean square signal to noise ratio – Root mean square (RMS) error • Requires knowledge only of mean and variance of noise – Easier to estimate than power spectrum, as required for Wiener filtering • Problem statement: find image that is – As smooth as possible – Difference between restored image and degraded image matches known noise parameters Matrix formulation Matrix formulation • Image of size M x N as a column vector of size (M x N) x 1 • Minimize Laplacian („result is as smooth as possible“) XX [∇2 fˆ(x, y)]2 – Euclidean vector norm denoted by k · k of • Convolution kernel applied to image size M x N as a matrix of size (M x N) x (M x N) • Image degradation model in matrix form g = Hf + η – Equivalent to g(x, y) = h(x, y) ? f (x, y) + η(x, y) x y • Under the constraint („estimate matches known noise level“) kg − Hf̂k2 = kηk2 • Solution ⎡ F̂ (u, v) = ⎣ ⎤ H ∗ (u, v) ⎦ G(u, v) |H(u, v)|2 + γ|P (u, v)|2 – Fourier transform of Laplacian P (u, v) – Parameter that needs to be adjusted γ 6 Finding gamma Obtaining noise estimate • Adjust gamma such that • Definition 2 2 kηk2 = 2 krk = kg − Hf̂k = kηk ± ² • Iterative procedure 1. Specify p y initial value for γ 2. Obtain estimate f̂ and compute krk2 3. If above equation not satisfied, adjust γ, go back to 2. • Variance ση2 = 1 MN • Mean mη = M −1 NX −1 X η(x, y)2 x=0 y=0 MX −1 NX −1 x=0 y=0 1 MN [η(x,, y) − mη ]2 [η( M −1 NX −1 X η(x, y) x=0 y=0 • Therefore: need only variance and mean! kηk2 = MN [ση2 + m2η ] Summary Richardson-Lucy deconvolution • Wiener and constrained least squares filter lead to very similar equations and results • Think of image as 2D probability distribution • Wiener filter designed to optimize restoration in statistical sense • Constrained least squares filter formulated as problem to find optimal restoration of one specific input • Detailed derivation, e.g., in „Image Processing: The Fundamentals“, Petrou & Bosdogianni, Wiley & Sons Bayes‘ theorem (discrete events) – Undegraded image represented by probability f(ξi) for photon to fall on pixel ξi – 1D Notation for convenience • Image degradation d d – Probability that we measure photon at pixel xj, although it really was at ξi, given by P (xj |ξi ) X – Degraded image g(xj ) = f (ξi )P (xj |ξi ) i – Conditional probability corresponds to blur kernel P (xj |ξi ) = h(xj − ξi ) Derivation • Probability of event A given B • P(A) prior probability of A • Similar – "prior“: p does not use information about B • P(A|B) conditional probability of A, given B • Together – Also “posterior probability”: depends upon the specified value of B • P(B|A) conditional probability of B given A • P(B) is the prior B 7 Alternative form Richardson-Lucy deconvolution • Given partition of the event space {Ai} • Undegraded image expressed using unknown „inverse filter“ f (ξi ) = X j P (ξi |xj )g(xj ) (#) • Using Bayes theorem • Bayes Bayes‘ theorem P (ξi |xj ) = P P (x ( j |ξi )f (ξi ) = P (xj |ξk )f (ξk ) k • Iterative procedure P h(xj − ξi )f (ξi ) h( (##) h(xj − ξk )f (ξk ) k – Estimate P (ξi |xj ) using (##) and current f (ξi ) – Update f (ξi ) using (#) Richardson-Lucy deconvolution Maximum likelihood • Compactly, iteration index r • Given: probability distribution with a set of parameters f r+1(ξi ) = f (ξi )r X j P h(xj − ξi )g(xj ) r k h(xj − ξk )f (ξk ) • Why does this make sense, i.e., converge to a desired solution? – Can show that converges to maximum likelihood solution for Poisson noise model – Derivation next • Likelihood function: given observation, tells us for each parameter setting how likely it is that these parameters lead to observation b i • Maximum likelihood solution: given observation, the one parameter setting that maximizes the likelihood function – Unknown parameters that most likely generated the observation In our context Poisson distribution • Assumption • Expected number of occurences (i.e., photons) λ – Photons, i.e. pixels, are independent – Measurement error of pixels obeys Poisson distribution • Observed number of occurences k • Likelihood function – Product of Poisson distributions – As a function of undegraded pixel values • Maximum likelhood – Most likely undegraded pixel values that led to observed image k 8 In equations Maximizing log-likelihood • Likelihood for observed image y given undegraded image x, blur kernel h, Poisson noise • Take derivative with respect to x • Maximize likelihood, or equivalently loglikelihood, with respect to undegraded image x • Using matrix notation • Nonlinear in unknown x, iterative solution Richardson-Lucy deconvolution Richardson-Lucy Summary • Non-blind deconvolution based on probabilistic model • Yields maximum likelihood solution under Poisson noise model • Standard method used widely – Astronomy, Hubble space telescope • Suffers from noise amplification Degraded image Richardson-Lucy, 10 iterations Literature Recent advances • Original papers • Progressive Inter-scale and Intra-scale Nonblind Image Deconvolution, Yuan et al., SIGGRAPH 2008 – “Bayesian-Based Iterative Method of Image Restoration”, Richardson, 1972 http://www.opticsinfobase.org/abstract.cfm?URI=josa-62-1-55 – “An iterative technique for the rectification of observed distributions”,, Lucy, y, 1974 http://articles.adsabs.harvard.edu/full/1974AJ.....79..745L • Adaptive method, derived by maximizing likelihood using Poisson noise model http://research.microsoft.com/en-us/um/people/jiansun/papers/ImageDeconv_Siggraph08.pdf • Extension of Richardson-Lucy – Edge preserving filter – Multiresolution approach – “An adaptively accelerated Lucy-Richardson method for image deblurring”, Singh et al. 2008 http://portal.acm.org/citation.cfm?id=1387867 9 Results Input with blur kernel (not to scale) Results Richardson-Lucy Yuan et al. Input with blur kernel (not to scale) Richardson-Lucy Yuan et al. Next time • Gradient based image manipulation 10