Download Slides

Introduction to
Sparse Representations
Sparse Representations and
their applications in signal and image processing
Raja Giryes
Tel Aviv University
October 30th, 2016
2
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Regulations
• Three HW assignments (submission in pairs) – 20%
• Exam on 23.2.2017 – 30%
• Final project (also in pairs) based on recently
published 1-3 papers. Projects will be assigned
within a month.
• The project will include:
▫ A final report (10-20 pages) summarizing these
papers, their contributions, and your own findings
(open questions, simulation results, etc.) – 25%.
▫ A Power-point presentation of the project in a miniworkshop on February 27 – 25%.
3
Sparse Representation
and their Applications in
Signal and Image
Processing
Course Website
web.eng.tau.ac.il/sparsity
Oct. 30, 2016
4
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Homework and Course Forum
• Course forum is active in the Moodle system
• Serves as a platform for online discussion
between students
• Asking questions about the homeworks
• TA in the course: Guy Leibovitz.
5
Sparse Representation
and their Applications in
Signal and Image
Processing
The sparsity model for signals and images
Oct. 30, 2016
6
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Why Sparsity?
• Emerging and fast growing field.
Published
items
in each year
Citations in each year
• New algorithms
and
theory.
• State-of-the-art results in many fields.
▫
▫
▫
▫
▫
▫
Audio processing
Video processing
Image processing
Radar
Medical imagingSearching ISI-Web-of-Science (October 30th 2016):
Topic=((spars* and (represent* or approx* or
Etc.
solution) and (dictionary or pursuit)) or (compres*
and sens* and spars*)). Years-2007-2016
7
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Image Denoising
Original
Noisy (20.43dB)
Result (30.75dB)
[Mairal, Elad & Sapiro, 2006]
8
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Image Deblurring
Original
Blury
Deblurred
[Dong, Shi, Ma & Li, 2015]
9
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Super-Resolution
Original
Blury
Deblurred
[Dong, Shi, Ma & Li, 2015]
10
Poisson Inpainting
29 October 2014
Poisson Denoising Problem
x0
peak  0.1
y
11
Poisson Inpainting
29 October 2014
Poisson Denoising Applications
•
•
•
•
•
•
Tomography – CT, PET and SPECT
Astrophysics
Fluorescence Microscopy
Night Vision
Spectral Imaging
etc.
12
Poisson Inpainting
29 October 2014
Tomography
Slices of skeletal SPECT image
[Takalo , Hytti and Ihalainen 2011]
13
Poisson Inpainting
29 October 2014
Fluorescence Microscopy
C. elegans embryo labeled with three fluorescent dyes
[Luisier, Vonesch, Blu and Unser 2010]
14
Poisson Inpainting
29 October 2014
Astrophysics
XMM/Newton image of the Kepler SN1604 supernova
[Starck, Donoho and Candès 2003]
15
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Poisson Noise Removal
Original image
Noisy image
Recovery
Max y value = 7
Peak = 1
[Giryes and Elad 2014].
16
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Poisson Noise Removal
Original image
Noisy image
Recovery
Max y value = 3
Peak = 0.2
[Giryes and Elad 2014].
17
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Poisson Noise Removal
Original image
Noisy image
Recovery
Max y value = 8
Peak = 2
[Giryes and Elad 2014].
18
Poisson Inpainting
29 October 2014
Inpainting Results
24.34dB
Peak = 1
19
Poisson Inpainting
29 October 2014
Inpainting Results
23.58dB
Peak = 2
20
Poisson Inpainting
29 October 2014
Inpainting Results
22.72dB
Peak = 2
21
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Image Compression Artifacts
Artifact
type
Architectural
Cause
Appears in
Blockiness
Independent
treatment of
blocks.
Block-based
methods (e.g.,
JPEG).
Ringing
Elimination of high
frequency
coefficients.
Visible for largeblock transforms as
in wavelet coding
(e.g., JPEG-2000).
JPEG compression at 0.189bpp
ringing
Appears along sharp
edges and spreads
in the transform
block.
Blurring
Loss of highfrequency
components.
Transform-coding at
low bit-rates.
blur
JPEG-2000 compression at 0.273bpp
22
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
JPEG Postprocessing
Bitrate
)bpp(
JPEG
Foi et al.
Zhang et al.
Proposed
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
0.363
32.96
0.874
34.06
0.893
34.24
0.895
34.32
0.896
0.511
34.76
0.904
35.55
0.912
35.82
0.916
35.87
0.916
0.638
35.81
0.919
36.44
0.924
37.77
0.927
36.81
0.927
0.807
36.86
0.931
37.34
0.933
37.73
0.937
37.79
0.937
0.537
28.25
0.856
28.91
0.877
30.19
0.886
29.67
0.890
0.764
30.89
0.906
31.42
0.918
32.80
0.926
32.59
0.929
0.938
32.54
0.927
33.02
0.935
34.32
0.942
34.33
0.944
1.149
34.22
0.944
34.64
0.949
35.81
0.955
35.94
0.956
[Dar, Bruckstein, Elad and Giryes, 2016]
23
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
JPEG Postprocessing
JPEG Result at 0.363bpp (32.96dB)
Postprocessing Result (34.32dB)
[Dar, Bruckstein, Elad and Giryes, 2016]
24
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
JPEG-2000 Postprocessing
JPEG-2000 Result at 0.40bpp (30.79dB)
Postprocessing Result (31.51dB)
[Dar, Bruckstein, Elad and Giryes, 2016]
25
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
HEVC Postprocessing
Bit-rate
(bpp)
HEVC
Proposed
Improvement
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
0.177
25.79
0.7278
25.89
0.7337
0.10
0.0059
0.340
28.91
0.8357
29.00
0.8359
0.09
0.0002
0.639
32.71
0.9180
32.92
0.9202
0.21
0.0022
1.046
36.31
0.9577
36.51
0.9583
0.20
0.000
6
0.120
27.55
0.7658
27.76
0.7749
0.21
0.0091
0.206
30.22
0.8420
30.45
0.8466
0.23
0.0046
0.401
33.30
0.9163
33.54
0.9187
0.24
0.0024
0.746
36.81
0.9554
37.12
0.9581
0.31
0.0027
[Dar, Bruckstein, Elad and Giryes, 2016]
26
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Segmentation
[Giryes, Elad and Bruckstein, 2016]
27
Sparse Representation
and their Applications in
Signal and Image
Processing
The sparsity model for signals and images
Oct. 30, 2016
28
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Our Problem Setup
• Given the linear measurements y  Mx  e, y 
y
m
e


M
md
x
d
• Target: Recover x from y.
md
• M
is the measurement matrix (m≤d).
m
m
29
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Denoising
• Given the linear measurements y  x  e, y 
y
m


Zero entries
x
e
m
d
One entries
• Target: Recover x from y with noise reduction.
• M=I is the identity matrix.
m
30
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Debluring/Deconvolution
• Given the linear measurements y  Mx  e, y 
y
m


Zero entries
x
e
d
• Target: Recover x from y.
• M  d d is a block circulant matrix.
m
m
31
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Super-Resolution
• Given the linear measurements y  Mx  e, y 
y
m
e


Zero entries
M
md
x
d
• Target: Recover x from y.
• M  md is the downsampling matrix (m<d).
m
m
32
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Compressed Sensing
• Given the linear measurements y  Mx  e, y 
y
m
e


M
md
x
d
• Target: Recover x from y.
• M  md is a (random) sensing matrix (𝑚 ≪ 𝑑).
m
m
33
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
A General Recipe
• Assume 𝑥 belongs to a certain low-dimensional
model Κ
• Given a noisy measurement 𝑦 of 𝑥, we may
recover 𝑥 by finding the vector 𝑥 that
▫ Closest to 𝑦
▫ Belongs to the model Κ
34
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Sparsity Prior (Synthesis)
x  D , 
x
d
0
k
km

Zero entries
Non-zero entries

D
d n
n
35
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Classical Problem Setup
• We look at the measurements now as
y
m


A  MD 
mn

x
d


D
d n
e
n
n
Signal is
recovered by
Our target
m
36
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Sparsity Minimization Problem
• Assume bounded adversarial noise
e 2 
• The problem we aim at solving is simply:
ˆ l  arg min w 0
0
• xˆl0  Dˆ l0
w
n
s.t.
y  Aw 2  
• What can we say about the recovery?
• Other models provide new setups
37
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
Other Low-Dimensional Models
•
•
•
•
•
Structure sparsity
Analysis cosparse model
Gaussian mixture models (GMM)
Low-rank matrices
Low-dimensional manifolds
38
Sparse Representation
and their Applications in
Signal and Image
Processing
Oct. 30, 2016
web.eng.tau.ac.il/sparsity