Download Phase imaging - EECS Berkeley

Computational illumination for high‐resolution 3D phase imaging
Laura Waller
Department of Electrical Engineering and Computer Sciences
Affiliate: Bioengineering, Applied Sciences & Technology
UC Berkeley
[email protected]
www.laurawaller.com
L. Waller, UC Berkeley
1/45
Computational imaging
The computer becomes a part of the imaging system
1) Modify optical design
2) Take picture(s)
3) Crunch Data
4) Final result
5) Start company
Raytrix Gmbh
L. Waller, UC Berkeley
2/45
Optics Abstractions
Quantum optics
Electromagnetic optics
Wave optics
Geometrical optics
(ray tracing)
Paraxial ray optics
(matrix methods)
B. Saleh, Photonics.
L. Waller, UC Berkeley
3/45
Phase imaging: seeing the invisible
30um
» light is a wave, so has intensity and phase
» optical phase is too fast to capture directly
Intensity
radians
Phase
L. Waller, UC Berkeley
4/45
Intensity
Applications: biological microscopy
1um
Phase
E-coli
1
1
0.5
0.5
0
0
-0.5
-0.5
rad
endothelial cells
Human cheek cells
HeLa cells
L. Waller, UC Berkeley
5/45
Applications: X‐ray and EUV phase imaging
Photomask phase edges
E
LBNL tomography beamline
E
Absorption
degrees
2.5um
A. Shanker, et al. Applied Optics (2014).
L. Waller, UC Berkeley
Phase contrast
Bech, Martin, et al. Scientific reports 3 (2013).
» bioimaging
» materials characterization
» lithography
6/45
Phase retrieval is a nonlinear problem
x
Optical System Design
complex
function
optical system
A
detector
(measures intensity)
Algorithm
Intensity 
L. Waller, UC Berkeley
Ax
2
7/45
7/20
Phase from defocus
Z. Jingshan, L. Tian, J. Dauwels,
L. Waller, Optics Express (2014).
1
z stage
0.5
0
-0.5
focus
stack
camera
Intensity ( z )  Az x
L. Waller, UC Berkeley
2
40µm
radians
recovered phase
Leverages existing hardware
 software add‐on only
8/45
8/20
Phase from defocus with many images
Intensity stack
z
phase – old methods
rad
solve for phase
+ noise (SNR~0.5)
phase – our methods
High noise causes most techniques to break down!
Kalman filter [1‐3] provides optimal noise performance
[1] L. Waller, M. Tsang, S. Ponda, G. Barbastathis, Opt. Express 19 (2011).
[2] Z. Jingshan, J. Dauwels, M. Vazquez, L Waller, IEEE ICASSP 1229 (2012).
[3] Z. Jingshan, J. Dauwels, M. Vasquez, L. Waller , Opt. Express (2013).
L. Waller, UC Berkeley
9/45
9/20
Covariance matrix is impractically large
N=number of pixels > 1 Mpxl
Kalman filter must store and manipulate covariance matrix of size N2  Terabytes of data!
Sparsity of covariance matrix can be exploited:
Complexity
Kalman filter
Sparse Kalman
filter
Storage
O( N 3 )
O(N2 )
O(N log N)
O( N )
phase –
phase ‐augmented Kalman
Kalman filter
For our data, we achieve 1011 speed‐up
Z. Jingshan, J. Dauwels, M. Vasquez, L. Waller , Opt. Express (2013).
L. Waller, UC Berkeley
10/45
10/20
Focus step positions can be optimized
Recovered phase (radians)
1
0.5
0
-45.6 μm
Intensity stack
45.6 μm
-0.5
equally spaced
129 images
Z. Jingshan, R. Claus, J. Dauwels, L. Tian, L. Waller, Opt. Express 22(9), 10661-10674 (2014).
L. Waller, UC Berkeley
11/45
11/20
Focus step positions can be optimized
Recovered phase (radians)
1
0.5
0
-45.6 μm
Intensity stack
45.6 μm
-0.5
non‐equally spaced
5 images
Z. Jingshan, R. Claus, J. Dauwels, L. Tian, L. Waller, Opt. Express 22(9), 10661-10674 (2014).
L. Waller, UC Berkeley
12/45
12/20
x
Optical System Design
complex
function
optical system
A
detector
(measures intensity)
Algorithm
Intensity 
L. Waller, UC Berkeley
Ax
2
13/45
Computational illumination with an LED array
LED array
200µm
L.Tian, X.Li,
K.Ramchandran, L.
Waller, Biomed. Opt.
Express (2014).
25µm
Gigapixel
phase
imaging
camera
Real-time multi-contrast
3D imaging
brightfield
L. Tian, L. Waller,
Optica (2015).
L. Waller, UC Berkeley
darkfield
phase contrast
Z. Liu, L. Tian, S. Liu, L. Waller, J. Biomed. Optics (2014).
14/45
Multi‐contrast in an LED array microscope
sample
existing microscope
Fourier space
camera
LED array
Brightfield Darkfield
» replace illumination unit with LED array
» Each LED corresponds to illumination by a different angle
Phase Contrast 200μm
L. Waller, UC Berkeley
15/45
Real‐time multi‐contrast imaging
Brightfield
Z. Liu, L. Tian, S. Liu, L. Waller, J. Biomed. Optics (2014).
L. Waller, UC Berkeley
Dark field
Phase contrast
» 20x, NA~0.4, 30 Hz
» C. Elegans on NGM petri plate (Dillin lab, UCB)
16/45
Differential phase contrast
contrast is proportional to the derivative of phase![1‐3]
Left
Right
Left – Right
Left + Right
B. Kachar, Science 227, 27 (1985).
[1] D. Hamilton & C. Sheppard, J. Microscopy 133, 27 (1984).
[2] T. Ford, K. Chu & J. Mertz, Nat. Methods 9, 1195 (2012).
[3] S. B. Mehta and C. J. Sheppard, Opt. Lett. 34, 1924 (2009).
L. Waller, UC Berkeley
17/45
Transfer function for differential phase contrast ky
sample
Pupil space
pupil
kx
LED array
existing microscope
camera
» Amplitude information is symmetric in Fourier space
» Phase information is asymmetric
Phase to intensity
transfer function
20µm

L. Waller, UC Berkeley

18/19
18/45
Transfer function for differential phase contrast existing microscope
sample
ky
Pupil space
pupil
kx
LED array
camera
Phase transfer function
NA 2NA
» Similar to derivative model within NA
» 2x better resolution from partial coherence
[1] D. Hamilton, C. Sheppard, T. Wilson, J. Microscopy 135, 766 (1984).
[2] D. Hamilton, C. Sheppard, J. Microscopy 133, 27 (1984).
L. Waller, UC Berkeley
19/19
19/45
Quantitative phase from DPC measurements
existing microscope
sample
ky
Pupil space
pupil
kx
camera
LED array
Phase transfer function
DPC image
Phase
rad
1
‐1
20µm
L. Waller, UC Berkeley
20/19
20/45
It’s easy to change the DPC angle
LED array
camera
L. Waller, UC Berkeley
21/45
2‐axis DPC is good for accuracy & speed
Top‐bottom DPC
Left‐right DPC
2‐axis DPC
Reconstructed phase
Phase transfer function
20µm
L. Waller, UC Berkeley
22/45
Computational illumination with an LED array
LED array
200µm
L.Tian, X.Li,
K.Ramchandran, L.
Waller, Biomed. Opt.
Express (2014).
25µm
Gigapixel
phase
imaging
camera
Real-time multi-contrast
3D imaging
brightfield
L. Tian, L. Waller,
Optica (2015).
L. Waller, UC Berkeley
darkfield
phase contrast
Z. Liu, L. Tian, S. Liu, L. Waller, J. Biomed. Optics (2014).
23/45
Darkfield images represent sub‐resolution features
sample’s Fourier space
ky
NAillumination
NA = NAobjective + NAillumination
NAobjective
kx
Can we stitch together all of Fourier space?
Our version of ideas in:
G.Zheng, R. Horstmeyer, C. Yang, Nat. Photon. (2013).
L. Waller, UC Berkeley
low-resolution image
24/45
Scanning through all LEDs provides full coverage in Fourier space + overlap redundancy
ky
NAillumination
NAeff  NAobj  NAillum
 0.1  0.6
 0.7
kx
L. Waller, UC Berkeley
25/45
Fourier Ptychography achieves resolution beyond the diffraction limit of the objective
Raw data, central LED on
Reconstructed image
Full Field of View image
20µm
300µm
4x 0.1NA  NAeffective = 0.6
Our version of ideas in:
G.Zheng, R. Horstmeyer, C. Yang, Nat. Photon. (2013).
L. Waller, UC Berkeley
26/45
Digital Aberration removal
Raw data,
central LED illumination
Reconstructed image
Full Field of View image
rad
4x 0.1NA Nikon, Synthesized NA = 0.6
recovered aberration
Our version of ideas in:
G.Zheng, R. Horstmeyer, C. Yang, Nat. Photon. (2013).
O. Xu, C. Yang, Opt. Express (2013).
L. Waller, UC Berkeley
27/45
Reconstruction algorithm is nonlinear optimization
# of LEDs = 1
start with low‐resolution object initial guess
repeat for each measurement
Check estimate against data
– forward problem
1) Impose intensity constraint
2) Fourier support constraint
– inverse problem
# of LEDs = 21
Background correction[2]
# of LEDs = 293
Update object estimate
Regularization for robustness to noise[2]
[1] G.Zheng, R. Horstmeyer, C. Yang, Nat. Photon. (2013).
[2] L.Tian, X.Li, K.Ramchandran, L. Waller, Biomed. Opt. Express (2014).
L. Waller, UC Berkeley
28/45
Comparing to other phase imaging methods
Full 4x FoV phase reconstruction (synthetic 0.7 NA)
Phase from TIE 40x 0.65 NA
1
20µm
200µm
‐0.6
40x 0.65 NA Phase contrast
40x FoV
Intensity
Phase
unstained human breast basal epithelial cell MCF10A
L. Waller, UC Berkeley
29/45
Unstained
Human Bone Osteosarcoma Epithelial U2OS
(phase) sample
200µm
1
‐0.6
20µm
FoV~2.1mmx1.7mm
resolution ~400nm
(~0.7 NA)
Phase
Intensity
L. Waller, UC Berkeley
13k x 11k pixels reconstructed
30/45
We collect a lot of data
» For current resolution/FoV:
» ~300 images (16 bit, 5.5 Mpxl each) » Full frame rate of camera = 50 fps
» Speed limit = 0.55 Gb/s!
» Phase retrieval requires ~10x more data collected than reconstructed (overlap in Fourier space)
How can we use data more efficiently?
L. Waller, UC Berkeley
32/45
Coding strategies for phase?
ky
LED
array
sample
objective
pupil
plane
kx
camera
f
existing microscope
object’s Fourier space
» Illuminate sample from multiple LEDs
» Want:
1) LEDs far apart
2) Asymmetry for phase contrast
Random coding is good!
L.Tian, X.Li, K.Ramchandran, L. Waller, Biomed. Opt. Express (2014).
L. Waller, UC Berkeley
33/45
Multiplexed illumination improves acquisition speed
sample’s Fourier space
ky
NAillumination
NAobjective
kx
L.Tian, X.Li, K.Ramchandran, L. Waller, Biomed. Opt. Express (2014).
L. Waller, UC Berkeley
low-resolution image
34/45
Multiplexing reduces acquisition time and data
low resolution zoom‐in
Sequential 1 LED 293 images
T = 586s
Multiplexed 8 LEDs
293 images
T= 293s
Multiplexed 8 LEDs
40 images
T = 40s
50µm
Only uses 14% of the data!
synthesized NA=0.6
L.Tian, X.Li, K.Ramchandran, L. Waller, Biomed. Opt. Express (2014).
L. Waller, UC Berkeley
35/45
35/20
Time‐lapse phase (in vitro)
200µm
60x FOV
15µm
1
‐0.6
80x FOV
40x FOV
1.7mm
10µm
20µm
20µm
40x FOV
~0.4µm
resolution
(0.7 NA)
Hela
Cells
L. Waller, UC Berkeley
2.1mm
36/45
LED array
3D intensity and phase imaging
camera
L. Waller, UC Berkeley
37/45
Scanning illumination angles relates to depth
LED array
scan illumination
in (x,y)
xi
camera
thick sample
zi
∆z
∆x
L. Waller, UC Berkeley
38/45
Light field dataset gives depth information
varying illumination angle
recovered 3D intensity
θx
y
x
θx
θx
x
x
Our version of ideas in:
G.Zheng, C. Kolner, C. Yang, Opt. Lett.. (2011).
L. Waller, UC Berkeley
39/45
Digital refocusing for phase contrast
LED
pattern
L
R
100µm
3D intensity
L. Tian, J. Wang, L. Waller, Opt. Lett. 39(5), 2014.
L. Waller, UC Berkeley
3D phase contrast
10x, NA = 0.25
Refocusing range: ‐100µm‐100µm
C. Elegans on NGM petri plate (Dillin lab)
40/45
Diffraction effects cause error in light field results
Light field refocus
z = -55 µm
Physical focus
z = 55 µm
50µm
We need phase to correct diffraction errors!
L. Tian, L. Waller, Optica (2015).
L. Waller, UC Berkeley
41/45
Diffraction effects cause error in light field results
z = -55 µm
Physical focus
Light field refocus
Our method
1
z = 55 µm
50µm
0
L. Tian, L. Waller, Optica (2015).
L. Waller, UC Berkeley
42/45
Multi‐slice model for 3D
LED array
3D sample
L. Waller, UC Berkeley
43/45
200µm
25µm
varying illumination
angle
θx
high
resolution
y
OR? AND?
x
3D
L. Waller, UC Berkeley
44/45
Can we get enhanced resolution + 3D?
Physical focus
3D Fourier Ptychography
z = -77 µm
Low‐resolution full FoV image
from 4 0.1 NA objective
z = 33 µm
50µm
300µm
L. Tian, L. Waller, Optica (2015).
L. Waller, UC Berkeley
10µm
45/45
Resolution depends on defocus distance
L. Tian, L. Waller, Optica (2015).
L. Waller, UC Berkeley
46/45
Resolution depends on defocus distance
L. Tian, L. Waller, Optica (2015).
L. Waller, UC Berkeley
47/19
47/45
3D reconstructions in phase and amplitude
x
y
z
L. Tian, L. Waller, Optica (2015).
L. Waller, UC Berkeley
Synthetic NA 0.7, Spirogyra algae
49/45
Current work: mobile microscopy
Brightfield
DPC Left/Right
Darkfield
LED array
DPC Top/Bottom
camera
L. Waller, UC Berkeley
51/45
Acknowledgements:
» Berkeley Computational Imaging Lab Find us on GigaPan: WallerLab_Berkeley
Twitter: @optrickster
Open-source code & datasets: www.laurawaller.com
» Samples from: Fletcher lab, Li Lab, Yildiz lab, UCB Stem Cell center, Dillin lab, Herr lab
» Funding:
L. Waller, UC Berkeley
52/45