Survey
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
Download Computational Camera & Photography: Coded Imaging Camera Culture Ramesh Raskar
Document related concepts
no text concepts found
Transcript
Computational Camera & Photography: Coded Imaging Camera Culture Ramesh Raskar MIT Media Lab http://cameraculture.media.mit.edu/ 6KDGRZ 5HIUDFWLYH 5HIOHFWLYH Image removed due to copyright restrictions. See Fig. 1, “Eight major types of optics in animal eyes.” In Fernald, R. D. “Casting a Genetic Light on the Evolution of Eyes.” Science 313, no. 5795 (September 29, 2006): 1914-1918. http://dx.doi.org/10.1126/science.1127889 )HUQDOG 6FLHQFH>6HSW@ Traditional Camera Shutter is OPEN Blurring |H(f)| Sinc Function == Convolution f Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Traditional Camera: Box Filter Blurring Process for Linear Motion T T = * $ 6PHDU 0DWUL[ Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. [ 8QNQRZQ ,PDJH E %OXUUHG ,PDJH Deblurring Process T ? = \ A Smear Matrix Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. \ b Blurred Image Unknown Image - Condition number for the smearing matrix is large - Thus invertibility is bad Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Input Image Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Rectified Crop Deblurred Result Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Input Photo Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Deblurred Result Traditional Camera Shutter is OPEN Our Camera Flutter Shutter Shutter is OPEN and CLOSED Comparison of Blurred Images Implementation Completely Portable Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera • Dragonfly2 camera • External Trigger – Mode 5 – Multiple Exposure Pulse Width Mode • On Chip Fluttered Integration, no extra cost Courtesy of MERL. Used with permission. Lab Setup Blurring |H(f)| Sinc Function == Convolution f Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Traditional Camera: Box Filter |H(f)| Preserves High Spatial Frequencies f Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Flutter Shutter: Coded Filter Comparison |H(f)| |H(f)| f Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. f |1/H(f)| |1/H(f)| Inverse Filter stable Inverse Filter Unstable f Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. f Short Exposure Long Exposure Coded Exposure Our result Matlab Lucy Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Ground Truth Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.” Proceedings of SIGGRAPH 2006. Motion Blur as Convolution Convolution == Linear System ? ? ? Solving Are all codes “good”? |H(f)| f All ones |H(f)| f Alternate f Random |H(f)| Our Code f Need to consider zero padded codes |H(f)| f |H(f)| f |H(f)| f License Plate Retrieval Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Camera Limitations and Tradeoffs • Low dynamic range • Resolution vs Noise • Motion blur – Reduce shutter speed, but lose light • Limited depth of field – Reduce aperture, but lose light – Increase shutter time, but motion blur Courtesy of MERL. Used with permission. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Changing Aperture Size Images removed due to copyright restrictions. • Samuel Hasinoff and Kiriakos Kutulakos, ‘A Layer-Based Restoration Framework for Variable-Aperture Photography’ ICCV 2007 Mitsubishi Electric Research Labs (MERL) How to handle focus blur? Coding and Modulation in Cameras Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Coded Exposure (Flutter Shutter) Coded Aperture with Veeraraghavan, Raskar, Tumblin, & Mohan, Raskar, Agrawal, Tumblin SIGGRAPH 2006 SIGGRAPH 2007 Temporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband code: Focus Deblurring Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras LED In Focus Photo Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Out of Focus Photo: Open Aperture Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Out of Focus Photo: Coded Aperture Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Out of Focus Photo: Coded Aperture Blurred Photos Open Aperture Coded Aperture, 7 * 7 Mask Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. Deblurred Photos Open Aperture Coded Aperture, 7 * 7 Mask Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. Mitsubishi Electric Research Labs (MERL) Captured Blurred Photo Coding and Modulation in Cameras Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Refocused on Person Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. MERL, Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Northwestern Univ. Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin Blur Estimation & Segmentation •Defocus blur dependent on depth •Assumptions •Layered Lambertian Scene •Constant blur within each layer k=1 Deblur at different blur sizes k Captured Blurred Photo k = 10 MERL, Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Northwestern Univ. Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin Define Cost Function k=1 k = 10 Deblurred Images k=1 k = 10 Cost Function Images Likelihood Error: (Blurred image - Sharp Image * PSFk)2 Gradient Error: Natural Image Statistics, Gradient Kurtosis MERL, Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Northwestern Univ. Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin Blur Estimation & Segmentation == Labeling • Graph cuts for labeling k=1 K=1 k = 10 Error Images K=7 Captured Photo Reblur Deblur, k = 7 Fusion Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. MERL, Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Northwestern Univ. Weighted Deconvolution Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin MERL, Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Northwestern Univ. Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007. Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin Mitsubishi Electric Research Labs (MERL) Blocking Light Coded Exposure Coding in Time Coding and Modulation in Cameras == More Information Coded Aperture Coding in Space Mask? Mask Sensor Sensor Mask Full Resolution Digital Refocusing: 4D Light Field from 2D Photo: Coded Aperture Camera Heterodyne Light Field Camera Conventional Lens: Limited Depth of Field Open Aperture Smaller Aperture Courtesy of Shree Nayar. Used with permission. Slide by Shree Nayar Wavefront Coding using Cubic Phase Plate "Wavefront Coding: jointly optimized optical and digital imaging systems“, E. Dowski, R. H. Cormack and S. D. Sarama , Aerosense Conference, April 25, 2000 Courtesy of Shree Nayar. Used with permission. Slide by Shree Nayar Depth Invariant Blur Conventional System Courtesy of Shree Nayar. Used with permission. Wavefront Coded System Slide by Shree Nayar Decoding depth via defocus blur • Design PSF that changes quickly through focus so that defocus can be easily estimated • Implementation using phase diffractive mask (Sig 2008, Levin et al used amplitude mask) Typical PSF changes slowly Images removed due to copyright restrictions. Phase mask Image removed due to copyright restrictions. Designed PSF changes fast Images removed due to copyright restrictions. R. Piestun, Y. Schechner, J. Shamir, “Propagation-Invariant Wave Fields with Finite Energy,” JOSA A 17, 294-303 (2000) R. Piestun, J. Shamir, “Generalized propagation invariant wave-fields,” JOSA A 15, 3039 (1998) Rotational PSF Images removed due to copyright restrictions. Two sequences showing rotating and standard PSF. R. Piestun, Y. Schechner, J. Shamir, “Propagation-Invariant Wave Fields with Finite Energy,” JOSA A 17, 294-303 (2000) R. Piestun, J. Shamir, “Generalized propagation invariant wave-fields,” JOSA A 15, 3039 (1998) Single Pixel Camera Several slides removed due to copyright restrictions. Slides by Shree Nayar MIT OpenCourseWare http://ocw.mit.edu MAS.531 / MAS.131 Computational Camera and Photography Fall 2009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
Related documents