Download Computational photography with plenoptic camera and light field

Tutorial
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
2021
Computational photography with plenoptic
camera and light field capture: tutorial
EDMUND Y. LAM
Department of Electrical and Electronic Engineering, The University of Hong Kong, Pokfulam, Hong Kong, China (e-mail: [email protected])
Received 14 July 2015; revised 7 September 2015; accepted 10 September 2015; posted 10 September 2015 (Doc. ID 245927);
published 14 October 2015
Photography is a cornerstone of imaging. Ever since cameras became consumer products more than a century ago,
we have witnessed great technological progress in optics and recording mediums, with digital sensors replacing
photographic films in most instances. The latest revolution is computational photography, which seeks to make
image reconstruction computation an integral part of the image formation process; in this way, there can be new
capabilities or better performance in the overall imaging system. A leading effort in this area is called the plenoptic
camera, which aims at capturing the light field of an object; proper reconstruction algorithms can then adjust the
focus after the image capture. In this tutorial paper, we first illustrate the concept of plenoptic function and light
field from the perspective of geometric optics. This is followed by a discussion on early attempts and recent
advances in the construction of the plenoptic camera. We will then describe the imaging model and computational
algorithms that can reconstruct images at different focus points, using mathematical tools from ray optics and
Fourier optics. Last, but not least, we will consider the trade-off in spatial resolution and highlight some research
work to increase the spatial resolution of the resulting images. © 2015 Optical Society of America
OCIS codes: (110.1758) Computational imaging; (110.2990) Image formation theory; (110.5200) Photography; (110.3010) Image
reconstruction techniques; (100.3020) Image reconstruction-restoration; (100.6640) Superresolution.
http://dx.doi.org/10.1364/JOSAA.32.002021
1. INTRODUCTION
A. Computational Optical Imaging
The world is rich in information. The goal of imaging is to
record, process, and display such information. Before the advent of digital sensors, what we strove to record on film was
often a faithful 2D projection of a scene or an object; except
perhaps for some darkroom techniques, such as burning and
dodging, to adjust the exposure, the photographic print mirrored what was captured on the film.
The popularization of digital sensors, including both CCD
and CMOS, together with the processing power in the accompanying electronics, has brought about a paradigm shift in imaging. On one hand, it is possible to consider the digital sensor
merely a replacement of the analog detector (film), keeping essentially intact the other components of the imaging pipeline. This
was the conventional wisdom when digital sensors began to enter
the mainstream in the 1990s. On the other hand, a much more
powerful question to ask is the following [1]: What kind of information should we record on the digital sensor so that, with suitable
processing, we can reconstruct an image that may have better quality or there may be new functionality in the imaging system?
Accordingly, this extra flexibility has three dimensions: the
imaging hardware—primarily the optics—to decide what to
1084-7529/15/112021-12$15/0$15.00 © 2015 Optical Society of America
capture, the computational algorithm to process the data,
and the application of such a computational imaging system.
It is critical to note that images only emerge from computing on
the information recorded; as such, the captured data do not
necessarily have to be the 2D projection of the scene or the
object, or anything visually identifiable at all. In fact, for
different applications, there is a lot of room to investigate what
to capture and to devise efficient algorithms for the image
reconstruction [2–4].
B. Computational Photography
Photography is likely the most common form of optical imaging. From pinhole image formation, which was first described
in writing during the Spring and Autumn Period in China
(roughly from the eighth century B.C. to the fifth century
B.C.) [1], to the first permanent photograph by the French
inventor Joseph Nicéphore Niépce (the nineteenth century),
photography has come a long way as an established technology
and art form. With computation, how can we experience
photography differently?
One answer, which is the focus of this tutorial, is light field
photography [5]. Conventionally, whether with films or digital
sensors, what is recorded is the amount of light, irrespective of
its direction. Light field photography attempts to use a new
2022
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
imaging system design, called a plenoptic camera [6], that can
indeed discriminate and capture light from a few directions.
Together with proper computational algorithms, capturing
the light field gives us at least three advantages over the traditional camera [7]: to reconstruct images with small viewpoint
changes, to compute a depth map using depth from focus, and
to reconstruct images as if they have been focused at different
planes, thus achieving refocus after data capture.
There has been significant interest in and development of
light field photography over the past decade, including the deployment of some commercial products, and it is now an opportune time to review the technology behind them. Here, we
will do so primarily for the benefit of one with a background in
geometric and Fourier optics.
Tutorial
design of a portable light field camera system. In explaining
this design, we will go in depth with a spatial and a Fourier
analysis of the image formation process. Up to this point, the
effect of diffraction is ignored, so we can use the simpler representation of light rays. Section 4 goes beyond this and relates
light field to the Wigner–Ville distribution under the scalar diffraction theory. The image formation process is also explained
using Fourier optics. Whether using geometric or Fourier optics, we will see there is a trade-off between angular and spatial
information, and providing a higher-resolution 2D image from
the light field is an important issue. Section 5 is devoted to
highlighting recent advances in light field photography, particularly in optical and digital superresolution.
C. Other Computational Imaging Systems
2. PLENOPTIC FUNCTION AND LIGHT FIELD
Before we delve into computational photography for the rest of
this paper, it is worth pointing out that computation has also
been explored for many forms of imaging, notably for machine
vision inspection and biomedical applications. For instance,
structured illumination with phase-shifting algorithms allows
for high-resolution height measurement of the object under
inspection [8–10]. Phase diversity is a powerful technique that
estimates wavefront aberrations based on multiple captured images, which are mainly used to reconstruct a clear 2D object,
but this principle can also be applied to 3D imaging [11]. In
computed tomography, the captured data are Fourier transformed to form slices of the object’s spectrum; a reconstruction
algorithm, whether it is filtered backprojection or a more advanced iterative method, delivers detailed images of the internal
structures of a person [12,13]. A similar approach is also taken
with modalities such as diffuse optical tomography (DOT)
[14,15]. Meanwhile, optical coherence tomography (OCT),
which is a popular bio-optical imaging modality based on
Michelson interferometry, also benefits substantially from
computation. A form known as spectral-domain OCT collects
the data uniformly in wavelength and ordinarily requires resampling and Fourier transform to reconstruct the images [16,17].
Digital holographic microscopy, by its very nature, requires
computational processing to deliver a visible image. The phase
of the wavefront is converted as a modulation in the magnitude
during the recording step; thus, a computational process
is needed to reconstruct, for example, quantitative phase
information or sectional images of the object [18,19]. These
are representative of the emerging subfield of computational
biophotonics [20].
A. Theory
D. Organization
This paper is organized as follows. In Section 2, we will explain
the theory behind plenoptic function and light field, from the
perspective of geometric optics. A ray-space representation will
then be introduced, which helps us discuss the image formation
process. The main purpose is to show the desirability of being
able to capture the light field. Section 3 is then mostly concerned with camera implementation, which has the ability
to discriminate light rays from several directions and therefore
provides a sampling of the incoming light field. We motivate
this from the historical perspective, leading to the first successful
Let us assume that we are dealing with incoherent light traveling around objects that are much larger than the wavelength
of light. This allows us to use “rays”—parameterized chiefly by
their locations and directions—to describe the image formation
process. This is known as ray optics or, more commonly, geometrical optics because light propagation obeys a set of geometrical rules [21].
Specifically, in a homogeneous medium (such as free space),
light rays propagate in straight lines. For the purpose of quantifying them and using them for computations, we need a
mathematical function to describe each ray. In an attempt to
understand the fundamental substances of human and machine
vision, Adelson and Bergen in the early 1990s devised the plenoptic function to capture the information carried by the light
rays [22]. The result is a 7D function,
Pθ; ϕ; λ; t; x; y; z;
where θ; ϕ are angular coordinates encoding the direction of
the ray, λ is the wavelength, t is time, and x; y; z are the spatial
coordinates of a point this ray passes through. This point is
taken as the origin of the ray.
The word “plenoptic” comes from “plenus” in Latin, meaning complete or full [22]. Thus, does the plenoptic function
indeed contain complete information about an object? On
one hand, we should remember that we are in the realm of
geometrical optics; therefore, information regarding initial
phase is absent in the plenoptic function. On the other hand,
for our human visual system, what we observe are normally
samples and projections of the plenoptic function. Arguably, we
do not “see” an object directly; rather, our retina responds to the
light rays emanated or reflected from such an object; therefore,
the plenoptic function encapsulates the full information communicated to the observer.
In fact, in the study of image formation, we often can reduce
the dimensionality of the plenoptic function by arguing the
following:
1. The function does not vary with time (so t can be
omitted).
2. The light is monochromatic (so λ can be omitted).
3. The radiance along a ray is a constant (so only two
parameters are needed for the location, effectively omitting z).
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
Furthermore, it is common to replace the angular coordinates θ; ϕ with Cartesian coordinates u; v. As depicted in
Fig. 1, we assume that all light rays travel from left to right
and pass by two planes along the way. A specific light ray intersects the first plane at coordinates u; v, which govern the
location of the ray; it continues to travel and intersects
the second plane at coordinates x; y, effectively specifying the
propagation direction.
Instead of calling this the plenoptic function, it is more
common, thanks to the work of Levoy and Hanrahan [23],
to refer to this 4D simplification as the light field, denoted
2023
represented by a point in the ray-space diagram. We can visualize a light field as a collection of such points.
Figure 2 shows four illustrative situations, each corresponding to a discrete set of light rays:
Lu; v; x; y;
which is also called the “lumigraph” in the computer graphics
community [24]. Note that the term “light field” was first used
in a translated work on illumination engineering in the 1930s
[23,25], but this is not in the sense of an electromagnetic field,
which underlines the principles of optics [26]. The value of
Lu; v; x; y is the amount of light—known as the radiance—
of the monochromatic light ray. When considering color image
formation, we can assume that the light field is a threecomponent vector with radiance information on the red, green,
and blue components or any corresponding wavelengths.
In a typical camera, the two-plane parameterization may be
applied where the first plane is the position of the main lens and
the second plane lies in the electronic sensor. This,
unfortunately, would not allow us to capture the light field;
as the photodetectors respond to light from all directions,
the angular information of the rays is lost. In Section 3, we
will discuss a setup known as the plenoptic camera, with a
somewhat more complicated and interesting geometry, that
can achieve this. Before that, however, let us now turn our attention to a diagrammatic representation of the light field.
(a)
(b)
B. Ray-Space Representation
An exemplary way to plot the light field is to use the so-called
ray-space diagram [5,27,28], which allows us to gain insight
about different collections of light rays in a compact manner.
For the purpose of illustration, we retain only one coordinate in
each of the planes in Fig. 1. The light field is then represented
as a 2D diagram, which is a plot of x, the spatial axis, versus u,
the directional axis. Furthermore, consider the light ray Lu; x:
it emanates from a specific u to a specific x and therefore is
(c)
(d)
Fig. 1. Two-plane parameterization of the light field [23].
Fig. 2. Illustrative plots of the ray-space diagram. (a) A regular array
of light rays, from a set of points in the u plane to a set of points in the
x plane. (b) A set of light rays arriving at the same x position. (c) A set
of light rays approaching a location behind the x 0 plane. (d) A set of
light rays diverging after converging at a location before the x 0 0 plane.
2024
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
1. As shown in Fig. 2(a), the light rays emanate from a set
of regularly spaced u to another set of regularly spaced x.
Accordingly, the ray-space diagram consists of a regular matrix
of points.
2. As shown in Fig. 2(b), a collection of rays at different
points on the u plane focusing onto the same point in the x
plane become, in the ray-space, a vertical line.
3. As shown in Fig. 2(c), a collection of rays at different
points on the u plane converge to a point at the x plane; thus,
on the x 0 plane, which is nearer, these rays are getting closer, but
their spatial locations are still different. In the ray-space diagram, these become a line tilted to the right. Another way
to describe the phenomenon is that the ray-space has undergone a shearing. This observation is fundamental to plenoptic
camera, and we will return to it below.
4. As shown in Fig. 2(d), a collection of rays at different
points on the u plane converge to a point at the x plane; thus,
on the x 0 0 plane, which is farther, these rays are again diverging
to different spatial locations. Similar to the discussion above, in
the ray-space diagram, they form a line tilted to the left; equivalently, the ray-space is also sheared but to a different direction.
Tutorial
(a)
C. From Light Field to Image
Suppose now we put a photodetector in the x; y plane. It
captures the irradiance of the light, which is a summation of
the incident radiance [29,30]; mathematically, this means we
integrate the light field, i.e.,
ZZ ∞
1
Lu; v; x; yT u; vdudv;
(1)
Ex; y 2
Z
−∞
where Ex; y is the resulting image intensity and Z is the separation between the u; v and x; y planes. The term T u; v
represents the aperture, which for a circular lens with radius R
can be modeled as a circ function [31,32]:
1 if u2 v2 < R 2
T u; v circu; v (2)
0 otherwise:
More precisely, there should be a cos4 θ term inside the integral,
where θ is the angle between the light ray and the normal to the
u; v and x; y planes; this is sometimes known as the cosine
fourth law, representing the physical effect that rays from more
oblique directions contribute less energy to the total irradiance
[33]. This term, however, can be conveniently absorbed into
the definition of the light field [5]. Equation (1) thus provides
the linkage between the 4D light field and the 2D image capture in a conventional camera: the image is a projection of the
light field onto the x; y axes.
Let us now return to the cases depicted in Figs. 2(b)–2(d).
Assume that we have placed a lens in the u plane; we can converge the light rays to a point in the x plane, as depicted in
Fig. 2(b), if it is placed in the focal plane of the lens, with the
use of the paraxial approximation. Now suppose we bring the x
plane closer to the u plane. From Fig. 2(c), we know this will
result in a tilted line in the ray-space; in addition, we have the
necessary mathematics to quantify the angle of the tilt, as
follows.
Consider the geometry depicted in Fig. 3(a), where we have
shrunk the distance between the two planes from Z to αZ . The
ray intersects the new plane at x 0 , where x 0 − u αx − u, and
therefore,
(b)
Fig. 3. Bringing the x plane closer to the u plane results in a tilted
line in the ray-space at an angle ψ. (a) Moving the second plane closer
to the first, by a factor of α. (b) Corresponding shearing in ray-space,
with ψ tan−1 1 − α.
x0 − u
:
α
Thus, this ray can be expressed as
x0 − u
y0 − v
;v :
Lu; v; x; y L u; v; u α
α
x u
(3)
(4)
We want to depict this in the ray-space diagram with x 0 and u.
From Eq. (3), we can also express x 0 in terms of x such that
x 0 αx 1 − αu:
(5)
In the ray-space diagram with x 0 and u, this is a straight line
with slope 1∕1 − α. We normally define the shear angle ψ as
the angle with the vertical line, which is
π
1
−1
tan−1 1 − α:
(6)
ψ − tan
2
1−α
This is depicted in Fig. 3(b).
Because we are concerned with light rays only, matrix optics
offers an alternative to computing the tilting angle [34]. Let us
refer back to Fig. 3(a). Now consider two light rays, one starting
at position u1 , emanating at an angle θ1 , passing through x 10 ,
before arriving at x; similarly, the second ray starts at u2 with
an angle θ2 , passing through x 20 , before arriving at the same
x. With the ray-transfer matrices [21], which make use of the
paraxial approximation, we can write
0 1 Z
x
1 αZ
x1
u1
;
(7)
0 1 θ1
0 1
θ1
θ1
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
u2
θ2
1 Z
0 1
x
θ2
1
0
αZ
1
x 20
:
θ2
(8)
Therefore,
u1 x Z θ1 x 10 αZ θ1 ;
(9)
u2 x Z θ2 x 20 αZ θ2 :
(10)
Δx 0
x 20
x 10 ,
−
then, taking the
If we define Δu u2 − u1 and
difference between the above two equations, we have
u2 − u1 Z θ2 − θ1 x 20 − x 10 αZ θ2 − θ1 ;
(11)
which means
Δu Δx 0 αΔu:
(12)
Referring back to Fig. 3(b), the tilting angle is controlled by Δu
and Δx 0 , where
0
Δx
tan−1 1 − α;
ψ tan−1
(13)
Δu
arriving at the same result as in Eq. (6).
We can now summarize the process of forming a 2D image
from a light field.
• Capturing an image is essentially integrating the light field
over the angular variables u; v, as Eq. (1) suggests; in the rayspace representation, this is projected onto the horizontal axis.
• If we adjust the placement of the electronic sensor, the
light field will undergo a shear; bringing the sensor closer to
the lens will result in a shear to the right by an angle ψ given
by Eq. (6); putting the sensor farther away will have the opposite effect.
• If x; y represents the ideal focusing plane with respect to
an object, then moving the sensor to another plane x 0 ; y 0 means we first shear the light field and then still project to
the horizontal axis. We know by experience that this gives a
blurred image.
Consequently, if we are able to capture the light field itself
instead of its projection, we can reconstruct what it would look
like at any plane by shearing it to a different extent and projecting it onto the horizontal axis or, alternatively, simply projecting the light field at various angles [35]. But how would we
capture the 4D light field using a 2D sensor? It turns out that
answering this question has a history that dates back to over a
century ago.
3. PLENOPTIC CAMERA SYSTEMS
A. Early Developments
The first attempt to capture the light field is traced to the work
of Gabriel Lippmann, the 1908 Nobel Laureate in physics for
his invention of color photography based on interference. In
that same year of his award, he also published a paper describing a method known as “photographies intégrales”—i.e., integral photography [36]. His chief concern was the development
of a stereoscopic display, but he needed a way to record an object for this purpose. He made use of an array of small lenses,
with images forming on the film behind the lenses, each having
a slightly different perspective. Later on, with backlit diffused
light, one can view these subimages together with a stereoscopic
2025
view. We now understand that, taken as a whole, the recorded
images in Lippmann’s scheme preserve a sampling of the light
field of the object, but the image formation process was not
studied in this way during his days.
Unfortunately, Lippmann’s idea did not take off for many
years, primarily because of the technical difficulties in creating
the lens array. (These days, though, with the widespread availability of high-resolution cameras and spatial light modulators,
integral imaging has become quite possible and is an active research area [37,38].) In the 1930s, Herbert Ives, a charter
member and past president of OSA, improved on Lippmann’s
design in two aspects. First, he added a main lens with a large
diameter [39]. Second, he analyzed the original approach and
found it to be pseudoscopic rather than stereoscopic. He then
proposed a secondary exposure to invert the depth [40]. It was
not until a few decades later that Chutjian and Collier developed a one-step imaging solution and made possible the first
computer-generated integral imagery [41].
The next significant phase of the development emerged
primarily from the computer vision and computer graphics
communities. Soon after defining the plenoptic function [22],
Adelson and Wang outlined an image capture system to collect
such data and named it the plenoptic camera [42]. Their objective was to infer the object depths, achieving what is known
as a single lens stereo. The “single lens” here refers to the single
main lens; in addition, there is a lenticular array (alternatively,
pinhole array) in front of the sensor, so that the data recorded
on the photodetectors correspond to images from slightly different perspectives. With these subimages, the authors were
able to produce depth maps and reconstruct images from different viewing positions [42].
Later, after the “light field” was defined as a simplification of
the plenoptic function, it was demonstrated that a camera array
could record the light field of an object [43]. Similar to the
above, one use of the data was to reproduce images from various
viewpoints, a method known as image-based rendering [44].
Another use was to simulate defocus blur, thus synthesizing
(or reconstructing) images as if they were captured with a large
aperture; this technique is called synthetic aperture photography [45]. The camera array, however, was very large [46,47]
and therefore not portable, thus limiting its use to a more controlled environment.
A very different idea from those noted above is the development of specialized sensors. In particular, recent advances in
complementary metal oxide semiconductor (CMOS) processes
have made angle-sensitive photodetectors possible [48]; these
photodetectors use pixel structures based on the Talbot effect
to permit the capturing of angular light radiance [49,50].
One additional line of technological developments should
also be mentioned here, although the purpose of these innovations was not to capture the light field. Instead, the researchers
were motivated to learn from nature and, thus, explored the
compound eye—commonly found in many insects—as an
alternative form of imaging systems [51]. Such an artificial
compound eye was realized using a microlens array in front of
the digital sensor [52,53], and developments in the design and
fabrication of these microlenses would prove to be useful for a
portable light field camera later on. Representative work in
2026
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
compound-eye imaging includes Tanida et al. [54,55], who
built the TOMBO system, short for thin observation module
by bound optics (the name TOMBO also means “butterfly” in
Japanese); Chan et al. [56–58], who introduced diversity in the
captured subimages and developed a superresolution algorithm
for the image reconstruction; and Plemmons et al. [59], who
built the complete optical hardware and software systems. They
also studied the theoretical and practical performance of the
resulting prototype called PERIODIC (practical enhancedresolution-integrated optical-digital imaging camera) [60]. A
major advantage was that one could achieve a very thin imaging
system; unfortunately, the image quality was often not comparable with that of a traditional camera system with a main lens
due to diffraction effects [5].
B. Portable Light Field Camera
1. Placement of the Lenses and Photodetectors
The first design of a portable light field camera can be thought
of as a far-field version of integral imaging [61], a miniaturization of the camera array system [43], or a hybridization of
simple and compound eyes. The optical system of such a camera is shown in Fig. 4. There are four planes of interest here:
1. The object plane, labeled ζ; η.
2. The aperture plane, which is also the location of the exit
pupil. We label it the u; v plane.
3. The imaging plane, with a regular matrix of microlenses. The size of the individual microlens determines the spatial sampling resolution [5]. We label it the x; y plane.
4. The photodetector array plane, labeled σ; τ.
The key to an effective light field capture is the precise positioning of these planes [6,62].
First, let us consider the red lines in Fig. 4, which trace the
light rays all the way from the object to the photodetector. The
object and the main lens are separated by a distance z 0 , and the
main lens and the lenslet array are separated by a distance z 1 . As
the figure illustrates, the lenslet array is placed where the object
is brought to focus; in other words, for a conventional camera,
in lieu of the microlenses, we would have the photodetectors
there in order to obtain a sharp image. Thus, if the main lens
has a focal length f 1 , then
Tutorial
1
1
1
;
z0 z1 f 1
(14)
thus obeying the well-known lens equation.
In this plenoptic camera, the photodetector array is placed a
certain distance z 2 behind the lenslet array. Accordingly, the
light rays that converge at a microlens will again diverge and
impinge on the photodetectors at different locations. The recorded image then provides directional resolution of the object.
To maximize this, we should ensure the sharpest microlens images, which is achieved by focusing the microlenses on the principal plane of the main lens [6]. This is illustrated by the blue
lines in Fig. 4. Because the microlenses are very small compared
with the main lens, z 1 is effectively at optical infinity.
Consequently, if the microlenses have a focal length f 2 , then
z2 f 2:
(15)
Furthermore, we must take care to ensure that the light rays
passing through one microlens should sufficiently cover the
photodetectors behind it, so we do not waste the pixels. On
the other hand, to avoid cross talk, the light rays also should
not extend to the photodetectors associated with the adjacent
microlenses. This can be achieved if the image side f -number
of the main lens (its diameter, d 1 , divided by z 1 ) and the
f -number of the microlenses (their diameters, d 2 , divided
by z 2 ) are equal [5], i.e.,
d1 d2
:
(16)
z1
z2
These specific optical configurations would now allow us to
capture a filtered and sampled light field of an object with the
photodetector array: the sampling is because there are only a
finite number of pixels and microlenses, and the filtering is
due to their finite dimensions. There are two complementary
perspectives to see why this is the case. First, associate the region on the photodetectors behind each microlens as a subarray
image and treat each microlens as a virtual pixel; the subarray
image then captures the directional information of the light rays
corresponding to that pixel. Mathematically, the subarray image
is a collection of the light field Lu; v; x; y with different u; v
but the same x; y. Second, consider the two sets of blue lines
in Fig. 4. They refer to two pixels on the photodetector array
that correspond to the same location in the aperture. We
can refer to the collection of such corresponding pixels as a
subaperture image. Mathematically, the subarray image is a collection of the light field Lu; v; x; y with the same u; v but
different x; y. Therefore, taken together, the photodetectors
are sampling the light field from the object. (Readers interested
in experimenting with light field data and visualizing these different image representations can download publicly available
light field data sets, such as from the Stanford Light Field
Archive, or create their own data using plenoptic cameras that
are now commercially available.)
2. Spatial Analysis of Image Formation
Fig. 4. Light field camera system. Light rays marked in red show
how the microlenses separate them, so the photodetector array can
capture a sampling of the light field. Light rays marked in blue show
that the photodetector array can also be thought of as recording the
images of the exit pupil.
Because the light field of an object is captured in the photodetectors in Fig. 4, we can make use of the discussions in
Section 2.C to compute a 2D image from such data. The simplest way is to sum up the values from the photodetectors in
each subarray image, which gives the intensity of the virtual
Tutorial
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
pixel at each microlens position; taking these virtual pixel values
together would form an image of the object, albeit often at a
low resolution. This corresponds to the case depicted in
Fig. 2(b).
However, a major advantage of recording the light field, as
opposed to the intensity image, is the ability to support postcapture refocus of the object [5,62]. Consider the situation
shown in Fig. 2(c), where we are recording the light field
on the x 0 plane. We know from the earlier discussion that
this light field Lu; x 0 is essentially the same as the light field
Lu; x after undergoing a shearing operation. Thus, if we first
compute the angle ψ using Eq. (6) and then project Lu; x 0 at
this angle, as illustrated in Fig. 5, we can reconstruct an image
as if the photodetectors were placed at the (x) plane. Overlaying
this analysis on Fig. 4, we can see that this tilted summation of
the captured light field can lead to an image as if it were taken
directly with a sensor a certain distance behind the lenslet array
plane. This increases the image distance and, consequently,
shortens the object distance, as a result of the lens equation.
Similarly, the situation depicted in Fig. 2(d) shows that we
can likewise reconstruct an image as if it were taken directly
with a sensor some distance in front of the lenslet array plane,
corresponding to focusing at the part of the object that is farther
from the main lens.
Figure 6 shows an example of post-capture refocus from a
light field captured directly in a commercial plenoptic camera
[6,63]. A set of tools has been made available to decode the
light field data and compute the image reconstruction with
different parameters [64,65].
A byproduct of such post-capture refocusing is the ability to
produce an extended depth-of-field (EDOF) image, which is
desirable in many applications [66–68]. In principle, each of
the subaperture images already has a higher depth of field compared with an ordinary photograph using the same main lens.
This is because the subaperture image, as the name suggests,
uses only a smaller part of the aperture, which leads to a larger
effective f -number. However, the signal-to-noise ratio (SNR)
is usually quite poor because the majority of the captured data is
discarded. A better alternative is to refocus the image on a pixel
basis, selecting the focus point to be on the depth of the closest
object. This can be achieved by reconstructing a stack of images
corresponding to different sensor locations and making use of a
digital photomontage algorithm to combine them to form a
single EDOF image [5,69].
Fig. 5. Tilted projection of the light field.
2027
Fig. 6. Example of post-capture refocus from custom light field
data. (a) Image reconstruction with focus at the front. (b) Image
reconstruction with focus at the back.
3. Fourier Analysis of Image Formation
The above discussion suggests that image reconstruction is
achieved by projecting the 4D light field onto a 2D space at an
angle that determines the focus distance of the resulting image.
This projection can be computed using numerical integration,
by identifying which photodetectors lie along the “line of projection” and summing them up accordingly. However, this
process often involves computing the appropriate weights of
the photodetector values and may even include resampling
of the data, which can be substantial with a large light field.
In this section, we explain how this projection of the light field
can alternatively be achieved in the Fourier domain, which can
lead to asymptotically lower complexity algorithm and further
insight into the imaging process. The method is called Fourier
slice photography [70].
The mathematical basis is the projection-slice theorem by
Bracewell [31,71]. In two dimensions, this well-known theorem states that a projection, i.e., line integration, of a 2D function along an angle ϕ is related to a slice, again at angle ϕ, of the
2D Fourier transform of the function through a 1D Fourier
transform. This is represented diagrammatically in Fig. 7. The
generalization to higher dimensions is rather straightforward,
and we will be especially concerned with projecting 4D
2028
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
reasonable to ask, then, if we can arrive at a similar conclusion
if we consider light as electromagnetic waves. Specifically, what
may be the effect of diffraction? This question is particularly
pertinent if we would like to scale down the dimension of
the plenoptic camera, such as in the construction of the light
field microscope [76–78]. It has been pointed out that the light
field can be identified as the phase-space of geometric optics
[79], and the Wigner–Ville distribution can then be used to
generalize the phase-space formalism. More specifically, we will
show how to relate the light field to the Wigner–Ville distribution, and we can also develop Fourier optics expressions
for the plenoptic camera system.
Fig. 7. Illustration of the projection-slice theorem. Projection at
angle ψ of a 2D function in the x − u plane, which becomes a 1D
function in ρ, is related to its 2D Fourier transform in the f x − f u
plane sliced at angle ψ by a 1D Fourier transform.
functions to 2D [72]. Remember that our interest is in a tilted
projection of the light field; the main result from Fourier slice
photography is that such image refocusing, in the Fourier
domain, is simply obtained by slicing the 4D Fourier transform
of the light field at different angles [70,73,74].
There is, however, an additional mathematical detail regarding the shearing operation. It involves a basis transform that is
not orthonormal [75]. Observe from Eq. (5) that
2u3 2 1
0
0 0 32 u 3
6v7 6 0
4 05 4
x
1−α
y0
0
1
0
1−α
0
α
0
and therefore the basis transform matrix
2 1
0
0
1
0
6 0
B4
1−α
0
α
0
1−α 0
0 76 v 7
54 5;
x
0
y
α
03
(18)
Thus, we need to generalize the projection-slice theorem to include such basis transform. Specifically, Ng showed that the
Fourier domain would need a change of basis involving the normalized inverse transpose of B, which is [70]
21 0 1 − 1
0 3
B −T
60
α2 4
0
jB −T j
0
1
0
0
0
1
α
0
α
1 − α1 7
5:
0
We make use of the scalar diffraction theory, which is accurate as long as the diffracting structures are large when compared with the wavelength of light [32]. For simplicity,
we also limit the discussion to monochromatic waves, where
the extension to polychromatic light with different degrees
of coherence is possible, though slightly more involved
[80,81].
Let us now consider a scalar field at a plane s; t and denote
it, in phasor notation, as Us; t. The 4D Wigner–Ville distribution (often simply called the Wigner distribution) is given by
[82,83]
ZZ
W U s; t; ξ; ν (17)
is
07
5:
0
α
A. Wigner–Ville Distribution
(19)
1
α
With this, the projection of the light field is computed in the
Fourier domain by first applying the full 4D Fourier transform
to the light field data. Then, the basis transform using Eq. (19)
is computed on the light field spectrum. We then extract a 2D
slice of the signal, which corresponds to setting the other two
dimensions to zero. Finally, we calculate an inverse 2D Fourier
transform on the slice. The result, by the projection-slice theorem, is equivalent to the direct projection of the light field data.
4. WAVE OPTICS
Thus far, we have analyzed the plenoptic imaging system
entirely with geometric optics by tracing the light rays. It is
∞
−∞
×e
α
β
α
β
U s ; t U s − ; t −
2
2
2
2
−j2παξβν
dαdβ;
(20)
where ξ; ν are spatial frequency coordinates. Our goal is to
relate the light field representation with this Wigner–Ville distribution. Unfortunately, this cannot be done directly because
light must propagate isotropically with an infinitesimal pinhole
[80]. Thus, while a light field can specify a precise direction,
with Wigner–Ville distribution, we must take into account a
range of spatial frequencies. The trick is to insert a finite aperture; moving it across the plane results in a generalization
known as the observable light field [80]. (The “augmented light
field” is another possible generalization of the light field and is
used in computational photography analysis as well; for details,
we refer readers to [84].)
This methodology is depicted in Fig. 8. Assume an aperture
function T s; t is translated to be centered at u; v, so the
scalar field after the aperture is
Ũs; t Us; tT s − u; t − v:
(21)
Its angular spectrum, which is the Fourier transform of the scalar field [32], is given by
ZZ
∞
y
x y
x
Us;tT s − u;t − ve −j2πλsλt dsdt:
A ; λ λ
−∞
(22)
The observable light field [from the point u; v at direction
x; y, denoted by Lobs u; v; x; y] is given by the magnitude
squared of the angular spectrum [80]; thus,
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
2029
Next, we consider what happens after passing through the
lenslet array. Assume there are N × N identical microlenses. As
each has diameter d 2 and focal length f 2 , the propagations
after the lenses are given by [32]
Ũx; y Ux; y
Z Z Z Z
π
× e −jλf 2 x−md 2 y−nd 2 ;
(23)
Us1 ;t 1 U s2 ;t 2 T s1 −u;t 1 −v
−∞
y
x
×T s 2 −u;t 2 −ve −j2π λs1 −s2 λt 1 −t 2 ×ds1 dt 1 ds 2 dt 2
(24)
x y
x y
W U u; v; ;
W T −u; −v; ; ; (25)
λ λ
λ λ
where we arrive at the last line by rewriting the integration
variables into averages and differences, as given in [80]. We
can therefore see that the observable light field is the smoothed
Wigner–Ville distribution, where the blur is given by a
Wigner–Ville distribution of the aperture. This trades off resolution in position with direction: the smaller the aperture, the
more precise we can be about the position, but the less definite
we are with the spatial frequency. Furthermore, when a wavelength becomes negligibly small (bringing us back to the domain of geometric optics), the Wigner–Ville distribution of
the aperture is a delta function, and the observable light field
is then equivalent to the Wigner–Ville distribution of the wave
function.
B. Image Formation
We can also derive wave optics expressions for the plenoptic
camera image formation process. For simplicity of discussion,
we consider the setup as given in Fig. 4 (expressions with more
general distance values can be found in [85]).
Let the scalar field at the object plane be denoted as Uζ; η.
The propagation from the object plane to the lenslet array follows that of a standard imaging setup, with the latter positioned
at the normal “image plane.” Thus, we can readily know that
the impulse response of this part of the imaging system is given
by the Fraunhofer diffraction pattern of the main lens pupil,
P 1 u; v, i.e., [32]
ZZ ∞
2π
P 1 u; ve −jλz 1 x−M ζuy−M ηv dudv;
h1 x; y; ζ; η K
−∞
(26)
where K is a constant and M is the magnification of the system
given by M −z 1 ∕z 0. The scalar field in front of the lenslet
array is then
ZZ ∞
h1 x; y; ζ; ηUζ; ηdζdη
(27)
Ux; y −∞
2
2
(28)
where P 2 x; y is the pupil function of the microlens. Finally,
the propagation from the lenslet array to the photodetector
array is given by a Fresnel propagation with impulse response
2π
∞
because of linearity of wave propagation.
P 2 x − md 2 ; y − nd 2 m1 n1
Fig. 8. Observable light field.
2
x y Lobs u;v;x;y A ; λ λ
N X
N
X
e j λ z 2 λzjπ σ2 τ2 e 2
;
h2 σ; τ jλz 2
and the scalar field at the photodetector array is
ZZ ∞
h2 σ − x; τ − yŨx; ydxdy;
Uσ; τ (29)
(30)
−∞
again because of superposition [32].
To reconstruct an image, a possible method is to backpropagate the field from the photodetector array to an arbitrary
object plane. Reference [86] describes how this can be done
with coherent and incoherent approaches.
5. LIGHT FIELD SUPERRESOLUTION
Whether we analyze the plenoptic camera system using rays or
waves, it is evident that the imaging system trades off spatial
resolution with angular information in the light field [87].
This is because it takes a collection of captured pixel values together to reconstruct the intensity at a specific location, due to
the projection operation described in Section 3.B.2. For example, one of the first consumer light field cameras is equipped
with a sensor having 11 million photodetectors, capturing “11
mega-rays,” but typically can only reconstruct an image under 1
megapixel [63].
To tackle this problem, various methods have been developed, which are aimed to achieve better resolution from a
plenoptic camera. We can, for example, capture more light field
data by multiplexing many views, an idea explored in the
programmable aperture camera [88]. This technique, however,
requires a long acquisition time, and the object needs to be
static. Combining a light field camera with a traditional camera
is also a possibility [7]. Alternatively, without increasing the
capture data volume, there are still broadly two major approaches: with changes in the way light field data are captured
and with more powerful image superresolution algorithms.
Below, we highlight the representative methods.
A. Optical Methods
The major limitation to spatial resolution is the physical dimensions of the microlens array. In addition, the microlenses require precise alignment and still may introduce errors such
as spherical aberration and coma as well as chromatic aberration. A very different method of capturing the light field is to
replace the microlens array with an attenuation mask placed
between the main lens and the sensor [74]. This design, known
as the dappled photography camera system, is shown in Fig. 9.
2030
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
The attenuation mask, which consists of a high-frequency
sinusoidal pattern, serves the purpose of heterodyne encoding
of the light rays onto the photodetector. In other words, instead
of recording each light ray with a pixel, dappled photography
captures a linearly independent weighted sum of light rays,
combined in a coded fashion, which can be decoded in the image reconstruction process [74]. A major advantage of this design, compared with the plenoptic camera system in Fig. 4, is
that it can reconstruct a high-resolution image at the plane in
focus. This comes at a price, however; the signal-to-noise ratio
(SNR) is further reduced because of the light attenuation at the
mask [89].
An alternative plenoptic camera system also exists with the
use of a lenslet array but with different placement of the optical
elements. Figure 10 shows such a design known as the focused
plenoptic camera [90] (the authors also call it the plenoptic 2.0
camera [91]). Recall that, in the plenoptic 1.0 camera (i.e.,
Fig. 4), the object is imaged at the microlens array plane,
and the microlenses focus at the main lens, which is effectively
at optical infinity. In contrast, with the focused plenoptic camera, the object is imaged at a plane in front of the lenslet array,
and the microlenses then focus on this image, producing a relay
system to reimage it onto the sensor. This means that the
photodetector has been moved farther from the main lens.
Remember that f 2 is the focal length of the microlenses; then
1
1
1
;
z 1B z 2 f 2
(31)
where z 1B is the distance between the main lens image plane
and the lenslet array and z 2 is the distance between the latter
and the photodetector, as shown in Fig. 10. Consequently, the
angular information for any spatial location is sampled by different microlenses. Furthermore, the spatioangular sampling is
determined by z 1B and z 2 , as long as Eq. (31) is satisfied; this
alleviates the constraint on packing more microlenses, as their
density no longer determines the spatial resolution. Instead, we
can use larger microlenses to produce higher-quality images,
with fewer boundary pixels resulting in a more efficient use of
the sensor [90]. This flexibility in spatioangular sampling also
results in higher-resolution reconstructed images [92,93].
B. Computational Methods
Following a long tradition in image reconstruction and restoration algorithms [94], we can similarly make use of prior
Tutorial
Fig. 10. Focused plenoptic camera system. Object is focused at a
plane in front of the lenslet array, and the microlenses then act as
a relay system with the main camera lens.
information about the light field and the image in processing
the raw data from the photodetectors [95,96]. For instance,
one can model the light field patches using a Gaussian mixture
model to achieve light field superresolution [97] or, alternatively, assume a Lambertian scene object in the image formation
model and use a Markov random field prior in the reconstruction [89]. One can also make use of the spatial relationship in
the subarray images [98,99] and make use of disparity maps to
generate superresolved views of an object, which correspond
to increasing the sampling rate of the light field in spatial and
angular directions [100].
Many computational approaches, in fact, are coupled with
changes in the optical system in the data capture. A representative method is the placement of a second attenuation mask at
the main lens aperture in Fig. 9. In this design, the two masks
together modulate and encode the light field spectrum, and the
computational reconstruction takes advantage of the coherence
inherent in the angular dimensions of the light field by optimizing with a total variation (TV) norm regularizer [73,101].
Along a similar vein, it has been argued that the natural light
field is sparse. Hence, making use of compressed sensing and
dictionary learning [102–104], it is possible to construct a compressive light field camera that captures optimized 2D light
field projections and design robust sparse reconstruction to
recover a higher-resolution light field [105].
6. CONCLUSIONS
The development of light field cameras, as well as the associated
image reconstruction and superresolution algorithms, has significantly enriched our arsenal of imaging tools, particularly
for computational photography. However, as pointed out by
Goodman, light field photography can also shed light on other
imaging modalities, such as digital holography, and vice versa
[106]. Looking into the future, it would not be surprising to
see more applications emerging because of our ability to capture
light fields, and biophotonics imaging is certainly an area that has
a lot of potential to benefit from this new imaging paradigm.
REFERENCES
Fig. 9. Dappled photography with heterodyned light field capture
using an attenuation mask. The mask is typically placed close to the
detector, i.e., with a small z 2 [74].
1. E. Y. Lam, “Computational photography: advances and challenges,”
Proc. SPIE 8122, 81220O (2011).
2. D. R. Gerwe, A. Harvey, and M. E. Gehm, “Computational optical
sensing and imaging: Introduction to feature issue,” Appl. Opt. 52,
COSI1–COSI2 (2013).
Tutorial
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
3. F. H. Imai, D. C. Linne von Berg, T. Skauli, S. Tominaga, and Z.
Zalevsky, “Imaging systems and applications: Introduction to the
feature,” Appl. Opt. 53, ISA1–ISA2 (2014).
4. E. Y. Lam, G. Bennett, C. Fernandez-Cull, D. Gerwe, M. Kriss, and
Z. Zalevsky, “Imaging systems and signal recovery: introduction to
feature issue,” Appl. Opt. 54, IS1–IS2 (2015).
5. R. Ng, “Digital light field photography,” Ph.D. thesis (Stanford
University, 2006).
6. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan,
“Light field photography with a hand-held plenoptic camera,” Tech.
Rep. CSTR 2005-02 (Stanford University Computer Science
Department, 2005).
7. V. Boominathan, K. Mitra, and A. Veeraraghavan, “Improving
resolution and depth-of-field of light field cameras using a hybrid imaging system,” in IEEE International Conference on Computational
Photography (IEEE, 2014), pp. 1–10.
8. F. Deng, W. F. Sze, J. Deng, K. S. Fung, W. Leung, and E. Y. Lam,
“Regularized multiframe phase-shifting algorithm for threedimensional profilometry,” Appl. Opt. 51, 33–42 (2012).
9. F. Deng, C. Liu, W. Sze, J. Deng, K. S. Fung, W. Leung, and E. Y.
Lam, “Illumination-invariant phase-shifting algorithm for three-dimensional profilometry of a moving object,” Opt. Eng. 51, 097001 (2012).
10. F. Deng, C. Liu, W. Sze, J. Deng, K. S. Fung, and E. Y. Lam, “An
INSPECT measurement system for moving objects,” IEEE Trans.
Instrum. Meas. 64, 63–74 (2015).
11. P. Kner, “Phase diversity for three-dimensional imaging,” J. Opt. Soc.
Am. A 30, 1980–1987 (2013).
12. R. N. Bracewell, Two-Dimensional Imaging (Prentice-Hall, 1995).
13. J. L. Prince and J. M. Links, Medical Imaging Signals and Systems,
2nd ed. (Prentice-Hall, 2014).
14. D. Boas, D. Brooks, E. Miller, C. DiMarzio, M. Kilmer, R. Gaudette,
and Q. Zhang, “Imaging the body with diffuse optical tomography,”
IEEE Signal Process. Mag. 18, 57–75 (2001).
15. P. Bonfert-Taylor, F. Leblond, R. W. Holt, K. Tichauer, B. W. Pogue,
and E. C. Taylor, “Information loss and reconstruction in diffuse
fluorescence tomography,” J. Opt. Soc. Am. A 29, 321–330 (2012).
16. J. F. de Boer, B. Cense, B. H. Park, M. C. Pierce, G. J. Tearney, and
B. E. Bouma, “Improved signal-to-noise ratio in spectral-domain
compared with time-domain optical coherence tomography,” Opt.
Lett. 28, 2067–2069 (2003).
17. J. Ke and E. Y. Lam, “Image reconstruction from nonuniformly
spaced samples in spectral-domain optical coherence tomography,”
Biomed. Opt. Exp. 3, 741–752 (2012).
18. M. K. Kim, Digital Holographic Microscopy: Principles, Techniques,
and Applications (Springer, 2011).
19. E. Y. Lam, X. Zhang, H. Vo, T.-C. Poon, and G. Indebetouw, “Threedimensional microscopy and sectional image reconstruction using
optical scanning holography,” Appl. Opt. 48, H113–H119 (2009).
20. L. V. Wang and H.-I. Wu, Biomedical Optics: Principles and Imaging
(Wiley, 2007).
21. B. E. Saleh and M. C. Teich, Fundamentals of Photonics, 2nd ed.
(Wiley, 2007).
22. E. H. Adelson and J. R. Bergen, “The plenoptic function and the
elements of early vision,” in Computational Models of Visual
Processing (MIT, 1991), pp. 3–20.
23. M. Levoy and P. Hanrahan, “Light field rendering,” in Proceedings of
SIGGRAPH (ACM, 1996), pp. 31–42.
24. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” in Proceedings of SIGGRAPH (ACM, 1996), pp. 43–54.
25. A. Gershun, “The light field,” J. Math. Phys. 18, 51–151 (1939).
Originally published in Moscow in 1936.
26. M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge,
1999).
27. T. Fujii, T. Kimoto, and M. Tanimoto, “Ray space coding for 3D visual
communication,” in Picture Coding Symposium (1996), pp. 447–451.
28. T. Fujii and M. Tanimoto, “Free viewpoint TV system based on rayspace representation,” Proc. SPIE 4864, 175–189 (2002).
29. R. Ramamoorthi and P. Hanrahan, “On the relationship between
radiance and irradiance: determining the illumination from images
of a convex Lambertian object,” J. Opt. Soc. Am. A 18, 2448–2459
(2001).
2031
30. H. H. Barrett and K. J. Myers, Foundations of Image Science (Wiley,
2004).
31. R. N. Bracewell, The Fourier Transform and Its Applications, 3rd ed.
(McGraw-Hill, 2000).
32. J. W. Goodman, Introduction to Fourier Optics, 3rd ed. (Roberts and
Company, 2004).
33. B. K. Horn, Robot Vision (MIT, 1986).
34. K. Halbach, “Matrix representation of Gaussian optics,” Am. J. Phys.
32, 90–108 (1964).
35. A. Isaksen, L. McMillan, and S. J. Gortler, “Dynamically reparameterized light fields,” in Proceedings of SIGGRAPH (ACM, 2000),
pp. 297–306.
36. M. Lippmann, “Photographie—Épreuves réversibles. photographies
intégrales,” Académie des Sci. 146, 446–451 (1908).
37. J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt.
48, H77–H94 (2009).
38. X. Xiao, B. Javidi, M. Martínez-Corral, and A. Stern, “Advances in
three-dimensional integral imaging: sensing, display, and applications,” Appl. Opt. 52, 546–560 (2013).
39. H. E. Ives, “Parallax panoramagrams made with a large diameter
lens,” J. Opt. Soc. Am. 20, 332–340 (1930).
40. H. E. Ives, “Optical properties of a Lippman lenticulated sheet,”
J. Opt. Soc. Am. 21, 171–176 (1931).
41. A. Chutjian and R. J. Collier, “Recording and reconstructing threedimensional images of computer-generated subjects by Lippmann
integral photography,” Appl. Opt. 7, 99–103 (1968).
42. E. H. Adelson and J. Y. Wang, “Single lens stereo with a plenoptic
camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992).
43. V. Vaish, B. Wilburn, N. Joshi, and M. Levoy, “Using plane + parallax
for calibrating dense camera arrays,” in IEEE Computer Society
Conference on Computer Vision and Pattern Recognition (2004),
Vol. I, pp. 2–9.
44. R. Szeliski, Computer Vision: Algorithms and Applications (Springer,
2011).
45. M. Levoy, B. Chen, V. Vaish, M. Horowitz, I. McDowall, and M. Bolas,
“Synthetic aperture confocal imaging,” in Proceedings of SIGGRAPH
(2004), Vol. 23, pp. 825–834.
46. B. Wilburn, N. Joshi, V. Vaish, M. Levoy, and M. Horowitz, “Highspeed videography using a dense camera array,” in IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition (2004), Vol. II, pp. 294–301.
47. B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A.
Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” ACM Trans. Graph. 24, 765–776 (2005).
48. M. Hirsch, S. Sivaramakrishnan, S. Jayasuriya, A. Wang, A. Molnar,
R. Raskar, and G. Wetzstein, “A switchable light field camera
architecture with angle sensitive pixels and dictionary-based
sparse coding,” in IEEE International Conference on Computational
Photography (IEEE, 2014), pp. 1–10.
49. A. Wang, P. Gill, and A. Molnar, “Light field image sensors based on
the Talbot effect,” Appl. Opt. 48, 5897–5905 (2009).
50. A. Wang and A. Molnar, “A light-field image sensor in 180 nm
CMOS,” IEEE J. Solid-State Circuits 47, 257–271 (2012).
51. M. Land, “The optics of animal eyes,” Contemp. Phys. 29, 435–455
(1988).
52. S. Ogata, J. Ishida, and T. Sasano, “Optical sensor array in an
artificial compound eye,” Opt. Eng. 33, 3649–3655 (1994).
53. J. S. Sanders and C. E. Halford, “Design and analysis of apposition
compound eye optical sensors,” Opt. Eng. 34, 222–235 (1995).
54. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T.
Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation
module by bound optics (TOMBO): concept and experimental
verification,” Appl. Opt. 40, 1806–1813 (2001).
55. J. Tanida, R. Shogenji, Y. Kitamura, K. Yamada, M. Miyamoto, and
S. Miyatake, “Color imaging with an integrated compound imaging
system,” Opt. Express 11, 2109–2117 (2003).
56. W.-S. Chan, E. Y. Lam, and M. K. Ng, “Investigation of computational
compound-eye imaging system with super-resolution reconstruction,”
in International Conference on Acoustics, Speech, and Signal
Processing (IEEE, 2006), pp. 1177–1180.
2032
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
57. W.-S. Chan, E. Y. Lam, and M. K. Ng, “Extending the depth of field in
a compound-eye imaging system with super-resolution reconstruction,” in International Conference on Pattern Recognition (IEEE,
2006), pp. 623–626.
58. W.-S. Chan, E. Y. Lam, M. K. Ng, and G. Y. Mak, “Super-resolution
reconstruction in a computational compound-eye imaging system,”
Multidimension. Syst. Signal Process. 18, 83–101 (2007).
59. R. J. Plemmons, S. Prasad, S. Matthews, M. Mirotznik, R. Barnard,
B. Gray, V. P. Pauca, T. C. Torgersen, J. van der Gracht, and G.
Behrmann, “PERIODIC: Integrated computational array imaging technology,” in OSA Topical Meeting in Computational Optical Sensing
and Imaging (Optical Society of America, 2007), paper CMA1.
60. S. Prasad, “Digital superresolution and the generalized sampling
theorem,” J. Opt. Soc. Am. A 24, 311–325 (2007).
61. H. Navarro, J. Barreiro, G. Saavedra, M. Martínez-Corral, and B.
Javidi, “High-resolution far-field integral-imaging camera by double
snapshot,” Opt. Express 20, 890–895 (2012).
62. Y.-R. Ng, P. M. Hanrahan, M. S. Levoy, and M. A. Horowitz, “Imaging
arrangements and methods therefor,” U.S. patent US8395696 B2
(March 12, 2013).
63. T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667,
86671J (2013).
64. D. G. Dansereau, O. Pizarro, and S. B. Williams, “Decoding, calibration and rectification for lenselet-based plenoptic cameras,” in IEEE
Conf. Comput. Vis. Pattern Recogn., Portland, Oregon, 2013,
pp. 1027–1034.
65. D. G. Dansereau, O. Pizarro, and S. B. Williams, “Linear volumetric
focus for light field cameras,” ACM Trans. Graph. 34, 1–20 (2015).
66. E. R. Dowski and W. T. Cathey, “Extended depth of field through
wave-front coding,” Appl. Opt. 34, 1859–1866 (1995).
67. E. Ben-Eliezer, E. Marom, N. Konforti, and Z. Zalevsky,
“Experimental realization of an imaging system with an extended
depth of field,” Appl. Opt. 44, 2792–2798 (2005).
68. P. Zammit, A. R. Harvey, and G. Carles, “Extended depth-of-field imaging and ranging in a snapshot,” Optica 1, 209–216 (2014).
69. A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B.
Curless, D. Salesin, and M. Cohen, “Interactive digital photomontage,” ACM Trans. Graph. 23, 292–300 (2004).
70. R. Ng, “Fourier slice photography,” ACM Trans. Graph. 24, 735–744
(2005).
71. R. N. Bracewell, “Strip integration in radio astronomy,” Aust. J. Phys.
9, 198–217 (1956).
72. D. H. Garces, W. T. Rhodes, and N. M. Peña, “Projection-slice theorem: a compact notation,” J. Opt. Soc. Am. A 28, 766–769 (2011).
73. Z. Xu, J. Ke, and E. Y. Lam, “High-resolution lightfield photography
using two masks,” Opt. Express 20, 10971–10983 (2012).
74. A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J.
Tumblin, “Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” in Proceedings of
SIGGRAPH (2007), Vol. 26, pp. 1–12.
75. R. N. Bracewell, K.-Y. Chang, A. Jha, and Y.-H. Wang, “Affine theorem
for two-dimensional Fourier transform,” Electron. Lett. 29, 304 (1993).
76. M. Broxton, L. Grosenick, S. Yang, N. Cohen, A. Andalman, K.
Deisseroth, and M. Levoy, “Wave optics theory and 3-D deconvolution
for the light field microscope,” Opt. Express 21, 25418–25439 (2013).
77. N. Cohen, S. Yang, A. Andalman, M. Broxton, L. Grosenick, K.
Deisseroth, M. Horowitz, and M. Levoy, “Enhancing the performance
of the light field microscope using wavefront coding,” Opt. Express
22, 24817–24839 (2014).
78. L. Tian and L. Waller, “3D intensity and phase imaging from light field
measurements in an LED array microscope,” Optica 2, 104–111 (2015).
79. M. E. Testorf and M. A. Fiddy, “Lightfield photography and phasespace tomography: A paradigm for computational imaging,” in
OSA Topical Meeting in Computational Optical Sensing and
Imaging (Optical Society of America, 2009), paper CTuB2.
80. Z. Zhang and M. Levoy, “Wigner distributions and how they relate to
the light field,” in IEEE International Conference on Computational
Photography (IEEE, 2009), pp. 1–10.
81. T. Cuypers, R. Horstmeyer, S. B. Oh, P. Bekaert, and R. Raskar,
“Validity of Wigner distribution function for ray-based imaging,” in
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.
103.
104.
105.
106.
Tutorial
IEEE International Conference on Computational Photography
(IEEE, 2011), pp. 1–9.
M. J. Bastiaans, “Wigner distribution function and its application to
first-order optics,” J. Opt. Soc. Am. 69, 1710–1716 (1979).
H. M. Ozaktas, Z. Zalevsky, and M. A. Kutay, The Fractional Fourier
Transform: With Applications in Optics and Signal Processing
(Wiley, 2001).
S. B. Oh, S. Kashyap, R. Garg, S. Chandran, and R. Raskar,
“Rendering wave effects with augmented light field,” Comput.
Graph. Forum 29, 507–516 (2010).
S. A. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Appl.
Opt. 52, D22–D31 (2013).
A. Junker, T. Stenau, and K.-H. Brenner, “Scalar wave-optical
reconstruction of plenoptic camera images,” Appl. Opt. 53, 5784–
5790 (2014).
C. Hahne, A. Aggoun, S. Haxha, V. Velisavljevic, and J. C. J.
Fernández, “Light field geometry of a standard plenoptic camera,”
Opt. Express 22, 26659–26673 (2014).
C.-K. Liang, T.-H. Lin, B.-Y. Wong, C. Liu, and H. H. Chen,
“Programmable aperture photography: multiplexed light field acquisition,” in Proceedings of SIGGRAPH (2008), Vol. 27, pp. 1–10.
T. E. Bishop and P. Favaro, “The light field camera: extended depth
of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal.
Mach. Intell. 34, 972–986 (2012).
T. Georgiev and A. Lumsdaine, “Focused plenoptic camera and rendering,” J. Electron. Imaging 19, 021106 (2000).
T. G. Georgiev, A. Lumsdaine, and S. Goma, “High dynamic range
image capture with plenoptic 2.0 camera,” in OSA Topical Meeting in
Signal Recovery and Synthesis (Optical Society of America, 2009),
paper SWA7P.
T. Georgiev, G. Chunev, and A. Lumsdaine, “Superresolution with
the focused plenoptic camera,” Proc. SPIE 7873, 78730X (2011).
T. Georgiev, “Plenoptic camera resolution,” in OSA Imaging and
Applied Optics Congress (Optical Society of America, 2015),
paper JTh4A.2.
E. Y. Lam, “Image restoration in digital photography,” IEEE Trans.
Consum. Electron. 49, 269–274 (2003).
A. Levin, W. T. Freeman, and F. Durand,“Understanding camera tradeoffs through a Bayesian analysis of light field projections,” in European
Conference on Computer Vision (Springer, 2008), pp. 88–101.
Z. Xu and E. Y. Lam, “A spatial projection analysis of light field capture,” in OSA Frontiers in Optics (Optical Society of America, 2010),
paper FWH2.
K. Mitra and A. Veeraraghavan, “Light field denoising, light field
superresolution and stereo camera based refocussing using a
GMM light field patch prior,” in IEEE Computer Society Conference
on Computer Vision and Pattern Recognition Workshops (IEEE,
2012), pp. 22–28.
Z. Xu and E. Y. Lam, “Light field superresolution reconstruction in computational photography,” in OSA Topical Meeting in Signal Recovery
and Synthesis (Optical Society of America, 2011), paper SMB3.
J. Xu, Z. Xu, and E. Y. Lam, “Method and apparatus for processing
light-field image,” U.S. patent 14,575,091 (June 18, 2015).
S. Wanner and B. Goldluecke, “Variational light field analysis for disparity estimation and super-resolution,” IEEE Trans. Pattern Anal.
Mach. Intell. 36, 606–619 (2014).
Z. Xu and E. Y. Lam, “A high-resolution lightfield camera with dualmask design,” Proc. SPIE 8500, 85000U (2012).
D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52,
1289–1306 (2006).
E. J. Candès and M. B. Wakin, “An introduction to compressive sampling,” IEEE Signal Process. Mag. 25, 21–30 (2008).
J. Ke and E. Y. Lam, “Object reconstruction in block-based compressive imaging,” Opt. Express 20, 22102–22117 (2012).
K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive
light field photography using overcomplete dictionaries and optimized projections,” ACM Trans. Graph. 32, 1–12 (2013).
J. W. Goodman, “Assessing a new imaging modality,” in OSA
Imaging and Applied Optics Congress (Optical Society of America,
2012), paper JM1A.1.