Download Theory of confocal fluorescence imaging in the

Journal of Microscopy, Vol. 189, Pt 3, March 1988, pp. 192–198.
Received 12 August 1997; accepted 1 December 1997
SHORT COMMUNICATION
Theory of confocal fluorescence imaging in the
programmable array microscope (PAM)
P. J. VERVEER,* Q. S. HANLEY,* P. W. VERBEEK,† L. J. VAN VLIET† & T. M. JOVIN*
* Department of Molecular Biology, Max Planck Institute for Biophysical Chemistry,
Am Fassberg 11, D-37077 Göttingen, Germany
† Faculty of Applied Physics, Pattern Recognition Group, Delft University of Technology,
Lorentzweg 1, 2628 CJ Delft, The Netherlands
Key words. Confocal microscope, spatial light modulator, digital micromirror
device (DMD), tandem scanning microscopy, pseudorandom sequences,
point-spread-function, optical sectioning.
Summary
The programmable array microscope (PAM) uses a spatial
light modulator (SLM) to generate an arbitrary pattern of
conjugate illumination and detection elements. The SLM
dissects the fluorescent light imaged by the objective into a
focal conjugate image, Ic, formed by the ‘in-focus’ light, and
a nonconjugate image, Inc, formed by the ‘out-of-focus’
light. We discuss two different schemes for confocal imaging
using the PAM. In the first, a grid of points is shifted to scan
the complete image. The second, faster approach, uses a
short tiled pseudorandom sequence of two-dimensional
patterns. In the first case, Ic is analogous to a confocal
image and Inc to a conventional image minus Ic. In the
second case Ic and Inc are the sum and the difference,
respectively, of a conventional and a confocal image. The
pseudorandom sequence approach requires post-processing
to retrieve the confocal part, but generates significantly
higher signal levels for an equivalent integration time.
1. Introduction
Confocal fluorescence microscopy has been widely used to
obtain images of biological specimens (Pawley, 1995). In
contrast to the conventional fluorescence microscope,
confocal configurations allow optical sectioning of the object
and thereby three-dimensional imaging (Wilson, 1989;
Correspondence to: T. M. Jovin; tel: +49-551-201-1382; fax: +49-551-2011467; E-mail: [email protected]
192
Brakenhoff et al., 1989). This is achieved by illumination of
the object with a point source and detection through a
pinhole at a conjugate position in the emission path. To
acquire a complete image, scanning of the light source or of
the sample is required. An inherent limitation of the widely
distributed confocal laser scanning microscope (CLSM) is its
low duty cycle; the laser illuminates only one point at a time.
In contrast, the conventional full-field microscope records
the images much faster (no scanning is required) and is also
more light efficient (no photons are rejected by a pinhole). A
critical discussion of the relative merits of wide-field
microscopy with deconvolution and confocal scanning
microscopy for three-dimensional imaging involves complex
issues (Shaw, 1995; Verveer et al., 1997).
An alternative to the sequential operation of the CLSM is
offered by the tandem scanning microscope (Egger &
Petráň, 1967) and related real-time scanning instruments
(Kino, 1995). These devices incorporate a rotating disc with
multiple apertures for the simultaneous illumination and
detection of a large number of points. However, since the
spacing between the pinholes must be large to maintain the
confocal effect, much of the illumination light is rejected by
the disc ($ 97%; Fewer et al., 1997). This problem can be
circumvented by using an array of closely packed apertures
modulated in time such that they open and close in a
completely uncorrelated fashion (Juškaitis et al., 1996;
Wilson et al., 1996). The result is the sum of a conventional
and a confocal image. A separate conventional image must
be taken and subtracted in order to obtain the desired
confocal image.
q 1998 The Royal Microscopical Society
P RO G R A M M A B L E A R R AY M IC RO SC O P E ( PA M )
Fig. 1. Schematic of a fluorescence PAM microscope based on a
digital micromirror device (DMD). A DMD is illuminated uniformly.
Only mirror elements orientated in the ‘on’ position (one is
depicted) reflect light into the back-aperture of the objective, exciting the sample at conjugate positions. The fluorescent emission is
imaged back to the DMD, where it is relayed to a 2-D detector (camera-1) by the same mirror elements, forming the focal conjugate
image Ic. Fluorescent light arising from ‘out-of-focus’ positions in
the sample is reflected from mirror elements in the ‘off ’ position,
and may optionally be relayed to a second detector (camera-2) in
order to record the nonconjugate image Inc simultaneously. In
the figure, the fluorescent light from an ‘out-of-focus’ point is
shown to be intercepted by both depicted mirror elements and is
therefore detected by both camera-2 (predominantly) and camera-1. The fluorescence from ‘in-focus’ points is recorded
exclusively on camera-1.
193
be tilted at an angle of 6 108 to the normal (Montamedi et
al., 1997). The use of a DMD for laser-based confocal
imaging in reflection mode was reported recently (Liang et
al., 1997). Figure 1 presents a schematic for a fluorescence
programmable array microscope system based on the DMD.
The latter is uniformly illuminated by a lamp or laser
source. Mirror elements switched to the ‘on’ position reflect
the light to the object at conjugate positions in the focal
plane of the objective. The fluorescent emission is imaged
back to the corresponding elements of the DMD and
reflected via relay optics to a 2-D detector (camera-1),
yielding the focal conjugate image Ic. The fluorescence
representing ‘out-of-focus’ contributions, i.e. originating
from nonfocal planes, is reflected by the mirror elements in
the ‘off ’ position and imaged onto a second detector
(camera-2), thereby obtaining the corresponding nonconjugate image Inc. Forming the nonconjugate image on
a second camera, a highly desirable option of the PAM, may
be difficult or impossible with other kinds of SLM devices.
During the frame integration time, the modulation pattern
of the SLM is changed N times in order to generate a confocal
image. We denote the ith modulation of the SLM by
¼
Si ðxd ; yd Þ
1 if ðxd ; yd Þ is on an SLM element that is ‘on’;
0 ifðxd ; yd Þ is on an SLM element that is ‘off ’;
ð1Þ
where xd and yd are continuous coordinates on the SLM. We
include the shape and finite size of the SLM elements in
Si(xd, yd). We assume an object O(xo, yo, zo) on a separate
coordinate system (xo, yo, zo). It is scanned in the axial
direction by displacing it over a distance zs. For each
modulation pattern Si(xd, yd) the object is illuminated with
an intensity given by integrating the contributions of all the
illumination sources:
… … þ∞
Si ðMu; MvÞHex ðxo ¹ u; yo ¹ v; zo Þ du dv;
Ii ðxo ; yo ; zo Þ ¼
¹∞
In this paper, we discuss two approaches for obtaining
confocal fluorescence images using the newly developed
programmable array microscope (PAM; Hanley, Verveer &
Jovin, submitted manuscript). The PAM uses a programmable spatial light modulator (SLM) for generating an
arbitrary pattern of conjugate illumination and detection
apertures. We present the theoretical basis for image
formation using the SLM-based system.
2. Theory
2.1. Confocal imaging with a PAM
SLMs can be classified according to whether they operate in
a transmission or reflection mode. A particular example of
the latter class is the digital micromirror device (DMD),
consisting of an array of square mirrors, each of which can
q 1998 The Royal Microscopical Society, Journal of Microscopy, 189, 192–198
ð2Þ
where M is the magnification of the lens, and Hex is the
excitation PSF of the lens. We sum the contributions of all
points in the object and over all different modulations of the
SLM to find the conjugate image Ic:
… … … þ∞
¹1
T NX
S ðx ; y Þ
Ic ðxd ; yd ; zs Þ ¼
N i¼0 i d d
¹∞
x
y
× Hem d ¹ u; d ¹ v; w Ii ðu; v; wÞOðu; v; w ¹ zs Þ du dv dw;
M
M
ð3Þ
where Hem is the emission PSF of the lens, and T is the total
integration time. The expression for the nonconjugate
image Inc is obtained by replacing the term Si(xd, yd) in Eq.
(3) with 1¹ Si(xd, yd).
The modulations Si(xd, yd) can be selected so as to
implement two approaches for fast confocal imaging.
194
P. J. V E RV E E R E T A L .
Switching SLM elements in a grid pattern and shifting them
systematically provides a system analogous to a tandem
scanning microscope. Alternatively, by modulating the SLM
elements using pseudorandom sequences, an aperture
correlation approach similar to that recently reported for
a real-time microscope can be implemented (Juškaitis et al.,
1996; Wilson et al., 1996).
In the first approach, we select the modulation patterns
according to
Sa;b ðxd ; yd Þ ¼ Gðxd ¹ ah; yd ¹ bhÞ;
ð4Þ
where h is the size of an SLM element (assuming square
elements with 100% fill factor) and the index i in Eq. (1) has
been replaced by two-dimensional integer indices a, b, where
0 #a < nx, 0 #b < ny. The grid is defined by the properties
Gðxd ; yd Þ ¼ Gðxd ¹ pdx ; yd ¹ qdy Þ
p; q [ N;
ð5Þ
and
nX
y ¹1
x ¹1 n
X
Gðxd ¹ ah; yd ¹ bhÞ ¼ 1;
ð6Þ
a¼0 b¼0
where dx ¼ nxh and dy ¼ nyh are the lattice distances of the
grid. The grid can have many different shapes, the simplest
being square, but pseudohexagonal grids or line patterns
are also possible and in some cases preferable.
In the second approach, all SLM elements are ideally
modulated in a completely uncorrelated fashion. This is
difficult to achieve in practice, since the number of times
that the SLM can change its pattern within a given frame
integration time is limited. Thus, we repeatedly use the
same sequence of relatively short length in order to
modulate the complete plane. Define a sequence of N twodimensional patterns Ri(a, b) with the property
N¹1
X
Ri ða; bÞRi ðc; dÞ ¼ Ndða ¹ c; b ¹ dÞ a; b; c; d [ N;
ð7Þ
i¼0
where Ri(a, b) has value ¹1 or 1. A pseudorandom sequence
of one-dimensional patterns having the desired property can
be found using the rows of cyclic Hadamard matrices
(Harwit & Sloane, 1979). These are mapped to twodimensional patterns so as to obtain the required property
(Eq. 7). Since Si(xd, yd) can only assume the values 0 and 1,
we use the sequence (1 þ Ri(a, b))/2 (see also Wilson et al.,
1996). In this case 50% of all elements will be in the ‘on’
state in each modulation pattern. Other sequences may also
be employed (Golay, 1961).
We use a grid of the form in Eq. (5) to tile the ith pattern
in order to form the complete modulation pattern Si(xd, yd)
(see panel C of Fig. 3 for an example). That is, all SLM
elements for which G(xd ¹ ah, yd ¹ bh) ¼ 1 are switched in
an identical manner using the sequence (1 þ Ri(a, b))/2.
Since we switch all SLM elements simultaneously, the ith
modulation is given by a summation over all values of a
and b:
Si ðxd ; yd Þ ¼
nX
y ¹1
x ¹1 n
X
a¼0 b¼0
1ÿ
1 þ Ri ða; bÞ Gðxd ¹ ah; yd ¹ bhÞ
2
nx ¹1 nX
y ¹1
1 1X
¼ þ
R ða; bÞGðxd ¹ ah; yd ¹ bhÞ:
2 2 a¼0 b¼0 i
ð8Þ
The resulting signal consists of two parts, the first one
corresponding to a conventional image.
We have derived the imaging model for these modulation
schemes by substituting Eqs. (4) or (8) into Eq. (3). An
outline of the derivations is given in Appendix A, and we
present the results here. For the shifting grid approach the
focal conjugate image is given by
… … … þ∞
x
T
y
Hem d ¹ u; d ¹ v; w
Ic;S ðxd ; yd ; zs Þ ¼
M
M
nx ny
¹∞
yd ¹ fy
x ¹ fx
;v ¹
; w Oðu; v; w ¹ zs Þ du dv dw;
× IG u ¹ d
M
M
ð9Þ
where the total illumination is defined by:
… … þ∞
IG ðxo ; yo ; zo Þ ¼
GðMu; MvÞHex ðxo ¹ u; yo ¹ v; zo Þ du dv:
¹∞
ð10Þ
The total illumination IG enters into equation Eq. (9) with
spatially varying shifts along the two orthogonal x and y
directions, given by
fx ¼ xd mod h;
ð11aÞ
fy ¼ yd mod h:
ð11bÞ
Equation (9) can be viewed as a three-dimensional convolution with a spatially varying PSF, due to the dependence of fx
and fy on xd and yd, respectively. This is a consequence of the
fact that the SLM cannot scan continuously. That is, G(xd, yd)
can shift only over integer multiples of h, as opposed to a
continuous motion in the tandem scanning microscope
(Wilson & Hewlett, 1991). The distinction is negligible if h/M
is small compared to the sampling density of the image. For
large h/M the effect is a perceptible pixelation in the resulting
image. The PSF could be made spatially invariant by
scanning the SLM over h (dithering).
For the pseudorandom sequence approach we obtain
… … … þ∞
x
T
y
Hem d ¹ u; d ¹ v; w
Ic;R ðxd ; yd ; zs Þ ¼
M
M
4
¹∞
… … … þ∞
x
T
y
× Oðu; v; w ¹ zs Þ du dv dw þ
Hem d ¹ u; d ¹ v; w
M
M
4
¹∞
yd ¹ fy
xd ¹ fx
× IG u ¹
;v ¹
; w Oðu; v; w ¹ zs Þ du dv dw;
M
M
ð12Þ
We can draw the following important conclusions about
the various strategies for confocal image formation with the
PAM. (i) In comparison with a CLSM, the PAM has the
potential for increasing the acquisition speed by two orders of
q 1998 The Royal Microscopical Society, Journal of Microscopy, 189, 192–198
P RO G R A M M A B L E A R R AY M IC RO SC O P E ( PA M )
magnitude with a concurrent increase of signal strength by
an order of magnitude. Although the lamp source of the PAM
yields a much lower irradiance (typically 100-fold) at the focal
plane than the laser in a CLSM, this is compensated for by the
large number of points that are scanned in parallel (allowing
longer dwell times) and the higher quantum efficiency of the
detector (a CCD sensor instead of a photomultiplier tube).
(ii) The pseudorandom sequence approach is very light
efficient, a result of the fact that 50% of all SLM elements are
‘on’ in each modulation pattern. This is also seen from the
second term in Eq. (12), which is equal to Eq. (9) except for
the coefficient T/4 instead of T/nxny. Thus, although both
approaches yield the same confocal image for the same grid
G(xd, yd), the pseudorandom sequence generates a much
larger signal, the strength of which is independent of the
length of the sequence. (iii) To obtain a confocal image by the
pseudorandom sequence approach, a compensation term
must be subtracted. This may be a conventional image
acquired separately (Juškaitis et al., 1996; Wilson et al., 1996)
or the nonconjugate image. The latter is preferable since it is
acquired simultaneously and yields twice the intensity (see
Section 2.2). However, we note that the subtraction operation
inevitably increases noise, since the variance is the sum of the
variances of the two images.
2.2. The nonconjugate image
The nonconjugate image Inc can be imaged on a second
camera, for instance using a DMD as the SLM. The model for
Inc can be derived rigorously, but it is easier to note that the
sum of the two images must be a conventional image, since
the two cameras together collect all of the fluorescent light
that falls on the DMD. Therefore, Inc is the difference of a
conventional image and the image given by Eqs. (9) or (12).
For the pseudorandom sequence, it follows that the difference
between Ic and Inc equals the desired confocal image.
In both approaches, the two images can be used with a
deconvolution algorithm to generate an improved image.
This includes simple enhancement algorithms such as the
nearest-neighbour approach using three or more sections to
generate a single enhanced image (Agard, 1984) or more
powerful algorithms (see, for instance, Carrington et al.,
1995) using 3-D stacks to generate a full 3-D reconstructed
object. Both are used extensively in fluorescent imaging. We
are currently working on a multichannel extension of a set
of deconvolution algorithms described in Verveer & Jovin
(1997) that will allow the use of both Ic and Inc for
generating a single optimal restoration.
3. Results
3.1. Simulations
Equation (9) was used to simulate numerically the response of
an infinitely thin fluorescent plane as a function of its position
q 1998 The Royal Microscopical Society, Journal of Microscopy, 189, 192–198
195
Fig. 2. The image of an infinitely thin plane as a function of its position relative to the focal plane obtained by confocal imaging using
an SLM. A square grid was used with lattice distances dx ¼ dx ¼
{2, 3, 5, 10, 20} × h. The responses for a conventional microscope
and an ideal confocal microscope are also plotted. Parameters: SLM
element size h ¼ 17 mm, NA ¼ 1·4, M ¼ 100, excitation wavelength
= 633 nm, emission wavelength = 665 nm, refractive index = 1·515.
relative to the plane of focus. Simulation using Eq. (3) is also
possible and yields identical results. We used a square grid
with varying lattice distances. The conventional excitation
and emission PSFs were computed using vector wave theory
(van der Voort & Brakenhoff, 1990). Figure 2 presents the
normalized results for different lattice distances in the PAM
compared to a conventional microscope and an ideal confocal
microscope. Even for relatively small lattice distances a
significant sectioning effect is apparent, although at the cost
of an increased background. With increased lattice spacing,
the response becomes sharper and suppression of the background is more efficient. The conventional response deviates
from the expected straight line as a result of simulating an
infinitely large object in a finite sized image (truncation effect).
Had we simulated the pseudorandom sequence approach, the
result would have been identical except for a constant offset.
These results are comparable to calculations for tandem
scanning microscopy (Wilson & Hewlett, 1991).
Figure 3 presents a simulation based on a more complex
small 3-D object, consisting of a thin (0·1 mm) spherical
shell with a diameter of 5 mm in which a solid sphere 1·6 mm
in diameter was embedded. The peak grey value of the shell
was twice that of the inner sphere. We have used a similar
object for evaluating deconvolution algorithms (Verveer &
Jovin, 1997). Equations (9) and (12) were used to simulate
PAM microscope images using calculated excitation and
emission PSFs. The panels in Fig. 3 show the following
images: (A) single section through the middle of the
simulated object; (B) and (C) single patterns generated by
the SLM for the shifting grid and the pseudorandom
196
P. J. V E RV E E R E T A L .
Fig. 3. Simulation of the imaging of a 3D object using the shifting grid approach and the pseudorandom sequence approach. The object consisted of a spherical shell with a thickness of 0·1 mm and a diameter of 5 mm and a small embedded 1·6-mm solid sphere. The peak grey value
of the shell was twice that of the embedded sphere. Parameters: SLM element size h ¼ 17 mm, NA ¼ 1·3, M ¼ 100, excitation wavelength ¼
488 nm, emission wavelength ¼ 520 nm, refractive index ¼ 1·515, sampling distances x and y ¼ 34 nm, z ¼ 100 nm. Shifting grid approach:
square grid with lattice distances dx ¼ dy ¼ 10 × h. Pseudorandom sequence length: 128, tiled on a square grid with the same lattice distances.
(A) A single slice through the middle of the simulated object. (B) Single SLM pattern for the shifting grid approach. (C) Single SLM pattern for
the pseudorandom sequence approach. (D) Conjugate image Ic generated by the shifting grid approach. (E) Wide-field image. (F) Conjugate
image generated by the pseudorandom sequence approach. (G) Nonconjugate image Inc generated by both approaches. D equals F ¹ G and E
equals F þ G. See text for further explanation.
sequence approaches, respectively; (D) conjugate image Ic for
the shifting grid method. The pixelation effect due to the
finite size of the SLM is clearly visible. For imaging of a real
object with a less than ideal PSF, this effect will not be so
apparent; (E) wide-field (conventional) image; (F) conjugate
image Inc for the pseudorandom sequence approach; (G)
nonconjugate image that apart from an intensity difference
is equal for both approaches. The difference between the
images of F and G is equal to the confocal image in D, and
their sum is equal to the wide-field image in E. Note that
images F and G are similar. That is, the difference that
constitutes the reconstructed confocal image is small in
magnitude compared to the underlying wide-field signal in
both signals, reflecting the properties of this particular object.
3.2. Experimental verification of the sectioning properties
The first implementation of the PAM is described in Hanley et
al. (submitted manuscript). The instrument used a DMD to
modulate the excitation light according to one of the two
schemes described in Section 2. In contrast to the set-up of
Fig. 1, the emission light was detected via a separate optical
path that was not modulated by the DMD. By taking a
separate exposure for each illumination pattern and applying
image processing techniques to reconstruct the image, we
were able to demonstrate both imaging modes of the PAM for
biological specimens. The simulations based on the imaging
of an infinitely thin fluorescent plane presented in the
previous section have not been verified experimentally as a
result of the technical difficulties in preparing a suitable
sample. However, a similar experiment was performed in
reflection mode by scanning a mirror through the plane of
focus. Although the imaging model in reflection is somewhat
different (Wilson & Hewlett, 1991), the effects of the lattice
period on the sectioning and background suppression can be
evaluated. Figure 4 presents the results of such an
experiment for several lattice distances. The DMD was used
to illuminate a mirror with a square pattern of points for a
q 1998 The Royal Microscopical Society, Journal of Microscopy, 189, 192–198
P RO G R A M M A B L E A R R AY M IC RO SC O P E ( PA M )
Fig. 4. Experimental verification of the sectioning and background
suppression in the PAM. A square grid was used with lattice distances dx ¼ dy ¼ {3, 5, 10, 20} × h. The response for conventional
imaging is also plotted. SLM parameters: h ¼ 17 mm (16 mm þ
1 mm between-mirror gap) (DMD Texas Instruments, Dallas, TX).
Objective NA ¼ 1·3, M ¼ 100 oil immersion, refractive index ¼
1·515 (Plan-NeoFluar, Zeiss). Wavelength range 450–490 nm
(bandpass filter, Zeiss). The apparatus for experimental verification
is described in Hanley et al. (submitted manuscript). The sample
was displaced along the z-axis with a computerized focus control
(Ludl Electronic Products, Hawthorne, NY) and data acquisition
was controlled with IPLAB Spectrum (Signal Analytics, Vienna,
VA). Camera system: CH220 (Photometrics, Tucson, AZ) with
KAF1400 sensor. All components were installed in an Axioplan
microscope (Zeiss).
given lattice distance. A single exposure was taken for each
axial displacement. We used the image in which the mirror
was at the focal plane to determine the centre positions of a
number of illumination spots from their intensity distributions. The intensity as a function of the axial displacement
was determined at these positions and averaged after
normalization with the value at the focal plane. Although
Fig. 2 models a fluorescent response and Fig. 4 is a
measurement in reflection, both follow the same general
trend. The background systematically increases as the lattice
distance decreases. Thus, as the lattice period decreases the
confocal signal becomes harder to distinguish from background noise. The peak widths in Fig. 4 are greater than
expected for a confocal microscope operation in reflection.
However, we expect this to improve in an instrument
corresponding more closely to Fig. 1.
4. Conclusions
Two modes of operation of the PAM for confocal imaging have
been described. Both provide substantial advantages in speed
and signal level compared to the widely used CLSM. The
shifting grid approach has the advantage of being simple to
q 1998 The Royal Microscopical Society, Journal of Microscopy, 189, 192–198
197
implement, as well as yielding a confocal image without postprocessing. The response is similar to that of the tandem
scanning microscope in that the lattice distance of the grid
plays the same role as the pinhole spacing in a Nipkow disc.
However, this parameter is freely programmable in the PAM, a
significant advantage. The use of a pseudorandom sequence
of finite size tiled on a grid produces an image consisting of a
conventional term plus a confocal term equal to that obtained
with a shifting grid of equivalent spacing. The length of the
sequence defines the lattice sizes of the grid upon which it is
tiled and, thereby, the resolution and sectioning strength as in
the shifting grid approach. Although some post-processing is
needed, the frame integration time for a given signal level can
be substantially shorter. The ‘out-of-focus’ light normally
rejected in the CLSM can be collected with a second camera
and used to enhance the primary ‘in-focus’ image. With the
pseudorandom sequence approach, the difference of the two
images constitutes a confocal image with twice the intensity
obtained by correcting with a separately acquired conventional image. We are investigating the use of deconvolution
algorithms for combining the two images obtained in the form
of z-stacks using both approaches so as to achieve improved
reconstructions.
Acknowledgments
We thank Dr Robert Clegg for constructive comments on the
manuscript. P.J.V. is the recipient of a fellowship award
(Starker–Werner Fonds) of the Max Planck Society. L.J.v.V.
is supported as a Postdoctoral Fellow by the Dutch Royal
Academy of Sciences (KNAW).
References
Agard, D.A. (1984) Optical sectioning microscopy: cellular
architecture in three dimensions. Annu. Rev. Biophys. Bioeng.
13, 191–219.
Brakenhoff, G.J., van Spronsen, E.A., van der Voort, H.T.M. &
Nanninga, N. (1989) Three-dimensional confocal fluorescence
microscopy. Meth. Cell Biol. 30, 379–398.
Carrington, W.A., Lynch, R.M., Moore, E.D.W., Isenberg, G.,
Fogarty, K.E. & Fay, F.S. (1995) Superresolution three-dimensional images of fluorescence in cells with minimal light
exposure. Science, 268, 1483–1487.
Egger, M.D. & Petráň, M. (1967) New reflected-light microscope for
viewing unstained brain and ganglion cells. Science, 157, 305–
307.
Fewer, D.T., Hewlett, S.J., McCabe, E.M. & Hegarty, J. (1997) Directview microscopy: experimental investigation of the dependence
of the optical sectioning characteristics on pinhole-array
configuration. J. Microsc. 187, 54–61.
Golay, M.J.E. (1961) Complementary series. IRE Trans. Inf. Theory,
7, 82–87.
Harwit, M. & Sloane, N.J.A. (1979) Hadamard Transform Optics.
Academic Press, New York.
Juškaitis, R., Wilson, T., Neil, M.A.A. & Kozubek, M. (1996)
198
P. J. V E RV E E R E T A L .
Efficient real-time confocal microscopy with white lightsources.
Nature, 383, 804–806.
Kino, G.S. (1995) Intermediate optics in Nipkow disk microscopes.
Handbook of Biological Confocal Microscopy (ed. by J. B. Pawley),
chap. 10, pp. 155–165. Plenum Press, New York, 2nd edn.
Liang, M., Stehr, R.L. & Krause, A.W. (1997) Confocal pattern
period in multiple-aperture confocal imaging systems with
coherent illumination. Opt. Lett. 22, 751–753.
Motamedi, M.E., Wu, M.C. & Pister, K.S.J. (1997) Micro-optoelectro-mechanical devices on-chip optical processing. Opt. Eng.
36, 1282–1297.
Pawley, J.B. (1995) Handbook of Biological Confocal Microscopy.
Plenum Press, New York, 2nd edn.
Shaw, P. (1995) Comparison of wide-field/deconvolution and
confocal microscopy for 3d imaging. Handbood of Biological
Confocal Microscopy (ed. by J. B. Pawley), chap. 23, pp. 373–387.
Plenum Press, New York, 2nd edn.
Verveer, P.J. & Jovin, T.M. (1997) Efficient superresolution
restoration algorithms using maximum a posteriori estimations
with application to fluorescence microscopy. J. Opt. Soc. Am. A,
14, 1696–1706.
Verveer, P.J., van Kempen, G.M.P. & Jovin, T.M. (1997) Superresolution MAP algorithms applied to fluorescence imaging.
Three-Dimensional Microscopy: Image Acquisition and Processing IV
(ed. by C. J. Cogswell, J.-A. Conchello & T. Wilson), Proc. SPIE,
2984, 125–135.
van der Voort, H.T.M. & Brakenhoff, G.J. (1990) 3-D image
formation in a high-aperture fluorescence confocal microscope:
a numerical analysis. J. Microsc. 158, 43–54.
Wilson, T. (1989) Optical sectioning in confocal fluorescent
microscopes. J. Microsc. 154, 143–156.
Wilson, T. & Hewlett, S.J. (1991) Optical sectioning strength of the
direct-view microscope employing finite-sized pin-hole arrays. J.
Microsc. 163, 131–150.
Wilson, T., Juškaitis, R., Neil, M.A.A. & Kozubek, M. (1996)
Confocal microscopy by aperture correlation. Opt. Lett. 21,
1879–1881.
⊗2 Hex ðu; v; wÞ Oðu; v; w ¹ zs Þdu dv dw;
ð13Þ
where ⊗2 denotes the two-dimensional convolution, defined
by Eq. (2). We rewrite the coordinates xd and yd in terms of
the SLM element size h and the lattice distances dx and dy by
setting
ð14aÞ
xd ¼ ah þ pdx þ fx
yd ¼ bh þ qdy þ fy
where fx and fy are given by Eq. (11). Using Eqs. (5) and
(6) we see that
nX
y ¹1
x ¹1 n
X
Derivation of the imaging equations
ÿ
¼ G Mu ¹ xd þ pdx þ fx ; Mv ¹ yd þ qdy þ fy Þ
×
×
¹∞
Hem
y
¹ u; d ¹ v; w
M
M
Ic;R ðxd ; yd ; zs Þ
!
nx ¹1 nX
y ¹1
¹1
T NX
1 1X
¼
R ða; bÞGðxd ¹ ah; yd ¹ bhÞ
þ
N i¼0 2 2 a¼0 b¼0 i
"… … …
!
nx ¹1 nX
y ¹1
þ∞
ÿ
1 1X
R ðc; dÞG Mu ¹ ch; Mv ¹ dh
þ
×
2 c¼0 d¼0 i
¹∞ 2
#
x
y
⊗2 Hex ðu; v; wÞ Oðu; v; w ¹ zs ÞHem d ¹ u; d ¹ v; w du dv dw
M
M
×
nX
y ¹1
x ¹1 n
X
T
N
x
y
× Oðu; v; w ¹ zs ÞHem d ¹ u; d ¹ v; w du dv dw
M
M
… … … þ∞
x
T
y
¼
Hem d ¹ u; d ¹ v; w
M
M
4
¹∞
1þ
nX
y ¹1
x ¹1 n
X
a¼0 b¼0
× ½GðMu ¹ ah; Mv ¹ bhÞ
⊗2 Hex ðu; v; wÞÿOðu; v; w ¹ zs Þ du dv dw
… … … þ∞
x
T
y
¼
Hem d ¹ u; d ¹ v; w
M
M
nx ny
¹∞
… … … þ∞ nx ¹1 nX
y ¹1 n
y ¹1
x ¹1 n
X
X
N NX
dða ¹ c; b ¹ dÞ
þ
4 a¼0 b¼0 c¼0 d¼0
¹∞ 4
ÿ
× Gðxd ¹ ah; yd ¹ bhÞG Mu ¹ ch; Mv ¹ dh ⊗2 Hex ðu; v; wÞ
×
d
ð15Þ
Substituting this into Eq. (13) yields Eq. (9).
To derive Eq. (12) we substitute Eq. (8) into Eq. (3):
"
x
Gðxd ¹ ah; yd ¹ bhÞ
ÿ
¼ G Mu ¹ xd þ fx ; Mv ¹ yd þ fy :
nx ¹1 nX
y ¹1
T X
Gðxd ¹ ah; yd ¹ bhÞ
nx ny a¼0 b¼0
… … … þ∞
nX
y ¹1
x ¹1 n
X
a¼0 b¼0
To derive Eq. (9) we substitute Eq. (4) into Eq. (3):
Ic;S ðxd ; yd ; zs Þ ¼
ÿ
Gðxd ¹ ah; yd ¹ bhÞG Mu ¹ ah; Mv ¹ bh
a¼0 b¼0
¼
Appendix
ð14bÞ
ÿ
Gðxd ¹ ah; yd ¹ bhÞG Mu ¹ ah; Mv ¹ bh
!
#
⊗2 Hex ðu; v; wÞ O ðu; v; w ¹ zs Þdu dv dw:
ð16Þ
Using Eq. (15) this reduces to Eq. (12).
!
Gðxd ¹ ah; yd ¹ bhÞGðMu ¹ ah; Mv ¹ bhÞ
a¼0 b¼0
q 1998 The Royal Microscopical Society, Journal of Microscopy, 189, 192–198