Download On the Identification of Fast Dynamics Using Slow Rate Camera

51st IEEE Conference on Decision and Control
December 10-13, 2012. Maui, Hawaii, USA
On the Identification of Fast Dynamics using Slow Rate Camera
Measurements
Jacopo Tani, Sandipan Mishra, John T. Wen
Rensselaer Polytechnic Institute
Troy, NY 12180
focuses the a portion of the incoming beam onto the image
sensor. The deviation of the beam centroid from the reference
location (for the parallel wavefront case) indicates the local
slope of the wavefront. A deformable mirror consists of
multiple actuators arranged in an array, each individually
driven. These actuators typically operate at high bandwidth
with sampling period at 1ms or less [7]. The image sensors,
however, is limited to much slower sampling rate. Such
systems are called multi-rate system, where components
within the system operate at different sampling rates.
Because of the integrative nature of the slow image sensor,
high speed motion results in an image blur. Instead of
treating the blur as undesirable and to be removed, we use
the blur to extract motion information. In essence, we treat
the image sensor as a temporal to spatial transformation.
Our goal is to invert the transformation, while imposing
additional constraints to ensure uniqueness.
In image processing, motion blur refers to the phenomenon
observed when an image sensor captures motion of an object
relative to the sensor during the exposure time [8]. Removal
of the blur in a motion-blurred image, called deblurring,
without the knowledge of the underlying object is an illposed inverse problem [9]. For example, the image trace
of an object moving left to right would be indistinguishable from that of the object moving right to left. Several
algorithms have been proposed for extracting motion from
blur [8], with various additional assumptions to eliminate
this ill-posedness (such as assuming a known motion profile,
constant velocity, constant acceleration, etc.). While these
algorithms are effective for image restoration, they are inadequate for accurate dynamics reconstruction since they focus
primarily on determining the deblurred image, but not the
motion profile.
An integrative intensity sensor is unique in that temporal
information of the evolution of the output during the exposure time is captured by the sensor, effectively making it a
non-linear integral transform. There is scarce literature on
such non-linear integral transforms. In [10], the relationship
between ocean wave spectra and the corresponding ATINSAR (along-track interferometric synthetic aperture radar)
phase image spectrum measurement is modeled as a nonlinear integral transform using Gaussian basis functions
(which are approximated by delta functions to simplify
analysis). This model was not, however, utilized in any
dynamic systems framework. In [11], a video based tracking
Abstract— This paper addresses the identification of the dynamics of a high bandwidth motion system using measurements
from a slow rate integrative sensor. The motivation of this work
is the need to characterize the dynamics of adaptive optics
elements, such as deformable mirrors, using image array based
sensors, such as CCD or CMOS wavefront sensors, which sample at a significantly lower rate than the actuator bandwidth.
The integrative nature of the sensor produces a blurred image
when the image moves much faster than the exposure period.
The key concept of this paper is to extract system dynamics
from the image blur. We consider the image blur as a nonlinear
temporal-to-spatial transformation under a known input excitation, such as a sinusoid. The output signal parameters, such
as amplitude and phase, characterize the frequency response
of the system at the specific excitation frequency. This problem
may be posed as a nonlinear minimization: finding output signal
parameters to match the predicted spatial distribution with the
measurement. However, the nonlinear mapping is not one-toone, or, equivalently, the solution of the nonlinear minimization
is non-unique. We propose two methods for avoiding this
aliasing problem: by imposing a continuity constraint or by
solving the minimization over two different exposure periods.
The efficacy of the proposed identification approach for a singleinput/single-output system is experimentally demonstrated by
accurately obtaining the frequency response of a fast steering
mirror up to 500Hz using a 30 Hz CCD camera.
I. I NTRODUCTION
Image sensors are used as the feedback measurement
mechanism in a wide variety of applications [1]–[3]. Typical
image sensors consist of an array of CMOS or CCD1
elements. A fundamental limitation of these sensors is their
update rate, typically at 30Hz for CCD and less than 500Hz
for CMOS. There are fast cameras, they are expensive and
buffer the images preventing their use in real-time feedback
control. An image sensor integrates the amount of photons
impinging on each sensor element over the exposure period.
Typically only the time averaged (over the exposure period)
image feature of interest is used for analysis and feedback
control.
Adaptive Optics (AO) system is an example of imagebased feedback control system [4], [5], where an image
sensor measures the wavefront and a deformable mirror
adjusts its shape based on the wavefront measurement. A
commonly used wavefront sensor is the Shack-Hartmann
sensor [6], which consists of an array of lenslets in front
of a CCD or CMOS image sensor. Every lenslet in the array
1 Complementary Metal-Oxide Semiconductor and Charge-Coupled Devices.
978-1-4673-2066-5/12/$31.00
978-1-4673-2064-1/12/$31.00 ©2012
©2012 IEEE
IEEE
913
method is used to quantify the oscillations of an optically
trapped microsphere beyond the Nyquist frequency. The
method requires the modulation of the intensity of a light
source (termed Intensity Modulation Spectral Analysis, or
IMSA). In [12], the nonlinear image sensor transformation
from time to pixel domain is used in an extended Kalman
filter scheme for state estimation of an AO system. In this
paper, we consider the problem of system identification of
an MIMO system with high rate actuator but slow rate image
sensor. To illustrate the approach and experimentally validate
our results, we consider a simple setup consisting of a tiptilt fast steering mirror and a CCD image sensor. We pose
the problem as a minimization to extract temporal signal
information based on a spatial error metric. To eliminate
the potential ambiguity of the solution (or aliasing, due to
possible time permutation of the input signal), we use a
method based on imposing a continuity constraint on the
identified parameters and propose a method based on the
acquisition of images with multiple different exposure times.
simply the image kernel centred at y(t). With the application
of wavefront sensor in mind, we will consider Ψ to be
a point, without loss of generality, centred at the origin:
Ψ(η) = δ (η). For multiple points as in wavefront sensor,
we will consider multiple outputs, each one with its own
image kernel. Fig. 1 shows an example of the relationship
between the time-domain output y(t) and the pixel domain
image Y (η).
II. I NTEGRATIVE I MAGE S ENSOR
Fig. 1. Top left: time domain signals. Top right: the corresponding 2D
intensity mapping in the pixel domain of the image sensor.
A. Image Sensor Triggering
The image acquisition process is characterized by the
following timing parameters:
• Acquisition time, Ta : the time instant at which the image
sensor is triggered for the first time.
• Exposure time, Te : the length of the actual image
acquisition window.
• Triggering delay, Td : the delay between the command
for image acquisition and the start of the exposure
window.
• Readout time, Tr : the time to transfer measured information from the sensor to processor memory.
• Sample rate, Ts : the time between two successive exposure windows. Clearly, Ts ≥ Td + Tr + Te .
The first two parameters are user-specified. The readout time
is usually the bottleneck that for the sensor operation. We
will define tcam = Ta + Td , the instant the sensor exposure
takes place.
C. Point Source and Point Spread Function
In the case of point light source, as generated by a focused
laser beam or a guide star (in adaptive optics applications),
the image kernel, Ψ, is simply the point spread function
(PSF). In the ideal case, the image of the point source is also
a point; Ψ is then a delta function. In the case of a laser light
source, the intensity profile is typically a 2D Gaussian. The
optical aberration of a point source may also be approximated
by a 2D Gaussian profile. The PSF Ψ is then
1
Yk (η) = CΨ,Te ,Td ,kTs (y)(η) + nk (η),
Ψ(η − y(t)) dt.
k = 0, 1, . . . .
(4)
The general identification problem posed is to estimate H
by using {Yk }Kk=0 , for some K. In this paper, we use a specific
approach: obtain the frequency response of H , H( jω), by
finding the gain and phase at each ω. Letting u(t) = A sin ωt.
At steady state, the output is given by
(1)
where n(η) includes the random and shot noise of the image
sensor and CΨ : L2 [Ta + Td , Ta + Td + Te ] → L2 (F ) is given
by
Z
Ta +Td +Te
(3)
III. P ROBLEM D ESCRIPTION
Consider a stable SISO linear time invariant (LTI) system,
H , with input u and output y. The output, y is not directly
measured; instead it is to be inferred from the integrative
sensor output, Yk :
The output of the image sensor is a 2D intensity map,
Y (η), where η = (ηx , ηy ) parametrizes the spatial dimension
with η ∈ F := [(0, 0), (ηxmax , ηymax )]. We assume that the
image is formed by integrating an image kernel, Ψ, corresponding to y(t) ≡ 0, over the exposure period with the
acquisition command issued at t = Ta :
CΨ(y),Te ,Td ,Ta :=
0)
where η0 is the center of the Gaussian, a is the height of
the peak, and Σ is the 2 × 2 covariance matrix. The PSF may
be obtained experimentally by holding y(t) at zero. The PSF
is then approximately Y (η)/Te . The effect of noise may be
reduced by averaging Y over multiple exposures.
B. Image Sensor Modeling
Y (η) = CΨ (y)(η) + n(η)
T Σ−1 (η−η
Ψ(η) = ae− 2 (η−η0 )
(2)
y(t) = M sin(ωt + φ )
Ta +Td
(5)
where M = A |H( jω)| and φ = ∠H( jω)). For this paper,
instead of using Yk , we will command the image acquisition
To simplify notation, we will only include the all or part of
the subscripts of C if necessary. Note that Ψ(η − y(t)) is
914
after a tcam = Ta + Td at each experimental cycle, where the
Ta is chosen so to capture only the steady state response of
the system, and then average the measured input to reduce
the effect of noise. Hence, in effect, we are considering the
estimation of |H( jω)| and ∠H( jω) (ω is known) based on
a single image output, Y (y).
The identification problem then becomes the following
minimization problem:
min kC (M sin(ωt + φ )) −Y k
M,φ
- Step 3: Data Collection. For each frequency in prespecified set, Ω = {ω1 , . . . , ωN }, apply the input ui (t) =
A sin ωit, and collect the corresponding output images
Yi (η), as shown in Fig. 3.
(6)
where Y , a function of η, is the measured image distribution
based on the same exposure time Te , and k·k is a chosen norm
over the spatial domain. We summarize several properties of
C below.
Proposition 1: An intensity map generated by the image
kernel Ψ may be expressed as the convolution of the image
kernel with the Dirac delta function:
Fig. 3. Example of experimental data acquisition. The picture on the left
shows the reference signal and the output of the steering mirror (measured
by the PSD sensors) within the exposure window of the image sensor. The
picture on the right shows the typical “peanut” shaped image acquired.
CΨ (y) = Ψ ∗ Cδ (y).
(7)
Proposition 2: Two signals, y1 and y2 , generate the same
image if and only if they are the permutation of each other,
i.e.,
C (y1 ) = C (y2 ) ⇐⇒ y1 (t) = y2 (π(t))
where π is the permutation operator.
Proposition 3: Given y1 (t) = M1 sin(ωt + φ1 ) and y2 (t) =
M2 sin(ωt + φ2 ), the following is true, ∀z ∈ Z:
C (y1 ) = C (y2 ) ⇐⇒
M1 = M2
φ1 = (2z + 1)π − ω(Te + 2tcam ) − φ2 .
- Step 4: Sensor saturation and High Dynamic Range
(HDR) correction. Real image sensors saturate – the
integrative sensor model in (2) is only an approximation.
The saturation curve is as shown in Fig. 4. The image
intensity will need to adjusted based on this curve
to match with the sensor model. Some cameras are
equipped with the HDR correction feature, which takes
multiple pictures with different exposure times and then
stitch all the unsaturated regions together. To use such
camera for our purpose, the HDR feature should be
turned off.
This expression of the aliased solution can be obtained
by equating the first two moments [12] of the intensity
distributions generated by a signal y1 (t) and his permuted
version y2 (t).
Remark 1: Note that there is no alias on the amplitude,
so the minimization always gives the correct solution. The
two aliased phase solutions are related by the exposure
time Te (Fig. 2). We will use this fact later to modify the
minimization to eliminate the aliased solution.
IV. I DENTIFICATION P ROCEDURE
Fig. 4. Predicted and experimental maximum light intensity profiles as a
function of the exposure time with stationary constant laser source.
We now summarize the procedure of identifying the magnitude and phase of a sinusoid based on the acquired image.
- Step 1: Identification of mean distribution of camera
noise. The image without any light input will consist
of camera noise. The average of multiple images will
converge to the mean distribution of the noise. All
acquired images will subtract from this mean. Further
noise reduction may be obtained by thresholding: zeroing all intensities less than a threshold value, e.g., 1%
of the maximum intensity.
- Step 2: Identification of the image kernel, Ψ. Under zero
input, the output y(t) is also zero. The image kernel, Ψ,
is found directly from the acquired image (scaled by
Te ). To reduce the effect of noise, average (of images
subtracting the mean from Step 1) over multiple images
should be used. Image cropping and fitting to a 2D
Gaussian can further reduce the effect of noise.
- Step 5: Magnitude and Phase measurements. The last
step is to determine amplitude and phase of the output
sine waves from the collected images Yi . This is done
by solving a minimization problem (6) for each Yi .
V. S OLUTION OF M INIMIZATION P ROBLEM
The key step in the identification is the solution of the
minimization problem (6). All finite dimensional norms are
equivalent, so the choice of norm in (6) does not affect the
solution. However, they have different curvatures (therefore
different convergence rate) and noise sensitivities as seen in
Fig. 5. Based on this comparison, we choose the Frobenius
norm (vector two-norm) as it is steeper near the solution.
As expected, we see the aliased solution discussed in
Proposition 3 in the form of two minima for the phase and
only one for the amplitude (Fig. 6).
915
(a)
(b)
Fig. 2. (a) Cost function as a function of φ for different exposure times of the image sensor in the perfect predictor scenario. Amplitudes of the sine
waves are assumed to be correctly estimated. (b) Detail near the exact phase solution φ1 = −8◦ showing the invariance of its position with respect to the
exposure time.
(a)
Fig. 5.
(b)
Evaluation of cost function for different norms assuming correct amplitude in the (a) simulated noise-free case and (b) experimental case.
To avoid convergence to the aliased solution, we may impose the continuity constraint, i.e., choose the solution closer
to the solution of a nearby frequency. In Fig. 2, two different
exposure times are used. Alternatively, noting that the aliased
solution is a function of the exposure time, Te , while the true
solution remains independent of the exposure time, we may
pose the minimization problem for two different exposure
times:
n
min CΨ,Te1 (M sin(ωt + φ )) −Y1 M,φ
o
+ CΨ,Te2 (M sin(ωt + φ )) −Y2 (8)
Fig. 6. Cost function normalized with respect to the measured image norm
in the SISO ideal case, for ∞-norm, as a function of both amplitude and
phase.
where (Y1 ,Y2 ) are the images obtained with exposure times
(Te1 , Te2 ), respectively. Fig. 2 shows the cost function when
two different exposure times are used. From Proposition 3,
considering the exposure period (Te ) and the camera acquisition time (tcam ) proportional to the known frequency, i.e.:
Te = p/ω and tcam = b/ω, then the expression of the alias
phase becomes (letting z = 0):
φ1 = π − p + 2b − φ2 .
p1 = p2 + 0.5. Fig. 7 shows the simulated noise-free case
using cost function (8) normalizing each addend with the
norm of the respective measured intensity distribution.
VI. E XPERIMENTAL A PPARATUS
To test the concept in this paper, we have constructed
an experimental setup shown in Fig. 8. A laser source is
bounced off a fast steering mirror (FSM) with the resulting
image captured by a CCD camera, as represented in Fig. 9.
(9)
the two exposures times can be chosen, letting b1 = b2 , so
that the distance between the two minima is maximum, i.e.
916
Fig. 9.
Fig. 7. Perfect predictor case using cost function with exposure times of
30% and 80% of the period of the sine wave. φ2 is no longer a global
minimum of the cost function.
•
•
The mirror is also equipped with a high bandwidth position
sensing diode (PSD) allowing the comparison between the
image based technique in this paper with a high quality
reference. The components of the system are described
below.
•
all the experiments. Variation in the laser intensity of
may be exploited to further extract system dynamical
information as in [11].
CCD Camera records the light intensity of the laser
beam deflecting off the mirror surface.
Beam Splitter deflects the steady laser beam onto the
mirror.
VII. R ESULTS
We first obtain the PSF by averaging 100 images of the
stationary laser beam acquired with exposure time of 0.1ms
(Fig. 10).
The minimization problem (6) is then solved to find the
gain and phase response at different frequencies. Twenty
frequency points were chosen within [1 − 500] Hz such that
the corresponding periods were multiples of the experimental
fast sampling rate, i.e. the time step at which the FSM was
controlled (0.1ms). The max-norm is used in the minimization, and the continuity constraint is imposed to remove the
alias solution. The resulting frequency response is shown in
Fig. 11.
All measurements were performed with p = 0.5, ωtcam =
π/2 and 4 images per frequency point were averaged.
The comparison of our image-based result with the PSD
result show consistently good match from DC to 500Hz
– an order of magnitude higher than the 30Hz sampling
frequency of the image sensor. Once the frequency response
is obtained, standard identification algorithm may be applied
(e.g., subspace algorithm) to fit a state space model.
Fig. 8. Experimental apparatus: a CCD camera, a laser source (bottom
left), a steerable mirror (right) and a beam splitter.
•
Experimental apparatus representation with optical path.
A two-degree-of-freedom (tip-tilt) FSM is used as the
plant, H , to be identified. It is driven by piezoelectric
actuators and the position is measured by PSD sensors.
The mirror is connected to an xPC Target real-time
control computer with analog PID closed loop control.
The inputs to the system are the commanded angular
deflection in the (x, y) direction. The outputs are the actual angular positions. The PSD measurements serve as
high quality reference for the image based identification.
Focused Laser Beam passes through a beam splitter,
reflects off the FSM surface which deflects it onto the
image sensor. Mirror motion causes the laser beam
to move on the image sensor plane. The laser has
been kept at constant (minimum) power throughout
VIII. C ONCLUSIONS AND F UTURE W ORK
This paper demonstrates that by using the integrative
nature of the image sensor, we are able to identify the
dynamics of a SISO LTI system well beyond the sampling
frequency of the sensor. The procedure we have used so
far is a simple one – sweep across the frequency of input
sinusoids, and use image sensor to identify the output gain
and phase response at each frequency. An alias solution is
always present in the identification problem. We proposed
two methods to disambiguate the solutions.
Our current work involves using a general input instead
of a series of sine wave inputs, analyzing the effect of errors
917
(a)
(b)
Fig. 10. (a) PSF cropped out of the full resolution image. Top right is the measured image, top left the PSF (3). Bottom left and right figures are the
absolute error and its absolute value. The colors are automatically normalized from red (highest value) to blue (lowest). (b) Three dimensional representation
of the superimposed top left and right images in (a). Figure on the right shows the absolute value of the fitting error.
Fig. 11.
SISO case: Identified frequency response of the FSM. Validation PSD measurements and Camera identified data.
in the image kernel Ψ, and the direct identification of the
state space parameters instead of using the intermediate step
of frequency response.
[5] A. Glindemann, S. Hippler, T. Berkefeld, and W. Hackenberg, “Adaptive optics on large telescopes,” Experimental Astronomy 10: 547,
2000.
[6] D. N. Neal, J. Copland, and D. Neal, “Shack-hartmann wavefront sensor precision and accuracy,” Advanced Characterization Techniques for
Optical, Semiconductor and Data Storage Components; Proceedings
of SPIE Vol. 4779, 2002.
[7] T. Bifano, J. Perreault, and P. Bierden, “A micromachined deformable
mirror for optical wavefront compensation,” in Proceedings of the
SPIE, vol. 4124, 2000, pp. 7–14.
[8] W.-G. Chen, N. Nandhakumar, and W. N. Martin, “Image motion
estimation from motion smear-a new computational model,” Pattern
Analysis and Machine Intelligence, IEEE Transactions on, vol. 18, pp.
412–425, 1996.
[9] J. Biemond, R. Lagendijk, and R. Mersereau, “Iterative methods for
image deblurring,” Proceedings of the IEEE, vol. 78, no. 5, pp. 856
–883, May 1990.
[10] M. Bao, W. Alpers, and C. Bruning, “A new nonlinear integral
transform relating ocean wave spectra to phase image spectra of an
along-track interferometric synthetic aperture radar,” Geoscience and
Remote Sensing, IEEE Transactions on, vol. 37, no. 1, pp. 461 –466,
Jan. 1999.
[11] W. P. Wong and K. Halvorsen, “Beyond the frame rate: Measuring
high-frequency fluctuations with light intensity modulation,” Optical
Society of America, 2009.
[12] S. Mishra and J. Wen, “Extracting dynamics from blur,” 50th IEEE
Conference on Decision and Control and European Control Conference, Dec. 2011.
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation through CMMI-1130231. This work was also supported
in part by the Center for Automation Technologies and Systems (CATS) under a block grant from the New York State
Empire State Development Division of Science, Technology
and Innovation (NYSTAR).
R EFERENCES
[1] P. I. Corke, Visual Control Of Robot Manipulators – A Review. World
Scientific, 1994.
[2] F. Chaumette, “Image moments: a general and useful set of features
for visual servoing,” Robotics, IEEE Transactions on, vol. 20, no. 4,
pp. 713 – 723, 2004.
[3] E. N. Malamas, E. G. M. Petrakis, M. Zervakis, L. Petit, and J.-D.
Legat, “A survey on industrial vision systems, applications and tools,”
Image and Vision Computing, vol. 21, no. 2, pp. 171 – 188, 2003.
[4] X. Tao, B. Fernandez, O. Azucena, M. Fu, D. Garcia, Y. Zuo, D. C.
Chen, and J. Kubby, “Adaptive optics confocal microscopy using direct
wavefront sensing,” Optics Letters, Vol. 36, No. 7, Apr. 2011.
918