Download Dynamic Scene Generation, Multimodal Sensor Design, and

Dynamic Scene Generation, Multimodal Sensor Design, and
Target Tracking Demonstration for Hyperspectral/Polarimetric
Performance-Driven Sensing
Michael D. Presnarab, Alan D. Raisanenc, David R. Pogorzalaa, John P. Kerekesa, Andrew C. Riced1
a
Chester F. Carlson Center for Imaging Science
Rochester Institute of Technology
54 Lomb Memorial Drive
Rochester, NY 14623-5604
b
Air Force Institute of Technology
2950 Hobson Way
Wright-Patterson AFB, OH 45433-7765
c
Semiconductor and Microsystems
Fabrication Laboratory
Rochester Institute of Technology
82 Lomb Memorial Drive
Rochester, NY 14623-5604
d
Numerica Corporation
2661 Commons Blvd., Suite 210
Beavercreek, OH, USA 45431-3704
ABSTRACT
Simulation of moving vehicle tracking has been demonstrated using hyperspectral and polarimetric imagery (HSI/PI).
Synthetic HSI/PI image cubes of an urban scene containing moving vehicle content were generated using the Rochester
Institute of Technology’s Digital Imaging and Remote Sensing Image Generation (DIRSIG) Megascene #1 model.
Video streams of sensor-reaching radiance frames collected from a virtual orbiting aerial platform’s imaging sensor were
used to test adaptive sensor designs in a target tracking application. A hybrid division-of-focal-plane imaging sensor
boasting an array of 2×2 superpixels containing both micromirrors and micropolarizers was designed for co-registered
HSI/PI aerial remote sensing. Pixel-sized aluminum wire-grid linear polarizers were designed and simulated to measure
transmittance, extinction ratio, and diattenuation responses in the presence of an electric field. Wire-grid spacings of 500
[nm] and 80 [nm] were designed for lithographic deposition and etching processes. Both micromirror-relayed
panchromatic imagery and micropolarizer-collected PI were orthorectified and then processed by Numerica
Corporation’s feature-aided target tracker to perform multimodal adaptive performance-driven sensing of moving
vehicle targets. Hyperspectral responses of selected target pixels were measured using micromirror-commanded slits to
bolster track performance. Unified end-to-end track performance case studies were completed using both panchromatic
and degree of linear polarization sensor modes.
Keywords: Synthetic scene simulation, DIRSIG, micromirror, wire-grid linear polarizer, multi-object spectrometer,
adaptive multimodal sensor, performance-driven sensing, feature-aided target tracking
1. INTRODUCTION
1.1 Motivation and objectives
Researchers at the Rochester Institute of Technology (RIT) and Numerica Corporation are working together in the area
of integrated multimodal sensing, processing, and exploitation in support of a U.S. Air Force Office of Scientific
Research (AFOSR) Discovery Challenge Thrust (DCT)1. Adaptive multimodal electro-optical (EO) sensor concepts are
being modeled and simulated in a “performance-driven” context by coupling together hyperspectral/polarimetric image
Further author information: (Send correspondence to J.P.K.)
J.P.K.: Email: [email protected], Telephone: 1 585 475-6996
Polarization: Measurement, Analysis, and Remote Sensing IX, edited by David B. Chenault, Dennis H. Goldstein,
Proc. of SPIE Vol. 7672, 76720T · © 2010 SPIE · CCC code: 0277-786X/10/$18 · doi: 10.1117/12.849469
Proc. of SPIE Vol. 7672 76720T-1
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
synthesis, multimodal sensor hardware models, and data exploitation algorithms. The team’s goal is to enhance deeplyhidden, high-clutter target recognition by optimally exploiting the phenomenology of multimodal target scene signatures.
1.2 Approach
The approach taken by the authors includes three major research veins, as shown below in Figure 1. Dynamic scene
generation of an urban environment that includes moving vehicle targets sensed by a moving aerial platform is used to
render video frames in the form of hyperspectral/polarimetric image cubes. An adaptive multimodal sensor model
processes these input frames while incorporating realistic sampling, optical aberrations, and scattering models to produce
orthorectified imagery and selected target spectra for exploitation by a target tracker. The target tracker evaluates the
stream of either grayscale pan or degree of linear polarization (DoLP) imagery using change detection and queries the
sensor for hyperspectral imagery (HSI) pixels where moving targets are potentially located. Image quality, spectral
quality, and track performance metrics are computed on a frame-by-frame basis to evaluate effects of modeling
parameter changes. The recent addition of the polarimetric modality into all three research veins has been a major
advancement over a previous simulation2 that incorporated synthetic hyperspectral imagery, a multi-object spectrometer
model, and an HSI-aided pan-imagery target tracker.
Figure 1. Unified end-to-end simulation flowchart of adaptive multimodal performance-driven sensing
1.3 Overview of paper
We first provide an introduction to the dynamic scene modeling tools used as inputs in the simulation of a performancedriven sensor. Next, we present the modeling efforts of a hybrid array of micromirrors and micropolarizers as a primary
component of a performance-driven adaptive sensor. Finally, conversion of a target tracker to use either pan or DoLP
imagery for track initiation of moving vehicles and its simulation results is discussed.
Proc. of SPIE Vol. 7672 76720T-2
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
2. SPECTROPOLARIMETRIC DYNAMIC SCENE MODELING
2.1 Spectropolarimetric Video Stream Synthesis
The Digital Imaging and Remote Sensing (DIRS) laboratory within the RIT Center for Imaging Science is the home of
the DIRS Image Generation (DIRSIG) model. DIRSIG is a first-principles, physics-based synthetic image simulation
software package3. It has the ability to produce imagery in a variety of modalities, including multispectral, hyperspectral,
polarimetric, and LIDAR in the visible through the thermal infrared regions of the electromagnetic spectrum. DIRSIG
performs ray-tracing calculations of “light” emanating from realistic models of electromagnetic sources (e.g., sun, moon,
skydome), propagating through an absorptive and scattering atmosphere, reflecting off of virtual 3-D objects with
polarized bi-directional reflectance distribution functions (pBRDFs) specific to material classes, and finally arriving at a
virtual imaging detector with a specific array size and optical field of view. Co-registered detector channels can be
specified to output a variety of spectral radiance imagery, including monochromatic, panchromatic, multispectral, and
hyperspectral. Bundled with DIRSIG is Megascene #1, a high-fidelity synthetic recreation of natural and man-made
objects comprising a metro region of Rochester, NY4. Over 200 moving vehicles have been added to the baseline scene
using the Simulation of Urban MObility (SUMO) traffic model.
Sensor-reaching spectral radiance imagery contains pixels in the form of Lλ [W/cm2/sr/μm], which is a
combination of direct reflected radiance, diffuse reflected radiance, and path radiance5. Each of these pixels can be
further specified by a Stokes vector, Lλ = [Lλ0 Lλ1 Lλ2 Lλ3]T whose four components represent total spectral radiance, 0°
vs. 90° linearly polarized spectral radiance, +45° vs. -45° linearly polarized spectral radiance, and right-handed vs. lefthanded circularly polarized spectral radiance, respectively6. The Stokes vector obeys the following inequality:
W
(1)
cm · sr · µm
When performing remote sensing of terrestrial objects, Lλ3 ≈ 0. The amount of partial polarization of an individual
pixel’s radiance can be represented by the DoLP, while the electric field’s oscillation direction of the completely
polarized fraction can be represented by the Angle of Polarization (AoP) in terms of spectral radiance Stokes vector
parameters:
·
(2)
1
deg
(3)
tan
2
A single DIRSIG virtual detector channel can output any spectral wavelength as a “4-band” Stokes vector
image when using the polarized version of the Moderate Spectral Resolution Atmospheric Transmittance Algorithm
(MODTRAN4-P) to simulate atmospheric scattering effects on polarization7. DIRSIG also has the capability to output
unpolarized radiance bands that have been preprocessed through ideal linear polarizer filters at any orientation (e.g., 0°,
60°, & 120°) with respect to the sensor array. These bands can be converted into Stokes vector bands using Fesenkov’s
method, followed by a conversion into a DoLP image whose pixel values range between 0 (random polarization) and 1
(complete polarization). Calibration panels simulating glass, paint, and Lambertian reflectors can also be placed in the
scene to confirm radiometric and polarimetric effects. A comparison of RGB, pan, and DoLP oblique synthetic imagery
looking toward the sun on a clear day is shown below in Figure 2. Here, the higher DoLP returns represent glass-topped
buildings, house windows, and vehicles with glossy paint.
Proc. of SPIE Vol. 7672 76720T-3
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
b
a
c
Figure 2. Synthetic oblique aerial sensor imagery at a 50° sensor declination angle from nadir
(a) RGB modality, (b) Pan-VNIR modality, and (c) DoLP-VNIR modality
2.2 Orthorectification Process and Uncertainty Effects
A navigation system’s precise position and orientation measurements are required for orthorectification of
oblique imagery on a moving platform. The tilted detector plane is related to the ground plane through a projective
transformation mapping, also known as collinearity8. Each individual frame captured by an oblique-viewing detector can
be digitally registered so that each pixel location (x,y) [m] on the array directly maps to an East-North-Up (ENU)
location (X,Y,Z) [m] on the ground. The sensor’s focal length, f [m], the camera lens center’s ENU coordinates,
(XL,YL,ZL) [m], and the detector array’s center point in the tilted detector frame, (x0,y0) [m], are all constants for a single
image frame. Orthorectification is accomplished by sweeping across all possible ground coordinate (X,Y) [m] pairs in the
local area (e.g., the entire synthetic tile or just an area of interest) at a constant ground sample distance (GSD),
meanwhile inserting the terrain’s elevation, Z [m], from a surveyed digital elevation map (DEM). If a pixel location (x,y)
[m] computed via the following equation pair falls within the detector array’s boundaries, then that location’s nearestneighbor pixel spectrum is chosen for injection into the resultant orthorectified image at location (X,Y):
m
(4)
m
(5)
Here, the m coefficients represent elements of the rotation matrix from the local-level ENU frame to the detector frame
in right-hand heading-roll-pitch order:
·
·
1
0
0
0
cos
sin
0
sin
cos
·
cos
0
sin
0
1
0
sin
0
cos
·
cos
sin
0
sin
cos
0
0
0
1
A comparison of RGB, pan, and DoLP pseudo-nadir imagery after orthorectification is shown below in Figure 3.
a
b
c
Figure 3. Orthorectified imagery for subset area of interest with fixed boundaries
(a) RGB modality, (b) Pan-VNIR modality, and (c) DoLP-VNIR modality
Proc. of SPIE Vol. 7672 76720T-4
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
(6)
Platform uncertainty due to GPS/inertial navigation system noise can be simulated using the location and Euler
angle RMS error specifications of a commercial airborne navigator. Platform uncertainty due to jitter at specific
harmonic frequencies can also be simulated. These uncertainties translate the camera and rotate its optical axis about the
stare point and cause incorrectly registered frames during the orthorectification process. Simulation of urban aerosols
can also be activated by MODTRAN4-P to increase scattering of rays in all wavelengths for a more realistic and
challenging remote sensing scenario.
3. HYBRID ADAPTIVE SENSOR MODELING
3.1 MOEMS Device Simulation
Simulation of micro-opto-electromechanical systems (MOEMS) devices has been accomplished by the RIT
Semiconductor and Microsystems Fabrication Laboratory via 3-D modeling tools and the COMSOL Multiphysics
modeling and simulation tool. Finite element analysis has taken into consideration the electromagnetic, mechanical,
thermal, and DC electrical effects when operating these devices under VNIR aerial remote sensing conditions. Agile
micromirror device modeling has evolved for the purpose of relaying light from selected pixels within an imaged scene
toward a spectrometer. Partially transmissive micromirrors have been modeled as tunable Fabry-Perot etalons to measure
pixel spectra directly underneath the MOEMS device without the need for a spectrometer relay channel. Micromirror
actuators allow three degree-of-freedom movement: translation in the optical axis (i.e., piston), and rotation about two
orthogonal axes (i.e., tip and tilt). A 2×2 superpixel that includes a micromirror and three pixel-sized wire-grid linear
polarizer device models oriented at 0°, 60°, and 120° angles has been modeled in a hybrid sensor array to obtain coregistered pan and polarimetric imagery as shown below in Figure 4.
a
b
c
Figure 4. Novel MOEMS pixel-sized optical device modeling concepts
(a) Agile micromirror with Fabry-Perot etalon functionality
(b) Superpixel concept of 3 linear polarizers and an agile micromirror
(c) Ray-tracing example of micropolarizer transmission and micromirror reflection
Wire grid polarizer arrays were simulated using the 2-D in-plane hybrid electromagnetic wave application mode
of the COMSOL Multiphysics RF modeling module, as shown in Figure 5(a). Two different wire-grid geometries were
modeled. A relatively coarse d = 500 [nm] period grid with 100 [nm] wide lines (i.e., 20% duty cycle or fill factor) that
was 100 [nm] thick was simulated as a geometry that could be fabricated in RIT’s Semiconductor and Microsystems
Fabrication Laboratory (SMFL) on conventional i-line (i.e., 365 [nm]) or deep ultraviolet (DUV) lithography equipment
with a moderate amount of effort. A d = 80 [nm] period grid with 20 [nm] wide lines (i.e., 25% duty cycle or fill factor)
that was 20 [nm] thick was chosen as a geometry that could be fabricated using much more aggressive lithographic
techniques such as electron beam or high numerical aperture immersion. The metal grid was specified to be aluminum
with 35.6×106 [S/m] conductivity and a square aperture size of 10 [μm] per side, yielding 20 lines for d = 500 [nm] and
125 lines for d = 80 [nm]. The modeling space was defined as 10 [μm] in total depth perpendicular to the wire grid, and
15 [μm] per side in the plane of the grid, with a solid metal plate surrounding the central 10 [μm] gridded opening. A 1
[μm] “Perfectly Matched Layer” was defined surrounding the modeling space, and scattering boundary conditions were
defined at the outer boundary.
Proc. of SPIE Vol. 7672 76720T-5
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
Input linearly polarized electromagnetic plane waves were excited at the leftmost boundary of the modeling
space, and output transmitted electromagnetic power was extracted at the rightmost boundary after propagating through
the metal grid. The relative intensities of the electric field (ε, [V/m]) and magnetic field (ℋ, [V⋅s/m2]) incident plane
waves were specified in vector component directions that were parallel and perpendicular to the wires to produce the
polarization condition desired. The AoP, or ψ, is defined here to be the angle between the incident electric field’s
oscillation direction and the transmission axis of the wire grid array that is perpendicular to the wires. The incident field
amplitudes perpendicular to the wires were thus defined as εx = ε0⋅cosψ and ℋx = ℋ0⋅sinψ. The field amplitudes parallel
to the wires were defined as εy = ε0⋅sinψ and ℋy = ℋ0⋅cosψ. Gridding of the COMSOL modeling space was refined until
the obtained solutions converged and were unaffected by further refinement. The input wave power was determined by
removing the wire grid and repeating the propagation analysis, allowing the input wave to be integrated without
interference by reflections or diffraction from the grid. Cross-sectional energy density for the horizontally polarized ψ =
0° case is shown in Figure 5(b), while oscillating electric field magnitude for the vertically polarized ψ = 90° case is
shown in Figure 5(c).
10 [μm]
Solid
Metal
Plate
Input
Wave
10 [μm]
Al
WireGrid
Input
Wave
Output
Wave
Output
Wave
Propagation
Propagation
x
z
a
15 [μm]
Perfectly
Matched
Layer
b
c
y
Figure 5. 2-D cross-section of pixel-sized aluminum wire-grid polarizer with 20 wires spaced 500 [nm] apart
(a) Aluminum wire grid shown in modeling space with dimensions
(b) Energy density magnitude when horizontally polarized electric field oscillates perpendicular to the wires (AoP = 0°)
(c) Electric field magnitude when vertically polarized electric field oscillates parallel to the wires (AoP = 90°)
This propagation analysis process was repeated for both the d = 500 [nm] and d = 80 [nm] cases at 211 discrete
wavelengths and 10 angles of polarization. Wavelength increments of Δλ = 10 [nm] were simulated between λmin = 0.4
[μm] and λmax = 2.5 [μm]. Angle of polarization increments of Δψ = 10° were simulated between ψmin= 0° and ψmax=
+90°. A transmission matrix for each polarizer was obtained according to the following boundary irradiance ratio:
,
(7)
·
,
The transmission results are plotted below in Figure 6. Both polarizers have a maximum transmission of approximately
0.7 at most wavelengths for horizontally polarized light. The d = 500 [nm] spacing polarizer has very limited polarizing
performance at wavelengths below 1.5 [μm], leaking most cross-polarized energy through its wires. Its transmission
response contains a point of convergence around λ = 0.5 [μm], attributed to the size and shape of the polarizer’s
individual wires designed with a square instead of circular cross-section. The d = 80 [nm] spacing polarizer boasts
respectable performance across the VNIR spectrum from 0.4 to 2.5 [μm].
,
Proc. of SPIE Vol. 7672 76720T-6
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
a
b
Figure 6. Pixel-sized wire-grid polarizer transmission vs. VNIR wavelengths for 10 AoP increments
(a) Spacing of d = 500 [nm], (b) Spacing of d = 80 [nm]
The transmission responses of the horizontal (ψ = 0°) and vertical (ψ = 90°) polarization input cases can be
further used to compute the extinction ratio, re(λ), as follows:
,
, 0°
(8)
·
,
, 90°
This metric represents the ratio of horizontally polarized to vertically polarized light that passes through from a
randomly polarized input beam. The extinction ratio results are plotted below in Figure 7.
a
b
Figure 7. Pixel-sized wire-grid polarizer extinction ratio vs. VNIR wavelengths
(a) Spacing of d = 500 [nm], (b) Spacing of d = 80 [nm]
The diattenuation response, D(λ), can be computed via the horizontal and vertical polarization transmission
responses as follows:
,
,
, 0°
, 90°
(9)
·
,
,
, 0°
, 90°
This metric represents the polarizing efficiency of the device, and is related to the extinction ratio according to
1 / 1
. Figure 8 shows that the d = 500 [nm] spacing is characterized by poor diattenuation
throughout most of the VNIR spectrum, while the d = 80 [nm] spacing boasts D > 95% throughout the entire VNIR.
a
b
Figure 8. Pixel-sized wire-grid polarizer diattenuation vs. VNIR wavelengths
(a) Spacing of d = 500 [nm], (b) Spacing of d = 80 [nm]
The transmission responses of each polarizer at each specific wavelength can be normalized and plotted against
AoP, which attenuates the incoming beam based on the absolute value of AoP. Transmission through ideal polarizers
falls off with cos2(ψ), and this is shown for comparison with the 211 simulated wavelengths in Figure 9 below.
Proc. of SPIE Vol. 7672 76720T-7
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
0.4 µm ,
0.4 µm , 0°
2.5 µm ,
2.5 µm , 0°
Ideal cos2(ψ) Falloff
a
b
Figure 9. Pixel-sized wire-grid polarizer cosine-squared falloff for 211 VNIR wavelengths vs. AoP
(a) Spacing of d = 500 [nm], (b) Spacing of d = 80 [nm]
The transmission response vs. wavelength can be applied on a pixel-by-pixel and band-by-band basis when
input Stokes vector spectral radiance imagery is processed. The input spectral radiance Stokes vector entering a polarizer
with its polarizing axis aligned horizontally can be split into an incoherent superposition of completely unpolarized and
completely (linearly) polarized light:
· ,
· ,
W
0
,
(10)
,
cm
·
sr · µm
0
,
0
0
can be decomposed into two equal quantities of linearly polarized light having
1
,
The Stokes vector
,
AoPs of ψ = 0° and ψ = 90°, respectively. The total spectral radiance exiting the polarizer is thus attenuated by
, 0°
, 90° /2. The Lλ,1 output component will be positive (i.e., horizontally polarized) and attenuated relative
to the total Lλ,0 output component by the diattenuation, D(λ). In the case of an ideal polarizer (i.e.,
, 90°
0 and
D(λ) = 1), all output light will be horizontally polarized and therefore (Lλ,0)output = (Lλ,1)output. Further, due to the
symmetric transmission response for ±45° linearly polarized light, (Lλ,2)output = 0.
1
·
2
,
1
·
2
The Stokes vector
,
, 0°
, 90°
· 1
·
,
, 0°
, 90°
· 1
·
,
W
cm · sr · µm
0
0
, 0°
, 90°
(11)
1
· 1
·
,
·
was defined in accordance with the equality
0
0
·
,
,
,
,
where DoLP is a property of the entire input spectral radiance Stokes vector,
. The polarizer will transmit a
fraction
,
of the completely polarized total spectral radiance according to the results shown in Figure 6 for the
particular AoP with respect to the grid’s polarizing axis. As in the previous unpolarized Stokes vector, the Lλ,1 output
component will be positive but attenuated by D(λ) relative to the total Lλ,0 output component.
Proc. of SPIE Vol. 7672 76720T-8
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
1
W
(12)
0
cm · sr · µm
0
The previous two equations can be summed to obtain the spectral radiance Stokes vector exiting the polarizer.
This procedure must be used on DIRSIG-rendered Stokes vector bands in order to incorporate the effects of nonideal
wire-grid polarizers. Applying a transmission response to an ideal polarizer output band will not simulate any
diattenuation effects. If DIRSIG is configured to output radiance bands that exit ideal linear polarizers (e.g., 0°, 60°, and
120° orientations), leakage of cross-polarized light that reduces remotely sensed Lλ,1 and DoLP cannot be simulated
without a conversion into a Stokes vector form. Slight variations of these equations using a relative angle between the
input AoP in the Stokes vector’s reference frame and the orientation of the polarizing axis (e.g., 60° and 120°
orientations) will produce appropriate output Stokes vectors with nonzero , components.
,
,
·
·
,
Figure 10 shows a visual example of the polarizer operation on an incoming monochromatic electric field.
Here, the polarizer transmission axis is the horizontal axis (x), although the polarizer orientation could be rotated atop
specific detector pixels to achieve sensitivity to other angles (e.g., 0°, 60°, 120°). At λ = 1.5 [μm], the d = 500 [nm]
spacing leaks through a significant 6.9% of the εy component, adding to the irradiance seen on a detector pixel placed
downstream of the polarizer. On the other hand, the d = 80 [nm] spacing leaks through a negligible 0.2% of the εy
component.
a
b
Figure 10. Pixel-sized (10 [μm]) wire-grid polarizers operating on an incoming electric field wave with λ = 1.5 [μm] & AoP = 60°
(a) Spacing of d = 500 [nm], duty cycle of 20%, and aluminum film thickness of 100 [nm]
(b) Spacing of d = 80 [nm], duty cycle of 25%, and aluminum film thickness of 20 [nm]
3.2 Hybrid Adaptive Spectropolarimeter Modeling
Incorporation of various MOEMS-based optical devices into adaptive sensor designs is currently under way at RIT. A
multi-object spectrometer (MOS) is one such adaptive sensor, whose MOEMS micromirrors at the primary focal plane
selectively reimage small subsets of the scene onto either a panchromatic imaging detector or a spectrometry detector. In
this manner, hyperspectral imagery (HSI) aiding has been tied into a target tracker for an end-to-end simulation.
Replacing the micromirror array with a superpixel array of micromirrors and micropolarizers adds the polarimetric
modality to the MOS-based adaptive sensor, although spatial resolution is reduced by a factor of 2. The optical design of
this sensor was optimized to minimize aberrations in the VNIR, and a 3-D sensor model was constructed via optical
modeling software. The micromirrors selectively vector light through a relay to a pan imager or through a collimator and
diffraction grating to a spectrometer2.
Proc. of SPIE Vol. 7672 76720T-9
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
Figure 11. Hybrid adaptive sensor 3-D optical model
4. PERFORMANCE-DRIVEN ALGORITHM
4.1 Feature-Aided Target Tracking
Numerica’s Algorithm simuLator for Tracking and Observations (ALTO) produces measurement data, hosts advanced
tracking algorithms, and supports performance evaluation9. A particular distribution of ALTO was prepared to study
conceptually derived sensor instruments with synthetic hyperspectral DIRSIG imagery. Figure 12 shows a single pan
observation image frame from a video set when staring nadir above Megascene #1 Tile #1 with all trees initially
removed to achieve maximum tracking performance. Here, ALTO commands the multi-object spectrometer (MOS)
based sensor to query HSI spectra from selected pixels throughout the scene in a one pixel-per-column fashion while
simultaneously performing frame-to-frame change detection for target track assignment.
Figure 12. Target tracking within nadir pan observation images with no occlusions on a 660 [m] × 420 [m] ground area
(a) HSI-pixels queried by the target tracker at a single time epoch
(b) Labeled targets under track with 1-second position predictions
Oblique image streams captured from a moving aerial platform have also been input to ALTO using an orthorectification
process. The MOEMS superpixel modification to the original MOS-based sensor design has allowed for sensor
processing of hyperspectral/polarimetric image cubes rendered by DIRSIG. ALTO was modified to compute and process
DoLP imagery instead of pan imagery used for change detection. The ALTO user has the choice to differentiate moving
vehicles the background based on either pan or DoLP imagery. Figure 13 shows an orthorectified DoLP image from a
single observation video frame with ALTO-computed tracks overlaid. Due to the moving platform’s 50° declination
angle from nadir and its present detector heading facing generally toward the sun, DoLP works well in differentiating
moving vehicles with shiny paint and glass materials from the background. In other cases where the sensor angles or
atmospheric conditions are not optimal to achieve high DoLP returns, a boost parameter can help amplify low values of
DoLP across the scene and improve grayscale image contrast. However, image-wide boosting of DoLP has the potential
of generating many false alarm tracks due to noise amplification.
Proc. of SPIE Vol. 7672 76720T-10
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
Figure 13. Target tracking within orthorectified 50° oblique DoLP observation images
(a) DoLP observation image with labeled tracks
(b) 5× boosted and thresholded DoLP observation image with labeled tracks and many false alarms
4.2 Target Tracking Parameter Trade Studies and Metrics
A unified approach to modeling and simulation provides the ability to vary sensor design parameters and determine
performance effects in a certain application such as target tracking. Trade studies can be accomplished by designing
synthetic image cubes, sensor models, and producing target tracker input imagery after changing one or more design
parameters. Parameters changed in this project thus far are listed below in Table 1.
Table 1. Design Parameters Varied for Pan & DoLP-Initiated Target Tracking Trade Study
Design Parameter
Cases/Values Tested
Spectral and Polarimetric Wavelength Region
0.4 to 1.0 [μm] & 0.4 to 2.5 [μm]
Spectral Bandwidths
0.01, 0.025, 0.05, 0.1 [μm] bandwidths
Sensor Clockrate
2 & 10 [Hz]
Time of Day
12 PM, 3 PM, 5 PM
Occlusions within Base Scene
Megascene #1 Tile #1 with and without trees
Relative Sensor Azimuth from the Sun
Incremental captures in a full circle
Sensor Declination Angle
0°, 20°, & 50° from Nadir
Sensor Elevation above Ground Stare Point
3000, 2819, & 1928 [m] AGL corresponding to dec. angles
Single MOEMS Device Pixel Pitch
17, 10.8, & 8.5 [μm] square (i.e., double for a superpixel)
Nearest-Neighbor Orthorectification GSD
0.75 & 1.5 [m]
Spatial Oversampling
1× & 3×
Atmospheric Aerosol Content
Clear atmosphere; hazy atmosphere with urban aerosols
Platform Position Uncertainty
Perfect flight path; navigation system noise; aircraft jitter
The “track completeness” metric can determine the performance of the target tracker given a stream of dynamic
imagery rendered using a set of design parameters. The position state of each track identified by the target tracker is
compared with the global vehicle movement truth data to determine validity. This metric is the ratio of the number of
valid tracks that are indeed moving vehicles to the total number of moving vehicles within the area of interest that should
be tracked under perfect conditions. This metric is computed at each time epoch and plotted against simulation time.
Track Completeness
# of Valid Tracks
# of "Should" Tracks
(13)
The “track purity” metric is an indicator of whether a valid track stays with a single vehicle or skips to another
object for one or more time epochs. This metric is reported after comparing the tracks with each individual truth vehicle
within the area of interest during the time of simulation. An aggregate track purity metric averages all vehicle-specific
track purities to provide a single scalar number as a simulation-wide performance indication.
Track Purity
# of epochs a valid track maintained the same truth vehicle
Total # of epochs in a valid track
(14)
Besides these metrics, receiver operating characteristic (ROC) curves can be generated to demonstrate the relationship
between the valid tracks (i.e., detections) and invalid tracks (i.e., false alarms) given a set of design parameters.
Proc. of SPIE Vol. 7672 76720T-11
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms
5. SUMMARY
A unified modeling and simulation approach has led to advances in dynamic scene simulation, optical sensor design, and
target tracking. Synthetic digital image cubes have been utilized for sensor application trade studies without having to
build and fly a sensor design. An optical sensor that can switch between imaging modalities depending on conditions has
been proven to pick out objects of interest that stand out from their backgrounds either spectrally or polarimetrically.
Finally, a target tracker that can pick out moving objects within grayscale pan imagery using change detection has been
modified to use DoLP imagery to track moving objects made up of highly polarizing reflective material surfaces.
6. ACKNOWLEDGMENTS
This material is based on research sponsored by the Air Force Office of Scientific Research (AFOSR) under agreement
number FA9550-08-1-0028 (AFOSR-BAA-2007-08). The U.S. Government is authorized to reproduce and distribute
reprints for Governmental purposes notwithstanding any copyright notation thereon.
Computational resources were provided by the RIT Research Computing and DIRS groups. Scene simulation
was accomplished using DIRSIG version 4.3.0, developed at RIT (http://www.dirsig.org). Polarized atmospheric
simulation was accomplished using the experimental MODTRAN4-P version, obtained from the Air Force Research
Laboratory Space Vehicles directorate. Traffic simulation was accomplished using SUMO version 0.10.1, developed by
the German Aerospace Centre (http://sumo.sourceforge.net/). Optical modeling of sensor components was accomplished
using OSLO Premium Edition revision 6.4.6, donated by Lambda Research Corporation (http://www.lambdares.com) for
thesis research. FRED Optimum version 9.10, produced by Photon Engineering LLC (http://www.photonengr.com) was
used to model the micromirror arrays and simulate scattering. COMSOL Multiphysics®, developed by COMSOL AB
(http://www.comsol.com) was used to model micromirror and micropolarizer design concepts. Carrara 7
(http://www.daz3d.com/carrara/) was used for 3-D modeling of the micropolarizers and micromirrors. Target tracking
was accomplished using ALTO (http://www.numerica.us).
DISCLAIMER
The views expressed in this article are those of the authors and do not reflect the official policy or position of the United
States Air Force, the Department of Defense, or the U.S. Government.
REFERENCES
[1] Air Force Office of Scientific Research, “Broad Agency Announcement (BAA), Discovery Challenge Thrusts
(DCTs),” AFOSR-BAA-2007-08, Arlington, Virginia (2007).
[2] Kerekes, J. P., Presnar, M. D., Fourspring, K. D., Ninkov, Z., Pogorzala, D. R., Raisanen, A. D., Rice, A. C.,
Vasquez, J. R., Patel, J. P., MacIntyre, R. T., and Brown, S. D., “Sensor modeling and demonstration of a multiobject spectrometer for performance-driven sensing,” Proc. SPIE 7334, 73340J (2009).
[3] Schott, J. R., Brown, S. D., Raqueño, R. V., Gross, H. N. and Robinson, G., “An advanced synthetic image
generation model and its application to multi/hyperspectral algorithm development,” Canadian Journal of Remote
Sensing 25(2), 99-111 (1999).
[4] Ientilucci, E. J. and Brown, S. D., “Advances in wide area hyperspectral image simulation,” Proc. SPIE 5075, 110–
121 (2003).
[5] Schott, J. R., [Remote Sensing: The Image Chain Approach], Oxford University Press, New York, 2 ed. (2007).
[6] Schott, J. R., [Fundamentals of Polarimetric Remote Sensing], SPIE Press, Bellingham, Washington (2009).
[7] Pogorzala, D., Brown, S., Messinger, D. and Devaraj, C., "Recreation of a nominal polarimetric scene using
synthetic modeling tools", Proc. SPIE 6565, 65650Z (2007).
[8] Wolf, P. R. and Dewitt, B. A., [Elements of Photogrammetry with Applications in GIS], McGraw-Hill, Boston, 3
ed. (2000).
[9] Rice, A. C., Vasquez, J. R., Kerekes, J., and Mendenhall, M. J., “Persistent hyperspectral adaptive multi-modal
feature-aided tracking,” Proc. SPIE 7334, 73340M (2009).
Proc. of SPIE Vol. 7672 76720T-12
Downloaded from SPIE Digital Library on 28 May 2010 to 129.21.58.33. Terms of Use: http://spiedl.org/terms