Download ibr-1 - Computer Sciences User Pages

Image-Based Rendering
• Geometry and light interaction may be
difficult and expensive to model
– Think of how hard radiosity is
– Imagine the complexity of modeling the exact
geometry of carpet (as just one example)
• Image based rendering seeks to replace
geometry and surface properties with images
Texture Mapping
• Replace small scale geometry and surface changes
with an image attached to the surface
• Define a mapping from surface points into a
texture image
• When the properties of that point are required,
look up the pixel in the texture image
• A research field in itself
– Various methods of sampling the texture
– Various ways of combining the samples with direct
lighting calculations
– Ways of arranging and accessing textures in memory
Environment Mapping
• Replace the world around an object with images
• Generally used for fast reflection computations
• Typically, define a cube around the object and
associate images with the inside of each face
• When seeking reflection color, shot ray from surface
point onto cube and take corresponding image point
• Images represent a view of the world from the center
of the object
– Assumes that the world wouldn’t look much different if
viewed from points on the object’s surface
Plenoptic Function
• Describes the intensity of light:
–
–
–
–
passing through a given point, x
in a given direction, (,)
with given wavelength, 
at a given time, t
• Many image-based rendering approaches can be
cast as sampling from and reconstructing the
plenoptic function
• Note, function is generally constant along
segments of a line (assuming vacuum)
Image-Based Rendering
• Aim: Generate an image from a desired view
using existing images from other views
– May or may not know the viewing parameters for
the existing images
– Existing images may be photographs or computer
generated renderings
A Plethora of Approaches
• Methods differ in many ways:
–
–
–
–
–
The range of new viewpoints allowed
The density of input images
The representation for samples (known images)
The amount of user help required
The amount of additional information required
(such as intrinsic camera parameters)
– The method for gathering the input images
Movie-Map Approaches
• Film views from fixed locations, closely spaced,
and store on video-disc (would be DVD now)
• Allow the user to jump from location to location,
and pan
• Appropriate images are retrieved from disk and
displayed
• No reprojection – just uses nearest existing sample
• Still used in video games today, but with computer
generated movies
Quicktime VR (Chen, 1995)
• Movie-maps in software
• Construct panoramic images by stitching
together a series of photographs
– Semi automatic process, based on correlation
• Scale/shift images so that they look most alike
– Works best with >50% overlap
• Finite set of panoramas – user jumps from
one to the other
View Interpolation (Chen and Williams, 1993)
• Input: A set of synthetic images with known depth
and camera parameters (location, focal length, etc)
• Computes optical flow maps related each pair of
images
– Optical flow map is the set of vectors describing where
each point in the first image moves to in the second
image
• Morphs between images by moving points along
flow vectors
• Intermediate views are “real” views only in special
cases
View Morphing (Seitz and Dyer, 1997)
• Uses interpolation to generate new views such that
the intermediate views represent real camera motion
• Observation: Interpolation gives correct intermediate
views if the initial and final images are parallel
views
View Morphing Process
• Basic algorithm:
– Pre-warp input images to get them into parallel views
– Interpolate for intermediate view
– Post-warp to get final result
• User specifies a camera path that rotates and translates initial
camera onto final camera
• Requires knowledge of projection matrices for the
input images
– Found with vision algorithms. User may supply
correspondences
• Intermediate motion can be specified by giving
trajectories of four points
Plenoptic Modeling (McMillan and Bishop, 1995)
• Input: Set of panoramic images gathered by
panning a video camera about its vertical optical
axis on a tripod
• The algorithm:
– Determines the properties of each camera and registers
the images associated with each pan
– Determines the relative locations of each camera
– Determines the depth of each point seen by multiple
cameras
– Generates new views by reconstructing the plenoptic
function from the available samples
Fitting Each Camera
• Problem:
– Given several images all taken with the same camera rotated
about its optical center
– Determine the camera parameters
– Determine the angle between each image
• Approach:
– Multi-stage optimization
– First stage estimates angle between images and focal length
– Second stage refines and gets remaining parameters
• When done, can map a pixel in one image to its
correct position in any other images
Locating the Cameras
• Given cylindrical projections (panoramic images)
for two cameras
• User identifies 100-500 tie-points (points seen in
both images)
• Each tie-point defines two rays – one from the
center of each camera through the tie-point
– These rays should intersect at the world location for the
point
• Minimization step finds the camera locations and
some other parameters that minimize the
perpendicular distance between all the rays
Determining Disparity
• The minimization algorithm gives us the world
location of the tie-points, but what about the rest
of the points in the image?
• Use standard computer vision techniques to find
the remaining disparities
– Disparity is the angular difference between the
locations of a point in two images
– Directly related to the depth of the point
• Makes heavy use of the epipolar constraint
The Epipolar Constraint
• The location of a point in one image constrains it
to lie on a ray in space passing through the camera
and image point
• This ray projects to a curve in the second image
– Line for planar projection
– Sine curve for cylindrical projection
• Since the point must lie on the ray in world space,
it must lie on the curve in the second image
• Reduces the search for correspondences to a linear
one along the line
Reconstructing a New View
• The disparity from a known cylinder pair can be
transferred to a new cylinder, and then reprojected
onto a plane (the image)
– Can all be done in one step
• Have to resolve occlusion problems
– Two points from the reference image could map to the
same point in the output image
– Solution: Define a fill ordering that guarantees correct
occlusion (an important contribution of this paper)
• Also have to fill holes
Summary
• Image-based rendering obviously relies heavily on
computer vision techniques
– Particularly: Depth from stereo
– The problem is very hard with real images
– These techniques are not perfect!
• Sampling remains a problem
– Images tend to appear blurry
– Relatively little work on reconstruction algorithms