Download IBR-slides.pdf

Traditional 3D Graphics
„
Input:
„
Image-Based Rendering
„
„
Process:
„
„
„
Computer Vision
Input:
„
„
„
„
„
„
„
„
A model
Input:
„
„
„
Output:
„
Rendered images
Image-Based Rendering (IBR)
Rendered images
Why Does This Make Sense?
„
Real images
Analysis + reprojection
Use Vision to reconstruct a model
Use Graphics to render
Output:
„
Process:
„
Real images + assumptions
Process:
„
Isn’t There a Simpler Way?
„
Input:
„
Analysis
Output:
Rendered images
Photorealistic Rendering (Z)
Process:
„
„
Real images
Assumptions
Simulation
Display
Output:
„
„
Geometric description
Physical properties
„
Photorealistic rendering computes the radiance
arriving into the camera through different
locations on the image plane.
The “Plenoptic function”: the radiance leaving
every point in every direction at any moment in
time:
P (x, y,z,θ, φ, λ,t )
Images as Plenoptic Samples
„
An image is just a collection of rays: a sample of
the Plenoptic function!
Image-Based Rendering
„
3D Warping of Planar Images
„
Reconstruct new images from Plenoptic
samples.
Planar Warping Equation
Consider the standard pinhole camera
model commonly used in CG:
G
G
C1 + t1 P1 x1 = C 2 + st2 P2 x2
G
G
t2 P2 x2 = C1 − C 2 + t1 P1 x1
G
X = C + t Px
⎡u x
P = ⎢⎢u y
⎢⎣ u z
vx
vy
vz
ox ⎤
o y ⎥⎥
oz ⎥⎦
t2
t1
(
)
)
(
)
Generalized Disparity
May be computed from correspondences:
G
G
G
P2 x2 = δ ( x1 ) ( C1 − C 2 ) + P1 x1
X
X
G
P1 x1
C 2 δ ( xG ) ( C − C )
1
(
„
P2 x2
1
t1
G
G
G
P2 x2 = δ ( x1 ) C1 − C 2 + P1 x1
Planar Warping Equation (2)
G
G
G
P2 x2 = δ ( x1 ) C1 − C 2 + P1 x1
G 1
G
P2 x2 = C1 − C 2 + P1 x1
2
C1
G
P1 x1
P2 x2
C 2 δ ( xG ) ( C − C )
1
1
2
C1
Generalized Disparity
G
P1 x1
r
=
G
C1 − C 2 δ ( x1 ) C1 − C 2
Planar Warping Equation
G
G
G
P2 x2 = δ ( x1 ) C1 − C 2 + P1 x1
(
X
G
P1 x1
G
δ ( x1 ) =
r
G
G
G
hx2 = P2−1 ⎡⎣δ ( x1 ) C1 − C 2 + P1 x1 ⎤⎦
(
G
hx2 = P2−1 ⎡⎣ P1
r
C1
C 2 δ ( xG ) ( C − C )
1
„
„
Perform a matrix-vector
multiplication, followed by
a homogeneous divide.
9 adds, 11 multiplies, 1
inverse.
Incremental: 6 adds, 5
multiplies.
G
⎡ x1 ⎤
⎤
C1 − C2 ⎦ ⎢ G ⎥
⎣δ ( x1 ) ⎦
)
2
Warping A Pixel
„
(
)
G
⎡ x1 ⎤
G
hx2 = W ⎢ G ⎥
⎣δ ( x1 ) ⎦
G
P1 x1
1
)
3D Warping
⎡ u1 ⎤
⎡r ⎤
⎢
⎥
⎢ s ⎥ = W ⎢ v1 ⎥
⎢ ⎥
⎢ 1 ⎥
⎢ G ⎥
⎣⎢ t ⎦⎥
⎣δ ( x1 ) ⎦
„
Two sub-problems:
„
• Depth
• Correspondences
or optical flows
„
r
u2 =
t
Where should pixels be moved to?
Visibility
s
v2 =
t
Tien-Tsin Wong (Nov. 1999)
Visibility
„
„
Epipolar Geometry
Traditional solution: depth-buffering
Depths may not be available (especially in
a real image)
„
Can we solve the visibility without depth?
„
Yes! Using epipolar geometry
„
Consider
another
pixel
i2
p2 will occlude
Configuration
ofp1two
only
cameras
when
p1, p2 &
e are coplanar, co-linear and p2 is in
between
Reference
image
Reference
camera
Tien-Tsin Wong (Nov. 1999)
Desired
image
Desired
camera
Tien-Tsin Wong (Nov. 1999)
Epipolar Geometry (2)
„
Pattern of Drawing Order
If
Although
always
we
draw
know
i1 before
where
ioccluding
p1line
&correct
pcan
are,occlude
visibility
their
projection
is
Towe
Only
identify
pixels
pixels
ondon’t
thepotentially
same
epipolar
other,
each
the
2, the
2 each
on the epipolar
determines
the
visibility
(e.g.
i1 willto
epipolar
other
plane
isline
intersected
the
reference
image
ensured
without
knowing
thewith
depth
information!
neverthe
occlude
i2) line
give
epipolar
„
By intersecting the epipolar planes with the reference
image, a pattern of drawing order is obtained:
Epipolar
plane
positive
epipole
Tien-Tsin Wong (Nov. 1999)
Tien-Tsin Wong (Nov. 1999)
Pattern of Drawing Order
„
A diverging pattern is formed if the
direction of the epipolar ray is reversed.
Pixel-based Drawing Order
„
In fact, there are only three kinds of
patterns
Tien-Tsin Wong (Nov. 1999)
Tien-Tsin Wong (Nov. 1999)
Drawing Order: Summary
„
„
Project the desired
center of projection
onto reference image
plane:
Perform warping on
each of the resulting
blocks in scanline order
with appropriately
chosen directions:
Reconstruction
„
„
Digital images are discrete, not continuous.
The same image could be produced by different
scene geometries.
Splatting
„
„
Each sample is treated
as a Gaussian cloud
density.
Problem: excessive
exposure errors.
Micropolygons
„
„
LDI: Layered Depth Images
Non-Planar Panoramic Cameras
„
„
Similar warping equations exist, but they
are non-linear.
Visibility ordering idea works here too.
Fit a bilinear patch to
each image grid cell.
Problem: excessive
occlusion errors.
„
A convenient representation that avoids
disocclusion errors with a single “image”:
LDI
Camera
Geometry
Representation
Splat Size Computation