Download Shape-From

3D Vision



1
Why?

The world is 3D

Not all useful information is readily available in 2D
Why so hard?

“Inverse problem”: one image = many scenes

Complex relationship between objects & pixels

Noise, occlusion, etc.
What can we do?

Use "hints" from our knowledge of the world

Add more information to the problem!
Ellen L. Walker
Labeling Image Edges

Many edge labels carry 3D information

Occluding blade (>)


Convex crease (+)


Edge is "horizon" of curving-away surface
Others are about reflectance or illumination changes

Mark (M)


Change due to paint or material boundary
Illumination Boundary (S)

2
Edge is further than both surfaces
Limb (>>)


Edge is closer than both surfaces
Concave crease (–)


Occluding surface to right along arrow
Shadow edge
Ellen L. Walker
Intrinsic Image Pixel Contents
3

Depth (range)

Orientation (surface normal)

Illumination

Albedo (reflectance)

Given this information, the picture (intensity values) can
be completely reconstructed
Ellen L. Walker
3D Cues in Single 2D Images


Occlusion

Occluding objects are closer to the camera

The crossbar of a T-junction belongs to the occluding
object
Perspective scaling and foreshortening

If two copies of the same object appear in a picture, the
smaller one is further away.



Texture Gradient

4
Scaling is parallel to the image plane
Foreshortening is perpendicular to the image plane
Since texture is composed of repeated patterns, changes
in size and density of texture convey depth cues
Ellen L. Walker
"Shape-From" Methods

Use a cue (e.g. texture, shading) from a small region of
an image

Cues generally give partial surface orientation
information

5
E.g. degree of tilt

Related cues can give "boundary conditions" to start
from

Solve for continuous surfaces that satisfy both the
general and boundary constraints
Ellen L. Walker
Example: Shape From Shading
Figure 12.2
6
Ellen L. Walker
Gradient Space Represents Surface Normal
q
p
7
Ellen L. Walker
Reflectance Geometry

Three directions are important: normal to the surface,
surface to light source, and surface to camera
light source
normal v ector
Camera
surface patch
8
Ellen L. Walker
Types of Reflection (Review)


9
Specular reflection (mirror)

Color depends on light source color

Limited scattering: angle of incidence = angle of reflection

Camera sees light if it’s pointed in the right direction

Nearly all light is reflected
Lambertian reflection (matte)

Color depends on material properties of object

Light evenly scattered throughout half-space

Camera sees light if surface is visible

Amount of light reflected is proportional to angle between
surface and light source
Ellen L. Walker
Shape From Shading

Lambertian surface, known light source direction


Reflective component (highlights) can be subtracted out in
preprocessing
Relative brightness of surface patches constrain their
directions

Darker patches are more tilted away

A given brightness value represents a circle in gradient
space

Boundary pixels indicate surface at 90 degrees from
normal (if smooth surface)

Solve an optimization problem: brightness term and
smoothness term
10
Ellen L. Walker
Shape From Shading (cont’d)

Brightness term

Intensity is a function of reflectance, and reflectance is a
function of surface normals (p,q) and light source direction
(vx, vy, vz)
I(x, y) R( p(x, y),q(x, y))
R( p,q)  max( 0, 

Smoothness term


11
pv x  qv y  v z
1 p  q
2
2
)
Try to minimize integral of partial derivatives of p and q in
x and y direction
ES 

px2  py2  qx2  qy2 dxdy
Ellen L. Walker
Photometric Stereo


12
Extension to multiple "images"

Lambertian surface, several light sources

Each image has one light source, constrains surfaces

Solve an overdetermined linear (matrix) system - like
camera calibration with extra points
Implementation

Surround your object with a frame containing inwardpointing lights

Take an image with each light in turn

Use images and known light directions to solve the
equations.
Ellen L. Walker
More Variations


13
SHINY

Photometric stereo system for highly reflective materials

Used to accurately characterize welds
Accurate color determination (plastic objects)

Separate highlights from matte portion

Determine illumination color from highlight

Determine object color

Create “matte object” for photometric stereo or shape from
shading
Ellen L. Walker
Shape From Texture
Figure 12.3
14
Ellen L. Walker
Shape from Texture

Transformation of original texture related to surface normal

Solve for affine transformation between original texel and viewed
texel

Transformation depends on surface normal & distance

(Assume camera is far enough to avoid worst perspective
distortion)

If the original texel is known, transformations can be computed
directly

If the original texel is unknown, assume the largest visible texel is
directly facing the camera

Use smoothness or shape constraints to eliminate alternatives
15
Ellen L. Walker
Shape from Focus

Vary the focal length of the camera (i.e. zoom lens)

Objects at different distances will become clear at
different focal lengths

Can use comparisons between pairs of images to get
relative distances
16
Ellen L. Walker
Choice Depends on Object

Solid, matte object (or matte separated from specular)


Highly reflective object


Shape from texture, stereo, focus
Irregular textured object

17
Shape from specular reflection
Regular textured object


Shape from shading, photometric stereo
Stereo, focus
Ellen L. Walker