Download Multiple View Geometry in Computer Vision

Geometric
Computer Vision
Marc Pollefeys
Fall 2009
http://www.inf.ethz.ch/personal/pomarc/courses/gcv/
Geometric Computer Vision course schedule
(tentative)
Lecture
Exercise
Sept 16
Introduction
-
Sept 23
Geometry & Camera model
Camera calibration
Sept 30
Single View Metrology
Measuring in images
(Changchang Wu)
Oct. 7
Feature Tracking/Matching
Correspondence computation
Oct. 14
Epipolar Geometry
F-matrix computation
Oct. 21
Shape-from-Silhouettes
Visual-hull computation
Oct. 28
Multi-view stereo matching
Project proposals
Nov. 4
Structure from motion and visual SLAM
Papers
Nov. 11
Multi-view geometry and
self-calibration
Papers
Nov. 18
Shape-from-X
Papers
Nov. 25
Structured light and active range
sensing
Papers
Dec. 2
3D modeling, registration
and range/depth fusion
Papers
(Christopher Zach?)
Dec. 9
Appearance modeling and imagebased rendering
Papers
Dec. 16
Final project presentations
Final project presentations
From Last Class
Varing focus does change the focal length for our
image formation (probably very slightly).
Single View Metrology
Class 3
Single View Metrology
Measuring in a plane
Need to compute H as well as uncertainty
Direct Linear Transformation
(DLT)
xxii  Hx i  0
 h 1T x 
 T i
T
xi  xi, yi, wi  Hx i   h 2 x i 
 3T 
 yh 3T x  wh 2 T x   h x i 
i
i
 i T i
1
3T

x i  Hx i  wih x i  xih x i 


2T
1T
 xih x i  yih x i 


 0T

T

 wi x i
 yix iT

T

 wi x i
T
1



yi x i
h 
 2
T
T
0
 xix i  h   0

T
T
3


xi x i
0  h 
Ai h  0
•
Direct Linear Transformation
(DLT)
Equations are linear in h
Ai h  0
• Only 2 out of 3 are linearly independent
(indeed, 2 eq/pt)
 0TT
 wix TiT
  0 T  wTix i
0T
 wwixxi T
0 T
T
i
i

 yix i
xix i

 1
 2
yix TiT  h1 
yix i T  2 
 xix Ti  h   0
 xiTx i  3 
0  h 
 3
xi A
wi Aifi w0i’≠0)
(only
drop
i  ythird
i A i row
• Holds for any homogeneous
representation, e.g. (xi’,yi’,1)
Direct Linear Transformation
(DLT)
• Solving for H
 A1 
A 
 2Ah
h  0
A 3 
 or
 12x9, but rank 8
size A is 8x9
A 4 
Trivial solution is h=09T is not interesting
1-D null-space yields solution of interest
pick for example the one with h  1
Direct Linear Transformation
(DLT)
• Over-determined solution
 A1 
A 
 2Ah
h  0
  
 of inexact measurement
No exact solution because
An 

i.e. “noise”
Find approximate solution
- Additional constraint needed to avoid 0, e.g.
- Ah  0 not possible, so minimize Ah
h 1
DLT algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) For each correspondence xi ↔xi’ compute Ai. Usually
only two first rows needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
(iii) Obtain SVD of A. Solution for h is last column of V
(iv) Determine H from h
Importance of normalization
0
x
 i
0
0  xi  yi  1
yi
1
~102 ~102 1
0
0
0
yixi
 xixi
~102
~102
1
~104
yi yi
 xi yi
~104
 h1 
yi  2 
h   0

 xi  3 
h 
~102
orders of magnitude difference!
Monte Carlo simulation
for identity computation based on 5 points
(not normalized ↔ normalized)
Normalized DLT algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the 2D homography matrix H such that xi’=Hxi
Algorithm
 xi
(i) Normalize points ~
x i  Tnormx i , ~
xi  Tnorm
(ii) Apply DLT algorithm to ~
xi  ~
xi ,
~
(iii) Denormalize solution H  T-1 H
T
norm
Tnorm
norm
0
w / 2
w  h
  0
w  h h / 2 
 0
0
1 
1
Geometric distance
x measured coordinates
x̂ estimated coordinates
x true coordinates
d(.,.) Euclidean distance (in image)
Error in one image
2
e.g. calibration pattern
Ĥ  argmin  d xi , Hx i 
H
i
Symmetric transfer error
Ĥ  argmin
H


 d x i , H xi  d xi , Hx i 
-1
2
2
i
Reprojection error
Ĥ, x̂ , x̂   argmin  d x , x̂   d x , x̂ 
2
i
i
H,x̂ i , x̂ i
i
i
i
2
i
i
subject to x̂i  Ĥx̂ i
Reprojection error


d x, H x  d x, Hx 
-1
2
2
d x, x̂   d x, x̂
2
2
Statistical cost function and
Maximum Likelihood Estimation
• Optimal cost function related to noise model
• Assume zero-mean isotropic Gaussian noise
(assume outliers removed)
Pr x  
1
2 πσ
2
e

 d  x, x 2 / 2 2

Error in one image
Pr xi | H   
i
1
2 πσ
log Pr xi | H   

xi ,Hx i 2 / 2 2 
e
2
d
1
2σ
2
2

 d x i , Hx i   constant
Maximum Likelihood Estimate
2



d
x
,
H
x
 i i
Statistical cost function and
Maximum Likelihood Estimation
• Optimal cost function related to noise model
• Assume zero-mean isotropic Gaussian noise
(assume outliers removed)
Pr x  
1
2 πσ
2
e

 d  x, x 2 / 2 2

Error in both images
Prxi | H   
i
1
2πσ
2
e
 d x i , x i 2  d

Maximum Likelihood Estimate
 d x , x̂ 
2
i
i
2


 d x i , x̂ i 
xi ,Hxi 2  / 2 2 
Gold Standard algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the Maximum Likelyhood Estimation of H
(this also implies computing optimal xi’=Hxi)
Algorithm
(i) Initialization: compute an initial estimate using
normalized DLT or RANSAC
(ii) Geometric minimization of reprojection error:
● Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:
● compute initial estimate for optimal {xi}
● minimize cost  d x i , x̂ i 2  d xi , x̂i 2 over {H,x1,x2,…,xn}
● if many points, use sparse method
Uncertainty: error in one image
(i) Estimate the transformation Ĥ from the data
(ii) Compute Jacobian J f  X / h , evaluated at ĥ
(iii) The covariance matrix of the estimated ĥ is

T

1
given by   J   J
h

x
x iT
1 ~
J i  xi / h  
wi  0

0
~
xT
i
 xi~
x iT 

 yi~
x iT 

T 1 
 h  J  J    J i i J i 
 i


T
1
x



Uncertainty: error in both images
J  A | B
P  h | x1x1 ...x n xn 
separate in homography and point parameters
T 1
T 1

A

A
A
 X B
T 1
X
J  X J   T 1
T 1 
 B X A B X B
Using covariance matrix in point transfer
Error in one image
 x  J h  J
T
h h
Error in two images
 x  J h  h J Th  J x  x J Tx
(if h and x independent, i.e. new points)
Example:
=1 pixel =0.5cm
(Criminisi’97)
Example:
=1 pixel =0.5cm
(Criminisi’97)
Example:
(Criminisi’97)
Monte Carlo estimation of
covariance
• To be used when previous assumptions
do not hold (e.g. non-flat within variance)
or to complicate to compute.
• Simple and general, but expensive
• Generate samples according to assumed
noise distribution, carry out computations,
observe distribution of result
Single view measurements:
3D scene
Background: Affine Geometry
• Invariant properties: Parallelism,
ratio of areas, ratio of lengths on
collinear or parallel lines
• Invariant properties from projective
transformation: cross ratio
Background: Projective geometry
of 1D
x1 , x2 T
x2  0
x'  H 22 x
3DOF (2x2-1)
The cross ratio
x ,x x ,x
x , x ; x , x  
x ,x x ,x
1
2
3
1
2
3
4
1
3
2
4
4
Invariant under projective transformations
 xi1
x i , x j  det 
 xi 2
x j1 
x j 2 
Vanishing points
• Under perspective projection points at infinity can have a
finite image
• The projection of 3D parallel lines intersect at vanishing
points in the image
Basic geometry
Basic geometry
(b, t : i , v ) (0, ht : hi ,  ) 
ht
hi
• Allows to relate height of point to height of camera
Homology mapping between
parallel planes
• Allows to transfer point from one plane to another
Single view measurements
Single view measurements
Forensic applications
190.6±4.1 cm
190.6±2.9 cm
A. Criminisi, I. Reid, and A. Zisserman.
Computing 3D euclidean distance from a single view.
Technical Report OUEL 2158/98, Dept. Eng. Science, University of Oxford, 1998.
Example
courtesy of Antonio Criminisi
La Flagellazione di Cristo (1460)
Galleria Nazionale delle Marche
by Piero della Francesca (1416-1492)
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/
More interesting stuff
• Criminisi demo
http://www.robots.ox.ac.uk/~vgg/presentations/
spie98/criminis/index.html
• work by Derek Hoiem on learning
single view 3D structure and apps
http://www.cs.cmu.edu/~dhoiem/
• similar work by Ashutosh Saxena on
learning single view depth
http://ai.stanford.edu/~asaxena/learningdepth/
Next class
• Feature tracking and matching