Download a4academics.com

MOTION DETAIL
PRESERVING OPTICAL
FLOW ESTIMATION
Li Xu, Jiaya Jia and Yasuyuki Matsushita
CONTENTS
INTRODUCTION
CONVENTIONAL OPTICAL FLOW
OPTICAL FLOW MODEL
ROBUST DATA FUNCTION
EDGE-PRESERVING REGULARIZATION
MEAN FIELD APPROXIMATION
OPTIMIZATION FRAMEWORK
EXTENDED FLOW INITIALIZATION
CONTINUOUS FLOW OPTIMIZATION
OCCLUSION-AWARE REFINEMENT
CONCLUSION
REFERENCE
INTRODUCTION
Optical flow is the apparent motion of brightness
patterns in the image

Ideally, optical flow would be the same as the motion field
The motion field … is the projection into the image of
three-dimensional motion vectors

3
CONVENTIONAL OPTICAL FLOW

Dominant Scheme: Coarse-to-Fine Warping Framework
 The input image is represented as a tree of
regions



The optical flow is estimated by optimizing an
energy function
optical flow estimation on the coarser level
region-tree is used for defining region-wise finer
displacement samplings for finer level regiontrees
Middlebury optical flow evaluation
MULTI-SCALE PROBLEM IN COARSE-TO-FINE WARPING
(a)
(b)
(c)
Coarse level (e)
(d)
Fine level
(f)
(a)-(b) Two input patches.
(c) Flow estimate using the coarse-to-fine variational setting. (d) Our
flow estimate. (e)-(f) Two consecutive levels in the pyramid. Flow fields
are visualized using the colour code
Large displacement optical flow may not be well estimated
• Inclination to diminish small motion structures when
spatially significant and abrupt change of the displacement
exists.
•
Solution
•
Improve flow initialization to reduce the reliance of the
initialization from coarser levels and enables recovering
many motion details at each scale
Our Work
Framework

Extended coarse-to-fine motion estimation for both large and
small displacement optical flow
Model

A new data term to selectively combine constraints
Solver

Efficient numerical solver for discrete-continous optimization
OPTICAL FLOW MODEL
ROBUST DATA FUNCTION
Objective function for development of a new optimization
procedure
u denotes the flow field that represents the displacement
between frames I1 and I2 ,x represents the 2D coordinates

Data Constraints
Color constraint
D1(u,x)=||I2(x+u)-I1(x)||

Gradient constraint
D I(u,x)=ζ|| I2(x+u)- I1(x)||
Data term
ED (u,x)=∑1/2 D1(u,x)+1/2 D

(u,x)
I
Data cost distributions w.r.t different displacement values
A good model should only use the more fitting constraint, but not
both of them
Define a binary weight map α(x):Z―>{0,1} to switch between the two
terms. When α(x)=1, the gradient constraint is favored. Otherwise,
we select color constraint

EDGE-PRESERVING REGULARIZATION
Smoothness term, it maintains motion discontinuity
The final objective function is defined as E(u,α)= ED(u,α)+λEs(u)
where λ is the regularization weight.

MEAN FIELD APPROXIMATION
The effective energy is written as Eeff(u)= EeffD(u)+λEs(u)
The effective data function

β is the inverse temperature
β plays a key role in shaping the data function.

Small β makes the distribution close to the original one with α=0.5
A relatively large β yields the distribution approaching the lower envelope of
the costs with α=0 and α=1

OPTIMIZATION FRAMEWORK
INPUT: A pair of images for optical flow estimation
1.
Construct pyramids for both of he images and set the initial level l=0 and
uɭ=0 for all pixels
2.
Propagate uɭ to level l+1
Extended Flow Initialization
3.1. Detect and match SIFT features in level l+1
3.2 Perform patch matching in level l+1
3.3 Generate multiple flow vectors as candidates
3.4 Optimize flow
1.
Continous Flow Optimization
4.1 Compute the ᾱ map
4.2 Solve the energy function
5. Occlusion-aware Refinement
6.
If l≠n-1 where n is the total number of levels, l=l+1 and go to step 2
OUTPUT: The optical flow field
1.
EXTENDED FLOW INITIALIZATION
Finding multiple extended displacements (denoted as {u0,u1,….,un}) to
improve estimation in uc



uc which is the flow field computed in the immediately coarser level.
The steps adopted to obtain the extended displacements.
 SIFT Feature Detection
 Selection
 Expansion
 Patch Matching
 Matching Field Fusion
A)SIFT Feature Detection
SIFT feature detection and matching can efficiently capture large
motion for objects undergoing translational and rotational motion

Employ only the sparse matching of discriminative points, which
avoids introducing many ambiguous correspondences and outliers.

employ discrete optimization to only select the most credible
candidates.

B) Selection
The computed displacement vectors by feature matching are
denoted as {s0, . . . ,sn}
 Robustly screen out the duplicated vectors
Compute the euclidean distance between each
si and all uc
 If all results are greater than 1 (pixel), we regard si as a new flow
candidate.
Repeat this process for all si, and denote the m remaining
candidate vectors {sk0, . . . ,skm-1}

C)Expansion
The m remaining vectors {sk0, . . . ,skm-1} represent possible
missing motion in the present flow field uc

Determine whether or not they are better estimates to replace the
original ones

D) Patch Matching
sometimes it still misses some motion vectors

because small texture-less objects may not have distinct
features

SIFT descriptors, the patches on which they operate
should at least contain 16x16 samples as suggested.

Compute the matching field un by minimizing energy

Total of five color and gradient channels used.

Noise can be quickly rejected in the following optimization step
with the collection of a set of flow candidates for each pixel.

E) Matching Field Fusion
The m+1 new motion fields {u0,..um-1,un} together with the
original uc, comprise several motion candidates for each pixel in
the present image scale.

Selection of the optimal flow among the m+2 candidates for
each pixel is a labeling problem

Solved by discrete optimization efficiently

Extended flow initialization.
CONTINUOUS FLOW OPTIMIZATION
Refine flow u0 through continuous optimization
We propose decomposing the optimization into three simpler problems,
each of which can have the globally optimal solution.
Auxiliary variables p and w, representing the substituted data cost and
flow derivatives
The optimal solution is given by the shrinkage
formula

INPUT: Images Ik ,initial flow field u0,weights αk
Perform linerization at u0
η= η0
repeat
Compute pk
θ= θ0
repeat
Compute w
Compute du
θ=θ/3
until θ=θmin
η= η/3
until η< ηmin
ur=u0+du
OUTPUT: Refined flow field ur
θ and η are critical parameters that should be small.

fixing them to constants typically results in slow convergence

Initially sets θ and η to large values to allow warm-starting and
then decreases them in iterations toward the desired
convergence

θ and η minimum values are set to 0.1

OCCLUSION-AWARE REFINEMENT
Multiple pixels mapping to the same point in the target image
using forward warping are possibly occluded by each other

detect occlusion using the mapping uniqueness criterion
expressed as
o(x)=T0.1(f(x+u(x))-1)

f(x+u(x)) is the count of reference pixels mapped to position
x+u(x) in the target view using forward warping

measure of the data confidence based on the occlusion detection
is expressed as c(x)=max(1-o(x),0.01)

CONCLUSION
A unified framework to preserve motion details in both
small and large-displacement scenarios.
It include the selective combination of the color and
gradient constraints, sparse feature matching, and dense
patchmatching to collect appropriate motion candidates


Limitations
--Texture-less motion details
--Large occlusions
REFERENCE
1.
2.
3.
4.
L. Alvarez, J. Esclarin, M. Lefebure, and J. Sanchez, “A PDE Model for
Computing the Optical Flow”
P. Anandan, “A Computational Framework and an Algorithm for the
Measurement of Visual Motion”
S. Baker, D. Scharstein, J. Lewis, S. Roth, M.J. Black, and R. Szeliski, “A
Database and Evaluation Methodology for Optical Flow ”
C. Barnes, E. Shechtman, A. Finkelstein, and D.B. Goldman,
“Patchmatch: A Randomized Correspondence Algorithm for Structural
Image Editing”