Download Hierarchical Processing

Multimedia Project
Optical flow:
Methods and Applications
Team : MediaICE
Kim, Min Hyun
Pua Jun Hong
Contents
1. Background
(i) Mathematical Background
(ii) Aperture Problem
2. Algorithms
(i) Lucas and Kanade’s Method
(ii) Bouguet’s Method
(iii) Eliete’s Method
(iv) Variations
3. Applications
(i) Analysis of facial expression
(ii) Gesture Recognition
(iii) Map Navigation Interface on PDA
4. References
1. Background.
Among the existing methods for optical flow estimation, gradient based techniques are
distinguished. The basic assumption of them is that image brightness of each pixel will not be
significantly changed during small displacement.
(i) Mathematical Background
The most basic equation of this assumption which is called constraint optical flow equation can
be defined as
Ixu+ lyv+ It = 0
Where u and v are optical flow components in x and y directions for a displacement d = (dx, dy),
Ix+ ly and It are partial derivatives of the image brightness, I(x,y) with regard to the horizontal(x)
and vertical(y) coordinates, and time(t).
(ii) Aperture Problem
However, optical flow cannot be estimated from this equation because of Aperture Problem(See
the reference). Aperture problem is one of our visual system’s problems. In our visual system,
each neuron responds to the image locally and integration is needed to estimate object. The
small window in which each neuron is looking at the visual field is aperture. When we see an
object through small window, sometimes we cannot infer the motion unless we see the whole
object. Mathematical Optical flow estimation I just mentioned suffers from same phenomenon.
Figure 1. When this black bar moves to somewhere, we cannot tell its movement direction
unless we can see the edge of the bar.
2. Algorithms for optical flow
(i) Lucas and Kanade’s Method
In this method, the image is divided in windows of size N x N, each one with p = NxN pixels. For
each pixel we apply constraint optical flow equation, and find the one vector Least Mean Square.
All window pixels will have same optical flow vector.
Figure 3. Optical flow vector corresponds to all window pixels.
(ii) Bouguet’s Method:
Bouguet’s method [2] uses hierarchical processing applied to Lucas and Kanade’s method [8]. A
justification for using of hierarchical processing is the necessity of better precision in measures
of the obtained optical flow vectors. This method uses pyramidal representation of gray image
frames. Bouguet algorithm consists of using down level estimations as initial guess of pyramidal
top level. The estimation of pyramidal highest level is the estimated optical flow.
Hierarchical Processing
Traditionally, optical flow was computed using only one scale of resolution, usually defined by
the visual sensor. However, low sampling rates and aliasing effects make this method
inappropriate. Thus, alternative to solve this problem is to apply optical flow techniques in a
hierarchical coarse-to-fine framework. Gaussian or Laplacian pyramids is used to decompose
images into the different scales of resolution.
Figure 2. Hierarchical computational model.
(iii). Eliete’s Method:
Eliete’s method [3] is a variation of Lucas and Kanade’s method (Section 2.1.1). Eliete uses a
bigger window for the brightness conservation model than the one considered by Lucas and
Kanade. Only some pixels of each window are randomly chosen for the flow vector estimation.
The over-constrained equation system is solved by the LMS method.
(iv) Variations
Using color information as well as brightness information
Kelson et al.(2008) used color information and resulted in 30% more amount of valid estimated
measures in comparison with methods that only use brightness information.
3. Application
(i). Analysis of Facial Expressions of Human Emotion
Carmen et al 08 tried to analyze human emotion using optical flow technique. They tried to find
specific set of facial movements which are common to most people when they experience a
particular emotion. Then, they created ‘Emotion vector maps’ for specific emotions. Their result
was satisfactory as far as they used questionnaire to map facial expressions to the emotion
vector. Their eventual aim is to be able to recognize facial expressions without the need for
specialized training system.
(ii). Gesture Recognition
Cutler et al. 98 developed a real-time, view-based gesture recognition system. Optical flow is
estimated and segmented into motion blobs (see figure 4). They used real-time optical flow to
segment a user’s dominant motions. The system has been applied to create an interactive
environment for children, and successfully applied to the interactive system for children.
Figure 4. Action motion blobs: (a) waving, (b) jumping, (c) marching, (d) clapping, (e) drumming,
(f) flapping.
(iii). Map Navigation interface
One of the students in ICU applied optical flow to track the face and used face position for map
navigation. The position of the face can be calculated by average position of all optical flow dots,
and zooming can be done by calculating variance of the optical flow dots. Those kinds of face
position, orientation, and distance information will be used to execute the several functions of
map navigation.
Figure 5. Left picture shows how to measure center of the face using dots, and Right picture
shows map navigation interface.
4. References
[1] On the estimation of optical flow: relations between different approaches and some new
results HH Nagel - Artificial Intelligence, 1987 - portal.acm.orga
[2] Optical flow using color information: preliminary results Kelson R. T. Aires, Andre M.
Santana, Adelardo A. D. Medeiros
[3] B. Lucas and T. Kanade. An iterative image registration technique with an
application to tereo vision. In IJCAI81, pages 674–679, 1981.
[4] Aperture Problem: http://en.wikipedia.org/wiki/Motion_perception
[5] Optical flow image analysis of facial expressions of human emotion: forensic
applications Carmen J. Duthoit, T. Sztynda, S. K. L. Lal, B. T. Jap, J. I. Agbinya January 2008
[6] View-based interpretation of real-time optical flow for gesturerecognition
R Cutler, M Turk
- Automatic Face and Gesture Recognition, 1998. Proceedings. …, 1998 - ieeexplore.ieee.org