Download PPT - Computer Vision Lab.

Parallel Tracking and Mapping
for Small AR Workspaces
Vision Seminar
2008. 9. 4 (Thu)
Young Ki Baik
Computer Vision Lab.
References
 Parallel Tracking and Mapping for Small AR
Workspaces
 Georg Klein, David Murray, ISMAR 2007
 Visual Odometry
 David Nister et. al. , CVPR 2004
Outline
 What is AR?
 Previous works
 Proposed methods
 Demo
 Conclusion
What is AR?
 AR : Augment Reality
A
ugument reality(AR) uses the real scene as the background, and
makes applications with putting 3D objects to the background.
Since the cost of augment reality is not as expensive as full 3D
virtual reality(VR), AR has been very popular research topic.
Full 3D? (O)
Expensive? (O)
Full 3D (X)
Expensive? (X)
Demonstration
The Aim
 AR with a hand-held camera
 Visual Tracking provides registration
The Aim
 AR with a hand-held camera
 Visual Tracking provides registration
 Track without prior model of world
The Aim
 AR with a hand-held camera
 Visual Tracking provides registration
 Track without prior model of world
 Challenges




Speed
Accuracy
Robustness
Interaction with real world
Existing attempts : SLAM
 SLAM : Simultaneous Localization and Mapping
can use many different types of sensor to acquire observation
data used in building the map such as laser rangefinders, sonar
sensors and cameras.
 Well-established in robotics (using a rich array of sensors)
 Demonstrated with a single hand-held camera by Davison at
2003 (Mono-SLAM).
 Mono-SLAM was applied to
AR system at 2004.
Existing attempts : Model based tracking
 Model-based tracking is
 More robust
 More accurate
 Proposed by Lepetit et. al.
at ISMAR 2003
Frame by Frame SLAM
 Why?
is SLAM fundamentally harder?
Time
One frame
Find features
Update camera pose and entire map
Draw graphics
Many DOF
Frame by Frame SLAM
 SLAM
 Updating entire map every frame is so expensive!!!
 Needs “sparse map of high-quality features”
- A. Davison
 Proposed approach
 Use dense map (of low quality features)
 Don’t update the map every frame : Keyframes
 Split the tracking and mapping into two threads
Parallel Tracking and Mapping
 Proposed method
- Split the tracking and mapping into two threads
Time
Thread #2
Mapping
Update map
One frame
Thread #1
Tracking
Find features
Update camera pose only
Draw graphics
Simple & easy
Parallel Tracking and Mapping
Tracking thread:
Mapping thread:
• Responsible estimation
of camera pose and
rendering augmented
graphics
• Responsible for
providing the map
• Must run at 30 Hz
• Make as robust and
accurate as possible
• Can take lots of time
per key frame
• Make as rich and
accurate as possible
Tracking thread
 Overall flow
Pre-process frame
Map
Project points
Project points
Measure points
Measure points
Update Camera Pose
Update Camera Pose
Coarse stage
Fine stage
Draw Graphics
Pre-process frame
 Make for pyramid levels
640x480
320x240
160x120
80x60
Pre-process frame
 Make for pyramid levels
 Detect Fast corners
 E. Rosten et al (ECCV 2006)
640x480
320x240
160x120
80x60
Project Points
 Use motion model to update camera pose
 Constant velocity model
Estimated
current
Previous pos
Previous pos
Pt
∇t’
Pt-1
∇t
Vt =(Pt – Pt-1)/∇t
Pt+1=Pt+∇t’(Vt)
Pt+1
Project Points
 Choose subset to measure
 ~ 50 biggest features for coarse stage
 1000 randomly selected for fine stage
~50
1000
640x480
320x240
160x120
80x60
Measure Points
 Generate 8x8 matching template (warped
from source key-frame:map)
 Search a fixed radius around projected
position
 Use Zero-mean SSD
 Only search at Fast corner points
Update caemra pose
 6-DOF problem
 Obtain by SFM (Three-point algorithm)
?
Dray graphics
 What can we draw in an unknown scene?
 Assume single plane visible at start
 Run VR simulation on the plane
Mapping thread
 Overall flow
Stereo Initialization
Wait for new key frame
Add new map points
Optimize map
Map maintenance
Tracker
Stereo Initialization
 Use five-point-pose algorithm
 D. Nister et. al. 2006
 Requires a pair of frames and feature
correspondences
 Provides initial map
 User input required:
 Two clicks for two key-frames
 Smooth motion for feature correspondence
Wait for new key frame
 Key frames are only added if :
 There is a sufficient baseline to the other key frame
 Tracking quality is good
 Key frame (4 level pyramid images and its corners)
 When a key frame is added :
 The mapping thread stops whatever it is doing
 All points in the map are measured in the key frame
 New map points are found and added to the map
Add new map points
 Want as many map points as possible
 Check all maximal FAST corners in the key
frame :
 Check score
 Check if already in map
 Epipolar search in a neighboring key frame
 Triangulate matches and add to map
 Repeat in four image pyramid levels
Optimize map
 Use batch SFM method: Bundle Adjustment
 Adjusts map point positions and key frame
poses
 Minimize reprojection error of all points in all
keyframes (or use only last N key frames)
Map maintenance
 When camera is not exploring, mapping
thread has idle time
 Data association in bundle adjustment is
reversible
 Re-attempt outlier measurements
 Try measure new map features in all old key
frames
Comparison to EKF-SLAM
 More Accurate
 More robust
 Faster tracking
<
SLAM based AR
Proposed AR
System and Results
 Environment
 Desktop PC (Intel Core 2 Duo 2.66 GHz)
 OS : Linux
 Language : C++
 Tracking speed
Total
19.2 ms
Key frame preparation
2.2 ms
Feature Projection
3.5 ms
Patch search
9.8 ms
Iterative pose update
3.7 ms
System and Results
 Mapping scalability and speed
 Practical limit
• 150 key frames
• 6000 points
 Bundle adjustment timing
Key frames
2-49
50-99
100-149
Local Bundle Adjustment
170 ms
270 ms
440 ms
Global Bundle Adjustment
380 ms
1.7 s
6.9 s
Demonstration
Remaining problem
 Outlier management
 Still brittle in some scenario
 Repeated texture
 Passive stereo initialization
 Occlusion problem
 Relocation problem
Conclusion
 Conclusion
 Parallel tracking and mapping process are presented
using multi-thread.
 Contribution
 Visual odometry system was well presented.
 Overcome computation by multi-thread
 Opinion
 The proposed algorithm can be applied to our research
• Navigation system
• 3D tracking system
Q
&
A