Download Skin Color Weighted Disparity Competition for Hand Segmentation

Skin Color Weighted
Disparity Competition for
Hand Segmentation from
Stereo Camera
Qi Wang, Xilin Chen, Wen Gao
Proceedings of the British Machine Vision Conference
(BMVC), 2010
Outline

Introduction
The Proposed Method




Mark hand pixels
Skin Color Weighted Disparity Competition
Experimental Results
Conclusion

2
Introduction

Human computer interaction (HCI)




Robustness
Accuracy
Efficiency
Hand segmentation and hand gesture



3
Clutter background
Lighting variation
Flexibility or concave character of the hand
Hand Segmentation
Monocular camera
 Skin color detection [3,7,14]
 Background subtraction [11,16]
 Active contour [21]
 Model-based method [5,17,18,19]
 Interactive segmentation [4,10,15]
Stereo camera [8,12]
 Skin color weighted disparity competition
4
Reference
[3] A. Argyros and M. Lourakis. Real-time tracking of multiple skin-colored objects with a possibly moving
camera. In European Conference on Computer Vision, 2004.
[7] W.Y. Chang, C.S. Chen, and Y.P. Hung. Appearance-guided particle filtering for articulated hand tracking. In
IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005.
[8] Q. Delamarre and O. Faugeras. 3d articulated models and multiview tracking with physical forces. Computer
Vision and Image Understanding, 2001.
[11] J. Letessier and F. Berard. Visual tracking of bare fingers for interactive surfaces. 17th Annual ACM
symposium on User Interface Software and Technology, 2004. ACM.
[12] C. Manders, F. Farbiz, J. H. Chong, K. Y. Tang, G. G. Chua, M. H. Loke, and M. L.Yuan. Robust hand
tracking using a skin tone and depth joint probability model. In IEEE International Conference on Automatic
Face & Gesture Recognition,2008.
[14] Z. Mo, J.P. Lewis, and U. Neumann. Smartcanvas: a gesture-driven intelligent drawing desk system. 10th
International Conference on Intelligent User Interfaces,ACM, 2005.
[16] M. Shahzad and L. Joe. Visual touchpad: a two-handed gestural input device. In Proceedings of the 6th
international conference on Multimodal interfaces, ACM.
[21] L.G. Zhang, X. Chen, C.Wang, andW. Gao. Robust automatic tracking of skin-colored objects with level set
based occlusion handling. In Proceedings of The 8th Int’l Gesture Workshop, GW’09, Bielefeld, Germany, 2009.
[17] B. Stenger, P. R. S. Mendonca, and R. Cipolla. Model-based 3d tracking of an articulated hand. In IEEE
Computer Society Conference on Computer Vision and Pattern Recognition, 2001.
[18] B. Stenger, A. Thayananthan, P. H. S. Torr, and R. Cipolla. Model-based hand tracking using a hierarchical
bayesian filter. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006.
[19] Y. Wu, J. Lin, and T. S. Huang. Analyzing and capturing articulated hand motion in image sequences. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 2005.
5
The Proposed Method
6
Marking the Confident Hand Pixels
7
1
• Computing disparity map
2
• Detecting face
3
• Finding human body regions
4
• Marking the confident hand pixels
5
• Camshift algorithm
Marking the Confident Hand Pixels

Computing disparity map


Gray images

Block matching

Zero : occlusion
Detecting face
Finding human body regions
Marking the confident hand pixels
Camshift algorithm



8
Marking the Confident Hand Pixels

Computing disparity map
Detecting face [20]





Locally Assembled Binary (LAB) Haar feature
Average disparity as threshold
Foreground(FG) map and background(BG) map
Finding human body regions
Marking the confident hand pixels
Camshift algorithm


[20] S. Yan, S. Shan, X. Chen, and W. Gao. Locally assembled binary (lab) feature with
feature-centric cascade for fast and accurate face detection. In IEEE Conference on
Computer Vision and Pattern Recognition, pages 1–7, 2008.
9
Locally Assembled Binary (LAB)
Haar feature

Haar feature (OpenCV)

An Integral Image for rapid feature detection


10
*http://cg2010studio.wordpress.com *wikipedi
Locally Assembled Binary (LAB)
Haar feature
11
Marking the Confident Hand Pixels


Computing disparity map
Detecting face [20]






Locally Assembled Binary (LAB) Haar feature
Average disparity as threshold
Foreground(FG) map and background(BG) map
Finding human body regions
Marking the confident hand pixels
Camshift algorithm
[20] S. Yan, S. Shan, X. Chen, and W. Gao. Locally assembled binary (lab) feature with
feature-centric cascade for fast and accurate face detection. In IEEE Conference on
Computer Vision and Pattern Recognition, pages 1–7, 2008.
12
Marking the Confident Hand Pixels



Computing disparity map
Detecting face
Finding human body regions



Marking the confident hand pixels



Union zero pixels in disparity map and FG map
Fill the wholes (occlusion)
Color close to the detected face
(Hue-Saturation Histogram)
Disparity fall in local maxima of the FG map
Camshift algorithm
13
http://nccuir.lib.nccu.edu.tw/bitstream/140.119/32676/7/300607.pdf
Marking the Confident Hand Pixels





Computing disparity map
Detecting face
Finding human body regions
Marking the confident hand pixels
Camshift algorithm

Histogram
http://www.hadron.in/my-findings/touch-me-not.html
14
Segmenting the Hand with Skin Color
Weighted Disparity Competition
Support reasons



The hand disparity changes slightly.
The hand disparity is different from background
disparity.
The hand is an enough simple object from color
aspect.
15
Segmenting the Hand with Skin Color
Weighted Disparity Competition
1
2
3
16
• The skin color probability
• The two competing disparity limits
• Skin color weighted disparity competition
Segmenting the Hand with Skin Color
Weighted Disparity Competition

The skin color probability.


HSV color space
Gaussian model



17
: the mean HSV vector of all marked hand pixels
: the standard deviation of the H, S or V
Segmenting the Hand with Skin Color
Weighted Disparity Competition

The two competing disparity limits





18
h: hand ; b: background
: the maximal bias (typically set to 2 or 1 respectively)
: : the mean disparity of all marked confident hand pixels
: the disparity of background (Watershed Algorithm)
Segmenting the Hand with Skin Color
Weighted Disparity Competition

Skin color weighted disparity competition

Disparity verification
●
N

Skin probability


19
N: neighborhood size (typically set to 2)
Segmenting the Hand with Skin Color
Weighted Disparity Competition

: Validate a hand pixel by competition



20
: less value means more satisfying
Ts : threshold for distinguishing the skin color from the non-skin color
a : a small tuning constant
Segmenting the Hand with Skin Color
Weighted Disparity Competition
• DC means disparity competition
• DLOH means the disparity limit of the hand
• DLOB means the disparity limit of the background
21
Experimental Results



Implemented in C++ by using OpenCV and the
Triclops Stereo Vision SDK.
Video device: the Bumblebee 2 stereo camera
The test sequence contains 225 frames with the
resolution of 1024x384 (512x384)
22
Experimental Results

Parameter

Skin color probability

Skin color weighted disparity competition
23
Experimental Results
Triclops Stereo Vision SDK
24
Experimental Results
25
Experimental Results
26
[4] Y. Y. Boykov and M. P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of
objects in n-d images. In IEEE International Conference on Computer Vision, volume 1, pages 105–112, 2001.
[12] C. Manders, F. Farbiz, J. H. Chong, K. Y. Tang, G. G. Chua, M. H. Loke, and M. L. Yuan. Robust hand
tracking using a skin tone and depth joint probability model. In IEEE International Conference on Automatic
Face & Gesture Recognition, pages 1–6, 2008.
Experimental Results

Quantitative comparison:


27
A: the segmentation result
B: the ground truth from PhotoShop
Method
Accuracy
The proposed method
83.8%
Manders et al.’s method
63.9%
GraphCut
59.2%
Experimental Results


Device : a Lenovo ThinkPad T400 with 2.5 GHz Dual
Intel CPU and 2GB RAM
Computing time(no capture time)

The Frame Rate: 5 fps >>19.2 fps


28
ROIs tracking (Camshift algorithm)
Disparity competition is necessary for the successive frames
ONLY.
Conclusion


This paper propose the skin color weighted disparity
competition method for hand segmentation in clutter
environment from stereo camera.
Limitation


hand-hand occlusion or hand-face occlusion
Future works


29
More accurate segmentation
Reconstruct the 3D appearance