Download Beyond One-Class Classification

Beyond One-Class
Classification
Amfeng
24 March 2009
Outline
Two models for One Class Classification
From One Class to Binary Class
From Binary Class to Multi Class
From Multi Class to Clustering
Conclustion
Two models for One Class Classification
One Class SVM
Find the optimal hyperplane to separate the
target class from the origin with maximum
margin
Support Vector Data Description
Use the minimum hyperspere to enclose the
target class
Interpretation of the above models
（a） OCSVM取高斯核时的最优超平面
（b）SVDD取高斯核时的最小超球
How to extend to Binary or
Multi Classification
For imbalance data
From SVDD to Binary SVDD with
negative data: B_SVDD_Neg
Objective function:
 ( R, a,  )  R 2  C1  i  C2   p
i
p
 || xi  a ||2  R 2  i
s.t. 
, i  0,  p  0,
2
2
|| x p  a ||  R   p
i  target class, p  negative class
Drawback: without considering the margin
between classes.
The other Version of B_SVDD_Neg
Dong, X., W. Zhaohui, et al. (2001). A new multi-class support
vector machines, Systems, Man, and Cybernetics, 2001 IEEE
International Conference on.
Embedding margin for B_SVDD_Neg
范炜 (2003). 支持向量机算法的研究及其应用, 浙江大学. PhD.
The dual form
Notice
How to get the radius R
Must find the support
vector on the
hypersphere that
negative lived
Does it work really?
n

i 1
i
 1 
here   0
according to KKT, if  >0   =0
n
then

i 1
No support
vector of
Negative data.
i
1
Can’t calculate R
Solutions for above problem
1. Modify the coefficient of R:
Biased support vector machine
In order to avoid the above
problem, b need to less than 1,
that is b  1
Chan, C.-H., K. Huang, and M.R.L.a.I. King. Biased support vector machine for
relevance feedback in image retrieval. in International Joint Conference on
Neural Networks 2004. Budapest, Hungary.
Equivalent style: Minimum Enclosing and
Maximum Excluding Machine
Liu, Y. and Y.F. Zheng. Minimum Enclosing and Maximum Excluding
Machine for Pattern Description and Discrimination Pattern Recognition.
in Proc of the 18th Int Conf on ICPR 2006.Loa Alamitos: IEEE Computer
Society
2. Modify the coefficient of margin
 Here,
here, K  1
Wang, J., N. P, et al. (2005). Pattern classification via single spheres,
Lecture notes in artificial intelligence.( briefly PCSS)
3. Modify the coefficients of margin and R
Generalized HyperSphere SVM(GHSSVM)
张新峰; 刘垚巍: 广义超球面SVM研究 ,计算机研究与发展 2008.11
Extend to Ellipsoid
Wei2007:Minimum Mahalanobis Enclosing Ellipsoid Machine for
Pattern Classification:ICIC 2007,CCIS2, pp. 1176-1185
SVDD with negative data for MultiClass:M_SVDD_Neg
Drawback: without considering the margin either .
Embedding margin for SVDD_Mulit:
MSM_SVM
Pei-Yi Hao, Jung Hsien Chiang, Yen Hsiu lin:A new maximal-margin
spherical-structured multi-class support vector machine, Appl Intell,
2009,30,P98-111
Dual formulation
Without the problem
discussed at the
former.
Illustration of the difference
How about the hypenplane
model
OCSVM with negative: Binary
OCSVM_Neg
Motivation: using the mean of the other
class instead of the optimal point.
Doesn’t considering margin either.
1 T
1 n
min w,ξ,  w w     i
2
 n i 1
1
T
s.t. w ( xi   zn )    i , i  0, i  1,..., n
t
From OCSVM to Asymmetric SVM:
margin embeded
Like the SVDD Multi with margin,
here also describe the target
class by core hyperplane, then
push the negative class by
maximized the margin.
S. H. Wu, K. P. Lin, C. M. Chen, M. S. Chen, Asymmetric support vector
machines: low false positive learning under the user tolerance, Proceeding
of the 14th ACM SIGKDD international conference on Knowledge
discovery and data mining, 749-757, 2008.
Summarize
Model
Hyperspere
Ellipsoid Hyperplane
One-Class
SVDD
MVEE , OCSVM
MVCE,
MELM
Binary-Class Without
margin
B_SVDD_Neg
Multi-Class
Embedding
margin
BSVM,
MEMEM
PCSS,
GHSSVM
Without
margin
Multi
SVDD_Neg
Embedding
margin
MSM SVM
B_OCSVM_Neg
Binary
MELM
ASVM
One against others
Or One-to One
?
?
One Class Classification for Clustering
Support Vector Clustering(JMLR2002)
Iterative strategy integrating two-stage
one-class SVM
Kernel Growth (PAMI 2005)
Soft Clustering for Kernel Growth
Support Vector Clustering
Clustering boundary: same as SVDD,
found the support vector to get the
boundary.
Clustering number: based on the
adjacency matrix which components
decided according to:
Ben-Hur, H. A., D., et al. (2002). "Support vector clustering
" Journal of Machine Learning Research 2 125-137.
The kernel width decided the clustering
number
Outlier enable makes the clustering
possible
Iterative strategy integrating two-stage
one-class SVM
 Different from SVC, need to know the clustering
number in advance, attribute to partition-based
clustering algorithm.
 First stage: using OCSVM for each cluster to
find the non-support vectors ;
 Second stage: retrain the OCSVM using those
non-support vector for representing each
clustering accurately by the optimal hyperplane.
Yeh, C.-Y. and S.-J. Lee (2007). A Kernel-Based Two-Stage One-Class
Support Vector Machines Algorithm. Advances in Neural Networks –
ISNN 2007.
Illustration
Clustering assignment: each pattern is
assign to the maximum projection value by:
Conclusion
 One Class Classifier of SVDD and OCSVM can
be used in many field, including:
 Binary/Multi Class for unbalance data
 Clustering
 Large scale problem: CVM &BVM
 De-noising
 Information processing
Document classification
Image retrieval
 ….