Download ppt - CUBS

CSE 473/573
Computer Vision and Image
Processing (CVIP)
Ifeoma Nwogu
Lecture 24 – Classifiers
1
Schedule
• Last class
– We continued on segmentation, specifically mean
field and graph cuts but could not finish 
• Today
– Classifiers
• Readings for today:
– Forsyth and Ponce chapter 16
2
Classifiers
• Given a feature representation for images,
how do we learn a model for distinguishing
features from different classes?
• Take a measurement x, predict a bit (yes/no;
1/-1; 1/0; etc)
• Today:
•
•
Nearest neighbor classifiers
Linear classifiers: support vector machines
• Other techniques:
•
Boosting, Decision trees and forests, Deep neural networks
Image classification
• The big problems
– Image classification
• eg this picture contains a parrot
– Object detection
• eg in this box in the picture is a parrot
• Strategy
– Generate features from image (there are many
quite complex strategies)
– Put in one or more classifiers
4
Image classification - scenes
Image classification - material
Classifiers
Given a feature representation for images, how do we learn a
model for distinguishing features from different classes?
Decision
boundary
Zebra
Non-zebra
Nearest Neighbor Classifier
• Assign label of nearest training data point to each
test data point
from Duda et al.
Classifiers
• Take a measurement x, predict a bit (yes/no; 1/-1;
1/0; etc)
• Strategies:
– non-parametric
• nearest neighbor
– probabilistic
• histogram
• logistic regression
– decision boundary
• SVM
K-Nearest Neighbors
• For a new point, find the k closest points from
training data
• Labels of the k points “vote” to classify
k=5
Histogram based classifiers
• Represent class-conditional densities with
histogram
• Advantage:
– estimates become quite good
• (with enough data!)
• Disadvantage:
– Histogram becomes big with high dimension
• but maybe we can assume feature independence?
Example: Finding skin
• Skin has a very small range of (intensity
independent) colours, and little texture
– Compute an intensity-independent colour
measure, check if colour is in this range, check if
there is little texture (median filter)
– See this as a classifier - we can set up the tests by
hand, or learn them.
Histogram classifier for skin
Curse of dimension
• Can’t build a histogram based classifier for
when feature space is high-dimensional
– try R, G, B, and some texture features
– Fails when there are too many histogram buckets
Distance functions for bags of features
• Euclidean distance: D(h , h )   (h (i)  h (i))
N
2
1
• L1 distance:
2
i 1
1
2
N
D(h1 , h 2 )   | h1 (i )  h 2 (i ) |
i 1
• χ2 distance:
N
h1 (i)  h 2 (i) 2
i 1
h1 (i )  h 2 (i )
D(h1 , h 2 )  
• Histogram intersection (similarity):
N
I (h1 , h 2 )   min(h1 (i ), h 2 (i ))
• Hellinger kernel (similarity):
i 1
N
K (h1 , h 2 )   h1 (i ) h 2 (i )
i 1
Linear classifiers
• Find linear function (hyperplane) to separate
positive and negative examples
xi positive :
xi  w  b  0
xi negative :
xi  w  b  0
Which hyperplane
is best?
Support vector machines
• Find hyperplane that maximizes the margin between
the positive and negative examples
xi positive ( yi  1) :
xi  w  b  1
xi negative ( yi  1) :
xi  w  b  1
For support vectors,
Distance between point
and hyperplane:
xi  w  b  1
| xi  w  b |
|| w ||
Therefore, the margin is 2 / ||w||
Support vectors
Margin
C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and
Knowledge Discovery, 1998
What if the data is not linearly
separable?
1
2
min w  C
w ,b 2
n
 max 0,1  y (w  x
i 1
i
i
 b) 
+1
0
Margin
-1
• Demo: http://cs.stanford.edu/people/karpathy/svmjs/demo
Nonlinear SVMs
• Datasets that are linearly separable work out great:
x
0
• But what if the dataset is just too hard?
x
0
• We can map it to a higher-dimensional space:
x2
0
x
Slide credit: Andrew Moore
Nonlinear SVMs
• General idea: the original input space can
always be mapped to some higherdimensional feature space where the training
set is separable:
Φ: x → φ(x)
Slide credit: Andrew Moore
Nonlinear SVMs
• The kernel trick: instead of explicitly
computing the lifting transformation φ(x),
define a kernel function K such that
K(x,y) = φ(x) · φ(y)
• This gives a nonlinear decision boundary in
the original feature space:
  y  ( x )   ( x)  b    y K ( x , x)  b
i
i
i
i
i
i
i
i
C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and
Knowledge Discovery, 1998
Nonlinear kernel: Example
2

(
x
)

(
x
,
x
)
• Consider the mapping
x2
 ( x)   ( y)  ( x, x 2 )  ( y, y 2 )  xy  x 2 y 2
K ( x, y)  xy  x 2 y 2
Polynomial kernel: K (x, y )  (c  x  y )
d
Gaussian kernel
• Also known as the radial basis function
(RBF) kernel:
2
 1
K (x, y )  exp  2 x  y 
 

• The corresponding mapping φ(x) is infinitedimensional!
Gaussian kernel
SV’s
Kernels for bags of features
• Histogram intersection kernel:
N
I (h1 , h 2 )   min(h1 (i ), h 2 (i ))
i 1
N
• Hellinger kernel: K (h1 , h 2 )   h1 (i) h 2 (i)
i 1
• Generalized Gaussian kernel:
 1
2
K (h1 , h 2 )  exp  D(h1 , h 2 ) 
 A

D can be L1, Euclidean, χ2 distance, etc.
J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, Local Features and Kernels for Classifcation of Texture and Object
Categories: A Comprehensive Study, IJCV 2007
Summary: SVMs for image classification
1. Pick an image representation (in our case, bag of
features)
2. Pick a kernel function for that representation
3. Compute the matrix of kernel values between every
pair of training examples
4. Feed the kernel matrix into your favorite SVM solver
to obtain support vectors and weights
5. At test time: compute kernel values for your test
example and each support vector, and combine them
with the learned weights to get the value of the
decision function
What about multi-class SVMs?
• Unfortunately, there is no “definitive” multi-class
SVM formulation
• In practice, we have to obtain a multi-class SVM by
combining multiple two-class SVMs
• One vs. others
– Traning: learn an SVM for each class vs. the others
– Testing: apply each SVM to test example and assign to it
the class of the SVM that returns the highest decision
value
• One vs. one
– Training: learn an SVM for each pair of classes
– Testing: each learned SVM “votes” for a class to assign to
the test example
SVMs: Pros and cons
• Pros
– Many publicly available SVM packages:
http://www.kernel-machines.org/software
– Kernel-based framework is very powerful, flexible
– SVMs work very well in practice, even with very small
training sample sizes
• Cons
– No “direct” multi-class SVM, must combine two-class
SVMs
– Computation, memory
• During training time, must compute matrix of kernel values for
every pair of examples
• Learning can take a very long time for large-scale problems
SVMs for large-scale datasets
• Efficient linear solvers
•
LIBLINEAR, PEGASOS
• Explicit approximate embeddings: define an
explicit mapping φ(x) such that φ(x) · φ(y)
approximates K(x,y) and train a linear SVM
on top of that embedding
•
•
Random Fourier features for the Gaussian kernel
(Rahimi and Recht, 2007)
Embeddings for additive kernels, e.g., histogram
intersection (Maji et al., 2013, Vedaldi and Zisserman,
2012)
Evaluating classifiers
• Always
– train on training set, evaluate on test set
• test set performance might/should be worse than
training set
• Options
– Total error rate
• always less than 50% for two class
– Receiver operating curve
• because we might use different thresholds
– Class confusion matrix
• for multiclass
Receiver operating curve
Classifiers: Crucial Points
• Classifiers accept features, return decision
– and are often trained using data
• Very mature training processes now
– Always try standard methods first
• SVM
• kNN
• Logistic regression
• Evaluate with separate test data
–
–
–
–
look at total error rate
ROC
class confusion matrix
(occasionally) total risk
• typically when one has a loss model
Summary: Classifiers
• Nearest-neighbor and k-nearest-neighbor classifiers
• Support vector machines
–
–
–
–
–
–
Linear classifiers
Margin maximization
Non-separable case
The kernel trick
Multi-class SVMs
Large-scale SVMs
• There are so many other classifiers out there
– Neural networks, boosting, decision trees/forests, …
Slide Credits
• Svetlana Lazebnik – UIUC
• David Forsyth - UIUC
37
Next class
• Object recognition
• Readings for next lecture:
– Forsyth and Ponce chapter 17
• Readings for today:
– Forsyth and Ponce chapter 16
38
Questions
39