Survey

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Document related concepts

Transcript

CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu Lecture 24 – Classifiers 1 Schedule • Last class – We continued on segmentation, specifically mean field and graph cuts but could not finish • Today – Classifiers • Readings for today: – Forsyth and Ponce chapter 16 2 Classifiers • Given a feature representation for images, how do we learn a model for distinguishing features from different classes? • Take a measurement x, predict a bit (yes/no; 1/-1; 1/0; etc) • Today: • • Nearest neighbor classifiers Linear classifiers: support vector machines • Other techniques: • Boosting, Decision trees and forests, Deep neural networks Image classification • The big problems – Image classification • eg this picture contains a parrot – Object detection • eg in this box in the picture is a parrot • Strategy – Generate features from image (there are many quite complex strategies) – Put in one or more classifiers 4 Image classification - scenes Image classification - material Classifiers Given a feature representation for images, how do we learn a model for distinguishing features from different classes? Decision boundary Zebra Non-zebra Nearest Neighbor Classifier • Assign label of nearest training data point to each test data point from Duda et al. Classifiers • Take a measurement x, predict a bit (yes/no; 1/-1; 1/0; etc) • Strategies: – non-parametric • nearest neighbor – probabilistic • histogram • logistic regression – decision boundary • SVM K-Nearest Neighbors • For a new point, find the k closest points from training data • Labels of the k points “vote” to classify k=5 Histogram based classifiers • Represent class-conditional densities with histogram • Advantage: – estimates become quite good • (with enough data!) • Disadvantage: – Histogram becomes big with high dimension • but maybe we can assume feature independence? Example: Finding skin • Skin has a very small range of (intensity independent) colours, and little texture – Compute an intensity-independent colour measure, check if colour is in this range, check if there is little texture (median filter) – See this as a classifier - we can set up the tests by hand, or learn them. Histogram classifier for skin Curse of dimension • Can’t build a histogram based classifier for when feature space is high-dimensional – try R, G, B, and some texture features – Fails when there are too many histogram buckets Distance functions for bags of features • Euclidean distance: D(h , h ) (h (i) h (i)) N 2 1 • L1 distance: 2 i 1 1 2 N D(h1 , h 2 ) | h1 (i ) h 2 (i ) | i 1 • χ2 distance: N h1 (i) h 2 (i) 2 i 1 h1 (i ) h 2 (i ) D(h1 , h 2 ) • Histogram intersection (similarity): N I (h1 , h 2 ) min(h1 (i ), h 2 (i )) • Hellinger kernel (similarity): i 1 N K (h1 , h 2 ) h1 (i ) h 2 (i ) i 1 Linear classifiers • Find linear function (hyperplane) to separate positive and negative examples xi positive : xi w b 0 xi negative : xi w b 0 Which hyperplane is best? Support vector machines • Find hyperplane that maximizes the margin between the positive and negative examples xi positive ( yi 1) : xi w b 1 xi negative ( yi 1) : xi w b 1 For support vectors, Distance between point and hyperplane: xi w b 1 | xi w b | || w || Therefore, the margin is 2 / ||w|| Support vectors Margin C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998 What if the data is not linearly separable? 1 2 min w C w ,b 2 n max 0,1 y (w x i 1 i i b) +1 0 Margin -1 • Demo: http://cs.stanford.edu/people/karpathy/svmjs/demo Nonlinear SVMs • Datasets that are linearly separable work out great: x 0 • But what if the dataset is just too hard? x 0 • We can map it to a higher-dimensional space: x2 0 x Slide credit: Andrew Moore Nonlinear SVMs • General idea: the original input space can always be mapped to some higherdimensional feature space where the training set is separable: Φ: x → φ(x) Slide credit: Andrew Moore Nonlinear SVMs • The kernel trick: instead of explicitly computing the lifting transformation φ(x), define a kernel function K such that K(x,y) = φ(x) · φ(y) • This gives a nonlinear decision boundary in the original feature space: y ( x ) ( x) b y K ( x , x) b i i i i i i i i C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998 Nonlinear kernel: Example 2 ( x ) ( x , x ) • Consider the mapping x2 ( x) ( y) ( x, x 2 ) ( y, y 2 ) xy x 2 y 2 K ( x, y) xy x 2 y 2 Polynomial kernel: K (x, y ) (c x y ) d Gaussian kernel • Also known as the radial basis function (RBF) kernel: 2 1 K (x, y ) exp 2 x y • The corresponding mapping φ(x) is infinitedimensional! Gaussian kernel SV’s Kernels for bags of features • Histogram intersection kernel: N I (h1 , h 2 ) min(h1 (i ), h 2 (i )) i 1 N • Hellinger kernel: K (h1 , h 2 ) h1 (i) h 2 (i) i 1 • Generalized Gaussian kernel: 1 2 K (h1 , h 2 ) exp D(h1 , h 2 ) A D can be L1, Euclidean, χ2 distance, etc. J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, Local Features and Kernels for Classifcation of Texture and Object Categories: A Comprehensive Study, IJCV 2007 Summary: SVMs for image classification 1. Pick an image representation (in our case, bag of features) 2. Pick a kernel function for that representation 3. Compute the matrix of kernel values between every pair of training examples 4. Feed the kernel matrix into your favorite SVM solver to obtain support vectors and weights 5. At test time: compute kernel values for your test example and each support vector, and combine them with the learned weights to get the value of the decision function What about multi-class SVMs? • Unfortunately, there is no “definitive” multi-class SVM formulation • In practice, we have to obtain a multi-class SVM by combining multiple two-class SVMs • One vs. others – Traning: learn an SVM for each class vs. the others – Testing: apply each SVM to test example and assign to it the class of the SVM that returns the highest decision value • One vs. one – Training: learn an SVM for each pair of classes – Testing: each learned SVM “votes” for a class to assign to the test example SVMs: Pros and cons • Pros – Many publicly available SVM packages: http://www.kernel-machines.org/software – Kernel-based framework is very powerful, flexible – SVMs work very well in practice, even with very small training sample sizes • Cons – No “direct” multi-class SVM, must combine two-class SVMs – Computation, memory • During training time, must compute matrix of kernel values for every pair of examples • Learning can take a very long time for large-scale problems SVMs for large-scale datasets • Efficient linear solvers • LIBLINEAR, PEGASOS • Explicit approximate embeddings: define an explicit mapping φ(x) such that φ(x) · φ(y) approximates K(x,y) and train a linear SVM on top of that embedding • • Random Fourier features for the Gaussian kernel (Rahimi and Recht, 2007) Embeddings for additive kernels, e.g., histogram intersection (Maji et al., 2013, Vedaldi and Zisserman, 2012) Evaluating classifiers • Always – train on training set, evaluate on test set • test set performance might/should be worse than training set • Options – Total error rate • always less than 50% for two class – Receiver operating curve • because we might use different thresholds – Class confusion matrix • for multiclass Receiver operating curve Classifiers: Crucial Points • Classifiers accept features, return decision – and are often trained using data • Very mature training processes now – Always try standard methods first • SVM • kNN • Logistic regression • Evaluate with separate test data – – – – look at total error rate ROC class confusion matrix (occasionally) total risk • typically when one has a loss model Summary: Classifiers • Nearest-neighbor and k-nearest-neighbor classifiers • Support vector machines – – – – – – Linear classifiers Margin maximization Non-separable case The kernel trick Multi-class SVMs Large-scale SVMs • There are so many other classifiers out there – Neural networks, boosting, decision trees/forests, … Slide Credits • Svetlana Lazebnik – UIUC • David Forsyth - UIUC 37 Next class • Object recognition • Readings for next lecture: – Forsyth and Ponce chapter 17 • Readings for today: – Forsyth and Ponce chapter 16 38 Questions 39