Support Vector Machine. Tutorial. (and Statistical Learning Theory)


 Irene Shepherd
 1 years ago
 Views:
Transcription
1 Support Vector Machine (and Statistical Learning Theory) Tutorial Jason Weston NEC Labs America 4 Independence Way, Princeton, USA.
2 1 Support Vector Machines: history SVMs introduced in COLT92 by Boser, Guyon & Vapnik. Became rather popular since. Theoretically well motivated algorithm: developed from Statistical Learning Theory (Vapnik & Chervonenkis) since the 60s. Empirically good performance: successful applications in many fields(bioinformatics,tet,imagerecognition,...)
3 2 Support Vector Machines: history II Centralized website: Several tetbooks, e.g. An introduction to Support Vector Machines by Cristianini and ShaweTaylor is one. A large and diverse community work on them: from machine learning, optimization, statistics, neural networks, functional analysis, etc.
4 3 Support Vector Machines: basics [Boser, Guyon, Vapnik 92],[Cortes & Vapnik 95] margin margin Nice properties: conve, theoretically motivated, nonlinear with kernels..
5 4 Preliminaries: Machine learning is about learning structure from data. Although the class of algorithms called SVM s can do more, in this talk we focus on pattern recognition. So we want to learn the mapping: X Y,where Xis some object and y Yis a class label. Let s take the simplest case: 2class classification. So: R n, y {±1}.
6 5 Eample: Suppose we have 50 photographs of elephants and 50 photos of tigers. vs. We digitize them into piel images, so we have R n where n =10, 000. Now, given a new (different) photograph we want to answer the question: is it an elephant or a tiger? [we assume it is one or the other.]
7 6 Training sets and prediction models input/output sets X, Y training set ( 1,y 1 ),...,( m,y m ) generalization : given a previously seen X, find a suitable y Y. i.e., want to learn a classifier: y = f(, α),whereα are the parameters of the function. For eample, if we are choosing our model from the set of hyperplanes in R n,thenwehave: f(, {w, b}) =sign(w + b).
8 7 Empirical Risk and the true Risk We can try to learn f(, α) by choosing a function that performs well on training data: R emp (α) = 1 m m l(f( i,α),y i ) = Training Error i=1 where l is the zeroone loss function, l(y, ŷ) =1,ify ŷ,and0 otherwise. R emp is called the empirical risk. By doing this we are trying to minimize the overall risk: R(α) = l(f(, α), y)dp (, y) = Test Error where P(,y) is the (unknown) joint distribution function of and y.
9 8 Choosing the set of functions What about f(, α) allowing all functions from X to {±1}? Training set ( 1,y 1 ),...,( m,y m ) X {±1} Test set 1,..., m X, such that the two sets do not intersect. For any f there eists f : 1. f ( i )=f( i ) for all i 2. f ( j ) f( j ) for all j Based on the training data alone, there is no means of choosing which function is better. On the test set however they give different results. So generalization is not guaranteed. = a restriction must be placed on the functions that we allow.
10 9 Empirical Risk and the true Risk Vapnik & Chervonenkis showed that an upper bound on the true risk can be given by the empirical risk + an additional term: h(log( 2m h R(α) R emp (α)+ +1) log( η 4 ) m where h is the VC dimension of the set of functions parameterized by α. The VC dimension of a set of functions is a measure of their capacity or compleity. If you can describe a lot of different phenomena with a set of functions then the value of h is large. [VC dim = the maimum number of points that can be separated in all possible ways by that set of functions.]
11 10 VC dimension: The VC dimension of a set of functions is the maimum number of points that can be separated in all possible ways by that set of functions. For hyperplanes in R n, the VC dimension can be shown to be n +1.
12 11 VC dimension and capacity of functions Simplification of bound: Test Error Training Error + Compleity of set of Models Actually, a lot of bounds of this form have been proved (different measures of capacity). The compleity function is often called a regularizer. If you take a high capacity set of functions (eplain a lot) you get low training error. But you might overfit. If you take a very simple set of models, you have low compleity, but won t get low training error.
13 12 Capacity of a set of functions (classification) [Images taken from a talk by B. Schoelkopf.]
14 13 Capacity of a set of functions (regression) y sine curve fit hyperplane fit true function
15 14 Controlling the risk: model compleity Bound on the risk Confidence interval Empirical risk (training error) h 1 h* h n S1 S* S n
16 15 Capacity of hyperplanes Vapnik & Chervonenkis also showed the following: Consider hyperplanes (w ) =0where w is normalized w.r.t a set of points X such that: min i w i =1. The set of decision functions f w () =sign(w ) defined on X such that w A has a VC dimension satisfying h R 2 A 2. where R is the radius of the smallest sphere around the origin containing X. = minimize w 2 and have low capacity = minimizing w 2 equivalent to obtaining a large margin classifier
17 <w, > + b > 0 <w, > + b < 0 w { <w, > + b = 0}
18 { <w, > + b = 1} y i = 1 w { <w, > + b = +1} 2 1, y i = +1 Note: <w, 1 > + b = +1 <w, 2 > + b = 1 => <w, ( 1 2 )> = 2 => < w, w ( 1 2 ) > 2 = w { <w, > + b = 0}
19 16 Linear Support Vector Machines (at last!) So, we would like to find the function which minimizes an objective like: Training Error + Compleity term We write that as: 1 m m l(f( i,α),y i ) + Compleity term i=1 For now we will choose the set of hyperplanes (we will etend this later), so f() =(w )+b: 1 m m l(w i + b, y i ) + w 2 i=1 subject to min i w i =1.
20 17 Linear Support Vector Machines II That function before was a little difficult to minimize because of the step function in l(y, ŷ) (either 1 or 0). Let s assume we can separate the data perfectly. Then we can optimize the following: Minimize w 2, subject to: (w i + b) 1, if y i =1 (w i + b) 1, if y i = 1 The last two constraints can be compacted to: y i (w i + b) 1 This is a quadratic program.
21 18 SVMs : nonseparable case To deal with the nonseparable case, one can rewrite the problem as: Minimize: subject to: w 2 + C m i=1 ξ i y i (w i + b) 1 ξ i, ξ i 0 This is just the same as the original objective: 1 m m l(w i + b, y i ) + w 2 i=1 ecept l is no longer the zeroone loss, but is called the hingeloss : l(y, ŷ) =ma(0, 1 yŷ). This is still a quadratic program!
22 margin ξ i
23 19 Support Vector Machines  Primal Decision function: Primal formulation: f() =w + b min P (w, b)= 1 2 w 2 + C H 1 [ y i f( i )] } {{ } i } {{ } maimize margin minimize training error Ideally H 1 would count the number of errors, approimate with: H 1(z) Hinge Loss H 1 (z) =ma(0, 1 z) 0 z
24 20 SVMs : nonlinear case Linear classifiers aren t comple enough sometimes. SVM solution: Map data into a richer feature space including nonlinear features, then construct a hyperplane in that space so all other equations are the same! Formally, preprocess the data with: Φ() and then learn the map from Φ() to y: f() =w Φ()+b.
25 21 SVMs : polynomial mapping Φ:R 2 R 3 ( 1, 2 ) (z 1,z 2,z 3 ):=( 2 1, (2) 1 2, 2 2) 1 2 z 1 z 3 z 2
26 22 SVMs : nonlinear case II For eample MNIST handwriting recognition. 60,000 training eamples, test eamples, Linear SVM has around 8.5% test error. Polynomial SVM has around 1% test error
27 23 SVMs : full MNIST results Classifier Test Error linear 8.4% 3nearestneighbor 2.4% RBFSVM 1.4 % Tangent distance 1.1 % LeNet 1.1 % Boosted LeNet 0.7 % Translation invariant SVM 0.56 % Choosing a good mapping Φ( ) (encoding prior knowledge + getting right compleity of function class) for your problem improves results.
28 24 SVMs : the kernel trick Problem: the dimensionality of Φ() can be very large, making w hard to represent eplicitly in memory, and hard for the QP to solve. The Representer theorem (Kimeldorf & Wahba, 1971) shows that (for SVMs as a special case): m w = α i Φ( i ) for some variables α. Instead of optimizing w directly we can thus optimize α. The decision rule is now: f() = i=1 m α i Φ( i ) Φ()+b i=1 We call K( i,)=φ( i ) Φ() the kernel function.
29 25 Support Vector Machines  kernel trick II We can rewrite all the SVM equations we saw before, but with the w = m i=1 α iφ( i ) equation: Decision function: f() = i α i Φ( i ) Φ()+b = i α i K( i, )+b Dual formulation: min P (w, b)= 1 m 2 α i Φ( i ) 2 + C H 1 [ y i f( i )] i=1 i } {{ } } {{ } maimize margin minimize training error
30 26 Support Vector Machines  Dual But people normally write it like this: Dual formulation: min α D(α) = 1 2 α i α j Φ( i ) Φ( j ) i,j i y i α i s.t. Èi α i=0 0 y i α i C Dual Decision function: f() = i α i K( i, )+b Kernel function K(, ) is used to make (implicit) nonlinear feature map, e.g. Polynomial kernel: K(, )=( +1) d. RBF kernel: K(, )=ep( γ 2 ).
31 27 PolynomialSVMs The kernel K(, )=( ) d gives the same result as the eplicit mapping + dot product that we described before: Φ:R 2 R 3 ( 1, 2 ) (z 1,z 2,z 3 ):=( 2 1, (2) 1 2, 2 2) Φ(( 1, 2 ) Φ(( 1, 2)=( 2 1, (2) 1 2, 2 2) ( 2 1, (2) 1 2, 2 2) is the same as: = K(, )=( ) 2 =(( 1, 2 ) ( 1, 2)) 2 =( ) 2 = Interestingly, if d is large the kernel is still only requires n multiplications to compute, whereas the eplicit representation may not fit in memory!
32 28 RBFSVMs The RBF kernel K(, )=ep( γ 2 ) is one of the most popular kernel functions. It adds a bump around each data point: m f() = α i ep( γ i 2 )+b i=1 Φ.. ' Φ() Φ(') Using this one can get stateoftheart results.
33 29 SVMs : more results There is much more in the field of SVMs/ kernel machines than we could cover here, including: Regression, clustering, semisupervised learning and other domains. Lots of other kernels, e.g. string kernels to handle tet. Lots of research in modifications, e.g. to improve generalization ability, or tailoring to a particular task. Lots of research in speeding up training. Please see tet books such as the ones by Cristianini & ShaweTaylor or by Schoelkopf and Smola.
34 30 SVMs : software Lots of SVM software: LibSVM (C++) SVMLight (C) As well as complete machine learning toolboes that include SVMs: Torch (C++) Spider (Matlab) Weka (Java) All available through
More Generality in Efficient Multiple Kernel Learning
Manik Varma manik@microsoft.com Microsoft Research India, Second Main Road, Sadashiv Nagar, Bangalore 560 080, India Bodla Rakesh Babu rakeshbabu@research.iiit.net CVIT, International Institute of Information
More informationLearning with Local and Global Consistency
Learning with Local and Global Consistency Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf Max Planck Institute for Biological Cybernetics, 7276 Tuebingen, Germany
More informationAn Introduction to Variable and Feature Selection
Journal of Machine Learning Research 3 (23) 11571182 Submitted 11/2; Published 3/3 An Introduction to Variable and Feature Selection Isabelle Guyon Clopinet 955 Creston Road Berkeley, CA 9478151, USA
More informationA RateDistortion OneClass Model and its Applications to Clustering
and its Applications to Clustering Koby Crammer crammer@cis.upenn.edu Partha Pratim Talukdar partha@cis.upenn.edu Department of Computer & Information Science, University of Pennsylvania, Philadelphia,
More informationLearning with Local and Global Consistency
Learning with Local and Global Consistency Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf Max Planck Institute for Biological Cybernetics, 7276 Tuebingen, Germany
More informationLearning Deep Architectures for AI. Contents
Foundations and Trends R in Machine Learning Vol. 2, No. 1 (2009) 1 127 c 2009 Y. Bengio DOI: 10.1561/2200000006 Learning Deep Architectures for AI By Yoshua Bengio Contents 1 Introduction 2 1.1 How do
More informationCovering Number Bounds of Certain Regularized Linear Function Classes
Journal of Machine Learning Research 2 (2002) 527550 Submitted 6/200; Published 3/2002 Covering Number Bounds of Certain Regularized Linear Function Classes Tong Zhang T.J. Watson Research Center Route
More informationObject Detection with Discriminatively Trained Part Based Models
1 Object Detection with Discriminatively Trained Part Based Models Pedro F. Felzenszwalb, Ross B. Girshick, David McAllester and Deva Ramanan Abstract We describe an object detection system based on mixtures
More informationHow to Use Expert Advice
NICOLÒ CESABIANCHI Università di Milano, Milan, Italy YOAV FREUND AT&T Labs, Florham Park, New Jersey DAVID HAUSSLER AND DAVID P. HELMBOLD University of California, Santa Cruz, Santa Cruz, California
More informationLearning to Select Features using their Properties
Journal of Machine Learning Research 9 (2008) 23492376 Submitted 8/06; Revised 1/08; Published 10/08 Learning to Select Features using their Properties Eyal Krupka Amir Navot Naftali Tishby School of
More informationRegression. Chapter 2. 2.1 Weightspace View
Chapter Regression Supervised learning can be divided into regression and classification problems. Whereas the outputs for classification are discrete class labels, regression is concerned with the prediction
More informationThe Pyramid Match Kernel: Discriminative Classification with Sets of Image Features
The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features Kristen Grauman and Trevor Darrell Massachusetts Institute of Technology Computer Science and Artificial Intelligence
More informationFlexible and efficient Gaussian process models for machine learning
Flexible and efficient Gaussian process models for machine learning Edward Lloyd Snelson M.A., M.Sci., Physics, University of Cambridge, UK (2001) Gatsby Computational Neuroscience Unit University College
More informationGaussian Processes for Machine Learning
Gaussian Processes for Machine Learning Adaptive Computation and Machine Learning Thomas Dietterich, Editor Christopher Bishop, David Heckerman, Michael Jordan, and Michael Kearns, Associate Editors Bioinformatics:
More informationGaussian Processes in Machine Learning
Gaussian Processes in Machine Learning Carl Edward Rasmussen Max Planck Institute for Biological Cybernetics, 72076 Tübingen, Germany carl@tuebingen.mpg.de WWW home page: http://www.tuebingen.mpg.de/ carl
More informationA Tutorial on Spectral Clustering
A Tutorial on Spectral Clustering Ulrike von Luxburg Max Planck Institute for Biological Cybernetics Spemannstr. 38, 7276 Tübingen, Germany ulrike.luxburg@tuebingen.mpg.de This article appears in Statistics
More informationOptimization with SparsityInducing Penalties. Contents
Foundations and Trends R in Machine Learning Vol. 4, No. 1 (2011) 1 106 c 2012 F. Bach, R. Jenatton, J. Mairal and G. Obozinski DOI: 10.1561/2200000015 Optimization with SparsityInducing Penalties By
More informationProbability Estimates for Multiclass Classification by Pairwise Coupling
Journal of Machine Learning Research 5 (2004) 975005 Submitted /03; Revised 05/04; Published 8/04 Probability Estimates for Multiclass Classification by Pairwise Coupling TingFan Wu ChihJen Lin Department
More informationTHE PROBLEM OF finding localized energy solutions
600 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 3, MARCH 1997 Sparse Signal Reconstruction from Limited Data Using FOCUSS: A Reweighted Minimum Norm Algorithm Irina F. Gorodnitsky, Member, IEEE,
More informationSteering User Behavior with Badges
Steering User Behavior with Badges Ashton Anderson Daniel Huttenlocher Jon Kleinberg Jure Leskovec Stanford University Cornell University Cornell University Stanford University ashton@cs.stanford.edu {dph,
More informationExploiting generative models discriminative classifiers
Exploiting generative models discriminative classifiers In Tommi S. Jaakkola* MIT Artificial Intelligence Laboratorio 545 Technology Square Cambridge, MA 02139 David Haussler Department of Computer Science
More informationFoundations of Data Science 1
Foundations of Data Science John Hopcroft Ravindran Kannan Version /4/204 These notes are a first draft of a book being written by Hopcroft and Kannan and in many places are incomplete. However, the notes
More informationSee All by Looking at A Few: Sparse Modeling for Finding Representative Objects
See All by Looking at A Few: Sparse Modeling for Finding Representative Objects Ehsan Elhamifar Johns Hopkins University Guillermo Sapiro University of Minnesota René Vidal Johns Hopkins University Abstract
More informationThe Backpropagation Algorithm
7 The Backpropagation Algorithm 7. Learning as gradient descent We saw in the last chapter that multilayered networks are capable of computing a wider range of Boolean functions than networks with a single
More informationRepresentation Learning: A Review and New Perspectives
1 Representation Learning: A Review and New Perspectives Yoshua Bengio, Aaron Courville, and Pascal Vincent Department of computer science and operations research, U. Montreal also, Canadian Institute
More informationA Fast Learning Algorithm for Deep Belief Nets
LETTER Communicated by Yann Le Cun A Fast Learning Algorithm for Deep Belief Nets Geoffrey E. Hinton hinton@cs.toronto.edu Simon Osindero osindero@cs.toronto.edu Department of Computer Science, University
More informationLearning over Sets using Kernel Principal Angles
Journal of Machine Learning Research 4 (2003) 913931 Submitted 3/03; Published 10/03 Learning over Sets using Kernel Principal Angles Lior Wolf Amnon Shashua School of Engineering and Computer Science
More informationVisual Categorization with Bags of Keypoints
Visual Categorization with Bags of Keypoints Gabriella Csurka, Christopher R. Dance, Lixin Fan, Jutta Willamowski, Cédric Bray Xerox Research Centre Europe 6, chemin de Maupertuis 38240 Meylan, France
More informationA Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine Learning Pedro Domingos Department of Computer Science and Engineering University of Washington Seattle, WA 981952350, U.S.A. pedrod@cs.washington.edu ABSTRACT
More informationWSABIE: Scaling Up To Large Vocabulary Image Annotation
WSABIE: Scaling Up To Large Vocabulary Image Annotation Jason Weston 1 and Samy Bengio 1 and Nicolas Usunier 2 1 Google, USA 2 Université Paris 6, LIP6, France {jweston,bengio}@google.com nicolas.usunier@lip6.fr
More information