# Multiclass Classification Class 06, 25 Feb 2008 Ryan Rifkin

Save this PDF as:

Size: px
Start display at page:

Download "Multiclass Classification. 9.520 Class 06, 25 Feb 2008 Ryan Rifkin"

## Transcription

1 Multiclass Classification Class 06, 25 Feb 2008 Ryan Rifkin

2 It is a tale Told by an idiot, full of sound and fury, Signifying nothing. Macbeth, Act V, Scene V

3 What Is Multiclass Classification? Each training point belongs to one of N different classes. The goal is to construct a function which, given a new data point, will correctly predict the class to which the new point belongs.

4 What Isn t Multiclass Classification? There are many scenarios in which there are multiple categories to which points belong, but a given point can belong to multiple categories. In its most basic form, this problem decomposes trivially into a set of unlinked binary problems, which can be solved naturally using our techniques for binary classification.

5 A First Idea Suppose we knew the density, p i (x), for each of the N classes. Then, we would predict using f(x) = arg max i 1,...,N p i(x). Of course we don t know the densities, but we could estimate them using classical techniques.

6 The Problem With Densities, and Motivation Estimating densities is hard, especially in high dimensions with limited data. For binary classification tasks, we have seen that directly estimating a smooth separating function gives better results than density estimation (SVM, RLSC). Can we extend these approaches usefully to the multiclass scenario?

7 A Simple Idea One-vs-All Classification Pick a good technique for building binary classifiers (e.g., RLSC, SVM). Build N different binary classifiers. For the ith classifier, let the positive examples be all the points in class i, and let the negative examples be all the points not in class i. Let f i be the ith classifier. Classify with f(x) = argmax i f i (x).

8 Another Simple Idea All-vs-All Classification Build N(N 1) classifiers, one classifier to distinguish each pair of classes i and j. Let f ij be the classifier where class i were positive examples and class j were negative. Note f ji = f ij. Classify using f(x) = argmax i j f ij (x). Also called all-pairs or one-vs-one classification.

9 The Truth OVA and AVA are so simple that many people invented them independently. It s hard to write papers about them. So there s a whole cottage industry in fancy, sophisticated methods for multiclass classification. To the best of my knowledge, choosing properly tuned regularization classifiers (RLSC, SVM) as your underlying binary classifiers and using one-vs-all (OVA) or all-vs-all (AVA) works as well as anything else you can do. If you actually have to solve a multiclass problem, I strongly urge you to simply use OVA or AVA, and not worry about anything else. The choice between OVA and AVA is largely computational.

10 OVA vs. AVA Viewed naively, AVA seems faster and more memory efficient. It requires O(N 2 ) classifiers instead of O(N), but each classifier is (on average) much smaller. If the time to build a classifier is superlinear in the number of data points, AVA is a better choice. With SVMs, AVA s probably best. However, if you can solve one RLS problem over your entire data set using a matrix factorization, you get multiclass classification essentially for free (see RLS lecture). So with RLS, OVA s a great choice.

11 Other Approaches There have been two basic approaches to extending regularization ideas to multiclass classification: Single Machine approaches try to solve a single optimization problem that trains many binary classifiers simultaneously. Error Correcting Code approaches try to combine binary classifiers in a way that lets you exploit decorrelations and correct errors. These approaches are not completely exclusive.

12 Weston and Watkins, Vapnik The first single machine approach: min f 1,...,f N H,ξ R l(n 1) Ni=1 f i 2 K + C l i=1 j y i ξ ij subject to : f yi (x i ) + b yi f j (x i ) + b j + 2 ξ ij ξ ij 0 Key idea. Suppose that point i is in class y i. Then, for j y i, we want (abusing our notation w.r.t. b), f yi (x i ) f j (x i ) 2, or we pay a linear penalty of ξ ij.

13 WW Analysis I This idea seems intuitively reasonable. Is it good? Weston and Watkins perform experiments. On 2 out of 5 datasets, they find that their approach performs substantially better than OVA, and about the same on the rest. However, they state that to enable comparison, for each algorithm C = was chosen (the training data must be classified without error), so they are performing ERM, not regularization (C = λ = 0). A Gaussian kernel was used, with σ the same for each method (not necessarily a good idea), and no information about how this σ was chosen.

14 WW Analysis II Under what circumstances would we expect this method to outperform a OVA approach? Tough to say. We d need a situation where it would be hard to actually separate the data, but where there exist meaningful subsets of the data where even though we can t assign a positive value to the correct class, we can assign a less negative value to it than other classes. Or, we need subsets where even though we re assigning positive values to some incorrect class, we re assigning even more strongly positive values to the correct classes. Challenge: Come up with a set of examples that actually has this property. Double challenge: Have it hold when the kernel is a properly-tuned Gaussian.

15 WW Analysis III There is no evidence that this scheme is any more accurate than an OVA scheme. It is, however, much slower. Instead of N problems of size l, we need to solve a problem of size (N 1)l. Moreover, although the authors derive a dual problem, they don t show any way to decompose it; the resulting problem is much more complex than a standard SVM.

16 Lee, Lin and Wahba: Motivation In an earlier paper, Lin showed that asymptotically, as we let l and λ 0, the solution of an SVM will tend to ( f(x) = sign p(x) 1 ). 2 This is good for binary classification. Now consider multiclass classification with an OVA scheme. In regions where there is a dominant class i for which p(x) > 1 2, all is good. If there isn t, then all N of the OVA functions will return 1, and we will be unable to recover the most likely class.

17 Lee, Lin and Wahba: Notation For i 1,..., N, define v i as the N dimensional vector with a 1 in the ith coordinate and N 1 1 elsewhere. The vector v i serves as the target for points in class i we try to get function outputs that are very close to the entries of v i. Given a training point x i, we try to make f j = 1 N 1 for j y i, and then we also require that N j=1 f(x i ) = 0. This leads us to...

18 Lee, Lin and Wahba: Formulation min 1 li=1 Nj=1,j yi f 1,...,f N H l (f j (x i ) + N 1 1 ) + + λ N j=1 f j 2 K K subject to : Nj=1 f j (x) = 0 x

19 Lee, Lin and Wahba: Analysis Asymptotically, this formulation gives f i (x) = { 1 iff i = argmax pj (x) N 1 1 otherwise In turn, this means that our final multiclass classification function will be f(x) = argmax p j (x).

20 Lee, Lin and Wahba: Issues I Even under the conditions of the analysis, this produces a different result from OVA only when argmax p j (x) < 1 2 the Bayes error rate must be > 1 2, indicating a very tough problem that should likely be modelled differently. The problem with an OVA-SVM scheme relies on the linearity of the SVM loss. If we use instead RLSC or a square-loss SVM, these problems disappear, and the Bayes rule is again recovered asymptotically.

21 Lee, Lin and Wahba: Issues II Like the WW formulation, this formulation is big, and no decomposition method is provided. This is an asymptotic analysis. It requires n and λ 0, and no rates are provided. But asymptotically, density estimation will allow us to recover the optimal Bayes rule. The burden is on the authors to show that there is a useful middle ground where this performs well.

22 Lee, Lin and Wahba: Experiments Two toy examples. In one, no comparison is made to other techniques. In the other, they compare to OVA. The data is 200 points in one-dimension in thre classes, constructed so that class 2 never has conditional probability > 50%. The LLW approach predicts class 2 in the region where it s more likely than any other class (total error rate.389, and the OVA system fails there (total error.4243). I cannot understand how their parameters were chosen.

23 Crammer and Singer: Formulation They consider a formulation that is a modified version of WW: min f 1,...,f N H,ξ R l subject to : Ni=1 f i 2 K + C l i=1 j y i ξ i f yi (x i ) f j (x i ) + 1 ξ i ξ i 0 Key difference: there is only one slack variable ξ i per data point, rather than N 1 slack variables per data point as in WW. Instead of paying for each class j i for which f i (x) < f j (x) + 1, they pay only for the largest f j (x).

24 Crammer and Singer: Development The majority of the C&S paper is devoted to efficiently solving the formulation. The Lagrangian dual is taken, dot products are replaced with kernel products, and a dual decomposition algorithm is developed. This algorithm is substantially more complicated than the SVM algortihm; the mathematics is fairly involved, but elegant. In the experimental section of their paper, C&S make claims as to both the speed and the accuracy of their method...

25 Crammer and Singer: Speed C&S claim that their method is orders of magnitude faster than than an OVA approach. However: They benchmark the OVA approach by using Platt s 1998 results, in which he implemented SMO with no caching of kernel products. In contrast, their system used 2 GB of RAM to cache kernel products. The paper states that the fastest version has two more technical improvements which are not discussed here but will be documented in the code that we will shortly make available ; the code was never made available.

26 Crammer and Singer: Accuracy C&S consider data sets from the UCI repository. Their chart shows the difference in error rates between their system and an OVA system, but not the actual error rates. The largest differences appeared to be on the satimage (difference of approximately 6.5%) and shuttle (difference of approximately 3%) data sets. Through personal communication, Crammer indicated that the actual error rates for his system on these two data sets were 8.1% and 0.1%, respectively. In my own one-vs-all experiments on these data sets, I achieved error rates of 7.9% for the satimage data and 0.35% on the shuttle data. These differences are much smaller than the differences reported by C&S.

27 Dietterich and Bakiri: Introducing the ECC Approach Consider a { 1,1}-valued matrix M of size N by F where F is the number of classifiers to be trained. The ith column of the matrix induces a partition of the classes into two metaclasses, where a point x i is placed in the positive metaclass for the jth classifier if and only if M yi j = 1. When faced with a new test point x, we compute f 1 (x),..., f F (x), take the signs of these values, and then compare the Hamming distance between the resulting vector and each row of the matrix, choosing the minimizer: f(x) = arg min r 1,...,N F i=1 ( 1 sign(mri f i (x)) 2 ).

28 Dietterich and Bakiri: Introducing the ECC Approach D&B suggest that M be constructed to have good errorcorrecting properties if the minimum Hamming distance between rows of M is d, then the resulting multiclass classification will be able to correct any d 1 2 errors. In learning, good column separation is also important two identical (or opposite) columns will make identical errors. This highlights the key difference between the multiclass machine learning framework and standard errorcorrecting code applications.

29 Dietterich and Bakiri: Experiments A large number of experiments were performed. Decision trees and feed-forward neural networks were used as base learners, and several methods of generating codes were tried. In general, it seems that their approach outperformed OVA approaches. However, differences were often small, the quality of the underlying binary classifiers is unknown, and often only relative performance results are given, so this work is difficult to evaluate.

30 Allwein, Schapere and Singer: Generalizing D&B AS&S generalize D&B in two main ways: Allow 0-entries (in addition to 1 and -1) in the M matrix if M ij = 0, then examples from class i are simply not used in classifier j. Use loss-based decoding to classify examples instead of taking the sign of the output of each classifier, compute the actual loss, using the training loss function (hinge loss for SVM, square loss for RLSC). AS&S observe that OVA approaches, all-pairs approaches, the ECC approach of D&B, and generalizations of this to include 0 in the matrix all fit into their framework.

31 Allwein, Schapere and Singer: Experimental Setup AS&S tested multiclass SVMs and AdaBoost using five different matrices: OVA and AVA: one-vs-all and all-pairs COMPLETE: O(2 N ) columns DENSE and SPARSE: Randomized codes with and without zeros in M. Their conclusion: For SVM, it is clear that the widely used one-against-all code is inferior to all the other codes we tested.

32 Allwein, Schapere and Singer: Experimental Conditions AS&S performed all SVM experiments using a polynomial kernel of degree 4; no justification for this choice is made. No information about regularization parameters is given. Fortunately, AS&S gave actual numerical results for their experiments. The difference in performance was large on only two data sets: yeast and satimage. I performed my own experiments on these data sets, using Gaussian kernels. The Guassian parameter σ was tuned by cheating on the test set, although the accuracy rates were not very sensitive to choice of σ (and see later).

33 Allwein, Schapere and Singer: Experimental Comparison OVA AVA COM DEN SPA Allwein et al Rifkin Comparison of results for the satimage data set. OVA AVA COM DEN SPA Allwein et al Rifkin Comparison of results for the yeast data set.

34 Fürnkrnaz: Round-Robin Classification Using Ripper, a rule-based learner, as an underlying binary learner, Fürnrkanz showed experimentally that allpairs substantially outperformed one-vs-all across a variety of data sets from UCI. His experiments include the satimage and yeast data sets discussed above, with his best all-pairs system achieving accuracy rates of 10.4% and 41.8%, respectively. These results cannot be directly compared to my numbers (7.9% and 39.0%) because I cheated and because the yeast setup was somewhat different, but see below.

35 Hsu and Lin: A Metastudy Hsu and Lin empirically compared different methods of multiclass classification using SVMs as binary learners. They tried five methods: OVA, AVA, DAG (similar to AVA, faster at testing time), the method of Crammer & Singer, and the method of Weston & Watkins. They used Gaussian kernels, tuned on validation sets, and reported all the numbers. They conclude that one-against-one and DAG methods are more suitable for practical use than the other methods. We take the numbers directly from their paper, and presented them in a format that suits us...

36 Hsu and Lin: Results Best Worst Diff Size Size * Diff iris wine glass vowel vehicle segment dna satimage letter shuttle A view of the multiclass results of Hsu and Lin for RBF kernels.

37 New Experiments With Aldebaro Klautau of UCSD, I performed a set of experiments on data from the UCI data set. There were three main things we wanted to explore, in a well-controlled, (hopefully) reproducible setting: Revisiting the data sets Fürnkranz used, but using welltuned SVMs as the base learners. Comparing the five different matrices of Allwein et al., across a range of data sets. Comparing RLSC and SVM.

38 Protocol Details of the protocol are found in the papers. All experiments were done with a Gaussian kernel. For a given experiments, the same parameters (σ and C (or λ)) were used for all machines. All parameters were found by crossvalidation. We focussed specifically on data sets on which Fürnkranz had found a significant difference between OVA and AVA. But what is significance?

39 McNemar s Test, I Fürnkranz used McNemar s test to assess the significance of the difference in performance of two classifiers. We compute the number of times (over the test set) each of these events occur: both classifiers were correct on (CC) both classifiers were incorrect (II) A was correct, B incorrect (CI) B was correct, A incorrect (IC)

40 McNemar s Test, II McNemar s Test uses the observation that, if the classifiers have equal error rates, then CI and IC should be equally frequent. This test is good because it requires very few assumptions (just that the observations are paired). It s bad because it ignores the number of examples on which the two systems agree, and (directly related) it does not provide a confidence interval. We therefore decided to introduce...

41 A Bootstrap Test For Comparing Classifiers We calculate the empirical probabilities of the four events CC, CI, IC, and II. Then, a large number (in our experiments, 10,000) of times, we generate bootstrap samples containing l data points, each of which is simply an occurrence of CC, CI, IC, or II with the appropriate probability. We then calculate confidence intervals (in our experiments, 90%) on the difference in performance (CI - IC). Other, better approaches may be possible.

42 Data Sets Used # att / average / min / max baseline Name train test class # nom # examples per class error (%) soybean-large / / 1 / letter / / 576 / satimage / / 409 / abalone / / 0 / optdigits / / 376 / glass / / 0 / car / / 65 / spectrometer / / 1 / yeast / / 5 / page-blocks / / 28 /

43 Reproducing Fürnkranz s Experiments Data Set Current Experiments Furnkranz s paper soybean-large letter satimage abalone optdigits glass car spectrometer yeast page-blocks

44 Code Matrix Sizes Name OVA AVA COM DEN SPA soybean-large letter satimage abalone optdigits glass car spectrometer e yeast page-blocks

45 Results: SVM AVA vs. OVA Data Set AVA OVA DIFF AGREE BOOTSTRAP soybean-large [-0.008, 0.019] letter [0.008, 0.015] satimage [-5E-4, 0.008] abalone [-0.102, ] optdigits [0.006, 0.016] glass [-0.047, 0.037] car [-0.016, ] spectrometer [-0.143, ] yeast [-0.006, 0.021] page-blocks [-0.002, 0.002]

46 Results: SVM DENSE vs. OVA Data Set DEN OVA DIFF AGREE BOOTSTRAP soybean-large [-0.019, 0.013] letter [5E-4, 0.004] satimage [-0.006, 0.003] abalone [-0.092, ] optdigits [-0.004, 0.002] glass [-0.042, 0.014] car spectrometer [-0.011, 0.026] yeast [-0.011, 0.011] page-blocks

47 Results: SVM SPARSE vs. OVA Data Set SPA OVA DIFF AGREE BOOTSTRAP soybean-large [-0.011, 0.016] letter [0.005, 0.011] satimage [0.003, 0.018] abalone [-0.067, ] optdigits [-0.002, 0.008] glass [-0.070, 0.033] car [-0.011, ] spectrometer [-0.038, 0.019] yeast [-0.015, 0.013] page-blocks [0.001, 0.007]

48 Results: SVM COMPLETE vs. OVA Data Set COM OVA DIFF AGREE BOOTSTRAP soybean-large letter satimage [-1E-3, 1E-3] abalone optdigits [-0.003, 0.002] glass [-0.042, 0.014] car [5.79E-4, 0.003] spectrometer yeast [-0.028, ] page-blocks [-0.002, 0.004]

49 Results: FÜRNRKANZ vs. SVM OVA Data Set FUR OVA DIFF AGREE BOOTSTRAP soybean-large [.056,.109] letter [.043,.057] satimage [.0345,.055] abalone [-.083, ] optdigits [0.035, 0.056] glass [-0.098, 0.005] car [0.006, 0.020] spectrometer [-0.060, 0.017] yeast [-0.005, ] page-blocks [-0.012, ]

50 Results: SVM OVA vs. RLSC OVA Data Set RLSC SVM DIFF AGREE BOOTSTRAP soybean-large [-0.008, 0.013] letter satimage [-0.004, 0.006] abalone [-0.099, ] optdigits [-0.007, 0.003] glass [-0.037, 0.047] car [0.009, 0.020] spectrometer [-0.036, 0.009] yeast [-0.016, 0.011] page-blocks [-0.004, 0.001]

51 Results: SVM AVA vs. RLSC AVA Data Set RLSC SVM DIFF AGREE BOOTSTRAP soybean-large [0.000, 0.037] letter satimage [-0.013, ] abalone [-0.009, 0.034] optdigits [-0.013, ] glass [-0.047, 0.028] car [0.013, 0.024] spectrometer [0.036, 0.092] yeast [-0.025, 0.005] page-blocks [-0.003, 0.003]

52 Results: SVM DENSE vs. RLSC DENSE Data Set RLSC SVM DIFF AGREE BOOTSTRAP soybean-large [0.011, 0.040] letter [-0.002, 0.009] abalone [-0.025, 0.017] optdigits [-0.006, 0.003] glass [-0.037, 0.042] car spectrometer [-0.038, 0.008] yeast [-0.016, 0.009] page-blocks

53 Results: SVM SPARSE vs. RLSC SPARSE Data Set RLSC SVM DIFF AGREE BOOTSTRAP soybean-large [0.000, 0.027] letter satimage [-0.011, 0.003] abalone [-0.043, ] optdigits [0.001, 0.011] glass [-0.037, 0.047] car [0.028, 0.043] spectrometer [-0.024, 0.021] yeast [-0.005, 0.020] page-blocks [-0.006, 1.83E-4]

54 Towards a Theory, I Recall that to solve an RLSC problem, we solve a linear system of the form (K + λli)c = y. Because this is a linear system, the vector c is a linear function of the right hand side y. Define the vector y i as y i j = { 1 if yj = i 0 otherwise

55 Towards a Theory, II Now, suppose that we solve N RLSC problems of the form (K + λli)c i = y i, and denote the associated functions f c 1,..., f c N. Now, for any possible right hand side y for which the y i and y j are equal, whenever x i and x j are in the same class, we can calculate the associated c vector from the c i without solving a new RLSC system. In particular, if we let m i (i {1,..., N}) be the y value for points in class i, then the associated solution vector c is given by c = N i=1 c i m i.

56 Towards a Theory, III For any code matrix M not containing any zeros, we can compute the output of the coding system using only the entries of the coding matrix and the outputs of the underlying one-vs-all classifiers. In particular, we do not need to actually train the classifiers associated with the coding matrix. We can simply use the appropriate linear combination of the underlying one-vs-all classifiers. For any r {1,..., N}, it can be shown that = F i=1 N j=1 j r L(M ri f i (x)) D rj f j (x), (1) where C rj F i=1 M ri M ji and D rj (F C rj ).

57 Towards a Theory, IV Noting that D rj is guaranteed to be positive under the basic assumption that no two rows of the coding matrix are identical, we see that f(x) = arg min r {1,...,N} = arg min r {1,...,N} F i=1 N j=1 j r L(M ri f i (x)) D rj f j (x).

58 Towards a Theory: Class Symmetry I We define a coding matrix to be class-symmetric if D ij (and therefore C ij as well) is independent of i and j (assuming i j). Note that a sufficient (but not necessary) condition for a coding-matrix to class symmetric is if, whenever it contains a column containing k 1 s and n k -1 s, all N! such columns are included. The OVA and COMk!(N k)! PLETE schemes are class-symmetric, while the DENSE scheme is in general not (the AVA and SPARSE schemes include zeros in the coding matrix, and are not addressed by this analysis).

59 Towards a Theory: Class Symmetry II For class-symmetric schemes, D rj is independent of the choice of r and j, and can be denoted as D. For these schemes, f(x) = arg min r 1,...,N D = arg min r 1,...,N N j=1 j r N j=1 j r = arg max r 1,...,N fr (x), f j (x) f j (x) When RLSC is used as the underlying binary learner for class-symmetric coding matrices containing no zeros, the predictions generated are identical to those of the one-vsall scheme.

### In Defense of One-Vs-All Classification

Journal of Machine Learning Research 5 (2004) 101-141 Submitted 4/03; Revised 8/03; Published 1/04 In Defense of One-Vs-All Classification Ryan Rifkin Honda Research Institute USA 145 Tremont Street Boston,

### Survey on Multiclass Classification Methods

Survey on Multiclass Classification Methods Mohamed Aly November 2005 Abstract Supervised classification algorithms aim at producing a learning model from a labeled training set. Various

### Reducing multiclass to binary by coupling probability estimates

Reducing multiclass to inary y coupling proaility estimates Bianca Zadrozny Department of Computer Science and Engineering University of California, San Diego La Jolla, CA 92093-0114 zadrozny@cs.ucsd.edu

### A fast multi-class SVM learning method for huge databases

www.ijcsi.org 544 A fast multi-class SVM learning method for huge databases Djeffal Abdelhamid 1, Babahenini Mohamed Chaouki 2 and Taleb-Ahmed Abdelmalik 3 1,2 Computer science department, LESIA Laboratory,

### SUPPORT VECTOR MACHINE (SVM) is the optimal

130 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 19, NO. 1, JANUARY 2008 Multiclass Posterior Probability Support Vector Machines Mehmet Gönen, Ayşe Gönül Tanuğur, and Ethem Alpaydın, Senior Member, IEEE

### Sub-class Error-Correcting Output Codes

Sub-class Error-Correcting Output Codes Sergio Escalera, Oriol Pujol and Petia Radeva Computer Vision Center, Campus UAB, Edifici O, 08193, Bellaterra, Spain. Dept. Matemàtica Aplicada i Anàlisi, Universitat

### Support Vector Machines with Clustering for Training with Very Large Datasets

Support Vector Machines with Clustering for Training with Very Large Datasets Theodoros Evgeniou Technology Management INSEAD Bd de Constance, Fontainebleau 77300, France theodoros.evgeniou@insead.fr Massimiliano

### Introduction to Support Vector Machines. Colin Campbell, Bristol University

Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### A Simple Introduction to Support Vector Machines

A Simple Introduction to Support Vector Machines Martin Law Lecture for CSE 802 Department of Computer Science and Engineering Michigan State University Outline A brief history of SVM Large-margin linear

### Chapter 6. The stacking ensemble approach

82 This chapter proposes the stacking ensemble approach for combining different data mining classifiers to get better performance. Other combination techniques like voting, bagging etc are also described

### Support Vector Machines

Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric

### Predict Influencers in the Social Network

Predict Influencers in the Social Network Ruishan Liu, Yang Zhao and Liuyu Zhou Email: rliu2, yzhao2, lyzhou@stanford.edu Department of Electrical Engineering, Stanford University Abstract Given two persons

### Which Is the Best Multiclass SVM Method? An Empirical Study

Which Is the Best Multiclass SVM Method? An Empirical Study Kai-Bo Duan 1 and S. Sathiya Keerthi 2 1 BioInformatics Research Centre, Nanyang Technological University, Nanyang Avenue, Singapore 639798 askbduan@ntu.edu.sg

### Comparing the Results of Support Vector Machines with Traditional Data Mining Algorithms

Comparing the Results of Support Vector Machines with Traditional Data Mining Algorithms Scott Pion and Lutz Hamel Abstract This paper presents the results of a series of analyses performed on direct mail

### Impact of Boolean factorization as preprocessing methods for classification of Boolean data

Impact of Boolean factorization as preprocessing methods for classification of Boolean data Radim Belohlavek, Jan Outrata, Martin Trnecka Data Analysis and Modeling Lab (DAMOL) Dept. Computer Science,

### Artificial Neural Networks and Support Vector Machines. CS 486/686: Introduction to Artificial Intelligence

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

### Large Margin DAGs for Multiclass Classification

S.A. Solla, T.K. Leen and K.-R. Müller (eds.), 57 55, MIT Press (000) Large Margin DAGs for Multiclass Classification John C. Platt Microsoft Research Microsoft Way Redmond, WA 9805 jplatt@microsoft.com

### CSCI567 Machine Learning (Fall 2014)

CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 22, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 22, 2014 1 /

### Adapting Codes and Embeddings for Polychotomies

Adapting Codes and Embeddings for Polychotomies Gunnar Rätsch, Alexander J. Smola RSISE, CSL, Machine Learning Group The Australian National University Canberra, 2 ACT, Australia Gunnar.Raetsch, Alex.Smola

### SVM Ensemble Model for Investment Prediction

19 SVM Ensemble Model for Investment Prediction Chandra J, Assistant Professor, Department of Computer Science, Christ University, Bangalore Siji T. Mathew, Research Scholar, Christ University, Dept of

### Employer Health Insurance Premium Prediction Elliott Lui

Employer Health Insurance Premium Prediction Elliott Lui 1 Introduction The US spends 15.2% of its GDP on health care, more than any other country, and the cost of health insurance is rising faster than

### Class #6: Non-linear classification. ML4Bio 2012 February 17 th, 2012 Quaid Morris

Class #6: Non-linear classification ML4Bio 2012 February 17 th, 2012 Quaid Morris 1 Module #: Title of Module 2 Review Overview Linear separability Non-linear classification Linear Support Vector Machines

### Multi-class protein fold recognition using adaptive codes

Eugene Ie EIE@CS.COLUMBIA.EDU Center for Computational Learning Systems, Columbia University, New York, NY 10115 USA Jason Weston JASONW@NEC-LABS.COM NEC Research Institute, 4 Independence Way, Princeton,

### One-sided Support Vector Regression for Multiclass Cost-sensitive Classification

One-sided Support Vector Regression for Multiclass Cost-sensitive Classification Han-Hsing Tu r96139@csie.ntu.edu.tw Hsuan-Tien Lin htlin@csie.ntu.edu.tw Department of Computer Science and Information

### Pattern Recognition 44 (2011) Contents lists available at ScienceDirect. Pattern Recognition. journal homepage:

Pattern Recognition 44 (2011) 1761 1776 Contents lists available at ScienceDirect Pattern Recognition journal homepage: www.elsevier.com/locate/pr An overview of ensemble methods for binary classifiers

### Multi-Class and Structured Classification

Multi-Class and Structured Classification [slides prises du cours cs294-10 UC Berkeley (2006 / 2009)] [ p y( )] http://www.cs.berkeley.edu/~jordan/courses/294-fall09 Basic Classification in ML Input Output

### Multi-class Classification: A Coding Based Space Partitioning

Multi-class Classification: A Coding Based Space Partitioning Sohrab Ferdowsi, Svyatoslav Voloshynovskiy, Marcin Gabryel, and Marcin Korytkowski University of Geneva, Centre Universitaire d Informatique,

### Simple and efficient online algorithms for real world applications

Simple and efficient online algorithms for real world applications Università degli Studi di Milano Milano, Italy Talk @ Centro de Visión por Computador Something about me PhD in Robotics at LIRA-Lab,

### Introducing diversity among the models of multi-label classification ensemble

Introducing diversity among the models of multi-label classification ensemble Lena Chekina, Lior Rokach and Bracha Shapira Ben-Gurion University of the Negev Dept. of Information Systems Engineering and

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Multi-class protein classification using adaptive codes

Journal of Machine Learning Research () Submitted ; Published Multi-class protein classification using adaptive codes Iain Melvin NEC Laboratories of America Princeton, NJ 08540, USA; and Center for Computational

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### GURLS: A Least Squares Library for Supervised Learning

Journal of Machine Learning Research 14 (2013) 3201-3205 Submitted 1/12; Revised 2/13; Published 10/13 GURLS: A Least Squares Library for Supervised Learning Andrea Tacchetti Pavan K. Mallapragada Center

### The Artificial Prediction Market

The Artificial Prediction Market Adrian Barbu Department of Statistics Florida State University Joint work with Nathan Lay, Siemens Corporate Research 1 Overview Main Contributions A mathematical theory

### Feature Selection with Decision Tree Criterion

Feature Selection with Decision Tree Criterion Krzysztof Grąbczewski and Norbert Jankowski Department of Computer Methods Nicolaus Copernicus University Toruń, Poland kgrabcze,norbert@phys.uni.torun.pl

### Support Vector Machine (SVM)

Support Vector Machine (SVM) CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin

### Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:

### A Survey of Kernel Clustering Methods

A Survey of Kernel Clustering Methods Maurizio Filippone, Francesco Camastra, Francesco Masulli and Stefano Rovetta Presented by: Kedar Grama Outline Unsupervised Learning and Clustering Types of clustering

### Basics of Statistical Machine Learning

CS761 Spring 2013 Advanced Machine Learning Basics of Statistical Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu Modern machine learning is rooted in statistics. You will find many familiar

### Programming Exercise 3: Multi-class Classification and Neural Networks

Programming Exercise 3: Multi-class Classification and Neural Networks Machine Learning November 4, 2011 Introduction In this exercise, you will implement one-vs-all logistic regression and neural networks

### Introduction to machine learning and pattern recognition Lecture 1 Coryn Bailer-Jones

Introduction to machine learning and pattern recognition Lecture 1 Coryn Bailer-Jones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 What is machine learning? Data description and interpretation

### Example: Credit card default, we may be more interested in predicting the probabilty of a default than classifying individuals as default or not.

Statistical Learning: Chapter 4 Classification 4.1 Introduction Supervised learning with a categorical (Qualitative) response Notation: - Feature vector X, - qualitative response Y, taking values in C

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### Machine Learning Final Project Spam Email Filtering

Machine Learning Final Project Spam Email Filtering March 2013 Shahar Yifrah Guy Lev Table of Content 1. OVERVIEW... 3 2. DATASET... 3 2.1 SOURCE... 3 2.2 CREATION OF TRAINING AND TEST SETS... 4 2.3 FEATURE

### large-scale machine learning revisited Léon Bottou Microsoft Research (NYC)

large-scale machine learning revisited Léon Bottou Microsoft Research (NYC) 1 three frequent ideas in machine learning. independent and identically distributed data This experimental paradigm has driven

### Online Classification on a Budget

Online Classification on a Budget Koby Crammer Computer Sci. & Eng. Hebrew University Jerusalem 91904, Israel kobics@cs.huji.ac.il Jaz Kandola Royal Holloway, University of London Egham, UK jaz@cs.rhul.ac.uk

### CSE 494 CSE/CBS 598 (Fall 2007): Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye

CSE 494 CSE/CBS 598 Fall 2007: Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye 1 Introduction One important method for data compression and classification is to organize

### Privacy-Preserving Outsourcing Support Vector Machines with Random Transformation

Privacy-Preserving Outsourcing Support Vector Machines with Random Transformation Keng-Pei Lin Ming-Syan Chen Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan Research Center

### Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05

Ensemble Methods Knowledge Discovery and Data Mining 2 (VU) (707004) Roman Kern KTI, TU Graz 2015-03-05 Roman Kern (KTI, TU Graz) Ensemble Methods 2015-03-05 1 / 38 Outline 1 Introduction 2 Classification

### CI6227: Data Mining. Lesson 11b: Ensemble Learning. Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore.

CI6227: Data Mining Lesson 11b: Ensemble Learning Sinno Jialin PAN Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore Acknowledgements: slides are adapted from the lecture notes

### Supervised Learning with Unsupervised Output Separation

Supervised Learning with Unsupervised Output Separation Nathalie Japkowicz School of Information Technology and Engineering University of Ottawa 150 Louis Pasteur, P.O. Box 450 Stn. A Ottawa, Ontario,

### C4 Computer Vision. 4 Lectures Michaelmas Term Tutorial Sheet Prof A. Zisserman. fundamental matrix, recovering ego-motion, applications.

C4 Computer Vision 4 Lectures Michaelmas Term 2004 1 Tutorial Sheet Prof A. Zisserman Overview Lecture 1: Stereo Reconstruction I: epipolar geometry, fundamental matrix. Lecture 2: Stereo Reconstruction

### 1 Solving LPs: The Simplex Algorithm of George Dantzig

Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.

### Classification algorithm in Data mining: An Overview

Classification algorithm in Data mining: An Overview S.Neelamegam #1, Dr.E.Ramaraj *2 #1 M.phil Scholar, Department of Computer Science and Engineering, Alagappa University, Karaikudi. *2 Professor, Department

### A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions

A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions Jing Gao Wei Fan Jiawei Han Philip S. Yu University of Illinois at Urbana-Champaign IBM T. J. Watson Research Center

### Discuss the size of the instance for the minimum spanning tree problem.

3.1 Algorithm complexity The algorithms A, B are given. The former has complexity O(n 2 ), the latter O(2 n ), where n is the size of the instance. Let n A 0 be the size of the largest instance that can

### Online learning of multi-class Support Vector Machines

IT 12 061 Examensarbete 30 hp November 2012 Online learning of multi-class Support Vector Machines Xuan Tuan Trinh Institutionen för informationsteknologi Department of Information Technology Abstract

### Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

### Search Taxonomy. Web Search. Search Engine Optimization. Information Retrieval

Information Retrieval INFO 4300 / CS 4300! Retrieval models Older models» Boolean retrieval» Vector Space model Probabilistic Models» BM25» Language models Web search» Learning to Rank Search Taxonomy!

### CSE 473: Artificial Intelligence Autumn 2010

CSE 473: Artificial Intelligence Autumn 2010 Machine Learning: Naive Bayes and Perceptron Luke Zettlemoyer Many slides over the course adapted from Dan Klein. 1 Outline Learning: Naive Bayes and Perceptron

### An Introduction to Machine Learning

An Introduction to Machine Learning L5: Novelty Detection and Regression Alexander J. Smola Statistical Machine Learning Program Canberra, ACT 0200 Australia Alex.Smola@nicta.com.au Tata Institute, Pune,

### Linear Codes. Chapter 3. 3.1 Basics

Chapter 3 Linear Codes In order to define codes that we can encode and decode efficiently, we add more structure to the codespace. We shall be mainly interested in linear codes. A linear code of length

### Large-Scale Similarity and Distance Metric Learning

Large-Scale Similarity and Distance Metric Learning Aurélien Bellet Télécom ParisTech Joint work with K. Liu, Y. Shi and F. Sha (USC), S. Clémençon and I. Colin (Télécom ParisTech) Séminaire Criteo March

### Introduction to Machine Learning NPFL 054

Introduction to Machine Learning NPFL 054 http://ufal.mff.cuni.cz/course/npfl054 Barbora Hladká hladka@ufal.mff.cuni.cz Martin Holub holub@ufal.mff.cuni.cz Charles University, Faculty of Mathematics and

### Recognizing Informed Option Trading

Recognizing Informed Option Trading Alex Bain, Prabal Tiwaree, Kari Okamoto 1 Abstract While equity (stock) markets are generally efficient in discounting public information into stock prices, we believe

### Transforming Classifier Scores into Accurate Multiclass Probability Estimates

Transforming Classifier Scores into Accurate Multiclass Probability Estimates Bianca Zadrozny & Charles Elkan Presenter: Myle Ott 1 Motivation (the same old story) Easy to rank examples in order of classmembership

### MAXIMIZING RETURN ON DIRECT MARKETING CAMPAIGNS

MAXIMIZING RETURN ON DIRET MARKETING AMPAIGNS IN OMMERIAL BANKING S 229 Project: Final Report Oleksandra Onosova INTRODUTION Recent innovations in cloud computing and unified communications have made a

### Multiple Kernel Learning on the Limit Order Book

JMLR: Workshop and Conference Proceedings 11 (2010) 167 174 Workshop on Applications of Pattern Analysis Multiple Kernel Learning on the Limit Order Book Tristan Fletcher Zakria Hussain John Shawe-Taylor

### Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics

Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics Part I: Factorizations and Statistical Modeling/Inference Amnon Shashua School of Computer Science & Eng. The Hebrew University

### Support Vector Machine. Tutorial. (and Statistical Learning Theory)

Support Vector Machine (and Statistical Learning Theory) Tutorial Jason Weston NEC Labs America 4 Independence Way, Princeton, USA. jasonw@nec-labs.com 1 Support Vector Machines: history SVMs introduced

### FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT MINING SYSTEM

International Journal of Innovative Computing, Information and Control ICIC International c 0 ISSN 34-48 Volume 8, Number 8, August 0 pp. 4 FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### LINEAR SYSTEMS. Consider the following example of a linear system:

LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x +2x 2 +3x 3 = 5 x + x 3 = 3 3x + x 2 +3x 3 = 3 x =, x 2 =0, x 3 = 2 In general we want to solve n equations in

### Distributed Machine Learning and Big Data

Distributed Machine Learning and Big Data Sourangshu Bhattacharya Dept. of Computer Science and Engineering, IIT Kharagpur. http://cse.iitkgp.ac.in/~sourangshu/ August 21, 2015 Sourangshu Bhattacharya

### DUOL: A Double Updating Approach for Online Learning

: A Double Updating Approach for Online Learning Peilin Zhao School of Comp. Eng. Nanyang Tech. University Singapore 69798 zhao6@ntu.edu.sg Steven C.H. Hoi School of Comp. Eng. Nanyang Tech. University

### Active Learning SVM for Blogs recommendation

Active Learning SVM for Blogs recommendation Xin Guan Computer Science, George Mason University Ⅰ.Introduction In the DH Now website, they try to review a big amount of blogs and articles and find the

### Recall this chart that showed how most of our course would be organized:

Chapter 4 One-Way ANOVA Recall this chart that showed how most of our course would be organized: Explanatory Variable(s) Response Variable Methods Categorical Categorical Contingency Tables Categorical

### Maximum Margin Clustering

Maximum Margin Clustering Linli Xu James Neufeld Bryce Larson Dale Schuurmans University of Waterloo University of Alberta Abstract We propose a new method for clustering based on finding maximum margin

### Final Exam, Spring 2007

10-701 Final Exam, Spring 2007 1. Personal info: Name: Andrew account: E-mail address: 2. There should be 16 numbered pages in this exam (including this cover sheet). 3. You can use any material you brought:

### Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data

CMPE 59H Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data Term Project Report Fatma Güney, Kübra Kalkan 1/15/2013 Keywords: Non-linear

### A crash course in probability and Naïve Bayes classification

Probability theory A crash course in probability and Naïve Bayes classification Chapter 9 Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s

### E-commerce Transaction Anomaly Classification

E-commerce Transaction Anomaly Classification Minyong Lee minyong@stanford.edu Seunghee Ham sham12@stanford.edu Qiyi Jiang qjiang@stanford.edu I. INTRODUCTION Due to the increasing popularity of e-commerce

### Solving polynomial least squares problems via semidefinite programming relaxations

Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose

### Relations Graphical View

Relations Slides by Christopher M. Bourke Instructor: Berthe Y. Choueiry Introduction Recall that a relation between elements of two sets is a subset of their Cartesian product (of ordered pairs). A binary

### Social Media Mining. Data Mining Essentials

Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers

### Car Insurance. Havránek, Pokorný, Tomášek

Car Insurance Havránek, Pokorný, Tomášek Outline Data overview Horizontal approach + Decision tree/forests Vertical (column) approach + Neural networks SVM Data overview Customers Viewed policies Bought

### Clustering. Danilo Croce Web Mining & Retrieval a.a. 2015/201 16/03/2016

Clustering Danilo Croce Web Mining & Retrieval a.a. 2015/201 16/03/2016 1 Supervised learning vs. unsupervised learning Supervised learning: discover patterns in the data that relate data attributes with

### Feature Selection using Integer and Binary coded Genetic Algorithm to improve the performance of SVM Classifier

Feature Selection using Integer and Binary coded Genetic Algorithm to improve the performance of SVM Classifier D.Nithya a, *, V.Suganya b,1, R.Saranya Irudaya Mary c,1 Abstract - This paper presents,

### Categorical Data Visualization and Clustering Using Subjective Factors

Categorical Data Visualization and Clustering Using Subjective Factors Chia-Hui Chang and Zhi-Kai Ding Department of Computer Science and Information Engineering, National Central University, Chung-Li,

### C19 Machine Learning

C9 Machine Learning 8 Lectures Hilary Term 25 2 Tutorial Sheets A. Zisserman Overview: Supervised classification perceptron, support vector machine, loss functions, kernels, random forests, neural networks

### Bayes and Naïve Bayes. cs534-machine Learning

Bayes and aïve Bayes cs534-machine Learning Bayes Classifier Generative model learns Prediction is made by and where This is often referred to as the Bayes Classifier, because of the use of the Bayes rule

### Data Mining Practical Machine Learning Tools and Techniques

Ensemble learning Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 8 of Data Mining by I. H. Witten, E. Frank and M. A. Hall Combining multiple models Bagging The basic idea

### Mathematics Course 111: Algebra I Part IV: Vector Spaces

Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 1996-7 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are

### D-optimal plans in observational studies

D-optimal plans in observational studies Constanze Pumplün Stefan Rüping Katharina Morik Claus Weihs October 11, 2005 Abstract This paper investigates the use of Design of Experiments in observational

### MACHINE LEARNING. Introduction. Alessandro Moschitti

MACHINE LEARNING Introduction Alessandro Moschitti Department of Computer Science and Information Engineering University of Trento Email: moschitti@disi.unitn.it Course Schedule Lectures Tuesday, 14:00-16:00

### A semi-supervised Spam mail detector

A semi-supervised Spam mail detector Bernhard Pfahringer Department of Computer Science, University of Waikato, Hamilton, New Zealand Abstract. This document describes a novel semi-supervised approach