# Several Views of Support Vector Machines

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007

2 Tikhonov Regularization We are considering algorithms of the form min f H n v i (Y i ) + λ 2 f 2 K. (1) i=1 Different loss functions lead to different learning problems. Last class, we discussed regularized least squares, by choosing v i (y i ) = 1 2 (Y i y i ) 2. Support vector machines are another Tikhonov regularization algorithm...

3 SVM Motivation: Problems with RLS RLS uses the square loss, which some might say does not make sense for classification. SVM uses the hinge loss (defined soon), which does makes sense. Nonlinear RLS does not scale easily to large data sets. The SVM can have better scaling properties. The SVM has a (in my opinion weak) geometric motivation: the idea of margin.

4 A loss function for classification The most natural loss for classification is probably the 0-1 loss: We pay zero if our prediction has the correct sign, and one otherwise (remember that functions in an RKHS make real-valued predictions). Unfortunately, the 0-1 loss is not convex. Therefore, we have little hope of being able to optimize this loss function in practice. (Note that the representer theorem does hold for the 0-1 loss.) A solution: the hinge loss, a convex loss that upper bounds the zero-one loss: v(y) = max(1 yy, 0) = (1 yy ) +.

5 The hinge loss Hinge Loss y * f(x)

6 Value-based SVM Substituting the loss function into the definition of Tikhonov regularization, we get an optimization problem min y R n i (1 y i Y i ) + + λy t K 1 y. This is (basically) an SVM. So what? How will you solve this problem (find the minimizing y)? The hinge loss is not differentiable, so you cannot take the derivative and set it to zero.

7 Coefficient-based SVM Remember that the representer theorem says the answer has the form f ( ) = c i k(x i, ). i Using the transformation y = Kc (or c = K 1 y), we can rewrite the SVM as Again: so what? min c R n i (1 (Kc) i ) + + λc t Kc.

8 The SVM: So What? The SVM has many interesting and desirable properties. These properties are not immediately apparent from the optimization problems we have just written. Optimization theory and geometry lead us to algorithms for solving the problem and insights in the nature of the solution. We will see that SVMs have a nice sparsity property: many (frequently most) of the c i s turn out to be zero.

9 Nondifferentiable Functions and Constraints We can rewrite a piecewise differentiable convex linear function as a sum of differentiable functions over constrained variables. Case in point: instead of minimizing (1 yy ) +, I can minimize ξ subject to the constraints that ξ 1 yy and ξ 0. Two different ways of looking at the same thing.

10 The Hinge Loss, Constrained Form If I want to take a Lagrangian, I need to rewrite the loss function in terms of constraints. These constraints are also called slack variables. This rewriting is orthogonal to the issue of whether I think about y or c. In terms of y, we rewrite as min y R n i min y R n,ξ R n (1 y i Y i ) + + λy t K 1 y. i ξ i + λy t K 1 y subject to : ξ (1 yy ) ξ 0

11 In terms of c In terms of the c, the constrained version of the problem is min c R n,ξ R n subject to : i ξ i + λc t Kc ξ (1 YKc) ξ 0 Note how we get rid of the (1 Kc) + by requiring that the ξ are nonnegative.

12 Solving an SVM, I Written in terms of c or y (and ξ), we have a problem where we re trying to minimize a convex quadratic function subject to linear constraints. In optimization theory, this is called a convex quadratic program. Algorithm I: Find or buy software that solves convex quadratic programs. This will work. However, this software generally needs to work with the matrix K. It will be slower than solving an RLS problem of the same size. As we will see, the SVM has special structure which leads to good algorithms.

13 The geometric approach The traditional approach to explaining the SVM is via separating hyperplanes and margin. Imagine the positive and negative examples are separable by a linear function (i.e.a hyperplane). Define the margin as the distance from the hyperplane to the nearest example. Intuitively, larger margin will generalize better.

14 Large and Small Margin Hyperplanes (a) (b)

15 Classification With Hyperplanes Denote the hyperplane by w. f (x) = w t x. A separating hyperplane satisfies y i (w t x i ) > 0 for the entire training set. We are considering homogeneous hyperplanes (i.e., hyperplanes that pass through the origin.) Geometrically, when we draw the hyperplane, we are drawing the set {x : w t x = 0}, and the vector w is normal to this set.

16 Maximizing Margin, I Given a separating hyperplane w, let x c be a training point closest to w, and define x w to be the unique point in {x : w t x = 0} that is closest to x. (Both x c and x w depend on w.) Finding a maximum margin hyperplane is equivalent to finding a w that maximizes x c x w. For some k (assume k > 0 for convenience), w t x c = k w t x w = 0 = w t (x c x w ) = k

17 Maximizing Margin, II Noting that the vector x c x w is parallel to the normal vector w, k = w t (x c x w ) = w x c x w = x c x w = k w

18 Maximizing Margin, III k is a nuisance parameter; WLOG, we fix it to 1. (Scaling a hyperparameter by a positive constant changes k and w, but not x c or x w.) With k fixed, maximizing x x w is equivalent to maximizing 1 w, or minimizing w, or minimizing w 2. The margin is now the distance between {x : w t x = 0} and {x : w t x = 1.} Fixing k is fixing the scale of the function.

19 The linear homogeneous separable SVM Phrased as an optimization problem, we have 2 min w R n w subject to : y i w t x i 1 0 i = 1,..., n Note that w 2 is the RKHS norm of a linear function. We are minimizing the RKHS norm, subject to a hard loss.

20 From hard loss to hinge loss. We can introduce slacks ξ i : min w R n,ξ R n w 2 + i ξ i subject to : ξ i 1 y i w t x i i = 1,..., n ξ i 0 i = 1,..., n What happened to our beautiful geometric argument? What is the margin if we don t separate the data? Because we are nearly always interested classification problems that are not separable, I think it makes more sense to start with the RKHS and the hinge loss, rather than the concept of margin.

21 Fenchel Duality, Main Theorem (Reminder) Theorem Given convex functions f and g, under minor technical conditions, inf {f (y) + g(y) + f (z) + g ( z)} = 0, y,z at least one minimizer exists, and all minimizers y, z satisfy the complementarity equations: f (y) y t z + f (z) = 0 g(y) + y t z + g ( z) = 0.

22 Regularization Optimality Condition We are looking for y and z satisfying For Tikhonov regularization, R(y) y t z + R (z) = 0. R(y) = λy t K 1 y. R (z) = λ 1 z t Kz. The optimality condition for the regularizer is: 1 2 λy t K 1 y y t z λ 1 z t Kz = (y λ 1 Kz) t (λk 1 y z) = 0 y = λ 1 Kz z = λk 1 y.

23 Regularization Optimality Condition For Tikhonov regularization, the optimal y and z satisfy y = λ 1 Kz, independent of the loss function. Modified regularizers will lead to modified optimality conditions, again independent of the loss. Key future example: unregularized bias terms. The z s are closely related to the expansion coefficients via c = λ 1 z.

24 Loss Optimality Conditions For a pointwise loss function V (y) = i v i (y i ), the conjugate of the sum is the sum of the conjugates: { } V (z) = sup y y t z i v i (y i ) = i = i sup y i {y i z i v i (y i )} v i (z i ). Therefore, for each data point, we get a constraint v i (y i ) + y i z i + v i ( z i ). The exact form of the constraint is dictated by the loss.

25 The Hinge Loss Conjugate We need to derive v ( z) for the hinge loss v(y) = (1 yy ) +. We could use the graphical method (maybe on board). Note that Y { 1, 1}, so yy = y/y. Alternate approach, a composition of functions...

26 The max(y, 0) nonlinearity. Suppose f (y) = max(y, 0) = (y) + f (z) = sup y {yz (y) + } Clearly, if z < 0 or z > 1, f (z) = Clearly, if z [0, 1], f (z) = 0 Conclusion: f (z) = δ [0,1] (z)

27 The 1 yy term. g(y) = f (1 yy ) g (z) = sup y {yz f (1 yy )} Substitute ŷ = 1 yy y = Y ŷy g (z) = supŷ{(y ŷy )z f (ŷ)} = Yz + f ( Yz)

28 Putting it together f (y) = (y) + f (z) = δ [0,1] (z) g(y) = f (1 yy ) g (z) = Yz + f ( Yz) v(y) = (1 yy ) + v (z) = Yz + f (Yz) = Yz + δ [0,1] ( Yz) v ( z) = δ [0,1] ( z Y ) z Y

29 The hinge loss optimality condition v(y) + yz + v ( z) = 0 ( z ) (1 yy ) + + yz + δ [0,1] z Y Y = 0 ( ) 1 ( z ) (1 yy ) + z Y y + δ [0,1] = 0 Y (1 yy ) + z ( z ) Y (1 yy ) + δ [0,1] = 0 Y

30 The complete SVM optimality conditions Training an SVM means (conceptually) finding y, z satisfying y = λ 1 Kz (1 yy ) + = z (1 yy ) Y z Y [0, 1]n.

31 Analyzing the Loss Optimality Condition Remember, the loss function optimality condition is: (1 y i Y i ) + z ( ) i zi (1 y i Y i ) + δ Y [0,1] = 0 i Suppose that at optimality, (1 y i Y i ) < 0. We pay no loss at the ith point. Clearly, z i Y i (1 y i Y i ) must be zero as well. But that means that z i = 0, and also that c i = 0 in the functional expansion. Similarly, if (1 y i Y i ) > 0, then z i Y i = 1. If 1 y i Y i = 0, we cannot say anything about z i. Y i

32 What are support vectors? We see that points that are well-classified (1 y i y I < 0) have z i = c i = 0. These points to do not contribute to the functional expansion. The other points do contribute. They are called support vectors. If we are lucky, the number of support vectors will be small relative to the size of the training set. It is precisely this fact that makes the SVM architecture especially useful. Other key point: non-support vectors can be added, removed, or moved without changing the solution (assuming they always satisfy (1 y i Y i < 0)).

33 Support Vectors: Graphical Interpretation

34 The primal and dual problems min R(y) + y i v i (y i ) min y λ 2 y t K 1 y + i (1 y i Y i ) + min z λ 1 2 zt Kz + i min R (z) + z i v i ( z i ) ( z ) i z i + δ Y [0,1] i Y i

35 A simple SVM algorithm We will develop a poor-man s but conceptually reasonable algorithm for solving λ 1 min z 2 zt Kz + ( z ) i z i + δ Y [0,1] i Y i i We work with the z s rather than the y s because we don t want to deal with K 1. Consider optimizing one of the z i, and holding the others fixed. We are now trying to minimize λ K iiz 2 i + j i (K ij z j )z i 1, Y i subject to the constraint z i Y i [0, 1]. This problem is easy to solve directly. Algorithm: Keep doing this until we re done.

36 A simple SVM algorithm, analyzed We start with the all-zero solution z = 0. Note that solving a subproblem for point i involves the kernel products between i and those j such that z j 0. If we have two points j and k such that neither z j nor z k ever become nonzero during the course of the algorithm, we never need to compute K jk. Real SVM algorithms are basically (almost) this idea, combined with schemes for caching kernel products.

37 An unregularized bias The representer theorem says the answer has the form f ( ) = i c i k(x i, ). Suppose we decide to look for a function of the form f ( ) = i c i k(x i, ) + b, and we do not regularize b. The modified problem is min c R n,ξ R n,b R Why would we do such a thing? i ξ i + λc t Kc ξ (1 Kc + b) ξ 0.

38 An unregularized bias, thoughts Why should my hyperplane have to go through the origin? I don t know that a priori. An unregularized bias says constant functions are not penalized. We are saying Find me a function in the RKHS, plus some constant function. Alternate strategy: add a dimension of all 1 s to the data, in feature space (e.g., k(x i, x j ) k(x i, x j ) + 1) The alternate strategy allows arbitrary hyperplanes, but penalizes the bias term.

39 Unregularized bias, pros and cons Pro: Some people think it feels better. Cons: The math gets more complicated. Suggestion: if you have a regularized bias, do it implicitly. Don t bother writing b 2 everywhere, that s a waste of ink. Suggestion: have a regularized bias. If you insist on an unregularized bias, Fenchel duality is a good way to talk about it...

40 Fenchel Bias, I Instead of y = Kc, we have y = Kc + b. Suppose we have regularizer R (with conjugate R (y)). Adding an unregularized bias is really saying I can shift all my values by some constant, and I consider that just as smooth. The new regularizer is R (y) = inf b R(y 1 nb)

41 The conjugate of a biased regularizer R (y) = inf R(y 1 nb) b R (z) = sup{y t z inf R(y 1 nb)} y b = sup{y t z R(y 1 n b)} y,b = sup{(ŷ + 1 n b) t z R(ŷ)} ŷ,b = sup{(1 t nz)b + sup{ŷ t z R(ŷ)}} b = δ {0} (1 t nz) + R (z). ŷ

42 The conjugate of a biased regularizer, thoughts R (y) = inf nb) b R (z) = δ {0} (1 t nz) + R (z). In the primal, we say allow a constant shift of the values. In the dual, we say i z i = 0. That s it!!!!!

43 The conjugate of a biased regularizer, more thoughts We don t need to rederive the whole dual from the beginning. This result is general across regularizers and loss functions. This is an example of infimal convolution, see the Fenchel paper for details. For algorithms, the constraint i z i = 0 means they modify two z s at a time rather than one.

44 Good Large-Scale SVM Solvers SVMLight: SVMTorch: LIBSVM: cjlin/libsvm

45 Musings on SVMs and RLS If we can solve one RLS problem, we can find a good λ (that minimizes LOO error.) There exists work on finding the regularization path of the SVM (Hastie et al. 04). The claim is they can find a good λ in the same time as it takes to solve one problem. The experiments do not convince me (and they do not do LOO error.) For large nonlinear problems, I cannot solve one RLS problem at all. The SVM is sparse. It is only a constant factor sparse, so it won t scale forever, but solving O(100,000) point nonlinear SVM problems is (somewhat) common.

46 The elephant in the room. There are many good methods to help us choose λ. However, choosing k is usually the hard part. Note that λ is about choosing how much smoothness to insist on in an RKHS, but choosing k is about deciding which RKHS to use. If we only have a small number of parameters, we can grid search. But what about kernels like ( k(x i, x j ) = exp ) γ d (x id x jd ) 2, d a generalization of the Gaussian where we have a lengthscale for each dimension? There are some recent attempts to deal with this, but nothing is too satisfactory in my opinion.

### Support Vector Machines

Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### Support Vector Machine (SVM)

Support Vector Machine (SVM) CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin

### Introduction to Support Vector Machines. Colin Campbell, Bristol University

Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.

### Linear Programming, Lagrange Multipliers, and Duality Geoff Gordon

lp.nb 1 Linear Programming, Lagrange Multipliers, and Duality Geoff Gordon lp.nb 2 Overview This is a tutorial about some interesting math and geometry connected with constrained optimization. It is not

### Support Vector Machines Explained

March 1, 2009 Support Vector Machines Explained Tristan Fletcher www.cs.ucl.ac.uk/staff/t.fletcher/ Introduction This document has been written in an attempt to make the Support Vector Machines (SVM),

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### An Introduction to Machine Learning

An Introduction to Machine Learning L5: Novelty Detection and Regression Alexander J. Smola Statistical Machine Learning Program Canberra, ACT 0200 Australia Alex.Smola@nicta.com.au Tata Institute, Pune,

### Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Support Vector Machines

Support Vector Machines Here we approach the two-class classification problem in a direct way: We try and find a plane that separates the classes in feature space. If we cannot, we get creative in two

### Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

### A Simple Introduction to Support Vector Machines

A Simple Introduction to Support Vector Machines Martin Law Lecture for CSE 802 Department of Computer Science and Engineering Michigan State University Outline A brief history of SVM Large-margin linear

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### 1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, In which we introduce the theory of duality in linear programming.

Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, 2011 Lecture 6 In which we introduce the theory of duality in linear programming 1 The Dual of Linear Program Suppose that we

### Class #6: Non-linear classification. ML4Bio 2012 February 17 th, 2012 Quaid Morris

Class #6: Non-linear classification ML4Bio 2012 February 17 th, 2012 Quaid Morris 1 Module #: Title of Module 2 Review Overview Linear separability Non-linear classification Linear Support Vector Machines

### Introduction to Machine Learning NPFL 054

Introduction to Machine Learning NPFL 054 http://ufal.mff.cuni.cz/course/npfl054 Barbora Hladká hladka@ufal.mff.cuni.cz Martin Holub holub@ufal.mff.cuni.cz Charles University, Faculty of Mathematics and

### From Maxent to Machine Learning and Back

From Maxent to Machine Learning and Back T. Sears ANU March 2007 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 1 / 36 50 Years Ago... The principles and mathematical methods of statistical

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### Lecture 2: The SVM classifier

Lecture 2: The SVM classifier C19 Machine Learning Hilary 2015 A. Zisserman Review of linear classifiers Linear separability Perceptron Support Vector Machine (SVM) classifier Wide margin Cost function

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### max cx s.t. Ax c where the matrix A, cost vector c and right hand side b are given and x is a vector of variables. For this example we have x

Linear Programming Linear programming refers to problems stated as maximization or minimization of a linear function subject to constraints that are linear equalities and inequalities. Although the study

### Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

### Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

### A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

### Chapter 15 Introduction to Linear Programming

Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of

### Linear smoother. ŷ = S y. where s ij = s ij (x) e.g. s ij = diag(l i (x)) To go the other way, you need to diagonalize S

Linear smoother ŷ = S y where s ij = s ij (x) e.g. s ij = diag(l i (x)) To go the other way, you need to diagonalize S 2 Online Learning: LMS and Perceptrons Partially adapted from slides by Ryan Gabbard

### Math 407A: Linear Optimization

Math 407A: Linear Optimization Lecture 4: LP Standard Form 1 1 Author: James Burke, University of Washington LPs in Standard Form Minimization maximization Linear equations to linear inequalities Lower

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### ME128 Computer-Aided Mechanical Design Course Notes Introduction to Design Optimization

ME128 Computer-ided Mechanical Design Course Notes Introduction to Design Optimization 2. OPTIMIZTION Design optimization is rooted as a basic problem for design engineers. It is, of course, a rare situation

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### By W.E. Diewert. July, Linear programming problems are important for a number of reasons:

APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization)

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

### To give it a definition, an implicit function of x and y is simply any relationship that takes the form:

2 Implicit function theorems and applications 21 Implicit functions The implicit function theorem is one of the most useful single tools you ll meet this year After a while, it will be second nature to

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### Zeros of Polynomial Functions

Zeros of Polynomial Functions The Rational Zero Theorem If f (x) = a n x n + a n-1 x n-1 + + a 1 x + a 0 has integer coefficients and p/q (where p/q is reduced) is a rational zero, then p is a factor of

### Least-Squares Intersection of Lines

Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a

### Lecture 2: August 29. Linear Programming (part I)

10-725: Convex Optimization Fall 2013 Lecture 2: August 29 Lecturer: Barnabás Póczos Scribes: Samrachana Adhikari, Mattia Ciollaro, Fabrizio Lecci Note: LaTeX template courtesy of UC Berkeley EECS dept.

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### Definition 8.1 Two inequalities are equivalent if they have the same solution set. Add or Subtract the same value on both sides of the inequality.

8 Inequalities Concepts: Equivalent Inequalities Linear and Nonlinear Inequalities Absolute Value Inequalities (Sections 4.6 and 1.1) 8.1 Equivalent Inequalities Definition 8.1 Two inequalities are equivalent

### Increasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.

1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.

### 1 Polyhedra and Linear Programming

CS 598CSC: Combinatorial Optimization Lecture date: January 21, 2009 Instructor: Chandra Chekuri Scribe: Sungjin Im 1 Polyhedra and Linear Programming In this lecture, we will cover some basic material

### Linear Programming. April 12, 2005

Linear Programming April 1, 005 Parts of this were adapted from Chapter 9 of i Introduction to Algorithms (Second Edition) /i by Cormen, Leiserson, Rivest and Stein. 1 What is linear programming? The first

### SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### Support Vector Machine. Tutorial. (and Statistical Learning Theory)

Support Vector Machine (and Statistical Learning Theory) Tutorial Jason Weston NEC Labs America 4 Independence Way, Princeton, USA. jasonw@nec-labs.com 1 Support Vector Machines: history SVMs introduced

### Bindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8

Spaces and bases Week 3: Wednesday, Feb 8 I have two favorite vector spaces 1 : R n and the space P d of polynomials of degree at most d. For R n, we have a canonical basis: R n = span{e 1, e 2,..., e

### Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### Support Vector Machines with Clustering for Training with Very Large Datasets

Support Vector Machines with Clustering for Training with Very Large Datasets Theodoros Evgeniou Technology Management INSEAD Bd de Constance, Fontainebleau 77300, France theodoros.evgeniou@insead.fr Massimiliano

### Walrasian Demand. u(x) where B(p, w) = {x R n + : p x w}.

Walrasian Demand Econ 2100 Fall 2015 Lecture 5, September 16 Outline 1 Walrasian Demand 2 Properties of Walrasian Demand 3 An Optimization Recipe 4 First and Second Order Conditions Definition Walrasian

### Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem

### Chap 4 The Simplex Method

The Essence of the Simplex Method Recall the Wyndor problem Max Z = 3x 1 + 5x 2 S.T. x 1 4 2x 2 12 3x 1 + 2x 2 18 x 1, x 2 0 Chap 4 The Simplex Method 8 corner point solutions. 5 out of them are CPF solutions.

### Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

### Minimize subject to. x S R

Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such

### Chapter 4. Duality. 4.1 A Graphical Example

Chapter 4 Duality Given any linear program, there is another related linear program called the dual. In this chapter, we will develop an understanding of the dual linear program. This understanding translates

### Solution based on matrix technique Rewrite. ) = 8x 2 1 4x 1x 2 + 5x x1 2x 2 2x 1 + 5x 2

8.2 Quadratic Forms Example 1 Consider the function q(x 1, x 2 ) = 8x 2 1 4x 1x 2 + 5x 2 2 Determine whether q(0, 0) is the global minimum. Solution based on matrix technique Rewrite q( x1 x 2 = x1 ) =

### Duality in Linear Programming

Duality in Linear Programming 4 In the preceding chapter on sensitivity analysis, we saw that the shadow-price interpretation of the optimal simplex multipliers is a very useful concept. First, these shadow

### Convex analysis and profit/cost/support functions

CALIFORNIA INSTITUTE OF TECHNOLOGY Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m

### constraint. Let us penalize ourselves for making the constraint too big. We end up with a

Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the

### Mathematical finance and linear programming (optimization)

Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may

### Math 4310 Handout - Quotient Vector Spaces

Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable

### Linear Programming Notes VII Sensitivity Analysis

Linear Programming Notes VII Sensitivity Analysis 1 Introduction When you use a mathematical model to describe reality you must make approximations. The world is more complicated than the kinds of optimization

### Lecture 1: Linear Programming Models. Readings: Chapter 1; Chapter 2, Sections 1&2

Lecture 1: Linear Programming Models Readings: Chapter 1; Chapter 2, Sections 1&2 1 Optimization Problems Managers, planners, scientists, etc., are repeatedly faced with complex and dynamic systems which

### 1 Gaussian Elimination

Contents 1 Gaussian Elimination 1.1 Elementary Row Operations 1.2 Some matrices whose associated system of equations are easy to solve 1.3 Gaussian Elimination 1.4 Gauss-Jordan reduction and the Reduced

### Mathematics Notes for Class 12 chapter 12. Linear Programming

1 P a g e Mathematics Notes for Class 12 chapter 12. Linear Programming Linear Programming It is an important optimization (maximization or minimization) technique used in decision making is business and

### Notes on Support Vector Machines

Notes on Support Vector Machines Fernando Mira da Silva Fernando.Silva@inesc.pt Neural Network Group I N E S C November 1998 Abstract This report describes an empirical study of Support Vector Machines

### v w is orthogonal to both v and w. the three vectors v, w and v w form a right-handed set of vectors.

3. Cross product Definition 3.1. Let v and w be two vectors in R 3. The cross product of v and w, denoted v w, is the vector defined as follows: the length of v w is the area of the parallelogram with

### 2x + y = 3. Since the second equation is precisely the same as the first equation, it is enough to find x and y satisfying the system

1. Systems of linear equations We are interested in the solutions to systems of linear equations. A linear equation is of the form 3x 5y + 2z + w = 3. The key thing is that we don t multiply the variables

### Fixed Point Theorems

Fixed Point Theorems Definition: Let X be a set and let T : X X be a function that maps X into itself. (Such a function is often called an operator, a transformation, or a transform on X, and the notation

### Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

### Lecture 5: Conic Optimization: Overview

EE 227A: Conve Optimization and Applications January 31, 2012 Lecture 5: Conic Optimization: Overview Lecturer: Laurent El Ghaoui Reading assignment: Chapter 4 of BV. Sections 3.1-3.6 of WTB. 5.1 Linear

### Artificial Neural Networks and Support Vector Machines. CS 486/686: Introduction to Artificial Intelligence

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

### 1 Solving LPs: The Simplex Algorithm of George Dantzig

Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.

### Lecture 6: Logistic Regression

Lecture 6: CS 194-10, Fall 2011 Laurent El Ghaoui EECS Department UC Berkeley September 13, 2011 Outline Outline Classification task Data : X = [x 1,..., x m]: a n m matrix of data points in R n. y { 1,

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### Support Vector Machines

CS229 Lecture notes Andrew Ng Part V Support Vector Machines This set of notes presents the Support Vector Machine (SVM) learning algorithm. SVMs are among the best (and many believe are indeed the best)

### Part II Redundant Dictionaries and Pursuit Algorithms

Aisenstadt Chair Course CRM September 2009 Part II Redundant Dictionaries and Pursuit Algorithms Stéphane Mallat Centre de Mathématiques Appliquées Ecole Polytechnique Sparsity in Redundant Dictionaries

### Big Data - Lecture 1 Optimization reminders

Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics

### Høgskolen i Narvik Sivilingeniørutdanningen STE6237 ELEMENTMETODER. Oppgaver

Høgskolen i Narvik Sivilingeniørutdanningen STE637 ELEMENTMETODER Oppgaver Klasse: 4.ID, 4.IT Ekstern Professor: Gregory A. Chechkin e-mail: chechkin@mech.math.msu.su Narvik 6 PART I Task. Consider two-point

### Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

### Arrangements And Duality

Arrangements And Duality 3.1 Introduction 3 Point configurations are tbe most basic structure we study in computational geometry. But what about configurations of more complicated shapes? For example,

### Constrained optimization.

ams/econ 11b supplementary notes ucsc Constrained optimization. c 2010, Yonatan Katznelson 1. Constraints In many of the optimization problems that arise in economics, there are restrictions on the values

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### Linear Least Squares

Linear Least Squares Suppose we are given a set of data points {(x i,f i )}, i = 1,...,n. These could be measurements from an experiment or obtained simply by evaluating a function at some points. One

### Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi

Neural Networks CAP5610 Machine Learning Instructor: Guo-Jun Qi Recap: linear classifier Logistic regression Maximizing the posterior distribution of class Y conditional on the input vector X Support vector

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Linear Algebra I. Ronald van Luijk, 2012

Linear Algebra I Ronald van Luijk, 2012 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents 1. Vector spaces 3 1.1. Examples 3 1.2. Fields 4 1.3. The field of complex numbers. 6 1.4.

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### Quiz 1 Sample Questions IE406 Introduction to Mathematical Programming Dr. Ralphs

Quiz 1 Sample Questions IE406 Introduction to Mathematical Programming Dr. Ralphs These questions are from previous years and should you give you some idea of what to expect on Quiz 1. 1. Consider the

### Tangent and normal lines to conics

4.B. Tangent and normal lines to conics Apollonius work on conics includes a study of tangent and normal lines to these curves. The purpose of this document is to relate his approaches to the modern viewpoints

### Loss Functions for Preference Levels: Regression with Discrete Ordered Labels

Loss Functions for Preference Levels: Regression with Discrete Ordered Labels Jason D. M. Rennie Massachusetts Institute of Technology Comp. Sci. and Artificial Intelligence Laboratory Cambridge, MA 9,

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of