# Lecture 2: August 29. Linear Programming (part I)

Size: px
Start display at page:

Transcription

1 10-725: Convex Optimization Fall 2013 Lecture 2: August 29 Lecturer: Barnabás Póczos Scribes: Samrachana Adhikari, Mattia Ciollaro, Fabrizio Lecci Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. Linear Programg (part I) 2.1 Goals of Lectures on Linear Programg The goals of this and the following lecture on Linear Programg are: 1. to define what a Linear Program is 2. to solve linear programs using the Simplex Algorithm 3. to understand why linear programs are useful by means of motivating examples and applications to Machine Learning 4. to understand the difficulties associated to linear programs (e.g. convergence of algorithms, solvability in polynomial vs. exponential many operations, approximate vs. exact solutions,... ). 2.2 Simplest Optimization Problems The simplest optimization problems concern the optimization (i.e. maximization or imization) of a given 1-dimensional constant or linear function f when no constraints or only bound constraints are placed on its argument. The optimization of f is trivial in these cases. Examples are: x f(x) = c ; c R (f constant: the optimal value c is achieved at any x in the domain of f) x f(x) = ax + b ; a, b R (f linear: the optimal values are + if a > 0 or if a < 0) max x f(x) = ax + b ; c x d a > 0 b, c, d R (f linear with bound constraint: the optimal value ad + b is achieved at x = d). See figure

2 2-2 Lecture 2: August 29 c d f(x) Figure 2.1: A linear objective function with bound constraints. The function f is maximized at x = d. 2.3 Linear Programs A more interesting class of optimization problems involves the optimization of a n-dimensional linear function when m linear constraints are placed on its arguments and the arguments are subject to bound constraints. For instance, the following problem corresponds to the imization of a linear cost function f subject to m linear constraints and n bound constraints: x 1,...,x n f(x 1,..., x n ) = c 1 x c n x n a 11 x a 1n x n b 1. (linear constraints) a m1 x a mn x n b m l 1 x 1 u 1. (bound constraints) l n x n u n. An optimization problem such as the one above is termed Linear Program. In principle, one can express a linear program in different ways. In this case, the linear program above is expressed in inequality form. More conveniently, the same problem can be rewritten in matrix notation as x 1,...,x n f(x 1,..., x n ) = c T x A T x b l x u

3 Lecture 2: August where c = (c 1,..., c n ) T R n x = (x 1,..., x n ) T R n b = (b 1,..., b m ) T R m l = (l 1,..., l n ) T R n u = (u 1,..., u n ) T R n and A T R m n is the m n matrix of coefficients appearing in the linear constraints. Example 2.1. The following linear program is expressed in inequality form: x 1,x 2 2x 1 x 2 x 1 + x 2 5 2x 1 + 3x 2 4 In this case, x 1 0 x 2 0. c = [ 2, 1] T b = [5, 4] T [ ] A T 1 1 =. 2 3 Example 2.2. Let s consider the following linear program: x 1,x 2 z = 2x 1 x 2 x 1 + x 2 5 2x 1 + 3x 2 12 x 1 4 x 1 0 x 2 0. The set of points (x 1, x 2 ) that simultaneously satisfy all the linear and bound constraint is called the set of feasible points or feasible set. The set of points lying at the boundary of the set of feasible points corresponding to intersections of linear and bound constraints is called the set of corner points or the set of corners.

4 2-4 Lecture 2: August 29 The set of feasible points which are not solutions of the linear program is called the set of suboptimal points. Figure 2.2 gives a graphical representation of the linear program. The set of feasible points is marked in pink. It is easy to guess that the the objective function z = z(x 1, x 2 ) is imized at the point C. Notice that at the point C the constraints x 1 + x 2 5 and x 1 4 are active (i.e. they hold with exact equality) whereas the remaining constraints are inactive (i.e. they hold as strict inequalities). Roughly speaking, the Simplex Algorithm jumps on the corners of the linear program and evaluates the objective function z = z(x 1, x 2 ) until the optimum is found x 1 -x 2 =z z A O x 1 +x 2 =5 Feasible set B C D x 1 =4 2x 1 +3x 2 = Figure 2.2: Graphical representation of the linear program of Example 2.2. From Figure 2.2 one can see that some difficulties can arise in a Linear Program. In fact, the feasible set might be empty there might be infinitely global optima if the optimum lies on an edge of the feasible set the optimum could be ±. Finally, notice that in higher dimensional linear programs (such as the general one described at the beginning of this Section), the feasible set is a polytope and the objective function is an hyperplane. 2.4 History and Motivation The interest in Linear Programg began during World War II in order to deal with problems of transportation, scheduling, and allocation of resources subject to certain restrictions such as costs and availability. Examples of these problems are:

5 Lecture 2: August how to efficiently allocate 70 men to 70 jobs (job scheduling) how to produce a particular blend (e.g. 30% Lead, 30% Zinc, 40% Tin) out of 9 different alloys that have different mixtures and different costs in such a way that the total cost is imize how to optimize a flow network (the max flow cut problem). Why Linear Programg? Because the objective functions to be optimized are linear and so are the constraints on their arguments programg = scheduling, and efficient scheduling is one of the focuses of this kind of optimization problems. Also, at the time of the development of the first computers, programg was a fancy word that attracted funds and grants from both military and non-military sources. George Bernard Dantzig ( ) is considered to be the father of Linear Programg. He was the inventor of the Simplex Algorithm. There are many reasons which make Linear Programg an interesting topic. First of all, linear programs represent the simplest (yet nontrivial) optimization problems. Also, many complex systems can be well approximated by linear equations and therefore Linear Programg is a key tool for a number of important applications. Last but not least, linear programs are very appealing from a computational perspective, especially because nowadays there exists many toolboxes that can solve them efficiently. Example 2.3 (The product mix problem). Here is an example of a real life problem that can be framed as a Linear Program. Suppose that a certain company manufactures four models of desks. In order to produce each desk, a certain amount of man hours are needed in the carpentry shop and in the finishing shop. For instance, desks of type 1 need 4 man hours of manufacturing in the carpentry shop and 1 man hour in the finishing shop before their production is complete. Each sold desk guarantees a certain profit to the company depending on the model. For instance, the company makes a 4\$ profit each time a desk of type 1 is sold. However, the number of available man hours in the carpentry shop and in the finishing shop are limited to 6000 and 4000 respectively. What is the optimal number of each type of desk that the company should produce? Figure 2.3: The number of man hours and the profits for the product mix problem. Based on Figure 2.3 which displays the parameters of this problem, we can state the question of interest in terms of a Linear Program. Let x 1,..., x 4 indicate the number of produced units of each of the four models of desks and let f(x 1, x 2, x 3, x 4 ) = 12x 1 +20x 2 +18x 3 +40x 4 be the company s profit function. The company

6 2-6 Lecture 2: August 29 needs to solve max x 1,x 2,x 3,x 4 f(x 1, x 2, x 3, x 4 ) = 12x x x x 4 4x 1 + 9x 2 + 7x x x 1 + x 2 + 3x x x 1 0 x 2 0 x 3 0 x Application: Pattern Classification via Linear Programg This section presents an application of Linear Programg to Pattern Classification 1. The intent is to demonstrate how Linear Programg can be used to perform linear classification (for instance as an alternative to Linear Discriant Analysis). Here are our goals. Given two sets H = {H 1,..., H h } R n and M = {M 1,..., M m } R n, we want to Problem 1) check if H and M are linearly separable Problem 2) if H and M are linearly separable, find a separating hyperplane. Before proceeding, it is worth clarifying what it means for two sets to be linearly separable. Definition 2.4 (linear separability). Two sets H R n and M R n are said to be (strictly) linearly separable if a R n, b R : H {x R n : a T x > b} and M {x R n : a T x b}. In words, H and M are linearly separable if there exists an hyperplane that leaves all the elements of H on one side and all the elements on M on the other side. Figure 2.4 illustrates linear separability. Figure 2.4: An example of linearly separable sets (left) and not linearly separable sets (right). The following Lemma gives an alternative characterization of linear separability. 1 See also for more information

7 Lecture 2: August Lemma 2.5. Two sets H = {H 1,..., H h } R n and M = {M 1,..., M m } R n are linearly separable if and only if there exist a R n and b R such that a T H i b 1 and a T M j b 1 for all i {1,..., h} and for all j {1,..., m}. Proof. ( =) Trivial. (= ) By Definition 2.4 there exist c R n and b R such that c T x > b for x H and c T x b for x M. Define p = x H ct x max x M ct x. It is clear from above that such p is strictly greater than 0. Let a = 2 p c and b = 1 p ( ) x H ct x + max x M ct x. Now, x H at x = x H = 1 p 2 1 ( p ct x = c T x + c T x ) ( ) x H p ) ( x H ct x + max x M ct x + p = x H ( ) 1 c T x + max p x M ct x + p = ( ) = 1 (pb + p) = b + 1, p which equivalently reads Similarly, max x M at x = max x M which equivalently reads = 1 p a T x b 1 x H. 2 1 ( p ct x = max c T x + c T x ) ( ) x M p ) ( max x M ct x + x H ct x p = max x M a T x b 1 x M. ( ) = 1 (pb p) = b 1 p ( ) 1 c T x + p x H ct x p = Notice that the ( ) follow from the definition of p and the ( ) follow from the definition of b. Figure 2.5 depicts the geometry of Lemma 2.5. Figure 2.5: Geometrical representation of Lemma 2.5.

8 2-8 Lecture 2: August 29 Now we show how Linear Programg can be used to solve Problem 1 and Problem 2 above for two given sets H = {H 1,..., H h } R n and M = {M 1,..., M m } R n. Consider the following linear program: y,z,a,b 1 h [y 1 + y y h ] + 1 m [z 1 + z z m ] y i a T H i + b + 1 for i = 1,..., h z i a T M i b + 1 for j = 1,..., m y i 0 z i 0 for i = 1,..., h for j = 1,..., m where y R h, z R m, a R n, b R. In the following we refer to the above program as LP. Theorem 2.6. H and M are linearly separable if and only if the optimal value of LP is 0. Theorem 2.7. If H and M are linearly separable and y, z, a, b f(x) = a T x + b is a separating hyperplane. is an optimal solution of LP, then Proof of Theorems 2.6 and 2.7. The optimal value of LP is 0 y = 0 z = 0 a T H i b + 1 a T M i b 1 for i = 1,..., h for j = 1,..., m H and M are linearly separable and a T x b = 0 is a separating hyperplane. The second implication follows from Lemma 2.5. The authors of [1] performed linear classification via Linear Programg for breast cancer diagnosis. They have been able to classify benign lumps and malignant lumps with 97.5% accuracy. Example 2.8 (Linearly Separable Case). Let H = {(0, 0), (1, 0)} R 2 and M = {(0, 2), (1, 2)} R 2. See

9 Lecture 2: August Figure 2.6 (left). The associated linear program is y,z,a,b 1 2 [y 1 + y 2 ] [z 1 + z 2 ] [ 0 y 1 [a 1, a 2 ] + b + 1 = b + 1 0] [ ] 1 y 2 [a 1, a 2 ] + b + 1 = a b + 1 [ 0 z 1 [a 1, a 2 ] b + 1 = 2a 2] 2 b + 1 [ 1 z 1 [a 1, a 2 ] b + 1 = a 2] 1 + 2a 2 b + 1 y i 0 for i = 1, 2 z i 0 for j = 1, 2. An optimal solution for this program is y 1 = y 2 = z 1 = z 2 = 0 a T = [1, 2], b = 1. By Theorem 2.7, x 1 2x = 0 is a separating hyperplane. Separable Case Non-Separable Case x x 1-2x 2 +1=0 x x 1 +x 2-3= x 1 x 1 Figure 2.6: Left: the separable case of Example 2.8. Right: the non-separable case of Example 2.9 Example 2.9 (Linearly Nonseparable Case). Let H = {(0, 0), (1, 2)} R 2 and M = {(0, 2), (1, 0)} R 2.

10 2-10 Lecture 2: August 29 See Figure 2.6 (right). The associated linear program is y,z,a,b 1 2 [y 1 + y 2 ] [z 1 + z 2 ] [ 0 y 1 [a 1, a 2 ] + b + 1 = b + 1 0] [ 1 y 2 [a 1, a 2 ] + b + 1 = a 2] 1 2a 2 + b + 1 [ ] 0 z 1 [a 1, a 2 ] b + 1 = 2a 2 2 b + 1 [ 1 z 1 [a 1, a 2 ] b + 1 = a 0] 1 b + 1 y i 0 for i = 1, 2 z i 0 for j = 1, 2. An optimal solution for this program is y 1 = 4, y 2 = z 1 = z 2 = 0 a T = [2, 1], b = 3. 2x 1 + x 2 3 = 0 is the proposed non-separating hyperplane. 2.6 Inequality Form vs. Standard Form We have already seen how a linear program is written in the inequality form. In matrix notation: c T x x 1,...,x n A T x b l x u where c = (c 1,..., c n ) T R n x = (x 1,..., x n ) T R n b = (b 1,..., b m ) T R m l = (l 1,..., l n ) T R n u = (u 1,..., u n ) T R n

11 Lecture 2: August and A T R m n is the m n matrix of coefficients appearing in the linear constraints. We say that a linear program is written in he standard form if the inequalities in the linear constraints are replaced by equalities and the variables are constrained to be non-negative. In matrix notation: c T x x 1,...,x n A T x = b x 0 where c = (c 1,..., c n ) T R n x = (x 1,..., x n ) T R n + b = (b 1,..., b m ) T R m + (sometimes b 0 is not required) Theorem Any linear program can be rewritten to an equivalent standard linear program. The proof of the previous theorem consists in a series of rules to convert constraints from one form to the other one: Getting rid of inequalities (except variable bounds). An inequality like x 1 + x 2 4 can be converted into an equality by introducing a non-negative slack variable x 3, i.e. x 1 + x 2 + x 3 = 4 x 3 0. Getting rid of equalities. An equality can be converted into two inequalities. For example x 1 + 2x 2 = 4 x 1 + 2x 2 4 x 1 + 2x 2 4. Getting rid of negative variables. A negative variable can be converted into the difference of two non-negative variables. For example Getting rid of bounded variables. x R = x = u v, for some u 0, v 0. l x u x l x u. Max to Min. A max problem can be easily converted into a problem by noting that max c T x = ( c) T x

12 2-12 Lecture 2: August 29 Getting rid of negative b. If b is negative in the constraint ax = b, we can consider the equivalent constraint ax = b. Remark. If the standard form has n variables and m equalities, then the inequality form has n m variables and m + (n m) = n inequalities. Example Consider the following linear program, expressed in inequality form: max x 1,x 2 2x 1 + 3x 2 x 1 + x 2 4 2x 1 + 5x 2 12 x 1 + 2x 2 5 The equivalent standard form is x 1 0 x 2 0. x 1,x 2 2x 1 3x 2 x 1 + x 2 + u = 4 2x 1 + 5x 2 + v = 12 x 1 + 2x 2 + w = 5 x 1, x 2, u, v, w 0. References [1] Mangasarian, Olvi L., W. Nick Street, and William H. Wolberg. Breast cancer diagnosis and prognosis via linear programg. Operations Research 43.4 (1995):

### Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

### Linear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.

Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### What is Linear Programming?

Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### 1 Solving LPs: The Simplex Algorithm of George Dantzig

Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.

### Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### 1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

### Special Situations in the Simplex Algorithm

Special Situations in the Simplex Algorithm Degeneracy Consider the linear program: Maximize 2x 1 +x 2 Subject to: 4x 1 +3x 2 12 (1) 4x 1 +x 2 8 (2) 4x 1 +2x 2 8 (3) x 1, x 2 0. We will first apply the

### Linear Programming Problems

Linear Programming Problems Linear programming problems come up in many applications. In a linear programming problem, we have a function, called the objective function, which depends linearly on a number

### 3. Evaluate the objective function at each vertex. Put the vertices into a table: Vertex P=3x+2y (0, 0) 0 min (0, 5) 10 (15, 0) 45 (12, 2) 40 Max

SOLUTION OF LINEAR PROGRAMMING PROBLEMS THEOREM 1 If a linear programming problem has a solution, then it must occur at a vertex, or corner point, of the feasible set, S, associated with the problem. Furthermore,

### Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

### LECTURE: INTRO TO LINEAR PROGRAMMING AND THE SIMPLEX METHOD, KEVIN ROSS MARCH 31, 2005

LECTURE: INTRO TO LINEAR PROGRAMMING AND THE SIMPLEX METHOD, KEVIN ROSS MARCH 31, 2005 DAVID L. BERNICK dbernick@soe.ucsc.edu 1. Overview Typical Linear Programming problems Standard form and converting

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Module1. x 1000. y 800.

Module1 1 Welcome to the first module of the course. It is indeed an exciting event to share with you the subject that has lot to offer both from theoretical side and practical aspects. To begin with,

### . P. 4.3 Basic feasible solutions and vertices of polyhedra. x 1. x 2

4. Basic feasible solutions and vertices of polyhedra Due to the fundamental theorem of Linear Programming, to solve any LP it suffices to consider the vertices (finitely many) of the polyhedron P of the

### Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

### This exposition of linear programming

Linear Programming and the Simplex Method David Gale This exposition of linear programming and the simplex method is intended as a companion piece to the article in this issue on the life and work of George

### 5.1 Bipartite Matching

CS787: Advanced Algorithms Lecture 5: Applications of Network Flow In the last lecture, we looked at the problem of finding the maximum flow in a graph, and how it can be efficiently solved using the Ford-Fulkerson

### Standard Form of a Linear Programming Problem

494 CHAPTER 9 LINEAR PROGRAMMING 9. THE SIMPLEX METHOD: MAXIMIZATION For linear programming problems involving two variables, the graphical solution method introduced in Section 9. is convenient. However,

### Operation Research. Module 1. Module 2. Unit 1. Unit 2. Unit 3. Unit 1

Operation Research Module 1 Unit 1 1.1 Origin of Operations Research 1.2 Concept and Definition of OR 1.3 Characteristics of OR 1.4 Applications of OR 1.5 Phases of OR Unit 2 2.1 Introduction to Linear

### LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS. 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method

LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method Introduction to dual linear program Given a constraint matrix A, right

### Mathematical finance and linear programming (optimization)

Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may

### International Doctoral School Algorithmic Decision Theory: MCDA and MOO

International Doctoral School Algorithmic Decision Theory: MCDA and MOO Lecture 2: Multiobjective Linear Programming Department of Engineering Science, The University of Auckland, New Zealand Laboratoire

### Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

### This unit will lay the groundwork for later units where the students will extend this knowledge to quadratic and exponential functions.

Algebra I Overview View unit yearlong overview here Many of the concepts presented in Algebra I are progressions of concepts that were introduced in grades 6 through 8. The content presented in this course

### Study Guide 2 Solutions MATH 111

Study Guide 2 Solutions MATH 111 Having read through the sample test, I wanted to warn everyone, that I might consider asking questions involving inequalities, the absolute value function (as in the suggested

### Chapter 5. Linear Inequalities and Linear Programming. Linear Programming in Two Dimensions: A Geometric Approach

Chapter 5 Linear Programming in Two Dimensions: A Geometric Approach Linear Inequalities and Linear Programming Section 3 Linear Programming gin Two Dimensions: A Geometric Approach In this section, we

### 24. The Branch and Bound Method

24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

### Linear Programming: Theory and Applications

Linear Programming: Theory and Applications Catherine Lewis May 11, 2008 1 Contents 1 Introduction to Linear Programming 3 1.1 What is a linear program?...................... 3 1.2 Assumptions.............................

### Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

### (Basic definitions and properties; Separation theorems; Characterizations) 1.1 Definition, examples, inner description, algebraic properties

Lecture 1 Convex Sets (Basic definitions and properties; Separation theorems; Characterizations) 1.1 Definition, examples, inner description, algebraic properties 1.1.1 A convex set In the school geometry

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

### Chapter 6. Linear Programming: The Simplex Method. Introduction to the Big M Method. Section 4 Maximization and Minimization with Problem Constraints

Chapter 6 Linear Programming: The Simplex Method Introduction to the Big M Method In this section, we will present a generalized version of the simplex method that t will solve both maximization i and

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

### Linear Programming. Solving LP Models Using MS Excel, 18

SUPPLEMENT TO CHAPTER SIX Linear Programming SUPPLEMENT OUTLINE Introduction, 2 Linear Programming Models, 2 Model Formulation, 4 Graphical Linear Programming, 5 Outline of Graphical Procedure, 5 Plotting

### Support Vector Machines

Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric

### 26 Linear Programming

The greatest flood has the soonest ebb; the sorest tempest the most sudden calm; the hottest love the coldest end; and from the deepest desire oftentimes ensues the deadliest hate. Th extremes of glory

### Lecture 5 Principal Minors and the Hessian

Lecture 5 Principal Minors and the Hessian Eivind Eriksen BI Norwegian School of Management Department of Economics October 01, 2010 Eivind Eriksen (BI Dept of Economics) Lecture 5 Principal Minors and

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### Optimization Modeling for Mining Engineers

Optimization Modeling for Mining Engineers Alexandra M. Newman Division of Economics and Business Slide 1 Colorado School of Mines Seminar Outline Linear Programming Integer Linear Programming Slide 2

### 6.4 Logarithmic Equations and Inequalities

6.4 Logarithmic Equations and Inequalities 459 6.4 Logarithmic Equations and Inequalities In Section 6.3 we solved equations and inequalities involving exponential functions using one of two basic strategies.

### Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### EQUATIONS and INEQUALITIES

EQUATIONS and INEQUALITIES Linear Equations and Slope 1. Slope a. Calculate the slope of a line given two points b. Calculate the slope of a line parallel to a given line. c. Calculate the slope of a line

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. #-approximation algorithm.

Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of three

### HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!

Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following

### The Graphical Method: An Example

The Graphical Method: An Example Consider the following linear program: Maximize 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2 0, where, for ease of reference,

### Linear Programming. April 12, 2005

Linear Programming April 1, 005 Parts of this were adapted from Chapter 9 of i Introduction to Algorithms (Second Edition) /i by Cormen, Leiserson, Rivest and Stein. 1 What is linear programming? The first

### SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. !-approximation algorithm.

Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of

### Simplex method summary

Simplex method summary Problem: optimize a linear objective, subject to linear constraints 1. Step 1: Convert to standard form: variables on right-hand side, positive constant on left slack variables for

### Lecture 7: Finding Lyapunov Functions 1

Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.243j (Fall 2003): DYNAMICS OF NONLINEAR SYSTEMS by A. Megretski Lecture 7: Finding Lyapunov Functions 1

### Walrasian Demand. u(x) where B(p, w) = {x R n + : p x w}.

Walrasian Demand Econ 2100 Fall 2015 Lecture 5, September 16 Outline 1 Walrasian Demand 2 Properties of Walrasian Demand 3 An Optimization Recipe 4 First and Second Order Conditions Definition Walrasian

### Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

### Solving Linear Programs

Solving Linear Programs 2 In this chapter, we present a systematic procedure for solving linear programs. This procedure, called the simplex method, proceeds by moving from one feasible solution to another,

### Duality in Linear Programming

Duality in Linear Programming 4 In the preceding chapter on sensitivity analysis, we saw that the shadow-price interpretation of the optimal simplex multipliers is a very useful concept. First, these shadow

### Can linear programs solve NP-hard problems?

Can linear programs solve NP-hard problems? p. 1/9 Can linear programs solve NP-hard problems? Ronald de Wolf Linear programs Can linear programs solve NP-hard problems? p. 2/9 Can linear programs solve

### MATH10040 Chapter 2: Prime and relatively prime numbers

MATH10040 Chapter 2: Prime and relatively prime numbers Recall the basic definition: 1. Prime numbers Definition 1.1. Recall that a positive integer is said to be prime if it has precisely two positive

### Chapter 3: Section 3-3 Solutions of Linear Programming Problems

Chapter 3: Section 3-3 Solutions of Linear Programming Problems D. S. Malik Creighton University, Omaha, NE D. S. Malik Creighton University, Omaha, NE Chapter () 3: Section 3-3 Solutions of Linear Programming

### 3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

### Max-Min Representation of Piecewise Linear Functions

Beiträge zur Algebra und Geometrie Contributions to Algebra and Geometry Volume 43 (2002), No. 1, 297-302. Max-Min Representation of Piecewise Linear Functions Sergei Ovchinnikov Mathematics Department,

### Lecture 6: Logistic Regression

Lecture 6: CS 194-10, Fall 2011 Laurent El Ghaoui EECS Department UC Berkeley September 13, 2011 Outline Outline Classification task Data : X = [x 1,..., x m]: a n m matrix of data points in R n. y { 1,

### Lecture 2. Marginal Functions, Average Functions, Elasticity, the Marginal Principle, and Constrained Optimization

Lecture 2. Marginal Functions, Average Functions, Elasticity, the Marginal Principle, and Constrained Optimization 2.1. Introduction Suppose that an economic relationship can be described by a real-valued

### 1 if 1 x 0 1 if 0 x 1

Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or

### Chapter 4. Duality. 4.1 A Graphical Example

Chapter 4 Duality Given any linear program, there is another related linear program called the dual. In this chapter, we will develop an understanding of the dual linear program. This understanding translates

### Optimal shift scheduling with a global service level constraint

Optimal shift scheduling with a global service level constraint Ger Koole & Erik van der Sluis Vrije Universiteit Division of Mathematics and Computer Science De Boelelaan 1081a, 1081 HV Amsterdam The

### Actually Doing It! 6. Prove that the regular unit cube (say 1cm=unit) of sufficiently high dimension can fit inside it the whole city of New York.

1: 1. Compute a random 4-dimensional polytope P as the convex hull of 10 random points using rand sphere(4,10). Run VISUAL to see a Schlegel diagram. How many 3-dimensional polytopes do you see? How many

### Support Vector Machine (SVM)

Support Vector Machine (SVM) CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin

### No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics

No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results

### Degeneracy in Linear Programming

Degeneracy in Linear Programming I heard that today s tutorial is all about Ellen DeGeneres Sorry, Stan. But the topic is just as interesting. It s about degeneracy in Linear Programming. Degeneracy? Students

### Week 1: Introduction to Online Learning

Week 1: Introduction to Online Learning 1 Introduction This is written based on Prediction, Learning, and Games (ISBN: 2184189 / -21-8418-9 Cesa-Bianchi, Nicolo; Lugosi, Gabor 1.1 A Gentle Start Consider

### POLYNOMIAL RINGS AND UNIQUE FACTORIZATION DOMAINS

POLYNOMIAL RINGS AND UNIQUE FACTORIZATION DOMAINS RUSS WOODROOFE 1. Unique Factorization Domains Throughout the following, we think of R as sitting inside R[x] as the constant polynomials (of degree 0).

### 2013 MBA Jump Start Program

2013 MBA Jump Start Program Module 2: Mathematics Thomas Gilbert Mathematics Module Algebra Review Calculus Permutations and Combinations [Online Appendix: Basic Mathematical Concepts] 2 1 Equation of

### Optimization in R n Introduction

Optimization in R n Introduction Rudi Pendavingh Eindhoven Technical University Optimization in R n, lecture Rudi Pendavingh (TUE) Optimization in R n Introduction ORN / 4 Some optimization problems designing

### MATH2210 Notebook 1 Fall Semester 2016/2017. 1 MATH2210 Notebook 1 3. 1.1 Solving Systems of Linear Equations... 3

MATH0 Notebook Fall Semester 06/07 prepared by Professor Jenny Baglivo c Copyright 009 07 by Jenny A. Baglivo. All Rights Reserved. Contents MATH0 Notebook 3. Solving Systems of Linear Equations........................

### A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

### Vector Spaces 4.4 Spanning and Independence

Vector Spaces 4.4 and Independence October 18 Goals Discuss two important basic concepts: Define linear combination of vectors. Define Span(S) of a set S of vectors. Define linear Independence of a set

### 1 Review of Least Squares Solutions to Overdetermined Systems

cs4: introduction to numerical analysis /9/0 Lecture 7: Rectangular Systems and Numerical Integration Instructor: Professor Amos Ron Scribes: Mark Cowlishaw, Nathanael Fillmore Review of Least Squares

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### Creating, Solving, and Graphing Systems of Linear Equations and Linear Inequalities

Algebra 1, Quarter 2, Unit 2.1 Creating, Solving, and Graphing Systems of Linear Equations and Linear Inequalities Overview Number of instructional days: 15 (1 day = 45 60 minutes) Content to be learned

### 4.1 Learning algorithms for neural networks

4 Perceptron Learning 4.1 Learning algorithms for neural networks In the two preceding chapters we discussed two closely related models, McCulloch Pitts units and perceptrons, but the question of how to

### Practice with Proofs

Practice with Proofs October 6, 2014 Recall the following Definition 0.1. A function f is increasing if for every x, y in the domain of f, x < y = f(x) < f(y) 1. Prove that h(x) = x 3 is increasing, using

### 3.1 Solving Systems Using Tables and Graphs

Algebra 2 Chapter 3 3.1 Solve Systems Using Tables & Graphs 3.1 Solving Systems Using Tables and Graphs A solution to a system of linear equations is an that makes all of the equations. To solve a system

### Introduction to Linear Programming (LP) Mathematical Programming (MP) Concept

Introduction to Linear Programming (LP) Mathematical Programming Concept LP Concept Standard Form Assumptions Consequences of Assumptions Solution Approach Solution Methods Typical Formulations Massachusetts

### 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS. The Jacobi Method

578 CHAPTER 1 NUMERICAL METHODS 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS As a numerical technique, Gaussian elimination is rather unusual because it is direct. That is, a solution is obtained after

### Linear Programming Supplement E

Linear Programming Supplement E Linear Programming Linear programming: A technique that is useful for allocating scarce resources among competing demands. Objective function: An expression in linear programming

### Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

### Discrete Optimization

Discrete Optimization [Chen, Batson, Dang: Applied integer Programming] Chapter 3 and 4.1-4.3 by Johan Högdahl and Victoria Svedberg Seminar 2, 2015-03-31 Todays presentation Chapter 3 Transforms using

### 56:171 Operations Research Midterm Exam Solutions Fall 2001

56:171 Operations Research Midterm Exam Solutions Fall 2001 True/False: Indicate by "+" or "o" whether each statement is "true" or "false", respectively: o_ 1. If a primal LP constraint is slack at the

### 5 INTEGER LINEAR PROGRAMMING (ILP) E. Amaldi Fondamenti di R.O. Politecnico di Milano 1

5 INTEGER LINEAR PROGRAMMING (ILP) E. Amaldi Fondamenti di R.O. Politecnico di Milano 1 General Integer Linear Program: (ILP) min c T x Ax b x 0 integer Assumption: A, b integer The integrality condition

### The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem