# Minimize subject to. x S R

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such as SHORTHEST PATHS, NETWORK FLOWS. But, as we have seen, most of practical graph or network problems are NP-complete and hard to solve. In such a case, it may be interesting to solve a simplified problem to obtain approximations or bounds on the initial hardest problem. Consider the following optimization problem where f : R n R and S R n : f (x) x S A relaxation of the above problem has the following form: f R (x) x S R where f R : R n R is such that f R (x) f (x) for any x S and S S R. It is clear that the optimal solution fr of the relaxation is a lower bound of the optimal solution of the initial problem. In previous section, the considered problems are such that S = X {0,1} n where X R n (or X Q n ) and the fractional relaxation corresponds to consider f R = f and S R = X. A large number of these problems have an underlying network structure. The idea of the Lagrangian Relaxation is to try to use the underlying network structure of these problems in order to use these efficient algorithms. The Lagrangian Relaxation is a method of decomposition: the constraints S = S 1 S 2 of the problems are separated into two groups, namely the easy constraints S 1 and the hard constraints S 2. The hard constraints are then removed, i.e., S R = S 1 and transferred into the objective function, i.e., f R depends on f and S 2. Since S R is a set of easy constraints, it will be possible to solve the relaxation problem. Moreover, the interest of the Lagrangian relaxation is that, in some cases, the optimal solution of the relaxed problem actually gives the optimal solution of the initial problem. We first illustrate the method on a classical example. 195

2 196 CHAPTER 12. LAGRANGIAN RELAXATION 12.1 Constrained Shortest Path. Consider a digraph D =(V,E) with a source s and a sink t. To each arc e E, we associate two values: c e the cost of the arc and and t e the time necessary to take the arc. See Figure 12.1-(a) where s = v 1 and t = v 6.. The CONSTRAINED SHORTEST PATH problem consists in finding an (s,t)-path of minimum cost, under the additional constraint that the path requires at most T units of time to traverse. Such problems arises frequently in practice since in many contexts a company (e.g. a package delivery firm) wants to provide its services at the lowest possible cost and yet ensure a certain level of service to its customers (as embodied in the time restriction). (1,10) v 2 (1,1) (5,7) v 4 (1,7) 21 v v 4 15 v 1 (1,2) (10,1) v 6 v v 6 (10,3) v 3 (2,3) (12,3) v 5 (2,2) 16 8 v 3 18 v 5 6 (a) (b) Figure 12.1: (a) An instance of CONSTRAINED SHORTEST PATH, and (b) its Lagrangian relaxation for µ = 2. Next to each arc e appears the pair (c e,t e ) in (a) and the value c e + µt e in (b). CONSTRAINED SHORTEST PATH can be formulated as the following integer linear programme, with x the indicator vector of the path. e entering v x e x e = e leaving v e E c e x e 1 t e x e T e E if v = s +1 if v = t 0 for all v V \{s,t} (12.1) x e {0,1} e E (12.1) can clearly be decomposed into a classical (and easily solvable) shortest path problem plus an extra time constraint. The idea of the Lagrangian Relaxation is to include this extra constraint as a penalty in the objective function.

3 12.1. CONSTRAINED SHORTEST PATH. 197 More precisely, let µ > 0 and consider the following integer linear programme. e E c e x e µ(t e E t e x e ) 1 if v = s x e x e = +1 if v = t e entering v e leaving v 0 for all v V \{s,t} (12.2) x e {0,1} e E The programme (12.2) is equivalent to finding a shortest (s,t)-path in D with the modified cost function c µ = c + t. See Figure 12.1-(b). e entering v x e x e = e leaving v e E (c e + µ t e )x e 1 x e {0,1} if v = s +1 if v = t 0 for all v V \{s,t} e E Therefore, µ > 0 being fixed, (12.2) can be easily solved. Let opt µ be the value of an optimal solution of (12.2) and let P µ be a path that achieves this solution. Now, let us informally describe how we can obtain the value opt of an optimal solution of (12.1). First, note that any feasible solution of (12.1) is an (s,t)-path P such that e E(P) t e T. Therefore, because µ > 0, any feasible solution P has a cost c µ (P) in (12.2) that is no larger than its cost c(p) in (12.1). In particular, opt µ c µ (P ) c(p )=opt for any µ > 0, where P is a feasible solution of (12.1) achieving the optimal value opt. That is, the optimal solution of (12.2) provides a lower bound on the optimal value of (12.1). Let us consider the example of Figure 12.1-(a) for T = 10. First, let us set µ = 0. P0 is the path (v 1,v 2,v 4,v 6 ) with cost opt0 = c(p 0 )=3. However, this path is not feasible in the initial problem. Therefore, the relaxation with µ = 0 only provides that 3 opt. For µ = 1, we get P1 =(v 1,v 2,v 5,v 6 ) and opt1 = c(p 1 )=20 µ T = 10. Again, this path is not feasible in the initial problem. However, we got a better lower bound: 10 opt. For µ = 2 (see Figure 12.1-(b)),two paths achieve the optimal value opt2 = 15 opt. But, one of them, namely P2 =(v 1,v 3,v 2,v 5,v 6 ) is such that e E(P 2 ) t e = T. Therefore, P2 is a feasible solution of (12.1). Moreover, 15 = c(p2 ) opt c 2 (P2 )=15. Hence, we have solved (12.1). In this example, we have shown that choosing the good value of µ allows to obtain the optimal solution of the initial problem. In the sequels, we show how to choose a value of µ that will provide a good lower bound for the initial problem and we show that, in some cases, it actually leads to the optimal solution.

4 198 CHAPTER 12. LAGRANGIAN RELAXATION 12.2 The Lagrangian Relaxation Technique In this section, we formally define the Lagrangian dual of an optimization problem and show that the solution of the Lagrangian dual provides a lower (resp., upper) bound of the initial minimization (resp., maximization) problem. Moreover, in the case of (convex) linear programmes, the optimal solution of the Lagrangian dual coincides with the optimal solution of the initial problem. Also, the bound obtained thanks to the Lagrangian relaxation is at least as good as the one obtained from fractional relaxation Lagrangian dual Consider the following integer linear programme: c T x Ax = b x X (12.3) The Lagrangian relaxation procedure uses the idea of relaxing the explicit linear constraints by bringing them into the objective function with associated vector µ called the Lagrange multiplier. We refer to the resulting problem c T x + µ T (Ax b) x X (12.4) as the Lagrangian relaxation or Lagrangian subproblem or the original problem (12.3), and we refer to the function L(µ)=min{c T x + µ T (Ax b) x X}, as the Lagrangian function. Lemma 12.1 (Lagrangian Bounding Principle). For any Lagrangian multiplier µ, the value L(µ) of the Lagrangian function is a lower bound on the optimal objective function value z of the original problem (12.3). Proof. Since Ax = b for every feasible solution x of 12.3, for any Lagrangian multiplier µ, z = min{c T x Ax = b,x X} = min{c T x + µ T (Ax b) Ax = b,x X}. Since removing the constraints Ax = b from the second formulation cannot lead to an increase in the value of the objective function (the value might decrease), z min{c T x+µ T (Ax b) x X} = L(µ) Bounds and optimality certificates To obtain the sharpest possible lower bound, we would need to solve the following optimization problem L = maxl(µ) (12.5) µ

5 12.2. THE LAGRANGIAN RELAXATION TECHNIQUE 199 which we refer to as the Lagrangian Dual problem associated with the original optimization problem (12.3). The Lagrangian Bounding Principle has the following immediate implication. Property 12.2 (Weak duality). The optimal solution L of the Lagrangian dual (12.5) is a lower bound on the value z of an optimal solution of (12.3), i.e., L z. Proof. For any Lagrangian multiplier µ and for any feasible solution x of Problem 12.3, we have L(µ) L c T x. Corollary 12.3 (Optimality test). Let µ be a Lagrangian multiplier. If x is a feasible solution of (12.3) satisfying L(µ)=c T x. Then L(µ) is the optimal value of the Lagrangian dual, i.e., L = L(µ), and x is an optimal solution of the primal (12.3). As indicated by the previous property and its corollary, the Lagrangian Bounding Principle immediately implies one advantage of the Lagrangian relaxation approach. It provides a certificate for guaranteeing that a given solution to the primal (12.3) is optimal. Indeed, in the next subsection, we describe a method to compute the optimal solution L of the Lagrangian dual. Even if L(µ) < c T x, having the lower bound permits us to state a bound on how far a given solution is from optimality: If (c T x L(µ))/L(µ) 0.05, for example, we know that the value of the feasible solution x is no more than 5% from optimality. This type of bound is very useful in practice. It permits us to assess the degree of sub-optimality of given solutions and it permits us to terminate our search for an optimal solution when we have a solution which is close enough to optimality for our purposes. Inequality constraints In (12.3), the hard constraints are all equalities. In practice, problems are described using inequalities. Consider the following optimization problem: c T x Ax b x X (12.6) In that case, we consider only Lagrangian multipliers with positive coefficients. The Lagrangian dual is L = maxl(µ) (12.7) µ 0 In this setting, the Lagrangian Bounding Principle and the weak duality property (12.2) are still valid. However, a vector x may not be an optimal solution of the primal problem even if

6 200 CHAPTER 12. LAGRANGIAN RELAXATION x is feasible for the primal problem and if x achieves the optimal solution of the Lagrangian dual L = L(µ) for some µ 0. The optimality test (Corollary ) may however be adapted in the following way: Property If L(µ) is achieved by a vector x such that x is a feasible solution of (12.6), and moreover x satisfies the complementary slackness condition µ T (Ax b)=0, then, L(µ) is the optimal value of the Lagrangian dual (12.7) and x is an optimal solution of the primal (12.6) Linear Programming All results presented above do not depend on the kind of considered optimization problem. More precisely, previously, the set X defining the easy constraints is arbitrary. We know focus on the linear programming case. More precisely, let us consider the following problem: c T x Ax = b Dx q x 0 Recall that the corresponding dual problem is (see Section 9.3.2): (12.8) Maximize b T y + q T z A T y + D T z c y 2 0 (12.9) For any vector µ, we set the Lagrangian function as L(µ)=min{c T x + µ T (Ax b) Dx q,x 0}. (12.10) Theorem Let (P) be any linear programme such as Problem The optimal value L = max µ L(µ) of the Lagrangian dual of (P) coincides with the optimal value opt of (P). Proof. Let x be a vector achieving the optimal value opt of (12.8) and let (y,z ) be an optimal solution of the dual Then A T y +D T z c 0. Moreover, by the complementary slackness conditions (Theorem 9.11): (A T y + D T z c)x = 0 y A z D x b q = z (Dx q)=0

7 12.2. THE LAGRANGIAN RELAXATION TECHNIQUE 201 Set µ = y, we have L( y )=min{c T x (y ) T (Ax b) Dx q,x 0}. That is, any vector achieves L( y ) if and only if it achieves the optimal value of the following linear programme: (c T (y ) T A)x Dx q and x 0. (12.11) The corresponding dual problem is Maximize q T z D T z c A T y and z 0. (12.12) Therefore, the complementary slackness conditions implies that if there is a feasible solution x to (12.11) and a feasible solution z feasible to (12.12) such that z(d x q)=0, and (D T z c + A T y ) x = 0, then x is an optimal solution of (12.11). Since setting x = x and z = z satisfies the complementary slackness conditions, x is an optimal solution to (12.11). Thus, L( y )=c T x + µ T (Ax b)=c T x = opt. The result follows Corollary Theorem Let X be a finite set in R n and let H (X) its convex hull. Then, the Lagrangian dual of c T x Ax = b and x X} has the same optimal value as c T x Ax = b and x H (X)}. Proof. Let L(µ) =min{c T x + µ(ax b) x X}. It is equivalent to L(µ) =min{c T x + µ(ax b) x H (X)} because the optimal values of the second formulation are reached at some extreme points of the polytope H (X) and that any vertex of H (X) belongs to X. Therefore, the solution of the Lagrangian dual of the initial problem equals the solution of the Lagrangian dual of c T x Ax = b and x H (X)}. The convex hull of a finite number of points can be defined as the intersection of a finite family of half-spaces, i.e., by a finite number of inequalities. Therefore, applying Theorem 12.5 to c T x Ax = b and x H (X)}, we get that its optimal value and the one of its Lagrangian dual coincide. Theorem Let (ILP) be an integer linear programme. Then the bound achieved by a Lagrangian relaxation of (ILP) is at least as good as the optimal value of its fractional relaxation.

8 202 CHAPTER 12. LAGRANGIAN RELAXATION Proof. Consider the integer linear programme (ILP): c T x Ax = b and x X Z n where X is convex since it is defined by linear inequalities. By Theorem 12.6, its Lagrangian relaxation has the same solution as (LR): c T x Ax = b and x H (X Z n ). Since the convex hull H = H (X Z n ) of X Z n is such that H X, we get that the solution of (LR) is not better than the one of (LP): c T x Ax = b and x X, the fractional relaxation of (ILP). Hence, opt(lp) opt(lr) opt(ilp) Solving the Lagrangian dual In this subsection, we describe a method to approximate the Lagrangian dual L = maxl(µ) (12.13) µ of the Lagrangian function which relaxes the constraints Ax = b, that is L(µ)=min{c T x + µ T (Ax b) x X}. Recall that the principle of the Lagrangian relaxation is to include the hard constraints into the objective function. In other words, optimizing a linear function over X is assumed to be easy. Therefore, µ being fixed, we can compute the value of L(µ) and an corresponding optimal solution x X. Note that the Lagrangian function is the lower envelope of the set of hyperplanes { c T x + µ T (Ax b) x X }. Therefore, L(µ) is a concave function. Say differently, it is equivalent to solve the following linear programme: Maximize w w c T x + µ T (Ax b) x X, µ R n Generally the number of constraints of such a programme is exponential, so we use a gradient descent method to compute a value as close as desired to the Lagrangian dual. More precisely, given a concave function f : R n R, a vector g is a subgradient of f at x if, for any y R n, f (y) f (x)+g T (y x). The function f is differentiable in x if and only if f admits a unique subgradient in x. If L were differentiable, we would use the gradient descent method to converge toward the optimal value. However, in our case, L is not everywhere differentiable because it is a piecewise linear function. So we compute a sequence of (µ k ) k N such that L(µ k ) converges to the optimal solution, using the following subgradient method.

9 12.3. APPLICATIONS 203 Algorithm 12.1 (Subgradient method). 0. Set k = 0 and choose µ 0 R n ; 1. Compute L(µ k ) and a vector x k X where it is achieved; 2. Choose a subgradient g k = Ax k b of the function L at µ k ; 3. If g k = 0, then stop, the optimal solution is L(µ k ) 4. Compute µ k+1 = µ k + θ T k.g k where θ k is the stepsize at this step. 5. Increment k and go to Step 2. In practice, the formula to calculate the stepsize is θ k = UB L(µ k) where UB is an upper Ax k b 2 bound on the optimal solution we want to compute. In the case when the constraints of the initial programme that we relax were Ax b, then the method must be slightly modified such that µ k 0 for any k 0. For this purpose, at Step 4., the ith coordinate of µ k+1 is taken as the maximum between 0 and the ith coordinate of µ k + θ T k.g k. In all cases, the number k of iterations depends on the desired accuracy of the result Applications Travelling Salesman Problem Let D =(V,E) be an arc-weighted complete digraph, i.e., for any u,v V there are both the arcs (u,v) and (v,u) and there is a cost function c : E R. The cost of a (directed) cycle is the sum of the cost of its arcs. The TRAVELLING SALESMAN problem is to find a minimumcost (directed) cycle passing exactly once through all vertices. This problem is a well known NP-hard problem. A first way to formulate this problem as an integer linear programme is the following, where the variable x e {0,1} indicates if the arc e belongs to the cycle or not.

10 204 CHAPTER 12. LAGRANGIAN RELAXATION c(e)x e e E x e = 1 e entering v for all v V x e = 1 e leaving v x e S 1 e E(S) x e {0,1} for all v V for all S V for all e E (12.14) The first two sets of constraints and the integrity constraint ensure that the solution is a disjoint union of cycles covering all vertices exactly once, thats is a cycle factor. So, if the third set of constraints is omitted, this programme solves the minimum-cost cycle-factor problem. The minimum cycle-factor problem consists in finding a cycle factor of a digraph D =(V, E) with minimum cost. This problem can be solved in polynomial time since it is equivalent to solve a minimum-cost perfect matching problem. Indeed, consider the bipartite graph G with vertex set {v,v + v V } in which there is an edge v u + of cost c(vu) for each arc (v,u) E. There is a one-to-one correspondence between the perfect matchings of G and the cycle-factors of D. A possibility would be to relax the third set of constraints and include it in the objective funcction, but the resulting objective function would be a sum of an exponential number of terms, which is not convenient. We shall now see how to replace the third set of constraints by some other, so that Lagraangian relaxation is practically possible. The third set of constraints ensures we have only one cycle. In other words, it ensures the connectivity of the solution. We now give a new linear programming formulation of TRAVEL- LING SALESMAN in which this set of constraints is replaced by two others guaranteeing the connectivity of any feasible solutions. The first of this two new sets of constraints ensures that there is a flow from some vertex s V to all other vertices. The second forces this flow to use the arcs of the solution, so that any feasible solution must be connected.

11 12.3. APPLICATIONS 205 e entering s c(e)x e e E x e = 1 e entering v x e = 1 e leaving v f e e entering s e leaving s f e e leaving s f e (n 1)x e f e = (n 1) f e = 1 for all v V for all v V for all v V \{s} e E x e {0,1} for all e E Using this formulation and relaxing the constraint f e (n 1)x e, the problem is equivalent to solving separately a flow problem and a minimum cycle-factor problem.

12 206 CHAPTER 12. LAGRANGIAN RELAXATION

13 Bibliography [1] R. K. Ahuja, T. L. Magnanti and J. B. Orlin. Network Flows: Theory, Algorithms, and Applications. Prentice Hall,

14 208 BIBLIOGRAPHY

### Lecture 7 - Linear Programming

COMPSCI 530: Design and Analysis of Algorithms DATE: 09/17/2013 Lecturer: Debmalya Panigrahi Lecture 7 - Linear Programming Scribe: Hieu Bui 1 Overview In this lecture, we cover basics of linear programming,

### CHAPTER 9. Integer Programming

CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral

### Inverse Optimization by James Orlin

Inverse Optimization by James Orlin based on research that is joint with Ravi Ahuja Jeopardy 000 -- the Math Programming Edition The category is linear objective functions The answer: When you maximize

### 10.1 Integer Programming and LP relaxation

CS787: Advanced Algorithms Lecture 10: LP Relaxation and Rounding In this lecture we will design approximation algorithms using linear programming. The key insight behind this approach is that the closely

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### Lecture 6: Approximation via LP Rounding

Lecture 6: Approximation via LP Rounding Let G = (V, E) be an (undirected) graph. A subset C V is called a vertex cover for G if for every edge (v i, v j ) E we have v i C or v j C (or both). In other

### Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, In which we introduce the theory of duality in linear programming.

Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, 2011 Lecture 6 In which we introduce the theory of duality in linear programming 1 The Dual of Linear Program Suppose that we

### Good luck, veel succes!

Final exam Advanced Linear Programming, May 7, 13.00-16.00 Switch off your mobile phone, PDA and any other mobile device and put it far away. No books or other reading materials are allowed. This exam

### Advanced Topics in Machine Learning (Part II)

Advanced Topics in Machine Learning (Part II) 3. Convexity and Optimisation February 6, 2009 Andreas Argyriou 1 Today s Plan Convex sets and functions Types of convex programs Algorithms Convex learning

### 1 Polyhedra and Linear Programming

CS 598CSC: Combinatorial Optimization Lecture date: January 21, 2009 Instructor: Chandra Chekuri Scribe: Sungjin Im 1 Polyhedra and Linear Programming In this lecture, we will cover some basic material

### By W.E. Diewert. July, Linear programming problems are important for a number of reasons:

APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization)

### Solving the ILP using branch-and-cut

Solving the ILP using branch-and-cut Solving ILPs is a main topic in combinatorial optimization. We will take a brief look at the branch-and-cut approach. Branch-and-cut makes use of two techniques: Cutting

### Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

### 3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

### 5.1 Bipartite Matching

CS787: Advanced Algorithms Lecture 5: Applications of Network Flow In the last lecture, we looked at the problem of finding the maximum flow in a graph, and how it can be efficiently solved using the Ford-Fulkerson

### 2.3 Scheduling jobs on identical parallel machines

2.3 Scheduling jobs on identical parallel machines There are jobs to be processed, and there are identical machines (running in parallel) to which each job may be assigned Each job = 1,,, must be processed

### max cx s.t. Ax c where the matrix A, cost vector c and right hand side b are given and x is a vector of variables. For this example we have x

Linear Programming Linear programming refers to problems stated as maximization or minimization of a linear function subject to constraints that are linear equalities and inequalities. Although the study

### Section Notes 4. Duality, Sensitivity, Dual Simplex, Complementary Slackness. Applied Math 121. Week of February 28, 2011

Section Notes 4 Duality, Sensitivity, Dual Simplex, Complementary Slackness Applied Math 121 Week of February 28, 2011 Goals for the week understand the relationship between primal and dual programs. know

### Min-cost flow problems and network simplex algorithm

Min-cost flow problems and network simplex algorithm The particular structure of some LP problems can be sometimes used for the design of solution techniques more efficient than the simplex algorithm.

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Linear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.

Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.

### princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora

princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora Scribe: One of the running themes in this course is the notion of

### 26 Linear Programming

The greatest flood has the soonest ebb; the sorest tempest the most sudden calm; the hottest love the coldest end; and from the deepest desire oftentimes ensues the deadliest hate. Th extremes of glory

Gradient, Subgradient and how they may affect your grade(ient) David Sontag & Yoni Halpern February 7, 2016 1 Introduction The goal of this note is to give background on convex optimization and the Pegasos

### Permutation Betting Markets: Singleton Betting with Extra Information

Permutation Betting Markets: Singleton Betting with Extra Information Mohammad Ghodsi Sharif University of Technology ghodsi@sharif.edu Hamid Mahini Sharif University of Technology mahini@ce.sharif.edu

### Shiqian Ma, SEEM 5121, Dept. of SEEM, CUHK 1. Chapter 2. Convex Optimization

Shiqian Ma, SEEM 5121, Dept. of SEEM, CUHK 1 Chapter 2 Convex Optimization Shiqian Ma, SEEM 5121, Dept. of SEEM, CUHK 2 2.1. Convex Optimization General optimization problem: min f 0 (x) s.t., f i (x)

### Linear Programming, Lagrange Multipliers, and Duality Geoff Gordon

lp.nb 1 Linear Programming, Lagrange Multipliers, and Duality Geoff Gordon lp.nb 2 Overview This is a tutorial about some interesting math and geometry connected with constrained optimization. It is not

### Lecture 7: Approximation via Randomized Rounding

Lecture 7: Approximation via Randomized Rounding Often LPs return a fractional solution where the solution x, which is supposed to be in {0, } n, is in [0, ] n instead. There is a generic way of obtaining

### Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

### Approximation Algorithms: LP Relaxation, Rounding, and Randomized Rounding Techniques. My T. Thai

Approximation Algorithms: LP Relaxation, Rounding, and Randomized Rounding Techniques My T. Thai 1 Overview An overview of LP relaxation and rounding method is as follows: 1. Formulate an optimization

### Chapter 15 Introduction to Linear Programming

Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of

### LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS. 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method

LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method Introduction to dual linear program Given a constraint matrix A, right

### Introduction to Convex Optimization

Convexity Xuezhi Wang Computer Science Department Carnegie Mellon University 10701-recitation, Jan 29 Outline Convexity 1 Convexity 2 3 Outline Convexity 1 Convexity 2 3 Convexity Definition For x, x X

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### Can linear programs solve NP-hard problems?

Can linear programs solve NP-hard problems? p. 1/9 Can linear programs solve NP-hard problems? Ronald de Wolf Linear programs Can linear programs solve NP-hard problems? p. 2/9 Can linear programs solve

### Absolute Value Programming

Computational Optimization and Aplications,, 1 11 (2006) c 2006 Springer Verlag, Boston. Manufactured in The Netherlands. Absolute Value Programming O. L. MANGASARIAN olvi@cs.wisc.edu Computer Sciences

### Integrating Benders decomposition within Constraint Programming

Integrating Benders decomposition within Constraint Programming Hadrien Cambazard, Narendra Jussien email: {hcambaza,jussien}@emn.fr École des Mines de Nantes, LINA CNRS FRE 2729 4 rue Alfred Kastler BP

### THEORY OF SIMPLEX METHOD

Chapter THEORY OF SIMPLEX METHOD Mathematical Programming Problems A mathematical programming problem is an optimization problem of finding the values of the unknown variables x, x,, x n that maximize

### Geometry of Linear Programming

Chapter 2 Geometry of Linear Programming The intent of this chapter is to provide a geometric interpretation of linear programming problems. To conceive fundamental concepts and validity of different algorithms

### Fairness in Routing and Load Balancing

Fairness in Routing and Load Balancing Jon Kleinberg Yuval Rabani Éva Tardos Abstract We consider the issue of network routing subject to explicit fairness conditions. The optimization of fairness criteria

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### Approximation Algorithms

Approximation Algorithms or: How I Learned to Stop Worrying and Deal with NP-Completeness Ong Jit Sheng, Jonathan (A0073924B) March, 2012 Overview Key Results (I) General techniques: Greedy algorithms

### ORF 523 Lecture 8 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Tuesday, March 8, 2016

ORF 523 Lecture 8 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Tuesday, March 8, 2016 When in doubt on the accuracy of these notes, please cross check with the instructor s

### Mathematics Notes for Class 12 chapter 12. Linear Programming

1 P a g e Mathematics Notes for Class 12 chapter 12. Linear Programming Linear Programming It is an important optimization (maximization or minimization) technique used in decision making is business and

### Week 5 Integral Polyhedra

Week 5 Integral Polyhedra We have seen some examples 1 of linear programming formulation that are integral, meaning that every basic feasible solution is an integral vector. This week we develop a theory

### 1 Digraphs. Definition 1

1 Digraphs Definition 1 Adigraphordirected graphgisatriplecomprisedofavertex set V(G), edge set E(G), and a function assigning each edge an ordered pair of vertices (tail, head); these vertices together

### Subgradients. S. Boyd and L. Vandenberghe Notes for EE364b, Stanford University, Winter April 13, 2008

Subgradients S. Boyd and L. Vandenberghe Notes for EE364b, Stanford University, Winter 2006-07 April 13, 2008 1 Definition We say a vector g R n is a subgradient of f : R n R at x domf if for all z domf,

CS787: Advanced Algorithms Topic: Online Learning Presenters: David He, Chris Hopman 17.3.1 Follow the Perturbed Leader 17.3.1.1 Prediction Problem Recall the prediction problem that we discussed in class.

### Arrangements And Duality

Arrangements And Duality 3.1 Introduction 3 Point configurations are tbe most basic structure we study in computational geometry. But what about configurations of more complicated shapes? For example,

### Definition 11.1. Given a graph G on n vertices, we define the following quantities:

Lecture 11 The Lovász ϑ Function 11.1 Perfect graphs We begin with some background on perfect graphs. graphs. First, we define some quantities on Definition 11.1. Given a graph G on n vertices, we define

### Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

### Introduction to Flocking {Stochastic Matrices}

Supelec EECI Graduate School in Control Introduction to Flocking {Stochastic Matrices} A. S. Morse Yale University Gif sur - Yvette May 21, 2012 CRAIG REYNOLDS - 1987 BOIDS The Lion King CRAIG REYNOLDS

### The multi-integer set cover and the facility terminal cover problem

The multi-integer set cover and the facility teral cover problem Dorit S. Hochbaum Asaf Levin December 5, 2007 Abstract The facility teral cover problem is a generalization of the vertex cover problem.

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. #-approximation algorithm.

Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of three

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### 1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### An Algorithm for Fractional Assignment Problems. Maiko Shigeno. Yasufumi Saruwatari y. and. Tomomi Matsui z. Department of Management Science

An Algorithm for Fractional Assignment Problems Maiko Shigeno Yasufumi Saruwatari y and Tomomi Matsui z Department of Management Science Science University of Tokyo 1-3 Kagurazaka, Shinjuku-ku, Tokyo 162,

### 1 Unrelated Parallel Machine Scheduling

IIIS 014 Spring: ATCS - Selected Topics in Optimization Lecture date: Mar 13, 014 Instructor: Jian Li TA: Lingxiao Huang Scribe: Yifei Jin (Polishing not finished yet.) 1 Unrelated Parallel Machine Scheduling

### Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach

MASTER S THESIS Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach PAULINE ALDENVIK MIRJAM SCHIERSCHER Department of Mathematical

### I. Solving Linear Programs by the Simplex Method

Optimization Methods Draft of August 26, 2005 I. Solving Linear Programs by the Simplex Method Robert Fourer Department of Industrial Engineering and Management Sciences Northwestern University Evanston,

### Optimization of Design. Lecturer:Dung-An Wang Lecture 12

Optimization of Design Lecturer:Dung-An Wang Lecture 12 Lecture outline Reading: Ch12 of text Today s lecture 2 Constrained nonlinear programming problem Find x=(x1,..., xn), a design variable vector of

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

### 24. The Branch and Bound Method

24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

### The Goldberg Rao Algorithm for the Maximum Flow Problem

The Goldberg Rao Algorithm for the Maximum Flow Problem COS 528 class notes October 18, 2006 Scribe: Dávid Papp Main idea: use of the blocking flow paradigm to achieve essentially O(min{m 2/3, n 1/2 }

### Identifying More-for-Less Paradox in the Linear Fractional Transportation Problem Using Objective Matrix

MATEMATIKA, 2012, Volume 28, Number 2, 173 180 c Department of Mathematical Sciences, UTM Identifying More-for-Less Paradox in the Linear Fractional Transportation Problem Using Objective Matrix 1 Vishwas

### Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing

### Big Data - Lecture 1 Optimization reminders

Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics

### Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

### 1 Solving LPs: The Simplex Algorithm of George Dantzig

Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.

### Scheduling Home Health Care with Separating Benders Cuts in Decision Diagrams

Scheduling Home Health Care with Separating Benders Cuts in Decision Diagrams André Ciré University of Toronto John Hooker Carnegie Mellon University INFORMS 2014 Home Health Care Home health care delivery

### Limit processes are the basis of calculus. For example, the derivative. f f (x + h) f (x)

SEC. 4.1 TAYLOR SERIES AND CALCULATION OF FUNCTIONS 187 Taylor Series 4.1 Taylor Series and Calculation of Functions Limit processes are the basis of calculus. For example, the derivative f f (x + h) f

### Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

### This exposition of linear programming

Linear Programming and the Simplex Method David Gale This exposition of linear programming and the simplex method is intended as a companion piece to the article in this issue on the life and work of George

### Scheduling Shop Scheduling. Tim Nieberg

Scheduling Shop Scheduling Tim Nieberg Shop models: General Introduction Remark: Consider non preemptive problems with regular objectives Notation Shop Problems: m machines, n jobs 1,..., n operations

### 6. Mixed Integer Linear Programming

6. Mixed Integer Linear Programming Javier Larrosa Albert Oliveras Enric Rodríguez-Carbonell Problem Solving and Constraint Programming (RPAR) Session 6 p.1/40 Mixed Integer Linear Programming A mixed

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### Partitioning edge-coloured complete graphs into monochromatic cycles and paths

arxiv:1205.5492v1 [math.co] 24 May 2012 Partitioning edge-coloured complete graphs into monochromatic cycles and paths Alexey Pokrovskiy Departement of Mathematics, London School of Economics and Political

### The Line Connectivity Problem

Konrad-Zuse-Zentrum für Informationstechnik Berlin Takustraße 7 D-14195 Berlin-Dahlem Germany RALF BORNDÖRFER MARIKA NEUMANN MARC E. PFETSCH The Line Connectivity Problem Supported by the DFG Research

### Math 443/543 Graph Theory Notes 4: Connector Problems

Math 443/543 Graph Theory Notes 4: Connector Problems David Glickenstein September 19, 2012 1 Trees and the Minimal Connector Problem Here is the problem: Suppose we have a collection of cities which we

### The chromatic spectrum of mixed hypergraphs

The chromatic spectrum of mixed hypergraphs Tao Jiang, Dhruv Mubayi, Zsolt Tuza, Vitaly Voloshin, Douglas B. West March 30, 2003 Abstract A mixed hypergraph is a triple H = (X, C, D), where X is the vertex

### Convex Programming Tools for Disjunctive Programs

Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible

### Markov Chains, part I

Markov Chains, part I December 8, 2010 1 Introduction A Markov Chain is a sequence of random variables X 0, X 1,, where each X i S, such that P(X i+1 = s i+1 X i = s i, X i 1 = s i 1,, X 0 = s 0 ) = P(X

### Lecture 5: Conic Optimization: Overview

EE 227A: Conve Optimization and Applications January 31, 2012 Lecture 5: Conic Optimization: Overview Lecturer: Laurent El Ghaoui Reading assignment: Chapter 4 of BV. Sections 3.1-3.6 of WTB. 5.1 Linear

### Deterministic Problems

Chapter 2 Deterministic Problems 1 In this chapter, we focus on deterministic control problems (with perfect information), i.e, there is no disturbance w k in the dynamics equation. For such problems there

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. !-approximation algorithm.

Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of

### Permutation Betting Markets: Singleton Betting with Extra Information

Permutation Betting Markets: Singleton Betting with Extra Information Mohammad Ghodsi Sharif University of Technology ghodsi@sharif.edu Hamid Mahini Sharif University of Technology mahini@ce.sharif.edu

### NOTES ON MEASURE THEORY. M. Papadimitrakis Department of Mathematics University of Crete. Autumn of 2004

NOTES ON MEASURE THEORY M. Papadimitrakis Department of Mathematics University of Crete Autumn of 2004 2 Contents 1 σ-algebras 7 1.1 σ-algebras............................... 7 1.2 Generated σ-algebras.........................

### Equilibrium computation: Part 1

Equilibrium computation: Part 1 Nicola Gatti 1 Troels Bjerre Sorensen 2 1 Politecnico di Milano, Italy 2 Duke University, USA Nicola Gatti and Troels Bjerre Sørensen ( Politecnico di Milano, Italy, Equilibrium

### Theory of Linear Programming

Theory of Linear Programming Debasis Mishra April 6, 2011 1 Introduction Optimization of a function f over a set S involves finding the maximum (minimum) value of f (objective function) in the set S (feasible

### What is Linear Programming?

Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to

### Minimum cost maximum flow, Minimum cost circulation, Cost/Capacity scaling

6.854 Advanced Algorithms Lecture 16: 10/11/2006 Lecturer: David Karger Scribe: Kermin Fleming and Chris Crutchfield, based on notes by Wendy Chu and Tudor Leu Minimum cost maximum flow, Minimum cost circulation,

### Some Notes on Taylor Polynomials and Taylor Series

Some Notes on Taylor Polynomials and Taylor Series Mark MacLean October 3, 27 UBC s courses MATH /8 and MATH introduce students to the ideas of Taylor polynomials and Taylor series in a fairly limited

### Minimum Makespan Scheduling

Minimum Makespan Scheduling Minimum makespan scheduling: Definition and variants Factor 2 algorithm for identical machines PTAS for identical machines Factor 2 algorithm for unrelated machines Martin Zachariasen,

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### Two General Methods to Reduce Delay and Change of Enumeration Algorithms

ISSN 1346-5597 NII Technical Report Two General Methods to Reduce Delay and Change of Enumeration Algorithms Takeaki Uno NII-2003-004E Apr.2003 Two General Methods to Reduce Delay and Change of Enumeration