# 6.854 Advanced Algorithms. Scribes: Jay Kumar Sundararajan. Duality

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Lecture 1: 10/13/ Advanced Algorithms Scribes: Jay Kumar Sundararajan Lecturer: David Karger Duality This lecture covers weak and strong duality, and also explains the rules for finding the dual of a linear program, with an example. Before we move on to duality, we shall first see some general facts about the location of the optima of a linear program. 1.1 Structure of LP solutions Some intuition in two dimensions Consider a linear program - Maximize y T b subject to y T A c The feasible region of this LP is in general, a convex polyhedron. Visualize it as a polygon in 2 dimensions, for simplicity. Now, maximizing y T b is the same as maximizing the projection of the vector y in the direction represented by vector b. For whichever direction b we choose, the point y that maximizes y T b cannot lie strictly in the interior of the feasible region. The reason is that, from an interior point, we can move further in any direction, and still be feasible. In particular, by moving along b, we can get to a point with a larger projection along b. This intuition suggests that the optimal solution of an LP will never lie in the interior of the feasible region, but only on the boundaries. In fact, we can say more. We can show that for any LP, the optimal solutions are always at the corners of the feasible region polyhedron. This notion is formalized in the next subsection Some definitions Definition 1 (Vertex of a Polyhedron) A point in the polyhedron which is uniquely optimal for some linear objective, is called a vertex of the polyhedron. Definition 2 (Extreme Point of a Polyhedron) A point in the polyhedron which is not 1-1

2 Lecture 1: 10/13/ a convex combination of two other points in the polyhedron is called an extreme point of the polyhedron. Definition 3 (Tightness) A constraint of the form a T x b, a T x = b or a T x b in a linear program is said to be tight for a certain point y, if a T y = b. Definition 4 (Basic Solution) For an n-dimensional linear program, a point is called a basic solution, if n linearly independent constraints are tight for that point. Definition 5 (Basic Feasible Solution) A point is a basic feasible solution, iff it is a basic solution that is also feasible. Note: If x is a basic feasible solution, then it is in fact, the unique point that is tight for all its tight constraints. This is because, there can be only one solution for a set of n linearly independent equalities, in n-dimensional space. Theorem 1 For a polyhedron P and a point x P, the following are equivalent: 1. x is a basic feasible solution 2. x is a vertex of P 3. x is an extreme point of P Proof: Assume the LP is in the canonical form. 1. Vertex Extreme Point Let v be a vertex. Then for some objective function c, c T x is uniquely minimized at v. Assume v is not an extreme point. Then, v can be written as v = λy + (1 λ)z for some y, z neither of which is v, and some λ satisfying 0 λ 1. Now, c T v = c T [λy + (1 λ)z] = λc T y + (1 λ)c T z This means c T y c T v c T z. But, since v is a minimum point, c T v c T y and c T v c T z. Thus, c T y = c T v = c T z. This is a contradiction, since v is the unique point at which c T x is minimized. 2. Extreme Point Basic Feasible Solution Let x be an extreme point. By definition, it lies in the polyhedron and is therefore feasible. Assume x is not a basic solution. Let T be the set of rows of the constraint matrix A for which the constraints are tight at x. Let a i (a 1 n vector) denote the

3 Lecture 1: 10/13/ i th row of A. For a i / T, a i.x > b i. Since x is not a basic solution, T does not span R n. So, there is a vector d 0 such that a i.d = 0 a i T. Consider y = x + ǫd and z = x ǫd. If a i T, then a i.y = a i.z = b i. If a i / T, a then, by choosing a sufficiently small ǫ: 0 < ǫ min i.x b i i/ T a i.d, we can ensure that a i.y b i and a i.z b i. Thus y and z are feasible. Since x = y/2 + z/2, x cannot be an extreme point a contradiction. 3. Basic Feasible Solution Vertex Let x be a basic feasible solution. Let T = {i a i.x = b i }. Consider the objective as minimizing c.y for c = i T a i. Then, c.x = i T (a i.x) = i T b i. For any x P, c.x = i T (a i.x ) i T b i with equality only if a i.x = b i i T. This implies that x = x and that x uniquely minimizes the objective c.y. This proves that vertex, extreme point and basic feasible solution are equivalent terms. Theorem 2 Any bounded LP in standard form has an optimum at a basic feasible solution. Proof: Consider an optimal x which is not a basic feasible solution. Being optimal, it is feasible, hence it is not basic. As in the previous proof, let T be the set of rows of the constraint matrix A for which the constraints are tight at x. Since x is not a basic solution, T does not span R n. So, there is a vector d 0 such that a i.d = 0 a i T. For a scalar ǫ with sufficiently small absolute value, y = x+ǫd is feasible, and represents a line containing x in the direction d. The objective function at y is c T x+ǫc T d. Since x is optimal, c T d = 0, as otherwise, an ǫ of the opposite sign can reduce the objective. This means, all feasible points on this line are optimal. One of the directions of motion on this line will reduce some x i. Keep going till some x i reduces to 0. This results in one more tight constraint than before. This technique can be repeated, till the solution becomes basic. Thus, we can convert any feasible solution to a basic feasible solution of no worse value. In fact, this proof gives an algorithm for solving a linear program: evaluate the objective at all basic feasible solutions, and take the best one. Suppose there are m constraints and n variables. Since a set of n constraints defines a basic feasible solution, there can be upto ( m ) n basic feasible solutions. For each set of n constraints, a linear system of inequalities has to be solved, which by Gaussian elimination, takes O(n 3 ) time. This is in general an exponential complexity algorithm in n. Note that the output size is polynomial in n, since the optimal solution is just the solution of a system of linear equalities.

4 Lecture 1: 10/13/ The dual of a linear program Given an LP in the standard form: Minimize c.x subject to: Ax = b;x 0 We call the above LP the primal LP. The decision version of the problem is: Is the optimum c.x δ? This problem is in NP, because, if we find a feasible solution with optimum value δ, we can verify that it satisfies these requirements, in polynomial time. A more interesting question is whether this problem is in co-np. We need to find an easily verifiable proof for the fact that there is no x which satisfies c.x < δ. To do this, we require the concept of duality Weak Duality We seek a lower bound on the optimum. Consider a vector y (treat is as a row vector here). For any feasible x, yax = yb holds. If we require that ya c, then yb = yax cx. Thus, yb is a lower bound on cx, and in particular on the optimum cx. To get the best lower bound, we need to maximize yb. This new linear program: Maximize yb subject to: ya c is called the dual linear program. (Note: The dual of a dual program is the primal). Thus primal optimum is lower bounded by the dual optimum. This is called weak duality. Theorem 3 (Weak Duality) Consider the LP z = Min{c.x Ax = b,x 0} and its dual w = max{y.b ya c}. Then z w. Corollary 1 If the primal is feasible and unbounded, then the dual is infeasible. 1.3 Strong Duality In fact, if either the primal or the dual is feasible, then the two optima are equal to each other. This is known as strong duality. In this section, we first present an intuitive explanation of the theorem, using a gravitational model. The formal proof follows that.

5 Lecture 1: 10/13/ A gravitational model Consider the LP min{y.b ya c}. We represent this feasible region as a hollow polytope, with the vector b pointing upwards. If a ball is dropped into the polytope, it will settle down at the lowest point, which is the optimum of the above LP. Note that any minimum is a global minimum, since the feasible region of an LP is a convex polyhedron. At the equilibrium point, there is a balance of forces the gravitational force and the normal reaction of the floors (constraints). Let x i represent the amount of force exerted by the i th constraint. The direction of this force is given by the i th column of A. Then the total force exerted by all the constraints Ax balances the gravity b: Ax = b. The physical world also gives the constraints that x 0, since the floors force is always outwards. Only those floors which the ball touches exert a force. This means that for the constraints which are not tight, the corresponding x i s are zero: x i = 0 if ya i > c i. This can be summarized as (c i ya i )x i = 0. This means x and y satisfy: y.b = ya i x i = c i x i = c.x But weak duality says that yb cx, for every x and y. Hence the x and y are the optimal solutions of their respective LP s. This implies strong duality the optima of the primal and dual are equal A formal proof Theorem 4 (Strong Duality) Consider w = min{y.b ya c} and z = min{c.x Ax = b,x 0}. Then z = w. Proof: Consider the LP min{y.b ya c}. Consider the optimal solution y. Without loss of generality, ignore all the constraints that are loose for y. If there are any redundant constraints, drop them. Clearly, these changes cannot alter the optimal solution. Dropping these constraints leads to a new A with fewer columns and a new shorter c. We will prove that the dual of the new LP has an optimum equal in value to the primal. This dual optimal solution can be extended to an optimal solution of the dual of the original LP, by filling in zeros at places corresponding to the dropped constraints. The point is that we do not need those constraints to come up with the dual optimal solution. After dropping those constraints, at most n tight constraints remain (where n is the length of the vector y). Since we have removed all redundancy, these constraints are linearly independent. In terms of the new A and c, we have new constraints ya = c. y is still the optimum.

6 Lecture 1: 10/13/ Claim: There exists an x, such that Ax = b. Proof: Assume such an x does not exist, i.e. Ax = b is infeasible. Then duality for linear equalities implies that there exists a z such that za = 0, but zb 0. Without loss of generality, assume z.b < 0 (otherwise, just negate the z). Now consider (y + z). A(y + z) = Ay + Az = Ay. Hence, it is feasible. (y + z).b = y.b + z.b < y.b, which is better than the assumed optimum a contradiction. So, there is an x such that Ax = b. Let this be called x. Claim: y.b = c.x. Proof: y.b = y.(ax ) = (y A).x = c.x (since Ax = b and y A = c) Claim: x 0 Proof: Assume the contrary. Then, for some i, x i < 0. Let c = c + e i, where e i is all 0 s except at the i th position, where it has a 1. Since A has full rank, ya c has a solution, say y. Besides, since c c, y is feasible for the original constraints ya c. But, y.b = y Ax = c x < cx = y b (since c i is now higher and x i < 0). This means y gives a better objective value than the optimal solution a contradiction. Hence, x 0. Thus, there is an x which is feasible in the dual, and whose objective is equal to the primal optimum. Hence, x must be the dual optimal solution, using weak duality. Thus, the optima of primal and dual are equal. Corollary 2 Checking for feasibility of a linear system of inequalities and optimizing an LP are equally hard. Proof: Optimizer Feasibility checker Use the optimizer to optimize any arbitrary function with the linear system of inequalities as the constraints. This will automatically check for feasibility, since every optimal solution is feasible. Feasibility checker Optimizer We construct a reduction from the problem of finding an optimal solution of LP 1 to the problem of finding a feasible solution of LP 2. LP 1 is min{c.x Ax = b,x 0}. Consider LP 2 = min{0.x Ax = b,x 0,yA c,c.x = b.y}. Any feasible solution of LP 2 gives an optimal solution of LP 1 due to the strong duality theorem. Finding an optimal solution is thus no harder than finding a feasible solution.

7 Lecture 1: 10/13/ Rules for duals Usually the primal is constructed as a minimization problem and hence the dual becomes a maximization problem. For the standard form, the primal is given by: z = min (c T x) Ax b x 0 while the dual is given by: w = max (b T y) A T y c y 0 For a mixed form of the primal, the following describes the dual: Primal: A 11 x 1 + A 12 x 2 + A 13 x 3 = b 1 A 21 x 1 + A 22 x 2 + A 23 x 3 b 2 A 31 x 1 + A 32 x 2 + A 33 x 3 b 3 x 1 0 z = min c 1 x 1 + c 2 x 2 + c 3 x 3 x 2 0 x 3 UIS (UIS = unrestricted in sign) Dual: w = max y 1 b 1 + y 2 b 2 + y 3 b 3 y 1 A 11 + y 2 A 21 + y 3 A 31 c 1 y 1 A 12 + y 2 A 22 + y 3 A 32 c 2 y 1 A 13 + y 2 A 23 + y 3 A 33 = c 3

8 Lecture 1: 10/13/ y 1 y 2 0 y 3 0 UIS These rules are summarized in the following table. PRIMAL Minimize Maximize DUAL Constraints b i 0 Variables b i 0 = b i Free Variables 0 c j Constraints 0 c j Free = c j Each variable in the primal corresponds to a constraint in the dual, and vice versa. For a maximization, an upper bound constraint is a natural constraint, while for a minimization, a lower bound constraint is natural. If the constraint is in the natural direction, then the corresponding dual variable is non-negative. An interesting observation is that, the tighter the primal gets, the looser the dual gets. For instance, an equality constraint in the primal leads to an unrestricted variable in the dual. Adding more constraints in the primal leads to more variables in the dual, hence more flexibility. 1.5 Shortest Path an example Consider the problem of finding the shortest path in a graph. Given a graph G, we wish to find the shortest path from a specified source node, to all other nodes. This can be formulated as a linear program: w = max (d t d s ) s.t. d j d i c ij, i,j In this formulation, d i represents the distance of node i from the source node s. The c ij constraints are essentially the triangle inequalities the distance from the source to a node i should not be more than the distance to some node j plus the distance from j to

9 Lecture 1: 10/13/ i. Intuitively, one can imagine stretching the network physically, to increase the sourcedestination distance. When we cannot pull any further without breaking an edge, we have found a shortest path. The dual to this program is found thus. The constraint matrix in the primal has a row for every pair of nodes (i, j), and a column for every node. The row corresponding to (i, j) has a +1 in the i th column and a -1 in the j th column, and zeros elsewhere. 1. Using this, we conclude that the dual has a variable for each pair (i,j), say y ij. 2. It has a constraint for each node i. The constraint has a coefficient of +1 for each edge entering node i and a -1 for each edge leaving i. The right side for the constraints are -1 for the node s constraint, 1 for the node t constraint, and 0 for others, based on the objective function in the primal. Moreover, all the constraints are equality constraints, since the d i variables were unrestricted in sign in the primal. 3. The dual variables will have to have a non-negativity constraint as well, since the constraints in the primal were natural (upper bounds for a maximization). 4. The objective is to minimize i,j c ijy ij, since the right side of the primal constraints are c ij. Thus the dual is: z = min i,j c ij y ij (y js y sj ) = 1 j (y jt y tj ) = 1 j (y ji y ij ) = 0, i s,t j y ij 0, i,j This is precisely the linear program to solve the minimum cost unit flow, in a gross flow formulation. The constraints correspond to the flow conservation at all nodes except at the source and sink. The value of the flow is forced to be 1. Intuitively, this says that we can use minimum cost unit flow algorithms to find the shortest path in a network. Duality is a very useful concept, especially because it helps to view the optimization problem on hand from a different perspective, which might be easier to handle.

10 Lecture 17: 10/20/ Advanced Algorithms Scribes: Michael Schnall-Levin Lecturer: David Karger Complementary Slackness and More Dual Examples 17.1 Complementary Slackness We begin by looking at the notion of complementary slackness. Consider the following primal LP and its dual: Primal: min cx, Ax = b, x 0 Dual: max yb, ya c We can rewrite the dual using slack variables s to put it in the form: Dual: max yb, ya + s = c, s 0 Using this formulation, we arrive at the following lemma. Lemma 1 The following are all equivalent: (i) x, y are optimal (ii) s x = 0 (iii) x j s j = 0 j (iv) s j 0 x j = 0 Proof: First note that (iii) and (iv) are just restatements of (ii). Therefore we only need to show (i) and (ii) are equivalent. (i) (ii): x, y are both optimal if and only if cx = yb (by strong duality) and cx = yb can be rewritten as (ya + s)x = yax which holds if and only if s x = 0 The interpretation of this lemma is the following. At the optimum, it is not possible to have x j and s j both slack. At least one of them has to be at the limit. Conversely, if at least one of the two is tight for every j, then the point is an optimum. We will see how this notion of complementary slackness can be used to gain insight in our analysis of the following dual problems. 17-1

11 Lecture 17: 10/20/ Two Dual Examples Max Flow Consider the max flow problem on a graph G with source node s and sink node t. In order to formulate this problem as an LP, we augment G with an extra edge from t to s having infinite capacity. Then the LP is to maximize the flow along that edge while requiring the flow to be both feasible (obey the capacity constraints) and a circulation: Letting x u,v be the flow value on edge (u, v), we can then write the primal LP as: Primal: max x t,s subject to: w x v,w x w,v = 0 v (flow is a circulation) x v,w u v,w (v, w) (flow values do not exceed capacities) x v,w 0 (v, w) (flow values are all positive) What is the dual for this LP? We introduce a variable for every constraint in the primal problem: a variable z v for each of the constraints on each node (the circulation constraints) and a variable y v,w for each of the capacity constraints. Then by following the cookbook method for finding duals as described in previous lectures we get the following dual problem. Dual: min u v,w y v,w subject to: y v,w 0 z v z w + y v,w 0 (v, w) (t, s) z t z s + y t,s 1 We can first simplify this by noting that since u t,s =, an optimal solution to the dual must have y t,s = 0. Using this and moving terms in the constraints allows us to rewrite them as: y v,w 0 z w z v + y v,w (v, w) (t, s) z t z s 1 Using the above, we can interpret the dual problem in the following way. Consider the u v,w as cross section areas, and the y v,w as lengths. Then the problem is to minimize the total volume, while maintaining a distance of at least 1 between nodes s and t. As a sanity check for what we have done, let (S,T) be a min s-t cut on G. Set y v,w = 1 for all edges crossing the cut and y v,w = 0 for all other edges. This is a feasible solution to the dual problem and the objective function in the dual takes on value v S,w T u v,w which is just the value of the min-cut. Therefore, by weak duality the value of the primal is no greater than the value of the dual, and so we have shown that max-flow min-cut. Now, let s see what further insight we can gain by using the complementary slackness results proven earlier. First, note that for any solution of the dual, we can subtract the same value from all z w without changing the value of the objective function or breaking any of the constraints. Therefore, we can choose to have z s = 0. This is equivalent to having the z w represent distances from node s.

12 Lecture 17: 10/20/ Now consider an optimal solution to the dual problem (with the choice as above of z s = 0). Let S = {v z v < 1} and T = V S. Then S,T is an s-t cut. For any edge (v,w) crossing the cut from S to T, we have y v,w z w z v > 0. Therefore, for these edges, the variables y v,w are not 0. Since we are at an optimum point, by complementary slackness, the corresponding constraint for the primal problem must be tight. The constraint corresponding to the variable y v,w is x v,w u v,w. Therefore we must have that x v,w = u v,w for all edges (v,w) crossing the cut from S to T. Similarly, consider any edge (v,w) crossing the cut from T to S. From the definition of S and T, we know that z v > z w and since y v,w 0 we must have z v + y v,w > z v. Therefore, the constraint z v + y v,w z v is not tight and so the corresponding variables in the primal must be zero and so x v,w = 0. Therefore, the flow on all edges from S to T is at capacity, and the flow on all edges from T to S is 0, and so the flow on the graph is equal to the capacity of the cut given by (S,T). Therefore, using complementary slackness we have proven the max flow = min-cut theorem Min-Cost Circulation We can quickly find an LP for min-cost circulation by noting that the formulation is very similar to that for max-flow. In particular, none of the constraints need to be changed. The only difference is that we now want a circulation and we want to minimize cost, and so only the objective function needs to be changed. The primal LP then becomes: Primal: min c v,w x v,w subject to: w x v,w x w,v = 0 v (flow is a circulation) x v,w u v,w (v, w) (flow values do not exceed capacities) x v,w 0 (v, w) (flow values are all positive) Because only the objective function on the primal has changed, the only change in the dual problem is on the constraints and the sign of the objective. The dual LP becomes: Dual: max u v,w y v,w subject to: y v,w 0 z v z w + y v,w c v,w Let p v = z v. Then the dual problem can be rewritten as: Dual: max u v,w y v,w subject to: y v,w 0 y v,w c v,w + p v p w If we consider the p v to be prices, then the dual problem is to maximize the weighted sum of the arcs of negative reduced cost.

13 Lecture 17: 10/20/ Again, we use the results of complementary slackness to see what further insight we can gain. Suppose that x v,w < u v,w. Then the constraint x v,w u v,w isn t tight and so the corresponding variable in the dual y v,w = 0. If the reduced cost c p v,w < 0 then y v,w < 0 and so x v,w = u v,w. So negative reduced cost arcs must be saturated. Similarly, suppose c p v,w > 0. Then the constraint y v,w c v,w + p v p w must be slack (since y v,w 0) and so the corresponding variable in the primal x v,w = 0. Therefore, positive reduced cost arcs must have zero flow. Therefore, using complementary slackness we have proven that in a min-cost flow negative reduced cost arcs are saturated and positive reduced cost arcs have zero flow.

### 1 Linear Programming. 1.1 Introduction. Problem description: motivate by min-cost flow. bit of history. everything is LP. NP and conp. P breakthrough.

1 Linear Programming 1.1 Introduction Problem description: motivate by min-cost flow bit of history everything is LP NP and conp. P breakthrough. general form: variables constraints: linear equalities

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### 3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

### 5.1 Bipartite Matching

CS787: Advanced Algorithms Lecture 5: Applications of Network Flow In the last lecture, we looked at the problem of finding the maximum flow in a graph, and how it can be efficiently solved using the Ford-Fulkerson

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

### Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

### 1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS. 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method

LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method Introduction to dual linear program Given a constraint matrix A, right

### Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

### 26 Linear Programming

The greatest flood has the soonest ebb; the sorest tempest the most sudden calm; the hottest love the coldest end; and from the deepest desire oftentimes ensues the deadliest hate. Th extremes of glory

### Approximation Algorithms

Approximation Algorithms or: How I Learned to Stop Worrying and Deal with NP-Completeness Ong Jit Sheng, Jonathan (A0073924B) March, 2012 Overview Key Results (I) General techniques: Greedy algorithms

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### Arrangements And Duality

Arrangements And Duality 3.1 Introduction 3 Point configurations are tbe most basic structure we study in computational geometry. But what about configurations of more complicated shapes? For example,

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### 1 Solving LPs: The Simplex Algorithm of George Dantzig

Solving LPs: The Simplex Algorithm of George Dantzig. Simplex Pivoting: Dictionary Format We illustrate a general solution procedure, called the simplex algorithm, by implementing it on a very simple example.

### What is Linear Programming?

Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to

### Minimum cost maximum flow, Minimum cost circulation, Cost/Capacity scaling

6.854 Advanced Algorithms Lecture 16: 10/11/2006 Lecturer: David Karger Scribe: Kermin Fleming and Chris Crutchfield, based on notes by Wendy Chu and Tudor Leu Minimum cost maximum flow, Minimum cost circulation,

### Mathematical finance and linear programming (optimization)

Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may

### International Doctoral School Algorithmic Decision Theory: MCDA and MOO

International Doctoral School Algorithmic Decision Theory: MCDA and MOO Lecture 2: Multiobjective Linear Programming Department of Engineering Science, The University of Auckland, New Zealand Laboratoire

### Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

### THE DIMENSION OF A VECTOR SPACE

THE DIMENSION OF A VECTOR SPACE KEITH CONRAD This handout is a supplementary discussion leading up to the definition of dimension and some of its basic properties. Let V be a vector space over a field

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

### Lecture 2: August 29. Linear Programming (part I)

10-725: Convex Optimization Fall 2013 Lecture 2: August 29 Lecturer: Barnabás Póczos Scribes: Samrachana Adhikari, Mattia Ciollaro, Fabrizio Lecci Note: LaTeX template courtesy of UC Berkeley EECS dept.

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. !-approximation algorithm.

Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of

### Linear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.

Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.

### Linear Programming Problems

Linear Programming Problems Linear programming problems come up in many applications. In a linear programming problem, we have a function, called the objective function, which depends linearly on a number

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. #-approximation algorithm.

Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of three

### Duality in Linear Programming

Duality in Linear Programming 4 In the preceding chapter on sensitivity analysis, we saw that the shadow-price interpretation of the optimal simplex multipliers is a very useful concept. First, these shadow

### Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

### Linear Programming. Solving LP Models Using MS Excel, 18

SUPPLEMENT TO CHAPTER SIX Linear Programming SUPPLEMENT OUTLINE Introduction, 2 Linear Programming Models, 2 Model Formulation, 4 Graphical Linear Programming, 5 Outline of Graphical Procedure, 5 Plotting

### Optimization in R n Introduction

Optimization in R n Introduction Rudi Pendavingh Eindhoven Technical University Optimization in R n, lecture Rudi Pendavingh (TUE) Optimization in R n Introduction ORN / 4 Some optimization problems designing

### 3 Does the Simplex Algorithm Work?

Does the Simplex Algorithm Work? In this section we carefully examine the simplex algorithm introduced in the previous chapter. Our goal is to either prove that it works, or to determine those circumstances

### OPRE 6201 : 2. Simplex Method

OPRE 6201 : 2. Simplex Method 1 The Graphical Method: An Example Consider the following linear program: Max 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix.

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. Nullspace Let A = (a ij ) be an m n matrix. Definition. The nullspace of the matrix A, denoted N(A), is the set of all n-dimensional column

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### Lecture 5: Conic Optimization: Overview

EE 227A: Conve Optimization and Applications January 31, 2012 Lecture 5: Conic Optimization: Overview Lecturer: Laurent El Ghaoui Reading assignment: Chapter 4 of BV. Sections 3.1-3.6 of WTB. 5.1 Linear

### The Goldberg Rao Algorithm for the Maximum Flow Problem

The Goldberg Rao Algorithm for the Maximum Flow Problem COS 528 class notes October 18, 2006 Scribe: Dávid Papp Main idea: use of the blocking flow paradigm to achieve essentially O(min{m 2/3, n 1/2 }

### princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora

princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora Scribe: One of the running themes in this course is the notion of

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### Linear Programming in Matrix Form

Linear Programming in Matrix Form Appendix B We first introduce matrix concepts in linear programming by developing a variation of the simplex method called the revised simplex method. This algorithm,

### Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

### Math 4310 Handout - Quotient Vector Spaces

Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable

### Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

### 1 Homework 1. [p 0 q i+j +... + p i 1 q j+1 ] + [p i q j ] + [p i+1 q j 1 +... + p i+j q 0 ]

1 Homework 1 (1) Prove the ideal (3,x) is a maximal ideal in Z[x]. SOLUTION: Suppose we expand this ideal by including another generator polynomial, P / (3, x). Write P = n + x Q with n an integer not

### Linear programming and reductions

Chapter 7 Linear programming and reductions Many of the problems for which we want algorithms are optimization tasks: the shortest path, the cheapest spanning tree, the longest increasing subsequence,

### CHAPTER 9. Integer Programming

CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral

### Fairness in Routing and Load Balancing

Fairness in Routing and Load Balancing Jon Kleinberg Yuval Rabani Éva Tardos Abstract We consider the issue of network routing subject to explicit fairness conditions. The optimization of fairness criteria

### Linear Programming Notes VII Sensitivity Analysis

Linear Programming Notes VII Sensitivity Analysis 1 Introduction When you use a mathematical model to describe reality you must make approximations. The world is more complicated than the kinds of optimization

### Actually Doing It! 6. Prove that the regular unit cube (say 1cm=unit) of sufficiently high dimension can fit inside it the whole city of New York.

1: 1. Compute a random 4-dimensional polytope P as the convex hull of 10 random points using rand sphere(4,10). Run VISUAL to see a Schlegel diagram. How many 3-dimensional polytopes do you see? How many

### Transportation Polytopes: a Twenty year Update

Transportation Polytopes: a Twenty year Update Jesús Antonio De Loera University of California, Davis Based on various papers joint with R. Hemmecke, E.Kim, F. Liu, U. Rothblum, F. Santos, S. Onn, R. Yoshida,

### 2.3 Scheduling jobs on identical parallel machines

2.3 Scheduling jobs on identical parallel machines There are jobs to be processed, and there are identical machines (running in parallel) to which each job may be assigned Each job = 1,,, must be processed

### Linear Programming. April 12, 2005

Linear Programming April 1, 005 Parts of this were adapted from Chapter 9 of i Introduction to Algorithms (Second Edition) /i by Cormen, Leiserson, Rivest and Stein. 1 What is linear programming? The first

### A Labeling Algorithm for the Maximum-Flow Network Problem

A Labeling Algorithm for the Maximum-Flow Network Problem Appendix C Network-flow problems can be solved by several methods. In Chapter 8 we introduced this topic by exploring the special structure of

### 8.1 Min Degree Spanning Tree

CS880: Approximations Algorithms Scribe: Siddharth Barman Lecturer: Shuchi Chawla Topic: Min Degree Spanning Tree Date: 02/15/07 In this lecture we give a local search based algorithm for the Min Degree

### Permutation Betting Markets: Singleton Betting with Extra Information

Permutation Betting Markets: Singleton Betting with Extra Information Mohammad Ghodsi Sharif University of Technology ghodsi@sharif.edu Hamid Mahini Sharif University of Technology mahini@ce.sharif.edu

### Linear Codes. In the V[n,q] setting, the terms word and vector are interchangeable.

Linear Codes Linear Codes In the V[n,q] setting, an important class of codes are the linear codes, these codes are the ones whose code words form a sub-vector space of V[n,q]. If the subspace of V[n,q]

### Linear Maps. Isaiah Lankham, Bruno Nachtergaele, Anne Schilling (February 5, 2007)

MAT067 University of California, Davis Winter 2007 Linear Maps Isaiah Lankham, Bruno Nachtergaele, Anne Schilling (February 5, 2007) As we have discussed in the lecture on What is Linear Algebra? one of

### 9.4 THE SIMPLEX METHOD: MINIMIZATION

SECTION 9 THE SIMPLEX METHOD: MINIMIZATION 59 The accounting firm in Exercise raises its charge for an audit to \$5 What number of audits and tax returns will bring in a maximum revenue? In the simplex

### We call this set an n-dimensional parallelogram (with one vertex 0). We also refer to the vectors x 1,..., x n as the edges of P.

Volumes of parallelograms 1 Chapter 8 Volumes of parallelograms In the present short chapter we are going to discuss the elementary geometrical objects which we call parallelograms. These are going to

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J. Olver 5. Inner Products and Norms The norm of a vector is a measure of its size. Besides the familiar Euclidean norm based on the dot product, there are a number

### T ( a i x i ) = a i T (x i ).

Chapter 2 Defn 1. (p. 65) Let V and W be vector spaces (over F ). We call a function T : V W a linear transformation form V to W if, for all x, y V and c F, we have (a) T (x + y) = T (x) + T (y) and (b)

### 3. Reaction Diffusion Equations Consider the following ODE model for population growth

3. Reaction Diffusion Equations Consider the following ODE model for population growth u t a u t u t, u 0 u 0 where u t denotes the population size at time t, and a u plays the role of the population dependent

### 1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

### Methods for Finding Bases

Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,

### Minimizing costs for transport buyers using integer programming and column generation. Eser Esirgen

MASTER STHESIS Minimizing costs for transport buyers using integer programming and column generation Eser Esirgen DepartmentofMathematicalSciences CHALMERS UNIVERSITY OF TECHNOLOGY UNIVERSITY OF GOTHENBURG

### IEOR 4404 Homework #2 Intro OR: Deterministic Models February 14, 2011 Prof. Jay Sethuraman Page 1 of 5. Homework #2

IEOR 4404 Homework # Intro OR: Deterministic Models February 14, 011 Prof. Jay Sethuraman Page 1 of 5 Homework #.1 (a) What is the optimal solution of this problem? Let us consider that x 1, x and x 3

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### Optimization Modeling for Mining Engineers

Optimization Modeling for Mining Engineers Alexandra M. Newman Division of Economics and Business Slide 1 Colorado School of Mines Seminar Outline Linear Programming Integer Linear Programming Slide 2

### Can linear programs solve NP-hard problems?

Can linear programs solve NP-hard problems? p. 1/9 Can linear programs solve NP-hard problems? Ronald de Wolf Linear programs Can linear programs solve NP-hard problems? p. 2/9 Can linear programs solve

### Vector Spaces II: Finite Dimensional Linear Algebra 1

John Nachbar September 2, 2014 Vector Spaces II: Finite Dimensional Linear Algebra 1 1 Definitions and Basic Theorems. For basic properties and notation for R N, see the notes Vector Spaces I. Definition

### Solutions to Math 51 First Exam January 29, 2015

Solutions to Math 5 First Exam January 29, 25. ( points) (a) Complete the following sentence: A set of vectors {v,..., v k } is defined to be linearly dependent if (2 points) there exist c,... c k R, not

### CS 598CSC: Combinatorial Optimization Lecture date: 2/4/2010

CS 598CSC: Combinatorial Optimization Lecture date: /4/010 Instructor: Chandra Chekuri Scribe: David Morrison Gomory-Hu Trees (The work in this section closely follows [3]) Let G = (V, E) be an undirected

### 160 CHAPTER 4. VECTOR SPACES

160 CHAPTER 4. VECTOR SPACES 4. Rank and Nullity In this section, we look at relationships between the row space, column space, null space of a matrix and its transpose. We will derive fundamental results

### LEARNING OBJECTIVES FOR THIS CHAPTER

CHAPTER 2 American mathematician Paul Halmos (1916 2006), who in 1942 published the first modern linear algebra book. The title of Halmos s book was the same as the title of this chapter. Finite-Dimensional

### Introduction to Algorithms Review information for Prelim 1 CS 4820, Spring 2010 Distributed Wednesday, February 24

Introduction to Algorithms Review information for Prelim 1 CS 4820, Spring 2010 Distributed Wednesday, February 24 The final exam will cover seven topics. 1. greedy algorithms 2. divide-and-conquer algorithms

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### 5 Homogeneous systems

5 Homogeneous systems Definition: A homogeneous (ho-mo-jeen -i-us) system of linear algebraic equations is one in which all the numbers on the right hand side are equal to : a x +... + a n x n =.. a m

### The Graphical Method: An Example

The Graphical Method: An Example Consider the following linear program: Maximize 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2 0, where, for ease of reference,

### 4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns

L. Vandenberghe EE133A (Spring 2016) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows

### Answers to some of the exercises.

Answers to some of the exercises. Chapter 2. Ex.2.1 (a) There are several ways to do this. Here is one possibility. The idea is to apply the k-center algorithm first to D and then for each center in D

### Diagonalisation. Chapter 3. Introduction. Eigenvalues and eigenvectors. Reading. Definitions

Chapter 3 Diagonalisation Eigenvalues and eigenvectors, diagonalisation of a matrix, orthogonal diagonalisation fo symmetric matrices Reading As in the previous chapter, there is no specific essential

### This exposition of linear programming

Linear Programming and the Simplex Method David Gale This exposition of linear programming and the simplex method is intended as a companion piece to the article in this issue on the life and work of George

### An Introduction to Linear Programming

An Introduction to Linear Programming Steven J. Miller March 31, 2007 Mathematics Department Brown University 151 Thayer Street Providence, RI 02912 Abstract We describe Linear Programming, an important

### LECTURE: INTRO TO LINEAR PROGRAMMING AND THE SIMPLEX METHOD, KEVIN ROSS MARCH 31, 2005

LECTURE: INTRO TO LINEAR PROGRAMMING AND THE SIMPLEX METHOD, KEVIN ROSS MARCH 31, 2005 DAVID L. BERNICK dbernick@soe.ucsc.edu 1. Overview Typical Linear Programming problems Standard form and converting

### The Market-Clearing Model

Chapter 5 The Market-Clearing Model Most of the models that we use in this book build on two common assumptions. First, we assume that there exist markets for all goods present in the economy, and that

### . P. 4.3 Basic feasible solutions and vertices of polyhedra. x 1. x 2

4. Basic feasible solutions and vertices of polyhedra Due to the fundamental theorem of Linear Programming, to solve any LP it suffices to consider the vertices (finitely many) of the polyhedron P of the

### a = bq + r where 0 r < b.

Lecture 5: Euclid s algorithm Introduction The fundamental arithmetic operations are addition, subtraction, multiplication and division. But there is a fifth operation which I would argue is just as fundamental

### Solving Linear Programs

Solving Linear Programs 2 In this chapter, we present a systematic procedure for solving linear programs. This procedure, called the simplex method, proceeds by moving from one feasible solution to another,

### 56:171 Operations Research Midterm Exam Solutions Fall 2001

56:171 Operations Research Midterm Exam Solutions Fall 2001 True/False: Indicate by "+" or "o" whether each statement is "true" or "false", respectively: o_ 1. If a primal LP constraint is slack at the