M AT H E M AT I C S O F O P - E R AT I O N S R E S E A R C H

Transcription

1 A N D R E W T U L L O C H M AT H E M AT I C S O F O P - E R AT I O N S R E S E A R C H T R I N I T Y C O L L E G E T H E U N I V E R S I T Y O F C A M B R I D G E

2

3 Contents 1 Generalities 5 2 Constrained Optimization Lagrangian Multipliers Lagrange Dual Supporting Hyperplanes 8 3 Linear Programming Convexity and Strong Duality Linear Programs Linear Program Duality Complementary Slackness 13 4 Simplex Method Basic Solutions Extreme Points and Optimal Solutions The Simplex Tableau The Simplex Method in Tableau Form 16

4 4 andrew tulloch 5 Advanced Simplex Procedures The Two-Phase Simplex Method Gomory s Cutting Plane Method 17 6 Complexity of Problems and Algorithms Asymptotic Complexity 19 7 The Complexity of Linear Programming A Lower Bound for the Simplex Method The Idea for a New Method 21 8 Graphs and Flows Introduction Minimal Cost Flows 23 9 Transportation and Assignment Problems Non-Cooperative Games Strategic Equilibrium Bibliography 31

5 1 Generalities

6

7 2 Constrained Optimization Minimize f (x) subject to h(x) = b, x X. Objective function f : R n R Vector x R n of decision variables, Functional constraint where h : R n > R m, b R m Regional constraint where X R n. Definition 2.1. The feasible set is X(b) = {x X : h(x) = b}. An inequality of the form g(x) b can be written as g(x) + z = b, where z R m called a slack variable with regional constraint z Lagrangian Multipliers Definition 2.2. Define the Lagrangian of a problem as L(x, λ) = f (x) λ T (h(x) b) (2.1) where λ R m is a vector of Lagrange multipliers Theorem 2.3 (Lagrangian Sufficiency Theorem). Let x X and λ R m such that and h(x) = b. Then x is optimal for P. L(x, λ) = inf x X L(x, λ) (2.2)

8 8 andrew tulloch Proof. min f x X(b) (x ) = min [ f (x) x X(b) λt (h(x ) b)] (2.3) min x X [ f (x ) λ T (h(x ) b)] (2.4) = f (x) λ T (h(x) b) (2.5) = f (x) (2.6) 2.2 Lagrange Dual Definition 2.4. Let φ(b) = inf f (x). (2.7) x X(b) Define the Lagrange dual function g : R m R with g(λ) = in f x X L(x, λ) (2.8) Then, for all λ R m, inf f (x) = inf L(x, λ) inf L(x, λ) = g(λ) (2.9) x X(b) x X(b) x X That is, g(λ) is a lower bound on our optimization function. This motivates the dual problem to maximize g(λ) subject to λ Y, where Y = {λ R m : g(λ) > }. Theorem 2.5 (Duality). From (2.9), we see that the optimal value of the primal is always greater than the optimal value of the dual. This is weak duality. 2.3 Supporting Hyperplanes Fix b R m and consider φ as a function of c R m. Further consider the hyperplane given by α : R m R with α(c) = β λ T (b c) (2.10) Now, try to find φ(b) as follow.

9 mathematics of operations research 9 (i) For each λ, find β λ = max{β : α(c) φ(c), c R m } (2.11) (ii) Choose λ to maximize β λ Definition 2.6. Call α : R m R a supporting hyperplane to φ at b if α(c) = φ(b) λ T (b c) (2.12) and φ(c) φ(b) λ T (b c) (2.13) for all c R m. Theorem 2.7. The following are equivalent (i) There exists a (non-vertical) supporting hyperplane to φ at b, (ii) XXXXXXXXXXXXXXXXXXXXXXXXXX

10

11 3 Linear Programming 3.1 Convexity and Strong Duality Definition 3.1. Let S R n. S is convex if for all δ [0, 1], x, y S implies that δx + (1 δ)y S. f : S R is convex if for all x, y S and δ [0, 1], δ f (x) + (1 δ) f (y) f (δx + (1 δ)y. Visually, the area under the function is a convex set. Definition 3.2. x S is an interior point of S if there exists ɛ > 0 such {y : y x 2 ɛ} S. x S is an extreme point of S if for all y, z S and S (0, 1), x = δy + (1 δ)z implies x = y = z. Theorem 3.3 (Supporting Hyperplane). Suppose that our function φ is convex and b lies in the interior of the set of points where φ is finite. Then there is a (non-vertical) supporting hyperplane to φ at b. Theorem 3.4. Let X(b) = {x X : h(x) b}, φ(b) = inf x X(b) f (x). Then φ is convex if X, f, and h are convex. Proof. Let b 1, b 2 R m such that φ(b 1 ), φ(b 2 ) are defined. Let δ [0, 1] and b = δb 1 + (1 δ)b 2. Consider x 1 X(b 1 ), x 2 X(b 2 ) and let x = δx 1 + (1 δ)x 2.

12 12 andrew tulloch By convexity of Y, x X. By convexity of h, h(x) = h(δx 1 + (1 δ)x 2 ) δh(x 1 ) + (1 δ)h(x 2 ) δb 1 + (1 δ)b 2 = b Thus x X(b), and by convexity of f, φ(b) f (x) = f (δx 1 + (1 δ)x 2 ) δ f (x 1 ) + (1 δ) f (x 2 ) δφ(b 1 ) + (1 δ)φ(b 2 ) 3.2 Linear Programs Definition 3.5. General form of a linear program is min{c T x : Ax b, x 0} (3.1) Standard form of a linear program is min{c T x : Ax = b, x 0} (3.2) 3.3 Linear Program Duality Introduce slack variables to turn problem P into the form min{c T x : Ax z = b, x, z 0} (3.3) We have X = {(x, z) : x, z 0} R m+n. The Lagrangian is L((x, z), λ) = c T x λ T (Ax z b) (3.4) = (c T λ T A)x + λ T z + λ T b (3.5)

13 mathematics of operations research 13 and has a finite minimum if and only if λ Y = {λ : c T λ T A 0, λ 0} (3.6) For a fixed λ Y, the minimum of L is satisfied when (c T λ T A)x = 0 and λ T z = 0, and thus g(λ) = We obtain that the dual problem inf L((x, z), λ) = λ T b (3.7) (x,z) X max{λ T b : A T λ c, λ 0} (3.8) and it can be shown (exercise) that the dual of the dual of a linear program is the original program. 3.4 Complementary Slackness Theorem 3.6. Let x and λ be feasible for P and its dual. Then x and λ are optimal if and only if (c T λ T A)x = 0 (3.9) λ T (Ax b) = 0 (3.10) Proof. If x, λ are optimal, then c T x = λ T b }{{} by strong duality (3.11) = inf x X (ct x λ T (Ax b)) c T x λ T (Ax b) }{{} c T x primal and dual optimality Then the inequalities must be equalities. Thus (3.12) λ T b = c T x λ T (Ax b) = (c T λ T A)x +λ T b (3.13) }{{} =0 and c T x λ T (Ax b) = c T x (3.14) }{{} =0

14 14 andrew tulloch If (c T λ T A)x = 0 and λ T (Ax b) = 0, then c T x = c T λ T (Ax b) = (c T λ T a)x + λ T b = λ T b (3.15) and so by weak duality, x and λ are optimal.

15 4 Simplex Method 4.1 Basic Solutions Maximize c T x subject to Ax = b, x 0, A R m n. Call a solution x R n of Ax = b basic if it has at most m non-zero entries, that is, there exists B {1,..., n} with B = m and x i = 0 if i / B. A basic solution x with x 0 is called a basic feasible solution (bfs). 4.2 Extreme Points and Optimal Solutions We make the following assumptions: (i) The rows of A are linearly independent (ii) Every set of m columns of A are linearly independent. (iii) Every basic solution is non-degenerate - that is, it has exactly m non-zero entries. Theorem 4.1. x is a bfs of Ax = b if and only if it is an extreme point of the set X(b) = {x : Ax = b, x 0}. Theorem 4.2. If the problem has a finite optimum (feasible and bounded), then it has an optimal solution that is a bfs.

16 16 andrew tulloch 4.3 The Simplex Tableau Let A R m n, b R m. Let B be a basis (in the bfs sense), and x R n, such that Ax = b. Then A B x B + A N x N = b (4.1) where A B R m m and A N R m (n m) respectively consist of the columns of A indexed by B and those not indexed by B. Moreover, if x is a basic solution, then there is a basic B such that x N = 0 and A B x B = b, and if x is a basic feasible solution, there is a basis B such that x n = 0, A B x B = b, and x B 0. For every x with Ax = b and every basis B, we have x B = A 1 B (b A Nx N ) (4.2) as we assume that A B has full rank. Thus, f (x) = c T x = c T B x B + c T N x N (4.3) = c T B A 1 B (b A NX N ) + c T N x N (4.4) = CB T A 1 B b + (ct N ct B A 1 B A N)x N (4.5) Assume we can guarantee that A 1 B and x N = 0 is a bfs with b = 0. Then x with x B = A 1 B b f (x ) = C T B A 1 B b (4.6) Assume that we are maximizing c T x. There are two different cases: (i) If C T N CT B A 1 B x is optimal. A N 0, then f (x) c T B A 1 B b for every feasible x, so (ii) If (c T N ct B A 1 B A N) i > 0, then we can increase the objective value by increasing the corresponding row of (x N ) i. 4.4 The Simplex Method in Tableau Form

17 5 Advanced Simplex Procedures 5.1 The Two-Phase Simplex Method Finding an initial bfs is easy if the constraints have the form Ax = b where b 0, as Ax + z = b, (x, z) = (0, b) (5.1) 5.2 Gomory s Cutting Plane Method Used in integer programming (IP). This is a linear program where in addition some of the variables are required to be integral. Assume that for a given integer program we have found an optimal fractional solution x with basis B and let a ij = (A 1 B A j) and a i0 = (A 1 B b) be the entries of the final tableau. If x is not integral, the for some row i, a i0 is not integral. For every feasible solution x, x i = a ij x j x i + a ij x j = a i0. (5.2) j N j N If x is integral, then the left hand side is integral as well, and the inequality must still hold if the right hand side is rounded down. Thus, x i + a ij x j a i0. (5.3) j N Then, we can add this constraint to the problem and solve the augmented program. One can show that this procedure converges in a finite number of steps.

18

19 6 Complexity of Problems and Algorithms 6.1 Asymptotic Complexity We measure complexity as a function of input size. The input of a linear programming problem: c R n, A R m n, b R m is represented in (n + m n + m) k bits if we represent each number using k bits. For two functions f : R R and g : R R write f (n) = O(g(n)) (6.1) if there exists c, n 0 such that for all n n 0, f (n) c g(n) (6.2),... (similarly for Ω, and Θ (Ω + O))

20

21 7 The Complexity of Linear Programming 7.1 A Lower Bound for the Simplex Method Theorem 7.1. There exists a LP of size O(n 2 ), a pivoting rule, and an initial BFS such that the simplex method requires 2 n 1 iterations. Proof. Consider the unit cube in R n, given by constraints 0 x i 1 for i = 1,..., n. Define a spanning path inductively as follows. In dimension 1, go from x 1 = 0 to x 1 = 1. In dimension k, set x k = 0 and follow the path for dimension 1,..., k 1. Then set x = 1, and follow the path for dimension 1,..., k 1 backwards. The objective x n currently increases only once. Instead consider the perturbed unit cube given by the constraints ɛ x 1 1, ɛx i 1 x i 1 ɛx i 1 with ɛ (0, 1 2 ). 7.2 The Idea for a New Method min{c T x : Ax = b, x 0} (7.1) max{b T λ : A T λ c} (7.2) By strong duality, each of these problems has a bounded optimal

22 22 andrew tulloch solution if and only if the following set of constraints is feasible: c T x = b T λ (7.3) Ax = b (7.4) x 0 (7.5) A T λ c (7.6) It is thus enough to decide, for a given A R m n and b R m, whether {x R n : Ax b} =. Definition 7.2. A symmetric matrix D R n n is called positive definite if x T Dx > 0 for every x R n. Alternatively, all eigenvalues of the matrix are strictly positive. Definition 7.3. A set E R n given by E = E(z, D) = {x R n : (x z) T D(x z) 1} (7.7) for a positive definite symmetric D R n n and z R n is called an ellipsoid with center z. Let P = {x R n : Ax b} for some A R m n and b R m. To decide whether P =, we generate a sequence {E t } of ellipsoids E t with centers x t. If x t P, then P is non-empty and the method stops. If x t / P, then one of the constraints is violated - so there exists a row j of A such that a T j x t < b j. Therefore, P is contained in the half-space {x R n : a T j x at j x t, and in particular the intersection of this half-space with E t, which we will call a half-ellipsoid. Theorem 7.4. Let E = E(z, D) be an ellipsoid in R n and a R n = 0. Consider the half-space H = {x R n a T x a T z}, and let z = z + 1 n + 1 D = n2 n 2 1 Da a T Da ( D 2 n + 1 Daa T ) D a T Da (7.8) (7.9) Then D is symmetric and positive definite, and therefore E = E(z, D ) is an ellipsoid. Moreover, E H E and Vol(E ) < e 2(n+1) 1 Vol(E).

23 8 Graphs and Flows 8.1 Introduction Consider a directed graph (network) G = (V, E), Vthe set of vertices, E V V a set of edges. Undirected if E is symmetric. 8.2 Minimal Cost Flows Fill in this stuff from lectures

24

25 9 Transportation and Assignment Problems

26

27 10 Non-Cooperative Games Theorem 10.1 (von Neumann, 1928). Let P R m n. Then max min x X y Y p(x, y) = min max y Y x X p(x, y) (10.1)

28

29 11 Strategic Equilibrium Definition x X is a best response to y Y if p(x, y) = max x X p(x, y). (x, y) X Y is a Nash equilibrium if x is a best response to y and y is a best response to x. Theorem (x, y) X Y is an equilibrium of the matrix game P if and only if min p(x, y ) = max min p(x, y ) (11.1) y Y x X y Y and max p(x, y) = min max p(x, y ). (11.2) x X y Y x X Theorem Let (x, y), (x, y ) X Y be equilibria of the matrix game with payoff matrix P. Then p(x, y) = p(x, y ) and (x, y ) and (x, y) are equilibria as well. Theorem 11.4 (Nash, 1951). Every bimatrix game has an equilibrium.

30

31 12 Bibliography