Linear & Quadratic Programming

Size: px

Start display at page:

Download "Linear & Quadratic Programming"

Alexina Lane
7 years ago
Views:

1 Linear & Quadratic Programming Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010

2 Outline Linear programming Norm minimization problems Dual linear programming Algorithms Quadratic constrained quadratic programming (QCQP) Least-squares Second order cone programming (SOCP) Dual quadratic programming Acknowledgement: Thanks to Mung Chiang (Princeton), Stephen Boyd (Stanford) and Steven Low (Caltech) for the course materials in this class. 1

3 Linear Programming Minimize linear function over linear inequality and equality constraints: c T x Gx h Ax = b Variables: x R n. Standard form LP: c T x Ax = b x 0 Most well-known, widely-used and efficiently-solvable optimization Appreciation-Application cycle starting for convex optimization 2

4 Transformation To Standard Form Introduce slack variables s i for inequality constraints: c T x Gx + s = h Ax = b s 0 Express x as difference between two nonnegative variables x +, x 0: x = x + x c T x + c T x Gx + Gx + s = h Ax + Ax = b x +, x, s 0 Now in LP standard form with variables x +, x, s 3

5 Linear Fractional Programming Minimize ratio of affine functions over polyhedron: c T x+d e T x+f Gx h Ax = b Domain of objective function: {x e T x + f > 0} Not an LP. But if nonempty feasible set, transformation into an equivalent LP with variables y, z: c T y + dz Gy hz 0 Ay bz = 0 e T y + fz = 1 z 0 Why: let y = x and z = 1 e T x+f e T x+f Charnes-Cooper Trick 4

6 Norm Minimization Problems l 1 norm: x 1 = n i=1 x i Minimize Ax b 1 is equivalent to this LP in x R n, s R n : 1 T s Ax b s Ax b s l norm: x = max i { x i } Minimize Ax b is equivalent to this LP in x R n, t R: t Ax b t1 Ax b t1 5

7 Dual Linear Programming 1. Primal problem in standard form: c T x Ax = b x 0 2. Write down Lagrangian using Lagrange multipliers λ, ν: L(x, λ, ν) = c T x n i=1 λ i x i +ν T (Ax b) = b T ν +(c+a T ν λ) T x 3. Find Lagrange dual function: g(λ, ν) = inf x L(x, λ, ν) = bt ν + inf x [(c + AT ν λ) T x] 6

8 Since a linear function is bounded below only if it is identically zero, we have g(λ, ν) = { b T ν A T ν λ + c = 0 otherwise. 7

9 Dual Linear Programming 4. Write down Lagrange dual problem: maximize g(λ, ν) = λ 0 { b T ν A T ν λ + c = 0 otherwise 5. Make equality constraints explicit: maximize b T ν A T ν λ + c = 0 λ 0 8

10 6. Simplify Lagrange dual problem: maximize b T ν A T ν + c 0 which is an inequality constrained LP 9

11 Basic Properties Definition: x in polyhedron P is an extreme point if there does not exist two other points y, z P such that x = θy + (1 θ)z for some θ [0, 1] Theorem: Assume that a LP in standard form is feasible and the optimal objective value is finite. There exists an optimal solution which is an extreme point P x c 10

12 Algorithms Simplex Method Interior-point Method Ellipsoid Method Cutting-plane Method Simplex method is very efficient in practice but specialized for LP: move from one vertex to another without enumerating all the vertices Interior point algorithms are fierce competitors of Simplex since

13 Convex QCQP (Convex) QP (with linear constraints) in x: (1/2)x T P x + q T x + r Gx h Ax = b where P S n +, G R m n, A R p n (Convex) QCQP in x: (1/2)x T P 0 x + q0 T x + r 0 (1/2)x T P i x + qi T x + r i 0, i = 1, 2,..., m Ax = b 12

14 where P S n +, i = 0,..., m f 0 (x ) x P 13

15 Least-squares Minimize Ax b 2 2 = x T A T Ax 2b T Ax + b T b over x. Unconstrained QP, Regression analysis, Least-squares approximation Analytic solution: x = A b where, for A R m n, A = (A T A) 1 A T if rank of A is n, and A = A T (AA T ) 1 if rank of A is m. If not full rank, then by singular value decomposition. Constrained least-squares (no general analytic solution). example: For Ax b 2 2 l i x i u i, i = 1,..., n 14

16 LP with Random Cost c T x Gx h Ax = b Cost c R n is random, with mean c and covariance Ω Expected cost: c T x. Cost variance x T Ωx Minimize both expected cost and cost variance (with a weight γ): c T x + γx T Ωx Gx h Ax = b 15

17 SOCP Second Order Cone Programming: f T x A i x + b i 2 c T i x + d i, i = 1,..., m F x = g Variables: x R n. And A i R n i n, F R p n If c i = 0, i, SOCP is equivalent to QCQP If A i = 0, i, SOCP is equivalent to LP 16

18 Robust LP Consider inequality constrained LP: c T x a T i x b i, i = 1,..., m Parameters a i are not accurate. They are only known to lie in given ellipsoids described by ā i and P i R n n : a i E i = {ā i + P i u u 2 1} Since sup{a T i x a i E} = ā T i x + P T i x 2, Robust LP (satisfy constraints for all possible a i ) formulated as 17

19 SOCP: c T x ā T i x + P i T x 2 b i, i = 1,..., m 18

20 Dual QCQP Primal (convex) QCQP (1/2)x T P 0 x + q0 T x + r 0 (1/2)x T P i x + qi T x + r i 0, i = 1, 2,..., m Ax = b Lagrangian: L(x, λ) = (1/2)x T P (λ)x + q(λ) T x + r(λ) where P (λ) = P 0 + m λ i P i, q(λ) = q 0 + m λ i q i, r(λ) = r 0 + m λ i r i i=1 i=1 i=1 Since λ 0, we have P (λ) 0 if P 0 0 and g(λ) = inf x L(x, λ) = (1/2)q(λ)T P (λ) 1 q(λ) + r(λ) 19

21 Lagrange dual problem: maximize (1/2)q(λ) T P (λ) 1 q(λ) + r(λ) λ 0 20

22 KKT Conditions for QP Primal (convex) QP with linear equality constraints: (1/2)x T P x + q T x + r Ax = b KKT conditions: Ax = b, P x + q + A T ν = 0 which can be written in matrix form: [ ] [ ] [ ] P A T x q A 0 ν = b Solving a system of linear equations is equivalent to solving equality constrained convex quadratic minimization 21

23 Summary LP covers a wide range of interesting problems and applications Dual LP is LP First type of nonlinearity: quadratic Least-squares Nonlinear problems that are or can be converted into convex optimization: QCQP (SOCP). Covers LP as special case Reading assignment: Sections and of textbook. 22

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725 Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T