Numerical Optimization

Size: px

Start display at page:

Download "Numerical Optimization"

Dale Oliver
7 years ago
Views:

1 Numerical Optimization Unit 8: Quadratic Programming, Active Set Method, and Sequential Quadratic Programming Che-Rung Lee Scribe: April 27, 2011 (UNIT 8) Numerical Optimization April 27, / 20

2 Quadratic programming The General form min g( x) = 1 x 2 x T G x + x T c The Lagrangian s.t. a T i x = b i i E a T i x b i i I L( x, λ) = 1 2 x T G x + x T c λ T (A x b) a 1 a 2 A =. Rm n (assuming m n) a m (UNIT 8) Numerical Optimization April 27, / 20

3 KKT Condition KKT condition L( x, λ) = 0 a T i x = b i a T i x b i λ i 0 i E i I i I λ i ( a T i x b i ) = 0, i I 1 If G is positive definite and x, λ satisfy KKT conditions, then x is the global solution of the optimization problem (Homework) (UNIT 8) Numerical Optimization April 27, / 20

4 KKT Matrix Let s first consider the equality constraints only L( x, λ) = 0 G x AT λ = c A x = b [ ] [ ] [ ] G A T x c A 0 λ = b [ ] [ ] [ ] G A T x c A 0 = λ (1) b [ ] G A T The matrix is called the KKT matrix. A 0 If A has full row-rank and the reduced Hessian Z T GZ is positive definite, where span{z} is the null space of span{a T } then the KKT matrix is nonsingular. (Homework) If there are only equality constraints, solve (1) directly can get optimal solution. (UNIT 8) Numerical Optimization April 27, / 20

5 Variable elimination 1/2 A general strategy for linear equality constraints is variable elimination. Variable elimination Let A x = b be the linear equality constraints. A R m n, x R n, and b R m. we assume m < n. We can choose m linearly independent columns to be basic variables and use them to solve the constraints. Others are called nonbasic variables, setting to 0. Let ( ) xb AP = [B N], = P T x x N where P is a permutation matrix. b = A x = APP T x = B x B + N x N. (UNIT 8) Numerical Optimization April 27, / 20

6 Variable elimination 2/2 Therefore, x B = B 1 b B 1 N x N. The original constrained problem becomes an unconstrained problem ([ min x f ( x) B s.t. A x = = min f 1 b B 1 ]) N x N b x N x N For nonlinear equality constraints, variable elimination may not feasible. For example, min x,y x 2 + y 2 s.t. (x 1) 3 = y 2 The solution is at (x, y) = (1, 0). Using variable elimination, the problems becomes min x x 2 + (x 1) 3 which is unbounded. For inequality constraints, we can use the active set method. (UNIT 8) Numerical Optimization April 27, / 20

7 Active set method Active set method Active set method solves constrained optimization problems by searching solutions in the feasible sets. If constraints are linear and one can guess the active constrains for the optimal solution, then one can use the active constraints to reduce the number of unknowns, and then perform algorithms for unconstrained optimization problems. Problem: how to guess the set of active constraints. Linear programming is an active set method. (UNIT 8) Numerical Optimization April 27, / 20

8 Active set method for convex QP Consider the following example Example min x g( x) = (x 1 1) 2 + (x 2 2.5) 2 y s.t. x 1 2x (1) x 1 2x (2) x 1 + 2x (3) x 1, x 2 0 (4),(5) ( ) 2 Initial step x 0 = 0 The working set W 0 = {(3), (5)} (1) (2) (4) (3) (5) x (UNIT 8) Numerical Optimization April 27, / 20

9 Solve the EQP Example min g( x) = x1 2 2x x2 2 5x + 25 x 4 ( ) ( ) T min x x + x + 29 x ( ) ( ) s.t. x = Since it has equality constraints only, using KKT system to solve the QP ( ) 2 K = x = 5 λ (UNIT 8) Numerical Optimization April 27, / 20

10 Optimality check Example The solution of the KKT system is x 1 = ( 2 0 ) (, 2 λ 1 = 1 Both Lagrangian multipliers are negative This is not the optimal solution. Remove one of the constraint W 1 = {(5)} and solve the KKT system again Let p 1 = x 2 x 1 = ( x2 λ 2 ( 1 0 ) ) ( ) =, x 2 = ( 1 0 direction, and search α 1 [0, 1], such that x + 2 ( 1 0 ) ), λ 2 = 5 ) be the search = x 1 + α 1 p 1 is feasible. (UNIT 8) Numerical Optimization April 27, / 20

11 Feasibility check Example ( 2 α1 The feasibility check: x 1 + α 1 p 1 = 0 ( ) Move to x 2 = x =. 0 ) α 1 = 1 But λ 2 < 0, it is not the optimal solution. Remove one more constraint W 2 = ( ) ( ) ( 2 0 ( ) 2 1 x3 = x = 2.5 ( ) ( ) ( ) Let p 2 = = be the search direction ( ) x = x 2 + α 2 p 2 =, α 2.5α 2 [0, 1] 2 ) (UNIT 8) Numerical Optimization April 27, / 20

12 Adding new constraints Example For α 2 = 1, constraint (1) will be invalided: x 1 2x α ( ) ( ) 1 1 Move to x 3 = =, and add (1) to the working 1.5 set, W 3 = {(1)}. Solve the KKT conditions: ( x λ 4 ) = ( ) 1.4 x 4 = 1.7 λ4 = 0.8 Since λ 0 and all constraints are satisfied, it is the optimal solution. (UNIT 8) Numerical Optimization April 27, / 20

13 Gradient projection method A special case of inequality constrains are bounded constraints min q( x) = 1 x 2 xg x + x T c s.t. l x u (which means li x i u i forall i) which can be solved by gradient project method. Algorithm: Gradient projection method 1 Given x 0. 2 For k = 0, 1, 2,... until converge (a) Find a search direction g. (b) Construct a piece wise linear function x(t) = p( x + t g, l, u) (c) In each line sequent of x(t) find the optimal solution x c (d) Use { x c as an initial guess to solve min x q( x) xi = x s.t. i c i A( x c ) l i x i u i i / A( x c ) (UNIT 8) Numerical Optimization April 27, / 20

14 In the algorithm 1/2 For 2(a), the search direction can be any descent direction, such as q. For 2(b), the piecewise linear function is computed as p( x, l, u) i = l i if x i < l i x i if x i [l i, u i ] u i if x i > u i For each element x i, compute t i as (l i x i )/g i if g i < 0, and l i > t i = (u i x i )/g i if g i > 0, and u i < + otherwise Sort { t 1, t 2,..., t n } to get t 0 = 0 < t 1 < t 2 <... < t m 1 < = t m For each { [t j 1, t j ], x(t) = x(t j 1 ) + (t t j 1 ) p j 1, where p j 1 gi if t i = j 1 < t i 0 otherwise (UNIT 8) Numerical Optimization April 27, / 20

15 In the algorithm 2/2 For 2(c), we search optimal solution segment by segment [t j 1, t j ]. Let t = t t j 1, x( t) = x(t j 1 ) + t p j 1. q(x( t)) = c T (x(t j 1 ) + t p j 1 ) (x(t j 1 + t p j 1 ) T G(x(t j 1 + t p j 1 )) = 1 2 a( pj 1 ) t 2 + b( p j 1 ) t + c( p j 1 ) for some function a, b, c of p j 1. Optimal t = b( pj 1 ) a( p j 1 and t = t j 1 + t. ) (UNIT 8) Numerical Optimization April 27, / 20

16 Sequential quadratic programming Recall the Newton s method for unconstrained problem. It builds a quadratic model at each x K and solve the quadratic problem at every step. SQP uses similar idea: It builds a QP at each step, f : R n R, c : R n R m min x f ( x) s.t. c( x) = 0 Let A( x) be the Jacobian of c( x): A( x) = ( c 1 c 2 c m ) T The Lagrangian L( x, λ) = f ( x ) λ T c( x) The KKT condition: f ( x ) A( x ) T λ = 0, c( x ) = 0 (UNIT 8) Numerical Optimization April 27, / 20

17 Newton s method [ x L( x, λ) c( x) ] [ Let F ( x, f ( x) A( x) T λ λ) = = c( x) solution x, λ must satisfy the KKT condition F ( x, λ ) = 0. Using Newton s method to solve F = 0. [ The Jacobian F ( x, xx L A( x) λ) = T ] A( x) 0 The Newton step [ ] [ ] [ ] xk+1 xk pk = λk+1 + λk, lk ]. The optimal where [ xx L A T k A k 0 ] [ ] [ pk fk + A = T k λ k lk c k (We use A k = A( x k ), f k = f ( x k ), and c k = c( x k ).) ] (2) (UNIT 8) Numerical Optimization April 27, / 20

18 Alternative formulation To simply that, we examine the first equation xx L p k A T k l k = f k + A T k λ k Since A(x k ) T ( λ k + l k ) = A(x k ) T λk+1, we can rewrite (2) as [ xx L A T k A k 0 ] [ ] [ ] p k fk = λ k+1 c k (3) If A k is of full row-rank and Z T 2 xxlz is positive definition, where Z is the null space of spana k, then the above equation solves the following QP 1 min p 2 pt xx L p + x fk T p s.t. A k p + c k = 0 For inequality, we can are the similar technique: A k p + c k 0 (UNIT 8) Numerical Optimization April 27, / 20

19 The sequential quadratic programming Algorithm: The sequential quadratic programming 1 Given x 0. 2 For k = 0, 1, 2,... until converge 1 Solve Subject to 2 Set x k+1 = x k + p k 1 min p k 2 pt k 2 xxl p k + fk T p k c i ( x k ) T p k + c i ( x k ) = 0 c i ( x k ) T p k + c i ( x k ) 0 i E i I (UNIT 8) Numerical Optimization April 27, / 20

20 Problem of SQP For inequality constraints, linearization may cause inconsistent problem. For example, consider the constraints The feasible region is x 2. c 1 : x 1 0 c 2 : x The linearization of c 1 and c 2 at x = 1 becomes c1 T p + c 1(x) 0 c2 T p + c 2(x) 0 p 0 2p 3 0, for which feasible region is empty. (UNIT 8) Numerical Optimization April 27, / 20

Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective