(We assume that x 2 IR n with n > m f g are twice continuously ierentiable functions with Lipschitz secon erivatives. The Lagrangian function `(x y) i

Transcription

1 An Analysis of Newton's Metho for Equivalent Karush{Kuhn{Tucker Systems Lus N. Vicente January 25, 999 Abstract In this paper we analyze the application of Newton's metho to the solution of systems of nonlinear equations arising from equivalent forms of the rst{orer Karush{Kuhn{Tucker necessary conitions for constraine optimization. The analysis is carrie out by using an abstract moel for the original system of nonlinear equations for an equivalent form of this system obtaine by a reformulation that appears often when ealing with rst{orer Karush{Kuhn{ Tucker necessary conitions. The moel is use to etermine the quantities that boun the ierence between the Newton steps corresponing to the two equivalent systems of equations. The moel is suciently abstract to inclue the cases of equality{constraine optimization, minimization with simple bouns, also a class of iscretize optimal control problems. Keywors. Nonlinear programming, Newton's metho, rst{orer Karush{Kuhn{Tucker necessary conitions. AMS subject classications. 49M37, 96, 93 Introuction A popular technique to solve constraine optimization problems is to apply Newton's metho to the system of nonlinear equations arising from the rst{orer necessary conitions. For instance, the system r x`(x y) = rf(x) + ry = (.) = correspons to the rst{orer necessary conitions of the equality{constraine optimization problem minimize f(x) subject to = g (x). g m (x) A = Departamento e Matematica, Universiae e oimbra, 3 oimbra, Portugal. lvicentemat.uc.pt. Support for this author has been provie by Instituto e Telecomunicac~oes, JNIT, Praxis XXI 2/2./MAT/346/94.

2 (We assume that x 2 IR n with n > m f g are twice continuously ierentiable functions with Lipschitz secon erivatives. The Lagrangian function `(x y) is ene as `(x y) = f(x) + y >.) The Newton step associate with the system (.) is given by 2 rxx`(x y) r r > x y =? r x`(x y) (.2) where r > represents the transpose of the Jacobian matrix g x (x) of, r x`(x y) 2 rxx`(x y) are the graient the Hessian of the Lagrangian with respect to x, respectively. There are cases in constraine optimization where the system of rst{orer necessary conitions is reformulate by eliminating variables /or equations. For the example given above, we know that (.) is equivalent (with x = x) to Z(x) > rf(x) = = where the columns of the orthogonal matrix Z(x) form a basis for the null space of r >. The matrix Z(x) shoul be compute as escribe in [7] so that it can be extene smoothly in a neighborhoo of x (see [7, Lemma 2.]). The Newton step associate with (.3) is ene by Z(x) > 2 rxx`(x y(x)) r > x =? Z(x)> rf(x) where y(x) is the vector of least squares multipliers obtaine by solving minimize kry + rf(x)k 2 2 (.3) (.4) with respect to y. See [7]. Two equivalent forms of the necessary conitions gave rise to two ierent Newton methos (computational issues relate to these methos are escribe, e.g., in the books [6] [8]). It is natural to ask how o these two methos compare, in other wors how close x x are from each other. The goal of this work is not to provie the answer only for this particular reformulation. We propose a moel where the answer can be given for the equality{constraine case but also for other cases. One case in which we are also intereste is minimization with simple bouns The rst{orer necessary conitions are minimize f(x) subject to x rf(x)? y = (.5) x > y = (.6) x y (.7) Since this problem involves inequality constraints, we will rather call conitions (.5){(.7), rst{ orer Karush{Kuhn{Tucker (KKT) necessary conitions. A step of Newton's metho applie to (.5){(.6) is the solution of r 2 f(x)?i Y X x y 2 =? rf(x)? y XY e (.8)

3 where X = ia Y = iag(y). (Given a vector u in IR n, iag(u) represents the iagonal matrix of orer n whose i{th iagonal element is u i. Also, e represents a vector of ones. We omit the imension of e since that will be etermine from the context.) The application of Newton's metho is mae by using an interior{point approach, where x y are positive x y are scale by x y so that x + x x y + y y are also positive. The equivalent form of (.5){(.7) that we woul like to consier (with x = x) is given by D(x) 2 rf(x) = (.9) x where D(x) is the iagonal matrix of orer n with i{th iagonal element given by 8 >< D(x) = ii > (x i ) 2 if (rf(x)) i if (rf(x)) i < This simple fact is prove in []. The iagonal element D(x) 2 even continuous, but if D(x) 2 is ierentiable for all i, then ii ii might not be ierentiable, or D(x) 2 rf(x) = D(x) 2 r 2 f(x) + x x (D(x)2 e) rf(x) The iagonal element D(x) 2 is not ierentiable if (rf(x)) i =, in which case the i{th component of the (iagonal) Jacobian matrix x (D(x)2 e) is articially set to zero. So, it makes sense ii to ene E(x) as the iagonal matrix of orer n whose i{th component is given by 8 >< E(x) = ii > (rf(x)) i if (rf(x)) i > otherwise to ene the Newton step corresponing to the equation (.9) as D(x) 2 r 2 f(x) + E(x) x =?D(x) 2 rf(x) (.) See [] [2] for more etails. For these other examples, we are intereste in comparing the two alternative Newton approaches by looking at the istance between the two alternative Newton steps x x. In the next section, we erive an abstract moel to provie a general answer that can be particularize for the ierent examples. The analysis will consier a relating conition U(x x y) =, will establish that the norm of the ierence of the steps is boune by a constant times ku(x x y)k. We will observe in the equality-constraine example (Section 3) that ku(x x y)k converges to zero if both Newton sequences are converging to a point satisfying the rst-orer necessary conitions. As a consequence of our analysis, the alternative Newton steps ten to be the same. However, this is not true in the secon type of examples (Sections 4 5) because ku(x x y)k oes not necessarily converge to zero even if both Newton sequences are converging to the same stationary point. 3

4 2 The abstract reformulation The original system of nonlinear equations is ene in the variables x y F (x y) = (2.) The equivalent form of this system that we consier is base on the equation G(x)H(x) = (2.2) where G satises at a solution (x y ) a ening conition of the form G(x )F (x y ) = G(x )H(x ) (2.3) The equivalence relates the variables x, x, y through the relating conition so that U(x x y) = F (x y) = is equivalent to G(x)H(x) = U(x x y) = The entrances in F, G, H are assume to have Lipschitz rst erivatives. Their meaning is clear for the examples we have given before but we postpone this for a while to analyze Newton's metho when applie to (2.) (2.2). A step of Newton's metho when applie to (2.) is the solution of F whereas a Newton's step for system (2.2) is given by x F (x y)x + (x y)y =?F (x y) (2.4) y x (G(x)H(x)) x =?G(x)H(x) (2.5) Our goal is to boun kk in terms of ku(x x y)k. First, we multiply (2.4) by G(x) y subtracting (2.5) to (2.6), we obtain G(x) F (x y)x + G(x)F (x y)y =?G(x)F (x y) (2.6) x y? x (G(x)H(x)) x =?G(x) F (x y)x? G(x)F x y (x y)y + G(x)H(x)? G(x)F (x y) Finally, we a x G(x)H(x) x to both sies, to get h x (G(x)H(x)) i () = Using the enitions x + F (G(x)H(x))? G(x) x (x y) G(x)H(x)? G(x)F (x y) x? G(x) F y (x y)y R(x) = x (G(x)H(x)) S(x y) = G(x)F (x y) x 4

5 we erive an upper boun for kk kk kr(x)? k kr(x)? S(x y)k + kg(x) F y (x y)k krf (x y)?> k kf (x y)k + kr(x)? k kg(x)h(x)? G(x)F (x y)k If GH, F, F are positive constants such that kr(x)? k GH krf (x y)?> k F kf (x y)k F then kx?xk F F GH kr(x)? S(x y)k + G(x)F y (x y) + GH kg(x)h(x)?g(x)f (x y)k We observe from this inequality that kk is boune above by three important terms. First, kk epens on how close the values for the functions G(x)H(x) G(x)F (x y) are from each other. It oes not matter how small the resiuals G(x)H(x) F (x y) are, but rather how close the function value G(x)H(x) is from the value of F (x y) reuce by G(x). The epenence on R(x)? S(x y) is about the consistency of the erivatives (G(x)H(x)) x G(x) F (x y), the former being the erivative of G(x)H(x) the latter the erivative of x F (x y) with respect to x reuce by the operator G(x). We conclue also that the norm of epens on the norm of G(x) F y quite interesting aspect of the analysis. In the examples given later, the term kg(x) F y (x y) which is a (x y)k is either zero or boune by ku(x x y)k with x = x. One can see that G(x) F y (x y) is the erivative of F (x y) with respect to y reuce by the operator G(x), its norm inuences the ierence between x x. From the inequality given above we can easily prove the following theorem. Theorem 2. onsier a Newton step (2.4) for F (x y) =, where rf (x y) is nonsingular. onsier a Newton step (2.5) for G(x)H(x) =, where R(x) is nonsingular. If there exist positive constants, 2, 3 such that then kr(x)? S(x y)k ku(x x y)k G(x)F y (x y) 2 ku(x x y)k (for some x), kg(x)h(x)? G(x)F (x y)k 3 ku(x x y)k x? x ku(x x y)k for some positive constant. The constant in Theorem 2. epens on x, x, y since, 2, 3 epen also on these points. We can assume that the star Newton assumptions [4] hol for the Newton methos ene by (2.4) (2.5) at the points (x y ) x, respectively. Then, if x y are suciently close to x y, x is suciently close to x, the constants, 2, 3, o not epen on the points x, x, y. We move rapily to the examples to illustrate our analysis. 5

6 3 Equality{constraine optimization In this case The choices for G(x) H(x) are Since G(x) = F (x y) = G(x)H(x) = Z(x) > I G(x)H(x)? G(x)F (x y) = rf(x) + ry Z(x) > rf(x) H(x) = rf(x) Z(x) > rf(x)? Z(x) > rf(x)? the ening conition (2.3) is satise for any pair (x y), even if it oes not verify the rst-orer necessary conitions. From the theory presente in [7], Also, R(x) = x (G(x)H(x)) = S(x y) = Z(x) > 2 rxx`(x y(x)) Z(x) > 2 rxx`(x y) r > r > the Lipschitz continuity of the secon erivatives of `(x y) imply for some positive constant. Moreover, kr(x)? S(x y)k It is natural to ene the relating conition as U(x x y) = G(x) F (x y) = y y? y(x) y? y(x) = Theorem 2. assures the existence of a positive constant such that kk y? y(x) where x x are given by (.2) (.4). It is obvious that F (x y) = is equivalent to G(x)H(x) = U(x x y) =. Also, if (x y ) is a stationary point then lim (x y)? (x y ) ku(x x y)k = (3.) the Newton steps x x ten in this situation to be the same step. 6

7 4 Minimization with simple bouns In this case, the equivalent KKT systems are F (x y) = rf(x)? y XY e = G(x)H(x) = D(x) 2 rf(x) = Of course, we have exclue from the KKT systems the nonnegativity of x, x, y. The choices for G(x) H(x) are G(x) = D(x) 2 I H(x) = rf(x) In this case the ening conition (2.3) is not satise unless we are at a point that veries the rst{orer KKT necessary conitions. In fact, we have G(x)H(x)? G(x)F (x y) = D(x) 2 rf(x)? D(x) 2 rf(x)? R(x) S(x y) are given by Moreover, X? D(x) 2 Y e R(x) = D(x) 2 r 2 f(x) + E(x) S(x y) = D(x) 2 r 2 f(x) + Y Thus, if the relating conition is ene as G(x) F y (x y) =?D(x)2 + X U(x x y) =? X? D(x) 2 e? X? D(x) 2 e (Y? E(x)) e A = then there exist positive constants, 2, 3 satisfying the assumptions of Theorem 2.. This theorem assures the existence of a positive constant such that kk? X? D(x) 2 e? X? D(x) 2 e (Y? E(x)) e A where x x are given by (.8) (.). Note that F (x y) = is equivalent to G(x)H(x) = U(x x y) = provie rf(x). However, in this example, if (x y ) is a point that satises the rst-orer KKT necessary conitions, the limit (3.) might not hol because lim X? D(x) 2 e lim x?x o not necessarily exist or equal zero. 7 x?x X? D(x) 2 e

8 5 Discretize optimal control problems with bouns on the control variables In this section, we consier the class of nonlinear programming problems analyze in [3]. See also [5]. A nonlinear programming problem of this class is formulate as minimize f(x x 2 ) subject to g(x x 2 ) = x 2 where x is in IR m x 2 is in IR n?m. In this class of problems, r is partitione as r = r x r x2 where r x is nonsingular. The rst{orer KKT necessary conitions are rf(x) + ry? = y 2 = x > 2 y 2 = x 2 y 2 A basis for the null space of r > is given by the columns of W (x) = The equivalent KKT system that we consier is?r x?> r x2 > I D(x) 2 W (x) > rf(x) = (5.) = (5.2) x 2 where D(x) is the iagonal matrix of orer n? m with i{th iagonal element given by D(x) = ii 8 >< > (x 2 ) i 2 if if W (x) > rf(x) i W (x) > rf(x) < i The Newton step associate with (5.){(5.2) is the solution of D(x) 2 W (x) > r 2 xx`(x y (x)) + E(x) x =?D(x) 2 W (x) > rf(x) 8

9 where y (x) =?r x? r x f(x) E(x) is a iagonal matrix of orer n?m with i{th iagonal element given by 8 >< E(x) = ii > W (x) > rf(x) i if otherwise W (x) > rf(x) > i See [3] for more etails. In this case, the KKT systems are represente by F (x y) = rf(x) + ry? X 2 Y 2 e y 2 A G(x)H(x) = D(x) 2 W (x) > rf(x) (We have exclue the nonnegativity of x 2, x 2, y 2.) The choices for G(x) H(x) are G(x) = D(x) 2 W (x) > I I H(x) = rf(x) The ening conition (2.3) is not satise unless we are at a point that veries the rst{orer KKT necessary conitions G(x)H(x)? G(x)F (x y) = A D(x) 2 W (x) > rf(x)? D(x) 2 W (x) > rf(x)?? X 2? D(x) 2 Y 2 e? Also, R(x) S(x y) are given by Moreover, R(x) = S(x y) = So, the relating conition is ene as D(x) 2 W (x) > r 2 xx`(x y (x)) + E(x) r > D(x) 2 W (x) > r 2 xx`(x y ) + Y 2 r > G(x) F y (x y) =?D(x) 2 + X 2 U(x x y) = y? y (x)? X2? D(x) 2 e? X2? D(x) 2 e (Y 2? E(x)) e A = (see [3]) 9

10 assuring the existence of the positive constants, 2, 3 in Theorem 2., which in turn guarantees the existence of a positive constant such that kk y? y (x)? X2? D(x) 2 e? X2? D(x) 2 e (Y 2? E(x)) e A Note that F (x y) = is equivalent to G(x)H(x) = U(x x y) = provie W (x) > rf(x). In this example, as in the previous one, even if (x y ) satises the rst-orer KKT conitions, there is no guarantee that the limit (3.) is true. A similar analysis for the nonlinear programming problem is also of interest. For instance, we coul consier the primal{ual, the ane{scaling, the reuce primal{ual algorithms escribe in [9]. References [] T. F. oleman Y. Li, On the convergence of interior{reective Newton methos for nonlinear minimization subject to bouns, Math. Programming, 67 (994), pp. 89{224. [2], An interior trust region approach for nonlinear minimization subject to bouns, SIAM J. Optim., 6 (996), pp. 48{445. [3] J. E. Dennis, M. Heinkenschloss, L. N. Vicente, Trust{region interior{point SQP algorithms for a class of nonlinear programming problems, SIAM J. ontrol Optim., 36 (998), pp. 75{794. [4] J. E. Dennis R.. Schnabel, Numerical Methos for Unconstraine Optimization Nonlinear Equations, Prentice{Hall, Englewoo lis, New Jersey, 983. Republishe by SIAM, Philaelphia, 996. [5] J. E. Dennis L. N. Vicente, On the convergence theory of general trust{region{base algorithms for equality{constraine optimization, SIAM J. Optim., 7 (997), pp. 927{95. [6] R. Fletcher, Practical Methos of Optimization, John Wiley & Sons, hichester, secon e., 987. [7] J. Gooman, Newton's metho for constraine optimization, Math. Programming, 33 (985), pp. 62{7. [8] S. G. Nash A. Sofer, Linear Nonlinear Programming, McGraw-Hill, New York, 996. [9] L. N. Vicente, On interior-point Newton algorithms for iscretize optimal control problems with state constraints, Optimization Methos & Software, 8 (998), pp. 249{275.