On the Solution of Indefinite Systems Arising in Nonlinear Optimization

On the Soluton of Indefnte Systems Arsng n Nonlnear Optmzaton Slva Bonettn, Valera Ruggero and Federca Tnt Dpartmento d Matematca, Unverstà d Ferrara Abstract We consder the applcaton of the precondtoned conjugate gradent (PCG method to the soluton of ndefnte lnear systems arsng n nonlnear optmzaton. Our approach s based on the choce of quasdefnte precondtoners and of a sutable factorzaton routne. Some theoretcal and numercal results about these precondtoners are obtaned. Furthermore, we show the behavour of the PCG method for dfferent representatons of the ndefnte systems and we compare the effectveness of the proposed varants. Keywords: Precondtoned Conjugate Gradent Method, Indefnte Precondtoners, Large Scale Optmzaton, Nonlnear Programmng Problems. 1 Introducton Ths work s concerned wth the soluton of an ndefnte lnear system whose coeffcent matrx has the followng form: ( H A T M = (1 A where H s an n n symmetrc matrx, whle the matrx A s m n. Ths system s related to the frst order Karush Kuhn Tucker (KKT condtons of the followng quadratc programmng problem mn 1 2 xt Hx c T x Ax = b (2 Ths research was supported by the Italan Mnstry for Educaton, Unversty and Research (MIUR, FIRB Project RBAU1JYPN. 1

2 whch can be expressed as follows: ( ( H A T x = A y ( c b. (3 It s well known that a suffcent condton for the nonsngularty of (1 s that (A1 A s full row rank and H s postve defnte on the null space of A, whch means p T Hp > for any p R n (p such that Ap =. The numercal soluton of systems as (3 s requred also n nonlnear programmng problems. Indeed, n the framework of the nteror pont methods, a varety of algorthms for lnearly and nonlnearly constraned optmzaton (see, for example, [2], [21], [19], [4], [6] requres, at each step, the soluton of the Newton system (or of a perturbaton to t appled to the KKT optmalty condtons of the problem: mn h(x = g(x f(x (4 where f(x : R n R, h(x : R n R neq, g(x : R n R m are twce contnuously dfferentable functons. The nequalty constrants are often reformulated by ntroducng the vector of the slack varables s as g(x s = s. (5 In ths case, the coeffcent matrx of the Newton s system s Q B T C T B C I, (6 S W wth Q = 2 f(x neq 1 λ 2 h (x m 1 w 2 g(x, B = h (x T and C = g (x T. Here Q s the Hessan matrx of the Lagrangan functon of the problem (4, 2 f(x, 2 h (x, 2 g (x are the Hessan matrces of the functon f(x and of the th component of the constrants h(x and g(x respectvely. Furthermore, λ R neq and w R m are the Lagrange multplers of the equalty and nequalty constrants respectvely.

3 The matrces S and W are dagonal matrces whose entres are the components of the vectors s and w respectvely. Snce n the nteror pont approach the slack varables and the multplers related to the nequaltes are forced to be postve, we wll assume that the dagonal entres of the matrces S and W are postve. The matrx (6 s not symmetrc; thus t s usual to obtan a symmetrc representaton of the Newton system by a sutable scalng of the equatons and by applyng elmnaton technques. From the lterature, we devsed four dfferent representatons of the Newton system, whch wll be descrbed n the secton 3. The coeffcent matrces of these reformulatons of the Newton system have the same block structure (1. Recently, many authors propose as effcent teratve lnear solver for the system (3, the precondtoned conjugate gradent (PCG method, wth an ndefnte precondtoner wth the same block structure of the matrx (1 ( G A T P =, (7 A where G s a postve defnte approxmaton of H. In [14], under sutable hypotheses, the authors prove that the PCG method wth such precondtoner appled to the system (3 termnates n a fnte number of steps n exact arthmetc, provdng also a spectral analyss of the matrx MP 1. The same precondtoner and ts varants have been further nvestgated (see for example [4], [15], [11], [7] and references theren. The matrx P has a very specal structure, whch yelds mportant propertes. Indeed, f we consder the augmented system related to the frst order condton of the least squares problem mn g r A T g 2 G, where 1 u 2 G = u T G 1 u (G 1 postve defnte, we obtan 1 ( G A T A ( ḡ g = ( r. (8 Here and n the followng, f g s a n + m vector, we ndcate wth ḡ the n vector whose entres are the frst n components of g and wth g the m vector whose entres are the last m components of g, so that g = ( ḡ g If the vector g solves the prevous system, then the component ḡ s the projecton of r n the null space of the matrx A..

4 More precsely, we have ḡ = G 1 (r A T g = (G 1 G 1 A T (AG 1 A T 1 AG 1 r = P A r where we denote by P A the projecton operator on the null space of A P A = G 1 G 1 A T (AG 1 A T 1 AG 1. (9 Ths mples Aḡ =. Ths property plays a crucal role n the analyss of the PCG method that enables us to derve n a more general way the theoretcal features of the PCG method, reported n the next secton. We focus on the connecton between the precondtonng technque wth the matrx P and the projecton operaton on the null space of the matrx A. Moreover, we show that the approach followed n [14] and the one suggested n [12], whch provdes to solve the quadratc problem (2 by projectng at each step the current resdual on the null space of A, are very smlar. Nevertheless, the frst approach could prevent the numercal nstablty problems whch may arse followng the second approach. In secton 4, a set of numercal experments enables us to compare the effectveness of the PCG method for the soluton of the Newton system n four dfferent formulatons. Furthermore, we employed the PCG method as nner solver for the Newton system arsng at each step of the nteror pont algorthm descrbed n [2], comparng the performance of the whole nteror pont algorthm wth respect to the dfferent representatons of the nner lnear system. 2 The precondtoned conjugate gradent method By puttng v = ( x y ( c, k = b the system (3 can be wrtten as Mv = k. The PCG method appled to (3 can be wrtten as follows: Algorthm 2.1 Choose an ntal pont v, compute r = k Mv, g = P 1 r, ν = r T g and put p = g ;,

5 for =, 1,... untl a stoppng crteron s satsfed δ p T Mp (1 α ν/δ (11 r +1 = r αmp (12 v +1 = v + αp (13 g +1 = P 1 r +1 (14 δ ν (15 ν r +1 T g +1 (16 β ν/δ (17 p +1 = g +1 + βp (18 Snce M s an ndefnte matrx, t can happen that, for some ndex, the quantty δ computed at the step (1 s zero: n ths case, we say that a breakdown occurs for the algorthm. It can be proved (Theorem 3.4 n [14] that, f P s defned as n (7, A s full row rank, M s a symmetrc nonsngular matrx and a breakdown does not occur, then, startng from any v R n+ m, the Algorthm 2.1 fnds a soluton of the system Mv = k n at most n m + 2 teratons. The approach descrbed above can be consdered a prmal dual approach to the soluton of the programmng problem (2, snce the Algorthm 2.1 s equvalent to the PCG method appled to the determnaton of a statonary pont of the Lagrangan functon of the problem (2 L(x, y = 1 2 xt Hx c T x + y T (Ax b. Indeed, at each step (12, we have ( x L(x r =, y y L(x, y 2.1 Precondtonng technques and projecton procedures The next theorem shows that, f the startng pont of the PCG procedure s chosen so that Ax = b, then the lnear system whch has to be solved at the step (14 has the form (8 at each terate. Theorem 2.1 Assume that P defned as n (7, A s full row rank and M s a symmetrc nonsngular matrx..

6 If the startng pont v = ( x y (19 s chosen such that x solves the system Ax = b and a breakdown does not occur, then the Algorthm 2.1 generates a sequence of ponts x such that Ax = b so that r = ( r. Moreover, the drecton p s gven by ( p p =, p where p belongs to the null space of A. Proof. The nverse of the precondtoner P s gven by ( P 1 G = 1 G 1 A T (AG 1 A T 1 AG 1 G 1 A T (AG 1 A T 1 (AG 1 A T 1 AG 1 (AG 1 A T. (2 The left up block of the matrx P 1 s the projecton operator on the null space of A defned n (9. If the startng pont (19 s such that x s a soluton of the system Ax = b, then we have ( c Hx A r = T y Then, when we compute g = P 1 r, we have ( P g = A r (AG 1 A T 1 AG 1 r = = ( r. ( ḡ g, where ḡ s the projecton of the vector r on the null space of A. Snce p = ḡ belongs to the null space of A, we have that and then Ax 1 = A(x + α p = A(x + αḡ = Ax = b r 1 = ( r1, where r 1 = c Hx 1 A T y 1. Let us proceed by nducton and assume that Ax = b and ( p p = p

7 where p belongs to the null space of A. Then, snce x +1 = x + α p, we have that Ax +1 = b and ( r+1 r +1 =. (21 Moreover, the precondtoned gradent g +1 s gven by ( g +1 = P 1 P r +1 = A r +1 (AG 1 A T 1 AG 1 r = +1 ( ḡ+1 g +1, where ḡ +1 belongs to the null space of A. Then we can conclude that p +1 belongs to the null space of A, snce, from the step (18 of the Algorthm 2.1, the drecton p +1 s computed as a lnear combnaton of the precondtoned gradent g +1 and the prevous drecton p. The prevous theorem suggests that, f we choose an approprate startng pont, the use of the precondtoner P and the projecton of the component r of the resdual vector on the null space of A are two strctly related operatons. Remark: We observe that the component A T y of the resdual r = c Hx A T y les n the range space of A T, whch s orthogonal to the null space of A; thus t does not affect, theoretcally, the result of the projecton P A r. The Example 1 n [12] shows that ths fact could not be true from the numercal pont of vew: ths fact wll be further nvestgated n the next secton. Let us ntroduce the followng notaton: f Z s a n ( n m matrx whose columns form a bass for the null space of A, and A s a full row rank matrx, then every vector x R n admts a unque representaton of the form x = A T x ν + Zx τ, (22 where x ν = (AA T 1 Ax s the normal component of x and x τ s the tangenzal component of x expressed n term of the bass Z. Indeed R n s the drect sum of the range space of A T and of the null space of A. If G s a symmetrc nonsngular matrx, then we have also that x ν = (AG 1 A T 1 AG 1 x, snce G 1 Z s agan a bass of the null space of A. Moreover we can also wrte x = A T x ν + GZx τ G

8 For a convenent x τ Z, we have GZx τ G = GZ(Z T GZ 1 x τ Z, then Z T x = x τ Z. Thus, we can wrte x = A T x ν + GZ(Z T GZ 1 x τ Z (23 and Zx τ = GZ(Z T GZ 1 x τ Z. We can obtan the tangental components wth respect to the bass Z, GZ and GZ(Z T GZ 1 by computng x τ = (Z T Z 1 Z T x x τ G = (Z T GZ 1 Z T x x τ Z = Z T x. The above relatons are useful for the proof of the followng theorem, where a dfferent nterpretaton of Algorthm 2.1 s derved (see proof of Theorem 3.5 n [14], under a weaker assumpton: here we do not requre that b =, y = and that Z s an orthonormal bass of the null space of A. Theorem 2.2 Let us assume that the hypothess A1 holds and let Z be a matrx whose columns form a bass for the null space of A. Let x = A T x ν + Zx τ the frst n components of the soluton of the system (3. If we choose a startng pont v such that Ax = b, then the tangental components {x τ k } of the elements of the sequence {x k} generated by Algorthm 2.1 appled to the system (3 are the elements of the sequence generated by the conjugate gradent method wth the precondtoner Z T GZ appled to the system where c z = c HA T x ν. Z T HZx τ = Z T c z (24 Proof. Snce the startng pont x solves the system Ax = b, the prevous theorem can be appled and the resduals r at each terate have the form (21. In partcular, for the vector r we have r = (c Hx A T y. (25 Snce x s a soluton of the system, we have that Ax = b whch mples AA T x ν = b. Moreover, we also have that Ax = b for every ndex, so that AA T x ν = b. Thus, we can conclude that x ν = x ν for each, because A s full row rank. By substtutng the expresson x = A T x ν + Zxτ n (25, we obtan r = c HZx τ HA T x ν A T y,

9 so that the tangental component of the resdual r expressed wth respect to the bass GZ(Z T GZ 1 s gven by r τ Z = Z T r = = Z T (c HA T x ν ZT HZx τ Z T c z Z T HZx τ whch s the resdual of the system (24. By explotng the decomposton (23 of r, we can wrte ( GZ(Z r = T GZ 1 r τ Z + A T r ν. The projecton operator P A defned n (9, can be expressed by means of the null space bass Z as P Z = Z(Z T GZ 1 Z T, snce, for every vector u R n, usng the formula (23, we have P A u = Z(Z T GZ 1 u τ Z = P Z u. Explotng the prevous formulaton of the projecton operator and takng nto account (2, t s easy to see that the precondtoned gradent computed at the step (14 has the followng form: Thus, t follows that g = P 1 r = ( Z(Z T GZ 1 r τ Z r ν. (26 r T g = r τ ZT (Z T GZ 1 r τ Z. (27 Furthermore, recallng the prevous theorem, the drecton p can be wrtten as ( Z p τ p = p snce p belongs to the null space of A; ths mples that ( HZ p τ Mp = + A T p from whch we obtan p T Mp = p τt (Z T HZ p τ. (28

1 The equaltes (27 and (28 show that the coeffcents α and β nvolved n the updatng steps (12 and (18 of the Algorthm 2.1 are the same as n the CG algorthm wth precondtoner (Z T GZ appled to the system (24. Moreover, the tangental component of the vector p +1 = Z p τ +1 can be obtaned as follows thus we can wrte p = Z p τ = ḡ + β p 1 = Z(Z T GZ 1 r τ Z + βz p τ 1 p τ = (Z T GZ 1 r τ Z + β p τ 1. The last equalty show that also the drecton p τ can be obtaned by the updatng rule of the CG algorthm wth precondtoner (Z T GZ appled to the system (24. The prevous remarks guarantee that the Algorthm 2.1 appled to the system (3 mplctly acts on the tangental components of the terates as the CG method appled to the system (24 wth precondtoner (Z T GZ. The prevous result can be employed to derve an estmaton of the absolute error and the fnte termnaton property of the Algorthm 2.1. Indeed the Algorthm 2.1 fnds the soluton vector x after at most n m teratons, and for 1 n m the followng estmaton holds x x 2 ( 1 k k 1 + x x (29 k where k = k(z T HZ(Z T GZ 1 and s the eucldean norm. Furthermore, under the assumptons of the prevous theorem, the Algorthm 2.1 does not break down: ndeed, the quantty δ computed at the step (1 actually has the form (27, and, f the matrx H s postve defnte on the null space of A, δ s strctly postve. Our mplementaton of the Algorthm 2.1 provdes the drect factorzaton of the precondtoner P. We observe that ths matrx can be factorzed n a Cholesky lke form L n+ m DL T n+ m, (3 where L n+ m s a lower trangular matrx wth dagonal entres equal to one and D s a nonsngular dagonal matrx wth n postve and m negatve dagonal entres. In order to reduce the fll ns n the lower trangular factor, we can perform a mnmum degree reorderng of the matrx P. But, t s not

11 assured that the symmetrcally permuted matrx UP U T can be factorzed n the Cholesky lke form. Nevertheless, we can obtan a factorzaton n the form (3 f we use for the matrx P the regularzaton technque descrbed n [1]; n other words, nstead of usng the precondtoner P, we compute the factorzaton of P = P + ( R1 R 2 where R 1 and R 2 are non negatve dagonal matrces such that U P U T admts a factorzaton of the form (3. The computaton of R 1 and R 2 can be obtaned durng the factorzaton procedure. If a pvot d s too small ( d < 1 15 max j< d j, we put d = ɛ f 1 n, or d = ɛ f n + 1 n + m, where ɛ s the machne precson. The dynamc computaton of the elements of R 1 and R 2 reduces the perturbaton to a mnmum. The Cholesky lke factorzaton of P can be obtaned by a modfcaton of the Ng and Peyton package. The modfcatons are descrbed n [3]. Ths new package, called BLKFCLT and downloadable from the web page http://dm.unfe.t/blkfclt/, s structured n two phases: the frst phase provdes an a pror reorderng routne for the sparsty preservng and the computaton of a symbolc factorzaton, whle, n the second phase, the Cholesky lke factorzaton s computed, employng the dynamc regularzaton strategy. 2.2 Another projecton algorthm It s mportant to notce that the proof of Theorem 2.2 and the estmaton (29 do not depend on the varable y, whch represents the Lagrange multpler of the equalty constrants of the problem (2. Indeed, the Algorthm 2.1 actually solves the problem mn 1 2 xτ T Z T HZx τ x τ T c Z that we obtan by substtutng (22 n (2. On the contrary of the methods proposed n [9], [13], [18], n ths case we do not have to determne Z. A smlar approach s used n the Algorthm 2 n [12], where the PCG teraton s appled only to the prmal varable x. In ths case, the resdual vector s defned as r + = Hx c, and, n general, n the algorthm, t wll be bounded away from zero, but, as the terates approach to the soluton, t wll become ncreasngly closer to the

12 range of A T. Indeed, f (x T, y T T s the soluton of the system (3, we have c Hx = A T y. Thus, the projecton of the resdual on the null space of A, the vector g, wll become ncreasngly closer to zero. Ths dfference n the magntudes of r + and g mght cause numercal dffcultes, snce g = P A r + = G 1 ( r + AT v, where v = argmn r + AT v G 1, and, n fnte arthmetc, r + A T v mght gve rse to a sgnfcant cancellaton of dgts. Ths roundoff error leads the projected resdual to do not belong exactly to the null space of A (see Example 1 n [12]. In order to avod ths drawback, n [12] the authors propose a varant of the PCG algorthm, whch, at each step, provdes a least squares estmate of the normal component of the resdual A T v and, then, t updates the resdual by subtractng ts normal component. Ths update leads the revsed resdual r to become ncreasngly closer to zero as the terates approach to the soluton. Algorthm 2.2 Algorthm III (Precondtoned CG wth resdual update n [12]. Choose an ntal pont x such that Ax = b and compute r + = Hx c, v = argmn r + AT v G 1, r = r + AT v, g = P A r, and put p = g ; for =, 1,... untl a stoppng crteron s satsfed α rt g p T H p x +1 = x + α p (31 r + +1 = r + + αh p v +1 = argmn r + +1 AT v G 1 (32 r +1 = r + +1 AT v +1 (33 g +1 = P A r +1 (34 β rt +1 p +1 r T g p +1 = g +1 + β p In order to compute the projecton (34, we have to solve two system: frst, we obtan v +1 n (32 by solvng the system ( ( ( G A T g r + = +1 A v +1

13 and then, to obtan the vector g +1 n (34, we solve ( ( ( G A T g+1 r + = +1 A T v +1 A u. In exact arthmetc, the vector r + A T v belongs to the null space of A, so that the component u of the soluton of the prevous system s zero. In other words, n exact arthmetc, the desred projecton of the resdual r s the vector r tself; thus, the steps (32 (33 can be consdered as an teratve refnement step. We observe that, at the terate of the Algorthm 2.2, we can obtan an estmate of the Lagrange multpler as y = k= v k and, furthermore, the vector r represents the resdual of the frst equaton of the system (3: ndeed we have r = Hx + A T y c. In exact arthmetc, the resdual of the second equaton of the system (3 should be the null vector, but operatng n fnte arthmetc, n general, ths s not true. In the Algorthm 2.1 the vector g +1 s obtaned by solvng one system only ( ( ( G A T ḡ+1 r+1 =, A g +1 r +1 where r +1 = c Hx +1 A T y +1 and r +1 = b Ax +1. Also n ths case, n exact arthmetc, the component r +1 of the resdual should be the null vector, but operatng n fnte arthmetc ths s not guaranteed. Thus, snce the Algorthm 2.1 works wth the prmal and dual varables, t takes nto account that ḡ +1 could not belong to the null space of A because of the roundoff errors n the component r +1 ; on the other hand, the Algorthm 2.2 does not take nto account of the dual varable, so that t partally controls the error on the projecton of the resdual vector r +1 + wth one step of teratve refnement. It s nterestng to observe the effects of the fnte precson on the Algorthms 2.1 and 2.2. The fgure 2.2 shows a comparson between the Algorthms 2.1 and 2.2 on the test problem CVXEQP3 of the CUTE collecton [5] wth n = 1 and m = 75. For each teraton, we have consdered the followng quanttes: the norm of the resduals r and r, whch ndcate the progress towards the soluton of the system(resduals; the scalar products r T g and r T g, whch are measurements of the angle between the resdual and g and g respectvely (Orthogonalty; the quanttes Aḡ and A g, whch tells us how precsely the projecton s computed (Projecton.

14 Fgure 1: Test problem CVXEQP3 1 1 1 5 Algorthm 2.1 1 1 1 5 Algorthm 2.2 Resdual Orthogonalty Projecton 1 1 1 5 1 5 1 1 1 1 1 15 1 15 1 2 1 2 1 25 5 1 15 Iteraton 1 25 5 1 15 Iteraton After 1 teratons, can observe that for the Algorthm 2.1, all the consdered quanttes are less than 1 8, whle for the Algorthm 2.2, the norm of the resdual, s greater than 1 3. Thus, the fnte precson does not sgnfcantly nfluence the Algorthm 2.1, whle t leads the Algorthm 2.2 to have less accuracy n the results. 3 Representatons of the Newton system In ths secton we descrbe four dfferent formulatons of the Newton system whose matrx s gven n (6. 3.1 Full System Followng the approach n [21], [2], [6], by multplyng the last block of equatons by S 1 and by performng a symmetrc permutaton on the second

15 and fourth columns, we obtan a block system whose coeffcent matrx s the followng Q C T B T F 1 I m C I m, (35 B where F 1 = S 1 W. (36 We can consder the matrx (35 as a specal case of (1 wth ( ( Q C Im H = F 1 and A =. (37 B A suffcent condton for the nonsngularty of the matrx (35 s that (A2 B s a full row rank matrx and Q + C T F 1 C s postve defnte on the null space of B. Let us prove that, under the hypothess (A2, the matrx (35 s nonsngular. If we have Q C T B T x F 1 I m y C I m w = B z then, from the fourth equalty, x belongs to the null space of B. From the thrd block of equatons, we have y = Cx and, from the second block of equatons, w = F 1 y + F 1 Cx, so that If we consder (Q + C T F 1 Cx + B T z = x T (Q + C T F 1 Cx + x T B T z = snce Bx =, from the hypothess (A2, we have x =, y =, w = and fnally B T z =. Snce B s a full row rank matrx, z =. Then the matrx (34 s nonsngular. A suffcent condton so that the hypothess (A2 s satsfed, s that the hypothess (A1 holds.

16 3.2 Reduced system From (35, by elmnatng the second block of equatons, we derve the second representaton of the Newton system. In ths case the coeffcent matrx has the form Q C T B T C F (38 B The block structure of the matrx (38 s the same as n (1, wth ( Q C T H = and A = (B (39 C F A specal case of the system (38 can be obtaned f we have neq = : ths occurs when there are no equalty constrants, or when the equalty constrants n (4 are treated as range constrants, wth upper and lower bounds that concde, as n [7], [19]. In ths case, the coeffcent matrx s ( Q C T M = H =. (4 C F A suffcent condton for the nonsngularty of the matrx (38 s that the hypothess (A2 s satsfed. Indeed, f we have Q C T B T x C F y = B z then, from the last equalty, x belongs to the null space of B. From the second block of equatons, we obtan y = F 1 Cx, so that, from the frst block of equatons, t follows that (Q + C T F 1 Cx + B T z =. Snce x belongs to the null space of B, the prevous equalty yelds x T (Q + C T F 1 Cx = whch, under the hypothess (A2, mples x = and z =. Ths proves the nonsngularty of the matrx (38. The condton (A2 s consstent wth the fact that the system Q C T B T C F B x y z = c d b (41

17 represents the optmalty condtons of the followng equalty constraned quadratc problem mn 1 2 xt Qx c T x + 1 2 (Cx dt F 1 (Cx d Bx = b (42 Indeed, the optmalty condtons for (42 are Qx + C T F 1 (Cx d + B T z = c Bx = b and by ntroducng a new varable y and the block of equatons y = F 1 (Cx d, we obtan (41. We pont out that matrx n the reduced form has the block F, whle n the full form t has the block F 1. 3.3 Condensed form By applyng elmnaton technques to the second block of equatons n (38 we obtan the followng coeffcents matrx ( Q B T (43 B wth Q = Q + C T F 1 C. Also n ths case, the hypothess A2 s a suffcent condton for the nonsngularty of the matrx (43. 3.4 Actve form We consder also the approach followed n [15]; n ths verson of the nteror pont method, a subdvson of the nequalty constrants n two dsjont subsets s performed at each step, the actve and the nactve constrants. The resultng system s obtaned by elmnatng only the equatons related to the nactve constrants n the second block. Thus, the coeffcent matrx has the followng form: ( ˆQ C T H = a and A = (B (44 C a F a where C a s the jacoban matrx of the actve nequalty constrants, C I ndcates the jacoban matrx of the nactve nequalty constrants and ˆQ =

18 Q + C T I F 1 I C I. Furthermore, F a and F I have the same meanng as n (36, but the slacks and the multplers are only the ones related to the actve and nactve constrants respectvely. In ths case, the szes of the blocks and the structure of the matrx can change at each teraton of the outer method. 3.5 The choce of the precondtoner The Newton system can be solved wth the PCG method descrbed n the secton 2. For the full and condensed representatons the matrx P can be chosen as D C T B T F 1 I m C I m B ( D and B B T (45 where D and D are postve dagonal approxmatons of Q and Q respectvely. For the reduced and actve form, the precondtoner can be chosen as D C T B T C F B or ˆD C T a B T C a F a B (46 where ˆD s a postve dagonal approxmaton of the matrx ˆQ. We remember that F = SW 1, where the elements of the dagonal matrces S and W are respectvely the slack varables and the multplers related to nequalty constrants. We observe that the precondtoner n (46 related to the reduced form of the Newton system contans the matrx F whle n the precondtoners n (45 related to the full and the condensed form of the Newton system, the nverse of F appears. The dagonal elements of F or F 1 affect the condton of the matrces M and P ; ndeed, n the last teratons of the outer optmzaton method, when we are near the soluton, the dagonal elements of S correspondng to actve nequalty constrants and the ones of W correspondng to nactve nequalty constrants assume gradually small values, very close to zero. If the method has a local superlnear convergence, ths stuaton s not crtcal, snce few outer steps produce the soluton wthn the requred accuracy [17]. Nevertheless the precondtoners n (45 can be more convenent when the problem has many nactve nequalty constrants, whle the precondtoner n (46 related to the reduced form can be more convenent n presence of many actve constrants. The precondtoner n (46 related to the actve form of the Newton system contans

19 the dagonal entres of F related to the actve nequalty constrants and the nverse of dagonal entres of F correspondng to the nactve nequalty constrants. Ths technque s used to assure that the entres of F a and these of F 1 I are bounded. For the formulaton (38, t s possble to prove some result analogous to the ones presented n the secton 2. Let us ntroduce the followng notaton: f v s a n vector, wth n = n + m, we wll ndcate ts frst n components as ˆv and ts last m components as ˇv. Thus, f v s a n + m + neq vector, we have v = ( v v = Theorem 3.1 Consder the PCG method appled to the system (41. Choosng the startng pont such that Bx = b and Cx F y = d, f a breakdown does not occur, then the Algorthm 2.1 generates a sequence of ponts {x } such that Bx = b and Cx F y = d, so that r = Moreover, at each teraton, we have ˆr ˆv ˇv v.. ˆr = c + C T F 1 d (Q + C T F 1 Cx B T z and the component ˆp of the drecton p belongs to the null space of B; we have also that ǧ = F 1 Cĝ (47 ˇp = F 1 C ˆp (48 We gve only a sketch of the proof. The frst resdual of the PCG procedure s a n + m + neq vector gven by ( r r = where r = ( ˆr,

2 and ˆr = c Qx C T y B T z. Snce y solves the second equaton of the system (41, then we have y = F 1 (Cx d, so that we can wrte ˆr = c + C T F 1 d (Q + C T F 1 Cx B T z. The vector g s computed as g = P 1 r, thus ḡ belongs to the null space of the matrx A defned n (39, but the null space of A s gven by the vectors (û T, ǔ T T such that û belongs to the null space of A and ǔ R m. It follows that ĝ belongs to the null space of B. Moreover, the component ǧ solves the equaton Cĝ F ǧ =, thus we have ǧ = F 1 Cĝ. Snce p = g, t follows that that Bx 1 = B(x + αĝ = Bx = b, whch mples r 1 =. Furthermore we have Cx 1 F y 1 = C(x + αĝ F (y + αǧ = C(x + αĝ F (y + αf 1 Cĝ = Cx F y = d whch yelds ř 1 = and ˆr 1 = c + C T F 1 d (Q + C T F 1 Cx 1 B T z 1. Moreover we have ˇp = F 1 C ˆp. If we assume that Bx = b and Cx F y = d so that ř = and r =, t follows that the component ǧ of the vector g = P 1 r satsfes the relaton (47. Moreover, assumng that ˇp = F 1 C ˆp and ˆp belongs to the null space of B, we have that Bx +1 = B(x + αˆp = b, whch mples r +1 =, and Cx +1 F y +1 = C(x + αˆp F (y + αf 1 C ˆp = d, whch mples ř =. Snce g +1 = P 1 r +1, we have that ĝ +1 = P Aˆr +1 belongs to the null space of B, ǧ +1 = F 1 Cĝ +1. Then ˆp +1 = ĝ +1 + β ˆp belongs to the null space of B. Furthermore we have whch yelds (48. ˇp +1 = ǧ +1 + β ˇp = F 1 Cĝ +1 + βf 1 C ˆp = F 1 C(ĝ +1 + β ˆp = F 1 C ˆp +1

21 If B s a full row rank matrx, Z s an n (n neq matrx whose columns form a bass of the null space of B and N s a symmetrc nonsngular matrx, we can wrte any n vector x as x = B T x ν + Zx τ (49 x = B T x ν + NZx τ N (5 x = B T x ν + NZ(Z T NZ 1 x τ Z. (51 Gven a symmetrc nonsngular matrx D, for the normal component we can wrte x ν = (BB T 1 Bx = (BD 1 B T 1 BD 1 x, whle the coeffcents of the tangental components wth respect to the dfferent bass of the null space of B satsfy x τ = (Z T Z 1 Z T x x τ N = (Z T NZ 1 Z T x x τ Z = Z T x. Theorem 3.2 Let Z a matrx whose columns form a bass for the null space of B, and assume that the hypothess (A2 holds, so that the matrx Z(Q + C T F 1 CZ T s postve defnte. Let x = B T x ν + Zx τ the frst n components of the soluton of the system (41. Then, the tangental components {x τ } of the sequence {x } generated by the Algorthm 2.1 appled to the system (41 wth a startng pont such that Bx = b and Cx F y = d are the elements of the sequence generated by the conjugate gradent method wth the precondtoner Z T NZ, N = (D + C T F 1 C, appled to the system Z T NZx τ = Z T c Z, (52 where N = Q + C T F 1 C and c Z = (c + C T F 1 d NB T x ν Thus, the conjugate gradent method fnds the vector x after at most n neq teratons, and for 1 n m the followng estmaton holds x x 2 ( 1 k k 1 + x x (53 k where k = k(z T NZ(Z T NZ 1. Proof. We can apply the prevous result, so that ˆr = (c + C T F 1 d Nx B T z,

22 from whch, explotng the formula (49 we obtan ˆr = (c + C T F 1 d NB T x ν NZx τ B T z. Snce Bx = Bx = b, t follows that x ν = x ν, thus the prevous formula becomes ˆr = c Z NZx τ B T z. Its tangental component expressed n the bass NZ(Z T NZ 1 can be obtaned as follows: ˆr τ Z = Z T ˆr = Z T c Z Z T NZx τ The nverse of the precondtoner P can be wrtten as P B P B C T F 1 D 1 B T (BD 1 B T 1 P 1 = F 1 C T P B F 1 CD 1 C T F 1 F 1 CD 1 B T (BD 1 B T 1, (BD 1 B T 1 BD 1 (BD 1 B T 1 BD 1 C T F 1 (BD 1 B T 1 where P B s the projecton operator on the null space of B whch can be expressed also as P B = N 1 N 1 B T (B N 1 B T 1 B N 1, P Z = Z(Z T NZ 1 Z T. If we wrte the resdual n the form (51, we obtan ˆr B T ˆr ν + NZ(Z T NZ 1ˆr τ Z r = = from whch we can wrte Ths mples that g = P 1 r = ĝ ǧ g = Z(Z T NZ 1ˆr τ Z F 1 C T ĝ ˆr ν r T g = ˆr τ ZT (Z T NZ 1ˆr τ Z. Furthermore, recallng that the component ˆp belongs to the null space of B and takng nto account of (48, we can wrte Z ˆp τ p = F 1 CZ ˆp τ, p

23 from whch we obtan Mp = QZ ˆp τ + CT F 1 CZ ˆp τ + BT p that mples p T Mp = ˆp τt (Z T NZˆp τ. Fnally, we observe that the vector ˆp τ can be wrtten as ˆp τ = (Z T NZ 1ˆr τ Z + β ˆp τ 1 whch s the updatng step of the PCG method wth precondtoner (Z T NZ appled to the system (52. Ths mples that the Algorthm 2.1 appled to the system (41 s the PCG method wth precondtoner (Z T NZ appled to the system (52; thus the estmate (53 holds. Fnally, for the specal case (4, the Algorthm 2.1 wth the followng precondtoner ( D C T, (54 C F appled to the system ( Q C T C F ( x y = ( c d s equvalent to the applcaton of the PCG method to the system, (Q + C T F 1 Cx = c + C T F 1 d wth precondtoner D + C T F 1 C (Theorem 3.1 n [7]. nonsngular under the hypothess (A2 The system s 4 Numercal results The am of our numercal experence s to compare the effectveness of the PCG method as solver for the Newton system formulated n the four cases descrbed n secton 3. In partcular, we are nterested on how much the representaton of the Newton system can nfluence the performance of the method. Furthermore, we use the PCG method as nner solver n an nteror pont method: we consder the nexact Newton nteror pont algorthm descrbed n [2], whch at each step has to solve the Newton system assocated

24 to the KKT condtons of the NLP problem (4. The numercal results presented n the followng have been carred out by codng ths algorthm n C++, on a HP zx6 workstaton wth Itanum2 processor 1.3 GHz and 2 Gb of RAM; the code s provded of an AMPL nterface. In order to obtan a sgnfcant comparson, we buld a set of NLP problems, wth a large number of nequalty constrants, lsted n table 1. The columns nl and nu report the number of lower and upper bounds on the components of the varable x. Except for the problem Svanberg, whch belongs to the CUTE collecton, the other test problems have been obtaned by modfyng the nonlnearly equalty constraned problems descrbed n [16]: some or all of the orgnal equalty constrants have been changed n nequalty constrants. The frst comparson n table 2 shows the performances of the PCG method on the soluton of the Newton system arsng at the last terate of the nteror pont code. The startng pont s the same n all cases. The stoppng crteron for the PCG procedure s r 1 12. The system has been formulated n the four dfferent representatons descrbed n the prevous secton: 3x3 denotes the reduced form, 2x2 s the condensed system, 4x4 s the full system whle Luk ndcates the actve approach. In ths case, we say that the th nequalty constrant s nactve f the followng condton s satsfed: w 1 5 s. The frst two columns of the table 2 ndcates the number of the actve (ma and nactve (m nequalty constrant, thus the dmenson of the system n the actve form s gven by n + ma + neq. The table reports the comparson n terms of teratons number and executon tme (n seconds, whle the symbol * ndcates that the tolerance of 1 12 was not satsfed after n teratons. The table 3 summarzes the results of the comparson, reportng the number of test problems n whch the Newton system n the dfferent representatons has been solved by the PCG method obtanng the best and worst performance. The representatons of the Newton system whch gves the best result are the reduced and the actve form, whle the less vald approach

25 seems to be the reducton n condensed form. An explanaton of ths fact could be that the precondtoner of the system n condensed form s obtaned by approxmatng the matrx ˆQ, whch n general s not sparse, wth a dagonal matrx, so that the approxmaton could be very poor. Furthermore, we can notce that the best performance obtaned wth the reducton n condensed form have been obtaned n the test problems wth many nactve constrants, for example the problem Lukvl12. Ths could be explaned by observng that n ths case the dagonal matrx F 1 has small elements, so that the dense part of ˆQ = Q + C T F 1 C s n some way weghted by small quanttes. The table 4 contans the tme comparson of four dfferent verson of the nteror pont method descrbed n [2] obtaned by representng the Newton system n the forms presented n the secton 3. In ths case, the termnaton rule for the PCG procedure explots an adaptve stoppng rule whch depends on the volaton of the KKT optmalty condtons at the current terate of the nteror pont algorthm. In the table 5 we report the summary of ths expermentaton: the nteror pont algorthm performs better f the Newton system s represented n reduced form or n actve form. The behavour of the PCG method for these forms s very smlar and we observe that, generally, the PCG method s convenent for test problems wth many actve nequalty constrants. Also the PCG method appled to the full system s convenent but, n general, has a worse behavour than that of the reduced and the actve form. For the condensed form of the Newton system, the PCG method s convenent when there are many nactve nequalty constrants (Lukvle17, Lukvle18, Lukvl1 ( Q s reduced to Q at the last teratons, but, n the opposte case, we observe worse performance or falure.

Table 1: Test Problems N TEST n neq m nl nu 1 Lukvle3 5 1 1 2 Lukvle4 5 24999 24999 3 Lukvle6 51 125 125 4 Lukvle7 5 2 2 5 Lukvle1 5 24999 24999 6 Lukvle11 49997 16665 16665 7 Lukvle14 49997 16665 16665 8 Lukvle16 49997 12499 24998 9 Lukvle17 49997 12499 24998 1 Lukvle18 49997 12499 24998 11 Lukvl2 5 49993 12 Lukvl3 5 2 13 Lukvl4 5 49998 14 Lukvl6 49999 24999 15 Lukvl7 5 4 16 Lukvl1 5 49998 17 Lukvl11 49997 3333 18 Lukvl12 49997 37497 19 Lukvl13 49997 3333 2 Lukvl14 49997 3333 21 Lukvl15 49997 37497 22 Lukvl16 49997 37497 23 Lukvl18 49997 37497 24 Svanberg 5 5 5 5 26

27 Table 2: Numercal soluton of one Newton system CG teratons Executon tme ma m Prob 3x3 2x2 4x4 Luk 3x3 2x2 4x4 Luk 1 Lukvle3 6 25 6 6.1.6.1.1 12498 1251 Lukvle4 14 233 15 14.8 9.4.9.8 125 Lukvle6 9 23 9 9.6 11.5.6.6 2 Lukvle7 4 13 1 4.1.2.1.1 24999 Lukvle1 3 16 3 3.2 4..2.2 16664 1 Lukvle11 1 15 22 11.5.5 1..5 16665 Lukvle14 2 * 2 2.1 *.1.1 24997 1 Lukvle16 1 21 14 9.4.6.6.4 1 24997 Lukvle17 1 48 1 1.4 1.4.4.3 24998 Lukvle18 3 5 3867 3.1.2 154..1 25 24993 Lukvl2 12 1266 35 13 1.2 78.6 26.8 1.4 1 1 Lukvl3 7 1 7 7.2.2.2.2 24998 25 Lukvl4 16 171 17 16 1 6.1 1 1 24999 Lukvl6 12 255 12 12.8 13.8.9.7 3 1 Lukvl7 4 27 7 4.1.3.1.1 3333 Lukvl1 23 24 34 25 1..7 1.6.8 37497 Lukvl11 * * * * * * * * 3333 Lukvl12 11 11 12 11.5.4.6.4 33329 1 Lukvl13 4 96 5 4.2 25,.3.1 37492 5 Lukvl14 36 68 46 37 1.7 2.2 2.4 1.9 37496 1 Lukvl15 11 21 26 11.5.6 1.2.5 24998 12499 Lukvl16 5 1 * 5.2.3 *.2 49998 3 Lukvl18 3 19 3 3.2 3.6.2.2 4387 9613 Svanberg 1 * 2 1.2 *.3.2 Table 3: Summary CG teratons Executon tme 3x3 2x2 4x4 Luk 3x3 2x2 4x4 Luk Best 22 1 8 19 18 4 9 2 Worst 16 5 14 8 Falures 1 3 2 1

28 Table 4: Executon tme n seconds for an nteror pont algorthm N Problem 3x3 2x2 4x4 Luk 1 Lukvle3 11.5 11.1 11.1 11.5 2 Lukvle4 51.7 * 57.7 51.7 3 Lukvle6 161.1 534 612 161.1 4 Lukvle7 6.6 8. 1.5 6.6 5 Lukvle1 * 46.4 * * 6 Lukvle11 24.3 27. 26.6 25.5 7 Lukvle14 * * 482.7 * 8 Lukvle16 21.8 24.1 22.1 2.8 9 Lukvle17 46.3 16.4 * 167.9 1 Lukvle18 12.1 11.7 12.7 12.1 11 Lukvl2 127.3 * 83.1 126.8 12 Lukvl3 8.6 14.4 9.3 8.7 13 Lukvl4 54.3 148.4 51.9 54.9 14 Lukvl6 177.7 564. 132.5 177. 15 Lukvl7 9. 11.9 1.8 9.1 16 Lukvl1 * 41.2 * * 17 Lukvl11 211.1 198.9 615. 189.5 18 Lukvl12 55.54 17.5 43.6 5.9 19 Lukvl13 31.1 349.2 22. 319.7 2 Lukvl14 2.2 * 22. 2.4 21 Lukvl15 187.4 * 12.5 25.5 22 Lukvl16 2. 2.7 22.6 19.9 23 Lukvl18 12.2 13.1 13.4 13.5 24 Svanberg 59. * 6.2 63.1 References [1] A. Altman and J. Gondzo, Regularzed symmetrc ndefnte systems n nteror pont methods for lnear and quadratc optmzaton, Optm. Methods Softw., vol. 11 12, 1999, 275 32. [2] S. Bonettn, E. Gallgan and V. Ruggero, Inner solvers for nteror pont methods for large scale nonlnear programmng, Techncal report n. 64, Dpartmento d Matematca Pura ed Applcata, Unverstà d Modena e Reggo Emla, 25. To appear on Computatonal Optmzaton and Applzatons.

29 Table 5: Summary 2x2 3x3 4x4 Luk Best 5 9 7 6 Worst 8 2 7 5 Falure 6 3 3 3 [3] S. Bonettn and V. Ruggero, Some teratve methods for the soluton of a reduced symmetrc ndefnte KKT system, Techncal report n. 6, Dpartmento d Matematca Pura ed Applcata, Unverstà d Modena e Reggo Emla, 25. To appear on Computatonal Optmzaton and Applzatons. [4] L. Bergamasch, J. Gondzo and G. Zll, Precondtonng ndefnte systems n nteror pont methods for optmzaton, Computatonal Optmzaton and Applcatons, vol. 28, 24, 149 171. [5] I. Bongartz, A.R. Conn, N. Gould and Ph. L. Tont (1995. CUTE: Constraned and Unconstraned Testng Envronnment, ACM Transactons on Mathematcal Software, 21, 123 16. [6] R. H. Byrd, J. C. Glbert and J. Nocedal, A trust regon method based on nteror pont technques for nonlnear programmng, Mathematcal Programmng, 89, 2, 149 185. [7] S. Cafer, M. D Apuzzo,V. De Smone and D. d Serafno, (24 On the teratve soluton of KKT systems n potental reducton software for large scale quadratc problems to appear on Computatonal Optmzaton and Applcatons. [8] E.D. Dolan, J. J. Moré and T. S. Munson (24. Benchmarkng optmzaton software wth COPS 3., Techncal Report ANL/MCS-TM- 273, Argonne Natonal Laboratory, Illnos, USA. [9] J. Dunn, (1993. Second order multpler calculatons for optmal control problems and related large scale nonlnear programs, SIAM Journal on Optmzaton, 3, 489 52. [1] I.S. Duff and J.K. Red, A set of Fortran subroutnes for solvng sparse symmetrc sets of lnear equatons. Tech. Report AERE R1533, HMSO, London, 1982.

3 [11] Durazz C., Ruggero V.; Indefntely precondtoned conjugate gradent method for large sparse equalty and nequalty constraned quadratc problems, Numer. Lnear Algebra Appl., 1 (23, 673-688. [12] N.I.M. Gould, M. E. Hrbar, and J. Nocedal, On the soluton of equalty constraned quadratc programmng problems arsng n optmzaton, SIAM J. Sc. Comput., vol. 23, 21, 1376 1395. [13] D. James (1992. Implct nullspace teratve methods for constraned least square problems, SIAM Journal on Matrx Analyss and Applcatons, 13, 962 978. [14] L. Lukšan and J. Vlček, Indefntely precondtoned truncated Newton method for large sparse equalty constraned nonlnear programmng problems, Numercal Lnear Algebra wth Applcatons, vol. 5, 1998, 219 247. [15] L. Lukšan, C. Matonoha and J. Vlček, Interor pont method for nonlnear nonconvex optmzaton, Numercal Lnear Algebra wth Applcatons, vol. 11, 5-6, 24, 431 453. [16] L. Lukšan and J. Vlček (1999. Sparse and partally separable test problems for unconstraned and equalty constraned optmzaton, Techncal report 767, Insttute of Computer Scence, Academy of Scence of the Czech Republc. [17] F.A. Potra F.A. and S.J. Wrght, Interor pont methods, Journal of Computatonal and Appled Mathematcs., 124 (2 281 32. [18] Ph. L. Tont and D. Tuyttens (199. On large scale nonlnear network optmzaton, Mathematcal Programmng, Seres B, 48, 125 159. [19] R. J. Vanderbe and D.F. Shanno, An nteror-pont algorthm for nonconvex nonlnear programmng, Computatonal Optmzaton and Applcatons, vol. 13, 1999, 231 252. [2] A. Wächter and L. T. Begler, On the mplementaton of an nterorpont flter lne-search algorthm for large-scale nonlnear programmng, Techncal Report RC 23149, IBM T.J. Watson Research Center, Yorktown Heghts, USA, 24. To appear n Mathematcal Programmng. [21] R.A.Waltz, J.L. Morales, J. Nocedal and D. Orban, An nteror pont algorthm for nonlnear optmzaton that combnes lne search and trust

regon steps, Report OTC 6/23, Optmzaton Technology Center, Northwestern Unversty, Evanston, IL 628, USA, 23. To appear n Mathematcal Programmng, Seres A. 31