A UNIFIED APPROACH TO MATHEMATICAL OPTIMIZATION AND LAGRANGE MULTIPLIER THEORY FOR SCIENTISTS AND ENGINEERS

Transcription

1 A UNIFIED APPROACH TO MATHEMATICAL OPTIMIZATION AND LAGRANGE MULTIPLIER THEORY FOR SCIENTISTS AND ENGINEERS RICHARD A. TAPIA Appendix E: Differeniaion in Absrac Spaces I should be no surprise ha he differeniaion of funcionals (real valued funcions) defined on absrac spaces plays a fundamenal role in coninuous opimizaion heory and applicaions. Differenial ools were he major par of he early calculus of variaions and acually i was heir use ha moivaed he erminology calculus of variaions. The ools ha we develop ha allow us o work in he full generaliy of vecor spaces are he so-called one-sided direcional variaion, he direcional variaion, and he direcional derivaive. Their developmen will be our firs ask. In some heory and applicaions we need more powerful differeniaion ools. Towards his end our second ask will be he presenaion of he Gâeaux and Fréche derivaives. Since hese derivaives offer more powerful ools, heir definiion and use will require sronger srucure han jus vecor space srucure. The price ha we have o pay is ha we will require a normed linear space srucure. While normed linear spaces are no as general as vecor spaces, he noion of he Fréche derivaive in a normed linear space is a useful heoreical ool and carries wih i a very saisfying heory. Hence, i is he preferred differeniaion noion in heoreical mahemaics. Moreover, a imes in our opimizaion applicaions we will have o urn o his elegan heory. However, we sress ha he less elegan differeniaion noions will ofen lead us o surprisingly general and useful applicaions; herefore, we promoe and embrace he sudy of all hese heories. We sress ha he various differeniaion noions ha we presen are no made one bi simpler, or proofs shorer, if we resric our aenion o finie dimensional spaces, indeed o IR n. Dimension of spaces is no an inheren par of differeniaion when properly presened. However, we quickly add ha we will ofen urn o IR n for examples because of is familiariy and rich base of examples. The reader ineresed in consuling references on differeniaion will find a hos of reamens in elemenary analysis books in he lieraure. However, we have been guided mosly by Orega and Rheinbold [2]. Our reamen is no he same as heirs bu i is similar. In consideraion of he many sudens ha will read our presenaion, we have included numerous examples hroughou our presenaion and have given hem in unusually complee deail. 1. Inroducion. The noion of he derivaive of a funcion from he reals o he reals has been known and undersood for cenuries. I can be raced back o Leibniz and Newon in he lae 1600 s and, a leas in undersanding, o Ferma and Descares in he early 1600 s. However, he exension from f : IR IR o f : IR n IR. and hen o f : IR n IR m was no handled well. A brief presenaion of his hisorical developmen follows. We moivae his discussion wih he leading quesion: Wha comes firs, he derivaive or he gradien? Keep in mind ha he derivaive should be a linear form, and no a number, as was believed incorrecly for so many years. I should be even more embarrassing ha his belief in he number inerpreaion persised even in he presence of he noions of he direcional derivaive, he Gâeaux derivaive, and he Fréche derivaive, all linear forms ha we sudy in his appendix. I mus have been ha hese laer noions were all viewed as differen animals, whose naural habia was he absrac worlds of vecor spaces and no IR n. Consider a funcion f : IR IR. From elemenary calculus we have he following noion of he derivaive of f a x: f (x) = lim x 0 f(x + x) f(x). (1.1) x If insead we consider f : IR n IR, hen (1.1) no longer makes sense, since we canno form x + x, i.e., we canno add scalars and vecors and we canno divide by vecors. So, hisorically as is so ofen done in mahemaics he problem a hand is reduced o a problem wih known soluion, i.e., o he known one dimensional case, by considering he parial derivaives of f a x i f(x) = lim x 0 f(x 1,..., x i + x,..., x n ) f(x 1,..., x n ), i = 1,..., n. (1.2) x 0

2 The parial derivaives are hen used o form f(x), he so-called gradien of f a x, i.e., 1 f(x) 2 f(x) f(x) =. (1.3). n f(x) Then he gradien is used o build he so-called oal derivaive or differenial of f a x as f (x)(η) = f(x), η. (1.4) These ideas were readily exended o F : IR n IR m by wriing F (x) = (f 1 (x 1,..., x n ),..., f m (x 1,..., x n )) T and defining F (x)(η) = f 1 (x), η. f m (x), η. (1.5) If we inroduce he so-called Jacobian marix, 1 f 1 (x)... n f 1 (x) JF (x) =..., (1.6) 1 f m (x)... n f m (x) hen we can wrie F (x)(η) = JF (x)η. (1.7) This sandard approach is acually backwards and is flawed. I is flawed because i depends on he noion of he parial derivaives which in urn depends on he noion of a naural basis for IR n, and he dependence becomes raher problemaic once we leave IR n. The backwardness follows from he realizaion ha he gradien and hence he Jacobian marix follow from he derivaive and no he oher way around. This will become readily apparen in 2 and 3. Hisorically, he large obsacle o producing saisfacory undersanding in he area of differeniaion came from he failure o view f (x) in (1.1) as he linear form f (x) η evaluaed a η = 1. In IR he number a and he linear form a η are easily idenified, or a leas no separaed. I followed ha hisorically a derivaive was always seen as a number and no a linear form. This led o he noion of parial derivaive, and in urn o he linear form f (x) given in (1.3). This linear form was no seen as a number, hence i was no seen as a derivaive. This moivaed he erminology differenial and a imes oal derivaive, for he linear form defined in (1.3). Much confusion resuled in erms of he usage derivaive, differenial, and oal derivaive. We reinforce our own criical saemens by quoing from he inroducory page o Chaper??, he differenial calculus chaper, in Dieudonne []. Tha presenaion, which hroughou adheres sricly o our general geomeric oulook on analysis, aims a keeping as close as possible o he fundamenal idea of calculus, namely he local approximaion of funcions by linear funcions. In he classical eaching of Calculus, his idea is immediaely obscured by he accidenal fac ha, on a one-dimensional vecor space, here is a one-o-one correspondence beween linear forms and numbers, and herefore he derivaive a a poin is defined as a number insead of a linear form. This slavish subservience o he Shibboleh of numerical inerpreaion a any cos becomes worse when dealing wih funcions of several variables. 2. Basic Differenial Noions. We begin wih a formal definiion of our differenial noions excep for he noion of he Fréche derivaive, i will be presened in 4. 1

3 Definiion 2.1. Consider f : X Y where X is a vecor space and Y is a opological linear space. Given x, η X, if he (one-sided) limi f +(x)(η) = lim 0 f(x + η) f(x) exiss, hen we call f +(x)(η) he righ or forward direcional variaion of f a x in he direcion η. If he (wo-sided) limi f (x)(η) = lim 0 f(x + η) f(x) exiss, hen we call f (x)(η) he direcional variaion of f a x in he direcion η. The erminology forward direcional variaion of f a x, and he noaion f +(x), imply ha f +(x)(η) exiss for all η X wih an analogous saemen for he direcional variaion. If f (x), he direcional variaion of f a x, is a linear form, i.e., f (x)(η) is linear in η, hen we call f (x) he direcional derivaive of f a x. If, in addiion, X and Y are normed linear spaces and f (x), he direcional derivaive of f a x, exiss as a bounded linear form (f (x) L[X, Y ]), hen we call f (x) he Gâeaux derivaive of f a x. Furhermore, we say ha f is direcionally, respecively Gâeaux, differeniable in D X o mean ha f has a direcional, respecively Gâeaux, derivaive a all poins x D. Remark 2.2. Clearly he exisence of he direcional variaion means ha he forward direcional variaion f +(x)(η), and he backward direcional variaion f (x)(η) = lim 0 f(x + η) f(x) exis and are equal. Remark 2.3. In he case where f : IR n IR m he noions of direcional derivaive and Gâeaux derivaive coincide, since in his seing all linear operaors are bounded. See Theorem C.5.4. Remark 2.4. The lieraure abounds wih uses of he erms derivaive, differenial, and variaion wih qualifiers direcional, Gâeaux, or oal. In his conex, we have srongly avoided he use of he confusing erm differenial and he even more confusing qualifier oal. Moreover, we have spen considerable ime and effor in selecing erminology ha we believe is consisen, a leas in flavor, wih ha found in mos of he lieraure. Remark 2.5. I should be clear ha we have used he erm variaion o mean ha he limi in quesion exiss, and have reserved he use of he erm derivaive o mean ha he eniy under consideraion is linear in he direcion argumen. Hence derivaives are always linear forms, variaions are no necessarily linear forms. Remark 2.6. The erm variaion comes from he early calculus of variaions and is usually associaed wih he name Lagrange. In all our applicaions he opological vecor space Y will be he reals IR. We presened our definiions in he more general seing o emphasize ha he definiion of variaion only requires a noion of convergence in he range space. In erms of homogeneiy of our variaions, we have he following properies. Proposiion 2.7. Consider f : X Y where X is a vecor space and Y is a opological vecor space. Also, consider x, η X. Then (i) The forward direcional variaion of f a x in he direcion η is non-negaively homogeneous in η, i.e., f +(x)(αη) = αf +(x)(η) for α 0. (ii) If f +(x)(η) is homogeneous in η, hen f (x)(η) exiss and he wo are equal. Hence, if f +(x)(η) is homogeneous in η, hen f +(x)(η) is he direcional variaion of f a x. (iii) The direcional variaion of f a x in he direcion η is homogeneous in η, i.e, f (x)(αη) = αf (x)(η) for all α IR. Proof. The proofs follow direcly from he definiions and he fac ha f (x)(η) = f +(x)( η). 2 (2.1) (2.2) (2.3)

4 Fig Absolue value funcion We now consider several examples. Figure E.1 shows he graph of he funcion f(x) = x. For his funcion f +(0)(1) = 1 and f (0)(1) = 1. More generally, if X is a normed linear space and we define f(x) = x, hen f +(0)(η) = η and f (0)(η) = η. This shows he obvious fac ha a forward (backward) variaion need no be a direcional variaion. The direcional variaion of f a x in he direcion η, f (x)(η), will usually be linear in η; however, he following example shows ha his is no always he case. Example 2.8. Le f : R 2 R be given by f(x) = x 1x 2 2 x 2 1 +, x = (x 1, x 2 ) 0 and f(0) = 0. Then x2 2 f (0)(η) = η 1η2 2 η η2 2 The following example shows ha he exisnce of he parial derivaives is no a sufficien condiion for he exisence of he direcional variaion. Example 2.9. Le f : R 2 R be given by f(x) = x 1x 2 x 2 1 +, x = (x 1, x 2 ) T 0 and f(0) = 0. x2 2 Then f 1 η 1 η 2 (0)(η) = lim 0 (η1 2 + η2 2 ) exiss if and only if η = (η 1, 0) or η = (0, η 2 ). In 5 we will show ha he exisence of coninuous parial derivaives is sufficien for he exisence of he Gâeaux derivaive, and more. The following observaion, which we sae formally as a proposiion, is exremely useful when one is acually calculaing variaions. Proposiion Consider f : X IR, where X is a real vecor space. Given x, η IR, le φ() = f(x+η). Then φ +(0) = f +(x)(η) and φ (0) = f (x)(η). Proof. The proof follows direcly once we noe ha φ() φ(0) = f(x + η) f(x). Example Given f(x) = x T A x, wih A IR n n and x, η IR n, find f (x)(η). Firs, we define Then, we have φ() = f(x + η)& = x T A x + x T A η + η T A x + 2 η T A η. (2.4) φ () = x T Aη + η T Ax + 2η T Aη, (2.5) 3

5 and by he proposiion φ (0) = x T Aη + η T Ax = f (x)(η). (2.6) Recall ha he ordinary mean-value heorem from differenial calculus says ha given a, b IR if he funcion f : IR IR is differeniable on he open inerval (a, b) and coninuous a x = a and x = b, hen here exiss θ (0, 1) so ha f(b) f(a) = f (a + θ(b a))(b a). (2.7) We now consider an analog of he mean-value heorem for funcions which have a direcional variaion. We will need his ool in proofs in 5. Proposiion Le X be a vecor space and Y a normed linear space. Consider an operaor f : X Y. Given x, y X assume f has a direcional variaion a each poin of he line {x + (y x) : 0 1} in he direcion y x. Then for each bounded linear funcional δ Y (i) δ(f(y) f(x)) = δ[f (x + θ(y x))(y x)], for some 0 < θ < 1. Moreover, (ii) f(y) f(x) sup f (x + θ(y x))(y x). 0<θ<1 (iii) If in addiion X is a normed linear space and f is Gâeaux differeniable on he given domain, hen (iv) f(y) f(x) sup f (x + θ(y x)) y x. 0<θ<1 If in addiion f has a Gâeaux derivaive a an addiional poin x 0 X, hen f(y) f(x) f (x 0 )(y x) sup f (x + θ(y x)) f (x 0 ) y x. 0<θ<1 Proof. Consider g() = δ(f(x + (y x))) for [0, 1]. Clearly g () = δ(f (x + (y x))(y x)). So he condiions of he ordinary mean-value heorem (2.7) are saisfied for g on [0, 1] and g(1) g(0) = g (θ) for some 0 < θ < 1. This is equivalen o (i). Using Corollary C.7.3 of he Hahn-Banach Theorem C.7.2 choose δ Y such ha δ = 1 and δ(f(y) f(x)) = f(y) f(x). Clearly (ii) now follows from (i) and (iii) follows from (ii). Inequaliy (iv) follows from (iii) by firs observing ha g(x) = f(x) f (x 0 )(x) saisfies he condiions of (iii) and hen replacing f in (iii) wih g. While he Gâeaux derivaive noion is quie useful; i has is shorcomings. For example, he Gâeaux noion is deficien in he sense ha a Gâeaux differeniable funcion need no be coninuous. The following example demonsraes his phenomenon. Example Le f : R 2 R be given by f(x) = x3 1 x 2, x = (x 1, x 2 ) 0, and f(0) = 0. Then f (0)(η) = 0 for all η X. Hence f (0) exiss and is a coninuous linear operaor, bu f is no coninuous a 0. We accep his deficiency, bu quickly add ha for our more general noions of differeniaion we have he following limied, bu acually quie useful, form of coninuiy. Orega and Rheinbold [2] call his hemiconinuiy. Proposiion Consider he funcion f : X IR where X is a vecor space. Also consider x, η X. If f (x)(η) (respecively f +(x)(η)) exiss, hen φ() = f(x + η) as a funcion from IR o IR is coninuous (respecively coninuous from he righ) a = 0. Proof. The proof follows by wriing f(x + η) f(x) = (f(x + η) f(x)) and hen leing approach 0 in he appropriae manner. Example Consider f : IR n IR. I should be clear ha he direcional variaion of f a x in he coordinae direcions e 1 = (1, 0,..., 0) T,..., e n = (0,..., 0, 1) T are he parial derivaives i f(x), i = 1,..., n given in (1.2). Nex consider he case where f (x) is linear. Then from Theorem?? we know ha f (x) mus have a represenaion in erms of he inner produc, i.e., here exiss a(x) IR n such ha f (x)(η) = a(x), η η IR n. (2.8) 4

6 Now by he lineariy of f (x)(η) in η = (η 1,..., η n ) T we have f (x)(η) = f (x)(η 1 e η n e n ) = η 1 1 f(x) η n n f(x) = f(x), η (2.9) where f(x) is he gradien vecor of f a x given in (1.3). Hence, a(x), he represener of he direcional derivaive a x, is he gradien vecor a x and we have f (x)(η) = f(x), η η IR n. (2.10) Example Consider F : IR n IR m which is direcionally differeniable a x. We wrie F (x) = (f 1 (x 1,..., x n ),..., f m (x 1,..., x n )) T. By Theorem C 5.4 we know ha here exiss an m n marix A(x) so ha F (x)(η) = A(x)η η IR n. The argumen used in Example 2.15 can be used o show ha f 1 (x), η F (x)(η) =.. f m (x), η I now readily follows ha he marix represener of F (x) is he so-called Jacobian marix, JF (x) = given in (1.5), i.e., F (x)(η) = JF (x)η. Remark We sress ha hese represenaion resuls are sraighforward because we have made he srong assumpion ha he derivaives exis. The challenge is o show ha he derivaives exis from assumpions on he parial derivaives. We do exacly his in The Gradien Vecor. While (1.4) and (2.8) end up a he same place, i is imporan o observe ha in (1.4) he derivaive was defined by firs obaining he gradien, while in (2.8) he gradien followed from he derivaive as is represener. This disincion is very imporan if we wan o exend he noion of he gradien o infinie dimensional spaces. We reinforce hese poins by exending he noion of gradien o inner produc spaces. Definiion 3.1. Consider f : H IR where H is an inner produc space wih inner produc,. Assume ha f is Gâeaux differeniable a x. Then if here exiss f(x) H such ha f (x)(η) = f(x), η for all η H, we call his (necessarily unique) represener of f (x)( ) he gradien of f a x. Remark 3.2. If our inner produc space H is complee, i.e. a Hilber space, hen by he Riesz represenaion heorem (Theorem C.6.1) he gradien f(x) always exiss. However, while he calculaion of f (x) is almos always a sraighforward maer, he deerminaion of f(x) may be quie challenging. Remark 3.3. Observe ha while f : H H we have ha f : H H. I is well known ha in his generaliy f(x) is he unique direcion of seepes ascen of f a x in he sense ha f(x) is he unique soluion of he opimizaion problem maximize subjec o f (x)(η) η f(x). The proof of his fac is a direc applicaion of he Cauchy-Schwarz inequaliy. The well-known mehod of seepes descen considers ieraes of he form x α f(x) for α > 0. This saemen is made o emphasize ha an expression of he form x αf (x) would be meaningless. This noion of gradien in Hilber space is due o Golomb [ ] in Exension of he noion of he gradien o normed linear spaces is an ineresing opic and has been considered by several auhors, see Golomb and Tapia [ ] 1972 for example. This noion is discussed in Chaper 14. 5

7 4. The Fréche Derivaive. The direcional noion of differeniaion is quie general. I allows us o work in he full generaliy of real-valued funcionals defined on a vecor space. In many problems, including problems from he calculus of variaions his generaliy is exacly wha we need and serves us well. However, for cerain applicaions we need a sronger noion of differeniaion. Moreover, many believe ha a saisfacory noion of differeniaion should possess he propery ha differeniable funcions are coninuous. This leads us o he concep of he Fréche derivaive. Recall ha by L[X, Y ] we mean he vecor space or bounded linear operaors from he normed linear space X o he normed linear space Y. We now inroduce he noion of he Fréche derivaive. Definiion 4.1. Consider f : X Y where boh X and Y are normed linear spaces. Given x X, if a linear operaor f (x) L[X, Y ] exiss such ha f(x + x) f(x) f (x)( x) lim = 0, x X, (4.1) x 0 x hen f (x) is called he Fréche derivaive of f a x. The operaor f : X L[X, Y ] which assigns f (x) o x is called he Fréche derivaive of f. Remark 4.2. Conrary o he direcional variaion a a poin, he Fréche derivaive a a poin is by definiion a coninuous linear operaor. Remark 4.3. By (4.1) we mean, given ɛ > 0 here exiss δ > 0 such ha f(x + x) f(x) f (x)( x) ɛ x (4.2) for every x X such ha x δ. Proposiion 4.4. Le X and Y be normed linear spaces and consider f : X Y. If f is Fréche differeniable a x, hen f is Gâeaux differeniable a x and he wo derivaives coincide. Proof. If he Fréche derivaive f (x) exiss, hen by replacing x wih x in (4.1) we have lim 0 f(x + x) f(x) f (x)( x) = 0. Hence f (x) is also he Gâeaux derivaive of f a x. Corollary 4.5. The Fréche derivaive, when i exiss, is unique. Proof. The Gâeaux derivaive defined as a limi mus be unique, hence he Fréche derivaive mus be unique. Proposiion 4.6. If f : X Y is Fréche differeniable a x, hen f is coninuous a x. Proof. Using he lef-hand side of he riangle inequaliy on (4.2) we obain f(x + x) f(x) (ɛ + f (x) ) x. The proposiion follows. Example 4.7. Consider a linear operaor L : X Y where X and Y are vecor spaces. For x, η X we have L(x + η) L(x) = L(η). Hence i is clear ha for a linear operaor L he direcional derivaive a each poin of X exiss and coincides wih L. 1 Moreover, if X and Y are normed linear spaces and L is bounded hen L (x) = L is boh a Gâeaux and a Fréche derivaive. Example 4.8. Le X be he infinie-dimensional normed linear space defined in Example C.5.1. Moreover, le δ be he unbounded linear funcional defined on X given in his example. Since we saw above ha δ (x) = δ, we see ha for all x, δ has a direcional derivaive which is never a Gâeaux or a Fréche derivaive, i.e., i is no bounded. Example 2.9 describes a siuaion where f (0)(η) is a Gâeaux derivaive. Bu i is no a Fréche derivaive since f is no coninuous a x = 0. For he sake of compleeness, we should idenify a siuaion where he funcion is coninuous and we have a Gâeaux derivaive a a poin which is no a Fréche derivaive. Such a funcion is given by (6.4). 1 There is a sligh echnicaliy here, for if X is no a opological linear space, hen he limiing process in he definiion of he direcional derivaive is no defined, even hough i is redundan in his case. Hence, formally we accep he saemen. 6

8 5. Condiions Implying he Exisence of he Fréche Derivaive. In his secion, we invesigae condiions implying ha a direcional variaion is acually a Fréche derivaive and ohers implying he exisence of he Gâeaux or Fréche derivaive. Our firs condiion is fundamenal. Proposiion 5.1. Consider f : X Y where X and Y are normed linear spaces. Suppose ha for some x X, f (x), is a Gâeaux derivaive. Then he following wo saemen are equivalen: (i) f (x) is a Fréche derivaive. (ii) The convergence in he direcional variaion defining relaion (2.2) is uniform wih respec o all η such ha η = 1. Proof. Guided by (4.2) we consider he saemen f(x + x) f(x) f (x)( x) x Consider he change of variables x = η in (5.1) o obain he saemen f(x + η) f(x) ɛ whenever x δ. (5.1) f (x)(η) ɛ whenever η X, η = 1 (5.2) and 0 < δ. Our seps can be reversed so (5.1) and (5.2) are equivalen saemens, and correspond o (i) and (ii) of he proposiion. There is value in idenifying condiions which imply ha a direcional variaion is acually a Gâeaux derivaive as we assumed in he previous proposiion. The following proposiion gives such condiions. Proposiion 5.2. Le X and Y be normed linear spaces. Assume f : X Y has a direcional variaion in X. Assume furher ha (i) for fixed x, f (x)(η) is coninuous in η a η = 0, and (ii) for fixed η, f (x)(η) is coninuous in x for all x X. Then f (x) L[X, Y ], i.e., f (x) is a bounded linear operaor. Proof. By Proposiion?? f (x) is a homogeneous operaor; hence f (x)(0) = 0. By (i) above here exiss r > 0 such ha f (x)(η) 1 whenever η r. I follows ha ( ) f (x)(η) = η rη r f (x) 1 η r η ; hence f (x) will be a coninuous linear operaor once we show i is addiive. Consider η 1, η 2 X. Given ɛ > 0 from Definiion 2.1 here exiss τ > 0 such ha f (x)(η 1 + η 2 ) f (x)(η 1 ) f (x)(η 2 ) [ ] [ ] [ ] f(x + η1 + η 2 ) f(x) f(x + η1 ) f(x) f(x + η2 ) f(x) 3ɛ, whenever τ. Hence f (x)(η 1 + η 2 ) f (x)(η 1 ) f (x)(η 2 ) 1 f( + η 1 + η 2 ) f(x + η 1 ) f(x + η 2 ) f(x) + 3ɛ. By (i) of Proposiion 2.12 and Corollary C.7.3 of he Hahn-Banach heorem here exiss δ Y such ha δ = 1 and 7

9 f(x + η 1 + η 2 ) f(x + η 1 ) f(x + η 2 ) f(x) = δ(f(x + η 1 + η 2 ) f(x + η 1 )) δ(f(x + η 2 ) f(x)) = δ(f (x + η 1 + θ 1 η 2 )(η 2 )) δ(f (x + θ 2 η 2 )(η 2 )) = δ[f (x + η 1 + θ 1 η 2 )(η 2 ) f (x)(η 2 ) +f (x)(η 2 ) f (x + θ 2 η 2 )(η 2 )] f (x + η 1 + θ 1 η 2 )(η 2 ) f (x)(η 2 ) + f (x)(η 2 ) f (x + θ 2 η 2 ), 0 < θ 1, θ 2 < 1, 2 ɛ if is sufficienly small. I follows ha f (x)(η 1 + η 2 ) f (x)(η 1 ) f (x)η 2 5ɛ since ɛ > 0 was arbirary f (x) is addiive. This proves he proposiion. We nex esablish he well-known resul ha a coninuous Gâeaux derivaive is a Fréche derivaive. Proposiion 5.3. Consider f : X Y where X and Y are normed linear spaces. Suppose ha f is Gâeaux differeniable in an open se D X. If f is coninuous a x D, hen f (x) is a Fréche derivaive. Proof. Consider x D a coninuiy poin of f. Since D is open here exiss ˆr > 0 saisfying for η X and η < ˆr, x + η D for (0, 1). Choose such an η. Then by par (i) of Proposiion 2.12 for any δ Y δ[f(x + η) f(x) f (x)(η)] = δ[f (x + θη)(η) f (x)(η)], for some θ (0, 1). By Corollary C.7.3 of he Hahn-Banach heorem we can choose δ so ha f(x + η) f(x) f (x)(η) f (x + θη) f (x) η. Since f is coninuous a x, given ɛ > 0 here exiss r, 0 < r < ˆr, such ha f(x + η) f(x) f (x)(η) ɛ η whenever η r. This proves he proposiion. Consider f : IR n IR. Example 2.9 shows ha he exisence of he parial derivaives does no imply he exisence of he direcional variaion, le alone he Gâeaux or Fréche derivaives. However, i is a well-known fac ha he exisence of coninuous parial derivaives does imply he exisence of he Fréche derivaive. Proofs of his resul can be found readily in advanced calculus and inroducory analysis exs. We found several such proofs, hese proofs are reasonably challenging and no a all sraighforward. For he sake of compleeness of he curren appendix we include an elegan and comparaively shor proof ha we found in he book by Fleming [1]. Proposiion 5.4. Consider f : D IR n IR where D is an open se. Then he following are equivalen. (i) f has coninuous firs-order parial derivaives in D. (ii) f is coninuously Fréche differeniable in D. Proof. [(i) (ii)] We proceed by inducion on he dimension n. For n = 1 he resul is clear. Assume ha he resul is rue in dimension n 1. Our ask is o esablish he resul in dimension n. Has will be used o denoe (n 1)-vecors. Hence ˆx = (x 1,..., x n 1 ) T. Le x 0 = (x 0,..., x 0 n) T be a poin in D and choose δ 0 > 0 so ha B(x 0, δ 0 ) D. Wrie φ(ˆx) = f(x 1,..., x n 1, x 0 n) = f(ˆx, x n 0 ) for ˆx such ha (ˆx, x 0 n) D. Clearly, i φ(ˆx) = i f(ˆx, x 0 n), i = 1,..., n 1 (5.3) 8

10 where i denoes he parial derivaive wih respec o he ih variable. Since i f is coninuous i φ is coninuous. By he inducion hypohesis φ is Fréche differeniable a ˆx 0. Therefore, given ɛ > 0 here exiss δ 1, saisfying 0 < δ 1 < δ 0 so ha φ(ˆx 0 + ĥ) φ(ˆx0 ) φ(ˆx 0 ), ĥ < ɛ ĥ whenever ĥ < δ 1. (5.4) Since n f is coninuous here exiss δ 2 saisfying 0 < δ 2 < δ 1 such ha n f(y) n f(x 0 ) < ɛ whenever y x 0 < δ 2. (5.5) Consider h IR n such ha h < δ 2. By he ordinary mean-value heorem, see (2.7), for some θ (0, 1). Seing gives Since f(x 0 ) = φ(ˆx 0 ) we can wrie f(x 0 + h) φ(ˆx 0 + ĥ) = f(ˆx + ĥ, x0 n + h n ) f(ˆx 0 + ĥ, x0 n) = n f(ˆx 0 + ĥ, x0 n + θh n )h n (5.6) y = (ˆx 0 + ĥ, x0 n + θh n ) (5.7) y x 0 = θ h n < h < δ 2. (5.8) f(x 0 + h) f(x 0 ) = [f(x 0 + h) φ(ˆx 0 + ĥ) + φ(ˆx0 + ĥ) φ(ˆx0 ). (5.9) Now, f(x 0 ), h is a bounded linear funcional in h. Our goal is o show ha i is he Fréche derivaive of f a x 0. Towards his end we firs use (5.6) and (5.7) o rewrie (5.9) as f(x 0 + h) f(x 0 ) = n f(y)h n + φ(ˆx 0 + ĥ). (5.10) Now we subrac f(x 0, h) from boh sides of (5.10) and recall (5.3) o obain f(x 0 + h) f(x 0 ) f(x 0 ), h = [ n f(y)h n n f(x 0 )h n ] + [φ(ˆx 0 + ĥ) φˆx0 φ(ˆx 0 ), ĥ ]. (5.11) Observing ha h n h and ĥ h and calling on (5.4) and (5.5) for bounds we see ha (5.11) readily leads o f(x 0 + h) f(x 0 ) f(x 0 ), h < ɛ h n + ɛ ĥ 2ɛ h (5.12) whenever h < δ 2. This shows ha f is Fréche differeniable a x 0 and f (x 0 )(h) = f(x 0 ), h. (5.13) Since he parials i f are coninuous a x 0, he gradien operaor f is coninuous a x 0 and i follows ha he Fréche derivaive f is coninuous a x 0. [(ii) (i)] Consider x 0 D. Since f is a Fréche derivaive we have he represenaion (5.13) (see (2.10)). Moreover, coninuous parials implies a coninuous gradien and in urn a coninuous derivaive. This proves he proposiion. Remark 5.5. The challenge in he above proof was esablishing lineariy. The proof would have been immediae if we had assumed he exisence of a direcional derivaive. Corollary 5.6. Consider F : D IR n IR m where D is an open se. Then he following are equivalen. (i) F has coninuous firs-order parial derivaives in D. 9

11 (ii) F is coninuously Fréche differeniable in D. Proof. Wrie F (x) = (f 1 (x),..., f m (x)) T. The proof follows from he proposiion by employing he relaionship F (x)(η) = f 1 (x), η. f m (x), η = JF (x)η where JF (x) represens he Jacobian marix of F a x. See Example This proof aciviy requires familiariy wih he maerial discussed in?? of Appendix C. 6. The Chain Rule. The chain rule is a very powerful and useful ool in analysis. We now invesigae condiions which guaranee a chain rule. Proposiion 6.1. (Chain Rule). Le X be a vecor space and le Y and Z be normed linear spaces. Assume: (i) h : X Y has a forward direcional variaion, respecively direcional variaion, direcional derivaive, a x X, and (ii) g : Y Z is Fréche differeniable a h(x) Y. Then f = g h : X Z has a forward direcional variaion, respecively direcional variaion, direcional derivaive, a x X and f +(x) = g (h(x))h +(x). (6.1) If X is also a normed linear space and h (x) is a Gâeaux, respecively Fréche, derivaive, hen f (x) is also a Gâeaux, respecively Fréche, derivaive. Hence Proof. Given x consider η X. Le y = h(x) and y = h(x + η) h(x). Then f(x + η) f(x) = g(y + y) g(y) = g (y)( y) + g(y + y) g(y) g (y)( y) = g (h(x + η) h(x)) (y) + g(y + y) g(y) g (y)( y) y h(x + η) h(x). [ ] f(x + η) f(x) g (h(x))h (x)(η) [ ] g (h(x)) h(x + η) h(x) h (x)(η) + g(y + y) g(y) g (y)( y) h(x + η) h(x) y. (6.2) Observe ha if h has a forward direcional variaion a x, hen alhough h may no be coninuous a x, we do have ha h is coninuous a x in each direcion (see Proposiion 2.14), i.e., y 0 as 0. By leing 0 in (6.2) we see ha f has a forward direcional variaion a x. Clearly, he same holds for wo-sided limis. To see ha variaion can be replaced by derivaive we need only recall ha he composiion of wo linear forms is a linear form. Suppose ha X is also a normed linear space. From (6.1) i follows ha f (x) is a bounded linear form if g (h(x)) and h (x) are bounded linear forms. Hence, f (x) is a Gâeaux derivaive whenever h (x) is a Gâeaux derivaive. Now suppose ha h (x) is a Fréche derivaive. Our ask is o show ha f (x) is Fréche. We do his by esablishing uniform convergence in (6.2) for η such ha η = 1. From Proposiion 4.4 h is coninuous a x. Hence given ɛ > 0 δ > 0 so ha y = h(x + η) h(x) < ɛ 10

12 whenever η = < δ. So as 0, y 0 uniformly in η saisfying η = 1. Using he lef-hand side of he riangle inequaliy we can wrie h(x + η) h(x) h (x)) η + h(x + η) h(x) h (x)(η). (6.3) Since h is Fréche differeniable a x he second erm on he righ-hand side of he inequaliy in (6.3) goes o zero as 0 uniformly for η saisfying η = 1. Hence he erm on he lef-hand side of he inequaliy (6.3) is bounded for small uniformly in η saisfying η = 1. I now follows ha as 0 he expression on he lef-hand side of he inequaliy (6.3) goes o zero uniformly in η saisfying η = 1. This shows by Proposiion 5.1 ha f (x) is a Fréche derivaive. We now give an example where in Proposiions 6.1 he funcion h is Fréche differeniable, he funcion g has a Gâeaux derivaive and ye he chain rule (6.1) does no hold. The requiremen ha g be Fréche differeniable canno be weakened. Example 6.2. Consider g : IR 2 IR defined by g(0) = 0 and for nonzero x = (x 1, x 2 ) T Also consider h : IR 2 IR defined by g(x) = x 2(x x 2 2) 3/2 [(x x2 2 )2 + x 2 (6.4) 2 ]. h(x) = (x 1, x 2 2) T. (6.5) We are ineresed in he composie funcion f : IR 2 IR obained as f(x) = g(h(x)). We can wrie and f(0) = g(h(0)) = g(0) = 0. To begin wih (see Example 2.16) ( ) ( ) h 1 0 η1 (x)(η) = 0 2x 2 η 2 so f(x) = x 2(x x 2 4) 3/2 [(x x2 4 )2 + x 4 2 ] x 0 (6.6) h (0)(η) = ( ) ( ) 1 0 η1 = 0 0 η 2 (6.7) ( ) η1. (6.8) 0 Moreover, h (0) in (6.8) is clearly Fréche since h (x) in (6.7) gives a bounded linear operaor which is coninuous in x (see Proposiion 5.3). Direc subsiuion shows ha Hence g (0) is a Gâeaux derivaive. I follows ha Again direc subsiuion shows ha g (g(η) g(0) g(η) (0)(η) = lim = lim = 0. (6.9) 0 0 g (h(0))(h (0)) = 0. (6.10) f (0)(η) = η3 1η2 2 η (6.11) η4 2 Since (6.10) and (6.11) do no give he same righ-hand side our chain rule fails. I mus be ha g (0) is no a Fréche derivaive. We now demonsrae his fac direcly. A his juncure here is value in invesigaing he coninuiy of g a x = 0. If g is no coninuous a x = 0, we will have demonsraed ha g (0) canno be a Fréche derivaive. On he oher hand, if we show ha g is coninuous a x = 0, hen our example of chain rule failure is even more compelling. We consider showing ha g(x) 0 as x 0. From he definiion of g in (6.4) we wrie g(x) = x 2δ 3 δ 4 + x 2 2 (6.12) 11

13 where x x 2 2 = δ 2. Since δ is he Euclidean norm of (x 1, x 2 ) T a resaemen of our ask is o show ha g(x) 0 as δ 0. We herefore sudy φ : IR IR defined by φ(x) = xδ3 δ 4 + x 2 for x 0. (6.13) Direc calculaions show ha φ(0) = 0, φ(x) > 0 for x > 0, φ (x) > 0 for x < δ 2, φ (x) = 0 for x = δ 2, and φ (x) < 0 for x > δ 2. I follows ha φ has a unique minimum a x = δ 2 and φ(δ 2 ) = δ 2. This means ha g(x) δ 2 x. Hence, g(x) 0 as δ 0 and g is coninuous a x = 0. We encouner his resul wih slighly mixed emoions. While i adds credibiliy o he example, i.e., we have an example where he funcion is coninuous and he Gâeaux derivaive is no a Fréche derivaive, we now have o do more work o show ha g (0) is no a Fréche derivaive. Our only remaining opion is o demonsrae ha he convergence in (6.9) is no uniform wih respec o η saisfying η = 1. Toward his end consider η = (η 1, η 2 ) T such ha η = 1, i.e., η η 2 2 = 1. Then If we choose η 2 =, hen g(η) g(η) = η η2 2. (6.14) = 1 2. (6.15) So for small we can always find an η wih norm one of he form η = (η 1, ). Hence he convergence (6.9) is no uniform wih respec o η saisfying η = 1. The choice η 2 = should follow somewha direcly by reflecing on (6.14); however, his is no he way we found i. Being so pleased wih our opimizaion analysis of he funcion φ in (6.13) we did he same hing wih he funcion given in (6.14). I urns ou ha i is maximized in η 2 wih he choice η 2 =. If any choice should demonsrae he lack of uniform convergence, he maximizer should cerainly do i. The suden should explore our approach. 7. A Schemaic Overview and an Example. We begin by presening our hierarchy of noions of differeniaion schemaically in Figure 7.1. For his purpose i suffices o work wih a funcional f : X IR where X is a vecor space. For a given x X we will consider arbirary η X. We now illusrae hese noions and implicaions wih an imporan example. Example 7.1. Le C 1 [a, b] be he vecor space of all real-valued funcions which are coninuously differeniable on he inerval [a, b]. Suppose f : R 3 R has coninuous parial derivaives wih respec o he second and hird variables. Consider he funcional J : C 1 [a, b] R defined by J(y) = b a f(x, y(x), y (x)) dx. (7.1) We denoe he parial derivaives of f by f 1, f 2, and f 3 respecively. Using he echnique described in Proposiion 2.10 for η C 1 [a, b], we define Then, φ() = J(y + η) = φ () = b a b a f(x, y(x) + η(x), y + η (x))dx. [f 2 (x, y(x) + η(x), y (x) + η (x))η(x) + f 3 (x, y(x) + η(x), y (x) + η (x))η (x)]dx, 12

14 Forward Direcional Variaion X is a vecor space f +(x)(η) Direcional Variaion (wo-sided limi) X is a vecor space f (x)(η) = f +(x)(η) = f (x)(η) Direcional Derivaive X is a vecor space f (x)(η), he direcional variaion is linear in η Gâeaux Derivaive X is a normed linear space f (x)(η) is a direcional derivaive f (x)(η) is bounded in η Fréche Derivaive X is a normed linear space f (x) is a Gâeaux derivaive convergence in he definiion of he Gâeaux variaion 2.2 is uniform wih respec o all η saisfying η = 1. Fig so ha J (y)(η) = φ (0) = b a [f 2 (x, y(x), y (x))η(x) + f 3 (x, y(x), y (x))η (x)]dx. (7.2) I is readily apparen ha he direcional variaion J (y)(η) is acually a direcional derivaive, i.e., J (y)(η) is linear in η. Before we can discuss Gâeaux or Fréche derivaives of J we mus inroduce a norm on C 1 [a, b]. Toward his end le η = max η(x) + max a x b a x b η (x) for each η C 1 [a, b]. Recall ha we have assumed f has coninuous parial derivaives wih respec o he second and hird variables. Also y and y are coninuous on [a, b]. Hence, he composie funcions 13

15 f i (x, y(x), y (x)) i = 2, 3 are coninuous on he compac se [a, b] and herefore aain heir maxima on [a, b]. I follows ha [ ] J (y)(η) (b a) max f 2(x, y(x), y (x)) + max f 3(x, y(x), y (x)) η ; a x b a x b hence J (y) is a bounded linear operaor, and is herefore a Gâeaux derivaive. We now show ha J is he Fréche derivaive. Using he ordinary mean-value heorem for real valued funcions of several real variables allows us o wrie J(y + η) J(y) = = Z b a Z b where 0 < θ(x) < 1. Using (7.2) and (7.3) we wrie J(y + η) J(y) J (y)(η) = Z b a a [f(x, y(x) + η(x), y (x) + η (x)) f(x, y(x), y (x))]dx [f 2(x, y(x) + θ(x)η(x), y (x) + θ(x)η (x)) + f 3(x, y(x) + θ(x)η(x), y (x) + θ(x)η (x), y (x))]dx (7.3) {f 2(x, y(x) + θ(x)η(x), y (x) + θ(x)η(x)) f 2(x, y(x), y (x))]η(x) + [f 3(x, y(x) + θ(x)η(x), y (x) + θ(x)η(x)) f 3(x, y(x), y (x))η (x)]}dx. (7.4) For our given y C 1 [a, b] choose a δ 0 and consider he se D IR 3 defined by D = [a, b] [ y δ 0, y + δ 0] [ y δ 0, y + δ 0]. (7.5) Clearly D is compac and since f 2 and f 3 are coninuous, hey are uniformly coninuous on D. So, given ɛ > 0, by uniform coninuiy of f 2 and f 3 on D here exiss δ > 0 such ha if η < δ hen all he argumens of he funcion f 2 and f 3 in (7.4) are conained in D and J(y + η) J(y) J (y)(η) 2ɛ(b a) η. (7.6) Hence, J (y) is he Fréche derivaive of J a y according o he definiion of he Fréche derivaive, see Definiion 4.1 and Remark 4.3. Alernaively, we can show ha J (y) is he Fréche derivaive by demonsraing ha he map J defined from C 1 [a, b] ino is opological dual space (see C.7) is coninuous a y and hen appealing o Proposiion 5.3. Since we are aemping o learn as much as possible from his example we pursue his pah and hen compare he wo approaches. Familiariy wih C.5 will faciliae he presenaion. From he definiions and (7.2) we obain for y, h C 1 [a, b] J (y + h) J (y) = sup J (y + h)(η) J (y)(η) η =1» (b a) max a x b y (x) + h (x)) f 2(x, y(x), y (x)) + max a x b y (x) + h (x)) f 3(x, y(x), y (x)). (7.7) Observe ha he uniform coninuiy argumen used o obain (7.4) and (7.6) can be applied o (7.7) o obain J (y + h) J (y) 2ɛ(b a) whenever h δ. Hence J is coninuous a y and J (y) is a Fréche derivaive. We now commen on wha we have learned from our wo approaches. Clearly, he second approach was more direc. Le s explore why his was so. For he purpose of adding undersanding o he role ha uniform coninuiy played in our applicaion we offer he following observaion. Consider he funcion g(y) = f i(x, y(x), y (x)) (for i = 1 or i = 2) viewed as a funcion of y C 1 [a, b] ino C 0 [a, b], he coninuous funcions on [a, b] wih he sandard max norm. I is exacly he uniform coninuiy of f i on he compac se D IR 3 ha enables he coninuiy of g a y. 14

16 Observe ha in our firs approach we worked wih he definiion of he Fréche derivaive which enailed working wih he expression J(y + η) J(y) J (y)(η). (7.8) The expression (7.8) involves funcions of differing differenial orders; hence can be problemaic. However, in our second approach we worked wih he expression J (y + h) J (y) (7.9) which involves only funcions of he same differenial order. This should be less problemaic and more direc. Now, observe ha i was he use of he ordinary mean-value heorem ha allowed us o replace (7.8) wih (J (y + θη) J (y))(η). (7.10) Clearly (7.10) has he same flavor as (7.9), So i should no be a surprise ha he second half of our firs approach and our second approach were essenially he same. However, he second approach gave us more for our money; in he process we also esablished he coninuiy of he derivaive. Our summarizing poin is ha if one expecs J o also be coninuous, as we should have here in our example, i is probably wise o consider working wih Proposiion 5.3 insead of working wih he definiion and Proposiion 5.1 when aemping o esablish ha a direcional derivaive is a Fréche derivaive. 8. The Second Direcional Variaion. Again consider f : X Y where X is a vecor space and Y is a opological vecor space. As we have shown his allows us sufficien srucure o define he direcional variaion. We now urn our aenion owards he ask of defining a second variaion. I is mahemaically saisfying o be able o define a noion of a second derivaive as he derivaive of he derivaive. If f is direcionally differeniable in X, hen we know ha f : X [X, Y ], where [X, Y ] denoes he vecor space of linear operaors from X ino Y. Since in his generaliy, [X, Y ] is no a opological vecor space we can no consider he direcional variaion of he direcional derivaive as he second direcional variaion. If we require he addiional srucure ha X and Y are normed linear spaces and f is Gâeaux differeniable, we have f : X L[X, Y ] where L[X, Y ] is he normed linear space of bounded linear operaors from X ino Y described in Appendix??. We can now consider he direcional variaion, direcional derivaive, and Gâeaux derivaive. However, requiring such srucure is excessively resricive for our vecor space applicaions. Hence we will ake he approach used successfully by workers in he early calculus of variaions. Moreover, for our purposes we do no need he generaliy of Y being a general opological vecor space and i suffices o choose Y = IR. Furhermore, because of (ii) of Proposiion 2.7 we will no consider he generaliy of one-sided second variaions. Definiion 8.1. Consider f : X IR, where X is a real vecor space and x X. For η 1, η 2 X by he second direcional variaion of f a x in he direcions η 1 and η 2, we mean f (x)(η 1, η 2) = f (x + η 1)(η 2) f (x)(η 2) lim, (8.1) 0 whenever hese direcional derivaives and he limi exis. When he second direcional variaion of f a x is defined for all η 1, η 2 X we say ha f has a second direcional variaion a x. Moreover, if f (x)(η 1, η 2) is bilinear, i.e., linear in η 1 and η 2, we call f (x) he second direcional derivaive of f a x and say ha f is wice direcionally differeniable a x. We now exend our previous observaion o include he second direcional variaion. Proposiion 8.2. Consider a funcional f : X IR. Given x, η X, le Then, and φ() = f(x + η). φ (0) = f (x)(η) (8.2) φ (0) = f (x)(η, η) (8.3) 15

17 Moreover, given x, η 1, η 2 X, le ω() = f (x + η 1)(η 2). Then, ω (0) = f (x)(η 1, η 2) (8.4) Proof. The proof follows direcly from he definiions. Example 8.3. Given f(x) = x T A x, wih A IR n n and x, η IR n, find f (x)(η, η). From Example 2.11, we have φ () = x T Aη + η T Ax + 2η T Aη. Thus, and Moreover, leing we have φ () = 2 η T Aη, φ (0) = 2 η T Aη = f (x)(η, η). ω() = (x + η 1) T Aη 2 + η T 2 A(x + η 1), ω (0) = η T 1 Aη 2 + η T 2 Aη 1 = f (x)(η 1, η 2). In essenially all our applicaions we will be working wih he second direcional variaion only in he case where he direcions η 1, and η 2 are he same. The erminology second direcional variaion a x in he direcion η will be used o describe his siuaion. Some auhors only consider he second variaion when he direcions η 1 and η 2 are he same. They hen use (8.4) as he definiion of f (x)(η, η). While his is raher convenien and would acually suffice in mos of our applicaions, we are concerned ha i masks he fac ha he second derivaive a a poin is inherenly a funcion of wo independen variables wih he expecaion ha i be a symmeric bilinear form under reasonable condiions. This will become apparen when we sudy he second Fréche derivaive in laer secions. The definiion (7.10) reains he flavor of wo independen argumens. The second direcional variaion is usually linear in each of η 1 and η 2, and i is usually symmeric, in he sense ha f (x)(η 1, η 2) = f (x)(η 2, η 1). This is he case in Example 8.3 above even when A is no symmeric. However, his is no always he case. Example 8.4. Consider he following example. Define f : IR 2 IR 3 by f(0) = 0 and by f(x) = x1x2(x2 1 x 2 2) x x2 2 for x 0. Clearly, f (0)(η) = 0 η IR 2. Now, for x 0, he parial derivaives of f are clearly coninuous. Hence f is Fréche differeniable for x 0 by Proposiion 5.3. I follows ha we can wrie A direc calculaion of he wo parial derivaives allows us o show ha f (x)(η) = f(x), η. (8.5) f (x)(η) = f(x), η = f(x), η = f (x)(η). (8.6) Using (8.6) we have ha» f f (u)(v) f (0)(v) (0)(u, v) = lim = f (u)(v). 0 Hence, we can wrie f (0)(u, v) = f(u), v. (8.7) Again, appealing o he calculaed parial derivaives we see ha f (0) as given by (8.7) is no linear in u and is no symmeric. We conclude ha f (0) as a second direcional variaion is neiher symmeric nor a second direcional derivaive (bilinear). However, as we shall demonsrae in 10 he second Fréche derivaive a a poin is always a symmeric bilinear form. 16

18 In he following secion of his appendix we will define he second Fréche derivaive. A cerain amoun of saisfacion will be gained from he fac ha, unlike he second direcional variaion (derivaive), i is defined as he derivaive of he derivaive. However, in order o make our presenaion complee, we firs define he second Gâeaux derivaive. I oo will be defined as he derivaive of he derivaive, and perhaps represens he mos general siuaion in which his can be done. We hen prove a proposiion relaing he second Gâeaux derivaive o Fréche differeniabiliy. Afer several examples concerning Gâeaux differeniabiliy in IR n, we close he secion wih a proposiion giving several basic resuls in IR n. Definiion 8.5. Le X and Y be normed linear spaces. Consider f : X Y. Suppose ha f is Gâeaux differeniable in an open se D X. Then by he second Gâeaux derivaive of f we mean f : D X L[X, L[X, Y ]], he Gâeaux derivaive of he Gâeaux derivaive f : D X L[X, Y ]. As we show in deail in he nex secion, L[X, L[X, Y ]] is isomerically isomorphic o [X 2, Y ] he bounded bilinear forms defined on X ha map ino Y. Hence he second Gâeaux derivaive can be viewed naurally as a bounded bilinear form. The following proposiion is essenially Proposiion 5.2 resaed o accommodae he second Gâeaux derivaive. Proposiion 8.6. Le X and Y be normed linear spaces. Suppose ha f : X Y has a firs and second Gâeaux derivaive in an open se D X and f is coninuous a x D. Then boh f (x) and f (x) are Fréche derivaives. Proof. By coninuiy f (x) is a Fréche derivaive (Proposiion 5.1). Hence f is coninuous a x (Proposiion 5.2) and is herefore he Fréche derivaive a x (Proposiion 5.1). Some concern immediaely arises. According o our definiion given in he nex secion, he second Fréche derivaive is he Fréche derivaive of he Fréche derivaive. So i seems as if we have idenified a slighly more general siuaion here where he second Fréche derivaive could be defined as he Fréche derivaive of he Gâeaux derivaive f : X X. However, his siuaion is no more general. For if f is he Fréche derivaive of f, hen by Proposiion 4.4 f is coninuous and, in urn by Proposiion 5.2 f is a Fréche derivaive. So, a second Gâeaux derivaive which is a Fréche derivaive is a second Fréche derivaive according o he definiion ha will be given. Recall ha if f : IR n IR, hen he noions of direcional derivaive and Gâeaux derivaive coincide since in heis seing all linear operaors are bounded, see??. Consider f : IR n IR. Suppose ha f is Gâeaux differeniable. Then we showed in he Example 2.15 ha he gradien vecor is he represener of he Gâeaux derivaive, i.e., for x, η IR n f (x)(η) = f(x), η. Now suppose ha f is wice Gâeaux differeniable. The firs hing ha we observe, via Example 2.16, is ha he Hessian marix 2 f(x) is he Jacobian marix of he gradien vecor; specifically 2 1 1f(x)... 3 n 1f(x) J f(x) = (8.8) 1 nf(x)... n n(x) Nex, observe ha f f (x + η 1)(η 2) f (x)(η 2) (x)(η 1, η 2) = lim 0 f(x + η 1), η 2 f(x), η 2 = lim 0 f(x + η 1) f(x) = lim, η 0 2 = 2 f(x)η 1, η 2 = η T 2 2 f(x)η 1. Following in his direcion consider wice Gâeaux differeniable f : IR n IR m. Turning o he componen funcions we wrie f(x) = (f 1(x)...., f m(x)) T. Then, and f (x)(η) = Jf(x)η = ( f 1(x), η,..., f m(x), η ) T, f (x)(η 1, η 2) = ( 2 f 1(x)η 1, η 2,..., 2 f m(x)η 1, η 2 ) T = (η T 1 2 f 1(x)η 2,..., η T 1 2 f m(x)η 2) T. (8.9) This discussion leads naurally o he following proposiion. Proposiion 8.7. Consider f : D IR n IR m where D is an open se in IR n. Suppose ha f is wice Gâeaux differeniable in D. Then for x D (i) f (x) is symmeric if and only if he Hessian marices of he componen funcions a x, 2 f 1(x),..., 2 f m(x), are symmeric. 17

19 (ii) f is coninuous a x if and only if all second-order parial derivaives of he componen funcions f 1,..., f m are coninuous a x. (iii) If f is coninuous a x, hen f (x) and f (x) are Fréche derivaives and f (x) is symmeric. Proof. The proof of (i) follows direcly from (8.9) since second-order parial derivaives are firs-order parial derivaives of he firs-order parial derivaives. Par (ii) follows from Corollary 5.6. Par (iii) follows Proposiion 5.3, Proposiion 4.6, Proposiion 5.3 again, and he ye o be proved Proposiion 9.4, all in ha order. Hence he Hessian marix is he represener, via he inner produc, of he bilinear form f (x). So, he gradien vecor is he represener of he firs derivaive and he Hessian marix is he represener of he second derivaive. This is saisfying. A his poin we pause o collec hese observaions in proposiion form, so hey can be easily referenced. Proposiion 8.8. Consider f : IR n IR. (i) If f is direcionally differeniable a x, hen f (x) = f(x), η where f(x) is he gradien vecor of f a x defined in (1.3). η IR n (ii) If f is direcionally differeniable in a neighborhood of x and wice direcionally differeniable a x, hen f (x)(η 1, η 2) = 2 f(x)η 1, η 2 where 2 f(x) is he Hessian marix of f a x, described in (8.8). η 1, η 2 X Remark 8.9. I is worh noing ha he Hessian marix can be viewed as he Jacobian marix of he gradien vecor funcion. Moreover, in his conex our direcional derivaives are acually Gâeaux derivaives, since we are working in IR n. Finally, in his generaliy we can no guaranee ha he Hessian marix is symmeric. This would follow if f (x) was acually a Fréche derivaive, see Proposiion The following proposiion plays a srong role in Chaper?? where we develop our fundamenal principles for second-order necessiy and sufficiency. Our presenaion follows ha of Orega and Rheinbold []. Proposiion Assume ha F : D X Y, where X and Y are normed linear spaces and D is an open se in X, is Gâeaux differeniable in D and has a second Gâeaux derivaive a x D. Then, 1 (i) lim 0 ˆF (x + h) F (x) F (x)(h) 1 F (x)(h, h) = for an h X. Moreover, if f (x) is a Fréche derivaive, hen 1 (ii) lim h 0 h ˆF (x + h) F (x) F (x)(h) 1 F (x)(h, h). 2 2 Proof. For given h X and sufficiency small we have ha x + h D and is well-defined. Clearly, G() = F (x + h) F (x) F (x)(h) 1 2 F (x)(h, h) G () = F (x + h)(h) F (x)(h) F (x)(h, h). From he definiion of f (x)(h, h) we have ha given ε > 0 δ > 0 so ha when < δ. Now, from (iii) of Proposiion 2.12 G () ε G() = G() G(0) sup G (θ) ε 2 0 θ 1 whenever < δ. Since ε > 0 was arbirary we have esablished (i). We herefore urn our aenion o (ii) by leing R(h) = F (x + h) F (x) F (x)(h) 1 2 F (x)(h, h). Noice ha R is well-defined for h sufficienly small and is Gâeaux differeniable in a neighborhood of h = 0, since F is defined in a neighborhood of x. I is acually Fréche differeniable in his neighborhood, bu we only need Gâeaux differeniabiliy. Since F (x) is a Fréche derivaive, by definiion, given ε > 0 δ > 0 so ha provided h < δ. As before, (iii) of Proposiion 2.12 gives R (h) = F (x + h) F (x) F (x)(h, ) ε h R(h) = R(h) R(0) sup R (θh) h ε h 2 0 θ 1 provided h < δ. Since ε was arbirary we have esablished (ii) and he proposiion. 18

20 9. Higher Order Fréche Derivaives. Le X and Y be normed linear spaces. Suppose f : X Y is Fréche differeniable in D an open subse of X. Le L 1[X, Y ] denoe L[X, Y ] he normed linear space of bounded linear operaors from X ino Y. Recall ha f : D X L 1[X, Y ]. Consider f, he Fréche derivaive of he Fréche derivaive of f. Clearly f : D X L 1 [X, L 1[X, Y ]]. (9.1) In general le L n[x, Y ] denoe L 1[X, L n 1[X, Y ]], n = 2, 3,. Then f (n), n = 2, 3,..., he n-h Fréche derivaive of f is by definiion he Fréche derivaive of f (n 1), he (n 1)-s Fréche derivaive of f. Clearly f (n) : D X L n[x, Y ]. I is no immediaely obvious how o inerpre he elemens of L n[x, Y ]. The following inerpreaion is very helpful. Recall ha he Caresian produc of wo ses U and V is by definiion U V = {(u, v) : u U, v V }. Also by U 1 we mean U and by U n we mean U U n 1, n = 2, 3,. Clearly U n is a vecor space in he obvious manner whenever U is a vecor space. An operaor K : X n Y is said o be an n-linear operaor from X ino Y if i is linear in each of he n variables, i.e., for real α and β K(x 1,, αx i + βx i,, x n) = αk(x 1,, x i,, x n) + βk(x 1,, x i,, x n), i = 1,, n. The n-linear operaor K is said o be bounded if here exiss M > 0 such ha K(x 1,, x n) M x 1 x 2 x n, for all (x 1,, x n) X n. (9.2) The vecor space of all bounded n-linear operaors from X ino Y becomes a normed linear space if we define K o be he infimum of all M saisfying (9.2). This normed linear space we denoe by [X n, Y ]. Clearly by a 1-linear operaor we mean a linear operaor. Also a 2-linear operaor is usually called a bilinear operaor or form. The following proposiion shows ha he spaces L n[x, Y ] and [X n, Y ] are essenially he same excep for noaion. Proposiion 9.1. The normed linear spaces L n[x, Y ] and [X n, Y ] are isomerically isomorphic. Proof. These wo spaces are isomorphic if here exiss T n : L n[x, Y ] [X n, Y ] which is linear, one-one, and ono. The isomorphism is an isomery if i is also norm preserving. Clearly Tn 1 will have he same properies. Since L 1[X, Y ] = [X 1, Y ], le T 1 be he ideniy operaor. Assume we have consruced T n 1 wih he desired properies. Define T n as follows. For W L n[x, Y ] le T n(w ) be he n-linear operaor from X ino Y defined by T n(w )(x 1,, x n) = T n 1(W (x 1))(x 2,, x n) for (x 1,, x n) X n. Clearly T n(w ) W ; hence T n(w ) [X n, Y ]. Also T n is linear. If U [X n, Y ], hen for each x X le W (x) = T 1 n 1 (U(x,,, )). I follows ha W : X Ln 1[X, Y ] is linear and W U ; hence W L n[x, Y ] and T n(w ) = U.This shows ha T n is ono and norm preserving. Clearly a linear norm preserving operaor mus be one-one. This proves he proposiion. Remark 9.2. The Proposiion impacs he Fréche derivaive in he following manner. The n-h Fréche derivaive of f : X Y a a poin can be viewed as a bounded n-linear operaor from X n ino Y. I is quie saisfying, ha wihou asking for coninuiy of he n-h Fréche derivaive, f n : X [X n, Y ] we acually have symmery as is described in he following proposiion. Proposiion 9.3. If f : X Y is n imes Fréche differeniable in an open se D X, hen he n-linear operaor f n (x) is symmeric for each x D. A proof of his symmery proposiion can be found in Chaper 8 of Dieudonné []. There he resul is proved by inducion on n. The inducion process is iniiaed by firs esablishing he resul for n = 2. We will include ha proof because i is insrucive, ingenious, and elegan. However, he version of he proof presened in Dieudonné is difficul o follow. Hence, we presen Orega and Rheinbold s [] adapaion of ha proof. Proposiion 9.4. Suppose ha f : X Y is Fréche differeniable in an open se D X and has a second Fréche derivaive a x D. Then he bilinear operaor f (x) is symmeric. Proof. The second Fréche derivaive is by definiion he Fréche derivaive of he Fréche derivaive. Hence applying (4.2) for second Fréche derivaives we see ha for given ɛ > 0 δ > 0 so ha y D and f (y) f (x) f (x)(y x, ) ɛ x y (9.3) whenever x y < δ. Consider u, v B(0, δ ). The funcion G : [0, 1] Y defined by 2 G() = f(x + u + v) f(x + u) 19