College of Management, NCTU Operaton Research II Sprng, 009 Chap Nonlnear Programmng General Form of Nonlnear Programmng Problems Ma f( S.T. g ( b for =,, m 0 No algorthm that wll solve every specfc problem fttng ths format s avalable. An Eample The Product-M Problem wth Prce Elastcty The amount of a product that can be sold has an nverse relatonshp to the prce charged. That s, the relatonshp between demand and prce s an nverse curve. The frm s proft from producng and sellng unts s the sales revenue p( mnus the producton costs. That s, P( = p( c. If each of the frm s products has a smlar proft functon, say, P ( for producng and sellng unts of product, then the overall obectve functon s n f( = P (, a sum of nonlnear functons. = Nonlneartes also may arse n the g ( constrant functon. Chap-
College of Management, NCTU Operaton Research II Sprng, 009 An Eample The Transportaton Problem wth Volume Dscounts Determne an optmal plan for shppng goods from varous sources to varous destnatons, gven supply and demand constrants. In actualty, the shppng costs may not be fed. Volume dscounts sometmes are avalable for large shpments, whch cause a pecewse lnear cost functon. Graphcal Illustraton of Nonlnear Programmng Problems Ma Z = 3 + 5 S.T. 4 9 + 5 6, 0 The optmal soluton s no longer a CPF anymore. (Sometmes, t s; sometmes, t sn t. But, t stll les on the boundary of the feasble regon. We no longer have the tremendous smplfcaton used n LP of lmtng the search for an optmal soluton to ust the CPF solutons. What f the constrants are lnear; but the obectve functon s not? Chap-
College of Management, NCTU Operaton Research II Sprng, 009 Ma Z = 6 9 + 8 3 S.T. 4 3 + 8, 0 What f we change the obectve functon to 54 9 + 78 3 The optmal soluton les nsde the feasble regon. That means we cannot only focus on the boundary of feasble regon. We need to look at the entre feasble regon. The local optmal needs not to be global optmal--complcate further Chap-3
College of Management, NCTU Operaton Research II Sprng, 009 Nonlnear programmng algorthms generally are unable to dstngush between a local optmal and a global optmal. It s desred to know the condtons under whch any local optmal s guaranteed to be a global optmal. If a nonlnear programmng problem has no constrants, the obectve functon beng concave (conve guarantees that a local mamum (mnmum s a global mamum (mnmum. What s a concave (conve functon? A functon that s always curvng downward (or not curvng at all s called a concave functon. A functon s always curvng upward (or not curvng at all, t s called a conve functon. Ths s nether concave nor conve. Defnton of concave and conve functons of a sngle varable A functon of a sngle varable f ( s a conve functon, f for each par of values of, say, and ( <, f [ λ + ( λ ] λf ( + ( λ f ( for all value of λ such that 0 < λ <. It s a strctly conve functon f can be replaced by <. It s a concave functon f ths statement holds when s replaced by (> for the case of strct concave. Chap-4
College of Management, NCTU Operaton Research II Sprng, 009 Chap-5 The geometrc nterpretaton of concave and conve functons. How to udge a sngle varable functon s conve or concave? Consder any functon of a sngle varable f( that possesses a second dervatve at all possble value of. Then f( s conve f and only f 0 ( d f d for all possble value of. concave f and only f 0 ( d f d for all possble value of. How to udge a two-varables functon s conve or concave? If the dervatves est, the followng table can be used to determne a two-varable functon s concave of conve. (for all possble values of and Quantty Conve Concave, (, (, ( f f f 0 0, ( f 0 0, ( f 0 0
College of Management, NCTU Operaton Research II Sprng, 009 Eample: f (, = + How to udge a mult-varables functon s conve or concave? The sum of conve functons s a conve functon, and the sum of concave functons s a concave functon. Eample: f(,, 3 = 4 ( 3 = [4 ] + [ ( 3 ] If there are constrants, then one more condton wll provde the guarantee, namely, that the feasble regon s a conve set. Conve set A conve set s a collecton of ponts such that, for each par of ponts n the collecton, the entre lne segment onng these two ponts s also n the collecton. In general, the feasble regon for a nonlnear programmng problem s a conve set whenever all the g ( (for the constrants g ( b are conve. Ma Z = 3 + 5 S.T. 4 9 + 5 6, 0 Chap-6
College of Management, NCTU Operaton Research II Sprng, 009 What happens when ust one of these g ( s a concave functon nstead? Ma Z = 3 + 5 S.T. 4 4 8 + 4 49, 0 The feasble regon s not a conve set. Under ths crcumstance, we cannot guarantee that a local mamum s a global mamum. Condton for local mamum = global mamum (wth g ( b constrants. To guarantee that a local mamum s a global mamum for a nonlnear programmng problem wth constrant g ( b and 0, the obectve functon f( must be a concave functon and each g ( must be a conve functon. Such a problem s called a conve programmng problem. One-Varable Unconstraned Optmzaton The dfferentable functon f( to be mamzed s concave. The necessary and suffcent condton for = * to be optmal (a global ma s df d = 0, at = *. It s usually not very easy to solve the above equaton analytcally. The One-Dmensonal Search Procedure. Fnng a sequence of tral solutons that leads toward an optmal soluton. Usng the sgns of dervatve to determne where to move. Postve dervatve ndcates that * s greater than ; and vce versa. Chap-7
College of Management, NCTU Operaton Research II Sprng, 009 The Bsecton Method Intalzaton: Select ε (error tolerance. Fnd an ntal (lower bound on * and (upper bound on * by nspecton. Set the ntal tral soluton Iteraton: df ( Evaluate at d df ( If 0, reset d df ( If 0, reset d Select a new =. =. =. + =. + =. Stoppng Rule: If ε, so that the new Otherwse, perform another teraton. * must be wthn ε of, stops. Eample: Ma f( = 3 4 6 df(/d New f ( 0 3 4.09 0.75 0.875 7.8439 4 -.9 0.75 0.875 0.85 7.867 5.3 0.85 0.875 0.84375 7.889 6-0.34 0.85 0.84375 0.885 7.885 7 0.5 0.885 0.84375 0.8359375 7.8839 Chap-8
College of Management, NCTU Operaton Research II Sprng, 009 Newton s Method The bsecton method converges slowly. Only take the nformaton of frst dervatve nto account. The basc dea s to appromate f( wthn the neghborhood of the current tral soluton by a quadratc functon and then to mamze (or mnmze the appromate functon eactly to obtan the new tral soluton. Ths appromatng quadratc functon s obtaned by truncatng the Taylor seres after the second dervatve term. f ( ( + f ( + f ( ( + + ( + f Ths quadratc functon can be optmzed n the usual way by settng ts frst dervatve to zero and solvng for +. Thus, f ( + =. f ( Stoppng Rule: If ε, stop and output +. + Eample: Ma f( = 3 4 6 (same as the bsecton eample f ( + = = f ( Select ε = 0.0000, and choose =. Iteraton f ( f ( f ( + 3 0.84003 7.8838-0.35-55.79 0.83763 4 0.83763 7.8839-0.0006-54.790 0.8376 Multvarable Unconstraned Optmzaton Usually, there s no analytcal method for solvng the system of equatons gven by settng the respectve partal dervatves equal to zero. Thus, a numercal search procedure must be used. Chap-9
College of Management, NCTU Operaton Research II Sprng, 009 The Gradent Search Procedure (for multvarable unconstraned mamzaton problems The goal s to reach a pont where all the partal dervatves are 0. A natural approach s to use the values of the partal dervatves to select the specfc drecton n whch to move. The gradent at pont = s f f f f ( = (,,..., at =. n The drecton of the gradent s nterpreted as the drecton of the drected lne f f f segment from the orgn to the pont (,,...,, whch s the drecton of n changng that wll mamze f( change rate. However, normally t would not be practcal to change contnuously n the drecton of f (, because ths seres of changes would requre contnuously f reevaluatng the and changng the drecton of the path. A better approach s to keep movng n a fed drecton from the current tral soluton, not stoppng untl f( stops ncreasng. The stoppng pont would be the net tral soluton and reevaluate gradent. The gradent would be recalculated to determne the new drecton n whch to move. Reset = + t * f (, where t * s the postve value that mamzes f( +t * f( = The teratons contnue untl f ( = 0 wth a small tolerance ε. Summary of the Gradent Search Procedures Intalzaton: Select ε and any ntal tral soluton. Go frst to the stoppng rule. Step : Epress f( +t f( f as a functon of t by settng = + t(, for = =,,, n, and then substtutng these epressons nto f(. Step : Use the one-dmensonal search procedure to fnd t = t* that mamzes f( +t f( over t 0. Step 3: Reset = + t * f(. Then go to the stoppng rule. Chap-0
College of Management, NCTU Operaton Research II Sprng, 009 Stoppng Rule: Evaluate f( at =. Check f f ε, for all =,,, n. If so, stop wth the current as the desred appromaton of an optmal soluton *. Otherwse, perform another teraton. Eample for multvarate unconstrant nonlnear programmng Ma f( = + f f =, = + 4 We verfy that f( s. Suppose pck = (0, 0 as the ntal tral soluton. f (0,0 = Iteraton : = (0, 0 + t(0, = (0, t f ( + t f ( = f(0, t = Iteraton : = (0, / + t(, 0 = (t, / Usually, we wll use a table for convenence purpose. Chap-
College of Management, NCTU Operaton Research II Sprng, 009 Iteraton f ( ( + t f f ( + t f ( * t + t * f ( For mnmzaton problem We move n the opposte drecton. That s = t * f(. Another change s t = t * that mnmze f( t f( over t 0 Necessary and Suffcent Condtons for Optmalty (Mamzaton Problem Necessary Condton Also Suffcent f: One-varable unconstraned = 0 d Multvarable unconstraned f = 0 (=,, n f( concave General constraned problem KKT condtons f( s concave and g ( s conve The Karush-Kuhn-Tucker (KKT Condtons for Constraned Optmzaton Assumed that f(, g (, g (,, g m ( are dfferentable functons. Then * = ( *, *,, n * can be an optmal soluton for the nonlnear programmng problem only f there est m numbers u, u,, u m such that all the followng KKT condtons are satsfed: m f g ( u 0, at = *, for =,,, n = m * f g ( ( u = 0, at = *, for =,,, n = (3 g ( * b 0, for =,,, m (4 u [g ( * b ] = 0, for =,,, m Chap-
College of Management, NCTU Operaton Research II Sprng, 009 (5 * 0, for =,,, m (6 u 0, for =,,, m Corollary of KKT Theorem (Suffcent Condtons Note that satsfyng these condtons does not guarantee that the soluton s optmal. Assume that f( s a concave functon and that g (, g (,, g m ( are conve functons. Then * = ( *, *,, n * s an optmal soluton f and only f all the KKT condtons are satsfed. An Eample Ma f( = ln( + + S.T. + 3, 0 n = ; m = ; g ( = + s conve; f( s concave.. ( = u 0 +. ( = ( u 0 + =. ( = u 0. ( = ( u = 0 3. + 3 0 4. u ( + 3 = 0 5. 0, 0 6. u 0 Therefore, There ests a u = such that = 0, = 3, and u = satsfy KKT condtons. The optmal soluton s (0, 3. How to solve the KKT condtons Sorry, there s no easy way. In the above eample, there are 8 combnatons for ( 0, ( 0, and u ( 0. Try each one untl fnd a ft one. What f there are lots of varables? Let s look at some easer (specal cases. Chap-3
College of Management, NCTU Operaton Research II Sprng, 009 Quadratc Programmng Ma f( = c / T Q S.T. A b 0 The obectve functon s f( = c / T Q = n c = = = n n q. The q are elements of Q. If =, then =, so /q s the coeffcent of. If, then /(q + q = q, so q s the coeffcent for the product of and (snce q = q. An eample Ma f(, = 5 + 30 + 4 4 S.T. + 30, 0 The KKT condtons for the above quadratc programmng problem.. ( = 5 + 4 4 u 0. ( = (5 + 4 4 u = 0. ( = 30 + 4 8 u 0. ( = (30 + 4 8 u = 0 3. + 30 0 4. u ( + 30 = 0 5. 0, 0 6. u 0 Chap-4
College of Management, NCTU Operaton Research II Sprng, 009 Introduce slack varables (y, y, and v for condton (=, (=, and 3.. ( = 4 + 4 u + y = 5. ( = 4 8 u + y = 30 3. + + v = 30 Condton ( = can be reepressed as. ( = y = 0 Smlarly, we have. ( = y = 0 4. u v = 0 For each of these pars (, y, (, y, (u, v the two varables are called complementary varables, because only one of them can be nonzero. Combne them nto one constrant y + y + u v = 0, called the complementary constrant. Rewrte the whole condtons 4 4 + u y = 5 4 + 8 + u y = 30 + + v = 30 y + y + u v = 0 0, 0, u 0, y 0, y 0, v 0 Ecept for the complementary constrant, they are all lnear constrants. For any quadratc programmng problem, ts KKT condtons have ths form Q + A T u y = c T A + v = b 0, u 0, y 0, v 0 T y + u T v = 0 Assume the obectve functon (of a quadratc programmng problem s concave and constrants are conve (they are all lnear. Thus, s optmal f and only f there est values of y, u, and v such that all four vectors together satsfy all these condtons. The orgnal problem s thereby reduced to the equvalent problem of fndng a feasble soluton to these constrants. These constrants are really the constrants of a LP ecept the complementary constrant. Why don t we ust modfy the Smple Method? Chap-5
College of Management, NCTU Operaton Research II Sprng, 009 The Modfed Smple Method The complementary constrant mples that t s not permssble for both complementary varables of any par to be basc varables. The problem reduces to fndng an ntal BF soluton to any lnear programmng problem that has these constrants, subect to ths addtonal restrcton on the dentfy of the basc varables. When c T 0 (unlkely and b 0, the ntal soluton s easy to fnd. = 0, u = 0, y = c T, v = b Otherwse, ntroduce artfcal varable nto each of the equatons where c > 0 or b < 0, n order to use these artfcal varables as ntal basc varables Ths choce of ntal basc varables wll set = 0 and u = 0 automatcally, whch satsfy the complementary constrant. Then, use phase of the two-phase method to fnd a BF soluton for the real problem. That s, apply the smple to (z s the artfcal varables Mn Z = z Subect to the lnear programmng constrants obtaned from the KKT condtons, but wth these artfcal varables ncluded. Stll need to modfy the smple method to satsfy the complementary constrant. Restrcted-Entry Rule: Eclude from consderaton any nonbasc varable to be the enterng varable whose complementary varable already s a basc varable. Choce the other nonbasc varables accordng to the usual crteron. Ths rule keeps the complementary constrant satsfed all the tme. When an optmal soluton *, u *, y *, v *, z = 0,, z n = 0 s obtaned for the phase problem, * s the desred optmal soluton for the orgnal quadratc programmng problem. Chap-6
College of Management, NCTU Operaton Research II Sprng, 009 A Quadratc Programmng Eample Ma 5 + 30 + 4 4 S.T. + 30, 0 Chap-7
College of Management, NCTU Operaton Research II Sprng, 009 Constraned Optmzaton wth Equalty Constrants Consder the problem of fndng the mnmum or mamum of the functon f(, subect to the restrcton that must satsfy all the equatons g ( = b g m ( = b m Eample: Ma f(, = + S.T. g(, = + = A classcal method s the method of Lagrange multplers. The Lagrangan functon h(, λ = f ( λ [ g ( b ], where λ, λ,..., λ are called Lagrange multplers. For the feasble values of, g ( b = 0 for all, so h(, λ = f(. m = ( m The method reduces to analyzng h(, λ by the procedure for unconstraned optmzaton. Set all partal dervatve to zero h f = m = g λ = 0, for =,,, n h = g ( + b = 0, for =,,, m λ Notce that the last m equatons are equvalent to the constrants n the orgnal problem, so only feasble solutons are consdered. Back to our eample h(, = + λ ( +. h = h = h = λ Chap-8
College of Management, NCTU Operaton Research II Sprng, 009 Other types of Nonlnear Programmng Problems Separable Programmng It s a specal case of conve programmng wth one addtonal assumpton: f( and g( functons are separable functons. A separable functon s a functon where each term nvolves ust a sngle varable. Eample: f(, = 6 9 + 8 3 = f ( + f ( f ( = f ( = Such problem can be closely appromated by a lnear programmng problem. Please refer to secton.8 for detals. Geometrc Programmng The obectve and the constrant functons take the form N a a a 3 g( = c P (, where P ( =... 3 for =,,, N = When all the c are strctly postve and the obectve functon s to be mnmzed, ths geometrc programmng can be converted to a conve y programmng problem by settng = e. Fractonal Programmng Suppose that the obectve functon s n the form of a (lnear fracton. Mamze f( = f ( / f ( = (c + c 0 / (d + d 0. Also assume that the constrants g ( are lnear. A b, 0. We can transform t to an equvalent problem of a standard type for whch effectve soluton procedures are avalable. We can transform the problem to an equvalent lnear programmng problem by lettng y = / (d + d 0 and t = / (d + d 0, so that = y/t. The orgnal formulaton s transformed to a lnear programmng problem. Ma Z = cy + c 0 t S.T. Ay bt 0 dy + d 0 t = y, t 0 Chap-9