Size: px
Start display at page:

Download ""

Transcription

1 A Globally Convergent Primal-Dual Interior Point Method for Constrained Optimization Hiroshi Yamashita 3 Abstract This paper proposes a primal-dual interior point method for solving general nonlinearly constrained optimization problems. The method is based on solving the barrier Karush-Kuhn-Tucker conditions for optimality by the Newton method. To globalize the iteration we introduce the barrier-penalty fucntion and the optimality condition for minimizing this function. Our basic iteration is the Newton iteration for solving the optimality conditions with respect to the barrier-penalty function which coincides with the Newton iteration for the barrier Karush-Kuhn- Tucker conditions if the penalty parameter is suciently large. It is proved that the method is globally convergent from an arbitrary initial point that strictly satises the bounds on the variables. Implementations of the given algorithm are done for small dense nonlinear programs. The method solves all the problems in Hock and Schittkowski's textbook eciently. Thus it is shown that the method given in this paper possesses a good theoretical convergence property and is ecient in practice. Key words: interior point method, primal-dual method, constrained optimization, nonlinear programming 3 Mathematical Systems, Inc., 2-4-3, Shinjuku, Shinjuku-ku, Tokyo, Japan 1

2 1 Introduction In this paper we propose a primal-dual interior point method that solves general nonlinearly constrained optimization problems. To obtain a fast algorithm for nonlinear optimization problems it is fairly clear from various experiences (for example the well known success of the SQP method) that we should eventually solve the Karush-Kuhn-Tucker conditions for optimality by a Newton-like method. However, solving the optimality conditions simply as a system of equations does not give an algorithm for solving optimization problems in general except for convex problems. Therefore it is not appropriate to treat the primal and dual variables equally to obtain globally convergent methods for general nonlinear optimization problems. Observing that there is a nice relation between the parameterized optimality conditions (barrier KKT condition) and the classical logarithmic barrier function of the primal variables, we can develop a globally convergent method for general nonlinear optimization problems. To prevent the possible divergence of the dual variables, we introduce the barrier penalty function and solve the optimality conditions by a Newton-like method. For nonlinear programs it has been believed long time that interior point methods are not practical because of inevitable numerical diculties which occur at the nal stage of iterations. See Fiacco and McCormick [4], Fletcher [5] and other standard textbooks on nonlinear optimization. Thus the state of the art method for nonlinear programs today is the SQP method (see Powell [7], Fletcher [5]) which can also be interpreted as a Newton-like method for Karush-Kuhn-Tucker conditions near a solution. It is known that the SQP method is ecient and stable for wide range of problems. The SQP method requires the quadratic programming subproblems which handle the combinatorial aspect of the problem caused by inequality constraints. A solution of the quadratic programming problem itself requires rather expensive cost especially for large scale problems. Therefore it is desired to have an ecient and stable interior point method for nonlinear problems in the light of success of that in the eld of large scale linear programs. This paper shows that it is in fact possible to construct such a method that is globally convergent theoretically, and ecient and stable in practice. The method is tested on 115 test problems in Hock and Schittkowski's collection [6]. For these problems our method solves all the problems with 2 to 3 function evaluations per problem and about 2 iterations per problem. Details are described in Section 5. A preliminary report of this paper has appeared in [11]. In this paper, problems to be solved are restricted to small to medium ones because we do not exploit the sparsity of the matrices here. However, recent report by Yamashita, Yabe and Tanabe [13] studies a trust region type method to utilize the sparsity of the Hessian of the Lagrangian. They report eciency of the method that uses the barrier penalty function as in this paper. See also [1] for a trust region type interior point method. It is to be noted that the recent studies by Yamashita and Yabe [12] and Yabe and Yamashita [9] show the superlinear and/or quadratic convergence of a class of primaldual interior point methods that use the Newton or quasi-newton iteration for solving the barrier KKT conditions. Other reports on the local behavior of primal-dual interior point methods include [2] and [3]. In Section 2, we describe basic concepts in the primal-dual interior point method. The 2

3 barrier penalty function which plays a key role in the method of this paper is introduced in Section 3, and analyzed there. A line search method that minimizes the barrier penalty function is described in Section 4, and is proved to be globally convergent. Section 5 reports the results of numerical experiment. Notation. The subscript k denotes an iteration count. Subscripts i and j denote components of vectors and matrices. The superscript t denotes transposes of vectors and matrices. The vector e denotes the vector of all ones and the matrix I the identity matrix. For simplicity of description, we assume k1k denotes the l 2 norm for vectors. The symbol R n denotes the n dimensional real vector space. The set R n + is dened by R n = fx + 2 jx> g. Rn 2 Primal-dual interior point method In this paper, we consider the following constrained optimization problem: (1) minimize f(x); x 2 R n ; subject to g(x) = ; x ; where we assume that the functions f : R n! R 1 and g : R n! R m are twice continuously dierentiable. Let the Lagrangian function of the above problem be dened by (2) L(w) =f(x) y t g(x) z t x; where w =(x; y; z) t, and y 2 R m and z 2 R n are the Lagrange multiplier vectors which correspond to the equality and inequality constraints respectively. Then Karush-Kuhn- Tucker (KKT) conditions for optimality of the above problem are given by 1 1 r (w) r xl(w) g(x) CA = (3) CA XZe and (4) where x ; z ; r x L(w) = rf(x) A(x) t y z; A(x) = rg 1(x) t. CA ; rg m (x) t 1 X = diag (x 1 ; 111;x n ) ; Z = diag (z 1 ; 111;z n ) : Now we approximate problem (1) by introducing the barrier function F B (; ) :R n +! R 1, (5) minimize subject to g(x) = ; F B (x; ) =f(x) np 3 log(x i );x 2 R n +

4 where the barrier parameter > is a given constant. It is well known that, if is suciently small, problem (5) is a good approximation to original problem (1) (see Fiacco and McCormick [4]). The optimality conditions for (5) are given by (6) rf (x) A(x) t y X 1 e = ; g(x) = and x>; where y 2 R m is the Lagrange multiplier for the equality constraints. If we introduce an auxiliary variable z 2 R n which is to be equal to X 1 e, then the above conditions become the conditions (7) and (8) r(w; ) r xl(w) g(x) XZe e x>; 1 CA = z>: The introduction of the variable z is essential to the numerical success of the barrier based algorithm in this paper. In this paper we call conditions (7) the barrier KKT conditions, and a point w() = (x();y();z()) that satises these conditions is called the barrier KKT point. We will use an interior point method for searching a point that approximately satises the above conditions, and nally obtain a point that satises the Karush-Kuhn-Tucker conditions by letting #. This means that we force x and z be strictly positive during iterations. Therefore we delete inequality conditions (8) hereafter, and always assume that x and z are strictly positive in what follows. Here we note that (9) where r(w; ) =r (w) ^e; ^e = e An algorithm of this paper approximately solves the sequence of conditions (7) with a decreasing sequence of the barrier parameter that tends to, and thus obtains an approximate solution to KKT conditions. For deniteness, we describe a prototype of such algorithm as follows. Algorithm IP Step. (Initialize) Set ">, M c > and k =. Let a positive sequence f k g ; k # be given. 1 CA : Step 1. (Termination) If kr (w)k ", then stop. 1 CA ; 4

5 Step 2. (Approximate barrier KKT point) Find a point w k+1 that satises (1) kr(w k+1 ; k )km c k : Step 3. (Update) Set k := k + 1 and go to Step 1. 2 The following theorem shows the global convergence property of Algorithm IP. Theorem 1 Let fw k g be an innite sequence generated by Algorithm IP. Suppose that the sequences fx k g and fy k g are bounded. Then fz k g is bounded, and any accumulation point of fw k g satises KKT conditions (3) and (4). Proof. Assume that there exists an i such that (z k ) i!1. Equation (1) yields (rf(x k) A(x k ) t y k ) i (z k ) i 1 M c k1 (z k ) i ; which isacontradiction because of the boundedness of fx k g and fy k g.thus the sequence fz k g is bounded. Let ^w be any accumulation point offw k g. Since the sequences fw k g and f k g satisfy (1) for each k and k approaches zero, r (^w) = follows from the denition of r(w; ). Therefore the proof is complete. 2 We note that the barrier parameter sequence f k g in Algorithm IP need not be determined beforehand. The value of each k may be set adaptively as the iteration proceeds. An example of updating method of k is described in Section 5. We call condition (1) the approximate barrier KKT condition, and call a point that satises this condition the approximate barrier KKT point. To nd an approximate barrier KKT point for a given >, we use the Newton-like method in this paper. Let 1w =(1x; 1y; 1z) t be dened by a solution of (11) where (12) J(w) = J(w)1w = r(w; ); G A(x)t I A(x) Z X Then the basic iteration of the Newton-like method may be described as 1 CA : (13) w k+1 = w k +3 k 1w k ; where 3 k = diag( xk I n ; yk I m ; zk I n ) is composed of step sizes in x, y and z variables. If G = r 2 L(w), then 1w becomes Newton's direction for solving (7). To solve (11), we x split the equations into two groups. Thus we solve! G + X 1 Z A(x) t 1x!! = r xl(w)+x 1 e z (14) ; A(x) 1y g(x) 5

6 for (1x; 1y) t, then we obtain 1z by the third equation in (11). If G + X 1 Z is positive denite and A(x) is of full rank, then the coecient matrix in (14) is nonsingular. It will be useful to note that (14) can be written as (15) where (16) G + X 1 Z A(x) t A(x)! 1x ey ~y = y +1y:! = rf(x)+x1 e g(x) In this paper, we will deal with the case in which the matrix G can be assumed nonnegative denite. Therefore, to solve the general nonlinear problems, we will use a positive denite quasi-newton approximation to the Hessian matrix of the Lagrangian function to obtain the desired property of the matrix G.! ; 3 Barrier penalty function To globalize the convergence property ofaninterior point algorithm based on the above iteration, we introduce two auxiliary problems. Firstly mp we dene the following problem: minimize F P (x; ) =f(x)+ jg i (x)j; x 2 R n ; (17) subject to x ; where the penalty parameter is a given positive constant. The necessary conditions for optimality of this problem are (see 14.2 of Fletcher [5]) r x L(w) = ; ( mx ) (18) y jg i (x)j ; XZe = ; x ;z ; where the means the subdierential of the function in the braces with respect to g. In our case the second condition in (18) is equivalent to (19) y i ; g i (x) =; y i = ; g i (x) > ; y i =; g i (x) < ; for each i =1; 111;m. This condition can be expressed as jg i (x)j = y i g i (x); y i ; i =1; 111;m; or jg i (x)j + y ig i (x) =; y i ; i =1; 111;m: 6

7 Therefore conditions (18) can be written as r (w) = r xl(w) (2) r E (w) XZe and (21) where 1 CA = x ; z ; y i ; i =1; 111;m; 1 CA r E (w) i = jg i (x)j + y ig i (x) ;; 111;m: Note that we are using the same symbol r (w) to denote the residual vector of the optimality conditions as in Section 2 for simplicity. Ifkyk 1 <, conditions (18) are equivalent to conditions (3) and (4). In this sense, problem (17) is equivalent to problem (1). Next we introduce the barrier penalty function F (; ; ) :R n +! R1 by (22) F (x; ; ) =f(x) nx log x i + mx jg i (x)j ; where and are given positive constants. This function plays an essential role in the method given in this paper. Let us approximate problem (17) by the following problem (23) minimize F (x; ; ); x 2 R n + : The necessary conditions for optimality of a solution to the above problem are r x L(w) = ; ( mx ) (24) y jg i (x)j XZe = e; x > ; z>; where we introduce an auxiliary variable z 2 R n as in (7). As above, conditions (24) can be written as r(w; ) xl(w) (25) r E (w) CA = CA XZe e and (26) 1 x>; z>; y i ; i =1; 111;m; where we use the same symbol r(w; ) as in Section 2 for simplicity. If>kyk 1 then conditions (25) and (26) coincide with (7) and (8). We call a point w() =(x();y();z()) 2 R n 2 R m 2 R n that satises (24) for a given > the barrier KKT point for this as before. We can use Algorithm IP to solve (17). The following theorem shows the global convergence property of Algorithm IP for solving (17). 1 7

8 Theorem 2 Let fw k g be an innite sequence generated byalgorithm IP for solving (17). Suppose that the sequences fx k g is bounded. Then fz k g is bounded, and any accumulation point of fw k g satises the optimality conditions (2) and (21) for problem (17). 2 Now we formulate a Newton-like iteration for solving the above conditions (25). Thus we calculate the rst order change of (24) with respect to a change in w. This gives (G + X 1 Z)1x A(x) t 1y = r x L(w)+X 1 e z; ( mx ~y y +1y g i (x)+rg i (x) 1x ) (27) t ; 1z = X 1 Z1x + X 1 e z: Following lemma gives a basic property of the iteration vector 1w =(1x; 1y; 1z) t. Lemma 1 Suppose that 1w satises (27) at an interior point w. (i) If >keyk 1, then 1w is identical to the one given by (11). (ii) If 1w =, then the point w isabarrier KKT point that satises (24). (iii) If 1x =, then the point (x; y +1y; z +1z) is a barrier KKT point that satises (24). 2 If we consider the subproblem: (28) minimize 1 2 1xt (G+X 1 Z)1x+(rf(x)X 1 e) t 1x+ mx g i (x)+rg i (x) t 1x ; 1x 2 R n ; the solution vector 1x and the corresponding multiplier vector ~y that satisfy the necessary conditions for optimality also satisfy conditions (27). If G + X 1 Z is positive denite we can solve the problem (28) by a straightforward active set method which starts with an active set that contains all the constraints g i (x)+rg i (x) t 1x =;; 111m. An example of the procedure is described in Fletcher [5]. It is important to note that, if the vectors 1x and ~y obtained by solving (15) satisfy >k~yk 1, then the desired iteration vector that satises (27) are also obtained. The procedure described above is devised instead of the simple Newton iteration given in Section 2 in order to show away of preventing possible divergence of the dual variable y. However for the practical purpose it seems sucient to solve only equation (11) once per iteration as shown in sections below in which practical experiences obtained by the author on general nonlinear programming problems are described. Therefore this procedure is of only theoretical importance at present. As shown above, optimality conditions (2) and (21) are identical to optimality conditions (3) and (4) if we set = 1. Also, Newton-like iteration (27) coincides with (11) when = 1. Therefore we will regard that the method explained below includes a method for solving (1) when = 1 for simplicity of exposition. We note that the penalty parameter which will appear below should be nite even if = 1. Another interesting point to be noted in the above iteration is that it can give a solution to an infeasible problem. Even if the constraints in problem (1) are incompatible, the 8

9 method described above will give a solution to problem (17) as shown below. A solution to problem (17) with infeasible constraints may give useful information about problem (1). Now we proceed to an analysis of the properties of the barrier penalty function. The directional derivative F (x; ; ; s) of the function F (x; ; ) along an arbitrary given direction s is dened by F F (x + s; ; ) F (x; ; ) (x; ; ; s) = lim # = rf(x) X X t s e t X 1 s + rg i (x) t s + + rg i (x) s X t rg i (x) t s; where the summations in the above equation are to be understood as X X a i = a i X X ; a i = a i X X ; a i = a i : + g i > We introduce a rst order approximation F l of F (x + s; ; ) by F l (x; ; ; s) f(x)+rf (x) t s log(x i )+ s i (29) + mx g i = nx g i (x)+rg i (x) t s; g i < x i and an estimate of the rst order change 1F l of F by (3) 1F l (x; ; ; s) F l (x; ; ; s) F (x; ; ); = rf(x) t s e t X 1 s + mx jg i (x)+rg i (x)sj mx jg i (x)j for an arbitrary given direction s. We have following properties for the quantities dened above which give an extension to the similar properties in the case of dierentiable functions (see for example Yamashita [1]). Lemma 2 Let >, > and s 2 R n be given. Then the following assertions hold. (i) The function F l (x; ; ; s) is convex with respect to the variable. (ii) There holds the relation (31) F (x; ; )+F (x; ; ; s) F l (x; ; ; s): (iii) Further, there exists a 2 (; 1) such that (32) F (x + s; ; ) F (x; ; )+F (x + s; ; ; s); whenever x + s>. 9

10 Proof. The rst statement of the lemma is obvious. If > is suciently small, we have F l (x; ; ; s) = F (x; ; )+F (x; ; ; s) = F (x; ; )+F (x; ; ; s): Since F l (x; ; ; s) isconvex with respect to and coincides with a linear function of > when is suciently small, we obtain (31). Now we show (32). If we consider F (x + s; ; ; s) as a function of the variable 2 [; 1], the number of discontinuous points of F is nite. Therefore there exists a 2 (; 1) such that F (x + s; ; ; s) Z1 F (x + s; ; ; s)d = F (x + s; ; ) F (x; ; ): This completes the proof. 2 Lemma 3 Let " 2 (; 1) be a given constant and s 2 R n be given. If 1F l (x; ; ; s) <, then (33) F (x + s; ; ) F (x; ; ) " 1F l (x; ; ; s); for suciently small >. Proof. From (32), there exists a 2 (; 1) such that (34) F (x + s; ; ) F (x; ; ) F (x + s; ; ; s) = F (x + s; ; ; s): From (31) we have (35) F (x + s; ; ; s) 1F l (x + s; ; ; s): Because 1F l (; ; ; s) is continuous, and 1F l (x; ; ; s) < by the assumption, we obtain (36) 1F l (x + s; ; ; s) " 1F l (x; ; ; s); for suciently small ksk. From (34) - (36) we obtain (33) for suciently small >. 2 Lemma 4 Suppose that 1w satises (27). If <, then (37) 1F l (x; ; ;1x) 1x t (G + X 1 Z)1x mx ( j~y i j)jg i (x)j: Further if, G is positive semidenite and keyk 1 1F l (x; ; ;1x) =yields 1x =., then 1F l (x; ; ;1x), and 1

11 Proof. From (27) and (3) we have 1F l (x; ; ;1x) =1x t (G+X 1 Z)1x+1x t A(x) t ~y+ The i-th components in the summations in the last three terms give mx g i (x)+rg i (x) t 1x mx jg i (x)j: (38) ~y i rg i (x) t 1x + g i (x)+rg i (x) 1x t jg i (x)j ~y i rg i (x) t 1x + g i (x)+rg i (x) 1x t jg i (x)j = ~y i g i (x) jg i (x)j ; where the inequality in the second line follows from, and the equality in the third line follows from the property ~y i ; g i (x)+rg i (x) t 1x =; ~y i = ; g i (x)+rg i (x) t 1x>; ~y i =; g i (x)+rg i (x) t 1x<: The relation (38) gives (37). 2 4 Line search algorithm To obtain a globally convergent algorithm to a barrier KKT point for a xed >, it is necessary to modify the basic Newton iteration with the unit step length somehow. Our iterations consist of (39) x k+1 = x k + xk 1x k ; y k+1 = y k + yk 1y k ; z k+1 = z k + zk 1z k ; where xk, yk and zk are step sizes determined by the line search procedures described below. The main iteration is to decrease the value of the barrier penalty function for xed. Thus the step size of the primal variable x is determined by the sucient decrease rule of the merit function. The step size of the dual variable z is determined so as to stabilize the iteration. The explicit rules follow in order. We adopt Armijo's rule as the line search rule for the variable x. At the point x k,we calculate the maximum allowed step to the boundary of the feasible region by (4) kmax = min i ( (x k) i (1x k ) i (1x k) i < i.e., the step size kmax gives an innitely large value of the barrier penalty function F if it exists, because of the barrier terms, and a step size 2 [; kmax ) gives a strictly feasible primal variable. A step to the next iterate is given by ) ; (41) xk = k l k ; k = min f kmax ; 1g ; 11

12 where 2 (; 1) and 2 (; 1) are xed constants and l k is the smallest nonnegative integer such that (42) F (x k + k l k 1x k; ; ) F (x k ; ; ) " k l k 1F l(x k ; ; ;1x k ); where " 2 (; 1). Typical values of these parameters are = :5, = :9995 and " =1 6. Therefore we will try the sequence x k +:9995 kmax 1x k ; x k +:5 2 :9995 kmax 1x k ; x k +:25 2 :9995 kmax 1x k ; 111 for example, and will nd a step size that satises (42). If G is positive semidenite, then 1F l (x k ; ; ;1x k ) by Lemma 4, and therefore the existence of such steps is assured by Lemma 3. For the variable z, we adopt the box constraints rule, i.e., we force x and z to satisfy the condition (43) c Lki ((x k ) i + xk (1x k ) i )((z k ) i + zk (1z k ) i ) c Uki ; i =1; 111;n at the end of each iteration, where the bounds c Lk and c Uk satisfy (44) <c Lki <<c Uki ; i =1; 111;n: To this end, we let (45) c Lki = minn M L ; ((x k ) i + xk (1x k ) i )(z k ) io ; c Uki = max fm U ; ((x k ) i + xk (1x k ) i )(z k ) i g ; where M L > 1 and M U > 1 are given constants. The construction of the above bounds shows that current z satises (46) (47) c Lki ((x k ) i + xk (1x k ) i ) (z k) i The step size z is determined by zk = min ( min i ( max i ( i c Uki ((x k ) i + xk (1x k ) i ) ; c Lki ((x k ) i + xk (1x k ) i ) c Uki ((z k ) i + i (1z k ) i ) ((x k ) i + xk (1x k ) i ) i =1; 111;n: )) ) ; 1 : The rule (47) means that the step size z is the maximal allowed step that satises the box constraints with the restriction of being not greater than the unit step length. Lemma 5 Suppose that an innite sequence fw k g is generated for xed >. Then if lim inf k!1(x k ) i > and lim sup k!1(x k ) i < 1, then lim inf k!1(c Lk ) i > and lim sup k!1(c Uk ) i < 1 for i =1; 111;n. 12

13 Proof. Suppose that (c Lk ) i! for an i and some subsequence K f; 1; 2; 111g. Then by the denition of (c Lk ) i in (45), (z k ) i! ;k 2 K. However, in order for a subsequence of f(z k ) i g to tend to, there must be an iteration k at which the lower bound (c Lk ) i =(x k+1 ) i of (z k ) i is arbitrary small and the value of (z k ) i at the iteration is strictly larger than that bound, i.e. at the iteration the value of (z k ) i decreases to a strictly smaller value. This means that at iteration k, (c Lk ) i = =M L from the denition (45), and therefore the value of (x k+1 ) i must be arbitrary large because =M L < (x k+1 ) i (z k ) i and (z k ) i! ;k 2 K. This is impossible because of the assumption of the lemma. The proof of the boundedness of (c Uk ) i is similar. 2 (48) In actual calculation we modify the direction 1z k (1z k ) i = 8>< >: ; if (z k) i = c Lki=(x k+1) i and (1z k ) i < ; ; if (z k ) i = c Uki =(x k+1 ) i and (1z k ) i > ; (1z k ) i ; otherwise: This modication means that we project the direction along the boundary of the box constraints if the point z k is on that boundary and the direction 1z k points outward of the box. This procedure is adopted because it gives better numerical results. The global convergence results shown in the following are equally valid for both unmodied and modied directions. For the variable y, there exist three obvious choices for the step length: by (49) yk =1 or xk or zk : The global convergence property given below holds for these choices. We choose yk = zk from numerical experiments. The following algorithm describes the iteration for xed > and >. We note that this algorithm corresponds to Step 2 of Algorithm IP in Section 2. Algorithm LS Step. (Initialize) Let w 2 R n + 2 Rm 2 R n +, and >, >. Set " >, 2 (; 1), 2 (; 1), " 2 (; 1), M L > 1 and M U > 1. Let k =. Step 1. (Termination) If kr(w k ;)k " ; then stop. Step 2. (Compute direction) Calculate the direction 1w k by (27). Step 3. (Stepsize) Set kmax = min i ( (x k) i ) (1x k ) i (1x k) i < ; k = min f kmax ; 1g : Find the smallest nonnegative integer l k that satises F (x k + k l k 1x k ;;) F (x k ;;) " k l k 1F l (x k ; ; ;1x k ): Calculate xk = k l k ; 13

14 c Lki = min ; (x k + xk 1x k ) i (z k ) i ; M L c Uki = max fm U ; (x k + xk 1x k ) i (z k ) i g ; ( ( ( i c Lki zk = min min max i i ((x k ) i + xk (1x k ) i c Uki )) ) ((z k ) i + i (1z k ) i ) ; 1 ; ((x k ) i + xk (1x k ) i ) yk = zk ; 3 k = diagf xk I n ; yk I m ; zk I n g: Step 4. (Update variables) Set w k+1 = w k +3 k 1w k : Step 5. Set k := k + 1 and go to Step 1. 2 To prove global convergence of Algorithm LS, we need the following assumptions. Assumption G (1) The functions f and g i ;i =1; :::; m, are twice continuously dierentiable. (2) The level set of the barrier penalty function at an initial point x 2 R n +, which is dened by nx 2 R n jf (x; ; ) + F (x ; ; )o, is compact for given >. (3) The matrix A(x) is of full rank on the level set dened in (2). (4) The matrix G k is positive semidenite and uniformly bounded. (5) The penalty parameter satises ky k +1y k k 1 for each k =; 1; :::. 2 We note that if a quasi-newton approximation is used for computing the matrix G k, then we need the continuity of only the rst order derivatives of functions in Assumption G-(1). We also note that if 1F l (x k ; ; ;1x k ) =, at an iteration k, then the step sizes xk = yk = zk = 1 are adopted and (x k+1 ;y k+1 ;z k+1 ) gives a barrier KKT point from Lemma 1 and Lemma 4. The following theorem gives a convergence of an innite sequence generated by Algorithm LS. Theorem 3 Let an innite sequence fw k g be generated byalgorithm LS. Then there exists at least one accumulation point of fw k g, and any accumulation point of the sequence fw k g is a barrier KKT point. 14

15 Proof. First we note that each component of the sequence fx k g is bounded away from zero and bounded above by the assumption and the existence of the log barrier term. Therefore the sequence fx k g has at least one accumulation point. The sequence fz k g also has these properties by Lemma 5. Thus there exists a positive number M such that (5) kpk 2 M pt (G k + X 1 k Z k)p M kpk 2 ; 8p 2 R n ; by the assumption. From (37) and (5), we have (51) and from (42), 1F l (x k ; ; ;1x k ) k1x kk 2 M < ; (52) F (x k+1 ; ; ) F (x k ; ; ) " k l k 1F l(x k ; ; ;1x k ) " k l k k1x kk 2 M < : Because the sequence ff (x k ; ; )g is decreasing and bounded below, the left hand side of (52) converges to. Since lim inf k!1(x k ) i > ;i =1; 111;n,wehave lim inf k!1 k >. Suppose that there exists a subsequence K f; 1; 111g and a such that (53) lim inf k!1 k1x kk>; k 2 K: Then we have l k!1;k 2 K from (52) because the left most expression tends to zero, and therefore we can assume l k > for suciently large k 2 K without loss of generality. If l k > then the point x k + xk 1x k = does not satisfy the condition (42). Thus, we have (54) F (x k + xk 1x k =; ; ) F (x k ; ; ) >" xk 1F l (x k ; ; ;1x k )=: By (32) and (31), there exists a k 2 (; 1) such that (55) F (x k + xk 1x k =; ; ) F (x k ; ; ) xk F (x k + k xk 1x k =; ; ;1x k )= xk 1F l (x k + k xk 1x k =; ; ;1x k )=; k 2 K: Then, from (54) and (55), we have " 1F l (x k ; ; ;1x k ) < 1F l (x k + k xk 1x k =; ; ;1x k ): This inequality yields (56) 1F l (x k + k xk 1x k =; ; ;1x k ) 1F l (x k ; ; ;1x k ) > (" 1)1F l (x k ; ; ;1x k ) > : Because 1x k is a solution of problem (28) and there holds (5), k1x k k is uniformly bounded above. Then by the property l k! 1,we have k k xk 1x k =k!;k 2 K. 15

16 Thus the left hand side of (56) and therefore 1F l (x k ; ; ;1x k ) converges to zero when k!1;k 2 K. This contradicts the assumption (53) because we have 1x k! ;k 2 K from (51). Therefore we proved (57) lim k1x k k =: k!1;k2k Let an arbitrary accumulation point of the sequence fx k g be ^x 2 R n + and let x k! ^x; k 2 K for K f; 1; 111g.Thus (58) x k! ^x; 1x k! ; x k+1! ^x; k 2 K: Because nx 1 Z k ko is bounded, we have lim k!1;k2kz k +1z k X e 1 = k from (27). If we dene ^z = ^X 1 e where ^X = diag(^x 1 ; 111; ^x n ), then we have Hence from (45) we have z k +1z k! ^z; k 2 K: (c Lk ) i M L (x k+1 ) i (z k +1z k ) i M U (C Uk ) i ; i =1; 111;n for k 2 K suciently large, which shows that the point z k +1z k is always accepted as z k+1 for suciently large k 2 K. Since zk = 1 is accepted for k 2 K suciently large, so is yk =1. Therefore we obtain lim r x L(^x; y k +1y k ; ^z) = ; k!1;k2k lim y k +1y k k!1;k2k ( nx ) jg i (^x)j : Because the matrix A(^x) is of full rank, the sequence fy k +1y k g ;k 2 K converges to a point ^y 2 R m which satises r x L(^x; ^y; ^z) = ; mx! ^y jg i (^x)j ; ^X ^z = e; ^x >; ^z >: This completes the proof because we proved that there exists at least one accumulation point offx k g, and for an arbitrary accumulation point ^x of fx k g, there exist unique ^y and ^z that satisfy the above. 2 16

17 5 Numerical Result In this section, we report numerical results of an implementation of the algorithm given in this paper for nonlinear programming problems. We set = 1 in this experiment. The software is called NUOPT and the code is written bytakahito Tanabe. In order to havean appropriate positive semidenite matrix G by a reasonable cost for nonlinear problems, we resort to a quasi-newton approximation to the Hessian matrix of the Lagrangian function. We use updating formula suggested by Powell[7] for the SQP method: G k+1 = G k G ks k s t k G k s t k G ks k + u ku t k s t u ; k k where u k is calculated by s k = x k+1 x k ; v k = r x L(x k+1 ;y k+1 ;z k+1 ) r x L(x k ;y k+1 ;z k+1 ); u k = k v k +(1 k )G k s k ; k = 8< : :8s t k G k s k s t k G k s ks t k v k 1; s t k v k :2s t k G ks k ; ; s t k v k :2s t k G ks k ; to satisfy s t u k k > for the hereditary positive deniteness of the update. Method for updating the barrier parameter k is as follows. Suppose we have an approximate barrier KKT point w k+1 that satises kr(w k+1 ; k )km c k ; in Step 2 of Algorithm IP. Then k+1 is dened by k+1 = max( kr(wk+1 ; k ) )k ; k ; M M where < M c < M and M > 1 should be satised. In our experiment weset M c =3;M =4;M = 5. As in the SQP method, we expect fast local convergence of the method if is suciently small near a solution because it is based on the Newton iteration for the optimality conditions and a quasi-newton approximation to the second derivative matrix. As noted in the above, this expectation is proved by Yamashita and Yabe [12]. Linear equation (14 ) is solved by using the Bunch-Parlett factorization. The test problems for nonlinear problems are adopted from the book by Hock and Schittkowski [6]. The results are summarized in Table 1 at the end of this paper. Following list explains the notations used in Table 1: n= number of variables. m=number of constraints. obj=nal objective function value. res=norm of nal KKT condition residual. itr=iteration count. 17

constraint. Let us penalize ourselves for making the constraint too big. We end up with a

constraint. Let us penalize ourselves for making the constraint too big. We end up with a Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the

More information

Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

More information

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing

More information

Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen (für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

More information

Date: April 12, 2001. Contents

Date: April 12, 2001. Contents 2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

More information

2.3 Convex Constrained Optimization Problems

2.3 Convex Constrained Optimization Problems 42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

More information

(Quasi-)Newton methods

(Quasi-)Newton methods (Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting

More information

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725 Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

More information

Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

More information

Interior Point Methods and Linear Programming

Interior Point Methods and Linear Programming Interior Point Methods and Linear Programming Robert Robere University of Toronto December 13, 2012 Abstract The linear programming problem is usually solved through the use of one of two algorithms: either

More information

Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

More information

10. Proximal point method

10. Proximal point method L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

More information

Adaptive Online Gradient Descent

Adaptive Online Gradient Descent Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

More information

Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.

Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U. Summer course on Convex Optimization Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota Interior-Point Methods: the rebirth of an old idea Suppose that f is

More information

Big Data - Lecture 1 Optimization reminders

Big Data - Lecture 1 Optimization reminders Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics

More information

Data analysis in supersaturated designs

Data analysis in supersaturated designs Statistics & Probability Letters 59 (2002) 35 44 Data analysis in supersaturated designs Runze Li a;b;, Dennis K.J. Lin a;b a Department of Statistics, The Pennsylvania State University, University Park,

More information

t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).

t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d). 1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction

More information

INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS

INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS HANDE Y. BENSON, DAVID F. SHANNO, AND ROBERT J. VANDERBEI Operations Research and Financial Engineering Princeton

More information

24. The Branch and Bound Method

24. The Branch and Bound Method 24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

More information

No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics

No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results

More information

Convex Programming Tools for Disjunctive Programs

Convex Programming Tools for Disjunctive Programs Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible

More information

Solving linear equations on parallel distributed memory architectures by extrapolation Christer Andersson Abstract Extrapolation methods can be used to accelerate the convergence of vector sequences. It

More information

Duality of linear conic problems

Duality of linear conic problems Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

More information

BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBERT SPACE REVIEW BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

More information

1 if 1 x 0 1 if 0 x 1

1 if 1 x 0 1 if 0 x 1 Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or

More information

Interior-Point Algorithms for Quadratic Programming

Interior-Point Algorithms for Quadratic Programming Interior-Point Algorithms for Quadratic Programming Thomas Reslow Krüth Kongens Lyngby 2008 IMM-M.Sc-2008-19 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK-2800

More information

Support Vector Machines

Support Vector Machines Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric

More information

NORTHWESTERN UNIVERSITY Department of Electrical Engineering and Computer Science LARGE SCALE UNCONSTRAINED OPTIMIZATION by Jorge Nocedal 1 June 23, 1996 ABSTRACT This paper reviews advances in Newton,

More information

Stochastic Inventory Control

Stochastic Inventory Control Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the

More information

Solving polynomial least squares problems via semidefinite programming relaxations

Solving polynomial least squares problems via semidefinite programming relaxations Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose

More information

Factorization Theorems

Factorization Theorems Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

More information

A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems

A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems Sunyoung Kim, Masakazu Kojima and Kim-Chuan Toh October 2013 Abstract. We propose

More information

4.6 Linear Programming duality

4.6 Linear Programming duality 4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

More information

CONSTRAINED NONLINEAR PROGRAMMING

CONSTRAINED NONLINEAR PROGRAMMING 149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach

More information

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem

More information

1 Norms and Vector Spaces

1 Norms and Vector Spaces 008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)

More information

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

More information

Lecture 13 Linear quadratic Lyapunov theory

Lecture 13 Linear quadratic Lyapunov theory EE363 Winter 28-9 Lecture 13 Linear quadratic Lyapunov theory the Lyapunov equation Lyapunov stability conditions the Lyapunov operator and integral evaluating quadratic integrals analysis of ARE discrete-time

More information

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem

More information

Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

More information

APPLYING NEW OPTIMIZATION CONTROL. Stephen J. Wright. Argonne, IL 60439

APPLYING NEW OPTIMIZATION CONTROL. Stephen J. Wright. Argonne, IL 60439 APPLYING NEW OPTIMIZATION ALGORITHMS TO MODEL PREDICTIVE CONTROL Stephen J Wright Mathematics and Computer Science Division Argonne National Laboratory Argonne, IL 9 Abstract The connections between optimization

More information

Notes on Factoring. MA 206 Kurt Bryan

Notes on Factoring. MA 206 Kurt Bryan The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor

More information

Ri and. i=1. S i N. and. R R i

Ri and. i=1. S i N. and. R R i The subset R of R n is a closed rectangle if there are n non-empty closed intervals {[a 1, b 1 ], [a 2, b 2 ],..., [a n, b n ]} so that R = [a 1, b 1 ] [a 2, b 2 ] [a n, b n ]. The subset R of R n is an

More information

A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION 1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

More information

Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

More information

OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN. E.V. Grigorieva. E.N. Khailov

OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN. E.V. Grigorieva. E.N. Khailov DISCRETE AND CONTINUOUS Website: http://aimsciences.org DYNAMICAL SYSTEMS Supplement Volume 2005 pp. 345 354 OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN E.V. Grigorieva Department of Mathematics

More information

Chapter 7. Continuity

Chapter 7. Continuity Chapter 7 Continuity There are many processes and eects that depends on certain set of variables in such a way that a small change in these variables acts as small change in the process. Changes of this

More information

Random graphs with a given degree sequence

Random graphs with a given degree sequence Sourav Chatterjee (NYU) Persi Diaconis (Stanford) Allan Sly (Microsoft) Let G be an undirected simple graph on n vertices. Let d 1,..., d n be the degrees of the vertices of G arranged in descending order.

More information

Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

More information

The Heat Equation. Lectures INF2320 p. 1/88

The Heat Equation. Lectures INF2320 p. 1/88 The Heat Equation Lectures INF232 p. 1/88 Lectures INF232 p. 2/88 The Heat Equation We study the heat equation: u t = u xx for x (,1), t >, (1) u(,t) = u(1,t) = for t >, (2) u(x,) = f(x) for x (,1), (3)

More information

Optimal shift scheduling with a global service level constraint

Optimal shift scheduling with a global service level constraint Optimal shift scheduling with a global service level constraint Ger Koole & Erik van der Sluis Vrije Universiteit Division of Mathematics and Computer Science De Boelelaan 1081a, 1081 HV Amsterdam The

More information

Measuring Rationality with the Minimum Cost of Revealed Preference Violations. Mark Dean and Daniel Martin. Online Appendices - Not for Publication

Measuring Rationality with the Minimum Cost of Revealed Preference Violations. Mark Dean and Daniel Martin. Online Appendices - Not for Publication Measuring Rationality with the Minimum Cost of Revealed Preference Violations Mark Dean and Daniel Martin Online Appendices - Not for Publication 1 1 Algorithm for Solving the MASP In this online appendix

More information

Solutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR

Solutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR Solutions Of Some Non-Linear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision

More information

PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5.

PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5. PUTNAM TRAINING POLYNOMIALS (Last updated: November 17, 2015) Remark. This is a list of exercises on polynomials. Miguel A. Lerma Exercises 1. Find a polynomial with integral coefficients whose zeros include

More information

HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!

HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)! Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following

More information

1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where. Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

More information

The Goldberg Rao Algorithm for the Maximum Flow Problem

The Goldberg Rao Algorithm for the Maximum Flow Problem The Goldberg Rao Algorithm for the Maximum Flow Problem COS 528 class notes October 18, 2006 Scribe: Dávid Papp Main idea: use of the blocking flow paradigm to achieve essentially O(min{m 2/3, n 1/2 }

More information

Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

More information

arxiv:1112.0829v1 [math.pr] 5 Dec 2011

arxiv:1112.0829v1 [math.pr] 5 Dec 2011 How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

More information

Advanced Microeconomics

Advanced Microeconomics Advanced Microeconomics Ordinal preference theory Harald Wiese University of Leipzig Harald Wiese (University of Leipzig) Advanced Microeconomics 1 / 68 Part A. Basic decision and preference theory 1 Decisions

More information

GenOpt (R) Generic Optimization Program User Manual Version 3.0.0β1

GenOpt (R) Generic Optimization Program User Manual Version 3.0.0β1 (R) User Manual Environmental Energy Technologies Division Berkeley, CA 94720 http://simulationresearch.lbl.gov Michael Wetter MWetter@lbl.gov February 20, 2009 Notice: This work was supported by the U.S.

More information

Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms

Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms Alexander Mafusalov, Stan Uryasev RESEARCH REPORT 2014-1 Risk Management and Financial Engineering Lab Department of

More information

Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

More information

Computing a Nearest Correlation Matrix with Factor Structure

Computing a Nearest Correlation Matrix with Factor Structure Computing a Nearest Correlation Matrix with Factor Structure Nick Higham School of Mathematics The University of Manchester higham@ma.man.ac.uk http://www.ma.man.ac.uk/~higham/ Joint work with Rüdiger

More information

Mathematics Course 111: Algebra I Part IV: Vector Spaces

Mathematics Course 111: Algebra I Part IV: Vector Spaces Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 1996-7 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are

More information

CHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e.

CHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e. CHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e. This chapter contains the beginnings of the most important, and probably the most subtle, notion in mathematical analysis, i.e.,

More information

BIG DATA PROBLEMS AND LARGE-SCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION

BIG DATA PROBLEMS AND LARGE-SCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION BIG DATA PROBLEMS AND LARGE-SCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION Ş. İlker Birbil Sabancı University Ali Taylan Cemgil 1, Hazal Koptagel 1, Figen Öztoprak 2, Umut Şimşekli

More information

Recall that the gradient of a differentiable scalar field ϕ on an open set D in R n is given by the formula:

Recall that the gradient of a differentiable scalar field ϕ on an open set D in R n is given by the formula: Chapter 7 Div, grad, and curl 7.1 The operator and the gradient: Recall that the gradient of a differentiable scalar field ϕ on an open set D in R n is given by the formula: ( ϕ ϕ =, ϕ,..., ϕ. (7.1 x 1

More information

Discrete Optimization

Discrete Optimization Discrete Optimization [Chen, Batson, Dang: Applied integer Programming] Chapter 3 and 4.1-4.3 by Johan Högdahl and Victoria Svedberg Seminar 2, 2015-03-31 Todays presentation Chapter 3 Transforms using

More information

We shall turn our attention to solving linear systems of equations. Ax = b

We shall turn our attention to solving linear systems of equations. Ax = b 59 Linear Algebra We shall turn our attention to solving linear systems of equations Ax = b where A R m n, x R n, and b R m. We already saw examples of methods that required the solution of a linear system

More information

Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach

Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach MASTER S THESIS Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach PAULINE ALDENVIK MIRJAM SCHIERSCHER Department of Mathematical

More information

Numerical methods for American options

Numerical methods for American options Lecture 9 Numerical methods for American options Lecture Notes by Andrzej Palczewski Computational Finance p. 1 American options The holder of an American option has the right to exercise it at any moment

More information

CHAPTER 9. Integer Programming

CHAPTER 9. Integer Programming CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral

More information

3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field

3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field 77 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field Overview: The antiderivative in one variable calculus is an important

More information

G.A. Pavliotis. Department of Mathematics. Imperial College London

G.A. Pavliotis. Department of Mathematics. Imperial College London EE1 MATHEMATICS NUMERICAL METHODS G.A. Pavliotis Department of Mathematics Imperial College London 1. Numerical solution of nonlinear equations (iterative processes). 2. Numerical evaluation of integrals.

More information

A FIRST COURSE IN OPTIMIZATION THEORY

A FIRST COURSE IN OPTIMIZATION THEORY A FIRST COURSE IN OPTIMIZATION THEORY RANGARAJAN K. SUNDARAM New York University CAMBRIDGE UNIVERSITY PRESS Contents Preface Acknowledgements page xiii xvii 1 Mathematical Preliminaries 1 1.1 Notation

More information

MixedÀ¾ нOptimization Problem via Lagrange Multiplier Theory

MixedÀ¾ нOptimization Problem via Lagrange Multiplier Theory MixedÀ¾ нOptimization Problem via Lagrange Multiplier Theory Jun WuÝ, Sheng ChenÞand Jian ChuÝ ÝNational Laboratory of Industrial Control Technology Institute of Advanced Process Control Zhejiang University,

More information

Introduction to Support Vector Machines. Colin Campbell, Bristol University

Introduction to Support Vector Machines. Colin Campbell, Bristol University Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.

More information

Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

More information

Overview of Math Standards

Overview of Math Standards Algebra 2 Welcome to math curriculum design maps for Manhattan- Ogden USD 383, striving to produce learners who are: Effective Communicators who clearly express ideas and effectively communicate with diverse

More information

Chapter 7 Nonlinear Systems

Chapter 7 Nonlinear Systems Chapter 7 Nonlinear Systems Nonlinear systems in R n : X = B x. x n X = F (t; X) F (t; x ; :::; x n ) B C A ; F (t; X) =. F n (t; x ; :::; x n ) When F (t; X) = F (X) is independent of t; it is an example

More information

Many algorithms, particularly divide and conquer algorithms, have time complexities which are naturally

Many algorithms, particularly divide and conquer algorithms, have time complexities which are naturally Recurrence Relations Many algorithms, particularly divide and conquer algorithms, have time complexities which are naturally modeled by recurrence relations. A recurrence relation is an equation which

More information

Metric Spaces. Chapter 7. 7.1. Metrics

Metric Spaces. Chapter 7. 7.1. Metrics Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some

More information

MA651 Topology. Lecture 6. Separation Axioms.

MA651 Topology. Lecture 6. Separation Axioms. MA651 Topology. Lecture 6. Separation Axioms. This text is based on the following books: Fundamental concepts of topology by Peter O Neil Elements of Mathematics: General Topology by Nicolas Bourbaki Counterexamples

More information

BX in ( u, v) basis in two ways. On the one hand, AN = u+

BX in ( u, v) basis in two ways. On the one hand, AN = u+ 1. Let f(x) = 1 x +1. Find f (6) () (the value of the sixth derivative of the function f(x) at zero). Answer: 7. We expand the given function into a Taylor series at the point x = : f(x) = 1 x + x 4 x

More information

Adaptive Search with Stochastic Acceptance Probabilities for Global Optimization

Adaptive Search with Stochastic Acceptance Probabilities for Global Optimization Adaptive Search with Stochastic Acceptance Probabilities for Global Optimization Archis Ghate a and Robert L. Smith b a Industrial Engineering, University of Washington, Box 352650, Seattle, Washington,

More information

Advanced Lecture on Mathematical Science and Information Science I. Optimization in Finance

Advanced Lecture on Mathematical Science and Information Science I. Optimization in Finance Advanced Lecture on Mathematical Science and Information Science I Optimization in Finance Reha H. Tütüncü Visiting Associate Professor Dept. of Mathematical and Computing Sciences Tokyo Institute of Technology

More information

By choosing to view this document, you agree to all provisions of the copyright laws protecting it.

By choosing to view this document, you agree to all provisions of the copyright laws protecting it. This material is posted here with permission of the IEEE Such permission of the IEEE does not in any way imply IEEE endorsement of any of Helsinki University of Technology's products or services Internal

More information

An Overview Of Software For Convex Optimization. Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.

An Overview Of Software For Convex Optimization. Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt. An Overview Of Software For Convex Optimization Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.edu In fact, the great watershed in optimization isn t between linearity

More information

Increasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.

Increasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all. 1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.

More information

Invertible elements in associates and semigroups. 1

Invertible elements in associates and semigroups. 1 Quasigroups and Related Systems 5 (1998), 53 68 Invertible elements in associates and semigroups. 1 Fedir Sokhatsky Abstract Some invertibility criteria of an element in associates, in particular in n-ary

More information

FACTORING POLYNOMIALS IN THE RING OF FORMAL POWER SERIES OVER Z

FACTORING POLYNOMIALS IN THE RING OF FORMAL POWER SERIES OVER Z FACTORING POLYNOMIALS IN THE RING OF FORMAL POWER SERIES OVER Z DANIEL BIRMAJER, JUAN B GIL, AND MICHAEL WEINER Abstract We consider polynomials with integer coefficients and discuss their factorization

More information

4: SINGLE-PERIOD MARKET MODELS

4: SINGLE-PERIOD MARKET MODELS 4: SINGLE-PERIOD MARKET MODELS Ben Goldys and Marek Rutkowski School of Mathematics and Statistics University of Sydney Semester 2, 2015 B. Goldys and M. Rutkowski (USydney) Slides 4: Single-Period Market

More information

Chapter 3 Nonlinear Model Predictive Control

Chapter 3 Nonlinear Model Predictive Control Chapter 3 Nonlinear Model Predictive Control In this chapter, we introduce the nonlinear model predictive control algorithm in a rigorous way. We start by defining a basic NMPC algorithm for constant reference

More information

The Envelope Theorem 1

The Envelope Theorem 1 John Nachbar Washington University April 2, 2015 1 Introduction. The Envelope Theorem 1 The Envelope theorem is a corollary of the Karush-Kuhn-Tucker theorem (KKT) that characterizes changes in the value

More information

15 Kuhn -Tucker conditions

15 Kuhn -Tucker conditions 5 Kuhn -Tucker conditions Consider a version of the consumer problem in which quasilinear utility x 2 + 4 x 2 is maximised subject to x +x 2 =. Mechanically applying the Lagrange multiplier/common slopes

More information

A New Interpretation of Information Rate

A New Interpretation of Information Rate A New Interpretation of Information Rate reproduced with permission of AT&T By J. L. Kelly, jr. (Manuscript received March 2, 956) If the input symbols to a communication channel represent the outcomes

More information

A Branch and Bound Algorithm for Solving the Binary Bi-level Linear Programming Problem

A Branch and Bound Algorithm for Solving the Binary Bi-level Linear Programming Problem A Branch and Bound Algorithm for Solving the Binary Bi-level Linear Programming Problem John Karlof and Peter Hocking Mathematics and Statistics Department University of North Carolina Wilmington Wilmington,

More information

Stationarity Results for Generating Set Search for Linearly Constrained Optimization

Stationarity Results for Generating Set Search for Linearly Constrained Optimization SANDIA REPORT SAND2003-8550 Unlimited Release Printed October 2003 Stationarity Results for Generating Set Search for Linearly Constrained Optimization Tamara G. Kolda, Robert Michael Lewis, and Virginia

More information