

 Warren McDowell
 2 years ago
 Views:
Transcription
1 A Globally Convergent PrimalDual Interior Point Method for Constrained Optimization Hiroshi Yamashita 3 Abstract This paper proposes a primaldual interior point method for solving general nonlinearly constrained optimization problems. The method is based on solving the barrier KarushKuhnTucker conditions for optimality by the Newton method. To globalize the iteration we introduce the barrierpenalty fucntion and the optimality condition for minimizing this function. Our basic iteration is the Newton iteration for solving the optimality conditions with respect to the barrierpenalty function which coincides with the Newton iteration for the barrier KarushKuhn Tucker conditions if the penalty parameter is suciently large. It is proved that the method is globally convergent from an arbitrary initial point that strictly satises the bounds on the variables. Implementations of the given algorithm are done for small dense nonlinear programs. The method solves all the problems in Hock and Schittkowski's textbook eciently. Thus it is shown that the method given in this paper possesses a good theoretical convergence property and is ecient in practice. Key words: interior point method, primaldual method, constrained optimization, nonlinear programming 3 Mathematical Systems, Inc., 243, Shinjuku, Shinjukuku, Tokyo, Japan 1
2 1 Introduction In this paper we propose a primaldual interior point method that solves general nonlinearly constrained optimization problems. To obtain a fast algorithm for nonlinear optimization problems it is fairly clear from various experiences (for example the well known success of the SQP method) that we should eventually solve the KarushKuhnTucker conditions for optimality by a Newtonlike method. However, solving the optimality conditions simply as a system of equations does not give an algorithm for solving optimization problems in general except for convex problems. Therefore it is not appropriate to treat the primal and dual variables equally to obtain globally convergent methods for general nonlinear optimization problems. Observing that there is a nice relation between the parameterized optimality conditions (barrier KKT condition) and the classical logarithmic barrier function of the primal variables, we can develop a globally convergent method for general nonlinear optimization problems. To prevent the possible divergence of the dual variables, we introduce the barrier penalty function and solve the optimality conditions by a Newtonlike method. For nonlinear programs it has been believed long time that interior point methods are not practical because of inevitable numerical diculties which occur at the nal stage of iterations. See Fiacco and McCormick [4], Fletcher [5] and other standard textbooks on nonlinear optimization. Thus the state of the art method for nonlinear programs today is the SQP method (see Powell [7], Fletcher [5]) which can also be interpreted as a Newtonlike method for KarushKuhnTucker conditions near a solution. It is known that the SQP method is ecient and stable for wide range of problems. The SQP method requires the quadratic programming subproblems which handle the combinatorial aspect of the problem caused by inequality constraints. A solution of the quadratic programming problem itself requires rather expensive cost especially for large scale problems. Therefore it is desired to have an ecient and stable interior point method for nonlinear problems in the light of success of that in the eld of large scale linear programs. This paper shows that it is in fact possible to construct such a method that is globally convergent theoretically, and ecient and stable in practice. The method is tested on 115 test problems in Hock and Schittkowski's collection [6]. For these problems our method solves all the problems with 2 to 3 function evaluations per problem and about 2 iterations per problem. Details are described in Section 5. A preliminary report of this paper has appeared in [11]. In this paper, problems to be solved are restricted to small to medium ones because we do not exploit the sparsity of the matrices here. However, recent report by Yamashita, Yabe and Tanabe [13] studies a trust region type method to utilize the sparsity of the Hessian of the Lagrangian. They report eciency of the method that uses the barrier penalty function as in this paper. See also [1] for a trust region type interior point method. It is to be noted that the recent studies by Yamashita and Yabe [12] and Yabe and Yamashita [9] show the superlinear and/or quadratic convergence of a class of primaldual interior point methods that use the Newton or quasinewton iteration for solving the barrier KKT conditions. Other reports on the local behavior of primaldual interior point methods include [2] and [3]. In Section 2, we describe basic concepts in the primaldual interior point method. The 2
3 barrier penalty function which plays a key role in the method of this paper is introduced in Section 3, and analyzed there. A line search method that minimizes the barrier penalty function is described in Section 4, and is proved to be globally convergent. Section 5 reports the results of numerical experiment. Notation. The subscript k denotes an iteration count. Subscripts i and j denote components of vectors and matrices. The superscript t denotes transposes of vectors and matrices. The vector e denotes the vector of all ones and the matrix I the identity matrix. For simplicity of description, we assume k1k denotes the l 2 norm for vectors. The symbol R n denotes the n dimensional real vector space. The set R n + is dened by R n = fx + 2 jx> g. Rn 2 Primaldual interior point method In this paper, we consider the following constrained optimization problem: (1) minimize f(x); x 2 R n ; subject to g(x) = ; x ; where we assume that the functions f : R n! R 1 and g : R n! R m are twice continuously dierentiable. Let the Lagrangian function of the above problem be dened by (2) L(w) =f(x) y t g(x) z t x; where w =(x; y; z) t, and y 2 R m and z 2 R n are the Lagrange multiplier vectors which correspond to the equality and inequality constraints respectively. Then KarushKuhn Tucker (KKT) conditions for optimality of the above problem are given by 1 1 r (w) r xl(w) g(x) CA = (3) CA XZe and (4) where x ; z ; r x L(w) = rf(x) A(x) t y z; A(x) = rg 1(x) t. CA ; rg m (x) t 1 X = diag (x 1 ; 111;x n ) ; Z = diag (z 1 ; 111;z n ) : Now we approximate problem (1) by introducing the barrier function F B (; ) :R n +! R 1, (5) minimize subject to g(x) = ; F B (x; ) =f(x) np 3 log(x i );x 2 R n +
4 where the barrier parameter > is a given constant. It is well known that, if is suciently small, problem (5) is a good approximation to original problem (1) (see Fiacco and McCormick [4]). The optimality conditions for (5) are given by (6) rf (x) A(x) t y X 1 e = ; g(x) = and x>; where y 2 R m is the Lagrange multiplier for the equality constraints. If we introduce an auxiliary variable z 2 R n which is to be equal to X 1 e, then the above conditions become the conditions (7) and (8) r(w; ) r xl(w) g(x) XZe e x>; 1 CA = z>: The introduction of the variable z is essential to the numerical success of the barrier based algorithm in this paper. In this paper we call conditions (7) the barrier KKT conditions, and a point w() = (x();y();z()) that satises these conditions is called the barrier KKT point. We will use an interior point method for searching a point that approximately satises the above conditions, and nally obtain a point that satises the KarushKuhnTucker conditions by letting #. This means that we force x and z be strictly positive during iterations. Therefore we delete inequality conditions (8) hereafter, and always assume that x and z are strictly positive in what follows. Here we note that (9) where r(w; ) =r (w) ^e; ^e = e An algorithm of this paper approximately solves the sequence of conditions (7) with a decreasing sequence of the barrier parameter that tends to, and thus obtains an approximate solution to KKT conditions. For deniteness, we describe a prototype of such algorithm as follows. Algorithm IP Step. (Initialize) Set ">, M c > and k =. Let a positive sequence f k g ; k # be given. 1 CA : Step 1. (Termination) If kr (w)k ", then stop. 1 CA ; 4
5 Step 2. (Approximate barrier KKT point) Find a point w k+1 that satises (1) kr(w k+1 ; k )km c k : Step 3. (Update) Set k := k + 1 and go to Step 1. 2 The following theorem shows the global convergence property of Algorithm IP. Theorem 1 Let fw k g be an innite sequence generated by Algorithm IP. Suppose that the sequences fx k g and fy k g are bounded. Then fz k g is bounded, and any accumulation point of fw k g satises KKT conditions (3) and (4). Proof. Assume that there exists an i such that (z k ) i!1. Equation (1) yields (rf(x k) A(x k ) t y k ) i (z k ) i 1 M c k1 (z k ) i ; which isacontradiction because of the boundedness of fx k g and fy k g.thus the sequence fz k g is bounded. Let ^w be any accumulation point offw k g. Since the sequences fw k g and f k g satisfy (1) for each k and k approaches zero, r (^w) = follows from the denition of r(w; ). Therefore the proof is complete. 2 We note that the barrier parameter sequence f k g in Algorithm IP need not be determined beforehand. The value of each k may be set adaptively as the iteration proceeds. An example of updating method of k is described in Section 5. We call condition (1) the approximate barrier KKT condition, and call a point that satises this condition the approximate barrier KKT point. To nd an approximate barrier KKT point for a given >, we use the Newtonlike method in this paper. Let 1w =(1x; 1y; 1z) t be dened by a solution of (11) where (12) J(w) = J(w)1w = r(w; ); G A(x)t I A(x) Z X Then the basic iteration of the Newtonlike method may be described as 1 CA : (13) w k+1 = w k +3 k 1w k ; where 3 k = diag( xk I n ; yk I m ; zk I n ) is composed of step sizes in x, y and z variables. If G = r 2 L(w), then 1w becomes Newton's direction for solving (7). To solve (11), we x split the equations into two groups. Thus we solve! G + X 1 Z A(x) t 1x!! = r xl(w)+x 1 e z (14) ; A(x) 1y g(x) 5
6 for (1x; 1y) t, then we obtain 1z by the third equation in (11). If G + X 1 Z is positive denite and A(x) is of full rank, then the coecient matrix in (14) is nonsingular. It will be useful to note that (14) can be written as (15) where (16) G + X 1 Z A(x) t A(x)! 1x ey ~y = y +1y:! = rf(x)+x1 e g(x) In this paper, we will deal with the case in which the matrix G can be assumed nonnegative denite. Therefore, to solve the general nonlinear problems, we will use a positive denite quasinewton approximation to the Hessian matrix of the Lagrangian function to obtain the desired property of the matrix G.! ; 3 Barrier penalty function To globalize the convergence property ofaninterior point algorithm based on the above iteration, we introduce two auxiliary problems. Firstly mp we dene the following problem: minimize F P (x; ) =f(x)+ jg i (x)j; x 2 R n ; (17) subject to x ; where the penalty parameter is a given positive constant. The necessary conditions for optimality of this problem are (see 14.2 of Fletcher [5]) r x L(w) = ; ( mx ) (18) y jg i (x)j ; XZe = ; x ;z ; where the means the subdierential of the function in the braces with respect to g. In our case the second condition in (18) is equivalent to (19) y i ; g i (x) =; y i = ; g i (x) > ; y i =; g i (x) < ; for each i =1; 111;m. This condition can be expressed as jg i (x)j = y i g i (x); y i ; i =1; 111;m; or jg i (x)j + y ig i (x) =; y i ; i =1; 111;m: 6
7 Therefore conditions (18) can be written as r (w) = r xl(w) (2) r E (w) XZe and (21) where 1 CA = x ; z ; y i ; i =1; 111;m; 1 CA r E (w) i = jg i (x)j + y ig i (x) ;; 111;m: Note that we are using the same symbol r (w) to denote the residual vector of the optimality conditions as in Section 2 for simplicity. Ifkyk 1 <, conditions (18) are equivalent to conditions (3) and (4). In this sense, problem (17) is equivalent to problem (1). Next we introduce the barrier penalty function F (; ; ) :R n +! R1 by (22) F (x; ; ) =f(x) nx log x i + mx jg i (x)j ; where and are given positive constants. This function plays an essential role in the method given in this paper. Let us approximate problem (17) by the following problem (23) minimize F (x; ; ); x 2 R n + : The necessary conditions for optimality of a solution to the above problem are r x L(w) = ; ( mx ) (24) y jg i (x)j XZe = e; x > ; z>; where we introduce an auxiliary variable z 2 R n as in (7). As above, conditions (24) can be written as r(w; ) xl(w) (25) r E (w) CA = CA XZe e and (26) 1 x>; z>; y i ; i =1; 111;m; where we use the same symbol r(w; ) as in Section 2 for simplicity. If>kyk 1 then conditions (25) and (26) coincide with (7) and (8). We call a point w() =(x();y();z()) 2 R n 2 R m 2 R n that satises (24) for a given > the barrier KKT point for this as before. We can use Algorithm IP to solve (17). The following theorem shows the global convergence property of Algorithm IP for solving (17). 1 7
8 Theorem 2 Let fw k g be an innite sequence generated byalgorithm IP for solving (17). Suppose that the sequences fx k g is bounded. Then fz k g is bounded, and any accumulation point of fw k g satises the optimality conditions (2) and (21) for problem (17). 2 Now we formulate a Newtonlike iteration for solving the above conditions (25). Thus we calculate the rst order change of (24) with respect to a change in w. This gives (G + X 1 Z)1x A(x) t 1y = r x L(w)+X 1 e z; ( mx ~y y +1y g i (x)+rg i (x) 1x ) (27) t ; 1z = X 1 Z1x + X 1 e z: Following lemma gives a basic property of the iteration vector 1w =(1x; 1y; 1z) t. Lemma 1 Suppose that 1w satises (27) at an interior point w. (i) If >keyk 1, then 1w is identical to the one given by (11). (ii) If 1w =, then the point w isabarrier KKT point that satises (24). (iii) If 1x =, then the point (x; y +1y; z +1z) is a barrier KKT point that satises (24). 2 If we consider the subproblem: (28) minimize 1 2 1xt (G+X 1 Z)1x+(rf(x)X 1 e) t 1x+ mx g i (x)+rg i (x) t 1x ; 1x 2 R n ; the solution vector 1x and the corresponding multiplier vector ~y that satisfy the necessary conditions for optimality also satisfy conditions (27). If G + X 1 Z is positive denite we can solve the problem (28) by a straightforward active set method which starts with an active set that contains all the constraints g i (x)+rg i (x) t 1x =;; 111m. An example of the procedure is described in Fletcher [5]. It is important to note that, if the vectors 1x and ~y obtained by solving (15) satisfy >k~yk 1, then the desired iteration vector that satises (27) are also obtained. The procedure described above is devised instead of the simple Newton iteration given in Section 2 in order to show away of preventing possible divergence of the dual variable y. However for the practical purpose it seems sucient to solve only equation (11) once per iteration as shown in sections below in which practical experiences obtained by the author on general nonlinear programming problems are described. Therefore this procedure is of only theoretical importance at present. As shown above, optimality conditions (2) and (21) are identical to optimality conditions (3) and (4) if we set = 1. Also, Newtonlike iteration (27) coincides with (11) when = 1. Therefore we will regard that the method explained below includes a method for solving (1) when = 1 for simplicity of exposition. We note that the penalty parameter which will appear below should be nite even if = 1. Another interesting point to be noted in the above iteration is that it can give a solution to an infeasible problem. Even if the constraints in problem (1) are incompatible, the 8
9 method described above will give a solution to problem (17) as shown below. A solution to problem (17) with infeasible constraints may give useful information about problem (1). Now we proceed to an analysis of the properties of the barrier penalty function. The directional derivative F (x; ; ; s) of the function F (x; ; ) along an arbitrary given direction s is dened by F F (x + s; ; ) F (x; ; ) (x; ; ; s) = lim # = rf(x) X X t s e t X 1 s + rg i (x) t s + + rg i (x) s X t rg i (x) t s; where the summations in the above equation are to be understood as X X a i = a i X X ; a i = a i X X ; a i = a i : + g i > We introduce a rst order approximation F l of F (x + s; ; ) by F l (x; ; ; s) f(x)+rf (x) t s log(x i )+ s i (29) + mx g i = nx g i (x)+rg i (x) t s; g i < x i and an estimate of the rst order change 1F l of F by (3) 1F l (x; ; ; s) F l (x; ; ; s) F (x; ; ); = rf(x) t s e t X 1 s + mx jg i (x)+rg i (x)sj mx jg i (x)j for an arbitrary given direction s. We have following properties for the quantities dened above which give an extension to the similar properties in the case of dierentiable functions (see for example Yamashita [1]). Lemma 2 Let >, > and s 2 R n be given. Then the following assertions hold. (i) The function F l (x; ; ; s) is convex with respect to the variable. (ii) There holds the relation (31) F (x; ; )+F (x; ; ; s) F l (x; ; ; s): (iii) Further, there exists a 2 (; 1) such that (32) F (x + s; ; ) F (x; ; )+F (x + s; ; ; s); whenever x + s>. 9
10 Proof. The rst statement of the lemma is obvious. If > is suciently small, we have F l (x; ; ; s) = F (x; ; )+F (x; ; ; s) = F (x; ; )+F (x; ; ; s): Since F l (x; ; ; s) isconvex with respect to and coincides with a linear function of > when is suciently small, we obtain (31). Now we show (32). If we consider F (x + s; ; ; s) as a function of the variable 2 [; 1], the number of discontinuous points of F is nite. Therefore there exists a 2 (; 1) such that F (x + s; ; ; s) Z1 F (x + s; ; ; s)d = F (x + s; ; ) F (x; ; ): This completes the proof. 2 Lemma 3 Let " 2 (; 1) be a given constant and s 2 R n be given. If 1F l (x; ; ; s) <, then (33) F (x + s; ; ) F (x; ; ) " 1F l (x; ; ; s); for suciently small >. Proof. From (32), there exists a 2 (; 1) such that (34) F (x + s; ; ) F (x; ; ) F (x + s; ; ; s) = F (x + s; ; ; s): From (31) we have (35) F (x + s; ; ; s) 1F l (x + s; ; ; s): Because 1F l (; ; ; s) is continuous, and 1F l (x; ; ; s) < by the assumption, we obtain (36) 1F l (x + s; ; ; s) " 1F l (x; ; ; s); for suciently small ksk. From (34)  (36) we obtain (33) for suciently small >. 2 Lemma 4 Suppose that 1w satises (27). If <, then (37) 1F l (x; ; ;1x) 1x t (G + X 1 Z)1x mx ( j~y i j)jg i (x)j: Further if, G is positive semidenite and keyk 1 1F l (x; ; ;1x) =yields 1x =., then 1F l (x; ; ;1x), and 1
11 Proof. From (27) and (3) we have 1F l (x; ; ;1x) =1x t (G+X 1 Z)1x+1x t A(x) t ~y+ The ith components in the summations in the last three terms give mx g i (x)+rg i (x) t 1x mx jg i (x)j: (38) ~y i rg i (x) t 1x + g i (x)+rg i (x) 1x t jg i (x)j ~y i rg i (x) t 1x + g i (x)+rg i (x) 1x t jg i (x)j = ~y i g i (x) jg i (x)j ; where the inequality in the second line follows from, and the equality in the third line follows from the property ~y i ; g i (x)+rg i (x) t 1x =; ~y i = ; g i (x)+rg i (x) t 1x>; ~y i =; g i (x)+rg i (x) t 1x<: The relation (38) gives (37). 2 4 Line search algorithm To obtain a globally convergent algorithm to a barrier KKT point for a xed >, it is necessary to modify the basic Newton iteration with the unit step length somehow. Our iterations consist of (39) x k+1 = x k + xk 1x k ; y k+1 = y k + yk 1y k ; z k+1 = z k + zk 1z k ; where xk, yk and zk are step sizes determined by the line search procedures described below. The main iteration is to decrease the value of the barrier penalty function for xed. Thus the step size of the primal variable x is determined by the sucient decrease rule of the merit function. The step size of the dual variable z is determined so as to stabilize the iteration. The explicit rules follow in order. We adopt Armijo's rule as the line search rule for the variable x. At the point x k,we calculate the maximum allowed step to the boundary of the feasible region by (4) kmax = min i ( (x k) i (1x k ) i (1x k) i < i.e., the step size kmax gives an innitely large value of the barrier penalty function F if it exists, because of the barrier terms, and a step size 2 [; kmax ) gives a strictly feasible primal variable. A step to the next iterate is given by ) ; (41) xk = k l k ; k = min f kmax ; 1g ; 11
12 where 2 (; 1) and 2 (; 1) are xed constants and l k is the smallest nonnegative integer such that (42) F (x k + k l k 1x k; ; ) F (x k ; ; ) " k l k 1F l(x k ; ; ;1x k ); where " 2 (; 1). Typical values of these parameters are = :5, = :9995 and " =1 6. Therefore we will try the sequence x k +:9995 kmax 1x k ; x k +:5 2 :9995 kmax 1x k ; x k +:25 2 :9995 kmax 1x k ; 111 for example, and will nd a step size that satises (42). If G is positive semidenite, then 1F l (x k ; ; ;1x k ) by Lemma 4, and therefore the existence of such steps is assured by Lemma 3. For the variable z, we adopt the box constraints rule, i.e., we force x and z to satisfy the condition (43) c Lki ((x k ) i + xk (1x k ) i )((z k ) i + zk (1z k ) i ) c Uki ; i =1; 111;n at the end of each iteration, where the bounds c Lk and c Uk satisfy (44) <c Lki <<c Uki ; i =1; 111;n: To this end, we let (45) c Lki = minn M L ; ((x k ) i + xk (1x k ) i )(z k ) io ; c Uki = max fm U ; ((x k ) i + xk (1x k ) i )(z k ) i g ; where M L > 1 and M U > 1 are given constants. The construction of the above bounds shows that current z satises (46) (47) c Lki ((x k ) i + xk (1x k ) i ) (z k) i The step size z is determined by zk = min ( min i ( max i ( i c Uki ((x k ) i + xk (1x k ) i ) ; c Lki ((x k ) i + xk (1x k ) i ) c Uki ((z k ) i + i (1z k ) i ) ((x k ) i + xk (1x k ) i ) i =1; 111;n: )) ) ; 1 : The rule (47) means that the step size z is the maximal allowed step that satises the box constraints with the restriction of being not greater than the unit step length. Lemma 5 Suppose that an innite sequence fw k g is generated for xed >. Then if lim inf k!1(x k ) i > and lim sup k!1(x k ) i < 1, then lim inf k!1(c Lk ) i > and lim sup k!1(c Uk ) i < 1 for i =1; 111;n. 12
13 Proof. Suppose that (c Lk ) i! for an i and some subsequence K f; 1; 2; 111g. Then by the denition of (c Lk ) i in (45), (z k ) i! ;k 2 K. However, in order for a subsequence of f(z k ) i g to tend to, there must be an iteration k at which the lower bound (c Lk ) i =(x k+1 ) i of (z k ) i is arbitrary small and the value of (z k ) i at the iteration is strictly larger than that bound, i.e. at the iteration the value of (z k ) i decreases to a strictly smaller value. This means that at iteration k, (c Lk ) i = =M L from the denition (45), and therefore the value of (x k+1 ) i must be arbitrary large because =M L < (x k+1 ) i (z k ) i and (z k ) i! ;k 2 K. This is impossible because of the assumption of the lemma. The proof of the boundedness of (c Uk ) i is similar. 2 (48) In actual calculation we modify the direction 1z k (1z k ) i = 8>< >: ; if (z k) i = c Lki=(x k+1) i and (1z k ) i < ; ; if (z k ) i = c Uki =(x k+1 ) i and (1z k ) i > ; (1z k ) i ; otherwise: This modication means that we project the direction along the boundary of the box constraints if the point z k is on that boundary and the direction 1z k points outward of the box. This procedure is adopted because it gives better numerical results. The global convergence results shown in the following are equally valid for both unmodied and modied directions. For the variable y, there exist three obvious choices for the step length: by (49) yk =1 or xk or zk : The global convergence property given below holds for these choices. We choose yk = zk from numerical experiments. The following algorithm describes the iteration for xed > and >. We note that this algorithm corresponds to Step 2 of Algorithm IP in Section 2. Algorithm LS Step. (Initialize) Let w 2 R n + 2 Rm 2 R n +, and >, >. Set " >, 2 (; 1), 2 (; 1), " 2 (; 1), M L > 1 and M U > 1. Let k =. Step 1. (Termination) If kr(w k ;)k " ; then stop. Step 2. (Compute direction) Calculate the direction 1w k by (27). Step 3. (Stepsize) Set kmax = min i ( (x k) i ) (1x k ) i (1x k) i < ; k = min f kmax ; 1g : Find the smallest nonnegative integer l k that satises F (x k + k l k 1x k ;;) F (x k ;;) " k l k 1F l (x k ; ; ;1x k ): Calculate xk = k l k ; 13
14 c Lki = min ; (x k + xk 1x k ) i (z k ) i ; M L c Uki = max fm U ; (x k + xk 1x k ) i (z k ) i g ; ( ( ( i c Lki zk = min min max i i ((x k ) i + xk (1x k ) i c Uki )) ) ((z k ) i + i (1z k ) i ) ; 1 ; ((x k ) i + xk (1x k ) i ) yk = zk ; 3 k = diagf xk I n ; yk I m ; zk I n g: Step 4. (Update variables) Set w k+1 = w k +3 k 1w k : Step 5. Set k := k + 1 and go to Step 1. 2 To prove global convergence of Algorithm LS, we need the following assumptions. Assumption G (1) The functions f and g i ;i =1; :::; m, are twice continuously dierentiable. (2) The level set of the barrier penalty function at an initial point x 2 R n +, which is dened by nx 2 R n jf (x; ; ) + F (x ; ; )o, is compact for given >. (3) The matrix A(x) is of full rank on the level set dened in (2). (4) The matrix G k is positive semidenite and uniformly bounded. (5) The penalty parameter satises ky k +1y k k 1 for each k =; 1; :::. 2 We note that if a quasinewton approximation is used for computing the matrix G k, then we need the continuity of only the rst order derivatives of functions in Assumption G(1). We also note that if 1F l (x k ; ; ;1x k ) =, at an iteration k, then the step sizes xk = yk = zk = 1 are adopted and (x k+1 ;y k+1 ;z k+1 ) gives a barrier KKT point from Lemma 1 and Lemma 4. The following theorem gives a convergence of an innite sequence generated by Algorithm LS. Theorem 3 Let an innite sequence fw k g be generated byalgorithm LS. Then there exists at least one accumulation point of fw k g, and any accumulation point of the sequence fw k g is a barrier KKT point. 14
15 Proof. First we note that each component of the sequence fx k g is bounded away from zero and bounded above by the assumption and the existence of the log barrier term. Therefore the sequence fx k g has at least one accumulation point. The sequence fz k g also has these properties by Lemma 5. Thus there exists a positive number M such that (5) kpk 2 M pt (G k + X 1 k Z k)p M kpk 2 ; 8p 2 R n ; by the assumption. From (37) and (5), we have (51) and from (42), 1F l (x k ; ; ;1x k ) k1x kk 2 M < ; (52) F (x k+1 ; ; ) F (x k ; ; ) " k l k 1F l(x k ; ; ;1x k ) " k l k k1x kk 2 M < : Because the sequence ff (x k ; ; )g is decreasing and bounded below, the left hand side of (52) converges to. Since lim inf k!1(x k ) i > ;i =1; 111;n,wehave lim inf k!1 k >. Suppose that there exists a subsequence K f; 1; 111g and a such that (53) lim inf k!1 k1x kk>; k 2 K: Then we have l k!1;k 2 K from (52) because the left most expression tends to zero, and therefore we can assume l k > for suciently large k 2 K without loss of generality. If l k > then the point x k + xk 1x k = does not satisfy the condition (42). Thus, we have (54) F (x k + xk 1x k =; ; ) F (x k ; ; ) >" xk 1F l (x k ; ; ;1x k )=: By (32) and (31), there exists a k 2 (; 1) such that (55) F (x k + xk 1x k =; ; ) F (x k ; ; ) xk F (x k + k xk 1x k =; ; ;1x k )= xk 1F l (x k + k xk 1x k =; ; ;1x k )=; k 2 K: Then, from (54) and (55), we have " 1F l (x k ; ; ;1x k ) < 1F l (x k + k xk 1x k =; ; ;1x k ): This inequality yields (56) 1F l (x k + k xk 1x k =; ; ;1x k ) 1F l (x k ; ; ;1x k ) > (" 1)1F l (x k ; ; ;1x k ) > : Because 1x k is a solution of problem (28) and there holds (5), k1x k k is uniformly bounded above. Then by the property l k! 1,we have k k xk 1x k =k!;k 2 K. 15
16 Thus the left hand side of (56) and therefore 1F l (x k ; ; ;1x k ) converges to zero when k!1;k 2 K. This contradicts the assumption (53) because we have 1x k! ;k 2 K from (51). Therefore we proved (57) lim k1x k k =: k!1;k2k Let an arbitrary accumulation point of the sequence fx k g be ^x 2 R n + and let x k! ^x; k 2 K for K f; 1; 111g.Thus (58) x k! ^x; 1x k! ; x k+1! ^x; k 2 K: Because nx 1 Z k ko is bounded, we have lim k!1;k2kz k +1z k X e 1 = k from (27). If we dene ^z = ^X 1 e where ^X = diag(^x 1 ; 111; ^x n ), then we have Hence from (45) we have z k +1z k! ^z; k 2 K: (c Lk ) i M L (x k+1 ) i (z k +1z k ) i M U (C Uk ) i ; i =1; 111;n for k 2 K suciently large, which shows that the point z k +1z k is always accepted as z k+1 for suciently large k 2 K. Since zk = 1 is accepted for k 2 K suciently large, so is yk =1. Therefore we obtain lim r x L(^x; y k +1y k ; ^z) = ; k!1;k2k lim y k +1y k k!1;k2k ( nx ) jg i (^x)j : Because the matrix A(^x) is of full rank, the sequence fy k +1y k g ;k 2 K converges to a point ^y 2 R m which satises r x L(^x; ^y; ^z) = ; mx! ^y jg i (^x)j ; ^X ^z = e; ^x >; ^z >: This completes the proof because we proved that there exists at least one accumulation point offx k g, and for an arbitrary accumulation point ^x of fx k g, there exist unique ^y and ^z that satisfy the above. 2 16
17 5 Numerical Result In this section, we report numerical results of an implementation of the algorithm given in this paper for nonlinear programming problems. We set = 1 in this experiment. The software is called NUOPT and the code is written bytakahito Tanabe. In order to havean appropriate positive semidenite matrix G by a reasonable cost for nonlinear problems, we resort to a quasinewton approximation to the Hessian matrix of the Lagrangian function. We use updating formula suggested by Powell[7] for the SQP method: G k+1 = G k G ks k s t k G k s t k G ks k + u ku t k s t u ; k k where u k is calculated by s k = x k+1 x k ; v k = r x L(x k+1 ;y k+1 ;z k+1 ) r x L(x k ;y k+1 ;z k+1 ); u k = k v k +(1 k )G k s k ; k = 8< : :8s t k G k s k s t k G k s ks t k v k 1; s t k v k :2s t k G ks k ; ; s t k v k :2s t k G ks k ; to satisfy s t u k k > for the hereditary positive deniteness of the update. Method for updating the barrier parameter k is as follows. Suppose we have an approximate barrier KKT point w k+1 that satises kr(w k+1 ; k )km c k ; in Step 2 of Algorithm IP. Then k+1 is dened by k+1 = max( kr(wk+1 ; k ) )k ; k ; M M where < M c < M and M > 1 should be satised. In our experiment weset M c =3;M =4;M = 5. As in the SQP method, we expect fast local convergence of the method if is suciently small near a solution because it is based on the Newton iteration for the optimality conditions and a quasinewton approximation to the second derivative matrix. As noted in the above, this expectation is proved by Yamashita and Yabe [12]. Linear equation (14 ) is solved by using the BunchParlett factorization. The test problems for nonlinear problems are adopted from the book by Hock and Schittkowski [6]. The results are summarized in Table 1 at the end of this paper. Following list explains the notations used in Table 1: n= number of variables. m=number of constraints. obj=nal objective function value. res=norm of nal KKT condition residual. itr=iteration count. 17
constraint. Let us penalize ourselves for making the constraint too big. We end up with a
Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the
More informationNonlinear Optimization: Algorithms 3: Interiorpoint methods
Nonlinear Optimization: Algorithms 3: Interiorpoint methods INSEAD, Spring 2006 JeanPhilippe Vert Ecole des Mines de Paris JeanPhilippe.Vert@mines.org Nonlinear optimization c 2006 JeanPhilippe Vert,
More informationFurther Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1
Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing
More informationNumerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen
(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained
More informationDate: April 12, 2001. Contents
2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........
More information2.3 Convex Constrained Optimization Problems
42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions
More information(Quasi)Newton methods
(Quasi)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable nonlinear function g, x such that g(x) = 0, where g : R n R n. Given a starting
More information(We assume that x 2 IR n with n > m f g are twice continuously ierentiable functions with Lipschitz secon erivatives. The Lagrangian function `(x y) i
An Analysis of Newton's Metho for Equivalent Karush{Kuhn{Tucker Systems Lus N. Vicente January 25, 999 Abstract In this paper we analyze the application of Newton's metho to the solution of systems of
More informationDuality in General Programs. Ryan Tibshirani Convex Optimization 10725/36725
Duality in General Programs Ryan Tibshirani Convex Optimization 10725/36725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T
More informationNonlinear Programming Methods.S2 Quadratic Programming
Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective
More informationIntroduction to Convex Optimization for Machine Learning
Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning
More informationInterior Point Methods and Linear Programming
Interior Point Methods and Linear Programming Robert Robere University of Toronto December 13, 2012 Abstract The linear programming problem is usually solved through the use of one of two algorithms: either
More information10. Proximal point method
L. Vandenberghe EE236C Spring 201314) 10. Proximal point method proximal point method augmented Lagrangian method MoreauYosida smoothing 101 Proximal point method a conceptual algorithm for minimizing
More informationAdaptive Online Gradient Descent
Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650
More informationSummer course on Convex Optimization. Fifth Lecture InteriorPoint Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.
Summer course on Convex Optimization Fifth Lecture InteriorPoint Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota InteriorPoint Methods: the rebirth of an old idea Suppose that f is
More informationINTERIORPOINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS
INTERIORPOINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS HANDE Y. BENSON, DAVID F. SHANNO, AND ROBERT J. VANDERBEI Operations Research and Financial Engineering Princeton
More informationBig Data  Lecture 1 Optimization reminders
Big Data  Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data  Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics
More informationData analysis in supersaturated designs
Statistics & Probability Letters 59 (2002) 35 44 Data analysis in supersaturated designs Runze Li a;b;, Dennis K.J. Lin a;b a Department of Statistics, The Pennsylvania State University, University Park,
More informationConvex Optimization SVM s and Kernel Machines
Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.
More informationt := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).
1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction
More information24. The Branch and Bound Method
24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NPcomplete. Then one can conclude according to the present state of science that no
More informationNo: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics
No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results
More informationConvex Programming Tools for Disjunctive Programs
Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible
More informationPRIMALDUAL METHODS FOR LINEAR PROGRAMMING
PRIMALDUAL METHODS FOR LINEAR PROGRAMMING Philip E. GILL, Walter MURRAY, Dulce B. PONCELEÓN and Michael A. SAUNDERS Technical Report SOL 913 Revised March 1994 Abstract Many interiorpoint methods for
More informationDuality of linear conic problems
Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least
More informationSolving linear equations on parallel distributed memory architectures by extrapolation Christer Andersson Abstract Extrapolation methods can be used to accelerate the convergence of vector sequences. It
More informationBANACH AND HILBERT SPACE REVIEW
BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but
More information1 if 1 x 0 1 if 0 x 1
Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or
More informationInteriorPoint Algorithms for Quadratic Programming
InteriorPoint Algorithms for Quadratic Programming Thomas Reslow Krüth Kongens Lyngby 2008 IMMM.Sc200819 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK2800
More informationSupport Vector Machines
Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric
More informationStochastic Inventory Control
Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the
More informationNORTHWESTERN UNIVERSITY Department of Electrical Engineering and Computer Science LARGE SCALE UNCONSTRAINED OPTIMIZATION by Jorge Nocedal 1 June 23, 1996 ABSTRACT This paper reviews advances in Newton,
More informationSolving polynomial least squares problems via semidefinite programming relaxations
Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose
More informationFactorization Theorems
Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization
More informationA LagrangianDNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems
A LagrangianDNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems Sunyoung Kim, Masakazu Kojima and KimChuan Toh October 2013 Abstract. We propose
More informationThe Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Linesearch Method
The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Linesearch Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem
More information4.6 Linear Programming duality
4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More information1 Norms and Vector Spaces
008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)
More informationSECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA
SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the
More informationLecture 13 Linear quadratic Lyapunov theory
EE363 Winter 289 Lecture 13 Linear quadratic Lyapunov theory the Lyapunov equation Lyapunov stability conditions the Lyapunov operator and integral evaluating quadratic integrals analysis of ARE discretetime
More informationAPPLYING NEW OPTIMIZATION CONTROL. Stephen J. Wright. Argonne, IL 60439
APPLYING NEW OPTIMIZATION ALGORITHMS TO MODEL PREDICTIVE CONTROL Stephen J Wright Mathematics and Computer Science Division Argonne National Laboratory Argonne, IL 9 Abstract The connections between optimization
More informationModern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh
Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem
More informationLinear Programming Notes V Problem Transformations
Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material
More informationRegression Using Support Vector Machines: Basic Foundations
Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering
More information3 Does the Simplex Algorithm Work?
Does the Simplex Algorithm Work? In this section we carefully examine the simplex algorithm introduced in the previous chapter. Our goal is to either prove that it works, or to determine those circumstances
More informationRi and. i=1. S i N. and. R R i
The subset R of R n is a closed rectangle if there are n nonempty closed intervals {[a 1, b 1 ], [a 2, b 2 ],..., [a n, b n ]} so that R = [a 1, b 1 ] [a 2, b 2 ] [a n, b n ]. The subset R of R n is an
More informationNotes on Factoring. MA 206 Kurt Bryan
The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor
More informationProblem 1 (10 pts) Find the radius of convergence and interval of convergence of the series
1 Problem 1 (10 pts) Find the radius of convergence and interval of convergence of the series a n n=1 n(x + 2) n 5 n 1. n(x + 2)n Solution: Do the ratio test for the absolute convergence. Let a n =. Then,
More informationA NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION
1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of
More informationRandom graphs with a given degree sequence
Sourav Chatterjee (NYU) Persi Diaconis (Stanford) Allan Sly (Microsoft) Let G be an undirected simple graph on n vertices. Let d 1,..., d n be the degrees of the vertices of G arranged in descending order.
More informationApplied Algorithm Design Lecture 5
Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design
More informationOPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN. E.V. Grigorieva. E.N. Khailov
DISCRETE AND CONTINUOUS Website: http://aimsciences.org DYNAMICAL SYSTEMS Supplement Volume 2005 pp. 345 354 OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN E.V. Grigorieva Department of Mathematics
More informationChapter 7. Continuity
Chapter 7 Continuity There are many processes and eects that depends on certain set of variables in such a way that a small change in these variables acts as small change in the process. Changes of this
More informationContinued Fractions and the Euclidean Algorithm
Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction
More informationThe Heat Equation. Lectures INF2320 p. 1/88
The Heat Equation Lectures INF232 p. 1/88 Lectures INF232 p. 2/88 The Heat Equation We study the heat equation: u t = u xx for x (,1), t >, (1) u(,t) = u(1,t) = for t >, (2) u(x,) = f(x) for x (,1), (3)
More informationSolutions Of Some NonLinear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR
Solutions Of Some NonLinear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision
More informationHOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!
Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following
More informationMeasuring Rationality with the Minimum Cost of Revealed Preference Violations. Mark Dean and Daniel Martin. Online Appendices  Not for Publication
Measuring Rationality with the Minimum Cost of Revealed Preference Violations Mark Dean and Daniel Martin Online Appendices  Not for Publication 1 1 Algorithm for Solving the MASP In this online appendix
More informationPUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5.
PUTNAM TRAINING POLYNOMIALS (Last updated: November 17, 2015) Remark. This is a list of exercises on polynomials. Miguel A. Lerma Exercises 1. Find a polynomial with integral coefficients whose zeros include
More informationDefinition of a Linear Program
Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1
More informationPractical Guide to the Simplex Method of Linear Programming
Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear
More information1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.
Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S
More informationThe Goldberg Rao Algorithm for the Maximum Flow Problem
The Goldberg Rao Algorithm for the Maximum Flow Problem COS 528 class notes October 18, 2006 Scribe: Dávid Papp Main idea: use of the blocking flow paradigm to achieve essentially O(min{m 2/3, n 1/2 }
More informationLecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method
Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming
More informationLowcomplexity minimization algorithms
NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS Numer. Linear Algebra Appl. 2005; 12:755 768 Published online 15 June 2005 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/nla.449 Lowcomplexity
More informationAdvanced Microeconomics
Advanced Microeconomics Ordinal preference theory Harald Wiese University of Leipzig Harald Wiese (University of Leipzig) Advanced Microeconomics 1 / 68 Part A. Basic decision and preference theory 1 Decisions
More informationOptimal shift scheduling with a global service level constraint
Optimal shift scheduling with a global service level constraint Ger Koole & Erik van der Sluis Vrije Universiteit Division of Mathematics and Computer Science De Boelelaan 1081a, 1081 HV Amsterdam The
More informationarxiv:1112.0829v1 [math.pr] 5 Dec 2011
How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly
More informationEcient approximation algorithm for minimizing makespan. on uniformly related machines. Chandra Chekuri. November 25, 1997.
Ecient approximation algorithm for minimizing makespan on uniformly related machines Chandra Chekuri November 25, 1997 Abstract We obtain a new ecient approximation algorithm for scheduling precedence
More information3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field
3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field 77 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field Overview: The antiderivative in one variable calculus is an important
More informationBuered Probability of Exceedance: Mathematical Properties and Optimization Algorithms
Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms Alexander Mafusalov, Stan Uryasev RESEARCH REPORT 20141 Risk Management and Financial Engineering Lab Department of
More informationGenOpt (R) Generic Optimization Program User Manual Version 3.0.0β1
(R) User Manual Environmental Energy Technologies Division Berkeley, CA 94720 http://simulationresearch.lbl.gov Michael Wetter MWetter@lbl.gov February 20, 2009 Notice: This work was supported by the U.S.
More informationTopic 1: Matrices and Systems of Linear Equations.
Topic 1: Matrices and Systems of Linear Equations Let us start with a review of some linear algebra concepts we have already learned, such as matrices, determinants, etc Also, we shall review the method
More informationMathematics Course 111: Algebra I Part IV: Vector Spaces
Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 19967 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are
More informationRecall that the gradient of a differentiable scalar field ϕ on an open set D in R n is given by the formula:
Chapter 7 Div, grad, and curl 7.1 The operator and the gradient: Recall that the gradient of a differentiable scalar field ϕ on an open set D in R n is given by the formula: ( ϕ ϕ =, ϕ,..., ϕ. (7.1 x 1
More informationCHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e.
CHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e. This chapter contains the beginnings of the most important, and probably the most subtle, notion in mathematical analysis, i.e.,
More informationBIG DATA PROBLEMS AND LARGESCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION
BIG DATA PROBLEMS AND LARGESCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION Ş. İlker Birbil Sabancı University Ali Taylan Cemgil 1, Hazal Koptagel 1, Figen Öztoprak 2, Umut Şimşekli
More informationMany algorithms, particularly divide and conquer algorithms, have time complexities which are naturally
Recurrence Relations Many algorithms, particularly divide and conquer algorithms, have time complexities which are naturally modeled by recurrence relations. A recurrence relation is an equation which
More informationDiscrete Optimization
Discrete Optimization [Chen, Batson, Dang: Applied integer Programming] Chapter 3 and 4.14.3 by Johan Högdahl and Victoria Svedberg Seminar 2, 20150331 Todays presentation Chapter 3 Transforms using
More information(57) (58) (59) (60) (61)
Module 4 : Solving Linear Algebraic Equations Section 5 : Iterative Solution Techniques 5 Iterative Solution Techniques By this approach, we start with some initial guess solution, say for solution and
More informationNumerical methods for American options
Lecture 9 Numerical methods for American options Lecture Notes by Andrzej Palczewski Computational Finance p. 1 American options The holder of an American option has the right to exercise it at any moment
More informationMA651 Topology. Lecture 6. Separation Axioms.
MA651 Topology. Lecture 6. Separation Axioms. This text is based on the following books: Fundamental concepts of topology by Peter O Neil Elements of Mathematics: General Topology by Nicolas Bourbaki Counterexamples
More informationWe shall turn our attention to solving linear systems of equations. Ax = b
59 Linear Algebra We shall turn our attention to solving linear systems of equations Ax = b where A R m n, x R n, and b R m. We already saw examples of methods that required the solution of a linear system
More informationMinimize subject to. x S R
Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such
More informationRecovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branchandbound approach
MASTER S THESIS Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branchandbound approach PAULINE ALDENVIK MIRJAM SCHIERSCHER Department of Mathematical
More informationCHAPTER 9. Integer Programming
CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral
More informationWeek 5 Integral Polyhedra
Week 5 Integral Polyhedra We have seen some examples 1 of linear programming formulation that are integral, meaning that every basic feasible solution is an integral vector. This week we develop a theory
More informationG.A. Pavliotis. Department of Mathematics. Imperial College London
EE1 MATHEMATICS NUMERICAL METHODS G.A. Pavliotis Department of Mathematics Imperial College London 1. Numerical solution of nonlinear equations (iterative processes). 2. Numerical evaluation of integrals.
More informationComputing a Nearest Correlation Matrix with Factor Structure
Computing a Nearest Correlation Matrix with Factor Structure Nick Higham School of Mathematics The University of Manchester higham@ma.man.ac.uk http://www.ma.man.ac.uk/~higham/ Joint work with Rüdiger
More informationA FIRST COURSE IN OPTIMIZATION THEORY
A FIRST COURSE IN OPTIMIZATION THEORY RANGARAJAN K. SUNDARAM New York University CAMBRIDGE UNIVERSITY PRESS Contents Preface Acknowledgements page xiii xvii 1 Mathematical Preliminaries 1 1.1 Notation
More informationLinear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.
1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that
More informationMixedÀ¾ Ð½Optimization Problem via Lagrange Multiplier Theory
MixedÀ¾ Ð½Optimization Problem via Lagrange Multiplier Theory Jun WuÝ, Sheng ChenÞand Jian ChuÝ ÝNational Laboratory of Industrial Control Technology Institute of Advanced Process Control Zhejiang University,
More informationIntroduction to Support Vector Machines. Colin Campbell, Bristol University
Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multiclass classification.
More informationChapter 7 Nonlinear Systems
Chapter 7 Nonlinear Systems Nonlinear systems in R n : X = B x. x n X = F (t; X) F (t; x ; :::; x n ) B C A ; F (t; X) =. F n (t; x ; :::; x n ) When F (t; X) = F (X) is independent of t; it is an example
More informationIncreasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.
1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.
More informationOverview of Math Standards
Algebra 2 Welcome to math curriculum design maps for Manhattan Ogden USD 383, striving to produce learners who are: Effective Communicators who clearly express ideas and effectively communicate with diverse
More informationLogistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression
Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max
More informationMetric Spaces. Chapter 7. 7.1. Metrics
Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some
More informationBX in ( u, v) basis in two ways. On the one hand, AN = u+
1. Let f(x) = 1 x +1. Find f (6) () (the value of the sixth derivative of the function f(x) at zero). Answer: 7. We expand the given function into a Taylor series at the point x = : f(x) = 1 x + x 4 x
More information