Size: px
Start display at page:

Download ""

Transcription

1 A Globally Convergent Primal-Dual Interior Point Method for Constrained Optimization Hiroshi Yamashita 3 Abstract This paper proposes a primal-dual interior point method for solving general nonlinearly constrained optimization problems. The method is based on solving the barrier Karush-Kuhn-Tucker conditions for optimality by the Newton method. To globalize the iteration we introduce the barrier-penalty fucntion and the optimality condition for minimizing this function. Our basic iteration is the Newton iteration for solving the optimality conditions with respect to the barrier-penalty function which coincides with the Newton iteration for the barrier Karush-Kuhn- Tucker conditions if the penalty parameter is suciently large. It is proved that the method is globally convergent from an arbitrary initial point that strictly satises the bounds on the variables. Implementations of the given algorithm are done for small dense nonlinear programs. The method solves all the problems in Hock and Schittkowski's textbook eciently. Thus it is shown that the method given in this paper possesses a good theoretical convergence property and is ecient in practice. Key words: interior point method, primal-dual method, constrained optimization, nonlinear programming 3 Mathematical Systems, Inc., 2-4-3, Shinjuku, Shinjuku-ku, Tokyo, Japan 1

2 1 Introduction In this paper we propose a primal-dual interior point method that solves general nonlinearly constrained optimization problems. To obtain a fast algorithm for nonlinear optimization problems it is fairly clear from various experiences (for example the well known success of the SQP method) that we should eventually solve the Karush-Kuhn-Tucker conditions for optimality by a Newton-like method. However, solving the optimality conditions simply as a system of equations does not give an algorithm for solving optimization problems in general except for convex problems. Therefore it is not appropriate to treat the primal and dual variables equally to obtain globally convergent methods for general nonlinear optimization problems. Observing that there is a nice relation between the parameterized optimality conditions (barrier KKT condition) and the classical logarithmic barrier function of the primal variables, we can develop a globally convergent method for general nonlinear optimization problems. To prevent the possible divergence of the dual variables, we introduce the barrier penalty function and solve the optimality conditions by a Newton-like method. For nonlinear programs it has been believed long time that interior point methods are not practical because of inevitable numerical diculties which occur at the nal stage of iterations. See Fiacco and McCormick [4], Fletcher [5] and other standard textbooks on nonlinear optimization. Thus the state of the art method for nonlinear programs today is the SQP method (see Powell [7], Fletcher [5]) which can also be interpreted as a Newton-like method for Karush-Kuhn-Tucker conditions near a solution. It is known that the SQP method is ecient and stable for wide range of problems. The SQP method requires the quadratic programming subproblems which handle the combinatorial aspect of the problem caused by inequality constraints. A solution of the quadratic programming problem itself requires rather expensive cost especially for large scale problems. Therefore it is desired to have an ecient and stable interior point method for nonlinear problems in the light of success of that in the eld of large scale linear programs. This paper shows that it is in fact possible to construct such a method that is globally convergent theoretically, and ecient and stable in practice. The method is tested on 115 test problems in Hock and Schittkowski's collection [6]. For these problems our method solves all the problems with 2 to 3 function evaluations per problem and about 2 iterations per problem. Details are described in Section 5. A preliminary report of this paper has appeared in [11]. In this paper, problems to be solved are restricted to small to medium ones because we do not exploit the sparsity of the matrices here. However, recent report by Yamashita, Yabe and Tanabe [13] studies a trust region type method to utilize the sparsity of the Hessian of the Lagrangian. They report eciency of the method that uses the barrier penalty function as in this paper. See also [1] for a trust region type interior point method. It is to be noted that the recent studies by Yamashita and Yabe [12] and Yabe and Yamashita [9] show the superlinear and/or quadratic convergence of a class of primaldual interior point methods that use the Newton or quasi-newton iteration for solving the barrier KKT conditions. Other reports on the local behavior of primal-dual interior point methods include [2] and [3]. In Section 2, we describe basic concepts in the primal-dual interior point method. The 2

3 barrier penalty function which plays a key role in the method of this paper is introduced in Section 3, and analyzed there. A line search method that minimizes the barrier penalty function is described in Section 4, and is proved to be globally convergent. Section 5 reports the results of numerical experiment. Notation. The subscript k denotes an iteration count. Subscripts i and j denote components of vectors and matrices. The superscript t denotes transposes of vectors and matrices. The vector e denotes the vector of all ones and the matrix I the identity matrix. For simplicity of description, we assume k1k denotes the l 2 norm for vectors. The symbol R n denotes the n dimensional real vector space. The set R n + is dened by R n = fx + 2 jx> g. Rn 2 Primal-dual interior point method In this paper, we consider the following constrained optimization problem: (1) minimize f(x); x 2 R n ; subject to g(x) = ; x ; where we assume that the functions f : R n! R 1 and g : R n! R m are twice continuously dierentiable. Let the Lagrangian function of the above problem be dened by (2) L(w) =f(x) y t g(x) z t x; where w =(x; y; z) t, and y 2 R m and z 2 R n are the Lagrange multiplier vectors which correspond to the equality and inequality constraints respectively. Then Karush-Kuhn- Tucker (KKT) conditions for optimality of the above problem are given by 1 1 r (w) r xl(w) g(x) CA = (3) CA XZe and (4) where x ; z ; r x L(w) = rf(x) A(x) t y z; A(x) = rg 1(x) t. CA ; rg m (x) t 1 X = diag (x 1 ; 111;x n ) ; Z = diag (z 1 ; 111;z n ) : Now we approximate problem (1) by introducing the barrier function F B (; ) :R n +! R 1, (5) minimize subject to g(x) = ; F B (x; ) =f(x) np 3 log(x i );x 2 R n +

4 where the barrier parameter > is a given constant. It is well known that, if is suciently small, problem (5) is a good approximation to original problem (1) (see Fiacco and McCormick [4]). The optimality conditions for (5) are given by (6) rf (x) A(x) t y X 1 e = ; g(x) = and x>; where y 2 R m is the Lagrange multiplier for the equality constraints. If we introduce an auxiliary variable z 2 R n which is to be equal to X 1 e, then the above conditions become the conditions (7) and (8) r(w; ) r xl(w) g(x) XZe e x>; 1 CA = z>: The introduction of the variable z is essential to the numerical success of the barrier based algorithm in this paper. In this paper we call conditions (7) the barrier KKT conditions, and a point w() = (x();y();z()) that satises these conditions is called the barrier KKT point. We will use an interior point method for searching a point that approximately satises the above conditions, and nally obtain a point that satises the Karush-Kuhn-Tucker conditions by letting #. This means that we force x and z be strictly positive during iterations. Therefore we delete inequality conditions (8) hereafter, and always assume that x and z are strictly positive in what follows. Here we note that (9) where r(w; ) =r (w) ^e; ^e = e An algorithm of this paper approximately solves the sequence of conditions (7) with a decreasing sequence of the barrier parameter that tends to, and thus obtains an approximate solution to KKT conditions. For deniteness, we describe a prototype of such algorithm as follows. Algorithm IP Step. (Initialize) Set ">, M c > and k =. Let a positive sequence f k g ; k # be given. 1 CA : Step 1. (Termination) If kr (w)k ", then stop. 1 CA ; 4

5 Step 2. (Approximate barrier KKT point) Find a point w k+1 that satises (1) kr(w k+1 ; k )km c k : Step 3. (Update) Set k := k + 1 and go to Step 1. 2 The following theorem shows the global convergence property of Algorithm IP. Theorem 1 Let fw k g be an innite sequence generated by Algorithm IP. Suppose that the sequences fx k g and fy k g are bounded. Then fz k g is bounded, and any accumulation point of fw k g satises KKT conditions (3) and (4). Proof. Assume that there exists an i such that (z k ) i!1. Equation (1) yields (rf(x k) A(x k ) t y k ) i (z k ) i 1 M c k1 (z k ) i ; which isacontradiction because of the boundedness of fx k g and fy k g.thus the sequence fz k g is bounded. Let ^w be any accumulation point offw k g. Since the sequences fw k g and f k g satisfy (1) for each k and k approaches zero, r (^w) = follows from the denition of r(w; ). Therefore the proof is complete. 2 We note that the barrier parameter sequence f k g in Algorithm IP need not be determined beforehand. The value of each k may be set adaptively as the iteration proceeds. An example of updating method of k is described in Section 5. We call condition (1) the approximate barrier KKT condition, and call a point that satises this condition the approximate barrier KKT point. To nd an approximate barrier KKT point for a given >, we use the Newton-like method in this paper. Let 1w =(1x; 1y; 1z) t be dened by a solution of (11) where (12) J(w) = J(w)1w = r(w; ); G A(x)t I A(x) Z X Then the basic iteration of the Newton-like method may be described as 1 CA : (13) w k+1 = w k +3 k 1w k ; where 3 k = diag( xk I n ; yk I m ; zk I n ) is composed of step sizes in x, y and z variables. If G = r 2 L(w), then 1w becomes Newton's direction for solving (7). To solve (11), we x split the equations into two groups. Thus we solve! G + X 1 Z A(x) t 1x!! = r xl(w)+x 1 e z (14) ; A(x) 1y g(x) 5

6 for (1x; 1y) t, then we obtain 1z by the third equation in (11). If G + X 1 Z is positive denite and A(x) is of full rank, then the coecient matrix in (14) is nonsingular. It will be useful to note that (14) can be written as (15) where (16) G + X 1 Z A(x) t A(x)! 1x ey ~y = y +1y:! = rf(x)+x1 e g(x) In this paper, we will deal with the case in which the matrix G can be assumed nonnegative denite. Therefore, to solve the general nonlinear problems, we will use a positive denite quasi-newton approximation to the Hessian matrix of the Lagrangian function to obtain the desired property of the matrix G.! ; 3 Barrier penalty function To globalize the convergence property ofaninterior point algorithm based on the above iteration, we introduce two auxiliary problems. Firstly mp we dene the following problem: minimize F P (x; ) =f(x)+ jg i (x)j; x 2 R n ; (17) subject to x ; where the penalty parameter is a given positive constant. The necessary conditions for optimality of this problem are (see 14.2 of Fletcher [5]) r x L(w) = ; ( mx ) (18) y jg i (x)j ; XZe = ; x ;z ; where the means the subdierential of the function in the braces with respect to g. In our case the second condition in (18) is equivalent to (19) y i ; g i (x) =; y i = ; g i (x) > ; y i =; g i (x) < ; for each i =1; 111;m. This condition can be expressed as jg i (x)j = y i g i (x); y i ; i =1; 111;m; or jg i (x)j + y ig i (x) =; y i ; i =1; 111;m: 6

7 Therefore conditions (18) can be written as r (w) = r xl(w) (2) r E (w) XZe and (21) where 1 CA = x ; z ; y i ; i =1; 111;m; 1 CA r E (w) i = jg i (x)j + y ig i (x) ;; 111;m: Note that we are using the same symbol r (w) to denote the residual vector of the optimality conditions as in Section 2 for simplicity. Ifkyk 1 <, conditions (18) are equivalent to conditions (3) and (4). In this sense, problem (17) is equivalent to problem (1). Next we introduce the barrier penalty function F (; ; ) :R n +! R1 by (22) F (x; ; ) =f(x) nx log x i + mx jg i (x)j ; where and are given positive constants. This function plays an essential role in the method given in this paper. Let us approximate problem (17) by the following problem (23) minimize F (x; ; ); x 2 R n + : The necessary conditions for optimality of a solution to the above problem are r x L(w) = ; ( mx ) (24) y jg i (x)j XZe = e; x > ; z>; where we introduce an auxiliary variable z 2 R n as in (7). As above, conditions (24) can be written as r(w; ) xl(w) (25) r E (w) CA = CA XZe e and (26) 1 x>; z>; y i ; i =1; 111;m; where we use the same symbol r(w; ) as in Section 2 for simplicity. If>kyk 1 then conditions (25) and (26) coincide with (7) and (8). We call a point w() =(x();y();z()) 2 R n 2 R m 2 R n that satises (24) for a given > the barrier KKT point for this as before. We can use Algorithm IP to solve (17). The following theorem shows the global convergence property of Algorithm IP for solving (17). 1 7

8 Theorem 2 Let fw k g be an innite sequence generated byalgorithm IP for solving (17). Suppose that the sequences fx k g is bounded. Then fz k g is bounded, and any accumulation point of fw k g satises the optimality conditions (2) and (21) for problem (17). 2 Now we formulate a Newton-like iteration for solving the above conditions (25). Thus we calculate the rst order change of (24) with respect to a change in w. This gives (G + X 1 Z)1x A(x) t 1y = r x L(w)+X 1 e z; ( mx ~y y +1y g i (x)+rg i (x) 1x ) (27) t ; 1z = X 1 Z1x + X 1 e z: Following lemma gives a basic property of the iteration vector 1w =(1x; 1y; 1z) t. Lemma 1 Suppose that 1w satises (27) at an interior point w. (i) If >keyk 1, then 1w is identical to the one given by (11). (ii) If 1w =, then the point w isabarrier KKT point that satises (24). (iii) If 1x =, then the point (x; y +1y; z +1z) is a barrier KKT point that satises (24). 2 If we consider the subproblem: (28) minimize 1 2 1xt (G+X 1 Z)1x+(rf(x)X 1 e) t 1x+ mx g i (x)+rg i (x) t 1x ; 1x 2 R n ; the solution vector 1x and the corresponding multiplier vector ~y that satisfy the necessary conditions for optimality also satisfy conditions (27). If G + X 1 Z is positive denite we can solve the problem (28) by a straightforward active set method which starts with an active set that contains all the constraints g i (x)+rg i (x) t 1x =;; 111m. An example of the procedure is described in Fletcher [5]. It is important to note that, if the vectors 1x and ~y obtained by solving (15) satisfy >k~yk 1, then the desired iteration vector that satises (27) are also obtained. The procedure described above is devised instead of the simple Newton iteration given in Section 2 in order to show away of preventing possible divergence of the dual variable y. However for the practical purpose it seems sucient to solve only equation (11) once per iteration as shown in sections below in which practical experiences obtained by the author on general nonlinear programming problems are described. Therefore this procedure is of only theoretical importance at present. As shown above, optimality conditions (2) and (21) are identical to optimality conditions (3) and (4) if we set = 1. Also, Newton-like iteration (27) coincides with (11) when = 1. Therefore we will regard that the method explained below includes a method for solving (1) when = 1 for simplicity of exposition. We note that the penalty parameter which will appear below should be nite even if = 1. Another interesting point to be noted in the above iteration is that it can give a solution to an infeasible problem. Even if the constraints in problem (1) are incompatible, the 8

9 method described above will give a solution to problem (17) as shown below. A solution to problem (17) with infeasible constraints may give useful information about problem (1). Now we proceed to an analysis of the properties of the barrier penalty function. The directional derivative F (x; ; ; s) of the function F (x; ; ) along an arbitrary given direction s is dened by F F (x + s; ; ) F (x; ; ) (x; ; ; s) = lim # = rf(x) X X t s e t X 1 s + rg i (x) t s + + rg i (x) s X t rg i (x) t s; where the summations in the above equation are to be understood as X X a i = a i X X ; a i = a i X X ; a i = a i : + g i > We introduce a rst order approximation F l of F (x + s; ; ) by F l (x; ; ; s) f(x)+rf (x) t s log(x i )+ s i (29) + mx g i = nx g i (x)+rg i (x) t s; g i < x i and an estimate of the rst order change 1F l of F by (3) 1F l (x; ; ; s) F l (x; ; ; s) F (x; ; ); = rf(x) t s e t X 1 s + mx jg i (x)+rg i (x)sj mx jg i (x)j for an arbitrary given direction s. We have following properties for the quantities dened above which give an extension to the similar properties in the case of dierentiable functions (see for example Yamashita [1]). Lemma 2 Let >, > and s 2 R n be given. Then the following assertions hold. (i) The function F l (x; ; ; s) is convex with respect to the variable. (ii) There holds the relation (31) F (x; ; )+F (x; ; ; s) F l (x; ; ; s): (iii) Further, there exists a 2 (; 1) such that (32) F (x + s; ; ) F (x; ; )+F (x + s; ; ; s); whenever x + s>. 9

10 Proof. The rst statement of the lemma is obvious. If > is suciently small, we have F l (x; ; ; s) = F (x; ; )+F (x; ; ; s) = F (x; ; )+F (x; ; ; s): Since F l (x; ; ; s) isconvex with respect to and coincides with a linear function of > when is suciently small, we obtain (31). Now we show (32). If we consider F (x + s; ; ; s) as a function of the variable 2 [; 1], the number of discontinuous points of F is nite. Therefore there exists a 2 (; 1) such that F (x + s; ; ; s) Z1 F (x + s; ; ; s)d = F (x + s; ; ) F (x; ; ): This completes the proof. 2 Lemma 3 Let " 2 (; 1) be a given constant and s 2 R n be given. If 1F l (x; ; ; s) <, then (33) F (x + s; ; ) F (x; ; ) " 1F l (x; ; ; s); for suciently small >. Proof. From (32), there exists a 2 (; 1) such that (34) F (x + s; ; ) F (x; ; ) F (x + s; ; ; s) = F (x + s; ; ; s): From (31) we have (35) F (x + s; ; ; s) 1F l (x + s; ; ; s): Because 1F l (; ; ; s) is continuous, and 1F l (x; ; ; s) < by the assumption, we obtain (36) 1F l (x + s; ; ; s) " 1F l (x; ; ; s); for suciently small ksk. From (34) - (36) we obtain (33) for suciently small >. 2 Lemma 4 Suppose that 1w satises (27). If <, then (37) 1F l (x; ; ;1x) 1x t (G + X 1 Z)1x mx ( j~y i j)jg i (x)j: Further if, G is positive semidenite and keyk 1 1F l (x; ; ;1x) =yields 1x =., then 1F l (x; ; ;1x), and 1

11 Proof. From (27) and (3) we have 1F l (x; ; ;1x) =1x t (G+X 1 Z)1x+1x t A(x) t ~y+ The i-th components in the summations in the last three terms give mx g i (x)+rg i (x) t 1x mx jg i (x)j: (38) ~y i rg i (x) t 1x + g i (x)+rg i (x) 1x t jg i (x)j ~y i rg i (x) t 1x + g i (x)+rg i (x) 1x t jg i (x)j = ~y i g i (x) jg i (x)j ; where the inequality in the second line follows from, and the equality in the third line follows from the property ~y i ; g i (x)+rg i (x) t 1x =; ~y i = ; g i (x)+rg i (x) t 1x>; ~y i =; g i (x)+rg i (x) t 1x<: The relation (38) gives (37). 2 4 Line search algorithm To obtain a globally convergent algorithm to a barrier KKT point for a xed >, it is necessary to modify the basic Newton iteration with the unit step length somehow. Our iterations consist of (39) x k+1 = x k + xk 1x k ; y k+1 = y k + yk 1y k ; z k+1 = z k + zk 1z k ; where xk, yk and zk are step sizes determined by the line search procedures described below. The main iteration is to decrease the value of the barrier penalty function for xed. Thus the step size of the primal variable x is determined by the sucient decrease rule of the merit function. The step size of the dual variable z is determined so as to stabilize the iteration. The explicit rules follow in order. We adopt Armijo's rule as the line search rule for the variable x. At the point x k,we calculate the maximum allowed step to the boundary of the feasible region by (4) kmax = min i ( (x k) i (1x k ) i (1x k) i < i.e., the step size kmax gives an innitely large value of the barrier penalty function F if it exists, because of the barrier terms, and a step size 2 [; kmax ) gives a strictly feasible primal variable. A step to the next iterate is given by ) ; (41) xk = k l k ; k = min f kmax ; 1g ; 11

12 where 2 (; 1) and 2 (; 1) are xed constants and l k is the smallest nonnegative integer such that (42) F (x k + k l k 1x k; ; ) F (x k ; ; ) " k l k 1F l(x k ; ; ;1x k ); where " 2 (; 1). Typical values of these parameters are = :5, = :9995 and " =1 6. Therefore we will try the sequence x k +:9995 kmax 1x k ; x k +:5 2 :9995 kmax 1x k ; x k +:25 2 :9995 kmax 1x k ; 111 for example, and will nd a step size that satises (42). If G is positive semidenite, then 1F l (x k ; ; ;1x k ) by Lemma 4, and therefore the existence of such steps is assured by Lemma 3. For the variable z, we adopt the box constraints rule, i.e., we force x and z to satisfy the condition (43) c Lki ((x k ) i + xk (1x k ) i )((z k ) i + zk (1z k ) i ) c Uki ; i =1; 111;n at the end of each iteration, where the bounds c Lk and c Uk satisfy (44) <c Lki <<c Uki ; i =1; 111;n: To this end, we let (45) c Lki = minn M L ; ((x k ) i + xk (1x k ) i )(z k ) io ; c Uki = max fm U ; ((x k ) i + xk (1x k ) i )(z k ) i g ; where M L > 1 and M U > 1 are given constants. The construction of the above bounds shows that current z satises (46) (47) c Lki ((x k ) i + xk (1x k ) i ) (z k) i The step size z is determined by zk = min ( min i ( max i ( i c Uki ((x k ) i + xk (1x k ) i ) ; c Lki ((x k ) i + xk (1x k ) i ) c Uki ((z k ) i + i (1z k ) i ) ((x k ) i + xk (1x k ) i ) i =1; 111;n: )) ) ; 1 : The rule (47) means that the step size z is the maximal allowed step that satises the box constraints with the restriction of being not greater than the unit step length. Lemma 5 Suppose that an innite sequence fw k g is generated for xed >. Then if lim inf k!1(x k ) i > and lim sup k!1(x k ) i < 1, then lim inf k!1(c Lk ) i > and lim sup k!1(c Uk ) i < 1 for i =1; 111;n. 12

13 Proof. Suppose that (c Lk ) i! for an i and some subsequence K f; 1; 2; 111g. Then by the denition of (c Lk ) i in (45), (z k ) i! ;k 2 K. However, in order for a subsequence of f(z k ) i g to tend to, there must be an iteration k at which the lower bound (c Lk ) i =(x k+1 ) i of (z k ) i is arbitrary small and the value of (z k ) i at the iteration is strictly larger than that bound, i.e. at the iteration the value of (z k ) i decreases to a strictly smaller value. This means that at iteration k, (c Lk ) i = =M L from the denition (45), and therefore the value of (x k+1 ) i must be arbitrary large because =M L < (x k+1 ) i (z k ) i and (z k ) i! ;k 2 K. This is impossible because of the assumption of the lemma. The proof of the boundedness of (c Uk ) i is similar. 2 (48) In actual calculation we modify the direction 1z k (1z k ) i = 8>< >: ; if (z k) i = c Lki=(x k+1) i and (1z k ) i < ; ; if (z k ) i = c Uki =(x k+1 ) i and (1z k ) i > ; (1z k ) i ; otherwise: This modication means that we project the direction along the boundary of the box constraints if the point z k is on that boundary and the direction 1z k points outward of the box. This procedure is adopted because it gives better numerical results. The global convergence results shown in the following are equally valid for both unmodied and modied directions. For the variable y, there exist three obvious choices for the step length: by (49) yk =1 or xk or zk : The global convergence property given below holds for these choices. We choose yk = zk from numerical experiments. The following algorithm describes the iteration for xed > and >. We note that this algorithm corresponds to Step 2 of Algorithm IP in Section 2. Algorithm LS Step. (Initialize) Let w 2 R n + 2 Rm 2 R n +, and >, >. Set " >, 2 (; 1), 2 (; 1), " 2 (; 1), M L > 1 and M U > 1. Let k =. Step 1. (Termination) If kr(w k ;)k " ; then stop. Step 2. (Compute direction) Calculate the direction 1w k by (27). Step 3. (Stepsize) Set kmax = min i ( (x k) i ) (1x k ) i (1x k) i < ; k = min f kmax ; 1g : Find the smallest nonnegative integer l k that satises F (x k + k l k 1x k ;;) F (x k ;;) " k l k 1F l (x k ; ; ;1x k ): Calculate xk = k l k ; 13

14 c Lki = min ; (x k + xk 1x k ) i (z k ) i ; M L c Uki = max fm U ; (x k + xk 1x k ) i (z k ) i g ; ( ( ( i c Lki zk = min min max i i ((x k ) i + xk (1x k ) i c Uki )) ) ((z k ) i + i (1z k ) i ) ; 1 ; ((x k ) i + xk (1x k ) i ) yk = zk ; 3 k = diagf xk I n ; yk I m ; zk I n g: Step 4. (Update variables) Set w k+1 = w k +3 k 1w k : Step 5. Set k := k + 1 and go to Step 1. 2 To prove global convergence of Algorithm LS, we need the following assumptions. Assumption G (1) The functions f and g i ;i =1; :::; m, are twice continuously dierentiable. (2) The level set of the barrier penalty function at an initial point x 2 R n +, which is dened by nx 2 R n jf (x; ; ) + F (x ; ; )o, is compact for given >. (3) The matrix A(x) is of full rank on the level set dened in (2). (4) The matrix G k is positive semidenite and uniformly bounded. (5) The penalty parameter satises ky k +1y k k 1 for each k =; 1; :::. 2 We note that if a quasi-newton approximation is used for computing the matrix G k, then we need the continuity of only the rst order derivatives of functions in Assumption G-(1). We also note that if 1F l (x k ; ; ;1x k ) =, at an iteration k, then the step sizes xk = yk = zk = 1 are adopted and (x k+1 ;y k+1 ;z k+1 ) gives a barrier KKT point from Lemma 1 and Lemma 4. The following theorem gives a convergence of an innite sequence generated by Algorithm LS. Theorem 3 Let an innite sequence fw k g be generated byalgorithm LS. Then there exists at least one accumulation point of fw k g, and any accumulation point of the sequence fw k g is a barrier KKT point. 14

15 Proof. First we note that each component of the sequence fx k g is bounded away from zero and bounded above by the assumption and the existence of the log barrier term. Therefore the sequence fx k g has at least one accumulation point. The sequence fz k g also has these properties by Lemma 5. Thus there exists a positive number M such that (5) kpk 2 M pt (G k + X 1 k Z k)p M kpk 2 ; 8p 2 R n ; by the assumption. From (37) and (5), we have (51) and from (42), 1F l (x k ; ; ;1x k ) k1x kk 2 M < ; (52) F (x k+1 ; ; ) F (x k ; ; ) " k l k 1F l(x k ; ; ;1x k ) " k l k k1x kk 2 M < : Because the sequence ff (x k ; ; )g is decreasing and bounded below, the left hand side of (52) converges to. Since lim inf k!1(x k ) i > ;i =1; 111;n,wehave lim inf k!1 k >. Suppose that there exists a subsequence K f; 1; 111g and a such that (53) lim inf k!1 k1x kk>; k 2 K: Then we have l k!1;k 2 K from (52) because the left most expression tends to zero, and therefore we can assume l k > for suciently large k 2 K without loss of generality. If l k > then the point x k + xk 1x k = does not satisfy the condition (42). Thus, we have (54) F (x k + xk 1x k =; ; ) F (x k ; ; ) >" xk 1F l (x k ; ; ;1x k )=: By (32) and (31), there exists a k 2 (; 1) such that (55) F (x k + xk 1x k =; ; ) F (x k ; ; ) xk F (x k + k xk 1x k =; ; ;1x k )= xk 1F l (x k + k xk 1x k =; ; ;1x k )=; k 2 K: Then, from (54) and (55), we have " 1F l (x k ; ; ;1x k ) < 1F l (x k + k xk 1x k =; ; ;1x k ): This inequality yields (56) 1F l (x k + k xk 1x k =; ; ;1x k ) 1F l (x k ; ; ;1x k ) > (" 1)1F l (x k ; ; ;1x k ) > : Because 1x k is a solution of problem (28) and there holds (5), k1x k k is uniformly bounded above. Then by the property l k! 1,we have k k xk 1x k =k!;k 2 K. 15

16 Thus the left hand side of (56) and therefore 1F l (x k ; ; ;1x k ) converges to zero when k!1;k 2 K. This contradicts the assumption (53) because we have 1x k! ;k 2 K from (51). Therefore we proved (57) lim k1x k k =: k!1;k2k Let an arbitrary accumulation point of the sequence fx k g be ^x 2 R n + and let x k! ^x; k 2 K for K f; 1; 111g.Thus (58) x k! ^x; 1x k! ; x k+1! ^x; k 2 K: Because nx 1 Z k ko is bounded, we have lim k!1;k2kz k +1z k X e 1 = k from (27). If we dene ^z = ^X 1 e where ^X = diag(^x 1 ; 111; ^x n ), then we have Hence from (45) we have z k +1z k! ^z; k 2 K: (c Lk ) i M L (x k+1 ) i (z k +1z k ) i M U (C Uk ) i ; i =1; 111;n for k 2 K suciently large, which shows that the point z k +1z k is always accepted as z k+1 for suciently large k 2 K. Since zk = 1 is accepted for k 2 K suciently large, so is yk =1. Therefore we obtain lim r x L(^x; y k +1y k ; ^z) = ; k!1;k2k lim y k +1y k k!1;k2k ( nx ) jg i (^x)j : Because the matrix A(^x) is of full rank, the sequence fy k +1y k g ;k 2 K converges to a point ^y 2 R m which satises r x L(^x; ^y; ^z) = ; mx! ^y jg i (^x)j ; ^X ^z = e; ^x >; ^z >: This completes the proof because we proved that there exists at least one accumulation point offx k g, and for an arbitrary accumulation point ^x of fx k g, there exist unique ^y and ^z that satisfy the above. 2 16

17 5 Numerical Result In this section, we report numerical results of an implementation of the algorithm given in this paper for nonlinear programming problems. We set = 1 in this experiment. The software is called NUOPT and the code is written bytakahito Tanabe. In order to havean appropriate positive semidenite matrix G by a reasonable cost for nonlinear problems, we resort to a quasi-newton approximation to the Hessian matrix of the Lagrangian function. We use updating formula suggested by Powell[7] for the SQP method: G k+1 = G k G ks k s t k G k s t k G ks k + u ku t k s t u ; k k where u k is calculated by s k = x k+1 x k ; v k = r x L(x k+1 ;y k+1 ;z k+1 ) r x L(x k ;y k+1 ;z k+1 ); u k = k v k +(1 k )G k s k ; k = 8< : :8s t k G k s k s t k G k s ks t k v k 1; s t k v k :2s t k G ks k ; ; s t k v k :2s t k G ks k ; to satisfy s t u k k > for the hereditary positive deniteness of the update. Method for updating the barrier parameter k is as follows. Suppose we have an approximate barrier KKT point w k+1 that satises kr(w k+1 ; k )km c k ; in Step 2 of Algorithm IP. Then k+1 is dened by k+1 = max( kr(wk+1 ; k ) )k ; k ; M M where < M c < M and M > 1 should be satised. In our experiment weset M c =3;M =4;M = 5. As in the SQP method, we expect fast local convergence of the method if is suciently small near a solution because it is based on the Newton iteration for the optimality conditions and a quasi-newton approximation to the second derivative matrix. As noted in the above, this expectation is proved by Yamashita and Yabe [12]. Linear equation (14 ) is solved by using the Bunch-Parlett factorization. The test problems for nonlinear problems are adopted from the book by Hock and Schittkowski [6]. The results are summarized in Table 1 at the end of this paper. Following list explains the notations used in Table 1: n= number of variables. m=number of constraints. obj=nal objective function value. res=norm of nal KKT condition residual. itr=iteration count. 17

constraint. Let us penalize ourselves for making the constraint too big. We end up with a

constraint. Let us penalize ourselves for making the constraint too big. We end up with a Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the

More information

Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

More information

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing

More information

Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen (für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

More information

Date: April 12, 2001. Contents

Date: April 12, 2001. Contents 2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

More information

2.3 Convex Constrained Optimization Problems

2.3 Convex Constrained Optimization Problems 42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

More information

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725 Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

More information

(Quasi-)Newton methods

(Quasi-)Newton methods (Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting

More information

(We assume that x 2 IR n with n > m f g are twice continuously ierentiable functions with Lipschitz secon erivatives. The Lagrangian function `(x y) i

(We assume that x 2 IR n with n > m f g are twice continuously ierentiable functions with Lipschitz secon erivatives. The Lagrangian function `(x y) i An Analysis of Newton's Metho for Equivalent Karush{Kuhn{Tucker Systems Lus N. Vicente January 25, 999 Abstract In this paper we analyze the application of Newton's metho to the solution of systems of

More information

ME128 Computer-Aided Mechanical Design Course Notes Introduction to Design Optimization

ME128 Computer-Aided Mechanical Design Course Notes Introduction to Design Optimization ME128 Computer-ided Mechanical Design Course Notes Introduction to Design Optimization 2. OPTIMIZTION Design optimization is rooted as a basic problem for design engineers. It is, of course, a rare situation

More information

Interior Point Methods and Linear Programming

Interior Point Methods and Linear Programming Interior Point Methods and Linear Programming Robert Robere University of Toronto December 13, 2012 Abstract The linear programming problem is usually solved through the use of one of two algorithms: either

More information

By W.E. Diewert. July, Linear programming problems are important for a number of reasons:

By W.E. Diewert. July, Linear programming problems are important for a number of reasons: APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization)

More information

Iterative Techniques in Matrix Algebra. Jacobi & Gauss-Seidel Iterative Techniques II

Iterative Techniques in Matrix Algebra. Jacobi & Gauss-Seidel Iterative Techniques II Iterative Techniques in Matrix Algebra Jacobi & Gauss-Seidel Iterative Techniques II Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin

More information

Introduction and message of the book

Introduction and message of the book 1 Introduction and message of the book 1.1 Why polynomial optimization? Consider the global optimization problem: P : for some feasible set f := inf x { f(x) : x K } (1.1) K := { x R n : g j (x) 0, j =

More information

Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

More information

Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

More information

Solving nonlinear equations in one variable

Solving nonlinear equations in one variable Chapter Solving nonlinear equations in one variable Contents.1 Bisection method............................. 7. Fixed point iteration........................... 8.3 Newton-Raphson method........................

More information

Andrew McLennan January 19, Winter Lecture 5. A. Two of the most fundamental notions of the dierential calculus (recall that

Andrew McLennan January 19, Winter Lecture 5. A. Two of the most fundamental notions of the dierential calculus (recall that Andrew McLennan January 19, 1999 Economics 5113 Introduction to Mathematical Economics Winter 1999 Lecture 5 Convergence, Continuity, Compactness I. Introduction A. Two of the most fundamental notions

More information

10. Proximal point method

10. Proximal point method L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

More information

Adaptive Online Gradient Descent

Adaptive Online Gradient Descent Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

More information

Optimization of Design. Lecturer:Dung-An Wang Lecture 12

Optimization of Design. Lecturer:Dung-An Wang Lecture 12 Optimization of Design Lecturer:Dung-An Wang Lecture 12 Lecture outline Reading: Ch12 of text Today s lecture 2 Constrained nonlinear programming problem Find x=(x1,..., xn), a design variable vector of

More information

Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.

Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U. Summer course on Convex Optimization Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota Interior-Point Methods: the rebirth of an old idea Suppose that f is

More information

Big Data - Lecture 1 Optimization reminders

Big Data - Lecture 1 Optimization reminders Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics

More information

Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

More information

t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).

t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d). 1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction

More information

No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics

No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results

More information

INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS

INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: FILTER METHODS AND MERIT FUNCTIONS HANDE Y. BENSON, DAVID F. SHANNO, AND ROBERT J. VANDERBEI Operations Research and Financial Engineering Princeton

More information

PRIMAL-DUAL METHODS FOR LINEAR PROGRAMMING

PRIMAL-DUAL METHODS FOR LINEAR PROGRAMMING PRIMAL-DUAL METHODS FOR LINEAR PROGRAMMING Philip E. GILL, Walter MURRAY, Dulce B. PONCELEÓN and Michael A. SAUNDERS Technical Report SOL 91-3 Revised March 1994 Abstract Many interior-point methods for

More information

NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice

NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice IC-32: Winter Semester 2006/2007 Benoît C. CHACHUAT Laboratoire d Automatique, École Polytechnique Fédérale de Lausanne CONTENTS 1 Nonlinear

More information

Duality of linear conic problems

Duality of linear conic problems Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

More information

Convex Programming Tools for Disjunctive Programs

Convex Programming Tools for Disjunctive Programs Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible

More information

Data analysis in supersaturated designs

Data analysis in supersaturated designs Statistics & Probability Letters 59 (2002) 35 44 Data analysis in supersaturated designs Runze Li a;b;, Dennis K.J. Lin a;b a Department of Statistics, The Pennsylvania State University, University Park,

More information

24. The Branch and Bound Method

24. The Branch and Bound Method 24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

More information

1 if 1 x 0 1 if 0 x 1

1 if 1 x 0 1 if 0 x 1 Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or

More information

max cx s.t. Ax c where the matrix A, cost vector c and right hand side b are given and x is a vector of variables. For this example we have x

max cx s.t. Ax c where the matrix A, cost vector c and right hand side b are given and x is a vector of variables. For this example we have x Linear Programming Linear programming refers to problems stated as maximization or minimization of a linear function subject to constraints that are linear equalities and inequalities. Although the study

More information

CONSTRAINED NONLINEAR PROGRAMMING

CONSTRAINED NONLINEAR PROGRAMMING 149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach

More information

Lecture 12 Basic Lyapunov theory

Lecture 12 Basic Lyapunov theory EE363 Winter 2008-09 Lecture 12 Basic Lyapunov theory stability positive definite functions global Lyapunov stability theorems Lasalle s theorem converse Lyapunov theorems finding Lyapunov functions 12

More information

BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBERT SPACE REVIEW BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

More information

Solving linear equations on parallel distributed memory architectures by extrapolation Christer Andersson Abstract Extrapolation methods can be used to accelerate the convergence of vector sequences. It

More information

Absolute Value Programming

Absolute Value Programming Computational Optimization and Aplications,, 1 11 (2006) c 2006 Springer Verlag, Boston. Manufactured in The Netherlands. Absolute Value Programming O. L. MANGASARIAN olvi@cs.wisc.edu Computer Sciences

More information

Interior-Point Algorithms for Quadratic Programming

Interior-Point Algorithms for Quadratic Programming Interior-Point Algorithms for Quadratic Programming Thomas Reslow Krüth Kongens Lyngby 2008 IMM-M.Sc-2008-19 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK-2800

More information

Support Vector Machines

Support Vector Machines Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric

More information

Factorization Theorems

Factorization Theorems Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

More information

Stochastic Inventory Control

Stochastic Inventory Control Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the

More information

Outline. Optimization scheme Linear search methods Gradient descent Conjugate gradient Newton method Quasi-Newton methods

Outline. Optimization scheme Linear search methods Gradient descent Conjugate gradient Newton method Quasi-Newton methods Outline 1 Optimization without constraints Optimization scheme Linear search methods Gradient descent Conjugate gradient Newton method Quasi-Newton methods 2 Optimization under constraints Lagrange Equality

More information

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem

More information

Solving polynomial least squares problems via semidefinite programming relaxations

Solving polynomial least squares problems via semidefinite programming relaxations Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose

More information

4.6 Linear Programming duality

4.6 Linear Programming duality 4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

More information

A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems

A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems Sunyoung Kim, Masakazu Kojima and Kim-Chuan Toh October 2013 Abstract. We propose

More information

NORTHWESTERN UNIVERSITY Department of Electrical Engineering and Computer Science LARGE SCALE UNCONSTRAINED OPTIMIZATION by Jorge Nocedal 1 June 23, 1996 ABSTRACT This paper reviews advances in Newton,

More information

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

More information

Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

More information

1 Norms and Vector Spaces

1 Norms and Vector Spaces 008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)

More information

Lecture 13 Linear quadratic Lyapunov theory

Lecture 13 Linear quadratic Lyapunov theory EE363 Winter 28-9 Lecture 13 Linear quadratic Lyapunov theory the Lyapunov equation Lyapunov stability conditions the Lyapunov operator and integral evaluating quadratic integrals analysis of ARE discrete-time

More information

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem

More information

3 Does the Simplex Algorithm Work?

3 Does the Simplex Algorithm Work? Does the Simplex Algorithm Work? In this section we carefully examine the simplex algorithm introduced in the previous chapter. Our goal is to either prove that it works, or to determine those circumstances

More information

Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

More information

Problem 1 (10 pts) Find the radius of convergence and interval of convergence of the series

Problem 1 (10 pts) Find the radius of convergence and interval of convergence of the series 1 Problem 1 (10 pts) Find the radius of convergence and interval of convergence of the series a n n=1 n(x + 2) n 5 n 1. n(x + 2)n Solution: Do the ratio test for the absolute convergence. Let a n =. Then,

More information

Ri and. i=1. S i N. and. R R i

Ri and. i=1. S i N. and. R R i The subset R of R n is a closed rectangle if there are n non-empty closed intervals {[a 1, b 1 ], [a 2, b 2 ],..., [a n, b n ]} so that R = [a 1, b 1 ] [a 2, b 2 ] [a n, b n ]. The subset R of R n is an

More information

since by using a computer we are limited to the use of elementary arithmetic operations

since by using a computer we are limited to the use of elementary arithmetic operations > 4. Interpolation and Approximation Most functions cannot be evaluated exactly: x, e x, ln x, trigonometric functions since by using a computer we are limited to the use of elementary arithmetic operations

More information

Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

More information

Theory of Linear Programming

Theory of Linear Programming Theory of Linear Programming Debasis Mishra April 6, 2011 1 Introduction Optimization of a function f over a set S involves finding the maximum (minimum) value of f (objective function) in the set S (feasible

More information

A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION 1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

More information

Notes on Factoring. MA 206 Kurt Bryan

Notes on Factoring. MA 206 Kurt Bryan The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor

More information

APPLYING NEW OPTIMIZATION CONTROL. Stephen J. Wright. Argonne, IL 60439

APPLYING NEW OPTIMIZATION CONTROL. Stephen J. Wright. Argonne, IL 60439 APPLYING NEW OPTIMIZATION ALGORITHMS TO MODEL PREDICTIVE CONTROL Stephen J Wright Mathematics and Computer Science Division Argonne National Laboratory Argonne, IL 9 Abstract The connections between optimization

More information

Calculus and linear algebra for biomedical engineering Week 4: Inverse matrices and determinants

Calculus and linear algebra for biomedical engineering Week 4: Inverse matrices and determinants Calculus and linear algebra for biomedical engineering Week 4: Inverse matrices and determinants Hartmut Führ fuehr@matha.rwth-aachen.de Lehrstuhl A für Mathematik, RWTH Aachen October 30, 2008 Overview

More information

Output feedback stabilization of the angular velocity of a rigid body

Output feedback stabilization of the angular velocity of a rigid body Systems & Control Letters 36 (1999) 181 192 Output feedback stabilization of the angular velocity of a rigid body A. Astol Department of Electrical and Electronic Engineering and Centre for Process Systems

More information

On generalized gradients and optimization

On generalized gradients and optimization On generalized gradients and optimization Erik J. Balder 1 Introduction There exists a calculus for general nondifferentiable functions that englobes a large part of the familiar subdifferential calculus

More information

Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, In which we introduce the theory of duality in linear programming.

Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, In which we introduce the theory of duality in linear programming. Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, 2011 Lecture 6 In which we introduce the theory of duality in linear programming 1 The Dual of Linear Program Suppose that we

More information

Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

More information

Introduction to Flocking {Stochastic Matrices}

Introduction to Flocking {Stochastic Matrices} Supelec EECI Graduate School in Control Introduction to Flocking {Stochastic Matrices} A. S. Morse Yale University Gif sur - Yvette May 21, 2012 CRAIG REYNOLDS - 1987 BOIDS The Lion King CRAIG REYNOLDS

More information

OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN. E.V. Grigorieva. E.N. Khailov

OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN. E.V. Grigorieva. E.N. Khailov DISCRETE AND CONTINUOUS Website: http://aimsciences.org DYNAMICAL SYSTEMS Supplement Volume 2005 pp. 345 354 OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN E.V. Grigorieva Department of Mathematics

More information

Chapter 7. Continuity

Chapter 7. Continuity Chapter 7 Continuity There are many processes and eects that depends on certain set of variables in such a way that a small change in these variables acts as small change in the process. Changes of this

More information

HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!

HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)! Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following

More information

Random graphs with a given degree sequence

Random graphs with a given degree sequence Sourav Chatterjee (NYU) Persi Diaconis (Stanford) Allan Sly (Microsoft) Let G be an undirected simple graph on n vertices. Let d 1,..., d n be the degrees of the vertices of G arranged in descending order.

More information

Chapter 15 Introduction to Linear Programming

Chapter 15 Introduction to Linear Programming Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of

More information

The Heat Equation. Lectures INF2320 p. 1/88

The Heat Equation. Lectures INF2320 p. 1/88 The Heat Equation Lectures INF232 p. 1/88 Lectures INF232 p. 2/88 The Heat Equation We study the heat equation: u t = u xx for x (,1), t >, (1) u(,t) = u(1,t) = for t >, (2) u(x,) = f(x) for x (,1), (3)

More information

EXPLICIT ABS SOLUTION OF A CLASS OF LINEAR INEQUALITY SYSTEMS AND LP PROBLEMS. Communicated by Mohammad Asadzadeh. 1. Introduction

EXPLICIT ABS SOLUTION OF A CLASS OF LINEAR INEQUALITY SYSTEMS AND LP PROBLEMS. Communicated by Mohammad Asadzadeh. 1. Introduction Bulletin of the Iranian Mathematical Society Vol. 30 No. 2 (2004), pp 21-38. EXPLICIT ABS SOLUTION OF A CLASS OF LINEAR INEQUALITY SYSTEMS AND LP PROBLEMS H. ESMAEILI, N. MAHDAVI-AMIRI AND E. SPEDICATO

More information

The Limit of a Sequence of Numbers: Infinite Series

The Limit of a Sequence of Numbers: Infinite Series Connexions module: m36135 1 The Limit of a Sequence of Numbers: Infinite Series Lawrence Baggett This work is produced by The Connexions Project and licensed under the Creative Commons Attribution License

More information

Lecture Notes on Engineering Optimization. Fraser J. Forbes and Ilyasse

Lecture Notes on Engineering Optimization. Fraser J. Forbes and Ilyasse 1 Lecture Notes on Engineering Optimization Fraser J. Forbes and Ilyasse Aksikas Department of Chemical and Materials Engineering University of Alberta aksikas@ualberta.ca 2 Contents Chapter 1 : Introduction

More information

Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

More information

Solutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR

Solutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR Solutions Of Some Non-Linear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision

More information

Definition of a Linear Program

Definition of a Linear Program Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

More information

Measuring Rationality with the Minimum Cost of Revealed Preference Violations. Mark Dean and Daniel Martin. Online Appendices - Not for Publication

Measuring Rationality with the Minimum Cost of Revealed Preference Violations. Mark Dean and Daniel Martin. Online Appendices - Not for Publication Measuring Rationality with the Minimum Cost of Revealed Preference Violations Mark Dean and Daniel Martin Online Appendices - Not for Publication 1 1 Algorithm for Solving the MASP In this online appendix

More information

PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5.

PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5. PUTNAM TRAINING POLYNOMIALS (Last updated: November 17, 2015) Remark. This is a list of exercises on polynomials. Miguel A. Lerma Exercises 1. Find a polynomial with integral coefficients whose zeros include

More information

Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

More information

Low-complexity minimization algorithms

Low-complexity minimization algorithms NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS Numer. Linear Algebra Appl. 2005; 12:755 768 Published online 15 June 2005 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/nla.449 Low-complexity

More information

Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms

Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms Buered Probability of Exceedance: Mathematical Properties and Optimization Algorithms Alexander Mafusalov, Stan Uryasev RESEARCH REPORT 2014-1 Risk Management and Financial Engineering Lab Department of

More information

The Goldberg Rao Algorithm for the Maximum Flow Problem

The Goldberg Rao Algorithm for the Maximum Flow Problem The Goldberg Rao Algorithm for the Maximum Flow Problem COS 528 class notes October 18, 2006 Scribe: Dávid Papp Main idea: use of the blocking flow paradigm to achieve essentially O(min{m 2/3, n 1/2 }

More information

Advanced Microeconomics

Advanced Microeconomics Advanced Microeconomics Ordinal preference theory Harald Wiese University of Leipzig Harald Wiese (University of Leipzig) Advanced Microeconomics 1 / 68 Part A. Basic decision and preference theory 1 Decisions

More information

Double Sequences and Double Series

Double Sequences and Double Series Double Sequences and Double Series Eissa D. Habil Islamic University of Gaza P.O. Box 108, Gaza, Palestine E-mail: habil@iugaza.edu Abstract This research considers two traditional important questions,

More information

Topic 1: Matrices and Systems of Linear Equations.

Topic 1: Matrices and Systems of Linear Equations. Topic 1: Matrices and Systems of Linear Equations Let us start with a review of some linear algebra concepts we have already learned, such as matrices, determinants, etc Also, we shall review the method

More information

Discrete Optimization

Discrete Optimization Discrete Optimization [Chen, Batson, Dang: Applied integer Programming] Chapter 3 and 4.1-4.3 by Johan Högdahl and Victoria Svedberg Seminar 2, 2015-03-31 Todays presentation Chapter 3 Transforms using

More information

5. Convergence of sequences of random variables

5. Convergence of sequences of random variables 5. Convergence of sequences of random variables Throughout this chapter we assume that {X, X 2,...} is a sequence of r.v. and X is a r.v., and all of them are defined on the same probability space (Ω,

More information

Ecient approximation algorithm for minimizing makespan. on uniformly related machines. Chandra Chekuri. November 25, 1997.

Ecient approximation algorithm for minimizing makespan. on uniformly related machines. Chandra Chekuri. November 25, 1997. Ecient approximation algorithm for minimizing makespan on uniformly related machines Chandra Chekuri November 25, 1997 Abstract We obtain a new ecient approximation algorithm for scheduling precedence

More information

ON THE APPROXIMATION OF INCONSISTENT INEQUALITY SYSTEMS

ON THE APPROXIMATION OF INCONSISTENT INEQUALITY SYSTEMS An. Şt. Univ. Ovidius Constanţa Vol. 11(), 003, 109 118 ON THE APPROXIMATION OF INCONSISTENT INEQUALITY SYSTEMS Elena Popescu Abstract In this paper is analyzed the minimal correction problem for an inconsistent

More information

Optimal shift scheduling with a global service level constraint

Optimal shift scheduling with a global service level constraint Optimal shift scheduling with a global service level constraint Ger Koole & Erik van der Sluis Vrije Universiteit Division of Mathematics and Computer Science De Boelelaan 1081a, 1081 HV Amsterdam The

More information

1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where. Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

More information

arxiv:1112.0829v1 [math.pr] 5 Dec 2011

arxiv:1112.0829v1 [math.pr] 5 Dec 2011 How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

More information

Tutorial on Convex Optimization for Engineers Part I

Tutorial on Convex Optimization for Engineers Part I Tutorial on Convex Optimization for Engineers Part I M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

More information