# 2.3 Convex Constrained Optimization Problems

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions is satisfied: (1) f is convex, h is nondecreasing and convex. (2) f is concave, h is nonincreasing and convex. By applying Theorem 15, we can see that: 1 f(x) e f(x) is convex if f is convex, is convex if f is concave and f(x) > 0 for all x. 2.3 Convex Constrained Optimization Problems In this section, we consider a generic convex constrained optimization problem. We introduce the basic terminology, and study the existence of solutions and the optimality conditions. We conclude this section with the projection problem and projection theorem. which is important for the subsequent algorithmic development Constrained Problem Consider the following constrained optimization problem minimize f(x) subject to g 1 (x) 0,..., g m (x) 0 Bx = d x X, (2.5) where f : R n R is an objective function, X R n is a given set, g i : R n R, i = 1,..., m are constraint functions, B is a p n matrix, and d R p. Let g(x) 0 compactly denote the inequalities g 1 (x) 0,..., g m (x) 0. Define C = {x R n g(x) 0, Bx = d, x X}. (2.6) We refer to the set C as constraint set or feasible set. The problem is feasible when C is nonempty. We refer to the value inf x C f(x) as the optimal value and denote it by f, i.e., f = inf x C f(x), where C is given by Eq. (2.6). A vector x is optimal (solution) when x is feasible and attains the optimal value f, i.e., g(x ) 0, Bx = d, x X, f(x ) = f. Before attempting to solve problem (2.5), there are some important questions to be answered, such as:

2 2.3. CONVEX CONSTRAINED OPTIMIZATION PROBLEMS 43 Is the problem infeasible, i.e., is C empty? Is f = +? 1. Is f =? If the answer is yes to any of the preceding questions, then it does not make sense to consider the problem in (2.5) any further. The problem is of interest only when f is finite. In this case, the particular instances when the problem has a solution are of interest in many applications. A feasibility problem is the problem of determining whether the constraint set C in Eq. (2.6) is empty or not. It can be posed as an optimization problem with the objective function f(x) = 0 for all x R n. In particular, a feasibility problem can be reduced to the following minimization problem: minimize 0 (2.7) subject to g(x) 0, Bx = d, x X. (2.8) In many applications, the feasibility problem can be a hard problem on its own. For example, the stability in linear time invariant systems often reduces to the feasibility problem where we want to determine whether there exist positive definite matrices P and Q such that A T P + P A = Q. Equivalently, the question is whether the set {(P, Q) S n ++ S n ++ A T P + P A = Q} is nonempty, where S n ++ denotes the set of n n positive definite matrices. From now on, we assume that the problem in Eq. (2.5) is feasible, and we focus on the issues of finiteness of the optimal value f and the existence of optimal solutions x. In particular, for problem (2.5), we use the following assumption. Assumption 1 The functions f and g i, i = 1,..., m are convex over R n. The set X is closed and convex. The set C = {x R n g(x) 0, Bx = d, x X} is nonempty. Under Assumption 1, the functions f and g i, i = 1,..., m are continuous over R n (see Theorem 10). In what follows,we denote the set of optimal solutions of problem (2.5) by X Existence of Solutions Here, we provide some results on the existence of solutions. Under Assumption 1, these results are consequences of Theorems 3 and 4 of Section This happens in general only when domf C =.

3 44 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 16 Let Assumption 1 hold. In addition, let X R n be bounded. Then, the optimal set X of problem (2.5) is nonempty, compact, and convex. Proof. At first, we show that the constraint set C is compact. The set C is the intersection of the level sets of continuous functions g i and hyperplanes (for j = 1,..., p, each set {x R n b T j x = d j } is a hyperplane), which are all closed sets. Therefore, C is closed. Since C X and X is bounded (because it is compact), the set C is also bounded. Hence, by Lemma 8 of Section 1.2.2, the set C is compact. The function f is continuous (by convexity over R n ). Hence, by Weierstrass Theorem (Theorem 3), the optimal value f of problem (2.5) is finite and its optimal set X is nonempty. The set X is closed since it can be represented as the intersection of closed sets: X = C {x R n f(x) f }. Furthermore, X is bounded since X C and C is bounded. Hence, X is compact. We now show that X is convex. Note that C is convex as it is given as the intersection of convex sets. Furthermore, the level set {x R n f(x) f } is convex by convexity of f. Hence, X is the intersection of two convex sets and, thus, X is convex. As seen in the proof of Theorem 16, the set C is closed and convex under Assumption 1. Also, under this assumption, the set X is closed and convex but possibly empty. The boundedness of X is the key assumption ensuring nonemptiness and boundedness of X. The following theorem is based on Theorem 4. We provide it without a proof. (The proof can be constructed similar to that of Theorem 16. The only new detail is in part (i), where using the coercivity of f, we show that the level sets of f are bounded.) Theorem 17 Let Assumption 1 hold. Furthermore, let any of the following conditions be satisfied: (i) The function f is coercive over C. (ii) For some γ R, the set {x C f(x) γ} is nonempty and compact. (iii) The set C is compact. Then, the optimal set X of problem (2.5) is nonempty, compact, and convex. For a quadratic convex objective and a linear constraint set, the existence of solutions is equivalent to finiteness of the optimal value. Furthermore, the issue of existence of solutions can be resolved by checking a linear condition, as seen in the following theorem. Theorem 18 Consider the problem minimize f(x) = x T P x + c T x subject to Ax b, where P is an n n positive semidefinite matrix, c R n, A is an m n matrix, and b R m. The following statements are equivalent:

4 2.3. CONVEX CONSTRAINED OPTIMIZATION PROBLEMS 45 (1) The optimal value f is finite. (2) The optimal set X is nonempty. (3) If Ay 0 and P y = 0 for some y R n, then c T y 0. The proof of Theorem 18 requires the notion of recession directions of convex closed sets, which is beyond the scope of these notes. The interested reader can find more discussion on this in Bertsekas, Nedić and Ozdaglar [9] (see there Proposition 2.3.5), or in Auslender and Teboulle [2]. As an immediate consequence of Theorem 18, we can derive the conditions for existence of solutions of linear programming problems. Corollary 1 Consider a linear programming problem minimize f(x) = c T x subject to Ax b. The following conditions are equivalent for the LP problem: (1) The optimal value f is finite. (2) The optimal set X is nonempty. (3) If Ay 0 for some y R n, then c T y 0. A linear programming (LP) problem that has a solution, it always has a solution of a specific structure. This specific solution is due to the geometry of the polyhedral constraint set. We describe this specific solution for an LP problem in a standard form: minimize subject to c T x Ax = b x 0, (2.9) where A is an m n matrix and b R m. The feasible set for the preceding LP problem is the polyhedral set {x R n Ax = b, x 0}. Definition 5 We say that x is a basic feasible solution for the LP in Eq. (2.9), when x is feasible and there are n linearly independent constraints among the constraints that x satisfies as equalities. The preceding definition actually applies to an LP problem in any form and not necessarily in the standard form. Furthermore, a basic solution of an LP is a vertex (or extreme point) of the (polyhedral) constraint set of the given LP, which are out of the scope of these lecture notes. The interested readers may find more on this, for example, in the textbook on linear optimization by Bertsimas and Tsitsiklis [11].

5 46 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Note that for a given polyhedral set, there can be only finitely many basic solutions. However, the number of such solutions may be very large. For example, the cube {x R n 0 x i 1, i = 1,..., n} is given by 2n inequalities, and it has 2 n basic solutions. We say that a vector x is a basic solution if it satisfies Definition 5 apart from being feasible. Specifically, x is a basic solution for (2.9) if there are n linearly independent constraints among the constraints that x satisfies as equalities. A basic solution x is degenerate if more than n constraints are satisfied as equalities at x (active at x). Otherwise, it is nondegenerate. Example 8 Consider the polyhedral set given by minimize x 1 + x 2 + x 3 2 subject to x 2 + 2x 3 2 x 1 1 x 3 1 x 1 0, x 2 0, x 3 0. The vector x = (1, 1, 0) is a nondegenerate basic feasible solution since there are exactly three linearly independent constraints that are active at x, specifically, x 1 + x 2 + x 3 2, x 1 1, x 3 0. The vector ˆx = (1, 0, 1) is a degenerate feasible solution since there are five constraints active at ˆx, namely x 1 + x 2 + x 3 2, x 2 + 2x 3 2, x 1 1, x 3 1, x 2 0. Out of these, for example, the last three are linearly independent. We are now ready to state a fundamental result for linear programming solutions. Theorem 19 Consider an LP problem. Assume that its constraint set has at least one basic feasible solution and that the LP has an optimal solution. Then, there exists an optimal solution which is also a basic feasible solution Optimality Conditions In this section, we deal with a differentiable convex function. We have the following. Theorem 20 Let f : R n R be a differentiable convex function, and let C R n be a nonempty closed convex set. Consider the problem minimize f(x) subject to x C. A vector x is optimal for this problem if and only if x C and f(x ) T (z x ) 0 for all z C.

6 2.3. CONVEX CONSTRAINED OPTIMIZATION PROBLEMS 47 Proof. For the sake of simplicity, we prove the result assuming that f is continuously differentiable. Let x be optimal. Suppose that for some ẑ C we have f(x ) T (ẑ x ) < 0. Since f is continuously differentiable, by the first-order Taylor expansion [Theorem 6(a)], we have for all sufficiently small α > 0, f(x + α(ẑ x )) = f(x ) + α f(x ) T (ẑ x ) + o(α) < f(x ), with x C and ẑ C. By the convexity of C, we have x + α(ẑ x ) C. Thus, this vector is feasible and has a smaller objective value than the optimal point x, which is a contradiction. Hence, we must have f(x ) T (z x ) 0 for all z C. Suppose now that x C and By convexity of f [see Theorem 12], we have implying that This and Eq. (2.10) further imply that f(x ) T (z x ) 0 for all z C. (2.10) f(x ) + f(x ) T (z x ) f(z) for all z C, Since x C, it follows that x is optimal. f(x ) T (z x ) f(z) f(x ). 0 f(z) f(x ) for all z C. We next discuss several implications of Theorem 20, by considering some special choices for the set C. Let C be the entire space, i.e., C = R n. The condition f(x ) T (z x ) 0 for all z C reduces to In turn, this is equivalent to f(x ) T d 0 for all d R n. (2.11) f(x ) = 0. Thus, by Theorem 20, a vector x is a minimum of f over R n if and only if f(x ) = 0. Let the set C be affine, i.e., the problem of interest is minimize f(x) subject to Ax = b, (2.12) where A is an m n matrix and b R m. In this case, the condition of Theorem 20 reduces to f(x ) T y 0 for all y N A,

7 48 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION where N A is the null space of the matrix A. Thus, the gradient f(x ) is orthogonal to the null space N A. Since the range of A T is orthogonal to N A [see Eq. (1.2)], it follows that f(x ) belongs to the range of A T, implying that f(x ) + A T λ = 0 for some λ R m. Hence, by Theorem 20, a vector x solves problem (2.12) if and only if Ax = b and there exists λ R m such that f(x ) + A T λ = 0. The relation f(x ) + A T λ = 0 is known as primal optimality condition. It is related to Lagrangian duality, which is the focus of Section 2.5. Let C be the nonnegative orthant in R n, i.e., the problem of interest is For this problem, the condition of Theorem 20 is equivalent to minimize f(x) subject to x 0. (2.13) f(x ) T x = 0. Therefore, a vector x solves problem (2.13) if and only if x 0 and f(x ) T x = 0. The relation f(x ) T x = 0 is known as complementarity condition, and the terminology comes again from the Lagrangian duality theory. Let C be a simplex in R n, i.e., the problem of interest is minimize f(x) subject to x 0, n x i = a, (2.14) i=1 where a > 0 is a scalar. By Theorem 20, x is optimal if and only if n i=1 f(x ) x i (x i x i ) 0 for all x i 0 with n i=1 x i = a. Consider an index i with x i > 0. Let j i and consider a feasible vector x with x i = 0, x j = x j + x i and all the other coordinates the same as those of x. By using this vector in the preceding relation, we obtain ( ) f(x ) f(x ) x i 0 for all i such that x i > 0, x j x i or equivalently f(x ) x i f(x ) x j for all i such that x i > 0. (2.15) Hence, x is an optimal solution to problem (2.14) if and only if x satisfies relation (2.15). Let us illustrate the optimality conditions for a simplical constraint set on the problem of optimal routing in a communication network (see [5] and [17]).

8 2.3. CONVEX CONSTRAINED OPTIMIZATION PROBLEMS 49 Example 9 (Optimal Routing) Consider a directed graph modeling a data communication network. Let S be a set of origin-destination pairs, i.e., each s S is an ordered pair (i s, j s ) of nodes i s and j s in the network, with i s being the origin and j s being the destination of s. Let y s be the traffic flow of s (data units/second) i.e., the arrival rate of traffic entering the network at the origin of s and exiting the network at the destination of s. The traffic flow of s is routed through different paths in the network. There is a cost associated with using the links L of the network, namely, the cost of sending a flow z ij on the link (i, j) L is f ij (z ij ), where f ij is convex and continuously differentiable. The problem is to decide on paths along which the flow y s should be routed, so as to minimize the total cost. To formalize the problem, we introduce the following notation: - P s is the set of all directed paths from the origin of s to the destination of s. - x s is the part of the flow y s routed through the path p with p P s. Let x denote a vector of path-flow variables, i.e., x = {x p p P s, s S}. Then, the routing problem can be casted as the following convex minimization: minimize f(x) = subject to (i,j) L f ij {p (i,j) p} x p p P s x p = y s for all s S x p 0 for all p P s and all s S. The cost on link (i, j) depends on the total flow through that link, i.e., the sum of all flows x p along paths p that contain the link (i, j). The problem is convex in variable x, with differentiable objective function and a constraint set given by a Cartesian product of simplices (one simplex per s). We now consider the optimality conditions for the routing problem. Note that f(x) x p = (i,j) p f ij(z ij ), with z ij being the total flow on the link (i, j). When f ij(z ij ) is viewed as the length of the link (i, j) evaluated at z ij, the partial derivative f(x) x p is the length of the path p. By the necessary and sufficient conditions for a simplex [cf. Eq. 2.14], for all s S, we have x p > 0 when f(x ) f(x ) for all p P s. x p x p This relation means that a set of path-flows is optimal if and only if the flow is positive only on the shortest paths (where link length is measured by the first derivative). It also means that at an optimum x, for an s S, all the paths p P s carrying a positive flow x p > 0 have the same length (i.e., the traffic y s is routed through the paths of equal length).

9 50 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION In the absence of convexity, the point x C satisfying the condition f(x ) T (z x ) 0 for all z C is referred to as a stationary point. Such a point may be a local or global minimum of f over C. A global minimum of f over C is any solution to the problem of minimizing f over C. A local minimum of f over C is a point x C for which there exists a ball B( x, r) such that there is no better point among the points that belong to the ball B( x, r) and the set C, i.e., a ball B( x, r) such that f(x) f( x) for all x C B( x, r). For convex problems (i.e., convex f and C), there is no distinction between local and global minima: every local minimum is also global in convex problems. This makes solving convex minimization problems easier than solving a more general nonconvex problems. Let us note that for a strictly convex function, the optimal solution to the problem of minimizing f over C is unique (of course, when a solution exists). We state this result in the following theorem, whose proof follows from the definition of strict convexity. Theorem 21 Let C R n be a nonempty closed convex set and let f be a strictly convex function over C. If the problem of minimizing f over C has a solution, then the solution is unique. Proof. To arrive at a contradiction, assume that the optimal set X has more than one point. Let x 1 and x 2 be two distinct solutions, i.e., f(x 1) = f(x 2) = f and x 1 x 2. Also, let α (0, 1). Since f is convex, the set X is also convex implying that αx 1 + (1 α)x 2 X. Hence, f(αx 1 + (1 α)x 2) = f. (2.16) At the same time, by strict convexity of f over C and the relation X C, we have that f is strictly convex over X, so that f(αx 1 + (1 α)x 2) < αf(x 1) + (1 α)f(x 2) = f, which contradicts relation (2.16). Therefore, the solution must be unique Projection Theorem One special consequence of Theorems 4 and 20 is the Projection Theorem. The theorem guarantees the existence and uniqueness of the projection of a vector on a closed convex set. This result has a wide range of applications. For a given nonempty set C R n and a vector ˆx, the projection problem is the problem of determining the point x C that is the closest to ˆx among all x C (with respect to the Euclidean distance). Formally, the problem is given by minimize x ˆx 2 subject to x C. (2.17) In general, such a problem may not have an optimal solution and the solution need not be unique (when it exists). However, when the set C is closed and convex set, the solution exists and it is unique, as seen in the following theorem.

10 2.3. CONVEX CONSTRAINED OPTIMIZATION PROBLEMS 51 Figure 2.9: The projection of a vector ˆx on the closed convex set C is the vector P C [ˆx] C that is the closest to ˆx among all x C, with respect to the Euclidean distance. Theorem 22 (Projection Theorem) Let C R n be a nonempty closed convex set and ˆx R n be a given arbitrary vector. (a) The projection problem in Eq. (2.17) has a unique solution. (b) A vector x C is the solution to the projection problem if and only if (x ˆx) T (x x ) 0 for all x C. Proof. (a) The function f(x) = x ˆx is coercive over R n, and therefore coercive over C (i.e., lim x, x C f(x) = ). The set C is closed, and therefore by Theorem 4, the optimal set X for projection problem (2.17) is nonempty. Furthermore, the Hessian of f is given by 2 f(x) = 2I, which is positively definite everywhere. Therefore, by Theorem 11(b), the function f is strictly convex and the optimal solution is unique [cf. Theorem 21]. (b) By the first-order optimality condition of Theorem 20), we have x C is a solution to the projection problem if and only if Since f(x) = 2(x ˆx), the result follows. f(x ) T (x x ) 0 for all x C. The projection of a vector ˆx to a closed convex set C is illustrated in Figure 2.9. The unique solution x to the projection problem is referred to as the projection of ˆx on C, and it is denoted by P C [ˆx]. The projection P C [ˆx] has some special properties, as given in the following theorem. Theorem 23 Let C R n be a nonempty closed convex set. (a) The projection mapping P C : R n C is nonexpansive, i.e., P C [x] P C [y] x y for all x, y R n.

11 52 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Figure 2.10: The projection mapping x P C [x] is nonexpansive, P C [x] P C [y] x y for all x, y. (b) The set distance function d : R n R given by is convex. dist(x, C) = P C [x] x Proof. (a) The relation evidently holds for any x and y with P C [x] = P C [y]. Consider now arbitrary x, y R n with P C [x] P C [y]. By Projection Theorem (b), we have (P C [x] x) T (z P C [x]) 0 for all z C, (2.18) (P C [y] y) T (z P C [y]) 0 for all z C. (2.19) Using z = P C [y] in Eq. (2.18) and z = P C [x] in Eq. (2.19), and by summing the resulting inequalities, we obtain, Consequently, (P C [y] y + x P C [x]) T (P C [x] P C [y]) 0. (x y) T (P C [x] P C [y]) P C [x] P C [y] 2. Since P C [x] P C [y], it follows that y x P C [x] P C [y]. (b) The distance function is equivalently given by dist(x, C) = min z C x z for all x Rn. The function h(x, z) = x z is convex in (x, z) over R n R n, and the set C is convex. Hence, by Theorem 13(d), the function dist(x, C) is convex.

### Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

### Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

### A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

### Shiqian Ma, SEEM 5121, Dept. of SEEM, CUHK 1. Chapter 2. Convex Optimization

Shiqian Ma, SEEM 5121, Dept. of SEEM, CUHK 1 Chapter 2 Convex Optimization Shiqian Ma, SEEM 5121, Dept. of SEEM, CUHK 2 2.1. Convex Optimization General optimization problem: min f 0 (x) s.t., f i (x)

### Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

### Lecture 3: Convex Sets and Functions

EE 227A: Convex Optimization and Applications January 24, 2012 Lecture 3: Convex Sets and Functions Lecturer: Laurent El Ghaoui Reading assignment: Chapters 2 (except 2.6) and sections 3.1, 3.2, 3.3 of

### THEORY OF SIMPLEX METHOD

Chapter THEORY OF SIMPLEX METHOD Mathematical Programming Problems A mathematical programming problem is an optimization problem of finding the values of the unknown variables x, x,, x n that maximize

### 1 Polyhedra and Linear Programming

CS 598CSC: Combinatorial Optimization Lecture date: January 21, 2009 Instructor: Chandra Chekuri Scribe: Sungjin Im 1 Polyhedra and Linear Programming In this lecture, we will cover some basic material

### Advanced Topics in Machine Learning (Part II)

Advanced Topics in Machine Learning (Part II) 3. Convexity and Optimisation February 6, 2009 Andreas Argyriou 1 Today s Plan Convex sets and functions Types of convex programs Algorithms Convex learning

### Introduction to Convex Optimization

Convexity Xuezhi Wang Computer Science Department Carnegie Mellon University 10701-recitation, Jan 29 Outline Convexity 1 Convexity 2 3 Outline Convexity 1 Convexity 2 3 Convexity Definition For x, x X

### Geometry of Linear Programming

Chapter 2 Geometry of Linear Programming The intent of this chapter is to provide a geometric interpretation of linear programming problems. To conceive fundamental concepts and validity of different algorithms

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing

### Absolute Value Programming

Computational Optimization and Aplications,, 1 11 (2006) c 2006 Springer Verlag, Boston. Manufactured in The Netherlands. Absolute Value Programming O. L. MANGASARIAN olvi@cs.wisc.edu Computer Sciences

### Cost Minimization and the Cost Function

Cost Minimization and the Cost Function Juan Manuel Puerta October 5, 2009 So far we focused on profit maximization, we could look at a different problem, that is the cost minimization problem. This is

### Convexity I: Sets and Functions

Convexity I: Sets and Functions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 See supplements for reviews of basic real analysis basic multivariate calculus basic linear algebra Last time:

### MATHEMATICAL BACKGROUND

Chapter 1 MATHEMATICAL BACKGROUND This chapter discusses the mathematics that is necessary for the development of the theory of linear programming. We are particularly interested in the solutions of a

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### 3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

### 24. The Branch and Bound Method

24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

### Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, In which we introduce the theory of duality in linear programming.

Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, 2011 Lecture 6 In which we introduce the theory of duality in linear programming 1 The Dual of Linear Program Suppose that we

### (Basic definitions and properties; Separation theorems; Characterizations) 1.1 Definition, examples, inner description, algebraic properties

Lecture 1 Convex Sets (Basic definitions and properties; Separation theorems; Characterizations) 1.1 Definition, examples, inner description, algebraic properties 1.1.1 A convex set In the school geometry

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### Convex analysis and profit/cost/support functions

CALIFORNIA INSTITUTE OF TECHNOLOGY Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m

### Walrasian Demand. u(x) where B(p, w) = {x R n + : p x w}.

Walrasian Demand Econ 2100 Fall 2015 Lecture 5, September 16 Outline 1 Walrasian Demand 2 Properties of Walrasian Demand 3 An Optimization Recipe 4 First and Second Order Conditions Definition Walrasian

### ME128 Computer-Aided Mechanical Design Course Notes Introduction to Design Optimization

ME128 Computer-ided Mechanical Design Course Notes Introduction to Design Optimization 2. OPTIMIZTION Design optimization is rooted as a basic problem for design engineers. It is, of course, a rare situation

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### max cx s.t. Ax c where the matrix A, cost vector c and right hand side b are given and x is a vector of variables. For this example we have x

Linear Programming Linear programming refers to problems stated as maximization or minimization of a linear function subject to constraints that are linear equalities and inequalities. Although the study

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).

1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction

### Lecture 5 Principal Minors and the Hessian

Lecture 5 Principal Minors and the Hessian Eivind Eriksen BI Norwegian School of Management Department of Economics October 01, 2010 Eivind Eriksen (BI Dept of Economics) Lecture 5 Principal Minors and

### 6.254 : Game Theory with Engineering Applications Lecture 5: Existence of a Nash Equilibrium

6.254 : Game Theory with Engineering Applications Lecture 5: Existence of a Nash Equilibrium Asu Ozdaglar MIT February 18, 2010 1 Introduction Outline Pricing-Congestion Game Example Existence of a Mixed

### No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics

No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results

### 1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### Min-cost flow problems and network simplex algorithm

Min-cost flow problems and network simplex algorithm The particular structure of some LP problems can be sometimes used for the design of solution techniques more efficient than the simplex algorithm.

### Optimization in R n Introduction

Optimization in R n Introduction Rudi Pendavingh Eindhoven Technical University Optimization in R n, lecture Rudi Pendavingh (TUE) Optimization in R n Introduction ORN / 4 Some optimization problems designing

### NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice

NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice IC-32: Winter Semester 2006/2007 Benoît C. CHACHUAT Laboratoire d Automatique, École Polytechnique Fédérale de Lausanne CONTENTS 1 Nonlinear

### Chapter 15 Introduction to Linear Programming

Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of

### Metric Spaces. Chapter 1

Chapter 1 Metric Spaces Many of the arguments you have seen in several variable calculus are almost identical to the corresponding arguments in one variable calculus, especially arguments concerning convergence

### Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

### 1 Norms and Vector Spaces

008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

CS787: Advanced Algorithms Topic: Online Learning Presenters: David He, Chris Hopman 17.3.1 Follow the Perturbed Leader 17.3.1.1 Prediction Problem Recall the prediction problem that we discussed in class.

### Increasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.

1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.

### On Minimal Valid Inequalities for Mixed Integer Conic Programs

On Minimal Valid Inequalities for Mixed Integer Conic Programs Fatma Kılınç Karzan June 27, 2013 Abstract We study mixed integer conic sets involving a general regular (closed, convex, full dimensional,

### Mathematical Economics - Part I - Optimization. Filomena Garcia. Fall Optimization. Filomena Garcia. Optimization. Existence of Solutions

of Mathematical Economics - Part I - Fall 2009 of 1 in R n Problems in Problems: Some of 2 3 4 5 6 7 in 8 Continuity: The Maximum Theorem 9 Supermodularity and Monotonicity of An optimization problem in

### GRA6035 Mathematics. Eivind Eriksen and Trond S. Gustavsen. Department of Economics

GRA635 Mathematics Eivind Eriksen and Trond S. Gustavsen Department of Economics c Eivind Eriksen, Trond S. Gustavsen. Edition. Edition Students enrolled in the course GRA635 Mathematics for the academic

### Course 221: Analysis Academic year , First Semester

Course 221: Analysis Academic year 2007-08, First Semester David R. Wilkins Copyright c David R. Wilkins 1989 2007 Contents 1 Basic Theorems of Real Analysis 1 1.1 The Least Upper Bound Principle................

### 1 Convex Sets, and Convex Functions

Convex Sets, and Convex Functions In this section, we introduce one of the most important ideas in the theory of optimization, that of a convex set. We discuss other ideas which stem from the basic definition,

### MA651 Topology. Lecture 6. Separation Axioms.

MA651 Topology. Lecture 6. Separation Axioms. This text is based on the following books: Fundamental concepts of topology by Peter O Neil Elements of Mathematics: General Topology by Nicolas Bourbaki Counterexamples

### Some representability and duality results for convex mixed-integer programs.

Some representability and duality results for convex mixed-integer programs. Santanu S. Dey Joint work with Diego Morán and Juan Pablo Vielma December 17, 2012. Introduction About Motivation Mixed integer

### By W.E. Diewert. July, Linear programming problems are important for a number of reasons:

APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization)

### α = u v. In other words, Orthogonal Projection

Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

### Convexity in R N Main Notes 1

John Nachbar Washington University December 16, 2016 Convexity in R N Main Notes 1 1 Introduction. These notes establish basic versions of the Supporting Hyperplane Theorem (Theorem 5) and the Separating

### Tutorial on Convex Optimization for Engineers Part I

Tutorial on Convex Optimization for Engineers Part I M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

### Error Bound for Classes of Polynomial Systems and its Applications: A Variational Analysis Approach

Outline Error Bound for Classes of Polynomial Systems and its Applications: A Variational Analysis Approach The University of New South Wales SPOM 2013 Joint work with V. Jeyakumar, B.S. Mordukhovich and

### TOPIC 4: DERIVATIVES

TOPIC 4: DERIVATIVES 1. The derivative of a function. Differentiation rules 1.1. The slope of a curve. The slope of a curve at a point P is a measure of the steepness of the curve. If Q is a point on the

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

### Introduction to Linear Programming.

Chapter 1 Introduction to Linear Programming. This chapter introduces notations, terminologies and formulations of linear programming. Examples will be given to show how real-life problems can be modeled

### Mathematical Methods of Engineering Analysis

Mathematical Methods of Engineering Analysis Erhan Çinlar Robert J. Vanderbei February 2, 2000 Contents Sets and Functions 1 1 Sets................................... 1 Subsets.............................

### Subgradients. S. Boyd and L. Vandenberghe Notes for EE364b, Stanford University, Winter April 13, 2008

Subgradients S. Boyd and L. Vandenberghe Notes for EE364b, Stanford University, Winter 2006-07 April 13, 2008 1 Definition We say a vector g R n is a subgradient of f : R n R at x domf if for all z domf,

### Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem

### Lecture 5: Conic Optimization: Overview

EE 227A: Conve Optimization and Applications January 31, 2012 Lecture 5: Conic Optimization: Overview Lecturer: Laurent El Ghaoui Reading assignment: Chapter 4 of BV. Sections 3.1-3.6 of WTB. 5.1 Linear

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### Network Traffic Modelling

University of York Dissertation submitted for the MSc in Mathematics with Modern Applications, Department of Mathematics, University of York, UK. August 009 Network Traffic Modelling Author: David Slade

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### Minimize subject to. x S R

Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such

### Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach

MASTER S THESIS Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branch-and-bound approach PAULINE ALDENVIK MIRJAM SCHIERSCHER Department of Mathematical

### International Doctoral School Algorithmic Decision Theory: MCDA and MOO

International Doctoral School Algorithmic Decision Theory: MCDA and MOO Lecture 2: Multiobjective Linear Programming Department of Engineering Science, The University of Auckland, New Zealand Laboratoire

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### Maximum and Minimum Values

Jim Lambers MAT 280 Spring Semester 2009-10 Lecture 8 Notes These notes correspond to Section 11.7 in Stewart and Section 3.3 in Marsden and Tromba. Maximum and Minimum Values In single-variable calculus,

### 10.1 Integer Programming and LP relaxation

CS787: Advanced Algorithms Lecture 10: LP Relaxation and Rounding In this lecture we will design approximation algorithms using linear programming. The key insight behind this approach is that the closely

### AND. Dimitri P. Bertsekas and Paul Tseng

SET INTERSECTION THEOREMS AND EXISTENCE OF OPTIMAL SOLUTIONS FOR CONVEX AND NONCONVEX OPTIMIZATION Dimitri P. Bertsekas an Paul Tseng Math. Programming Journal, 2007 NESTED SET SEQUENCE INTERSECTIONS Basic

### LECTURE 6: INTERIOR POINT METHOD. 1. Motivation 2. Basic concepts 3. Primal affine scaling algorithm 4. Dual affine scaling algorithm

LECTURE 6: INTERIOR POINT METHOD 1. Motivation 2. Basic concepts 3. Primal affine scaling algorithm 4. Dual affine scaling algorithm Motivation Simplex method works well in general, but suffers from exponential-time

### Separation Properties for Locally Convex Cones

Journal of Convex Analysis Volume 9 (2002), No. 1, 301 307 Separation Properties for Locally Convex Cones Walter Roth Department of Mathematics, Universiti Brunei Darussalam, Gadong BE1410, Brunei Darussalam

### 4. Expanding dynamical systems

4.1. Metric definition. 4. Expanding dynamical systems Definition 4.1. Let X be a compact metric space. A map f : X X is said to be expanding if there exist ɛ > 0 and L > 1 such that d(f(x), f(y)) Ld(x,

### Lecture 7 - Linear Programming

COMPSCI 530: Design and Analysis of Algorithms DATE: 09/17/2013 Lecturer: Debmalya Panigrahi Lecture 7 - Linear Programming Scribe: Hieu Bui 1 Overview In this lecture, we cover basics of linear programming,

### Metric Spaces. Chapter 7. 7.1. Metrics

Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. !-approximation algorithm.

Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of

### Ri and. i=1. S i N. and. R R i

The subset R of R n is a closed rectangle if there are n non-empty closed intervals {[a 1, b 1 ], [a 2, b 2 ],..., [a n, b n ]} so that R = [a 1, b 1 ] [a 2, b 2 ] [a n, b n ]. The subset R of R n is an

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. #-approximation algorithm.

Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of three

### Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.

Summer course on Convex Optimization Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota Interior-Point Methods: the rebirth of an old idea Suppose that f is

### On subdifferential calculus

On subdifferential calculus Erik J. Balder 1 Introduction The main purpose of these lectures is to familiarize the student with the basic ingredients of convex analysis, especially its subdifferential

### Inner products on R n, and more

Inner products on R n, and more Peyam Ryan Tabrizian Friday, April 12th, 2013 1 Introduction You might be wondering: Are there inner products on R n that are not the usual dot product x y = x 1 y 1 + +

### NP-hardness of Deciding Convexity of Quartic Polynomials and Related Problems

NP-hardness of Deciding Convexity of Quartic Polynomials and Related Problems Amir Ali Ahmadi, Alex Olshevsky, Pablo A. Parrilo, and John N. Tsitsiklis Abstract We show that unless P=NP, there exists no

### SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

### LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS. 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method

LECTURE 5: DUALITY AND SENSITIVITY ANALYSIS 1. Dual linear program 2. Duality theory 3. Sensitivity analysis 4. Dual simplex method Introduction to dual linear program Given a constraint matrix A, right

### Solutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR

Solutions Of Some Non-Linear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision

### Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

### Convex Functions. Convex Functions. Optimization Carlos Guestrin Carnegie Mellon University. March 17 th, Function f:r n R is convex if

Convex Functions Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 17 th, 2010 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

### minimal polyonomial Example

Minimal Polynomials Definition Let α be an element in GF(p e ). We call the monic polynomial of smallest degree which has coefficients in GF(p) and α as a root, the minimal polyonomial of α. Example: We

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and