B553 Lecture 10: Convex Optimization and Interior-Point Methods

Transcription

1 B553 Lecture 10: Convex Optimization and Interior-Point Methods Kris Hauser February 8, 2012 Convex optimization is an important class of constrained optimization problems that subsumes linear and quadratic programming. Such problems are specified in the form: min f(x) x R n x S (1) where the feasible set S R n is a convex closed subset of R n and f is a convex function. Convex optimization problems are important because all local minima are guaranteed to be globally minimal. (This value may not necessarily be reached at a unique point). Furthermore, there are an important class of optimization algorithms, called interior point methods, that work particularly well for convex problems. In fact, interior point methods are competitive with the simplex algorithm for LPs and QPs. (Practical experience finds that interior point methods work better on some problems, while simplex algorithms work better for others). 1 Convex Optimization 1.1 Convex sets Convex set. A convex set S satisfies the following property. For any two points x and y in S, the point (1 u)x + uy (2) 1

2 for u [0, 1] lies in S as well. In other words, the line segment between any two points in S must also lie in S. Convex Hull. The smallest convex set containing another set A is known as the convex hull of A, CH(A). 1.2 Convex functions A function or scalar field f is convex if it satisfies the following property: f((1 u)x + uy) (1 u)f(x) + uf(y). (3) for all x and y and any u [0, 1]. In other words, the function value between any two points x and y lies below the line of the graph of f connecting f(x) and f(y). An equivalent definition is that on the line segment between x and y, the area over the graph of f forms a convex set. Convex functions are always somewhat bowl-shaped, have no local maxima (unless constant), and have no more than one local minimum value. If a local minimum exists, then it is also the global minimum. Hence, gradient descent and Newton methods (with line search) are guaranteed to produce the global minimum when applied to such functions. 1.3 Some Properties of Convex Sets and Functions Convex sets are connected. The intersection of any number of convex sets is also a convex set. The empty set, the whole space, any linear subspaces, and halfspaces are convex. If a convex set S is of lower dimension than its surrounding space R n, then S lies on a linear subspace of R n. Some common convex functions include e x, log(x), and x k where k is an even number. Convex functions are closed under addition, the max operator, monotonic transformations of the x variable, and scaling by a nonnegative constant. Constraints of the form x T Ax c produce a closed convex set (an ellipsoid) if A is positive semidefinite and c is a nonnegative number. The set of positive semidefinite matrices is a closed convex set. Optimization problems with these constraints are known as semidefinite programs (SDPs). A surprising number of problems, including LPs and QPs, can be transformed into SDPs. 2

3 1.4 Convex Optimization Problems An optimization problem in general form: min f(x) x R n g i (x) 0, i = 1,..., m h j (x) = 0, j = 1,..., p is convex as long as f is convex, all of the g i for i = 1,..., m are convex, and all of the h j for j = 1,..., p are linear. In convex optimization we will typically eliminate the equalities before optimizing, either by converting them into two inequalities h j (x) 0 and h j (x) 0, or by performing a null-space transformation. Henceforth we will drop the h j s for the rest of this class. 2 Interior Point Methods The main idea of interior point methods is to iterate in the interior of the feasible set and then progressively approach the boundary (if the minimum does indeed lie on the boundary). They do so by transforming the original problem into a sequence of unconstrained optimization problems, in which the objective uses a barrier function that goes to infinity at the boundaries of the feasible region. By reducing the strength of this barrier at each subsequent optimization, the sequence of minima approach arbitrarily closely toward the minimum of the original problem. 2.1 Objective functions with logarithmic barriers For the inequality constrained problem min f(x) x R n g i (x) 0, i = 1,..., m we modify f to obtain a barrier function: f α (x) = f(x) α ln( g i (x)) (6) 3 (4) (5)

4 Since the logarithm goes to as its argument approach 0, the modified objective function becomes arbitrarily large as x approaches the boundaries of the feasible region. The parameter α gives the barrier strength. Its significance is that as α approaches 0, the optimum of f α (x) approaches the optimum of (5). More precisely, if f optimizes (5), and x α arg min x f α (x): 2.2 Optimizing the Barrier Function lim α 0 f(x α) = f. (7) We consider using Newton s method to find the minimum of f α (x). The gradient is g i (x) f α (x) = f(x) α g i (x). (8) One concern is that as x α proceeds closer and closer to the boundary, the numerical stability of the unconstrained optimization problem becomes worse and worse because the denominator approaches zero. So, it would be very challenging to apply gradient descent to a small tolerance. Instead, we cleverly introduce another set of KKT multiplier-like variable λ = (λ 1,..., λ m ) with λ i α/g i (x). So, in the (x, λ) space, we want to find a root of the equation x f α (x, λ) = f(x) g i (x)λ i = 0 (9) subject to the equality λ i g i (x) = α for i = 1,..., m. (10) This set of equations is far more numerically stable than (8) near the boundary. Note the similarity between these equations and the KKT equations! In fact, if α were zero we would get the KKT conditions except with the equalities stripped away. To find a root (x α, λ α) where both of the functions are satisfied, we will apply Newton s method. First, we derive the partial derivatives of the above functions: 4

5 and 2 xxf α (x, λ) = 2 f(x) λ x f α (x) = 2 g i (x)λ i, (11) g i (x). (12) x (λ i g i (x) α) = λ i g i (x) (13) λ i (λ i g i (x) α) = g i (x) for i = 1,..., m (14) At the current iterate (x t, λ t ), we derive the Newton step ( x, λ) from the following system of equations: [ ] [ ] [ ] H G x f(xt ) + Gλ diag(λ t )G T = t (15) diag(g) λ diag(g)λ t α1 where H denotes (11) evaluated at (x t, λ t ), g denotes the m 1 vector of g i s, G denotes the n m matrix whose i th column is g i (x t ), diag(v) produces a square matrix whose diagonal is the vector v, and 1 denotes the m 1 vector of all 1 s. Solving this system gives us a search direction which is then updated via line search to avoid divergence. Note that to keep x from drifting out of the feasible set we must enforce for all i the condition g i (x) 0, or equivalently λ i 0, during the line search. Once the unconstrained search has converged, µ may be reduced (e.g., by multiplication by a small number) and the optimization can begin again. There is a tradeoff in the convergence thresholds for each unconstrained search: too small, and the algorithm must perform a lot of work even when µ is high, but too large, and sequence of unconstrained optima does not converge quickly. A more balanced approach is to choose the threshold proportional to µ. 2.3 Special cases Note that in this interior point method we did not require that the problem be convex. Interior point methods can certainly be used in general nonconvex problems but like any local optimization they are not guaranteed to converge 5

6 to a minimum. Furthermore, computing the Hessian matrix is often quite expensive. They do however work quite well in convex problems. For example, in quadratic programming, the Hessian is constant, and in linear programming, the Hessian is zero. 2.4 Initialization at a feasible point Note that the interior point methods must be initialized at an interior point, or else the barrier function is undefined. To find an initial feasible point x we can use the following optimization: min x R n,s 1,...,s m s i g i (x) s i 0, i = 1,..., ms i 0 (16) If the problem is feasible, then the optimal s i will all be 0. It is easy to find an initial set of s i for any given x simply by setting s i = max(g i (x), 0). 3 Exercises 1. 6