Introduction to Optimization, and Optimality Conditions for Unconstrained Problems

Transcription

1 Introduction to Optimization, and Optimality Conditions for Unconstrained Problems Robert M. Freund February, Massachusetts Institute of Technology.

2 Preliminaries. Types of optimization problems Unconstrained Optimization Problem: (P) min x f(x) s.t. x X, where x = (x,...,x n ) R n, f(x) : R n R,and X is an open set (usually X = R n ). We say that x is a feasible solution of (P) if x X. Constrained Optimization Problem: (P) min x s.t. f(x) g i (x) 0 i =,...,m h i (x) =0 i =,...,l x X, where g (x),...,g m (x),h (x),...,h l (x) : R n R. Let g(x) =(g (x),...,g m (x)) : R n R m, h(x) =(h (x),...,h l (x)) :

3 R n R l. Then (P) can be written as (P) min x f(x) s.t. g(x) 0 h(x) =0 () x X. We say that x is a feasible solution of (P) if g(x) 0,h(x) = 0, and x X.. Local, Global, and Strict Optima The ball centered at x with radius ɛ is the set: B( x, ɛ) :={x x x ɛ}. Consider the following optimization problem over the set F: P : min x or max x f(x) s.t. x F We have the following definitions of local/global, strict/non-strict minima/maxima. Definition. x F is a local minimum of P if there exists ɛ> 0 such that f(x) f(y) for all y B(x, ɛ) F. Definition. x F is a global minimum of P if f(x) f(y) for all y F. 3

4 Definition.3 x F is a strict local minimum of P if there exists ɛ> 0 such that f(x) < f(y) for all y B(x, ɛ) F, y x. Definition.4 x F is a strict global minimum of P if f(x) < f(y) for all y F, y = x. Definition.5 x F is a local maximum of P if there exists ɛ> 0 such that f(x) f(y) for all y B(x, ɛ) F. Definition.6 x F is a global maximum of P if f(x) f(y) for all y F. Definition.7 x F is a strict local maximum of P if there exists ɛ > 0 such that f(x) > f(y) for all y B(x, ɛ) F, y x. Definition.8 x F is a strict global maximum of P if f(x) > f(y) for all y F, y = x..3 Gradients and Hessians Let f(x) : X R, where X R n is open. f(x) is differentiable at x X if there exists a vector f( x) (the gradient of f(x) at x) such that for each x X f(x) = f( x)+ f( x) t (x x)+ x x α( x, x x), and lim y 0 α( x, y) =0. f(x) is differentiable on X if f(x) is differentiable for all x X. The gradient vector is the vector of partial derivatives: f( x) f( x) t f( x) =,...,. x x n Example Let f(x) =3(x ) (x ) 3 +(x ) (x 3 ) 3. Then T f(x) = 6(x )(x ) 3, 9(x ) (x ) +(x )(x 3 ) 3, 3(x ) (x 3 ). 4

5 The directional derivative of f(x) at x in the direction d is: f( x + λd) f( x) lim λ 0 λ = f( x) t d The function f(x) is twice differentiable at x X if there exists a vector f( x) and an n n symmetric matrix H( x) (the Hessian of f(x) at x) such that for each x X f(x) = f( x)+ f( x) t (x x)+ (x x) t H( x)(x x)+ x x α( x, x x), and lim y 0 α( x, y) =0. f(x) is twice differentiable on X if f(x) istwice differentiable for all x X. The Hessian is the matrix of second partial derivatives: H( x) ij = f( x). x i x j Example Continuing Example, we have 6(x ) 3 8(x )(x ) 0 H(x) = 8(x )(x ) 8(x ) (x )+(x 3 ) 3 6(x )(x 3 ). 0 6(x )(x 3 ) 6(x ) (x 3 ).4 Positive Semidefinite and Positive Definite Matrices An n n matrix M is called positive definite if x t Mx > 0 for all x R n, x 0 positive semidefinite if x t Mx 0 for all x R n negative definite if x t Mx < 0 for all x R n, x 0 negative semidefinite if x t Mx 0 for all x R n, x 0 5

6 indefinite if there exists x, y R n for which x t Mx > 0and y t My < 0 We say that M is SPD if M is symmetric and positive definite. Similarly, we say that M is SPSD if M is symmetric and positive semi-definite. Example 3 is positive definite. 0 M = 0 3 Example 4 8 M = is positive definite. To see this, note that for x 0, x T Mx =8x x x + x =7x +(x x ) > 0..5 Existence of Optimal Solutions Most of the topics of this course are concerned with existence of optimal solutions, characterization of optimal solutions, and algorithms for computing optimal solutions. To illustrate the questions arising in the first topic, consider the following optimization problems: (P) min x + x x s.t. x. 6

7 Here there is no optimal solution because the feasible region is unbounded (P) min x x s.t. x<. Here there is no optimal solution because the feasible region is not closed. (P) min x f(x) s.t. x, where { /x, x < f(x) =, x =. Here there is no optimal solution because the function f( ) is not sufficiently smooth. Theorem (Weierstrass Theorem for sequences) Let {x k },k be an infinite sequence of points in the compact (i.e., closed and bounded) set F. Then some infinite subsequence of points x kj converges to a point contained in F. Theorem (Weierstrass Theorem for functions) Let f(x) be a continuous real-valued function on the compact nonempty set F R n. Then F contains a point that minimizes (maximizes) f(x) on the set F. Proof: Since the set F is bounded, f(x) is bounded below on F. Since F, there exists v =inf x F f(x). By definition, for any ɛ> 0, the set F ɛ = {x F : v f(x) v+ɛ} is non-empty. Let ɛ k 0as k,and let x k F ɛ k.since F is bounded, there exists a subsequence of {x k } converging to some x F. By continuity of f(x), we have f( x) = limk f(x k ), and, since v f(x k ) v + ɛ k, it follows that f( x) = lim k f(x k )= v. 7

8 The assumptions of Weierstrass Theorem can be somewhat relaxed. For example, for a minimization problem, we can assume that the set {x F : f (x) f (x )} is compact for some x F, and f (x) is lower semi-continuous, i.e., for any constant c, the set {x F : f (x) c} is closed. The proof is similar to the proof of the Weierstrass Theorem. 8

9 Optimality Conditions for Unconstrained Problems (P) min f(x) s.t. x X, where x = (x,...,x n ) R n, f(x) : R n R,and X is an open set (usually X = R n ). Definition. The direction d is called a descent direction of f(x) at x = x if f( x + ɛd ) < f( x) for all ɛ > 0 and sufficiently small. A necessary condition for local optimality is a statement of the form: if x is a local minimum of (P), then x must satisfy... Such a condition helps us identify all candidates for local optima. Theorem 3 Suppose that f(x) is differentiable at x. If there is a vector d such that f( x) t d< 0, then for all λ > 0 and sufficiently small, f( x+λd) < f( x), and hence d is a descent direction of f(x) at x. Proof: We have: f( x + λd) = f( x)+ λ f( x) t d + λ d α( x, λd), where α( x, λd) 0as λ 0. Rearranging, f( x + λd) f( x) = f( x) t d + d α( x, λd). λ Since f( x) t d< 0and α( x, λd) 0as λ 0, f( x + λd) f( x) < 0 for all λ> 0 sufficiently small. Corollary 4 Suppose f(x) is differentiable at x. If x is a local minimum, then f( x) =0. 9

10 Proof: If it were true that f ( x) = 0, then d = f ( x) would be a descent direction, whereby x would not be a local minimum. The above corollary is a first order necessary optimality condition for an unconstrained minimization problem. The following theorem is a second order necessary optimality condition Theorem 5 Suppose that f (x) is twice continuously differentiable at x X. If x is a local minimum, then f ( x) =0 and H ( x) is positive semidefinite. Proof: From the first order necessary condition, f ( x) = 0. Suppose H ( x) is not positive semi-definite. Then there exists d such that d t H( x)d < 0. We have: f ( x + λd) = f ( x)+ λ f ( x) t d + λ d t H ( x)d + λ d α( x, λd) = f ( x)+ λ d t H( x)d + λ d α( x, λd), where α( x, λd) 0as λ 0. Rearranging, f ( x + λd) f ( x) = d t H( x)d + d α( x, λd). λ Since d t H( x)d < 0and α( x, λd) 0as λ 0, f ( x + λd) f ( x) < 0 for all λ> 0 sufficiently small, yielding the desired contradiction. Example 5 Let Then and f (x) = x + x x +x 4x 4x x 3. f (x) = ( x + x 4,x +4x 4 3x ) T, H (x) =. 4 6x f (x) =0 has exactly two solutions: x = (4, 0) and x = (3, ). But H ( x) = 0

11 is indefinite, therefore, the only possible candidate for a local minimum is x = (4, 0). A sufficient condition for local optimality is a statement of the form: if x satisfies..., then x is a local minimum of (P). Such a condition allows us to automatically declare that x is indeed a local minimum. Theorem 6 Suppose that f (x) is twice differentiable at x. If f ( x) =0 and H ( x) is positive definite, then x is a (strict) local minimum. Proof: f (x) = f ( x)+ (x x) t H( x)(x x)+ x x α( x, x x). Suppose that x is not a strict local minimum. Then there exists a sequence x k x k x such that x x k = x and f (x k ) f ( x) for all k. Define d k = xk x. Then f (x k )= f ( x)+ x k x dk t H( x)d k + α( x, x k x), and so f (x k ) f ( x) d t k H ( x)d k + α( x, x k x) = 0. x x k Since d k = for any k, there exists a subsequence of {d k } converging to some point d such that d =. Assume without loss of generality that d k d. Then 0 lim d t k H ( x)d k + α( x, x x) = d t k H( x)d, k which is a contradiction of the positive definiteness of H ( x). Note: If f ( x) =0 and H( x) is negative definite, then x is a local maximum. If f ( x) =0 and H( x) is positive semidefinite, we cannot be sure if x is a local minimum.

12 Example 6 Continuing Example 5, we compute H( x) = 4 is positive definite. To see this, note that for any d = (d,d ), we have d T H( x)d = d +d =(d + d ) +3d d +4d > 0 for all d =0. Therefore, x satisfies the sufficient conditions to be a local minimum, and so x is a local minimum. Example 7 Let 3 f(x) =x + x. Then T f(x) = 3x, x, and 6x H(x) = 0. 0 At x = (0, 0), we have f( x) =0 and 0 0 H( x) = 0 is positive semi-definite, but x is not a local minimum, since f( ɛ, 0) = ɛ 3 < 0 =f(0, 0) = f( x) for all ɛ > 0. Example 8 Let 4 f(x) =x + x. Then T 3 f(x) = 4x, x, and ( ) x 0 H(x) =. 0 At x = (0, 0), we have f( x) =0 and 0 0 H( x) = 0 is positive semi-definite. Furthermore, x is a local minimum, since for all x we have f(x) 0 =f(0, 0) = f( x).

13 . Convexity and Minimization Let x, y R n. Points of the form λx +( λ)y for λ [0, ] are called convex combinations of x and y. Aset S R n is called a convex set if for all x, y S and for all λ [0, ] it holds that λx +( λ)y S. A function f (x) : S R, where S is a nonempty convex set, is a convex function if f (λx +( λ)y) λf (x)+( λ)f (y) for all x, y S and for all λ [0, ]. A function f (x) as above is called a strictly convex function if the inequality above is strict for all x = y and λ (0, ). A function f (x) : S R, where S is a nonempty convex set, is a concave function if f (λx +( λ)y) λf (x)+( λ)f (y) for all x, y S and for all λ [0, ]. A function f (x) as above is called a strictly concave function if the inequality above is strict for all x y and λ (0, ). Consider the problem: (CP) min x f (x) s.t. x S. 3

14 Theorem 7 Suppose S is a convex set, f (x) : S Ris a convex function, and x is a local minimum of (CP). Then x is a global minimum of f (x) over S. Proof: Suppose x is not a global minimum, i.e., there exists y S for which f (y) < f ( x). Let y(λ) := λ x +( λ)y, which is a convex combination of x and y for λ [0, ] (and therefore, y(λ) S for λ [0, ]). Note that y(λ) x as λ 0. From the convexity of f (x), f (y(λ)) = f (λ x+( λ)y) λf ( x)+( λ)f (y) < λf ( x)+( λ)f ( x) = f ( x) for all λ (0, ). Therefore, f (y(λ)) < f ( x) for all λ (0, ), and so x is not a local minimum, resulting in a contradiction. Note: If f (x) is strictly convex, a local minimum is the unique global minimum. If f (x) is concave, a local maximum is a global maximum. If f (x) is strictly concave, a local maximum is the unique global maximum. Theorem 8 Suppose S is a non-empty open convex set, and f (x) : S R is differentiable. Then f (x) is a convex function if and only if f (x) satisfies the following gradient inequality: f (y) f (x)+ f (x) t (y x) for all x, y S. Proof: Suppose f (x) is convex. Then for any λ [0, ], which implies that f (λy +( λ)x) λf (y)+( λ)f (x) f (x + λ(y x)) f (x) f (y) f (x). λ 4

15 Letting λ 0, we obtain: f (x) t (y x) f (y) f (x), establishing the only if part. Now, suppose that the gradient inequality holds for all x, y S. Let w and z be any two points in S. Let λ [0, ], and set x = λw + ( λ)z. Then f (w) f (x)+ f (x) t (w x) and f (z) f (x)+ f (x) t (z x). Taking a convex combination of the above inequalities, we obtain λf (w)+( λ)f (z) f (x)+ f (x) t (λ(w x)+( λ)(z x)) = f (x)+ f (x) t 0 = f (λw +( λ)z), which shows that f (x) is convex. Theorem 9 Suppose S is a non-empty open convex set, and f (x) : S R is twice differentiable. Let H (x) denote the Hessian of f (x). Then f (x) is convex if and only if H(x) is positive semidefinite for all x S. Proof: Suppose f (x) is convex. Let x S and d be any direction. Then for λ > 0 sufficiently small, x + λd S. We have: f ( x + λd) = f ( x)+ f ( x) t (λd)+ (λd) t H( x)(λd)+ λd α( x, λd), where α( x, y) 0as y 0. Using the gradient inequality, we obtain λ d t H ( x)d + d α( x, λd) 0. Dividing by λ > 0 and letting λ 0, we obtain d t H ( x)d 0, proving the only if part. 5

16 Conversely, suppose that H(z) is positive semidefinite for all z S. Let x, y S be arbitrary. Invoking the second-order version of Taylor s theorem, we have: f(y) = f(x)+ f(x) t (y x)+ (y x) t H(z)(y x) for some z which is a convex combination of x and y (and hence z S). Since H(z) is positive semidefinite, this means that f(y) f(x)+ f(x) t (y x). Therefore the gradient inequality holds, and hence f(x) is convex. Returning to the optimization problem (P), knowing that the function f(x) is convex allows us to establish a global optimality condition that is both necessary and sufficient: Theorem 0 Suppose f(x) : X Ris convex and differentiable on X. Then x X is a global minimum if and only if f( x) =0. Proof: The necessity of the condition f( x) = 0 was established in Corollary 4 regardless of the convexity of the function f(x). Suppose f( x) = 0. Then by the gradient inequality we have f(y) f( x) t (y x) = f( x is a global minimum. x)+ f( x) for all y X, and so Example 9 Continuing Example 5, recall that H(x) =. 4 6x Suppose that the domain of of f( ) is X = {(x,x ) x < 0}. Then f( ) is a convex function on this domain. Example 0 Let f(x) = ln( x x ) ln x ln x. 6

17 Then x x x f(x) =, x x x and x x + x x x H(x) =. x x x x + x It is actually easy to prove that f(x) is a strictly convex function, and hence that H(x) is positive definite on its domain X = {(x,x ) x > 0,x > 0,x x = 3, + x < }. At 3 we have f( x) =0,and so x is the unique global minimum of f(x). 3 Exercises on Unconstrained Optimization. Find points satisfying necessary conditions for extrema (i.e., local minima or local maxima) of the function x + x f(x) = 3+ x. + x + x x Try to establish the nature of these points by checking sufficient conditions.. Find minima of the function f(x) =(x x ) among all the points satisfying necessary conditions for an extremum. 3. Consider the problem to minimize Ax b, where A is an m n matrix and b is an m vector. a. Give a geometric interpretation of the problem. b. Write a necessary condition for optimality. Is this also a sufficient condition? 7

18 c. Is the optimal solution unique? Why or why not? d. Can you give a closed form solution of the optimal solution? Specify any assumptions that you may need. e. Solve the problem for A and b given below: 0 0 A =, b = Let S be a nonempty set in R n. Show that S is convex if and only if for each integer k the following holds true: x,...,x k S k λ j x j S j= k whenever λ,...,λ k satisfy λ,...,λ k 0and λ j =. j= 5. Bertsekas, Exercise.., page 6. (Note: x is called a stationary point of f( ) if f(x ) = 0.) 6. Bertsekas, Exercise.., page 6, parts (a), (b), (c), and (d). (Note: x is called a stationary point of f( ) if f(x ) = 0.) 8