Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Transcription

1 (für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11

2 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Problem Statement Nonlinear Programming We consider the problem where min f(x) x X f : R n R is a continuous (and usually differentiable) function of n variables x R n X = R n or (more generally) X is a subset of R n. If X = R n, the problem is called unconstrained If f is linear and X is polyhedral, the problem is a linear programming problem. Otherwise it is a nonlinear programming problem.

3 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Problem Statement Constrained Inequality Constrained Problem We consider the problem min x R n f(x) subject to h(x) = 0, g(x) 0 where f : R n R, h : R n R m, and g : R n R r are continuously differentiable functions. Here h = (h 1, h 2,..., h m ) are the equality constraints, and g = (g 1, g 2,..., g r ) are the inequality constraints.

4 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Topics Covered Unconstrained Optimization Derivative-Free Optimization Gradient Methods Newton s Method and Variations Least-Squares Problems Conjugate Gradient Method Constrained Optimization Conditional Gradient Method Gradient Projection Method Penalty and Augmented Lagrangian Methods Interior-Point Methods

5 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions LOCAL AND GLOBAL MINIMA Local and Global Minima f(x) Strict Local Minimum Local Minima Strict Global Minimum Quelle: Bertsekas Unconstrained local and global minima in one dimension Unconstrained IGPM, local RWTH and Aachen global Numerisches minima Rechnen in one dimension. x

6 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Necessary Optimality Conditions First Order Necessary Conditions Let x be an unconstrained local minimum of f : R n R, and assume that f is continuously differentiable in an open neighbourhood of x, then f(x ) = 0. Second Order Necessary Conditions Let x be an unconstrained local minimum of f : R n R, and assume that f is twice continuously differentiable in an open neighbourhood of x, then and f(x ) = 0 2 f is positive semidefinite.

7 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Sufficient Optimality Conditions Second Order Sufficient Conditions Let f : R n R be twice continuously differentiable in an open neighbourhood of x and suppose that x satisfies the conditions and f(x ) = 0 2 f is positive definite. Then x is a strict unconstrained local minimum of f. In particular, there exists scalars γ > 0 and ɛ > 0 such that f(x) f(x ) + γ 2 x x 2, x with x x < ɛ.

8 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Problem Statement Constrained Optimization Problem We consider the problem where min f(x) x X X R n is nonempty, convex, and closed f : R n R is a continuously differentiable function over X.

9 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Problem Statement Constrained Optimization Problem We consider the problem where min f(x) x X X R n is nonempty, convex, and closed f : R n R is a continuously differentiable function over X. Proposition If f is a convex function, then a local minimum of f over X is a global minimum. If in addition f is strictly convex over X, then there exists at most one global minimum of f over X.

10 also sufficient for x Proposition (Optimality Condition) to Constrained Optimization Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Necessary (a) and If x Constraint set X Sufficient is a local Conditions minimum of f over X, then Optimality Conditions f(x ) (x x ) 0, x X. (a) If x is a local minimum of f over X, then f(x ) T (x x ) 0, x X. (b) If f is convex over X, then this condition is also sufficient for x to minimize f over X. Surfaces of equal cost f(x) (b) If f is convex over X, then the condition of part (a) is also sufficient for x to minimize f over X. x!f(x * ) x * At the an a to 9 sibl X. x Constraint set X!f(x * ) x * Surfaces of equal cost f(x) Constraint set X At a local!f(x * ) minimum x, the gradient f (x ) makes x an angle less * than or equal to 90 degrees with x all feasible variations x x, x X. Quelle: Bertsekas Illu opt X i is a f the

11 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Optimization Subject to Bounds Consider a positive orthant constraint X = {x x 0}. The necessary optimality condition for x = (x 1,..., x n ) to be a local minimum is n f(x ) (x i x i x ) 0, x i 0, i = 1,..., n. i i=1 Consider two cases: Fix i. Let x j = x j for j i and x i = x i + 1: f(x ) x i 0, i. If x i > 0, let also x j = x j for j i and x i = 1 2 x i. Then f(x ) x i 0, so f(x ) x i = 0, if x i > 0.

12 0, i. Constrained Optimization Problem Statement Optimization over x a Convex i Set Unconstrained Optimality Conditions Constrained Optimality Conditions If x i > 0, let also x j = x j for j i and x i = 1 2 x i. Optimization Then f(xsubject )/ x i to Bounds 0, so Optimality conditions for an orthant constraint: at a minimum, all partial derivatives f(x ) x i are nonnegative, and they are zero for the inactive constraint indices, i.e., the indices with x i > 0. x i =0, if x i > 0. f(x * ) f(x * ) x * x* = 0 Quelle: Bertsekas Note: if all constraints are inactive, we obtain the unconstrained optimality condition f(x ) = 0.

13 Problem Statement Unconstrained Optimality Conditions Constrained Optimality Conditions Optimization Subject to Bounds Consider the constraints X = {x α i x i β i, i = 1,..., n} where α i and β i are scalars. If x is a local minimum, then f(x ) x i 0, if x i = α i, f(x ) x i 0, if x i = β i, f(x ) x i = 0, if α i < x i < β i.

14 Conditional Gradient Method Constrained Optimization Gradient Projection Methods Feasible Direction Methods Feasible Directions Conditional Gradient Method Gradient Projection Method A feasible direction at an x X is a vector d 0 such that x + αd is feasible for all suff. small α > 0 A feasible direction at an x X is a vector d 0 such that x + αd is feasible for all sufficiently small α > 0. x 2 Feasible directions at x x Constraint set X d x 1 Quelle: Bertsekas Note: the set of feasible directions at x is the set of all α(z x) Note: the set of feasible directions at x is the where z X, z x, and α > 0. set of all α(z x) where z X, z x, and α > 0

15 Feasible Directions Conditional Gradient Method Gradient Projection Method Feasible Direction Methods A feasible direction method x k+1 = x k + α k d k where d k is a feasible descent directions ( f(x k ) T d k < 0), and α k > 0 and such that x k+1 X. Alternative definition x k+1 = x k + α k ( x k x k ) where α k (0, 1] and if x k is nonstationary, x k X, f(x k ) T ( x k x k ) < 0. Stepsize rules: Limited minimization, constant α k = 1, Armijo rule.

16 Conditional Gradient Method Feasible Directions Conditional Gradient Method Gradient Projection Method x k+1 = x k + α k (x k Iteration where x k+1 = x k + α k ( x k x k ) x k = arg min x X f(xk ) T (x x k ) x k = arg m x Assume that X is c to exist by Weierstra x k is a point in X that lies "furthest along" the negative gradient direction f(x k ). Subproblem simpler to solve than original if f is nonlinear X specified by linear equality or inequality constraints Linear Programm f(x) Constraint set X x _ x Surfaces of equal cost Quelle: Bertsekas

17 Constraint Feasible set X Directions x Conditional Gradient Method Conditional Gradient Method Gradient Projection Method Operation of the method with limited minimization stepsize Surfaces of equal cost rule. Possibly slow convergence _ x Illustration of the d of the conditional g method. Constraint set X _ x 1 x 0 x 1 x 2 x* _ x 0 Operation of the m Slow (sublinear) co Surfaces of equal cost Quelle: Bertsekas

18 Feasible Directions Conditional Gradient Method Gradient Projection Method Gradient Projection Method Gradient projection methods determine the feasible direction by using a quadratic cost subproblem Simplest variant: where x k+1 = x k + α k ( x k x k ) x k = [x k s k f(x k )] + and [ ] + denotes the projection on the set X, α k (0, 1] is a stepsize, and s k is a positive scalar Stepsize rules for α k (assuming s k s): Limited minimization, Armijo along the feasible direction, constant stepsize. Also, Armijo along the projection arc (α k 1, s k : variable).

19 Gradient Projection Method Feasible Directions Conditional Gradient Method Gradient Projection Method Illustration of gradient projection method for the case where α k = 1 for all k, and thus x k+1 = x k = [x k s k f(x k )] +. x k+1 = x k + α k (x k x k ) x k = [ x k s k f(x k ) ] + where, [ ] + denotes projection on the se (0, 1] is a stepsize, and s k is a positive s x k+1 = x k - s k f(x k ) x k+2 - s k+2 f(x k+2 ) Constraint set X x k+3 x k+2 x k+1 x k x k+1 - s k+1 f(x k+1 ) Gradient projectio tions for the case α k 1, x k+1 If α k < 1, x k+1 i line segment conne and x k. Quelle: Bertsekas Note: if x k X we obtain the unconstrained steepest descent iteration. Stepsize rules for α k (assuming s k s minimization, Numerisches Armijo Rechnen along the feasible

20 Feasible Directions Conditional Gradient Method Gradient Projection Method Gradient Projection Method For practical purposes, the projection operation should be fairly simple. Example: Constraints are bounds on the variables, X = {x α i x i β i, i = 1,..., n} where α i and β i are scalars. The ith component of the projection of a vector x is given by α i if x i α i, [x] + i = β i if x i β i, otherwise. x i

21 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Lagrangian Function Equality Constrained Problem We consider the problem min x R n f(x) subject to h(x) = 0, where f : R n R and h : R n R m are continuously differentiable functions.

22 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Lagrangian Function Equality Constrained Problem We consider the problem min x R n f(x) subject to h(x) = 0, where f : R n R and h : R n R m are continuously differentiable functions. We then define the Lagrangian function L : R n R m R given by L(x, λ) = f(x) + m λ i h i (x) i=1 where the scalars λ 1,..., λ m are the Lagrange multipliers.

23 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Lagrange Multiplier Theorem Lagrange Multiplier Theorem Necessary Conditions Let x be a local minimum of f subject to h(x) = 0, and assume that the constraint gradients h 1 (x ),..., h m (x ) are linearly independent. Then there exists a unique vector λ = (λ 1,..., λ m ) called a Lagrange multiplier vector, such that x L(x, λ ) = f(x ) + m λ i h i(x ) = 0 and i=1 λ L(x, λ ) = 0. If in addition f and h are twice continuously differentiable, we have y T 2 xx y 0, y V (x ), where V (x ) is the subspace of first order feasible variations V (x ) = {y h i (x ) T y = 0, i = 1,..., m}.

24 If in addition f and h are twice cont. differentiable, ( ) Constrained Optimization m Lagrange y Multiplier 2 f (x )+ Theorem λ Example x 2 i=1 i 2 h i (x ) Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods y 0, y s.t. h(x ) y = 0 2 h(x) = 0 minimize x 1 + x 2 f(x * ) = (1,1) 0 x * = (-1,-1) 2 x 1 subject to x x 2 =2. The Lagrange multiplier is λ = 1/2. h(x * ) = (-2,-2) x 2 h 2 (x) = 0 minimize x 1 + x 2 h 2 (x * ) = (-4,0) h 1 (x * ) = (-2,0) f(x * ) = (1,1) 1 2 h 1 (x) = 0 x 1 2 s. t. (x 1 1) 2 + x 2 1 =0 2 (x 1 2) 2 + x 2 4 =0 Quelle: Bertsekas

25 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Lagrange Multiplier Theorem Lagrange Multiplier Theorem Sufficient Conditions Assume that f and h are twice continuously differentiable, and let x R n and λ R m satisfy x L(x, λ ) = 0, λ L(x, λ ) = 0, y T 2 xx y 0, y 0, y V (x ). Then x is a strict local minimum of f subject to h(x) = 0. In fact, there exists scalars γ > 0 and ɛ > 0 such that f(x) f(x ) + γ 2 x x 2, x with h(x) = 0 and x x < ɛ.

26 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Lagrange Multiplier Theorem Lagrange Multiplier Theorem Sufficient Conditions Assume that f and h are twice continuously differentiable, and let x R n and λ R m satisfy x L(x, λ ) = 0, λ L(x, λ ) = 0, y T 2 xx y 0, y 0, y V (x ). Then x is a strict local minimum of f subject to h(x) = 0. In fact, there exists scalars γ > 0 and ɛ > 0 such that f(x) f(x ) + γ 2 x x 2, x with h(x) = 0 and x x < ɛ. Approach can be extended to treat both equality and inequality constraints Karush-Kuhn-Tucker (KKT) necessary optimality conditions More general: Fritz John optimality conditions

27 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Lagrangian Function Inequality Constrained Problem We consider the problem min x X f(x) subject to g(x) 0, where f : R n R and g : R n R r are continuously differentiable functions and X is a closed set. The interior of the set is defined by S = {x X g j (x) < 0, j = 0,..., r}. We assume that S is nonempty and any feasible point is in the closure of S.

28 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Barrier Method Consider a barrier function, that is continuous and goes to as any one of the constraints g j (x) approaches 0 from negative values. The two most common examples are B(x) = r ln ( g j (x)), Barrier Method: j=1 B(x) = r j=1 1 g j (x), logarithmic inverse x k = arg min x S {f(x) + ɛk B(x)}, k = 0, 1,... where the parameter sequence {ɛ k } satisfies 0 < ɛ k+1 < ɛ k for all k and ɛ k 0.

29 Barrier Method { } Constrained r Optimization Lagrange Multiplier Theory r Barrier and Interior Point Methods 1 Penalty and Augmented Lagrangian Methods B(x) = ln g j (x), B(x) =. g j (x) j=1 j=1 Barrier Method: Barrier term ɛ k B(x) { goes to zero } for all interior points x S as ɛ k x k = arg min f (x)+ 0 k B(x), k = 0, 1,..., x S Every limit point of a sequence {x k } generated by a barrier method where the is a global parameter minimum sequence of the original { k } satisfies contrained0 < problem. k+1 < k for all k and k 0. ε B(x) ε' B(x) ε' < ε Boundary of S Boundary of S S Quelle: Bertsekas

30 Decrease faster than dictated by complexity Optimization analysis. over a Convex Set Barrier and Interior Point Methods Constrained Optimization Lagrange Multiplier Theory Penalty and Augmented Lagrangian Methods Require more than one Newton step per (approximate) Long-step minimization. methods Short-step and Use line search as in unconstrained Newton s method. Require much smaller number of (approximate) minimizations. Following approximately the central path by decreasing ɛ k slowly as in (a) or quickly as in (b). In (a) a single Newton step is required in each approximate minimization at the expense of a large number of approximate minimizations. x * x * Central Path Central Path x k+2 x(ε k+2 ) x k+1 x(ε k+1 ) x k x(ε k ) x S x k+2 x(εk+2 ) x k+1 x(ε k+1 ) x k x(ε k ) x S (a) (b) Quelle: Bertsekas

31 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Quadratic Penalty Method Equality Constrained Problem We consider the problem min x X f(x) subject to h(x) = 0, where f : R n R and h : R n R m are continuously differentiable functions, and X R n.

32 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Quadratic Penalty Method Equality Constrained Problem We consider the problem min x X f(x) subject to h(x) = 0, where f : R n R and h : R n R m are continuously differentiable functions, and X R n. We then define the augmented Lagrangian function L c : R n R m R given by L c (x, λ) = f(x) + λ T h(x) + c 2 h(x) 2 where c is a positive penalty parameter.

33 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Two Convergence Mechanisms Unconstrained minimization of L c (, λ) can yield points close to x by: Taking λ close to λ. For c sufficiently large, x is a strict local minimum of the augmented Lagragian L c (, λ ) corresponding to λ, i.e., L c (x, λ ) L c (x, λ ) + γ 2 x x 2, for all x with x x < ɛ, and for some γ > 0 and ɛ > 0. Taking c very large. For high c, there is a high cost for infeasibility, so the unconstrained minima of L c (, λ) will be nearly feasible. We have { f(x) if x X and h(x) = 0, L c (, λ) otherwise.

34 EXAMPLE CONTINUED Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Example 2 2 min x 1 + x 2, x =1, λ = 1 x 1 =1 Problem min 1 2 (x2 1 + x2 2 ) s.t. x 1 = 1 Augmented Lagrangian L c (x, λ) = 1 2 (x2 1 + x2 2 ) +λ(x 1 1) + c 2 (x 1 1) 2 Unconstrained Minimum x 1 (λ, c) = c λ c+1 x 2 (λ, c) = 0 For c > 0 (λ = 1): lim λ λ x 1(λ, c) = 1 = x 1 lim λ λ x 2(λ, c) = 0 = x 2 x 2 c = 1 λ = 0 1/2 0 1 x 1 x 2 0 1/2 c = 1 λ = 0 1 x 2 x 2 c = 1 λ = - 1/2 3/4 0 1 x 1 c = 10 λ = 0 x 1 0 x 10/ Quelle: Bertsekas

35 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods Multiplier Methods The multiplier method finds x k = arg min x R n L c k(x, λk ) f(x) + (λ k ) T h(x) + ck 2 h(x) 2 and update λ k using Key advantages λ k+1 = λ k + c k h(x k ) Less ill-conditioning: it is not necessary that c k (only that c k exceeds some threshold) Faster convergence when λ k is updated than when λ k is kept constant (wether c k or not)

36 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods The End

37 Lagrange Multiplier Theory Barrier and Interior Point Methods Penalty and Augmented Lagrangian Methods The End Ich danke Ihnen für die Aufmerksamkeit und wünsche Ihnen viel Glück bei der Prüfung und im weiteren Studium. Bei Fragen, Kommentaren,... : Tel.: grepl@igpm.rwth-aachen.de