Lecture 3. Convex Functions. September 2, 2008

Transcription

1 Convex Functions September 2, 2008

2 Outline Lecture 3 Convex Functions Examples Verifying Convexity of a Function Operations on Functions Preserving Convexity Convex Optimization 1

3 Convex Functions Informally: f is convex when for every segment [x 1, x 2 ], as x α = αx 1 +(1 α)x 2 varies over the line segment [x 1, x 2 ], the points (x α, f(x α )) lie below the segment connecting (x 1, f(x 1 )) and (x 2, f(x 2 )) Let f be a function from R n to R, f : R n R The domain of f is a set in R n defined by dom(f) = {x R n f(x) is well defined (finite)} Def. A function f is convex if (1) Its domain dom(f) is a convex set in R n and (2) For all x 1, x 2 dom(f) and α (0, 1) f(αx 1 + (1 α)x 2 ) αf(x 1 ) + (1 α)f(x 2 ) Convex Optimization 2

4 More on Convex Function Def. A function f is strictly convex when dom(f) is convex and f(αx 1 + (1 α)x 2 ) < αf(x 1 ) + (1 α)f(x 2 ) for all x 1, x 2 dom(f) and α (0, 1) Def. A function f is concave when f is convex, i.e., (1) Its domain dom(f) is a convex set in R n and (2) For all x 1, x 2 dom(f) and α (0, 1) f(αx 1 + (1 α)x 2 ) αf(x 1 ) + (1 α)f(x 2 ) Def. A function f is strictly concave when f is strictly convex Convex Optimization 3

5 Examples on R Convex: Affine: ax + b over R for any a, b R Exponential: e ax over R for any a R Power: x p over (0, + ) for p 1 or p 0 Powers of absolute value: x p over R for p 1 Negative entropy: x ln x over (0, + ) Concave: Affine: ax + b over R for any a, b R Powers: x p over (0, + ) for 0 p 1 Logarithm: ln x over (0, + ) Convex Optimization 4

6 Examples: Affine Functions and Norms Affine functions are both convex and concave Norms are convex Examples on R n Affine function f(x) = a x + b with a R n and b R Euclidean, l 1, and l norms General l p norms x p = ( n i=1 x i p ) 1/p for p 1 Convex Optimization 5

7 Examples on R m n The space R m n is the space of m n matrices Affine function f(x) = tr(a T X) + b = m n i=1 j=1 a ij x ij + b Spectral (maximum singular value) norm f(x) = X 2 = σ max (X) = λ max (X T X) where λ max (A) denotes the maximum eigenvalue of a matrix A Convex Optimization 6

8 Verifying Convexity of a Function We can verify that a given function f is convex by Using the definition Applying some special criteria Second-order conditions First-order conditions Reduction to a scalar function Showing that f is obtained through operations preserving convexity Convex Optimization 7

9 Second-Order Conditions Let f be twice differentiable and let dom(f) = R n required that dom(f) is open] [in general, it is The Hessian 2 f(x) is a symmetric n n matrix whose entries are the second-order partial derivatives of f at x: [ 2 f(x) ] ij = 2 f(x) x i x j for i, j = 1,..., n 2nd-order conditions: For a twice differentiable f with convex domain f is convex if and only if 2 f(x) 0 for all x dom(f) f is strictly convex if 2 f(x) 0 for all x dom(f) Convex Optimization 8

10 Examples Quadratic function: f(x) = (1/2)x P x + q x + r with a symmetric n n matrix P f(x) = P x + q, 2 f(x) = P Convex for P 0 Least-squares objective: f(x) = Ax b 2 with an m n matrix A Convex for any A f(x) = 2A T (Ax b), Quadratic-over-linear: f(x, y) = x 2 /y 2 f(x) = 2A T A Convex for y > 0 2 f(x, y) = 2 y 3 [ y x ] [ y x ] T 0 Convex Optimization 9

11 Verifying Convexity of a Function We can verify that a given function f is convex by Using the definition Applying some special criteria Second-order conditions First-order conditions Reduction to a scalar function Showing that f is obtained through operations preserving convexity Convex Optimization 10

12 First-Order Condition f is differentiable if dom(f) is open and the gradient f(x) = ( f(x) x 1, f(x),..., f(x) ) x 2 x n exists at each x domf 1st-order condition: differentiable f is convex if and only if its domain is convex and f(x) + f(x) T (z x) f(z) for all x, z dom(f) A first order approximation is a global underestimate of f Very important property used in algorithm designs and performance analysis Convex Optimization 11

13 Restriction of a convex function to a line f is convex if and only if domf is convex and the function g : R R, g(t) = f(x + tv), domg = {t x + tv dom(f)} is convex (in t) for any x domf, v R n Checking convexity of multivariable functions can be done by checking convexity of functions of one variable Example f : S n R with f(x) = ln det X, domf = S n ++ g(t) = ln det(x + tv ) = ln det X ln det(i + tx 1/2 V X 1/2 ) = ln det X n i=1 ln(1 + tλ i ) where λ i are the eigenvalues of X 1/2 V X 1/2 g is convex in t (for any choice of V and any X 0); hence f is concave Convex Optimization 12

14 Operations Preserving Convexity Positive Scaling Sum Composition with affine function Pointwise maximum and supremum Composition Minimization Convex Optimization 13

15 Scaling, Sum, & Composition with Affine Function Lecture 3 Positive multiple For a convex f and λ > 0, the function λf is convex Sum: For convex f 1 and f 2, the sum f 1 + f 2 is convex (extends to infinite sums, integrals) Composition with affine function: For a convex f and affine g [i.e., g(x) = Ax + b], the composition f g is convex, where (f g)(x) = f(ax + b) Examples Log-barrier for linear inequalities f(x) = m i=1 ln(b i a T i x), domf = {x at i x < b i, i = 1,..., m} (Any) Norm of affine function: f(x) = Ax + b Convex Optimization 14

16 Pointwise maximum For convex functions f 1,..., f m, the pointwise-max function F (x) = max {f 1 (x),..., f m (x)} is convex (What is domain of F?) Examples Piecewise-linear function: f(x) = max i=1,...,m (a T i x + b i ) is convex Sum of r largest components of a vector x R n : f(x) = x [1] + x [2] + + x [r] is convex (x [i] is i-th largest component of x) f(x) = max {x i1 + x i2 + + x ir } (i 1,...,i r ) I r I r = {(i 1,..., i r ) i 1 <... < i r, i j {1,..., m}, j = 1,..., n} Convex Optimization 15

17 Pointwise Supremum Let A R p and f : R n R p R. Let f(x, z) be convex in x for each z A. Then, the supremum function over the set A is convex: Examples g(x) = sup z A f(x, z) Set support function is convex for a set C R n, S C : R n R, S C (x) = sup z C z T x Set farthest-distance function is convex for a set C R n, f : R n R, f(x) = sup z C x z Maximum eigenvalue function of a symmetric matrix is convex λ max : S n R, λ max (X) = sup z =1 z T Xz Convex Optimization 16

18 Composition with Scalar Functions Composition of g : R n R and h : R R with dom(g) = R n dom(h) = R: f(x) = h(g(x)) f is convex if and (1) g is convex, h is nondecreasing and convex (2) g is concave, h is nonincreasing and convex Examples e g(x) is convex if g is convex 1 g(x) is convex if g is concave and positive Convex Optimization 17

19 Composition with Vector Functions Composition of g : R n R p and h : R p R with dom(g) = R n and dom(h) = R p : f(x) = h(g(x)) = h(g 1 (x), g 2 (x),..., g p (x)) f is convex if (1) each g i is convex, h is convex and nondecreasing in each argument (2) each g i is concave, h is convex and nonincreasing in each argument Example m i=1 e g i(x) is convex if g i are convex Convex Optimization 18

20 Extended-Value Functions A function f is an extended-value function if f : R n R {, + } Example: consider f(x) = inf y 0 xy for x R Def. The epigraph of a function f over R n is the following set in R n+1 : epif = {(x, w) R n+1 x R n, f(x) w} General Convex Function Def. epif is a convex set in R n+1 A function f is convex if its epigraph This definition is equivalent to the one we have used so far (when reduced to the function class we have considered thus far). How? For an f with domain domf, we associate an extended-value function f defined by f(x) if x domf f(x) = + otherwise domf is the projection of epif on R n ; convexity of f by letting w = f(x) Convex Optimization 19

21 Minimization Let C R n R p be a nonempty convex set Let f : R n R p R be a convex function [in (x, z) R n R p ]. Then g(x) = inf f(x, z) z C is convex Example Distance to a set: for a nonempty convex C R n, dist(x, C) = inf z C x z is convex Proof: Let x 1, x 2 R n and α (0, 1) be arbitrary. Let ɛ > 0 be arbitrarily small. Then, there exist z 1, z 2 C such that f(x 1, z 1 ) g(x 1 ) + ɛ and f(x 2, z 2 ) g(x 2 ) + ɛ. Consider f(αx 1 + (1 α)x 2, αz 1 + (1 α)z 2 ) and use convexity of f and C. Convex Optimization 20