Convex Optimization 3. Convex Functions

Transcription

1 Convex Optimization 3. Convex Functions Goele Pipeleers KU Leuven, Feb. 9-13, 2015

2 Overview definition properties of convex functions Jensen s inequality restriction to line first-order condition second-order condition epigraph operations that preserve convexity calculus of convex functions quasiconvex functions convexity with respect to generalized inequalities

3 Definition function f : R n R is convex if dom f is convex and f (θx 1 + (1 θ)x 2 ) θf (x 1 ) + (1 θ)f (x 2 ) for all x 1, x 2 dom f, 0 θ 1 f (x) f (x 2) f (x 1) x 1 x 2 x function f is concave if f is convex

4 Examples on R convex: affine: ax + b on R, for any a, b R exponential: e ax, for any a R powers: x p on R ++, for p 1 or p 0 powers of absolute value: x p on R, for p 1 negative entropy: x log x on R ++ concave: affine: ax + b on R, for any a, b R powers: x p on R ++, for 0 p 1 logarithm: log x on R ++

5 Examples on R n and R m n affine functions are convex and concave; all norms are convex examples on R n affine function: f (x) = a T x + b ni=1 norms: x p = p x i p for p 1; x = max x i i examples on R n m n n affine function: f (X) = tr(a T X) + b = A ij X ij + b i=1 j=1 spectral norm: f (X) = X 2 = σ max (X) = λ max (X T X)

6 extended-value extension f of f is Extended-value extension f (x) = f (x), for x dom f ; f (x) =, for x / dom f often simplifies notation: f is convex if for all x 1, x 2 R n, 0 θ 1 f (θx 1 + (1 θ)x 2 ) θ f (x 1 ) + (1 θ) f (x 2 )

7 Overview definition properties of convex functions Jensen s inequality restriction to line first-order condition second-order condition epigraph operations that preserve convexity calculus of convex functions quasiconvex functions convexity with respect to generalized inequalities

8 Jensen s inequality basic inequality: if f is convex, then f (θ 1 x 1 + θ 2 x 2 ) θ 1 f (x 1 ) + θ 2 f (x 2 ) for all x 1, x 2 dom f, θ 1, θ 2 R +, θ 1 + θ 2 = 1 extension to arbitrary convex combinations: if f is convex, then for any random variable z f (Ez) Ef (z) basic inequality is special case with discrete distribution prob(z = x 1 ) = θ 1, prob(z = x 2 ) = θ 2

9 Restriction of a convex function to a line f : R n R is convex if and only if the function g : R R g(t) = f (x + tv), dom g = {t x + tv dom f } is convex for any x dom f, v R n check convexity of f by checking convexity of functions of one variable example: f : S n R with f (X) = log det X on dom f = S n ++ g(t) = log det(x + tv ) = log det X + log det(i + tx 1/2 VX 1/2 ) n = log det X + log(1 + tλ i ) where λ i are the eigenvalues of X 1/2 VX 1/2 g is concave in t for any choice of X 0, V ; hence f is concave i=1

10 First-order condition f is differentiable if dom f is open and the gradient f (x) R n ( ) f (x) f (x) f (x) f (x) =,,..., x 1 x 2 x n exists at each x dom f 1st-order condition: differentiable f with convex domain is convex iff f (x) f (x 0 ) + f (x 0 ) T (x x 0 ), first-order approximation of f is global underestimator f (x) for all x 0, x dom f f (x 0) f (x 0 )+ f (x 0 ) T (x x 0 ) x 0 x

11 Second-order condition f is twice differentiable if dom f is open and the Hessian 2 f (x) S n exists at each x dom f 2 f (x) ij = 2 f (x) x i x j, i, j = 1,..., n 2nd-order condition: twice differentiable f with convex domain is convex iff 2 f (x) 0, for all x dom f

12 Examples quadratic function: f (x) = (1/2)x T Px + q T x + r, with P S n + is convex f (x) = Px + q, 2 f (x) = P least-squares objective: f (x) = Ax b 2 is convex f (x) = 2A T (Ax b), 2 f (x) = 2A T A quadratic-over-linear: f (x, y) = x 2 /y on R R ++ is convex 2 f (x) = 2 y 3 [ y x ] [ y x ] T 0 2 f y x

13 geometric mean: f (x) = ( n i=1 x i ) 1/n on R n ++ is concave [ ] [ ] T 2 1 x2 x2 f (x 1, x 2 ) = 4(x 1 x 2 ) 3/2 0 x 1 x 1 log-sum-exp: f (x) = log n i=1 exp(x i ) is convex 2 f (x 1, x 2 ) = e x 1 e x [ ] [ ] T (e x e x 2) f x x 1 2

14 Epigraph epigraph of f : R n R: epi f = {(x, t) R n+1 x dom f, f (x) t} epi f f f is convex if and only if epi f is a convex set

15 1st-order condition for convexity: t f (x) f (x 0 ) + f (x 0 ) T (x x 0 ) for all x 0 dom f and (x, t) epi f ; hence [ ] T ([ ] [ ]) f (x0 ) x x0 1 t f (x 0 ) 0 epi f (x 0, f (x 0)) ( f (x 0), 1)

16 Overview definition properties of convex functions Jensen s inequality restriction to line first-order condition second-order condition epigraph operations that preserve convexity calculus of convex functions quasiconvex functions convexity with respect to generalized inequalities

17 Operations that preserve convexity analyzing convexity of a function verify definition often simplified by restricting to a line use properties 2nd-order condition epigraph calculus of convex functions: obtain f from simple convex functions by operations that preserve convexity nonnegative weighted sum composition with affine function pointwise maximum and supremum partial minimization perspective composition

18 Weighted sum & composition with affine function for convex functions f, f 1, f 2 nonnegative multiple: αf with α 0 is convex sum: f 1 + f 2, is convex; extends to infinite sums, integrals composition with affine function: f (Ax + b) is convex examples log barrier for linear inequalities m f (x) = log(b i ai T x), dom f = {x ai T x < b i, i = 1,..., m} i=1 any norm of affine function: f (x) = Ax + b

19 Pointwise maximum if f 1,..., f m are convex, then f (x) = max{f 1 (x),..., f m (x)} is convex epi f is intersection of epi f i, i = 1,..., m f epi f 1 epi f epi f 2 x

20 examples piecewise-linear function: f (x) = sum of r largest components of x R n : is convex since max i=1,...,m {at i x + b i } is convex f (x) = x [1] + x [2] + + x [r] f (x) = max{x i1 + x i2 + + x ir 1 i 1 < i 2 < < i r n}

21 Pointwise supremum if f (x, y) is convex in x for each y C, then is convex g(x) = sup f (x, y) y C examples support function of a set C: S C (x) = sup y T x is convex y C distance to farthest point in a set C: f (x) = sup x y y C maximum eigenvalue of symmetric matrix: for X S n, λ max = sup y T Xy y 2 =1

22 Partial minimization if f (x, y) is convex in (x, y) and C is a convex set, then is convex g(x) = inf f (x, y) y C epi g is projection of epi f {(x, y, t) y C} g f epi f epi g y C x x

23 examples f (x, y) = x T Ax + 2x T By + y T Cy with [ ] A B B T 0, C 0 C minimizing over y gives g(x) = inf y f (x, y) = x T (A BC 1 B T )x g is convex, hence Schur complement A BC 1 B T 0 distance to a set: dist(x, S) = inf x y is convex if S is convex y S

24 Perspective the perspective of a function f : R n R is the function g : R n R R, g(x, t) = tf (x/t), dom g = {(x, t) x/t dom f, t > 0} g is convex if f is convex epi g inverse image of epi f under perspective function g(x, t) s f (x/t) s/t

25 examples f (x) = x T x is convex; hence g(x, t) = x T x/t is convex for t > 0 negative logarithm f (x) = log x is convex; hence relative entropy g(x, t) = t log t t log x is convex on R 2 ++ if f is convex, then ( ) Ax + b g(x) = (c T x + d) f c T x + d is convex on {x c T x + d > 0, (Ax + b)/(c T x + d) dom f }

26 Composition with scalar function composition of g : R n R and h : R R: f (x) = h(g(x)) { g convex, h convex and nondecreasing f is convex if g concave, h convex and nonincreasing proof for n = 1, differentiable g, h f = h (g(x))g (x) 2 + h (g(x))g (x) note: monotonicity must hold for extended-value extension h examples exp g(x) is convex if g is convex 1/g(x) is convex if g is concave and positive

27 Vector composition composition of g : R n R k and h : R k R: f (x) = h(g(x)) = h(g 1 (x), g 2 (x),..., g k (x)) { gi convex, h convex, h nondecreasing in each argument f is convex if g i concave, h convex, h nonincreasing in each argument proof for n = 1, differentiable g, h f (x) = g (x) T 2 h(g(x))g (x) + h(g(x)) T g (x) examples m i=1 log g i (x) is concave if g i are concave and positive log m i=1 exp g i (x) is convex if g i are convex

28 Overview definition properties of convex functions Jensen s inequality restriction to line first-order condition second-order condition epigraph operations that preserve convexity calculus of convex functions quasiconvex functions convexity with respect to generalized inequalities

29 Quasiconvex functions α-sublevel set of f : R n R: C α = {x dom f f (x) α} sublevel sets of convex functions are convex (converse is false) f : R n R is quasiconvex if dom f is convex and the sublevel sets C α are convex for all α f (x) β α a b c x f is quasiconcave if f is quasiconvex f is quasilinear if it is quasiconvex and quasiconcave

30 Examples x is quasiconvex on R ceil(x) = inf{z Z z x} is quasilinear log x is quasilinear on R ++ f (x 1, x 2 ) = x 1 x 2 is quasiconcave on R 2 ++ linear-fractional function is quasilinear distance ratio is quasiconvex f (x) = at x + b c T x + d, dom f = {x ct x + d > 0} f (x) = x a 2 x b 2, dom f = {x x a 2 x b 2 }

31 Convex representation of sublevel sets if f is quasiconvex, there exists a family of functions φ α such that: φ α (x) is convex in x for fixed α α-sublevel set of f is 0-sublevel set of φ α, i.e., f (x) α φ α (x) 0 examples: f (x) = x φ α (x) = x α 2 f (x) = ceil(x) φ α (x) = x floor(α) f (x 1, x 2 ) = x 1 x 2 φ α (x) = x 1 α x 2 (α < 0) f (x) = at x + b c T x + d φ α(x) = (a T x + b) α(c T x + d)

32 Convexity with respect to generalized inequalities f : R n R m is K-convex if dom f is convex and for x, y dom f, 0 θ 1 f (θx + (1 θ)y) K θf (x) + (1 θ)f (y) examples R m +-convexity means componentwise convexity f : S m S m, f (X) = X 2 is S m +-convex for fixed z R m, z T X 2 z = Xz 2 is convex in X, i.e., z T (θx + (1 θ)y ) 2 z θz T X 2 z + (1 θ)z T Y 2 z for X, Y S m, 0 θ 1 therefore (θx + (1 θ)y ) 2 θx 2 + (1 θ)y 2

33 Overview definition properties of convex functions Jensen s inequality restriction to line first-order condition second-order condition epigraph operations that preserve convexity calculus of convex functions quasiconvex functions convexity with respect to generalized inequalities