Lecture Note 2: Convex function. Xiaoqun Zhang Shanghai Jiao Tong University

Transcription

1 Lecture Note 2: Convex function Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 10, 2016

2 2

3 Contents 1.1 Convex functions Elemental properties of convex function Operations that preserving convexity of functions Differential criteria of convexity Conjugacy Convex functions Definition on one variable: f defined on a nonempty interval I is said to be convex on I if f(αx + (1 α)x ) αf(x) + (1 α)f(x ) for all pairs of points (x, x ) in I and all α (0, 1). Criterion of increasing slopes: A function f is convex on an interval I iff for all x 0 I, the slope-function x f(x) f(x0) x x 0 =: s(x) is increasing on I/{x 0 }. Proof. Let x 0 = αx 1 + (1 α)x 2. By convexity, we have αf(x 0 ) + (1 α)f(x 0 ) αf(x 1 ) + (1 α)f(x 2 ), then s(x 1 ) s(x 2 ). Let a function f be differentiable with an increasing derivative on an open interval I, then f is convex on I. Let a function f be twice differentiable with a non-negative second derivative on an open interval I. Then f is convex on I. Definition 1 A function f : C R, where C R n and C, is convex if C is convex; For every x, y C and every λ [0, 1] one has f(λx + (1 λ)y) λf(x) + (1 λ)f(y) More definitions: 3

4 Concave: if f is convex, f is called concave. Strictly convex: if the inequality is strict whenever x y, and 0 < λ < 1, f is called strictly convex. Strongly convex: if there exist c > 0 s.t. f(αx + (1 α)x ) αf(x) + (1 α)f(x ) 1 2 cα(1 α) x x 2 for all x, x C and α (0, 1). We say that f is strongly convex on C with modulus of strong convexity c. Theorem 1 The function f is strongly convex on C with modulus c if and only if the function f 1 2 c 2 is convex on C. Extended value of f. Define { f(x), x dom(f); f = +, x / dom(f). 1. For all x, y R n, all α [0, 1], f(αx + (1 α)y) α f(x) + (1 α) f(y). 2. Arithmetic operations involving : (+ ) + (+ ) = + ; ( ) + ( ) = ; (+ ) + ( ) undefined. (+ ) (± ) = ± ; The domain (or effective domain) of f is the nonempty set such as f(x) <, denoted as dom(f) = {x C f(x) < } and the epigraph f is a subset in R n R defined as epi(f)(x) = {(x, t) R n+1 : x dom(f); f(x) t}. Strict epigraph: is replaced by <. Theorem 2 Let f : R n R {+ } be not identically equal to +. The three properties below are equivalent: 1. f is convex ( in the sense of definition ). 2. its epigraph epi(f) is a convex set in R n R. 3. its strict epigraph is a convex set in R n R. Proof. only if : If f is convex, f(λx + (1 λ)y) λf(x) + (1 λ)f(y). let (x, t) epi(f) and (y, t 2 ) epi(f). then f(λx + (1 λ)y) λf(x) + (1 λ)f(y) λt 1 + (1 λ)t 2. Thus (λx + (1 λ)y, λt 1 + (1 λ)t 2 ) epi(f). It yields epi(f) is convex. if part : if epi(f) is convex, for (x, f(x)) epi(f) and (y, f(y)) epi(f), f(λx + (1 λ)y) λf(x) + (1 λ)f(y)). Thus f is convex. 4

5 The sub level sets S r (f) := {x R n : f(x) r} of a (proper) convex function f are convex (possibly empty). Conversely it is not necessarily true (such a function is called quasi-convex). Closed convex function: If we want to minimize a function f on some compact set K, we do not need to bother with existence if f is known to be closed (or l.s.c) and this holds even if K is not contained in dom(f). Lower semi-continuous (ls.c): a function f is l.s.c if, for each x R n, lim inf f(y) f(x). y x Proposition 1 For f : R n R {+ }, the following three properties are equivalent: f is l.s.c on R n ; epi(f) is a closed set in R n ; the sublevel-sets S r (f) are closed (possibly empty) for all r R. Definition 2 The function f : R n R { } is said to be closed if it is l.s.c everywhere, or if its epigraph is closed, or if its sublevel-sets are closed. Examples of convex functions: 1. Example on R: Linear function f(x) = ax + b e ax for any a R. x α on R ++ for α 1 or α 0. (f (x) = αx α 1 and f (x) = α(α 1)x α 2. x p on R for p 1. x ln x on R ++ (negative entropy). Some concave functions: ax + b, x α on R ++ for 0 α 1, log(x) on R Example on R n. All affine function f(x) = a x + b. All norms are convex x p = ( p x i p ) 1/p, for p 1 i=1 x = max x i i 5

6 Max function f(x) = max{x 1,, x n } is convex on R n. Quadratic-over-linear function: f(x, y) = x2 y, y > 0 domf = R R ++ Log-sum-exp: f(x) = log(e x1 + + e xn ) is convex on R n. Note: Log-sum-exp function is a differentiable approximation of max function f β (x) = 1 β log(eβx1 + + e βxn ) max f β (x) max{x 1,, x n } + 1/β log n x 1,,x n Geometry mean f(x) = (Π n i=1 x i) 1 n is concave on dom(f) = R n Examples in R m n i Affine function X R m n : f(x) = i = i A ij X ij + b j (A X) ii + b = tr(a T X) + b Spectral norm f(x) = x 2 = (λ 1 (x x)) 1/2 Nuclear norm f(x) = x Sum of largest eigenvalues of a symmetric matrix A. f m (A) := m j=1 λ j(a) has also the representation f m (A) = sup {Q Q=I m} trace(qaq T ) = trace(qq T A). Volume of ellipsoids: still in symmetric matrices S n (R), define the function f(a) := log(det(a 1 )) for positive definite A, otherwise +. is convex. 4. Indicator and support functions: given a nonempty subset S R n, the function χ S : R n R n { } defined by { 0; if x S, χ S := if not (1.1) is called the indicator function of S. χ S is (closed and) convex if and only if S is (closed and) convex. Indeed, epiχ S = S R by definition. The support function of a nonempty subset S is defined as σ S (x) := sup{ s, x : s S} 6

7 Theorem 3 Let π(x) be a real-valued function on R which is positively homogeneous of degree 1: π(tx) = tπ(x) x R n, t > 0 π is convex iff it is subadditive,i.e. In particular, a norm is convex. π(x + y) π(x) + π(y), x, y. 1.2 Elemental properties of convex function Jensen s inequality Proposition 2 Let f be convex and dom(f) = Q, then for every convex combination i λ ix i of points from Q, one has f( i λ i x i ) i λ i f(x i ) Proof. Using the equivalence definition epi(f) is convex. Convexity of sub-level sets. Let f be a convex function with the domain Q, then any real α, the set is convex. Lev α (f) := {x Q, f(x) α} (convexity convex sub-level sets, but converse isn t true). (any sub-level set is convex quasiconvex). 1.3 Operations that preserving convexity of functions Non-negative weighted sum: if f and g are convex, then λf + µg is convex for any λ, µ 0. Affine substitutions is convex: If f is convex, then f(ax + b) is convex. Pointwise max/sup: let {f j } j J for J be an arbitray familly of conve functions. Then f : sup{f j : j J} os convex (assume it is not identically ). Convex monotone superposition : Let f : R n R be convex, and g : R R be convex and increasing. Then the composite g f : x g(f(x)) is convex (set g( ) = and g f is not identically ). 7

8 Partial (marginal) minimization: if f(x, y) is jointly convex, and let C be nonempty closed convex set and let g(x) = inf y C f(x, y)is proper and if f is bounded below on the set {x} C for any x R n, then g is convex. Proof. By the boundedness of {x} C for any x R n. The inf on y is attained on any x. Thus epi(g) = {(x, t) for somey C, (x, y, t) epi(f)} and it is the projection of epi(f) onto R n R. Therefore epi(g) is the image of a convex under a linear mapping and it is convex. Example: ( ) A B consider f(x, y) = x T Ax+2x T By +y T Cy with B T positive semidefinite, C is positive definite minimizing over y gives g(x) = inf y f(x, y) = C x T (A BC 1 B T )x, g is convex, hence Schur complement A BC 1 B T ) is positive semi-definite. Distance to a set: dist(x, S) = inf y S x y is convex if S is convex. Dilation and perspectives of a function: Dilation: f u = uf(x/u) is still convex (epi(f u ) = uepi(f), S r (f u ) = us r/u (f).) Perspectives: f(u, x) : R R n R {+ } defined by is convex in R n+1. f(u, x) := { uf(x/u) u > 0 + if not Proof. It is better to look at f with epigraph. epi( f) = {(u, x, r) R + R n R : f(x/u) r/u} = {u(1, x, r ) : u > 0, (x, r ) epi(f)} = u>0 {u({1} epi(f)} = R + ({1} epi(f)) and it is therefore a convex cone. Example: f(x) = x T x convex, f(x, t) = x T x/t = tf(x/t) is convex. f(x) = log(x) is convex, hence relative entropy g(x, t) = t log t t log x is convex on R

9 KullbackLeibler (KL) divergence divergence between u, v R + ++: D kl (u, v) = n (u i log(u i /v i ) u i + v i ) i=1 is convex since it is negative entropy plus linear function of u and v. If f is convex, then g(x) = (c T x + d)f((ax + b)/(c T x + d)) is convex on dom(g) = {c T x + d > 0(Ax + b)/(c T x + d) dom(f)}. Restriction of a convex function to a line: If f : R n R is convex if and only if the function g : R R, g(t) = f(x + tv), dom(g) = {t x + tv domf} is convex for any x dom(f), v R n. This means that one can check convexity of f by checking convexity of functions of one variable. Proof. When f(x) is convex, derive g(t) is convex by checking the definition. Conversely, for any x 0, x 1, consider g(t) = f(x 0 + t(x 1 x 0 )) and let t = 0 and t = 1. Example: Consider f : S n R with f(x) = log det(x), dom(f) = S n ++. g(t) = log det(x + tv ) = log det(x) + log det(i + tx 1 2 V X 1 2 ) = log det(x) + n i=1 log(1 + tλ i) where λ i are the eigenvalues of X 1 2 V X 1 2. g is concave in t (for any choice of X S n ++, V ); hence f is concave. 1.4 Differential criteria of convexity Consider C R n be nonempty and convex and a function f is defined on C(f(x) < ) for all x C. We consider f is convex and differentiable on C. Theorem 4 (First order differentiation) Let f be a function differentiable on an open set Ω R n, and let C be a convex subset of Ω. Then f is convex on C if and only if f(x) f(x 0 ) + f(x 0 ), x x 0, for all(x 0, x) C C f is strictly convex on C if and only if strict inequality holds in the above inequality whenever x x 0. f is strongly convex with modulus c on C if and only if for all (x 0, x) C C, f(x) f(x 0 ) + f(x 0 ), x x c x x 0 2 9

10 Proof. as 1. Let f be convex on C, for arbitrary (x 0, x) C C and α (0, 1), we have f(αx + (1 α)x 0 ) f(x 0 ) α(f(x) f(x 0 )) Divide by α and let α, the lefthand side reduces to f(x 0 + α(x x 0 )) f(x 0 ) f(x 0 ), x x 0 α and the inequality is established. Conversely, take x 1 and x 2 in C, α (0, 1) and set x 0 = αx 1 + α(x 2 x1), by assumption f(x i ) f(x 0 ) + f(x 0 ), x i x 0 we obtain the convex combination αf(x 1 ) + (1 α)f(x 2 ) f(x 0 ) + f(x 0 ), αx 1 + (1 α)x 2 x 0 which is the definition of convex function. 2. Similar for strictly convex for x 0 x 3. Apply the proof in 1) on the function f 1 2 c 2. Example: Non-negativity of KullbackLeibler (KL) divergence D kl (u, v) = n (u i log(u i /v i ) u i + v i ) i=1 where φ(x) = i u i log u i is convex. D kl (u, v) = φ(u) φ(v) φ(v), u v Definition 3 Let C R n be convex. The mapping F : C R n is said to be monotone [resp. strictly monotone, resp. strongly monotone with modulus c > 0] on C when, for all x, x C, F (x) F (x), x x 0 [resp. F (x) F (x), x x > 0 whenever x x, resp. F (x) F (x), x x c x x 2 ]. Theorem 5 (monotonicity of gradient) Let f be a function differentiable on an open set Ω R n, and let C be convex subset of Ω. Then f is convex [resp. strictly convex, resp. strongly convex with modulus c] on C if and only if its gradient f is monotone [resp. strictly monotone, strongly monotone with modulus c]. 10

11 Proof. Combine the case for convex monotone and strongly convex strongly monotone allowing c = 0. Let f be strongly convex on C, and for arbitrary x, x 0 C, we have f(x) f(x 0 ) + f(x 0 ), x x c x x 0 2 f(x 0 ) f(x) f(x), x x c x x 0 2 and mere addition leads to f is strongly monotone. Conversely, let (x 0, x 1 ) be a pair in C and consider φ(t) := f(x t ) where x t := x 0 + t(x 1 x 0 ) for t in an open interval containing [0, 1] and φ is well-defined differentiable. Its derivative at t is φ (t) = f(x t ), x 1 x 0. Thus for 0 t < t 1. We have φ (t) φ (t ) = f(x t ) f(x t ), x 1 x 0 = 1 t t f(x t) f(x t ), x t x t and the monotonicity for f shows that φ is increasing, φ is therefore convex. For strongly convex, set t = 0, and use the strongly monotonicity φ (t ) φ (0) 1 t c x t x 0 2 = tc x 1 x 0 2 On the other hand, we have φ(1) φ(0) φ (0) = 1 0 [φ (t) φ (0)]dt 1 2 c x 1 x 0 2 which by definition of φ, is just the definition of strongly convex function. Theorem 6 (Second order differentiation) Let f be twice diff. convex set Ω R n. Then on an open f is convex on Ω iff 2 f(x 0 ) is positive semi-definite for all x 0 Ω. if f 2 (x 0 ) is positive definite for all x 0 Ω, then f is strictly convex on Ω. f is strongly convex with modulus c on Ω if and only if the smallest eigenvalue of 2 f( ) is minorized by c on Ω: for all x 0 Ω and all d R n, 2 d, d c d 2 Note: similarly f is concave iff dom(f) is convex and 2 f(x) is negative semidefinite for x dom(f). Examples: Quadratic function f(x) = 1 2 xt P x + q T x + r with P S n is convex if P S n + ( 2 f = P ). Special case: f(x) = Ax b 2 is convex as 2 f(x) = 2A T A is positive semi-definite for any A. 11

12 Quadratic over linear f(x, y) = x2 y for y > 0. Show the Hessian matrix is ( ) y 2 f(x, y) = 2 2 yx ( ) ( ) y 3 yx x 2 = 2 y y y x. 3 x Log-sum-exp: f(x) = log( n k=1 exp x k) is convex. Show the Hessian matrix is positive semi-definite. 2 f(x) = 1 1 T z diag(z) 1 (1 T z) 2 zzt Geometric mean: f(x) = ( n k=1 x k) 1/n on R n ++ is concave. 1.5 Conjugacy The conjugate of f: f (y) := sup y, x f(x) x dom(f) Figure 1.1: A function f : R R, and a value y R. The conjugate function f (y) is the maximum gap between the linear function yx and f(x). If f is differentiable, this occurs at a point x where f (x) = y. The domain of f (y) consists of y R n for which the supermum is finite. f is convex even f is not convex since f is a set supermum of a convex (affine function of y). For differentiable function, the mapping f f is called Legendre-Fenchel transform. Examples: f (y) = sup x y T x ax b is bounded iff y = a. dom(f ) = {a} and f (a) = b. 12

13 Negative logarithm f(x) = log(x), dom(f) = R ++. f (y) = sup y T x + log(x) = x dom(f) (for x = 1 y ) when dom(f ) = {y < 0}. { unbounded y 0 log( y) 1 y < 0 f(x) = e x. For y < 0, xy e x is unbounded (let x ); For y > 0, when x = log y, xy e x reaches its maximum and f (y) = y log y y. For y = 0, f (y) = sup e x = 0, thus f (y) = y log y y (with 0 log 0 = 0). Strictly convex quadratic: f(x) = 1/2x T Qx, Q S++. n y T x 1 2 xt Qx is bounded above for all x, maxium at Q 1 y = x, thus f (y) = 1 2 yt Q 1 y. { 0 x S Indicator function: let I S (x) =, thus I + if not S (y) = sup x S y T x (support function of S. Log-sum-sup: f(x) = log( { n n i=1, f (y) == i=1 y i log y i y 0, 1 T y = 1 exi + if not Norm: let be a norm in R n, with dual norm. We will show that the conjugate of f(x) = x is { f 0 y 1 (y) = + if not Recall that y = sup{ y T x, y 1}. For y > 1, by definition, there exists z scuh that z 1 and y T z > 1. Let x = tz amd t, then y T x x = t(y z z ), thus f (y) is unbounded. Conversly, if y 1. then y T x y x and y T x x 0 and the maximum is atteined at x = 0. Thus { f 0 y 1 (y) = + if not Norm square: let f(x) = 1 2 x 2 for some norm. Then f (y) = 1 2 y 2. Properties of Conjugate functions Fenchel s inequality: f (y)+f(x) x T y (Young s inequality for differentiable f). Conjugate of conjugate: if f is convex and f is closed, then f = f. Suppose f is convex and differentiable with dom(f) = R n. Any maximizer x s.t. f(x ) = y and conversely if x s,t y = f(x ), then x maximize y T x f(x) and f (y) = f(x ) T x f(x ) = y T 1 (y) f( 1 (y)). Thus df (y) = y, dx + x, y f(x ), dx = dy, x. Thus f (y) = x. f (y) = x y = f(x ) 13

14 Scaling and composition with affine transformation: for a > 0, b R g(x) = ax + b, thus g (y) = af (y/a) b. Suppose that A R n n is nonsingular and b R n, then the conjugate of f(ax + b) is g (y) = f (A T y) b T A T y with domg = A T domf. Sum of independent function f(u, v) = f 1 (u) + f 2 (v) where f 1, f 2 are both convex with conjugate f (w, z) = f 1 (w) + f 2 (z) Convexity of the conjugation: if dom(f 1 ) dom(f 2 ), and α (0, 1), then [αf 1 + (1 α)f 2 ] αf 1 + (1 α)f 2 14