Convex Optimization: Modeling and Algorithms

Convex Optimization: Modeling and Algorithms Lieven Vandenberghe Electrical Engineering Department, UC Los Angeles Tutorial lectures, 21st Machine Learning Summer School Kyoto, August 29-30, 2012

Introduction Convex optimization MLSS 2012 mathematical optimization linear and convex optimization recent history 1

Mathematical optimization minimize f 0 (x 1,...,x n ) subject to f 1 (x 1,...,x n ) 0 f m (x 1,...,x n ) 0 a mathematical model of a decision, design, or estimation problem finding a global solution is generally intractable even simple looking nonlinear optimization problems can be very hard Introduction 2

The famous exception: Linear programming minimize c 1 x 1 + c 2 x 2 subject to a 11 x 1 + +a 1n x n b 1... a m1 x 1 + +a mn x n b m widely used since Dantzig introduced the simplex algorithm in 1948 since 1950s, many applications in operations research, network optimization, finance, engineering, combinatorial optimization,... extensive theory (optimality conditions, sensitivity analysis,... ) there exist very efficient algorithms for solving linear programs Introduction 3

Convex optimization problem minimize f 0 (x) subject to f i (x) 0, i = 1,...,m objective and constraint functions are convex: for 0 θ 1 f i (θx+(1 θ)y) θf i (x)+(1 θ)f i (y) can be solved globally, with similar (polynomial-time) complexity as LPs surprisingly many problems can be solved via convex optimization provides tractable heuristics and relaxations for non-convex problems Introduction 4

History 1940s: linear programming minimize c T x subject to a T i x b i, i = 1,...,m 1950s: quadratic programming 1960s: geometric programming 1990s: semidefinite programming, second-order cone programming, quadratically constrained quadratic programming, robust optimization, sum-of-squares programming,... Introduction 5

New applications since 1990 linear matrix inequality techniques in control support vector machine training via quadratic programming semidefinite programming relaxations in combinatorial optimization circuit design via geometric programming l 1 -norm optimization for sparse signal reconstruction applications in structural optimization, statistics, signal processing, communications, image processing, computer vision, quantum information theory, finance, power distribution,... Introduction 6

Advances in convex optimization algorithms interior-point methods 1984 (Karmarkar): first practical polynomial-time algorithm for LP 1984-1990: efficient implementations for large-scale LPs around 1990 (Nesterov & Nemirovski): polynomial-time interior-point methods for nonlinear convex programming since 1990: extensions and high-quality software packages first-order algorithms fast gradient methods, based on Nesterov s methods from 1980s extend to certain nondifferentiable or constrained problems multiplier methods for large-scale and distributed optimization Introduction 7

Overview 1. Basic theory and convex modeling convex sets and functions common problem classes and applications 2. Interior-point methods for conic optimization conic optimization barrier methods symmetric primal-dual methods 3. First-order methods (proximal) gradient algorithms dual techniques and multiplier methods

Convex sets and functions Convex optimization MLSS 2012 convex sets convex functions operations that preserve convexity

Convex set contains the line segment between any two points in the set x 1,x 2 C, 0 θ 1 = θx 1 +(1 θ)x 2 C convex not convex not convex Convex sets and functions 8

Basic examples affine set: solution set of linear equations Ax = b halfspace: solution of one linear inequality a T x b (a 0) polyhedron: solution of finitely many linear inequalities Ax b ellipsoid: solution of positive definite quadratic inquality (x x c ) T A(x x c ) 1 (A positive definite) norm ball: solution of x R (for any norm) positive semidefinite cone: S n + = {X S n X 0} the intersection of any number of convex sets is convex Convex sets and functions 9

Example of intersection property C = {x R n p(t) 1 for t π/3} where p(t) = x 1 cost+x 2 cos2t+ +x n cosnt 2 p(t) 1 0 1 x2 1 0 C 1 0 π/3 2π/3 π t 2 2 1 0 1 2 x 1 C is intersection of infinitely many halfspaces, hence convex Convex sets and functions 10

Convex function domain domf is a convex set and Jensen s inequality holds: for all x,y domf, 0 θ 1 f(θx+(1 θ)y) θf(x)+(1 θ)f(y) (x,f(x)) (y,f(y)) f is concave if f is convex Convex sets and functions 11

Examples linear and affine functions are convex and concave expx, logx, xlogx are convex x α is convex for x > 0 and α 1 or α 0; x α is convex for α 1 norms are convex quadratic-over-linear function x T x/t is convex in x, t for t > 0 geometric mean (x 1 x 2 x n ) 1/n is concave for x 0 logdetx is concave on set of positive definite matrices log(e x 1 + e x n ) is convex Convex sets and functions 12

Epigraph and sublevel set epigraph: epif = {(x,t) x domf, f(x) t} epif a function is convex if and only its epigraph is a convex set f sublevel sets: C α = {x domf f(x) α} the sublevel sets of a convex function are convex (converse is false) Convex sets and functions 13

Differentiable convex functions differentiable f is convex if and only if domf is convex and f(y) f(x)+ f(x) T (y x) for all x,y domf f(y) f(x) + f(x) T (y x) (x,f(x)) twice differentiable f is convex if and only if domf is convex and 2 f(x) 0 for all x domf Convex sets and functions 14

Establishing convexity of a function 1. verify definition 2. for twice differentiable functions, show 2 f(x) 0 3. show that f is obtained from simple convex functions by operations that preserve convexity nonnegative weighted sum composition with affine function pointwise maximum and supremum minimization composition perspective Convex sets and functions 15

Positive weighted sum & composition with affine function nonnegative multiple: αf is convex if f is convex, α 0 sum: f 1 +f 2 convex if f 1,f 2 convex (extends to infinite sums, integrals) composition with affine function: f(ax+b) is convex if f is convex examples logarithmic barrier for linear inequalities f(x) = m log(b i a T i x) i=1 (any) norm of affine function: f(x) = Ax+b Convex sets and functions 16

Pointwise maximum is convex if f 1,..., f m are convex f(x) = max{f 1 (x),...,f m (x)} example: sum of r largest components of x R n f(x) = x [1] +x [2] + +x [r] is convex (x [i] is ith largest component of x) proof: f(x) = max{x i1 +x i2 + +x ir 1 i 1 < i 2 < < i r n} Convex sets and functions 17

Pointwise supremum g(x) = supf(x,y) y A is convex if f(x,y) is convex in x for each y A examples maximum eigenvalue of symmetric matrix λ max (X) = sup y T Xy y 2 =1 support function of a set C S C (x) = supy T x y C Convex sets and functions 18

Minimization h(x) = inf y C f(x,y) is convex if f(x,y) is convex in (x,y) and C is a convex set examples distance to a convex set C: h(x) = inf y C x y optimal value of linear program as function of righthand side follows by taking h(x) = inf y:ay x ct y f(x,y) = c T y, domf = {(x,y) Ay x} Convex sets and functions 19

Composition composition of g : R n R and h : R R: f is convex if f(x) = h(g(x)) g convex, h convex and nondecreasing g concave, h convex and nonincreasing (if we assign h(x) = for x domh) examples expg(x) is convex if g is convex 1/g(x) is convex if g is concave and positive Convex sets and functions 20

Vector composition composition of g : R n R k and h : R k R: f is convex if f(x) = h(g(x)) = h(g 1 (x),g 2 (x),...,g k (x)) g i convex, h convex and nondecreasing in each argument g i concave, h convex and nonincreasing in each argument (if we assign h(x) = for x domh) example log m i=1 expg i (x) is convex if g i are convex Convex sets and functions 21

Perspective the perspective of a function f : R n R is the function g : R n R R, g(x,t) = tf(x/t) g is convex if f is convex on domg = {(x,t) x/t domf, t > 0} examples perspective of f(x) = x T x is quadratic-over-linear function g(x,t) = xt x t perspective of negative logarithm f(x) = logx is relative entropy g(x,t) = tlogt tlogx Convex sets and functions 22

the conjugate of a function f is Conjugate function f (y) = sup (y T x f(x)) x dom f f(x) xy x (0, f (y)) f is convex (even if f is not) Convex sets and functions 23

convex quadratic function (Q 0) Examples f(x) = 1 2 xt Qx f (y) = 1 2 yt Q 1 y negative entropy norm f(x) = n x i logx i f (y) = i=1 f(x) = x f (y) = n e y i 1 i=1 { 0 y 1 + otherwise indicator function (C convex) f(x) = I C (x) = { 0 x C + otherwise f (y) = supy T x x C Convex sets and functions 24

Convex optimization problems Convex optimization MLSS 2012 linear programming quadratic programming geometric programming second-order cone programming semidefinite programming

Convex optimization problem minimize f 0 (x) subject to f i (x) 0, i = 1,...,m Ax = b f 0, f 1,..., f m are convex functions feasible set is convex locally optimal points are globally optimal tractable, in theory and practice Convex optimization problems 25

Linear program (LP) minimize c T x+d subject to Gx h Ax = b inequality is componentwise vector inequality convex problem with affine objective and constraint functions feasible set is a polyhedron P x c Convex optimization problems 26

Piecewise-linear minimization minimize f(x) = max i=1,...,m (at i x+b i ) f(x) a T i x + b i equivalent linear program an LP with variables x, t R minimize t subject to a T i x+b i t, i = 1,...,m x Convex optimization problems 27

l 1 -Norm and l -norm minimization l 1 -norm approximation and equivalent LP ( y 1 = k y k ) minimize Ax b 1 n minimize i=1 y i subject to y Ax b y l -norm approximation ( y = max k y k ) minimize Ax b minimize y subject to y1 Ax b y1 (1 is vector of ones) Convex optimization problems 28

example: histograms of residuals Ax b (with A is 200 80) for x ls = argmin Ax b 2, x l1 = argmin Ax b 1 10 8 6 4 2 0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 (Ax ls b) k 100 80 60 40 20 0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 (Ax l1 b) k 1-norm distribution is wider with a high peak at zero Convex optimization problems 29

Robust regression 25 20 15 10 f(t) 5 0 5 10 15 20 10 5 0 5 10 t 42 points t i, y i (circles), including two outliers function f(t) = α+βt fitted using 2-norm (dashed) and 1-norm Convex optimization problems 30

Linear discrimination given a set of points {x 1,...,x N } with binary labels s i { 1,1} find hyperplane a T x+b = 0 that strictly separates the two classes a T x i +b > 0 if s i = 1 a T x i +b < 0 if s i = 1 homogeneous in a, b, hence equivalent to the linear inequalities (in a, b) s i (a T x i +b) 1, i = 1,...,N Convex optimization problems 31

Approximate linear separation of non-separable sets minimize N max{0,1 s i (a T x i +b)} i=1 a piecewise-linear minimization problem in a, b; equivalent to an LP can be interpreted as a heuristic for minimizing #misclassified points Convex optimization problems 32

Quadratic program (QP) minimize (1/2)x T Px+q T x+r subject to Gx h P S n +, so objective is convex quadratic minimize a convex quadratic function over a polyhedron f 0 (x ) x P Convex optimization problems 33

Linear program with random cost minimize c T x subject to Gx h c is random vector with mean c and covariance Σ hence, c T x is random variable with mean c T x and variance x T Σx expected cost-variance trade-off minimize Ec T x+γvar(c T x) = c T x+γx T Σx subject to Gx h γ > 0 is risk aversion parameter Convex optimization problems 34

Robust linear discrimination H 1 = {z a T z +b = 1} H 1 = {z a T z +b = 1} distance between hyperplanes is 2/ a 2 to separate two sets of points by maximum margin, minimize a 2 2 = a T a subject to s i (a T x i +b) 1, i = 1,...,N a quadratic program in a, b Convex optimization problems 35

Support vector classifier minimize γ a 2 2+ N max{0,1 s i (a T x i +b)} i=1 γ = 0 γ = 10 equivalent to a quadratic program Convex optimization problems 36

Kernel formulation minimize f(xa)+ a 2 2 variables a R n X R N n with N n and rank N change of variables y = Xa, a = X T (XX T ) 1 y a is minimum-norm solution of Xa = y gives convex problem with N variables y minimize f(y)+y T Q 1 y Q = XX T is kernel matrix Convex optimization problems 37

Total variation signal reconstruction minimize ˆx x cor 2 2+γφ(ˆx) x cor = x+v is corrupted version of unknown signal x, with noise v variable ˆx (reconstructed signal) is estimate of x φ : R n R is quadratic or total variation smoothing penalty φ quad (ˆx) = n 1 (ˆx i+1 ˆx i ) 2, φ tv (ˆx) = i=1 n 1 i=1 ˆx i+1 ˆx i Convex optimization problems 38

example: x cor, and reconstruction with quadratic and t.v. smoothing 2 xcor 0 2 0 500 1000 1500 2000 2 i quad. 0 2 0 500 1000 1500 2000 2 i t.v. 0 2 0 500 1000 1500 2000 quadratic smoothing smooths out noise and sharp transitions in signal total variation smoothing preserves sharp transitions in signal i Convex optimization problems 39

Geometric programming posynomial function f(x) = K k=1 c k x a 1k 1 xa 2k 2 x a nk n, domf = R n ++ with c k > 0 geometric program (GP) with f i posynomial minimize f 0 (x) subject to f i (x) 1, i = 1,...,m Convex optimization problems 40

Geometric program in convex form change variables to y i = logx i, and take logarithm of cost, constraints geometric program in convex form: ( K ) minimize log exp(a T 0ky +b 0k ) ( k=1 K ) subject to log exp(a T iky +b ik ) 0, i = 1,...,m k=1 b ik = logc ik Convex optimization problems 41

Second-order cone program (SOCP) minimize f T x subject to A i x+b i 2 c T i x+d i, i = 1,...,m 2 is Euclidean norm y 2 = y 2 1 + +y2 n constraints are nonlinear, nondifferentiable, convex 1 constraints are inequalities w.r.t. second-order cone: { y } y1 2+ +y2 p 1 y p y3 0.5 0 1 0 0 1 y 2 1 1 y 1 Convex optimization problems 42

Robust linear program (stochastic) minimize c T x subject to prob(a T i x b i) η, i = 1,...,m a i random and normally distributed with mean ā i, covariance Σ i we require that x satisfies each constraint with probability exceeding η η = 10% η = 50% η = 90% Convex optimization problems 43

SOCP formulation the chance constraint prob(a T i x b i) η is equivalent to the constraint ā T i x+φ 1 (η) Σ 1/2 i x 2 b i Φ is the (unit) normal cumulative density function 1 η Φ(t) 0.5 0 0 t Φ 1 (η) robust LP is a second-order cone program for η 0.5 Convex optimization problems 44

Robust linear program (deterministic) minimize c T x subject to a T i x b i for all a i E i, i = 1,...,m a i uncertain but bounded by ellipsoid E i = {ā i +P i u u 2 1} we require that x satisfies each constraint for all possible a i SOCP formulation minimize c T x subject to ā T i x+ PT i x 2 b i, i = 1,...,m follows from sup (ā i +P i u) T x = ā T i x+ Pi T x 2 u 2 1 Convex optimization problems 45

Examples of second-order cone constraints convex quadratic constraint (A = LL T positive definite) x T Ax+2b T x+c 0 L T x+l 1 b 2 (b T A 1 b c) 1/2 extends to positive semidefinite singular A hyperbolic constraint [ x T x yz, y,z 0 2x y z ] 2 y +z, y,z 0 Convex optimization problems 46

Examples of SOC-representable constraints positive powers x 1.5 t, x 0 z : x 2 tz, z 2 x, x,z 0 two hyperbolic constraints can be converted to SOC constraints extends to powers x p for rational p 1 negative powers x 3 t, x > 0 z : 1 tz, z 2 tx, x,z 0 two hyperbolic constraints on r.h.s. can be converted to SOC constraints extends to powers x p for rational p < 0 Convex optimization problems 47

Semidefinite program (SDP) minimize c T x subject to x 1 A 1 +x 2 A 2 + +x n A n B A 1, A 2,..., A n, B are symmetric matrices inequality X Y means Y X is positive semidefinite, i.e., z T (Y X)z = i,j (Y ij X ij )z i z j 0 for all z includes many nonlinear constraints as special cases Convex optimization problems 48

Geometry 1 [ x y y z ] 0 z 0.5 1 0 0 y 1 0 0.5 x 1 a nonpolyhedral convex cone feasible set of a semidefinite program is the intersection of the positive semidefinite cone in high dimension with planes Convex optimization problems 49

Examples A(x) = A 0 +x 1 A 1 + +x m A m (A i S n ) eigenvalue minimization (and equivalent SDP) minimize λ max (A(x)) minimize t subject to A(x) ti matrix-fractional function minimize b T A(x) 1 b subject to A(x) 0 minimize t[ A(x) b subject to b T t ] 0 Convex optimization problems 50

Matrix norm minimization A(x) = A 0 +x 1 A 1 +x 2 A 2 + +x n A n (A i R p q ) matrix norm approximation ( X 2 = max k σ k (X)) minimize A(x) 2 minimize t[ subject to ti A(x) A(x) T ti ] 0 nuclear norm approximation ( X = k σ k(x)) minimize A(x) minimize (tru +trv)/2 [ U A(x) subject to T A(x) V ] 0 Convex optimization problems 51

Semidefinite relaxation semidefinite programming is often used to find good bounds for nonconvex polynomial problems, via relaxation as a heuristic for good suboptimal points example: Boolean least-squares minimize Ax b 2 2 subject to x 2 i = 1, i = 1,...,n basic problem in digital communications could check all 2 n possible values of x { 1,1} n... an NP-hard problem, and very hard in general Convex optimization problems 52

Lifting Boolean least-squares problem minimize x T A T Ax 2b T Ax+b T b subject to x 2 i = 1, i = 1,...,n reformulation: introduce new variable Y = xx T minimize tr(a T AY) 2b T Ax+b T b subject to Y = xx T diag(y) = 1 cost function and second constraint are linear (in the variables Y, x) first constraint is nonlinear and nonconvex... still a very hard problem Convex optimization problems 53

Relaxation replace Y = xx T with weaker constraint Y xx T to obtain relaxation minimize tr(a T AY) 2b T Ax+b T b subject to Y xx T diag(y) = 1 convex; can be solved as a semidefinite program Y xx T [ Y x x T 1 ] 0 optimal value gives lower bound for Boolean LS problem if Y = xx T at the optimum, we have solved the exact problem otherwise, can use randomized rounding generate z from N(x,Y xx T ) and take x = sign(z) Convex optimization problems 54

Example 0.5 0.4 SDP bound LS solution frequency 0.3 0.2 0.1 0 1 1.2 Ax b 2 /(SDP bound) n = 100: feasible set has 2 100 10 30 points histogram of 1000 randomized solutions from SDP relaxation Convex optimization problems 55

Conic optimization Convex optimization MLSS 2012 definitions and examples modeling duality

Generalized (conic) inequalities conic inequality: a constraint x K with K a convex cone in R m we require that K is a proper cone: closed pointed: does not contain a line (equivalently, K ( K) = {0} with nonempty interior: intk (equivalently, K +( K) = R m ) notation x K y x y K, x K y x y intk subscript in K is omitted if K is clear from the context Conic optimization 56

Cone linear program minimize c T x subject to Ax K b if K is the nonnegative orthant, this is a (regular) linear program widely used in recent literature on convex optimization modeling: a small number of primitive cones is sufficient to express most convex constraints that arise in practice algorithms: a convenient problem format when extending interior-point algorithms for linear programming to convex optimization Conic optimization 57

Norm cone K = { (x,y) R m 1 R x y } 1 y 0.5 1 0 0 x 2 1 1 x 1 0 1 for the Euclidean norm this is the second-order cone (notation: Q m ) Conic optimization 58

Second-order cone program minimize c T x subject to B k0 x+d k0 2 B k1 x+d k1, k = 1,...,r cone LP formulation: express constraints as Ax K b K = Q m 1 Q m r, A = B 10 B 11. B r0, b = d 10 d 11. d r0 B r1 d r1 (assuming B k0, d k0 have m k 1 rows) Conic optimization 59

Vector notation for symmetric matrices vectorized symmetric matrix: for U S p vec(u) = 2 ( U11, U 21,..., U p1, U 22, U 32,..., U p2,..., U ) pp 2 2 2 inverse operation: for u = (u 1,u 2,...,u n ) R n with n = p(p+1)/2 mat(u) = 1 2 2u1 u 2 u p u 2 2up+1 u 2p 1... u p u 2p 1 2up(p+1)/2 coefficients 2 are added so that standard inner products are preserved: tr(uv) = vec(u) T vec(v), u T v = tr(mat(u)mat(v)) Conic optimization 60

Positive semidefinite cone S p = {vec(x) X S p +} = {x R p(p+1)/2 mat(x) 0} 1 z 0.5 0 1 0 y 1 0 x 0.5 1 S 2 = { (x,y,z) [ x y/ 2 y/ 2 z ] } 0 Conic optimization 61

Semidefinite program minimize c T x subject to x 1 A 11 +x 2 A 12 + +x n A 1n B 1... x 1 A r1 +x 2 A r2 + +x n A rn B r r linear matrix inequalities of order p 1,..., p r cone LP formulation: express constraints as Ax K B K = S p 1 S p 2 S p r A = vec(a 11 ) vec(a 12 ) vec(a 1n ) vec(a 21 ) vec(a 22 ) vec(a 2n )... vec(a r1 ) vec(a r2 ) vec(a rn ), b = vec(b 1 ) vec(b 2 ). vec(b r ) Conic optimization 62

Exponential cone the epigraph of the perspective of expx is a non-proper cone { } K = (x,y,z) R 3 ye x/y z, y > 0 the exponential cone is K exp = clk = K {(x,0,z) x 0,z 0} 1 z 0.5 0 3 2 y 1 0 2 1 x 0 1 Conic optimization 63

Geometric program minimize c T x subject to log n i k=1 exp(a T ik x+b ik) 0, i = 1,...,r cone LP formulation minimize c T x subject to at ik x+b ik 1 z ik n i k=1 K exp, k = 1,...,n i, i = 1,...,r z ik 1, i = 1,...,m Conic optimization 64

Power cone definition: for α = (α 1,α 2,...,α m ) > 0, m i=1 α i = 1 K α = { (x,y) R m + R y x α } 1 1 xα m m examples for m = 2 α = ( 1 2, 1 2 ) α = (2 3, 1 3 ) α = (3 4, 1 4 ) 0.5 0.5 0.4 0.2 y 0 y 0 y 0 0.2 0.4 0 0.5 1 0 0.5 x 1 x 2 1 0.5 0 0.5 1 0 0.5 x 1 x 2 1 0.5 0 0.5 1 0 0.5 1 x 1 x 2 Conic optimization 65

Outline definition and examples modeling duality

Modeling software modeling packages for convex optimization CVX, YALMIP (MATLAB) CVXPY, CVXMOD (Python) assist the user in formulating convex problems, by automating two tasks: verifying convexity from convex calculus rules transforming problem in input format required by standard solvers related packages general-purpose optimization modeling: AMPL, GAMS Conic optimization 66

CVX example minimize Ax b 1 subject to 0 x k 1, k = 1,...,n MATLAB code cvx_begin variable x(3); minimize(norm(a*x - b, 1)) subject to x >= 0; x <= 1; cvx_end between cvx_begin and cvx_end, x is a CVX variable after execution, x is MATLAB variable with optimal solution Conic optimization 67

Modeling and conic optimization convex modeling systems (CVX, YALMIP, CVXPY, CVXMOD,... ) convert problems stated in standard mathematical notation to cone LPs in principle, any convex problem can be represented as a cone LP in practice, a small set of primitive cones is used (R n +, Q p, S p ) choice of cones is limited by available algorithms and solvers (see later) modeling systems implement set of rules for expressing constraints f(x) t as conic inequalities for the implemented cones Conic optimization 68

Examples of second-order cone representable functions convex quadratic f(x) = x T Px+q T x+r (P 0) quadratic-over-linear function f(x,y) = xt x y with domf = R n R + (assume 0/0 = 0) convex powers with rational exponent f(x) = x α, f(x) = { x β x > 0 + x 0 for rational α 1 and β 0 p-norm f(x) = x p for rational p 1 Conic optimization 69

Examples of SD cone representable functions matrix-fractional function f(x,y) = y T X 1 y with domf = {(X,y) S n + R n y R(X)} maximum eigenvalue of symmetric matrix maximum singular value f(x) = X 2 = σ 1 (X) [ ] ti X X 2 t X T ti 0 nuclear norm f(x) = X = i σ i(x) X t U,V : [ U X X T V ] 0, 1 (tru +trv) t 2 Conic optimization 70

Functions representable with exponential and power cone exponential cone exponential and logarithm entropy f(x) = xlogx power cone increasing power of absolute value: f(x) = x p with p 1 decreasing power: f(x) = x q with q 0 and domain R ++ p-norm: f(x) = x p with p 1 Conic optimization 71

Outline definition and examples modeling duality

Linear programming duality primal and dual LP (P) minimize c T x subject to Ax b (D) maximize b T z subject to A T z +c = 0 z 0 primal optimal value is p (+ if infeasible, if unbounded below) dual optimal value is d ( if infeasible, + if unbounded below) duality theorem weak duality: p d, with no exception strong duality: p = d if primal or dual is feasible if p = d is finite, then primal and dual optima are attained Conic optimization 72

Dual cone definition K = {y x T y 0 for all x K} K is a proper cone if K is a proper cone dual inequality: x y means x K y for generic proper cone K note: dual cone depends on choice of inner product: H 1 K is dual cone for inner product x,y = x T Hy Conic optimization 73

Examples R p +, Q p, S p are self-dual: K = K dual of a norm cone is the norm cone of the dual norm dual of exponential cone K exp = { (u,v,w) R R R + ulog( u/w)+u v 0 } (with 0log(0/w) = 0 if w 0) dual of power cone is K α = { (u,v) R m + R v (u 1 /α 1 ) α1 (u m /α m ) α m } Conic optimization 74

Primal and dual cone LP primal problem (optimal value p ) minimize c T x subject to Ax b dual problem (optimal value d ) maximize b T z subject to A T z +c = 0 z 0 weak duality: p d (without exception) Conic optimization 75

Strong duality p = d if primal or dual is strictly feasible slightly weaker than LP duality (which only requires feasibility) can have d < p with finite p and d other implications of strict feasibility if primal is strictly feasible, then dual optimum is attained (if d is finite) if dual is strictly feasible, then primal optimum is attained (if p is finite) Conic optimization 76

Optimality conditions minimize c T x subject to Ax+s = b s 0 maximize b T z subject to A T z +c = 0 z 0 optimality conditions [ 0 s ] = [ 0 A T A 0 ][ x z ] + [ c b ] s 0, z 0, z T s = 0 duality gap: inner product of (x,z) and (0,s) gives z T s = c T x+b T z Conic optimization 77

Barrier methods Convex optimization MLSS 2012 barrier method for linear programming normal barriers barrier method for conic optimization

History 1960s: Sequentially Unconstrained Minimization Technique (SUMT) solves nonlinear convex optimization problem minimize f 0 (x) subject to f i (x) 0, i = 1,...,m via a sequence of unconstrained minimization problems minimize tf 0 (x) m i=1 log( f i (x)) 1980s: LP barrier methods with polynomial worst-case complexity 1990s: barrier methods for non-polyhedral cone LPs Barrier methods 78

Logarithmic barrier function for linear inequalities barrier for nonnegative orthant R m +: φ(s) = m barrier for inequalities Ax b: i=1 logs i ψ(x) = φ(b Ax) = m log(b i a T i x) i=1 convex, ψ(x) at boundary of domψ = {x Ax < b} gradient and Hessian ψ(x) = A T φ(s), 2 ψ(x) = A T φ 2 (s)a with s = b Ax and ( 1 φ(s) =,..., s 1 1 s m ) ( 1, φ 2 (s) = diag s 2,..., 1 1 s 2 m ) Barrier methods 79

Central path for linear program minimize c T x subject to Ax b central path: minimizers x (t) of c f t (x) = tc T x+φ(b Ax) t is a positive parameter x x (t) optimality conditions: x = x (t) satisfies f t (x) = tc A T φ(s) = 0, s = b Ax Barrier methods 80

Central path and duality dual feasible point on central path for x = x (t) and s = b Ax, z (t) = 1 t φ(s) = ( 1 ts 1, 1 ts 2,..., 1 ts m z = z (t) is strictly dual feasible: c+a T z = 0 and z > 0 can be corrected to account for inexact centering of x x (t) duality gap between x = x (t) and z = z (t) is ) c T x+b T z = s T z = m t gives bound on suboptimality: c T x (t) p m/t Barrier methods 81

Barrier method starting with t > 0, strictly feasible x make one or more Newton steps to (approximately) minimize f t : x + = x α 2 f t (x) 1 f t (x) step size α is fixed or from line search increase t and repeat until c T x p ǫ complexity: with proper initialization, step size, update scheme for t, #Newton steps = O ( mlog(1/ǫ) ) result follows from convergence analysis of Newton s method for f t Barrier methods 82

Outline barrier method for linear programming normal barriers barrier method for conic optimization

Normal barrier for proper cone φ is a θ-normal barrier for the proper cone K if it is a barrier: smooth, convex, domain intk, blows up at boundary of K logarithmically homogeneous with parameter θ: φ(tx) = φ(x) θlogt, x intk, t > 0 self-concordant: restriction g(α) = φ(x + αv) to any line satisfies g (α) 2g (α) 3/2 (Nesterov and Nemirovski, 1994) Barrier methods 83

Examples nonnegative orthant: K = R m + φ(x) = m logx i (θ = m) i=1 second-order cone: K = Q p = {(x,y) R p 1 R x 2 y} φ(x,y) = log(y 2 x T x) (θ = 2) semidefinite cone: K = S m = {x R m(m+1)/2 mat(x) 0} φ(x) = logdetmat(x) (θ = m) Barrier methods 84

exponential cone: K exp = cl{(x,y,z) R 3 ye x/y z, y > 0} φ(x,y,z) = log(ylog(z/y) x) logz logy (θ = 3) power cone: K = {(x 1,x 2,y) R + R + R y x α 1 1 xα 2 2 } φ(x,y) = log ( x 2α 1 1 x 2α 2 2 y 2) logx 1 logx 2 (θ = 4) Barrier methods 85

Central path conic LP (with inequality with respect to proper cone K) minimize c T x subject to Ax b barrier for the feasible set where φ is a θ-normal barrier for K φ(b Ax) central path: set of minimizers x (t) (with t > 0) of f t (x) = tc T x+φ(b Ax) Barrier methods 86

Newton step centering problem minimize f t (x) = tc T x+φ(b Ax) Newton step at x x = 2 f t (x) 1 f t (x) Newton decrement λ t (x) = ( x T 2 f t (x) x ) 1/2 = ( f t (x) T x ) 1/2 useful as a measure of proximity of x to x (t) Barrier methods 87

Damped Newton method minimize f t (x) = tc T x+φ(b Ax) algorithm (with parameters ǫ (0,1/2), η (0,1/4]) select a starting point x domf t repeat: 1. compute Newton step x and Newton decrement λ t (x) 2. if λ t (x) 2 ǫ, return x 3. otherwise, set x := x+α x with α = 1 1+λ t (x) if λ t (x) η, α = 1 if λ t (x) < η stopping criterion λ t (x) 2 ǫ implies f t (x) inff t (x) ǫ alternatively, can use backtracking line search Barrier methods 88

Convergence results for damped Newton method damped Newton phase: f t decreases by at least a positive constant γ f t (x + ) f t (x) γ if λ t (x) η where γ = η log(1+η) quadratic convergence phase: λ t rapidly decreases to zero 2λ t (x + ) (2λ t (x)) 2 if λ t (x) < η implies λ t (x + ) 2η 2 < η conclusion: the number of Newton iterations is bounded by f t (x (0) ) inff t (x) γ +log 2 log 2 (1/ǫ) Barrier methods 89

Outline barrier method for linear programming normal barriers barrier method for conic optimization

Central path and duality x (t) = argmin ( tc T x+φ(b Ax) ) duality point on central path: x (t) defines a strictly dual feasible z (t) z (t) = 1 t φ(s), s = b Ax (t) duality gap: gap between x = x (t) and z = z (t) is c T x+b T z = s T z = θ t, ct x p θ t extension near central path (for λ t (x) < 1): c T x p ( 1+ λ t(x) θ ) θ t (results follow from properties of normal barriers) Barrier methods 90

Short-step barrier method algorithm (parameters ǫ (0,1), β = 1/8) select initial x and t with λ t (x) β repeat until 2θ/t ǫ: ( ) 1 t := 1+ 1+8 t, x := x f t (x) 1 f t (x) θ properties increases t slowly so x stays in region of quadratic region (λ t (x) β) iteration complexity ( ( )) θlog θ #iterations = O ǫt 0 best known worst-case complexity; same as for linear programming Barrier methods 91

Predictor-corrector methods short-step barrier methods stay in narrow neighborhood of central path (defined by limit on λ t ) make small, fixed increases t + = µt as a result, quite slow in practice predictor-corrector method select new t using a linear approximation to central path ( predictor ) re-center with new t ( corrector ) allows faster and adaptive increases in t; similar worst-case complexity Barrier methods 92

Primal-dual methods Convex optimization MLSS 2012 primal-dual algorithms for linear programming symmetric cones primal-dual algorithms for conic optimization implementation

Primal-dual interior-point methods similarities with barrier method follow the same central path same linear algebra cost per iteration differences more robust and faster (typically less than 50 iterations) primal and dual iterates updated at each iteration symmetric treatment of primal and dual iterates can start at infeasible points include heuristics for adaptive choice of central path parameter t often have superlinear asymptotic convergence Primal-dual methods 93

Primal-dual central path for linear programming minimize c T x subject to Ax+s = b s 0 maximize b T z subject to A T z +c = 0 z 0 optimality conditions (s z is component-wise vector product) Ax+s = b, A T z +c = 0, (s,z) 0, s z = 0 primal-dual parametrization of central path Ax+s = b, A T z +c = 0, (s,z) 0, s z = µ1 solution is x = x (t), z = z (t) for t = 1/µ µ = (s T z)/m for x, z on the central path Primal-dual methods 94

Primal-dual search direction current iterates ˆx, ŝ > 0, ẑ > 0 updated as ˆx := ˆx+α x, ŝ := ŝ+α s, ẑ := ẑ +α z primal and dual steps x, s, z are defined by A(ˆx+ x)+ŝ+ s = b, A T (ẑ + z)+c = 0 ẑ s+ŝ z = σˆµ1 ŝ ẑ where ˆµ = (ŝ T ẑ)/m and σ [0,1] last equation is linearization of (ŝ+ s) (ẑ + z) = σˆµ1 targets point on central path with µ = σˆµ i.e., with gap σ(ŝ T ẑ) different methods use different strategies for selecting σ α (0,1] selected so that ŝ > 0, ẑ > 0 Primal-dual methods 95

Linear algebra complexity at each iteration solve an equation A I 0 0 0 A T 0 diag(ẑ) diag(ŝ) x s z = b Aˆx ŝ c A T ẑ σˆµ1 ŝ ẑ after eliminating s, z this reduces to an equation A T DA x = r, with D = diag(ẑ 1 /ŝ 1,...,ẑ m /ŝ m ) similar equation as in simple barrier method (with different D, r) Primal-dual methods 96

Outline primal-dual algorithms for linear programming symmetric cones primal-dual algorithms for conic optimization implementation

Symmetric cones symmetric primal-dual solvers for cone LPs are limited to symmetric cones second-order cone positive semidefinite cone direct products of these primitive symmetric cones (such as R p +) definition: cone of squares x = y 2 = y y for a product that satisfies 1. bilinearity (x y is linear in x for fixed y and vice-versa) 2. x y = y x 3. x 2 (y x) = (x 2 y) x 4. x T (y z) = (x y) T z not necessarily associative Primal-dual methods 97

Vector product and identity element nonnegative orthant: component-wise product x y = diag(x)y identity element is e = 1 = (1,1,...,1) positive semidefinite cone: symmetrized matrix product x y = 1 2 vec(xy +YX) with X = mat(x), Y = mat(y) identity element is e = vec(i) second-order cone: the product of x = (x 0,x 1 ) and y = (y 0,y 1 ) is x y = 1 2 [ identity element is e = ( 2,0,...,0) x T y x 0 y 1 +y 0 x 1 ] Primal-dual methods 98

Classification symmetric cones are studied in the theory of Euclidean Jordan algebras all possible symmetric cones have been characterized list of symmetric cones the second-order cone the positive semidefinite cone of Hermitian matrices with real, complex, or quaternion entries 3 3 positive semidefinite matrices with octonion entries Cartesian products of these primitive symmetric cones (such as R p +) practical implication can focus on Q p, S p and study these cones using elementary linear algebra Primal-dual methods 99

Spectral decomposition with each symmetric cone/product we associate a spectral decomposition x = θ λ i q i, i=1 with θ q i = e and q i q j = i=1 { qi i = j 0 i j semidefinite cone (K = S p ): eigenvalue decomposition of mat(x) θ = p, mat(x) = p λ i v i vi T, q i = vec(v i vi T ) i=1 second-order cone (K = Q p ) θ = 2, λ i = x 0± x 1 2 2, q i = 1 2 [ ] 1, i = 1,2 ±x 1 / x 1 2 Primal-dual methods 100

Applications nonnegativity x 0 λ 1,...,λ θ 0, x 0 λ 1,...,λ θ > 0 powers (in particular, inverse and square root) x α = i λ α i q i log-det barrier φ(x) = logdetx = θ logλ i i=1 a θ-normal barrier, with gradient φ(x) = x 1 Primal-dual methods 101

Symmetric parametrization of central path centering problem minimize tc T x+φ(b Ax) optimality conditions (using φ(s) = s 1 ) Ax+s = b, A T z +c = 0, (s,z) 0, z = 1 t s 1 equivalent symmetric form (with µ = 1/t) Ax+b = s, A T z +c = 0, (s,z) 0, s z = µe Primal-dual methods 102

Scaling with Hessian linear transformation with H = 2 φ(u) has several important properties preserves conic inequalities: s 0 Hs 0 if s is invertible, then Hs is invertible and (Hs) 1 = H 1 s 1 preserves central path: s z = µe (Hs) (H 1 z) = µe example (K = S p ): transformation w = 2 φ(u)s is a congruence W = U 1 SU 1, W = mat(w), S = mat(s), U = mat(u) Primal-dual methods 103

Primal-dual search direction steps x, s, z at current iterates ˆx, ŝ, ẑ are defined by A(ˆx+ x)+ŝ+ s = b, A T (ẑ + z)+c = 0 (Hŝ) (H 1 z)+(h 1 ẑ) (H s) = σˆµe (Hŝ) (H 1 ẑ) where ˆµ = (ŝ T ẑ)/θ, σ [0,1], and H = 2 φ(u) last equation is linearization of (H(ŝ+ s)) ( H 1 (ẑ + z) ) = σˆµe different algorithms use different choices of σ, H Nesterov-Todd scaling: choose H = 2 φ(u) such that Hŝ = H 1 ẑ Primal-dual methods 104

Software implementations general-purpose software for nonlinear convex optimization several high-quality packages (MOSEK, Sedumi, SDPT3, SDPA,... ) exploit sparsity to achieve scalability customized implementations can exploit non-sparse types of problem structure often orders of magnitude faster than general-purpose solvers Primal-dual methods 105

Example: l 1 -regularized least-squares A is m n (with m n) and dense quadratic program formulation minimize Ax b 2 2+ x 1 minimize Ax b 2 2+1 T u subject to u x u coefficient of Newton system in interior-point method is [ A T A 0 0 0 ] [ ] D1 +D + 2 D 2 D 1 D 2 D 1 D 1 +D 2 (D 1, D 2 positive diagonal) expensive for large n: cost is O(n 3 ) Primal-dual methods 106

customized implementation can reduce Newton equation to solution of a system (AD 1 A T +I) u = r cost per iteration is O(m 2 n) comparison (seconds on 2.83 Ghz Core 2 Quad machine) m n custom general-purpose 50 200 0.02 0.32 50 400 0.03 0.59 100 1000 0.12 1.69 100 2000 0.24 3.43 500 1000 1.19 7.54 500 2000 2.38 17.6 custom solver is CVXOPT; general-purpose solver is MOSEK Primal-dual methods 107

Gradient methods Convex optimization MLSS 2012 gradient and subgradient method proximal gradient method fast proximal gradient methods 108

Classical gradient method to minimize a convex differentiable function f: choose x (0) and repeat x (k) = x (k 1) t k f(x (k 1) ), k = 1,2,... step size t k is constant or from line search advantages every iteration is inexpensive does not require second derivatives disadvantages often very slow; very sensitive to scaling does not handle nondifferentiable functions Gradient methods 109

Quadratic example f(x) = 1 2 (x2 1+γx 2 2) (γ > 1) with exact line search and starting point x (0) = (γ,1) x (k) x 2 x (0) x 2 = ( ) k γ 1 γ +1 4 x2 0 4 100 10 x 1 Gradient methods 110

Nondifferentiable example f(x) = x 2 1 +γx2 2 ( x 2 x 1 ), f(x) = x 1+γ x 2 1+γ ( x 2 > x 1 ) with exact line search, x (0) = (γ,1), converges to non-optimal point 2 x2 0 2 20 2 4 x 1 Gradient methods 111

First-order methods address one or both disadvantages of the gradient method methods for nondifferentiable or constrained problems smoothing methods subgradient method proximal gradient method methods with improved convergence variable metric methods conjugate gradient method accelerated proximal gradient method we will discuss subgradient and proximal gradient methods Gradient methods 112

Subgradient g is a subgradient of a convex function f at x if f(y) f(x)+g T (y x) y domf f(x) f(x 1 ) + g T 1 (x x 1) f(x 2 ) + g T 2 (x x 2) f(x 2 ) + g T 3 (x x 2) x 1 x 2 generalizes basic inequality for convex differentiable f f(y) f(x)+ f(x) T (y x) y domf Gradient methods 113

Subdifferential the set of all subgradients of f at x is called the subdifferential f(x) absolute value f(x) = x f(x) = x f(x) 1 x 1 x Euclidean norm f(x) = x 2 f(x) = 1 x 2 x if x 0, f(x) = {g g 2 1} if x = 0 Gradient methods 114

Subgradient calculus weak calculus rules for finding one subgradient sufficient for most algorithms for nondifferentiable convex optimization if one can evaluate f(x), one can usually compute a subgradient much easier than finding the entire subdifferential subdifferentiability convex f is subdifferentiable on dom f except possibly at the boundary example of a non-subdifferentiable function: f(x) = x at x = 0 Gradient methods 115

Examples of calculus rules nonnegative combination: f = α 1 f 1 +α 2 f 2 with α 1,α 2 0 g = α 1 g 1 +α 2 g 2, g 1 f 1 (x), g 2 f 2 (x) composition with affine transformation: f(x) = h(ax + b) g = A T g, g h(ax+b) pointwise maximum f(x) = max{f 1 (x),...,f m (x)} g f i (x) where f i (x) = max k f k (x) conjugate f (x) = sup y (x T y f(y)): take any maximizing y Gradient methods 116

Subgradient method to minimize a nondifferentiable convex function f: choose x (0) and repeat x (k) = x (k 1) t k g (k 1), k = 1,2,... g (k 1) is any subgradient of f at x (k 1) step size rules fixed step size: t k constant fixed step length: t k g (k 1) 2 constant (i.e., x (k) x (k 1) 2 constant) diminishing: t k 0, t k = k=1 Gradient methods 117

Some convergence results assumption: f is convex and Lipschitz continuous with constant G > 0: f(x) f(y) G x y 2 x,y results fixed step size t k = t converges to approximately G 2 t/2-suboptimal fixed length t k g (k 1) 2 = s converges to approximately Gs/2-suboptimal decreasing k t k, t k 0: convergence rate of convergence is 1/ k with proper choice of step size sequence Gradient methods 118

Example: 1-norm minimization minimize Ax b 1 (A R 500 100,b R 500 ) subgradient is given by A T sign(ax b) (f (k) best f )/f 10 0 0.1 0.01 0.001 10-1 10-2 10-3 0.01/ k 0.01/k 10-1 10-2 10-3 10-4 10-4 0 500 1000 1500 2000 2500 3000 k fixed steplength s = 0.1, 0.01, 0.001 10-5 0 1000 2000 3000 4000 5000 k diminishing step size t k = 0.01/ k, t k = 0.01/k Gradient methods 119

Outline gradient and subgradient method proximal gradient method fast proximal gradient methods

Proximal operator the proximal operator (prox-operator) of a convex function h is prox h (x) = argmin u h(x) = 0: prox h (x) = x ( h(u)+ 1 ) 2 u x 2 2 h(x) = I C (x) (indicator function of C): prox h is projection on C prox h (x) = argmin u C u x 2 2 = P C (x) h(x) = x 1 : prox h is the soft-threshold (shrinkage) operation prox h (x) i = x i 1 x i 1 0 x i 1 x i +1 x i 1 Gradient methods 120

Proximal gradient method unconstrained problem with cost function split in two components minimize f(x) = g(x)+h(x) g convex, differentiable, with domg = R n h convex, possibly nondifferentiable, with inexpensive prox-operator proximal gradient algorithm x (k) = prox tk h ( ) x (k 1) t k g(x (k 1) ) t k > 0 is step size, constant or determined by line search Gradient methods 121

Examples minimize g(x) + h(x) gradient method: h(x) = 0, i.e., minimize g(x) x + = x t g(x) gradient projection method: h(x) = I C (x), i.e., minimize g(x) over C x x + = P C (x t g(x)) C x + x t g(x) Gradient methods 122

iterative soft-thresholding: h(x) = x 1 x + = prox th (x t g(x)) where prox th (u) i = u i t u i t 0 t u i t u i +t u i t t prox th (u) i t u i Gradient methods 123

Properties of proximal operator prox h (x) = argmin u ( h(u)+ 1 ) 2 u x 2 2 assume h is closed and convex (i.e., convex with closed epigraph) prox h (x) is uniquely defined for all x prox h is nonexpansive prox h (x) prox h (y) 2 x y 2 Moreau decomposition x = prox h (x)+prox h (x) Gradient methods 124

Moreau-Yosida regularization h (t) (x) = inf u ( h(u)+ 1 ) 2t u x 2 2 (with t > 0) h (t) is convex (infimum over u of a convex function of x, u) domain of h (t) is R n (minimizing u = prox th (x) is defined for all x) h (t) is differentiable with gradient h (t) (x) = 1 t (x prox th(x)) gradient is Lipschitz continuous with constant 1/t can interpret prox th (x) as gradient step x t h (t) (x) Gradient methods 125

Examples indicator function (of closed convex set C): squared Euclidean distance h(x) = I C (x), h (t) (x) = 1 2t dist(x)2 1-norm: Huber penalty h(x) = x 1, h (t) (x) = n φ t (x k ) k=1 φ t (z) = { z 2 /(2t) z t z t/2 z t φt(z) t/2 z t/2 Gradient methods 126

Examples of inexpensive prox-operators projection on simple sets hyperplanes and halfspaces rectangles {x l x u} probability simplex {x 1 T x = 1,x 0} norm ball for many norms (Euclidean, 1-norm,... ) nonnegative orthant, second-order cone, positive semidefinite cone Gradient methods 127

Euclidean norm: h(x) = x 2 prox th (x) = ( 1 t ) x if x 2 t, x 2 prox th (x) = 0 otherwise logarithmic barrier h(x) = n i=1 logx i, prox th (x) i = x i+ x 2 i +4t, i = 1,...,n 2 Euclidean distance: d(x) = inf y C x y 2 (C closed convex) prox td (x) = θp C (x)+(1 θ)x, θ = t max{d(x), t} generalizes soft-thresholding operator Gradient methods 128

Prox-operator of conjugate prox th (x) = x tprox h /t(x/t) follows from Moreau decomposition of interest when prox-operator of h is inexpensive example: norms h(x) = x, h (y) = I C (y) where C is unit ball for dual norm prox h /t is projection on C formula useful for prox-operator of if projection on C is inexpensive Gradient methods 129

Support function many convex functions can be expressed as support functions with C closed, convex h(x) = S C (x) = supx T y y C conjugate is indicator function of C: h (y) = I C (y) hence, can compute prox th via projection on C example: h(x) is sum of largest r components of x h(x) = x [1] + +x [r] = S C (x), C = {y 0 y 1,1 T y = r} Gradient methods 130

Convergence of proximal gradient method minimize f(x) = g(x)+h(x) assumptions g is Lipschitz continuous with constant L > 0 g(x) g(y) 2 L x y 2 x,y optimal value f is finite and attained at x (not necessarily unique) result: with fixed step size t k = 1/L f(x (k) ) f L 2k x(0) x 2 2 compare with 1/ k rate of subgradient method can be extended to include line searches Gradient methods 131

Outline gradient and subgradient method proximal gradient method fast proximal gradient methods

Fast (proximal) gradient methods Nesterov (1983, 1988, 2005): three gradient projection methods with 1/k 2 convergence rate Beck & Teboulle (2008): FISTA, a proximal gradient version of Nesterov s 1983 method Nesterov (2004 book), Tseng (2008): overview and unified analysis of fast gradient methods several recent variations and extensions this lecture: FISTA (Fast Iterative Shrinkage-Thresholding Algorithm) Gradient methods 132

FISTA unconstrained problem with composite objective minimize f(x) = g(x)+h(x) g convex differentiable with domg = R n h convex with inexpensive prox-operator algorithm: choose any x (0) = x ( 1) ; for k 1, repeat the steps y = x (k 1) + k 2 k +1 (x(k 1) x (k 2) ) x (k) = prox tk h(y t k g(y)) Gradient methods 133

Interpretation first two iterations (k = 1,2) are proximal gradient steps at x (k 1) next iterations are proximal gradient steps at extrapolated points y x (k) = prox tk h(y t k g(y)) x (k 2) x (k 1) y sequence x (k) remains feasible (in domh); y may be outside domh Gradient methods 134

Convergence of FISTA minimize f(x) = g(x)+h(x) assumptions domg = R n and g is Lipschitz continuous with constant L > 0 h is closed (implies prox th (u) exists and is unique for all u) optimal value f is finite and attained at x (not necessarily unique) result: with fixed step size t k = 1/L f(x (k) ) f 2L (k +1) 2 x(0) f 2 2 compare with 1/k convergence rate for gradient method can be extended to include line searches Gradient methods 135

10 0 gradient 10 0 gradient Example minimize log m i=1 exp(a T i x+b i) randomly generated data with m = 2000, n = 1000, same fixed step size 10-1 FISTA 10-1 FISTA f(x (k) ) f f 10-2 10-3 10-4 10-2 10-3 10-4 10-5 10-5 10-6 0 50 100 150 200 FISTA is not a descent method k 10-6 0 50 100 150 200 k Gradient methods 136

Dual methods Convex optimization MLSS 2012 Lagrange duality dual decomposition dual proximal gradient method multiplier methods

Dual function convex problem (with linear constraints for simplicity) minimize f(x) subject to Gx h Ax = b Lagrangian L(x,λ,ν) = f(x)+λ T (Gx h)+ν T (Ax b) dual function g(λ,ν) = inf x L(x,λ,ν) = f ( G T λ A T ν) h T λ b T ν f (y) = sup x (y T x f(x)) is conjugate of f Dual methods 137

Dual problem maximize g(λ, ν) subject to λ 0 a convex optimization problem in λ, ν duality theorem (p is primal optimal value, d is dual optimal value) weak duality: p d (without exception) strong duality: p = d if a constraint qualification holds (for example, primal problem is feasible and dom f open) Dual methods 138

Norm approximation minimize Ax b reformulated problem minimize y subject to y = Ax b dual function ( g(ν) = inf y +ν T y ν T Ax+b T ν ) x,y { b = T ν A T ν = 0, ν 1 otherwise dual problem maximize b T z subject to A T z = 0, z 1 Dual methods 139

Karush-Kuhn-Tucker optimality conditions if strong duality holds, then x, λ, ν are optimal if and only if 1. x is primal feasible x domf, Gx h, Ax = b 2. λ 0 3. complementary slackness holds λ T (h Gx) = 0 4. x minimizes L(x,λ,ν) = f(x)+λ T (Gx h)+ν T (Ax b) for differentiable f, condition 4 can be expressed as f(x)+g T λ+a T ν = 0 Dual methods 140

Outline Lagrange dual dual decomposition dual proximal gradient method multiplier methods

Dual methods primal problem minimize f(x) subject to Gx h Ax = b dual problem maximize h T λ b T ν f ( G T λ A T ν) subject to λ 0 possible advantages of solving the dual when using first-order methods dual problem is unconstrained or has simple constraints dual is differentiable dual (almost) decomposes into smaller problems Dual methods 141

(Sub-)gradients of conjugate function f (y) = sup x ( y T x f(x) ) subgradient: x is a subgradient at y if it maximizes y T x f(x) if maximizing x is unique, then f is differentiable at y this is the case, for example, if f is strictly convex strongly convex function: f is strongly convex with modulus µ > 0 if f(x) µ 2 xt x is convex implies that f (x) is Lipschitz continuous with parameter 1/µ Dual methods 142

Dual gradient method primal problem with equality constraints and dual minimize f(x) subject to Ax = b dual ascent: use (sub-)gradient method to minimize g(ν) = b T ν +f ( A T ν) = sup x ( (b Ax) T ν f(x) ) algorithm x = argmin ˆx ν + = ν +t(ax b) ( f(ˆx)+ν T (Aˆx b) ) of interest if calculation of x is inexpensive (for example, separable) Dual methods 143