# Advances in Convex Optimization: Interior-point Methods, Cone Programming, and Applications

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Advances in Convex Optimization: Interior-point Methods, Cone Programming, and Applications Stephen Boyd Electrical Engineering Department Stanford University (joint work with Lieven Vandenberghe, UCLA) CDC 02 Las Vegas 12/11/02

2 Easy and Hard Problems

3 Least squares (LS) minimize Ax b 2 2 A R m n, b R m are parameters; x R n is variable have complete theory (existence & uniqueness, sensitivity analysis... ) several algorithms compute (global) solution reliably can solve dense problems with n = 1000 vbles, m = terms by exploiting structure (e.g., sparsity) can solve far larger problems... LS is a (widely used) technology CDC 02 Las Vegas 12/11/02 1

4 Linear program (LP) minimize c T x subject to a T i x b i, i =1,...,m c, a i R n are parameters; x R n is variable have nearly complete theory (existence & uniqueness, sensitivity analysis...) several algorithms compute (global) solution reliably can solve dense problems with n = 1000 vbles, m = constraints by exploiting structure (e.g., sparsity) can solve far larger problems... LP is a (widely used) technology CDC 02 Las Vegas 12/11/02 2

5 Quadratic program (QP) minimize Fx g 2 2 subject to a T i x b i, i =1,...,m a combination of LS & LP same story... QP is a technology solution methods reliable enough to be embedded in real-time control applications with little or no human oversight basis of model predictive control CDC 02 Las Vegas 12/11/02 3

6 The bad news LS, LP, and QP are exceptions most optimization problems, even some very simple looking ones, are intractable CDC 02 Las Vegas 12/11/02 4

7 Polynomial minimization minimize p(x) p is polynomial of degree d; x R n is variable except for special cases (e.g., d =2) this is a very difficult problem even sparse problems with size n =20,d =10are essentially intractable all algorithms known to solve this problem require effort exponential in n CDC 02 Las Vegas 12/11/02 5

8 What makes a problem easy or hard? classical view: linear is easy nonlinear is hard(er) CDC 02 Las Vegas 12/11/02 6

9 What makes a problem easy or hard? emerging (and correct) view:... the great watershed in optimization isn t between linearity and nonlinearity, but convexity and nonconvexity. R. Rockafellar, SIAM Review 1993 CDC 02 Las Vegas 12/11/02 7

10 Convex optimization minimize subject to f 0 (x) f 1 (x) 0,...,f m (x) 0 x R n is optimization variable; f i : R n R are convex: f i (λx +(1 λ)y) λf i (x)+(1 λ)f i (y) for all x, y, 0 λ 1 includes LS, LP, QP, and many others like LS, LP, and QP, convex problems are fundamentally tractable CDC 02 Las Vegas 12/11/02 8

11 Example: Robust LP minimize c T x subject to Prob(a T i x b i) η, i =1,...,m coefficient vectors a i IID, N (a i, Σ i ); η is required reliability for fixed x, a T i x is N (at i x, x T Σ i x) so for η = 50%, robust LP reduces to LP minimize c T x subject to a T i x b i, i =1,...,m and so is easily solved what about other values of η, e.g., η = 10%? η = 90%? CDC 02 Las Vegas 12/11/02 9

12 Hint {x Prob(a T i x b i ) η, i =1,...,m} η = 10% η = 50% η = 90% CDC 02 Las Vegas 12/11/02 10

13 That s right robust LP with reliability η = 90% is convex, and very easily solved robust LP with reliability η = 10% is not convex, and extremely difficult moral: very difficult and very easy problems can look quite similar (to the untrained eye) CDC 02 Las Vegas 12/11/02 11

14 Convex Analysis and Optimization

15 Convex analysis & optimization nice properties of convex optimization problems known since 1960s local solutions are global duality theory, optimality conditions simple solution methods like alternating projections convex analysis well developed by 1970s Rockafellar separating & supporting hyperplanes subgradient calculus CDC 02 Las Vegas 12/11/02 12

16 What s new (since 1990 or so) primal-dual interior-point (IP) methods extremely efficient, handle nonlinear large scale problems, polynomial-time complexity results, software implementations new standard problem classes generalizations of LP, with theory, algorithms, software extension to generalized inequalities semidefinite, cone programming... convex optimization is becoming a technology CDC 02 Las Vegas 12/11/02 13

17 Applications and uses lots of applications control, combinatorial optimization, signal processing, circuit design, communications,... robust optimization robust versions of LP, LS, other problems relaxations and randomization provide bounds, heuristics for solving hard problems CDC 02 Las Vegas 12/11/02 14

18 Recent history : interior-point methods for LP 1984: Karmarkar s interior-point LP method theory Ye, Renegar, Kojima, Todd, Monteiro, Roos,... practice Wright, Mehrotra, Vanderbei, Shanno, Lustig, : Nesterov & Nemirovsky s self-concordance analysis 1989 : LMIs and semidefinite programming in control 1990 : semidefinite programming in combinatorial optimization Alizadeh, Goemans, Williamson, Lovasz & Schrijver, Parrilo, : interior-point methods for nonlinear convex problems Nesterov & Nemirovsky, Overton, Todd, Ye, Sturm, : robust optimization Ben Tal, Nemirovsky, El Ghaoui,... CDC 02 Las Vegas 12/11/02 15

19 New Standard Convex Problem Classes

20 Some new standard convex problem classes second-order cone program (SOCP) geometric program (GP) (and entropy problems) semidefinite program (SDP) for these new problem classes we have complete duality theory, similar to LP good algorithms, and robust, reliable software wide variety of new applications CDC 02 Las Vegas 12/11/02 16

21 Second-order cone program second-order cone program (SOCP) has form minimize c T 0 x subject to A i x + b i 2 c T i x + d i, i =1,...,m with variable x R n includes LP and QP as special cases nondifferentiable when A i x + b i =0 new IP methods can solve (almost) as fast as LPs CDC 02 Las Vegas 12/11/02 17

22 Example: robust linear program where a i N(ā i, Σ i ) equivalent to minimize c T x subject to Prob(a T i x b i) η, i =1,...,m minimize c T x subject to ā T i x +Φ 1 (η) Σ 1/2 i x 2 1, i =1,...,m where Φ is (unit) normal CDF robust LP is an SOCP for η 0.5 (Φ(η) 0) CDC 02 Las Vegas 12/11/02 18

23 log-sum-exp function: Geometric program (GP) lse(x) = log (e x 1 + +e x n )... a smooth convex approximation of the max function geometric program: minimize lse(a 0 x + b 0 ) subject to lse(a i x + b i ) 0, i =1,...,m A i R m i n, b i R m i ; variable x R n CDC 02 Las Vegas 12/11/02 19

24 Entropy problems unnormalized negative entropy is convex function entr(x) = n x i log(x i /1 T x) i=1 defined for x i 0, 1 T x>0 entropy problem: minimize entr(a 0 x + b 0 ) subject to entr(a i x + b i ) 0, i =1,...,m A i R m i n, b i R m i CDC 02 Las Vegas 12/11/02 20

25 Solving GPs (and entropy problems) GP and entropy problems are duals (if we solve one, we solve the other) new IP methods can solve large scale GPs (and entropy problems) almost as fast as LPs applications in many areas: information theory, statistics communications, wireless power control digital and analog circuit design CDC 02 Las Vegas 12/11/02 21

26 CMOS analog/mixed-signal circuit design via GP given circuit cell: opamp, PLL, D/A, A/D, SC filter,... specs: power, area, bandwidth, nonlinearity, settling time,... IC fabrication process: TSMC 0.18µm mixed-signal,... find electronic design: device L & W, bias I & V, component values,... physical design: placement, layout, routing, GDSII,... CDC 02 Las Vegas 12/11/02 22

27 The challenges complex, multivariable, highly nonlinear problem dominating issue: robustness to model errors parameter variation unmodeled dynamics (sound familiar?) CDC 02 Las Vegas 12/11/02 23

28 Two-stage op-amp V dd M 8 M 5 M 7 I bias V in+ M 1 M 2 M 3 M 4 V in R c C c M 6 C L design variables: device lengths & widths, component values V ss constraints/objectives: power, area, bandwidth, gain, noise, slew rate, output swing,... CDC 02 Las Vegas 12/11/02 24

29 Op-amp design via GP express design problem as GP (using change of variables, and a few good approximations... ) 10s of vbles, 100s of constraints; solution time 1sec robust version: take 10 (or so) different parameter values ( PVT corners ) replicate all constraints for each parameter value get 100 vbles, 1000 constraints; solution time 2sec CDC 02 Las Vegas 12/11/02 25

30 Minimum noise versus power & BW w c =30MHz w c =60MHz w c =90MHz Minimum noise in nv/hz Power in mw CDC 02 Las Vegas 12/11/02 26

31 Cone Programming

32 Cone programming general cone program: minimize subject to c T x Ax K b generalized inequality Ax K b means b Ax K, a proper convex cone LP, QP, SOCP, GP can be expressed as cone programs CDC 02 Las Vegas 12/11/02 27

33 Semidefinite program semidefinite program (SDP): minimize subject to c T x x 1 A 1 + +x n A n B B, A i are symmetric matrices; variable is x R n constraint is linear matrix inequality (LMI) inequality is matrix inequality, i.e., K is positive semidefinite cone SDP is special case of cone program CDC 02 Las Vegas 12/11/02 28

34 Early SDP applications (around 1990 on) control (many) combinatorial optimization & graph theory (many) CDC 02 Las Vegas 12/11/02 29

35 More recent SDP applications structural optimization: Ben-Tal, Nemirovsky, Kocvara, Bendsoe,... signal processing: Vandenberghe, Stoica, Lorenz, Davidson, Shaked, Nguyen, Luo, Sturm, Balakrishnan, Saadat, Fu, de Souza,... circuit design: El Gamal, Vandenberghe, Boyd, Yun,... algebraic geometry: Parrilo, Sturmfels, Lasserre, de Klerk, Pressman, Pasechnik,... communications and information theory: Rasmussen, Rains, Abdi, Moulines,... quantum computing: Kitaev, Waltrous, Doherty, Parrilo, Spedalieri, Rains,... finance: Iyengar, Goldfarb,... CDC 02 Las Vegas 12/11/02 30

36 Convex optimization heirarchy convex problems cone problems more general SDP SOCP QP LS GP LP more specific CDC 02 Las Vegas 12/11/02 31

37 Relaxations & Randomization

38 Relaxations & randomization convex optimization is increasingly used to find good bounds for hard (i.e., nonconvex) problems, via relaxation as a heuristic for finding good suboptimal points, often via randomization CDC 02 Las Vegas 12/11/02 32

39 Example: Boolean least-squares Boolean least-squares problem: minimize Ax b 2 subject to x 2 i =1, i =1,...,n basic problem in digital communications could check all 2 n possible values of x... an NP-hard problem, and very hard in practice many heuristics for approximate solution CDC 02 Las Vegas 12/11/02 33

40 Boolean least-squares as matrix problem Ax b 2 = x T A T Ax 2b T Ax + b T b = Tr A T AX 2b T A T x + b T b where X = xx T hence can express BLS as minimize Tr A T AX 2b T Ax + b T b subject to X ii =1, X xx T, rank(x) =1... still a very hard problem CDC 02 Las Vegas 12/11/02 34

41 SDP relaxation for BLS ignore rank one constraint, and use X xx T [ X x x T 1 ] 0 to obtain SDP relaxation (with variables X, x) minimize Tr A T AX 2b T A T x + b T b [ ] X x subject to X ii =1, x T 0 1 optimal value of SDP gives lower bound for BLS if optimal matrix is rank one, we re done CDC 02 Las Vegas 12/11/02 35

42 Interpretation via randomization can think of variables X, x in SDP relaxation as defining a normal distribution z N(x, X xx T ), with E z 2 i =1 SDP objective is E Az b 2 suggests randomized method for BLS: find X, x, optimal for SDP relaxation generate z from N (x,x x x T ) take x = sgn(z) as approximate solution of BLS (can repeat many times and take best one) CDC 02 Las Vegas 12/11/02 36

43 Example (randomly chosen) parameters A R , b R 150 x R 100, so feasible set has points LS approximate solution: minimize Ax b s.t. x 2 = n, then round yields objective 8.7% over SDP relaxation bound randomized method: (using SDP optimal distribution) best of 20 samples: 3.1% over SDP bound best of 1000 samples: 2.6% over SDP bound CDC 02 Las Vegas 12/11/02 37

44 SDP bound LS solution frequency Ax b /(SDP bound) CDC 02 Las Vegas 12/11/02 38

45 Interior-Point Methods

46 Interior-point methods handle linear and nonlinear convex problems Nesterov & Nemirovsky based on Newton s method applied to barrier functions that trap x in interior of feasible region (hence the name IP) worst-case complexity theory: # Newton steps problem size in practice: # Newton steps between 10 & 50 (!) over wide range of problem dimensions, type, and data 1000 variables, constraints feasible on PC; far larger if structure is exploited readily available (commercial and noncommercial) packages CDC 02 Las Vegas 12/11/02 39

47 Typical convergence of IP method 10 2 duality gap SOCP GP LP 10 6 SDP # Newton steps LP, GP, SOCP, SDP with 100 variables CDC 02 Las Vegas 12/11/02 40

48 Typical effort versus problem dimensions 35 LPs with n vbles, 2n constraints 100 instances for each of 20 problem sizes avg & std dev shown Newton steps n CDC 02 Las Vegas 12/11/02 41

49 Computational effort per Newton step Newton step effort dominated by solving linear equations to find primal-dual search direction equations inherit structure from underlying problem equations same as for least-squares problem of similar size and structure conclusion: we can solve a convex problem with about the same effort as solving 30 least-squares problems CDC 02 Las Vegas 12/11/02 42

50 Problem structure common types of structure: sparsity state structure Toeplitz, circulant, Hankel; displacement rank Kronecker, Lyapunov structure symmetry CDC 02 Las Vegas 12/11/02 43

51 Exploiting sparsity well developed, since late 1970s direct (sparse factorizations) and iterative methods (CG, LSQR) standard in general purpose LP, QP, GP, SOCP implementations can solve problems with 10 5, 10 6 vbles, constraints (depending on sparsity pattern) CDC 02 Las Vegas 12/11/02 44

52 Exploiting structure in SDPs in combinatorial optimization, major effort to exploit structure structure is mostly (extreme) sparsity IP methods and others (bundle methods) used problems with LMIs, variables can be solved Ye, Wolkowicz, Burer, Monteiro... CDC 02 Las Vegas 12/11/02 45

53 Exploiting structure in SDPs in control structure includes sparsity, Kronecker/Lyapunov substantial improvements in order, for particular problem classes Balakrishnan & Vandenberghe, Hansson, Megretski, Parrilo, Rotea, Smith, Vandenberghe & Boyd, Van Dooren,......butno general solution yet CDC 02 Las Vegas 12/11/02 46

54 Conclusions

55 Conclusions convex optimization theory fairly mature; practice has advanced tremendously last decade qualitatively different from general nonlinear programming becoming a technology like LS, LP (esp., new problem classes), reliable enough for embedded applications cost only 30 more than least-squares, but far more expressive lots of applications still to be discovered CDC 02 Las Vegas 12/11/02 47

56 Some references Semidefinite Programming, SIAM Review 1996 Applications of Second-order Cone Programming, LAA 1999 Linear Matrix Inequalities in System and Control Theory, SIAM 1994 Interior-point Polynomial Algorithms in Convex Programming, SIAM 1994, Nesterov & Nemirovsky Lectures on Modern Convex Optimization, SIAM 2001, Ben Tal & Nemirovsky CDC 02 Las Vegas 12/11/02 48

57 Shameless promotion Convex Optimization, Boyd & Vandenberghe to be published 2003 good draft available at Stanford EE364 (UCLA EE236B) class web site as course reader CDC 02 Las Vegas 12/11/02 49

### Conic optimization: examples and software

Conic optimization: examples and software Etienne de Klerk Tilburg University, The Netherlands Etienne de Klerk (Tilburg University) Conic optimization: examples and software 1 / 16 Outline Conic optimization

### Optimisation et simulation numérique.

Optimisation et simulation numérique. Lecture 1 A. d Aspremont. M2 MathSV: Optimisation et simulation numérique. 1/106 Today Convex optimization: introduction Course organization and other gory details...

### An Overview Of Software For Convex Optimization. Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.

An Overview Of Software For Convex Optimization Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.edu In fact, the great watershed in optimization isn t between linearity

### LAGRANGIAN RELAXATION TECHNIQUES FOR LARGE SCALE OPTIMIZATION

LAGRANGIAN RELAXATION TECHNIQUES FOR LARGE SCALE OPTIMIZATION Kartik Sivaramakrishnan Department of Mathematics NC State University kksivara@ncsu.edu http://www4.ncsu.edu/ kksivara SIAM/MGSA Brown Bag

### An Introduction on SemiDefinite Program

An Introduction on SemiDefinite Program from the viewpoint of computation Hayato Waki Institute of Mathematics for Industry, Kyushu University 2015-10-08 Combinatorial Optimization at Work, Berlin, 2015

### Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.

Summer course on Convex Optimization Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota Interior-Point Methods: the rebirth of an old idea Suppose that f is

### Solving polynomial least squares problems via semidefinite programming relaxations

Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose

### An Accelerated First-Order Method for Solving SOS Relaxations of Unconstrained Polynomial Optimization Problems

An Accelerated First-Order Method for Solving SOS Relaxations of Unconstrained Polynomial Optimization Problems Dimitris Bertsimas, Robert M. Freund, and Xu Andy Sun December 2011 Abstract Our interest

### Convex Optimization. Lieven Vandenberghe University of California, Los Angeles

Convex Optimization Lieven Vandenberghe University of California, Los Angeles Tutorial lectures, Machine Learning Summer School University of Cambridge, September 3-4, 2009 Sources: Boyd & Vandenberghe,

### [1] F. Jarre and J. Stoer, Optimierung, Lehrbuch, Springer Verlag 2003.

References Lehrbuch: [1] F. Jarre and J. Stoer, Optimierung, Lehrbuch, Springer Verlag 2003. Referierte Zeitschriftenbeiträge: [2] F. Jarre, On the Convergence of the Method of Analytic Centers when applied

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### Real-Time Embedded Convex Optimization

Real-Time Embedded Convex Optimization Stephen Boyd joint work with Michael Grant, Jacob Mattingley, Yang Wang Electrical Engineering Department, Stanford University Spencer Schantz Lecture, Lehigh University,

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### TOMLAB - For fast and robust largescale optimization in MATLAB

The TOMLAB Optimization Environment is a powerful optimization and modeling package for solving applied optimization problems in MATLAB. TOMLAB provides a wide range of features, tools and services for

### Lecture 5: Conic Optimization: Overview

EE 227A: Conve Optimization and Applications January 31, 2012 Lecture 5: Conic Optimization: Overview Lecturer: Laurent El Ghaoui Reading assignment: Chapter 4 of BV. Sections 3.1-3.6 of WTB. 5.1 Linear

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

1 CONNECTIONS BETWEEN SEMI-INFINITE AND SEMIDEFINITE PROGRAMMING Lieven Vandenberghe 1) and Stephen boyd 2) 1) Electrical Engineering Department, University of California, Los Angeles 2) Electrical Engineering

### Completely Positive Cone and its Dual

On the Computational Complexity of Membership Problems for the Completely Positive Cone and its Dual Peter J.C. Dickinson Luuk Gijben July 3, 2012 Abstract Copositive programming has become a useful tool

### Las Vegas and Monte Carlo Randomized Algorithms for Systems and Control

Las Vegas and Monte Carlo Randomized Algorithms for Systems and Control Roberto Tempo IEIIT-CNR Politecnico di Torino roberto.tempo@polito.it BasarFest,, Urbana RT 2006 1 CSL UIUC Six months at CSL in

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### Advanced Lecture on Mathematical Science and Information Science I. Optimization in Finance

Advanced Lecture on Mathematical Science and Information Science I Optimization in Finance Reha H. Tütüncü Visiting Associate Professor Dept. of Mathematical and Computing Sciences Tokyo Institute of Technology

### Robust Geometric Programming is co-np hard

Robust Geometric Programming is co-np hard André Chassein and Marc Goerigk Fachbereich Mathematik, Technische Universität Kaiserslautern, Germany Abstract Geometric Programming is a useful tool with a

### Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

### Interior Point Methods and Linear Programming

Interior Point Methods and Linear Programming Robert Robere University of Toronto December 13, 2012 Abstract The linear programming problem is usually solved through the use of one of two algorithms: either

### Solving Large Scale Optimization Problems via Grid and Cluster Computing

Solving Large Scale Optimization Problems via Grid and Cluster Computing Katsuki Fujisawa, Masakazu Kojima Akiko Takeda and Makoto Yamashita Abstract. Solving large scale optimization problems requires

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### Interior-Point Methods for Linear Optimization

1 Interior-Point Methods for Linear Optimization Kees Roos Faculty of Information Technology and Systems (ITS) Optimization Group TU Delft, Mekelweg 4 P.O. Box 5031, 2628 CD Delft The Netherlands URL:

### Big Data Optimization at SAS

Big Data Optimization at SAS Imre Pólik et al. SAS Institute Cary, NC, USA Edinburgh, 2013 Outline 1 Optimization at SAS 2 Big Data Optimization at SAS The SAS HPA architecture Support vector machines

### The Application of Semidefinite Programming for Detection in CDMA

1442 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 19, NO. 8, AUGUST 2001 The Application of Semidefinite Programming for Detection in CDMA Peng Hui Tan, Student Member, IEEE, and Lars K. Rasmussen,

### Optimization Methods in Finance

Optimization Methods in Finance Gerard Cornuejols Reha Tütüncü Carnegie Mellon University, Pittsburgh, PA 15213 USA January 2006 2 Foreword Optimization models play an increasingly important role in financial

### Semidefinite characterisation of invariant measures for one-dimensional discrete dynamical systems

Semidefinite characterisation of invariant measures for one-dimensional discrete dynamical systems Didier Henrion 1,2,3 April 24, 2012 Abstract Using recent results on measure theory and algebraic geometry,

### 9. Numerical linear algebra background

Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### Lecture 6: Logistic Regression

Lecture 6: CS 194-10, Fall 2011 Laurent El Ghaoui EECS Department UC Berkeley September 13, 2011 Outline Outline Classification task Data : X = [x 1,..., x m]: a n m matrix of data points in R n. y { 1,

### INTERIOR POINT POLYNOMIAL TIME METHODS IN CONVEX PROGRAMMING

GEORGIA INSTITUTE OF TECHNOLOGY SCHOOL OF INDUSTRIAL AND SYSTEMS ENGINEERING LECTURE NOTES INTERIOR POINT POLYNOMIAL TIME METHODS IN CONVEX PROGRAMMING ISYE 8813 Arkadi Nemirovski On sabbatical leave from

### 1. spectral. Either global (e.g., Cheeger inequality,) or local.

CS369M: Algorithms for Modern Massive Data Set Analysis Lecture 12-11/04/2009 Introduction to Graph Partitioning Lecturer: Michael Mahoney Scribes: Noah Youngs and Weidong Shao *Unedited Notes 1 Graph

### A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems

A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems Sunyoung Kim, Masakazu Kojima and Kim-Chuan Toh October 2013 Abstract. We propose

### On SDP- and CP-relaxations and on connections between SDP-, CP and SIP

On SDP- and CP-relaations and on connections between SDP-, CP and SIP Georg Still and Faizan Ahmed University of Twente p 1/12 1. IP and SDP-, CP-relaations Integer program: IP) : min T Q s.t. a T j =

### Can linear programs solve NP-hard problems?

Can linear programs solve NP-hard problems? p. 1/9 Can linear programs solve NP-hard problems? Ronald de Wolf Linear programs Can linear programs solve NP-hard problems? p. 2/9 Can linear programs solve

### Scheduling Home Health Care with Separating Benders Cuts in Decision Diagrams

Scheduling Home Health Care with Separating Benders Cuts in Decision Diagrams André Ciré University of Toronto John Hooker Carnegie Mellon University INFORMS 2014 Home Health Care Home health care delivery

### A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

### 6. Cholesky factorization

6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

### CHAPTER 9. Integer Programming

CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral

### Model Predictive Control Lecture 5

Model Predictive Control Lecture 5 Klaus Trangbæk ktr@es.aau.dk Automation & Control Aalborg University Denmark. http://www.es.aau.dk/staff/ktr/mpckursus/mpckursus.html mpc5 p. 1 Exercise from last time

### A QCQP Approach to Triangulation. Chris Aholt, Sameer Agarwal, and Rekha Thomas University of Washington 2 Google, Inc.

A QCQP Approach to Triangulation 1 Chris Aholt, Sameer Agarwal, and Rekha Thomas 1 University of Washington 2 Google, Inc. 2 1 The Triangulation Problem X Given: -n camera matrices P i R 3 4 -n noisy observations

### MOSEK modeling manual

MOSEK modeling manual August 12, 2014 Contents 1 Introduction 1 2 Linear optimization 3 2.1 Introduction....................................... 3 2.1.1 Basic notions.................................. 3

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### Linear Matrix Inequalities in System and Control Theory

Linear Matrix Inequalities in System and Control Theory SIAM Studies in Applied Mathematics This series of monographs focuses on mathematics and its applications to problems of current concern to industry,

### Interior-Point Algorithms for Quadratic Programming

Interior-Point Algorithms for Quadratic Programming Thomas Reslow Krüth Kongens Lyngby 2008 IMM-M.Sc-2008-19 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK-2800

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Lecture 7: Finding Lyapunov Functions 1

Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.243j (Fall 2003): DYNAMICS OF NONLINEAR SYSTEMS by A. Megretski Lecture 7: Finding Lyapunov Functions 1

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### Lecture 11: 0-1 Quadratic Program and Lower Bounds

Lecture : - Quadratic Program and Lower Bounds (3 units) Outline Problem formulations Reformulation: Linearization & continuous relaxation Branch & Bound Method framework Simple bounds, LP bound and semidefinite

### A Log-Robust Optimization Approach to Portfolio Management

A Log-Robust Optimization Approach to Portfolio Management Dr. Aurélie Thiele Lehigh University Joint work with Ban Kawas Research partially supported by the National Science Foundation Grant CMMI-0757983

### . P. 4.3 Basic feasible solutions and vertices of polyhedra. x 1. x 2

4. Basic feasible solutions and vertices of polyhedra Due to the fundamental theorem of Linear Programming, to solve any LP it suffices to consider the vertices (finitely many) of the polyhedron P of the

### Big Data - Lecture 1 Optimization reminders

Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics

### Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

### Dual Methods for Total Variation-Based Image Restoration

Dual Methods for Total Variation-Based Image Restoration Jamylle Carter Institute for Mathematics and its Applications University of Minnesota, Twin Cities Ph.D. (Mathematics), UCLA, 2001 Advisor: Tony

### Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

### Absolute Value Programming

Computational Optimization and Aplications,, 1 11 (2006) c 2006 Springer Verlag, Boston. Manufactured in The Netherlands. Absolute Value Programming O. L. MANGASARIAN olvi@cs.wisc.edu Computer Sciences

### A Negative Result Concerning Explicit Matrices With The Restricted Isometry Property

A Negative Result Concerning Explicit Matrices With The Restricted Isometry Property Venkat Chandar March 1, 2008 Abstract In this note, we prove that matrices whose entries are all 0 or 1 cannot achieve

### Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

### How will the programme be delivered (e.g. inter-institutional, summerschools, lectures, placement, rotations, on-line etc.):

Titles of Programme: Hamilton Hamilton Institute Institute Structured PhD Structured PhD Minimum 30 credits. 15 of Programme which must be obtained from Generic/Transferable skills modules and 15 from

### LECTURE: INTRO TO LINEAR PROGRAMMING AND THE SIMPLEX METHOD, KEVIN ROSS MARCH 31, 2005

LECTURE: INTRO TO LINEAR PROGRAMMING AND THE SIMPLEX METHOD, KEVIN ROSS MARCH 31, 2005 DAVID L. BERNICK dbernick@soe.ucsc.edu 1. Overview Typical Linear Programming problems Standard form and converting

### Lecture Topic: Low-Rank Approximations

Lecture Topic: Low-Rank Approximations Low-Rank Approximations We have seen principal component analysis. The extraction of the first principle eigenvalue could be seen as an approximation of the original

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. #-approximation algorithm.

Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of three

### Virtual Landmarks for the Internet

Virtual Landmarks for the Internet Liying Tang Mark Crovella Boston University Computer Science Internet Distance Matters! Useful for configuring Content delivery networks Peer to peer applications Multiuser

### (Quasi-)Newton methods

(Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting

### Introduction to Inverse Problems (2 lectures)

Introduction to Inverse Problems (2 lectures) Summary Direct and inverse problems Examples of direct (forward) problems Deterministic and statistical points of view Ill-posed and ill-conditioned problems

### Theory and applications of Robust Optimization

Theory and applications of Robust Optimization Dimitris Bertsimas, David B. Brown, Constantine Caramanis July 6, 2007 Abstract In this paper we survey the primary research, both theoretical and applied,

### Linear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.

Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### Derive 5: The Easiest... Just Got Better!

Liverpool John Moores University, 1-15 July 000 Derive 5: The Easiest... Just Got Better! Michel Beaudin École de Technologie Supérieure, Canada Email; mbeaudin@seg.etsmtl.ca 1. Introduction Engineering

### Maximum-Margin Matrix Factorization

Maximum-Margin Matrix Factorization Nathan Srebro Dept. of Computer Science University of Toronto Toronto, ON, CANADA nati@cs.toronto.edu Jason D. M. Rennie Tommi S. Jaakkola Computer Science and Artificial

### Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they

### NP-hardness of Deciding Convexity of Quartic Polynomials and Related Problems

NP-hardness of Deciding Convexity of Quartic Polynomials and Related Problems Amir Ali Ahmadi, Alex Olshevsky, Pablo A. Parrilo, and John N. Tsitsiklis Abstract We show that unless P=NP, there exists no

### Online Learning and Competitive Analysis: a Unified Approach

Online Learning and Competitive Analysis: a Unified Approach Shahar Chen Online Learning and Competitive Analysis: a Unified Approach Research Thesis Submitted in partial fulfillment of the requirements

### The Fourth International DERIVE-TI92/89 Conference Liverpool, U.K., 12-15 July 2000. Derive 5: The Easiest... Just Got Better!

The Fourth International DERIVE-TI9/89 Conference Liverpool, U.K., -5 July 000 Derive 5: The Easiest... Just Got Better! Michel Beaudin École de technologie supérieure 00, rue Notre-Dame Ouest Montréal

### A Working Knowledge of Computational Complexity for an Optimizer

A Working Knowledge of Computational Complexity for an Optimizer ORF 363/COS 323 Instructor: Amir Ali Ahmadi TAs: Y. Chen, G. Hall, J. Ye Fall 2014 1 Why computational complexity? What is computational

### Computing a Nearest Correlation Matrix with Factor Structure

Computing a Nearest Correlation Matrix with Factor Structure Nick Higham School of Mathematics The University of Manchester higham@ma.man.ac.uk http://www.ma.man.ac.uk/~higham/ Joint work with Rüdiger

### Linear Algebra Review Part 2: Ax=b

Linear Algebra Review Part 2: Ax=b Edwin Olson University of Michigan The Three-Day Plan Geometry of Linear Algebra Vectors, matrices, basic operations, lines, planes, homogeneous coordinates, transformations

### Sensor networks, Internet, power grids, biological networks,

Big Data Sketching with Model Mismatch Sundeep Prabhakar Chepuri,YuZhang,GeertLeus,andG.B.Giannakis Faculty of EEMCS, Delft University of Technology, The Netherlands. Dept. of ECE and the DTC, University

### Algebra Unpacked Content For the new Common Core standards that will be effective in all North Carolina schools in the 2012-13 school year.

This document is designed to help North Carolina educators teach the Common Core (Standard Course of Study). NCDPI staff are continually updating and improving these tools to better serve teachers. Algebra

Quadratic Functions, Optimization, and Quadratic Forms Robert M. Freund February, 2004 2004 Massachusetts Institute of echnology. 1 2 1 Quadratic Optimization A quadratic optimization problem is an optimization

### Approximating Semidefinite Programs in Sublinear Time

Approximating Semidefinite Programs in Sublinear Time Dan Garber Technion - Israel Institute of Technology Haifa 3000 Israel dangar@cs.technion.ac.il Elad Hazan Technion - Israel Institute of Technology

### Optimal Design of a CMOS Op-Amp via Geometric Programming

IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. 20, NO. 1, JANUARY 2001 1 Optimal Design of a CMOS Op-Amp via Geometric Programming Maria del Mar Hershenson, Stephen

### A New Method for Estimating Maximum Power Transfer and Voltage Stability Margins to Mitigate the Risk of Voltage Collapse

A New Method for Estimating Maximum Power Transfer and Voltage Stability Margins to Mitigate the Risk of Voltage Collapse Bernie Lesieutre Dan Molzahn University of Wisconsin-Madison PSERC Webinar, October

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Solving Curved Linear Programs via the Shadow Simplex Method

Solving Curved Linear rograms via the Shadow Simplex Method Daniel Dadush 1 Nicolai Hähnle 2 1 Centrum Wiskunde & Informatica (CWI) 2 Bonn Universität CWI Scientific Meeting 01/15 Outline 1 Introduction

### Operations Research and Financial Engineering. Courses

Operations Research and Financial Engineering Courses ORF 504/FIN 504 Financial Econometrics Professor Jianqing Fan This course covers econometric and statistical methods as applied to finance. Topics

### Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

### Some Optimization Fundamentals

ISyE 3133B Engineering Optimization Some Optimization Fundamentals Shabbir Ahmed E-mail: sahmed@isye.gatech.edu Homepage: www.isye.gatech.edu/~sahmed Basic Building Blocks min or max s.t. objective as

### Solutions Of Some Non-Linear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR

Solutions Of Some Non-Linear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision

### 3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

### Fast Model Predictive Control Using Online Optimization Yang Wang and Stephen Boyd, Fellow, IEEE

IEEE TRANSACTIONS ON CONTROL SYSTEMS TECHNOLOGY, VOL 18, NO 2, MARCH 2010 267 Fast Model Predictive Control Using Online Optimization Yang Wang and Stephen Boyd, Fellow, IEEE Abstract A widely recognized

### Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering