# Subgradients. S. Boyd and L. Vandenberghe Notes for EE364b, Stanford University, Winter April 13, 2008

Save this PDF as:

Size: px
Start display at page:

## Transcription

2 f(z) f(x 1 ) + g T 1 (z x 1) f(x 2 ) + g T 2 (z x 2 ) f(x 2 ) + g T 3 (z x 2) x 1 x 2 Figure 1: At x 1, the convex function f is differentiable, and g 1 (which is the derivative of f at x 1 ) is the unique subgradient at x 1. At the point x 2, f is not differentiable. At this point, f has many subgradients: two subgradients, g 2 and g 3, are shown. epi f (g, 1) Figure 2: A vector g R n is a subgradient of f at x if and only if (g, 1) defines a supporting hyperplane to epi f at (x, f(x)). f(z) = z f(x) 1 z 1 x Figure 3: The absolute value function (left), and its subdifferential f(x) as a function of x (right). 2

3 2.1 Existence of subgradients If f is convex and x intdomf, then f(x) is nonempty and bounded. To establish that f(x), we apply the supporting hyperplane theorem to the convex set epi f at the boundary point (x, f(x)), to conclude the existence of a R n and b R, not both zero, such that [ a b ] T ([ z t ] [ x f(x) ]) for all (z, t) epi f. This implies b 0, and that = a T (z x) + b(t f(x)) 0 a T (z x) + b(f(z) f(x)) 0 for all z. If b 0, we can divide by b to obtain f(z) f(x) (a/b) T (z x), which shows that a/b f(x). Now we show that b 0, i.e., that the supporting hyperplane cannot be vertical. If b = 0 we conclude that a T (z x) 0 for all z domf. This is impossible since x int dom f. This discussion shows that a convex function has a subgradient at x if there is at least one nonvertical supporting hyperplane to epi f at (x, f(x)). This is the case, for example, if f is continuous. There are pathological convex functions which do not have subgradients at some points, but we will assume in the sequel that all convex functions are subdifferentiable (at every point in domf). 2.2 Subgradients of differentiable functions If f is convex and differentiable at x, then f(x) = { f(x)}, i.e., its gradient is its only subgradient. Conversely, if f is convex and f(x) = {g}, then f is differentiable at x and g = f(x). 2.3 The minimum of a nondifferentiable function A point x is a minimizer of a convex function f if and only if f is subdifferentiable at x and 0 f(x ), i.e., g = 0 is a subgradient of f at x. This follows directly from the fact that f(x) f(x ) for all x domf. This condition 0 f(x ) reduces to f(x ) = 0 if f is differentiable at x. 3

5 Example. Maximum of differentiable functions. Suppose f(x) = max i=1,...,m f i (x), where f i are convex and differentiable. Then we have f(x) = Co{ f i (x) f i (x) = f(x)}. At a point x where only one of the functions, say f k, is active, f is differentiable and has gradient f k (x). At a point x where several of the functions are active, f(x) is a polyhedron. Example. l 1 -norm. The l 1 -norm f(x) = x 1 = x x n is a nondifferentiable convex function of x. To find its subgradients, we note that f can expressed as the maximum of 2 n linear functions: x 1 = max{s T x s i { 1,1}}, so we can apply the rules for the subgradient of the maximum. The first step is to identify an active function s T x, i.e., find an s { 1,+1} n such that s T x = x 1. We can choose s i = +1 if x i > 0, and s i = 1 if x i < 0. If x i = 0, more than one function is active, and both s i = +1, and s i = 1 work. The function s T x is differentiable and has a unique subgradient s. We can therefore take +1 x i > 0 g i = 1 x i < 0 1 or + 1 x i = 0. The subdifferential is the convex hull of all subgradients that can be generated this way: f(x) = {g g 1,g T x = x 1 }. 3.5 Supremum Next we consider the extension to the supremum over an infinite number of functions, i.e., we consider f(x) = sup f α (x), α A where the functions f α are subdifferentiable. We only discuss the weak property. Suppose the supremum in the definition of f(x) is attained. Let β A be an index for which f β (x) = f(x), and let g f β (x). Then g f(x). If the supremum in the definition is not attained, the function may or may not be subdifferentiable at x, depending on the index set A. Assume however that A is compact (in some metric), and that the function α f α (x) is upper semi-continuous for each x. Then f(x) = Co { f α (x) f α (x) = f(x)}. 5

6 Example. Maximum eigenvalue of a symmetric matrix. Let f(x) = λ max (A(x)), where A(x) = A 0 + x 1 A x n A n, and A i S m. We can express f as the pointwise supremum of convex functions, f(x) = λ max (A(x)) = sup y T A(x)y. y 2 =1 Here the index set A is A = {y R n y 2 1 1}. Each of the functions f y (x) = y T A(x)y is affine in x for fixed y, as can be easily seen from y T A(x)y = y T A 0 y + x 1 y T A 1 y + + x n y T A n y, so it is differentiable with gradient f y (x) = (y T A 1 y,...,y T A n y). The active functions y T A(x)y are those associated with the eigenvectors corresponding to the maximum eigenvalue. Hence to find a subgradient, we compute an eigenvector y with eigenvalue λ max, normalized to have unit norm, and take g = (y T A 1 y,y T A 2 y,...,y T A n y). The index set in this example is {y y = 1} is a compact set. Therefore f(x) = Co { f y A(x)y = λ max (A(x))y, y = 1}. 3.6 Minimization over some variables The next subgradient calculus rule concerns functions of the form f(x) = inf y F(x, y) where F(x, y) is subdifferentiable and jointly convex in x and y. Again we only discuss the weak property. Suppose the infimum over y in the definition of f(ˆx) is attained at y = ŷ, i.e., f(ˆx) = F(ˆx, ŷ) and F(x, ŷ) F(ˆx, ŷ) for all x. Then there exists a g such that (g, 0) F(ˆx, ŷ), and any such g is a subgradient of f at ˆx. Strong property. Let x 2 be such that f(x 1 ) = inf x2 F(x 1, x 2 ). Then f(x 1 ) = {g 1 (g 1, 0) F(x 1, x 2 )}, (and the resulting subdifferential is independent of the choice of x 2 ). 3.7 Optimal value function of a convex optimization problem Suppose f : R m R p R is defined as the optimal value of a convex optimization problem in standard form, with z R n as optimization variable, minimize f 0 (z) subject to f i (z) x i, i = 1,...,m Az = y. (2) 6

7 In other words, f(x, y) = inf z F(x, y, z) where { f0 (z) f F(x, y, z) = i (z) x i, i = 1,..., m, Az = y + otherwise, which is jointly convex in x, y, z. Subgradients of f can be related to the dual problem of (2) as follows. Suppose we are interested in subdifferentiating f at (ˆx, ŷ). We can express the dual problem of (2) as maximize g(λ) x T λ y T ν (3) subject to λ 0. where ( ) m g(λ) = inf f 0 (z) + λ i f i (z) + ν T Az. z i=1 Suppose strong duality holds for problems (2) and (3) at x = ˆx and y = ŷ, and that the dual optimum is attained at λ, ν (for example, because Slater s condition holds). From the global perturbation inequalities we know that f(x, y) f(ˆx, ŷ) λ T (x ˆx) ν T (y ŷ) In other words, the dual optimal solution provides a subgradient: 4 Quasigradients (λ, ν ) f(ˆx, ŷ). If f(x) is quasiconvex, then g is a quasigradient at x 0 if g T (x x 0 ) 0 f(x) f(x 0 ), Geometrically, g defines a supporting hyperplane to the sublevel set {x f(x) f(x 0 )}. Note that the set of quasigradients at x 0 form a cone. Example. Linear fractional function. f(x) = at x+b c T x+d. Let ct x 0 + d > 0. Then g = a f(x 0 )c is a quasigradient at x 0. If c T x + d > 0, we have a T (x x 0 ) f(x 0 )c T (x x 0 ) = f(x) f(x 0 ). Example. Degree of a polynomial. Define f : R n R by f(a) = min{i a i+2 = = a n = 0}, i.e., the degree of the polynomial a 1 + a 2 t + + a n t n 1. Let a 0, and k = f(a), then g = sign(a k+1 )e k+1 is a quasigradient at a To see this, we note that implies b k+1 0. g T (b a) = sign(a k+1 )b k+1 a k+1 0 7

### Convex analysis and profit/cost/support functions

CALIFORNIA INSTITUTE OF TECHNOLOGY Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m

### 3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions

3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### {f 1 (U), U F} is an open cover of A. Since A is compact there is a finite subcover of A, {f 1 (U 1 ),...,f 1 (U n )}, {U 1,...

44 CHAPTER 4. CONTINUOUS FUNCTIONS In Calculus we often use arithmetic operations to generate new continuous functions from old ones. In a general metric space we don t have arithmetic, but much of it

### Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

### Convex Programming Tools for Disjunctive Programs

Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible

### by the matrix A results in a vector which is a reflection of the given

Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

### 3. Linear Programming and Polyhedral Combinatorics

Massachusetts Institute of Technology Handout 6 18.433: Combinatorial Optimization February 20th, 2009 Michel X. Goemans 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the

### Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

### 1 if 1 x 0 1 if 0 x 1

Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### Matrix Norms. Tom Lyche. September 28, Centre of Mathematics for Applications, Department of Informatics, University of Oslo

Matrix Norms Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 28, 2009 Matrix Norms We consider matrix norms on (C m,n, C). All results holds for

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### What is Linear Programming?

Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

Algorithms for large-scale convex optimization DTU 2010 3. Proximal gradient method introduction proximal mapping proximal gradient method convergence analysis accelerated proximal gradient method forward-backward

### MOSEK modeling manual

MOSEK modeling manual August 12, 2014 Contents 1 Introduction 1 2 Linear optimization 3 2.1 Introduction....................................... 3 2.1.1 Basic notions.................................. 3

### PART I. THE REAL NUMBERS

PART I. THE REAL NUMBERS This material assumes that you are already familiar with the real number system and the representation of the real numbers as points on the real line. I.1. THE NATURAL NUMBERS

### Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

### Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

### Undergraduate Notes in Mathematics. Arkansas Tech University Department of Mathematics

Undergraduate Notes in Mathematics Arkansas Tech University Department of Mathematics An Introductory Single Variable Real Analysis: A Learning Approach through Problem Solving Marcel B. Finan c All Rights

### Vectors, Gradient, Divergence and Curl.

Vectors, Gradient, Divergence and Curl. 1 Introduction A vector is determined by its length and direction. They are usually denoted with letters with arrows on the top a or in bold letter a. We will use

### ON A GLOBALIZATION PROPERTY

APPLICATIONES MATHEMATICAE 22,1 (1993), pp. 69 73 S. ROLEWICZ (Warszawa) ON A GLOBALIZATION PROPERTY Abstract. Let (X, τ) be a topological space. Let Φ be a class of realvalued functions defined on X.

### Vector Spaces II: Finite Dimensional Linear Algebra 1

John Nachbar September 2, 2014 Vector Spaces II: Finite Dimensional Linear Algebra 1 1 Definitions and Basic Theorems. For basic properties and notation for R N, see the notes Vector Spaces I. Definition

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### Ri and. i=1. S i N. and. R R i

The subset R of R n is a closed rectangle if there are n non-empty closed intervals {[a 1, b 1 ], [a 2, b 2 ],..., [a n, b n ]} so that R = [a 1, b 1 ] [a 2, b 2 ] [a n, b n ]. The subset R of R n is an

### 1 Norms and Vector Spaces

008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)

### 2 Complex Functions and the Cauchy-Riemann Equations

2 Complex Functions and the Cauchy-Riemann Equations 2.1 Complex functions In one-variable calculus, we study functions f(x) of a real variable x. Likewise, in complex analysis, we study functions f(z)

### Solutions to Linear Algebra Practice Problems 1. form (because the leading 1 in the third row is not to the right of the

Solutions to Linear Algebra Practice Problems. Determine which of the following augmented matrices are in row echelon from, row reduced echelon form or neither. Also determine which variables are free

### Optimisation et simulation numérique.

Optimisation et simulation numérique. Lecture 1 A. d Aspremont. M2 MathSV: Optimisation et simulation numérique. 1/106 Today Convex optimization: introduction Course organization and other gory details...

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

### A Problem With The Rational Numbers

Solvability of Equations Solvability of Equations 1. In fields, linear equations ax + b = 0 are solvable. Solvability of Equations 1. In fields, linear equations ax + b = 0 are solvable. 2. Quadratic equations

### (Basic definitions and properties; Separation theorems; Characterizations) 1.1 Definition, examples, inner description, algebraic properties

Lecture 1 Convex Sets (Basic definitions and properties; Separation theorems; Characterizations) 1.1 Definition, examples, inner description, algebraic properties 1.1.1 A convex set In the school geometry

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### 1. Prove that the empty set is a subset of every set.

1. Prove that the empty set is a subset of every set. Basic Topology Written by Men-Gen Tsai email: b89902089@ntu.edu.tw Proof: For any element x of the empty set, x is also an element of every set since

### 4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns

L. Vandenberghe EE133A (Spring 2016) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows

### No: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics

No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results

### Separation Properties for Locally Convex Cones

Journal of Convex Analysis Volume 9 (2002), No. 1, 301 307 Separation Properties for Locally Convex Cones Walter Roth Department of Mathematics, Universiti Brunei Darussalam, Gadong BE1410, Brunei Darussalam

### 1 Eigenvalues and Eigenvectors

Math 20 Chapter 5 Eigenvalues and Eigenvectors Eigenvalues and Eigenvectors. Definition: A scalar λ is called an eigenvalue of the n n matrix A is there is a nontrivial solution x of Ax = λx. Such an x

### Walrasian Demand. u(x) where B(p, w) = {x R n + : p x w}.

Walrasian Demand Econ 2100 Fall 2015 Lecture 5, September 16 Outline 1 Walrasian Demand 2 Properties of Walrasian Demand 3 An Optimization Recipe 4 First and Second Order Conditions Definition Walrasian

### x a x 2 (1 + x 2 ) n.

Limits and continuity Suppose that we have a function f : R R. Let a R. We say that f(x) tends to the limit l as x tends to a; lim f(x) = l ; x a if, given any real number ɛ > 0, there exists a real number

### One side James Compactness Theorem

One side James Compactness Theorem 1 1 Department of Mathematics University of Murcia Topological Methods in Analysis and Optimization. On the occasion of the 70th birthday of Prof. Petar Kenderov A birthday

### TOPIC 3: CONTINUITY OF FUNCTIONS

TOPIC 3: CONTINUITY OF FUNCTIONS. Absolute value We work in the field of real numbers, R. For the study of the properties of functions we need the concept of absolute value of a number. Definition.. Let

### Mathematical Methods of Engineering Analysis

Mathematical Methods of Engineering Analysis Erhan Çinlar Robert J. Vanderbei February 2, 2000 Contents Sets and Functions 1 1 Sets................................... 1 Subsets.............................

### DIFFERENTIABILITY OF COMPLEX FUNCTIONS. Contents

DIFFERENTIABILITY OF COMPLEX FUNCTIONS Contents 1. Limit definition of a derivative 1 2. Holomorphic functions, the Cauchy-Riemann equations 3 3. Differentiability of real functions 5 4. A sufficient condition

### Math 4310 Handout - Quotient Vector Spaces

Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable

### x if x 0, x if x < 0.

Chapter 3 Sequences In this chapter, we discuss sequences. We say what it means for a sequence to converge, and define the limit of a convergent sequence. We begin with some preliminary results about the

### Arrangements And Duality

Arrangements And Duality 3.1 Introduction 3 Point configurations are tbe most basic structure we study in computational geometry. But what about configurations of more complicated shapes? For example,

### TECHNICAL NOTE. Proof of a Conjecture by Deutsch, Li, and Swetits on Duality of Optimization Problems1

JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 102, No. 3, pp. 697-703, SEPTEMBER 1999 TECHNICAL NOTE Proof of a Conjecture by Deutsch, Li, and Swetits on Duality of Optimization Problems1 H. H.

### Chapter 1. Metric Spaces. Metric Spaces. Examples. Normed linear spaces

Chapter 1. Metric Spaces Metric Spaces MA222 David Preiss d.preiss@warwick.ac.uk Warwick University, Spring 2008/2009 Definitions. A metric on a set M is a function d : M M R such that for all x, y, z

### ALMOST COMMON PRIORS 1. INTRODUCTION

ALMOST COMMON PRIORS ZIV HELLMAN ABSTRACT. What happens when priors are not common? We introduce a measure for how far a type space is from having a common prior, which we term prior distance. If a type

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### Notes on Symmetric Matrices

CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.

### Convex Optimization. Lieven Vandenberghe University of California, Los Angeles

Convex Optimization Lieven Vandenberghe University of California, Los Angeles Tutorial lectures, Machine Learning Summer School University of Cambridge, September 3-4, 2009 Sources: Boyd & Vandenberghe,

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### LS.6 Solution Matrices

LS.6 Solution Matrices In the literature, solutions to linear systems often are expressed using square matrices rather than vectors. You need to get used to the terminology. As before, we state the definitions

### Error Bound for Classes of Polynomial Systems and its Applications: A Variational Analysis Approach

Outline Error Bound for Classes of Polynomial Systems and its Applications: A Variational Analysis Approach The University of New South Wales SPOM 2013 Joint work with V. Jeyakumar, B.S. Mordukhovich and

### Multiplicity. Chapter 6

Chapter 6 Multiplicity The fundamental theorem of algebra says that any polynomial of degree n 0 has exactly n roots in the complex numbers if we count with multiplicity. The zeros of a polynomial are

### Chapter 4. Duality. 4.1 A Graphical Example

Chapter 4 Duality Given any linear program, there is another related linear program called the dual. In this chapter, we will develop an understanding of the dual linear program. This understanding translates

### Limits and Continuity

Math 20C Multivariable Calculus Lecture Limits and Continuity Slide Review of Limit. Side limits and squeeze theorem. Continuous functions of 2,3 variables. Review: Limits Slide 2 Definition Given a function

### CHAPTER 9. Integer Programming

CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral

### MTH4100 Calculus I. Lecture notes for Week 8. Thomas Calculus, Sections 4.1 to 4.4. Rainer Klages

MTH4100 Calculus I Lecture notes for Week 8 Thomas Calculus, Sections 4.1 to 4.4 Rainer Klages School of Mathematical Sciences Queen Mary University of London Autumn 2009 Theorem 1 (First Derivative Theorem

### Additional Exercises for Convex Optimization

Additional Exercises for Convex Optimization Stephen Boyd Lieven Vandenberghe February 11, 2016 This is a collection of additional exercises, meant to supplement those found in the book Convex Optimization,

### HOMEWORK 4 SOLUTIONS. All questions are from Vector Calculus, by Marsden and Tromba

HOMEWORK SOLUTIONS All questions are from Vector Calculus, by Marsden and Tromba Question :..6 Let w = f(x, y) be a function of two variables, and let x = u + v, y = u v. Show that Solution. By the chain

### Section 6.1 - Inner Products and Norms

Section 6.1 - Inner Products and Norms Definition. Let V be a vector space over F {R, C}. An inner product on V is a function that assigns, to every ordered pair of vectors x and y in V, a scalar in F,

### MINIMIZATION OF ENTROPY FUNCTIONALS UNDER MOMENT CONSTRAINTS. denote the family of probability density functions g on X satisfying

MINIMIZATION OF ENTROPY FUNCTIONALS UNDER MOMENT CONSTRAINTS I. Csiszár (Budapest) Given a σ-finite measure space (X, X, µ) and a d-tuple ϕ = (ϕ 1,..., ϕ d ) of measurable functions on X, for a = (a 1,...,

### International Doctoral School Algorithmic Decision Theory: MCDA and MOO

International Doctoral School Algorithmic Decision Theory: MCDA and MOO Lecture 2: Multiobjective Linear Programming Department of Engineering Science, The University of Auckland, New Zealand Laboratoire

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete

### Class Meeting # 1: Introduction to PDEs

MATH 18.152 COURSE NOTES - CLASS MEETING # 1 18.152 Introduction to PDEs, Fall 2011 Professor: Jared Speck Class Meeting # 1: Introduction to PDEs 1. What is a PDE? We will be studying functions u = u(x

### Lecture 13 Linear quadratic Lyapunov theory

EE363 Winter 28-9 Lecture 13 Linear quadratic Lyapunov theory the Lyapunov equation Lyapunov stability conditions the Lyapunov operator and integral evaluating quadratic integrals analysis of ARE discrete-time

### PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5.

PUTNAM TRAINING POLYNOMIALS (Last updated: November 17, 2015) Remark. This is a list of exercises on polynomials. Miguel A. Lerma Exercises 1. Find a polynomial with integral coefficients whose zeros include

### Lecture 2: August 29. Linear Programming (part I)

10-725: Convex Optimization Fall 2013 Lecture 2: August 29 Lecturer: Barnabás Póczos Scribes: Samrachana Adhikari, Mattia Ciollaro, Fabrizio Lecci Note: LaTeX template courtesy of UC Berkeley EECS dept.

### Mathematical finance and linear programming (optimization)

Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may

### Monotone maps of R n are quasiconformal

Monotone maps of R n are quasiconformal K. Astala, T. Iwaniec and G. Martin For Neil Trudinger Abstract We give a new and completely elementary proof showing that a δ monotone mapping of R n, n is K quasiconformal

### Summer course on Convex Optimization. Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.

Summer course on Convex Optimization Fifth Lecture Interior-Point Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota Interior-Point Methods: the rebirth of an old idea Suppose that f is

### 1 Local Brouwer degree

1 Local Brouwer degree Let D R n be an open set and f : S R n be continuous, D S and c R n. Suppose that the set f 1 (c) D is compact. (1) Then the local Brouwer degree of f at c in the set D is defined.

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### Continuity of the Perron Root

Linear and Multilinear Algebra http://dx.doi.org/10.1080/03081087.2014.934233 ArXiv: 1407.7564 (http://arxiv.org/abs/1407.7564) Continuity of the Perron Root Carl D. Meyer Department of Mathematics, North

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### Linear Algebra I. Ronald van Luijk, 2012

Linear Algebra I Ronald van Luijk, 2012 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents 1. Vector spaces 3 1.1. Examples 3 1.2. Fields 4 1.3. The field of complex numbers. 6 1.4.

### TOPIC 4: DERIVATIVES

TOPIC 4: DERIVATIVES 1. The derivative of a function. Differentiation rules 1.1. The slope of a curve. The slope of a curve at a point P is a measure of the steepness of the curve. If Q is a point on the

### The Mean Value Theorem

The Mean Value Theorem THEOREM (The Extreme Value Theorem): If f is continuous on a closed interval [a, b], then f attains an absolute maximum value f(c) and an absolute minimum value f(d) at some numbers

### Linear Algebra Notes

Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note

### Direct Algorithms for Checking Coherence and Making Inferences from Conditional Probability Assessments

Direct Algorithms for Checking Coherence and Making Inferences from Conditional Probability Assessments Peter Walley, Renato Pelessoni a and Paolo Vicig a a Dipartimento di Matematica Applicata B. de Finetti,

### Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain

Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain 1. Orthogonal matrices and orthonormal sets An n n real-valued matrix A is said to be an orthogonal

### Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

### Review for Calculus Rational Functions, Logarithms & Exponentials

Definition and Domain of Rational Functions A rational function is defined as the quotient of two polynomial functions. F(x) = P(x) / Q(x) The domain of F is the set of all real numbers except those for

### Roots and Coefficients of a Quadratic Equation Summary

Roots and Coefficients of a Quadratic Equation Summary For a quadratic equation with roots α and β: Sum of roots = α + β = and Product of roots = αβ = Symmetrical functions of α and β include: x = and

### n 2 + 4n + 3. The answer in decimal form (for the Blitz): 0, 75. Solution. (n + 1)(n + 3) = n + 3 2 lim m 2 1

. Calculate the sum of the series Answer: 3 4. n 2 + 4n + 3. The answer in decimal form (for the Blitz):, 75. Solution. n 2 + 4n + 3 = (n + )(n + 3) = (n + 3) (n + ) = 2 (n + )(n + 3) ( 2 n + ) = m ( n

### some algebra prelim solutions

some algebra prelim solutions David Morawski August 19, 2012 Problem (Spring 2008, #5). Show that f(x) = x p x + a is irreducible over F p whenever a F p is not zero. Proof. First, note that f(x) has no

Quadratic Functions, Optimization, and Quadratic Forms Robert M. Freund February, 2004 2004 Massachusetts Institute of echnology. 1 2 1 Quadratic Optimization A quadratic optimization problem is an optimization

### On Minimal Valid Inequalities for Mixed Integer Conic Programs

On Minimal Valid Inequalities for Mixed Integer Conic Programs Fatma Kılınç Karzan June 27, 2013 Abstract We study mixed integer conic sets involving a general regular (closed, convex, full dimensional,