10. Proximal point method


 Edward Phelps
 2 years ago
 Views:
Transcription
1 L. Vandenberghe EE236C Spring ) 10. Proximal point method proximal point method augmented Lagrangian method MoreauYosida smoothing 101
2 Proximal point method a conceptual algorithm for minimizing a closed convex function f: x k) = prox tk fx k 1) ) = argmin u fu)+ 1 ) u x k 1) 2 2 2t k can be viewed as proximal gradient method page 62) with gx) = 0 of interest if prox evaluations are much easier than minimizing f directly a practical algorithm if inexact prox evaluations are used step size t k > 0 affects number of iterations, cost of prox evaluations basis of the augmented Lagrangian method Proximal point method 102
3 Convergence assumptions f is closed and convex hence, prox tf x) is uniquely defined for all x) optimal value f is finite and attained at x result fx k) ) f x 0) x 2 2 for k 1 2 k t i i=1 implies convergence if i t i rate is 1/k if t i is fixed or variable but bounded away from zero t i is arbitrary; however cost of prox evaluations will depend on t i Proximal point method 103
4 proof: apply analysis of proximal gradient method lect. 6) with gx) = 0 since g is zero, inequality 1) on page 612 holds for any t > 0 from page 614, fx i) ) is nonincreasing and t i fx i) ) f ) 1 2 ) x i) x 2 2 x i 1) x 2 2 combine inequalities for i = 1 to i = k to get k t i ) i=1 ) fx k) ) f ) k i=1 t i fx i) ) f ) 1 2 x0) x 2 2 Proximal point method 104
5 Accelerated proximal point algorithms FISTA take gx) = 0 on p.78): choose x 0) = x 1) and for k 1 x k) = prox tk f ) x k 1) 1 θ k 1 +θ k x k 1) x k 2) ) θ k 1 Nesterov s 2nd method p. 723): choose x 0) = v 0) and for k 1 v k) = prox tk /θ k )fv k 1) ), x k) = 1 θ k )x k 1) +θ k v k) possible choices of parameters fixed steps: t k = t and θ k = 2/k +1) variable steps: choose any t k > 0, θ 1 = 1, and for k > 1, solve θ k from 1 θ k )t k θ 2 k = t k 1 θ 2 k 1 Proximal point method 105
6 Convergence assumptions f is closed and convex hence, prox tf x) is uniquely defined for all x) optimal value f is finite and attained at x result fx k) ) f 2 x 0) x ) t 1 + k 2 for k 1 ti i=2 implies convergence if i ti rate is 1/k 2 if t i is fixed or variable but bounded away from zero Proximal point method 106
7 proof: follows from analysis in lecture 7 with gx) = 0 since g is zero, first inequalities on p and p hold for any t > 0 therefore the conclusion on p and p holds: fx k) ) f θ2 k 2t k x 0) x 2 2 for fixed step size t k = t, θ k = 2/k +1), θ 2 k 2t k = 2 k +1) 2 t for variable step size, we proved on page 719 that θ 2 k 2t k 2 2 t 1 + k ti ) 2 i=2 Proximal point method 107
8 Outline proximal point method augmented Lagrangian method MoreauYosida smoothing
9 Standard problem form minimize fx) + gax) f : R n R and g : R m R are closed convex functions; A R m n equivalent formulation with auxiliary variable y: minimize fx) + gy) subject to Ax = y examples g is indicator function of {b}: minimize fx) subject to Ax = b g is indicator function of C: minimize fx) subject to Ax C gy) = y b : minimize fx)+ Ax b Proximal point method 108
10 Dual problem Lagrangian of reformulated problem) dual problem Lx,y,z) = fx)+gy)+z T Ax y) maximize inf x,y Lx,y,z) = f A T z) g z) optimality conditions: x, y, z are optimal if x, y are feasible: x domf, y domg, and Ax = y x and y minimize Lx,y,z): A T z fx) and z gy) augmented Lagrangian method: proximal point method applied to dual Proximal point method 109
11 Proximal mapping of dual function proximal mapping of hz) = f A T z)+g z) is defined as prox th z) = argmin u f A T u)+g u)+ 1 ) 2t u z 2 2 dual expression: prox th z) = z +taˆx ŷ) where ˆx,ŷ) = argmin x,y fx)+gy)+z T Ax y)+ t ) 2 Ax y 2 2 ˆx, ŷ minimize augmented Lagrangian Lagrangian + quadratic penalty) Proximal point method 1010
12 proof write augmented Lagrangian minimization as minimize over x, y, w) fx)+gy)+ t 2 w 2 2 subject to Ax y +z/t = w optimality conditions u is multiplier for equality): Ax y + 1 t z = w, AT u fx), u gy), tw = u eliminating x, y, w gives u = z +tax y) and 0 A f A T u)+ g u)+ 1 t u z) this is the optimality condition for problem in definiton of u = prox th z) Proximal point method 1011
13 choose initial z 0) and repeat: Augmented Lagrangian method 1. minimize augmented Lagrangian ˆx,ŷ) = argmin x,y fx)+gy)+ t k 2 Ax y +1/t k )z k 1) 2 2 ) 2. dual update z k) = z k 1) +t k Aˆx ŷ) also known as method of multipliers, Bregman iteration this is the proximal point method applied to the dual problem as variants, can apply the fast proximal point methods to the dual usually implemented with inexact minimization in step 1 Proximal point method 1012
14 Examples minimize fx) + gax) equality constraints g is indicator of {b}): ˆx = argmin x z := z +taˆx b) fx)+z T Ax+ t ) 2 Ax b 2 2 set constraint g indicator of convex set C): ˆx = argmin fx)+ t2 ) dax+z/t)2 z := z +taˆx PAˆx+z/t)) Pu) is projection of u on C, du) = u Pu) 2 is Euclidean distance Proximal point method 1013
15 Outline proximal point method augmented Lagrangian method MoreauYosida smoothing
16 MoreauYosida smoothing MoreauYosida regularization Moreau envelope) of closed convex f is f t) x) = inf u fu)+ 1 ) 2t u x 2 2 = f prox tf x) ) + 1 2t with t > 0) proxtf x) x 2 2 immediate properties f t) is convex infimum over u of a convex function of x, u) domain of f t) is R n recall that prox tf x) is defined for all x) Proximal point method 1014
17 Examples indicator function: smoothed f is squared Euclidean distance fx) = I C x), f t) x) = 1 2t dx)2 1norm: smoothed function is Huber penalty fx) = x 1, f t) x) = n φ t x k ) k=1 φ t z) = { z 2 /2t) z t z t/2 z t φtz) t/2 z t/2 Proximal point method 1015
18 Conjugate of Moreau envelope f t) x) = inf u fu)+ 1 ) 2t u x 2 2 f t) is infimal convolution of fu) and v 2 2/2t) see page 814): f t) x) = inf fu)+ 1 ) u+v=x 2t v 2 2 from page 814, conjugate is sum of conjugates of fu) and v 2 2/2t): f t) ) y) = f y)+ t 2 y 2 2 hence, conjugate is strongly convex with parameter t Proximal point method 1016
19 Gradient of Moreau envelope f t) x) = sup y x T y f y) t ) 2 y 2 2 maximizer in definition is unique and satisfies x ty f y) y fx ty) maximizing y is the gradient of f t) : from pages 67 and 94, f t) x) = 1 t x proxtf x) ) = prox 1/t)f x/t) gradient f t) is Lipschitz continuous with constant 1/t follows from nonexpansiveness of prox; see page 69) Proximal point method 1017
20 Interpretation of proximal point algorithm apply gradient method to minimize Moreau envelope minimize f t) x) = inf u fu)+ 1 ) 2t u x 2 2 this is an exact smooth reformulation of problem of minimizing fx): solution x is minimizer of f f t) is differentiable with Lipschitz continuous gradient L = 1/t) gradient update: with fixed t k = 1/L = t x k) = x k 1) t f t) x k 1) ) = prox tf x k 1) )... the proximal point update with constant step size t k = t Proximal point method 1018
21 Interpretation of augmented Lagrangian algorithm ˆx, ŷ) = argmin x,y z := z +taˆx ŷ) fx)+gy)+ t ) 2 Ax y +1/t)z 2 2 with fixed t, dual update is gradient step applied to smoothed dual if we eliminate y, primal step can be interpreted as smoothing g: ˆx = argmin x fx)+g1/t) Ax+1/t)z) ) example: minimize fx)+ Ax b 1 ˆx = argmin x fx)+φ1/t Ax b+1/t)z) ) with φ 1/t the Huber penalty applied componentwise page 1015) Proximal point method 1019
22 References proximal point algorithm and fast proximal point algorithm O. Güler, On the convergence of the proximal point algorithm for convex minimization, SIAM J. Control and Optimization 1991) O. Güler, New proximal point algorithms for convex minimization, SIOPT 1992) O. Güler, Augmented Lagrangian algorithm for linear programming, JOTA 1992) augmented Lagrangian algorithm D.P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods 1982) Proximal point method 1020
3. Proximal gradient method
Algorithms for largescale convex optimization DTU 2010 3. Proximal gradient method introduction proximal mapping proximal gradient method convergence analysis accelerated proximal gradient method forwardbackward
More information2.3 Convex Constrained Optimization Problems
42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions
More informationDuality in General Programs. Ryan Tibshirani Convex Optimization 10725/36725
Duality in General Programs Ryan Tibshirani Convex Optimization 10725/36725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T
More informationGI01/M055 Supervised Learning Proximal Methods
GI01/M055 Supervised Learning Proximal Methods Massimiliano Pontil (based on notes by Luca Baldassarre) (UCL) Proximal Methods 1 / 20 Today s Plan Problem setting Convex analysis concepts Proximal operators
More informationIntroduction to Convex Optimization for Machine Learning
Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning
More informationFurther Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1
Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing
More informationProximal mapping via network optimization
L. Vandenberghe EE236C (Spring 234) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:
More informationAdaptive Online Gradient Descent
Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650
More informationNonlinear Optimization: Algorithms 3: Interiorpoint methods
Nonlinear Optimization: Algorithms 3: Interiorpoint methods INSEAD, Spring 2006 JeanPhilippe Vert Ecole des Mines de Paris JeanPhilippe.Vert@mines.org Nonlinear optimization c 2006 JeanPhilippe Vert,
More informationDownloaded 08/02/12 to 132.68.160.18. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.
SIAM J. OPTIM. Vol. 22, No. 2, pp. 557 580 c 2012 Society for Industrial and Applied Mathematics SMOOTHING AND FIRST ORDER METHODS: A UNIFIED FRAMEWORK AMIR BECK AND MARC TEBOULLE Abstract. We propose
More informationA NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION
1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of
More informationError Bound for Classes of Polynomial Systems and its Applications: A Variational Analysis Approach
Outline Error Bound for Classes of Polynomial Systems and its Applications: A Variational Analysis Approach The University of New South Wales SPOM 2013 Joint work with V. Jeyakumar, B.S. Mordukhovich and
More informationFirst Order Algorithms in Variational Image Processing
First Order Algorithms in Variational Image Processing M. Burger, A. Sawatzky, and G. Steidl today Introduction Variational methods in imaging are nowadays developing towards a quite universal and flexible
More informationBig Data  Lecture 1 Optimization reminders
Big Data  Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data  Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics
More informationSummer course on Convex Optimization. Fifth Lecture InteriorPoint Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.
Summer course on Convex Optimization Fifth Lecture InteriorPoint Methods (1) Michel Baes, K.U.Leuven Bharath Rangarajan, U.Minnesota InteriorPoint Methods: the rebirth of an old idea Suppose that f is
More informationMinimize subject to. x S R
Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such
More informationRegression Using Support Vector Machines: Basic Foundations
Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering
More informationOptical Flow. Thomas Pock
Optical Flow Thomas Pock 1 Optical Flow (II) Content Global approaches (HornSchunck, TVL1) Coarsetofine warping 2 The Horn and Schunck (HS) Method 3 The Horn and Schunck (HS) Method Global energy to
More informationConvex Optimization SVM s and Kernel Machines
Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.
More informationConvex analysis and profit/cost/support functions
CALIFORNIA INSTITUTE OF TECHNOLOGY Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m
More informationNonlinear Programming Methods.S2 Quadratic Programming
Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective
More informationSeveral Views of Support Vector Machines
Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min
More informationOnline Convex Optimization
E0 370 Statistical Learning heory Lecture 19 Oct 22, 2013 Online Convex Optimization Lecturer: Shivani Agarwal Scribe: Aadirupa 1 Introduction In this lecture we shall look at a fairly general setting
More informationOnline Learning of Optimal Strategies in Unknown Environments
1 Online Learning of Optimal Strategies in Unknown Environments Santiago Paternain and Alejandro Ribeiro Abstract Define an environment as a set of convex constraint functions that vary arbitrarily over
More informationDate: April 12, 2001. Contents
2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........
More informationAbsolute Value Programming
Computational Optimization and Aplications,, 1 11 (2006) c 2006 Springer Verlag, Boston. Manufactured in The Netherlands. Absolute Value Programming O. L. MANGASARIAN olvi@cs.wisc.edu Computer Sciences
More informationCourse 221: Analysis Academic year , First Semester
Course 221: Analysis Academic year 200708, First Semester David R. Wilkins Copyright c David R. Wilkins 1989 2007 Contents 1 Basic Theorems of Real Analysis 1 1.1 The Least Upper Bound Principle................
More informationmax cx s.t. Ax c where the matrix A, cost vector c and right hand side b are given and x is a vector of variables. For this example we have x
Linear Programming Linear programming refers to problems stated as maximization or minimization of a linear function subject to constraints that are linear equalities and inequalities. Although the study
More information1 Norms and Vector Spaces
008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)
More informationDistributed Machine Learning and Big Data
Distributed Machine Learning and Big Data Sourangshu Bhattacharya Dept. of Computer Science and Engineering, IIT Kharagpur. http://cse.iitkgp.ac.in/~sourangshu/ August 21, 2015 Sourangshu Bhattacharya
More informationDefinition of a Linear Program
Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1
More informationStanford University CS261: Optimization Handout 6 Luca Trevisan January 20, In which we introduce the theory of duality in linear programming.
Stanford University CS261: Optimization Handout 6 Luca Trevisan January 20, 2011 Lecture 6 In which we introduce the theory of duality in linear programming 1 The Dual of Linear Program Suppose that we
More informationA LagrangianDNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems
A LagrangianDNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems Sunyoung Kim, Masakazu Kojima and KimChuan Toh October 2013 Abstract. We propose
More informationNumerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen
(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained
More informationAM 221: Advanced Optimization Spring Prof. Yaron Singer Lecture 7 February 19th, 2014
AM 22: Advanced Optimization Spring 204 Prof Yaron Singer Lecture 7 February 9th, 204 Overview In our previous lecture we saw the application of the strong duality theorem to game theory, and then saw
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions logconcave and logconvex functions
More informationBy W.E. Diewert. July, Linear programming problems are important for a number of reasons:
APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization)
More information(Quasi)Newton methods
(Quasi)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable nonlinear function g, x such that g(x) = 0, where g : R n R n. Given a starting
More informationNotes on weak convergence (MAT Spring 2006)
Notes on weak convergence (MAT4380  Spring 2006) Kenneth H. Karlsen (CMA) February 2, 2006 1 Weak convergence In what follows, let denote an open, bounded, smooth subset of R N with N 2. We assume 1 p
More informationTable 1: Summary of the settings and parameters employed by the additive PA algorithm for classification, regression, and uniclass.
Online PassiveAggressive Algorithms Koby Crammer Ofer Dekel Shai ShalevShwartz Yoram Singer School of Computer Science & Engineering The Hebrew University, Jerusalem 91904, Israel {kobics,oferd,shais,singer}@cs.huji.ac.il
More information1 if 1 x 0 1 if 0 x 1
Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or
More information1 Fixed Point Iteration and Contraction Mapping Theorem
1 Fixed Point Iteration and Contraction Mapping Theorem Notation: For two sets A,B we write A B iff x A = x B. So A A is true. Some people use the notation instead. 1.1 Introduction Consider a function
More informationThe Heat Equation. Lectures INF2320 p. 1/88
The Heat Equation Lectures INF232 p. 1/88 Lectures INF232 p. 2/88 The Heat Equation We study the heat equation: u t = u xx for x (,1), t >, (1) u(,t) = u(1,t) = for t >, (2) u(x,) = f(x) for x (,1), (3)
More informationConvex Programming Tools for Disjunctive Programs
Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible
More informationMATH 425, PRACTICE FINAL EXAM SOLUTIONS.
MATH 45, PRACTICE FINAL EXAM SOLUTIONS. Exercise. a Is the operator L defined on smooth functions of x, y by L u := u xx + cosu linear? b Does the answer change if we replace the operator L by the operator
More informationVariational approach to restore pointlike and curvelike singularities in imaging
Variational approach to restore pointlike and curvelike singularities in imaging Daniele Graziani joint work with Gilles Aubert and Laure BlancFéraud Roma 12/06/2012 Daniele Graziani (Roma) 12/06/2012
More information2. Norm, distance, angle
L. Vandenberghe EE133A (Spring 2016) 2. Norm, distance, angle norm distance angle hyperplanes complex vectors 21 Euclidean norm (Euclidean) norm of vector a R n : a = a 2 1 + a2 2 + + a2 n = a T a if
More informationOnline Learning. 1 Online Learning and Mistake Bounds for Finite Hypothesis Classes
Advanced Course in Machine Learning Spring 2011 Online Learning Lecturer: Shai ShalevShwartz Scribe: Shai ShalevShwartz In this lecture we describe a different model of learning which is called online
More informationA QUICK GUIDE TO THE FORMULAS OF MULTIVARIABLE CALCULUS
A QUIK GUIDE TO THE FOMULAS OF MULTIVAIABLE ALULUS ontents 1. Analytic Geometry 2 1.1. Definition of a Vector 2 1.2. Scalar Product 2 1.3. Properties of the Scalar Product 2 1.4. Length and Unit Vectors
More informationLecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method
Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming
More information24. The Branch and Bound Method
24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NPcomplete. Then one can conclude according to the present state of science that no
More informationMath 2602 Finite and Linear Math Fall 14. Homework 9: Core solutions
Math 2602 Finite and Linear Math Fall 14 Homework 9: Core solutions Section 8.2 on page 264 problems 13b, 27a27b. Section 8.3 on page 275 problems 1b, 8, 10a10b, 14. Section 8.4 on page 279 problems
More informationMetric Spaces. Chapter 1
Chapter 1 Metric Spaces Many of the arguments you have seen in several variable calculus are almost identical to the corresponding arguments in one variable calculus, especially arguments concerning convergence
More informationLECTURE 6: INTERIOR POINT METHOD. 1. Motivation 2. Basic concepts 3. Primal affine scaling algorithm 4. Dual affine scaling algorithm
LECTURE 6: INTERIOR POINT METHOD 1. Motivation 2. Basic concepts 3. Primal affine scaling algorithm 4. Dual affine scaling algorithm Motivation Simplex method works well in general, but suffers from exponentialtime
More informationQuasistatic evolution and congested transport
Quasistatic evolution and congested transport Inwon Kim Joint with Damon Alexander, Katy Craig and Yao Yao UCLA, UW Madison Hard congestion in crowd motion The following crowd motion model is proposed
More informationMathematical finance and linear programming (optimization)
Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may
More informationPractice with Proofs
Practice with Proofs October 6, 2014 Recall the following Definition 0.1. A function f is increasing if for every x, y in the domain of f, x < y = f(x) < f(y) 1. Prove that h(x) = x 3 is increasing, using
More informationMathematical Methods of Engineering Analysis
Mathematical Methods of Engineering Analysis Erhan Çinlar Robert J. Vanderbei February 2, 2000 Contents Sets and Functions 1 1 Sets................................... 1 Subsets.............................
More informationMOSEK modeling manual
MOSEK modeling manual August 12, 2014 Contents 1 Introduction 1 2 Linear optimization 3 2.1 Introduction....................................... 3 2.1.1 Basic notions.................................. 3
More information3. INNER PRODUCT SPACES
. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.
More information1 Homogenous and Homothetic Functions
1 Homogenous and Homothetic Functions Reading: [Simon], Chapter 20, p. 483504. 1.1 Homogenous Functions Definition 1 A real valued function f(x 1,..., x n ) is homogenous of degree k if for all t > 0
More informationLecture 12 Basic Lyapunov theory
EE363 Winter 200809 Lecture 12 Basic Lyapunov theory stability positive definite functions global Lyapunov stability theorems Lasalle s theorem converse Lyapunov theorems finding Lyapunov functions 12
More informationModern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh
Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem
More informationconstraint. Let us penalize ourselves for making the constraint too big. We end up with a
Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the
More information(a) We have x = 3 + 2t, y = 2 t, z = 6 so solving for t we get the symmetric equations. x 3 2. = 2 y, z = 6. t 2 2t + 1 = 0,
Name: Solutions to Practice Final. Consider the line r(t) = 3 + t, t, 6. (a) Find symmetric equations for this line. (b) Find the point where the first line r(t) intersects the surface z = x + y. (a) We
More informationNumerical methods for finding the roots of a function
Numerical methods for finding the roots of a function The roots of a function f (x) are defined as the values for which the value of the function becomes equal to zero. So, finding the roots of f (x) means
More information! Solve problem to optimality. ! Solve problem in polytime. ! Solve arbitrary instances of the problem. #approximation algorithm.
Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NPhard problem What should I do? A Theory says you're unlikely to find a polytime algorithm Must sacrifice one of three
More informationDuality of linear conic problems
Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least
More informationLinear Programming, Lagrange Multipliers, and Duality Geoff Gordon
lp.nb 1 Linear Programming, Lagrange Multipliers, and Duality Geoff Gordon lp.nb 2 Overview This is a tutorial about some interesting math and geometry connected with constrained optimization. It is not
More informationRational Polynomial Functions
Rational Polynomial Functions Rational Polynomial Functions and Their Domains Today we discuss rational polynomial functions. A function f(x) is a rational polynomial function if it is the quotient of
More informationTutorial on Convex Optimization for Engineers Part I
Tutorial on Convex Optimization for Engineers Part I M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D98684 Ilmenau, Germany jens.steinwandt@tuilmenau.de
More informationLecture 7  Linear Programming
COMPSCI 530: Design and Analysis of Algorithms DATE: 09/17/2013 Lecturer: Debmalya Panigrahi Lecture 7  Linear Programming Scribe: Hieu Bui 1 Overview In this lecture, we cover basics of linear programming,
More informationOn generalized gradients and optimization
On generalized gradients and optimization Erik J. Balder 1 Introduction There exists a calculus for general nondifferentiable functions that englobes a large part of the familiar subdifferential calculus
More informationA Simple Introduction to Support Vector Machines
A Simple Introduction to Support Vector Machines Martin Law Lecture for CSE 802 Department of Computer Science and Engineering Michigan State University Outline A brief history of SVM Largemargin linear
More informationHøgskolen i Narvik Sivilingeniørutdanningen STE6237 ELEMENTMETODER. Oppgaver
Høgskolen i Narvik Sivilingeniørutdanningen STE637 ELEMENTMETODER Oppgaver Klasse: 4.ID, 4.IT Ekstern Professor: Gregory A. Chechkin email: chechkin@mech.math.msu.su Narvik 6 PART I Task. Consider twopoint
More informationLAGRANGIAN RELAXATION TECHNIQUES FOR LARGE SCALE OPTIMIZATION
LAGRANGIAN RELAXATION TECHNIQUES FOR LARGE SCALE OPTIMIZATION Kartik Sivaramakrishnan Department of Mathematics NC State University kksivara@ncsu.edu http://www4.ncsu.edu/ kksivara SIAM/MGSA Brown Bag
More informationTECHNICAL NOTE. Proof of a Conjecture by Deutsch, Li, and Swetits on Duality of Optimization Problems1
JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 102, No. 3, pp. 697703, SEPTEMBER 1999 TECHNICAL NOTE Proof of a Conjecture by Deutsch, Li, and Swetits on Duality of Optimization Problems1 H. H.
More information1 Formulating The Low Degree Testing Problem
6.895 PCP and Hardness of Approximation MIT, Fall 2010 Lecture 5: Linearity Testing Lecturer: Dana Moshkovitz Scribe: Gregory Minton and Dana Moshkovitz In the last lecture, we proved a weak PCP Theorem,
More informationLinear Threshold Units
Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear
More informationPractical Guide to the Simplex Method of Linear Programming
Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear
More informationSECONDORDER LINEAR HOMOGENEOUS DIFFERENTIAL EQUATIONS
L SECONDORDER LINEAR HOOGENEOUS DIFFERENTIAL EQUATIONS SECONDORDER LINEAR HOOGENEOUS DIFFERENTIAL EQUATIONS WITH CONSTANT COEFFICIENTS A secondorder linear differential equation is one of the form d
More informationBIG DATA PROBLEMS AND LARGESCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION
BIG DATA PROBLEMS AND LARGESCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION Ş. İlker Birbil Sabancı University Ali Taylan Cemgil 1, Hazal Koptagel 1, Figen Öztoprak 2, Umut Şimşekli
More informationSMOOTHING APPROXIMATIONS FOR TWO CLASSES OF CONVEX EIGENVALUE OPTIMIZATION PROBLEMS YU QI. (B.Sc.(Hons.), BUAA)
SMOOTHING APPROXIMATIONS FOR TWO CLASSES OF CONVEX EIGENVALUE OPTIMIZATION PROBLEMS YU QI (B.Sc.(Hons.), BUAA) A THESIS SUBMITTED FOR THE DEGREE OF MASTER OF SCIENCE DEPARTMENT OF MATHEMATICS NATIONAL
More informationBANACH AND HILBERT SPACE REVIEW
BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but
More informationSolutions Of Some NonLinear Programming Problems BIJAN KUMAR PATEL. Master of Science in Mathematics. Prof. ANIL KUMAR
Solutions Of Some NonLinear Programming Problems A PROJECT REPORT submitted by BIJAN KUMAR PATEL for the partial fulfilment for the award of the degree of Master of Science in Mathematics under the supervision
More informationSTORM: Stochastic Optimization Using Random Models Katya Scheinberg Lehigh University. (Joint work with R. Chen and M. Menickelly)
STORM: Stochastic Optimization Using Random Models Katya Scheinberg Lehigh University (Joint work with R. Chen and M. Menickelly) Outline Stochastic optimization problem black box gradient based Existing
More informationNo: 10 04. Bilkent University. Monotonic Extension. Farhad Husseinov. Discussion Papers. Department of Economics
No: 10 04 Bilkent University Monotonic Extension Farhad Husseinov Discussion Papers Department of Economics The Discussion Papers of the Department of Economics are intended to make the initial results
More information! Solve problem to optimality. ! Solve problem in polytime. ! Solve arbitrary instances of the problem. !approximation algorithm.
Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NPhard problem What should I do? A Theory says you're unlikely to find a polytime algorithm Must sacrifice one of
More information3 Does the Simplex Algorithm Work?
Does the Simplex Algorithm Work? In this section we carefully examine the simplex algorithm introduced in the previous chapter. Our goal is to either prove that it works, or to determine those circumstances
More informationOptimal shift scheduling with a global service level constraint
Optimal shift scheduling with a global service level constraint Ger Koole & Erik van der Sluis Vrije Universiteit Division of Mathematics and Computer Science De Boelelaan 1081a, 1081 HV Amsterdam The
More informationInner product. Definition of inner product
Math 20F Linear Algebra Lecture 25 1 Inner product Review: Definition of inner product. Slide 1 Norm and distance. Orthogonal vectors. Orthogonal complement. Orthogonal basis. Definition of inner product
More informationNumerical Solutions to Differential Equations
Numerical Solutions to Differential Equations Lecture Notes The Finite Element Method #2 Peter Blomgren, blomgren.peter@gmail.com Department of Mathematics and Statistics Dynamical Systems Group Computational
More informationBasics of Polynomial Theory
3 Basics of Polynomial Theory 3.1 Polynomial Equations In geodesy and geoinformatics, most observations are related to unknowns parameters through equations of algebraic (polynomial) type. In cases where
More informationLecture 4: Equality Constrained Optimization. Tianxi Wang
Lecture 4: Equality Constrained Optimization Tianxi Wang wangt@essex.ac.uk 2.1 Lagrange Multiplier Technique (a) Classical Programming max f(x 1, x 2,..., x n ) objective function where x 1, x 2,..., x
More informationRecovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branchandbound approach
MASTER S THESIS Recovery of primal solutions from dual subgradient methods for mixed binary linear programming; a branchandbound approach PAULINE ALDENVIK MIRJAM SCHIERSCHER Department of Mathematical
More informationLecture 7: Finding Lyapunov Functions 1
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.243j (Fall 2003): DYNAMICS OF NONLINEAR SYSTEMS by A. Megretski Lecture 7: Finding Lyapunov Functions 1
More informationThe Proximity Operator
The Proximity Operator YaoLiang Yu Machine Learning Department Carnegie Melon University Pittsburgh, PA, 523, USA yaoliang@cs.cmu.edu March 4, 204 Abstract We present some basic properties of the proximity
More informationCost Minimization and the Cost Function
Cost Minimization and the Cost Function Juan Manuel Puerta October 5, 2009 So far we focused on profit maximization, we could look at a different problem, that is the cost minimization problem. This is
More informationOptimization of Design. Lecturer:DungAn Wang Lecture 12
Optimization of Design Lecturer:DungAn Wang Lecture 12 Lecture outline Reading: Ch12 of text Today s lecture 2 Constrained nonlinear programming problem Find x=(x1,..., xn), a design variable vector of
More informationPROBLEM SET. Practice Problems for Exam #2. Math 2350, Fall Nov. 7, 2004 Corrected Nov. 10 ANSWERS
PROBLEM SET Practice Problems for Exam #2 Math 2350, Fall 2004 Nov. 7, 2004 Corrected Nov. 10 ANSWERS i Problem 1. Consider the function f(x, y) = xy 2 sin(x 2 y). Find the partial derivatives f x, f y,
More information