# (Quasi-)Newton methods

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 (Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting point x 0, Newton method consists in iterating: x k+1 = x k g (x k ) 1 g(x k ) where g (x) is the derivative of g at point x. Applying this method to the optimization problem: min f(x) x R n consists in setting g(x) = f(x), i.e. looking for stationary points. The iterations read: x k+1 = x k 2 f(x k ) 1 f(x k ). Newton method is particularly interesting as its convergence is quadratic locally around x, i.e.: x k+1 x γ x k x 2, γ > 0. However the convergence is guaranteed only if x 0 is sufficiently close to x. The method may diverge if the initial point is too far from x or if the Hessian is not positive definite. In order to address this issue of local convergence, Newton method can be combined with a line search method in the direction d k = 2 f(x k ) 1 f(x k ). proof. We now prove the quadratic local convergence of Newton method. TODO 1.2 Variable metric methods The idea behind variable metric methods consists in using iterations of the form { d k = B k g k, where g k = f(x k ) and B k is a positive definite matrix. If B k = I n, it corresponds to gradient descent. Fixing B k = B leads to the following remark. Remark. When minimizing min f(x) x R n one can set x = Cy with C invertible (change of variable). Let us denote f(y) = f(cy). This leads to: Gradient descent applied to f(y) reads: which is equivalent to f(y) = C T f(cy). y k+1 = y k ρ k C T f(cy k ) x k+1 = x k ρ k CC T f(x k ). This amounts to using B = CC T. In the case where f is a quadratic form, it is straightforward to observe that this will improve convergence. page 1

3 2.3 Broyden formula Broyden formula is a rank 1 correction. Let us write B k+1 = B k + vv T. The vector v R n should verify the quasi-newton relation: B k+1 y k = s k, where y k = g k+1 g k and s k = x k+1 x k. It follows that: B k y k + vv T y k = s k, which leads by application of the dot product with y k to: Using the equality (y T k v) 2 = (s k B k y k ) T y k, vv T = vvt y k (vv T y k ) T (v T y k ) 2, one can write after replacing vv T y k by s k B k y k, and (v T y k ) 2 by y T k (s k B k y k ), the correction formula also known as Broyden formula. B k+1 = B k + (s k B k y k )(s k B k y k ) T (s k B k y k ) T y k, Theorem 2. Let f a quadratic form positive definite. Let us consider the method that, starting for x 0, iterates: x k+1 = x k + s k, where the vectors s k are linearly independent. Then the sequence of matrices starting by B 0 and defined as: B k+1 = B k + (s k B k y k )(s k B k y k ) T (s k B k y k ) T y k, where y k = f(x k+1 ) f(x k ), converges in less than n iterations towards A 1, the inverse of the Hessian of f. proof. to: Since we consider here a quadratic function, the Hessian is constant and equal to A. This leads We have seen that B k+1 is constructed such that: Let us show that: By recurrence, let us assume that it is true for B k : Let i k 2. One has y i = f(x i+1 ) f(x i ), i. B k+1 y k = s k. B k+1 y i = s i, i = 0,..., k 1. B k y i = s i, i = 0,..., k 2. B k+1 y i = B k y i + (s k B k y k )(s T k y i B k y T k y i) (s k B k y k ) T y k. (1) By hypothesis, one has B k y i = s i which implies that y T k B ky i = y T k s i, but since As j = y j for all j, it leads to: y T k s i = s T k As i = s T k y i, page 3

4 The numerator in (1) is therefore 0, which leads to: B k+1 y i = B k y i = s i. One therefore has: After n iterations, one has B k+1 y i = s i, i = 0,..., k. B n y i = s i, i = 0,..., n 1. but since y i = As i, this last equation is equivalent to: B n As i = s i, i = 0,..., n 1. As the s i form a basis of R n this implies that B n A = I n or B n = A 1. The issue with Broyden s formula is that there is no guarantee that the matrices B k are positive definite, even if the function f is quadratic and B 0 = I n. It is nevertheless interesting to have B n = A Davidon, Fletcher and Powell formula The formula from Davidon, Fletcher and Powell is a rank 2 correction. It reads: B k+1 = B k + s ks T k s T k y B ky k yk T B k k yk T B. (2) ky k The following theorem states that under certain conditions, the formula guarantees to have B k positive definite. Theorem 3. Let us consider the update { dk = B k g k, x k+1 = x k + ρ k Bg k, ρ k optimal where B 0 > 0 is provided as well as x 0. Then the matrices B k defined as in (2) are positive definite for all k > 0. proof. Let x R n. One has: x T B k+1 x = x T B k x + (st k x)2 (yt k B kx) 2 y T k B ky k. = yt k B ky k x T B k x (y T k B kx) 2 y T k B ky k + (st k x)2. (3) If one defines the dot product x, y as x T B k y the equation above reads: x T B k+1 x = y k, y k x, x y k, x 2 y k, y k + (st k x)2. The first term is positive by Cauchy-Schwartz inequality. Regarding the second term, as the step size is optimal one has: g T k+1d k = 0, which implies = ρ k (g k+1 g k ) T d k = ρ k g T k B k g k > 0, and so x T B k+1 x 0. Both terms being positive, the sum is zero only if both terms are 0. This implies that x = λy k with λ 0. In this case the second term cannot be zero as s T k x = λst k y k. This implies that B k+1 > 0. Remark. In the proof we used the fact that an optimal step size is used. The result still holds if one uses an approximate line search strategy for example using Wolf and Powell s rule. In this case the point x k+1 is such that: φ (ρ k ) = f(x k+1 ) T d k m 2 f(x k ) T d k, 0 < m 2 < 1, which guarantees: and therefore (g k+1 g k ) T (x k+1 x k ) > 0. gk+1 T x k+1 x k ρ k > g T k x k+1 x k ρ k, page 4

5 Algorithm 1 Davidon-Fletcher-Powell algorithm Require: ε > 0 (tolerance), K (maximum number of iterations) 1: x 0 R n, B 0 > 0 (for example I n ) 2: for k = 0 to K do 3: if g k < ε then 4: break 5: end if 6: d k = B k f(x k ) 7: Compute optimal step size ρ k 8: x k+1 = x k + ρ k d k 9: s k = ρ k d k 10: y k = g k+1 g k 11: B k+1 = B k + s ks T k 12: end for 13: return x k+1 B ky k y T k B k y T k B ky k 2.5 Davidon-Fletcher-Powell algorithm One can know use the later formula in algorithm 1. This algorithm has a remarkable property when the function f is quadratic. Theorem 4. When f is a quadratic form, the algorithm 1 generates a sequence of directions s 0,..., s k which verify: s i A T s j = 0, 0 j < j k + 1, (4) B k+1 As i = s i, 0 i k. proof. TODO This theorem says that in the quadratic case, the algorithm 1 is like a conjugate gradient method, which therefore converges in at most n iterations. One can also notice that for k = n 1 the equalities B n As i = s i, i = 0,..., n 1, and the fact that all s i are linearly independent imply that B n = A 1. Remark. One can show that in the general case (non-quadratic), if the direction d k is reinitialized to g k periodically, this algorithm converges to a local minimum ˆx of f and that: lim B k = 2 f(ˆx) 1. k The implies that close to the optimum, if the line search is exact, the method behaves like a Newton method. This justifies the use of ρ k = 1 when using approximate line search. 2.6 Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm The BFGS formula is a correction formula of rank 2 which is derived from the formula of DFP by swapping the roles of s k and y k. BFGS is named for the four people who independently discovered it in 1970: Broyden, Fletcher, Goldfarb and Shanno. The formula obtained allows to maintain an approximation H k of the Hessian which satisfies the same properties: H k+1 > 0 if H k > 0 and satisfying the quasi-newton relation: y k = H k s k. The formula therefore reads: H k+1 = H k + y kyk T yk T s H ks k s T k H k k s T k H. ks k The algorithm is detailed in algorithm 2. Note that the direction d k is obtained by solving a linear system. However in practice the update of H k is done on Cholesky factorization of H k = C k Ck T which implies that the complexity of BFGS is the same as DFP. The use of a Cholesky factorization is useful to check that H k states numerically positive definite, as this property can be lost due to numerical errors. page 5

6 Algorithm 2 Broyden-Davidon-Goldfarb-Shanno (BFGS) algorithm Require: ε > 0 (tolerance), K (maximum number of iterations) 1: x 0 R n, H 0 > 0 (for example I n ) 2: for k = 0 to K do 3: if g k < ε then 4: break 5: end if 6: d k = H 1 k f(x k) 7: Compute optimal step size ρ k 8: x k+1 = x k + ρ k d k 9: s k = ρ k d k 10: y k = g k+1 g k 11: H k+1 = H k + y ky T k yk T s k 12: end for 13: return x k+1 H ks k s T k H k s T k H ks k Remark. The BFGS algorithm has the same property as the DFP method: in the quadratic case it produces conjugate directions, converges in less than n iterations and H n = A. Compared to DFP, BFGS convergence speed is much less sensitive to the use of approximate step size. It is therefore a particularly good candidate when combined with Wolfe and Powell s or Goldstein s rule. Remark. The BFGS method is available in scipy as scipy.optimize.fmin_bfgs. 2.7 Limited-memory BFGS (L-BFGS) algorithm The limited-memory BFGS (L-BFGS) algorithm is a variant of the BFGS algorithm that limits memory usage. While BFGS requires to store in memory a matrix of the size of the Hessian, n n, which can be prohibitive in applications such as computer vision or machine learning, the L-BFGS algorithm only stores a few vectors that are used to approximate the matrix H 1 k. As a consequence the memory usage is linear in the dimension of the problem. The L-BFGS is an algorithm of the quasi-newton family with d k = B k f(x k ). The difference is in the computation of the product between B k and f(x k ). The idea is to keep in memory the last few low rank corrections, more specifically the last m updates s k = x k+1 x k and y k = g k+1 g k. Often in practice m < 10, yet it might be necessary to modify this parameter on specific problems to speed up convergence. We denote by µ k = 1 yk T s. The algorithm to compute the descent direction is given in k algorithm 3. Algorithm 3 Direction finding in L-BFGS algorithm Require: m (memory size) 1: q = g k 2: for i = k 1 to k m do 3: α i = µ i s T i q 4: q = q α i y i 5: end for 6: z = B 0 k q 7: for i = k m to k 1 do 8: β = µ i y T i z 9: z = z + s i (α i β) 10: end for 11: d k = z Commonly, the inverse Hessian Bk 0 is represented as a diagonal matrix, so that initially setting z requires only an element-by-element multiplication. Bk 0 can change at each iteration but has however to be positive definite. Like BFGS, L-BFGS does not need exact line search to converge. In machine learning, it is almost always the best approach to solve l 2 regularized Logistic regression and conditional random fields (CRF). page 6

7 Remark. L-BFGS is for smooth unconstrained problem. Yet, it can be extended to handle simple box constraints (a.k.a. bound constraints) on variables; that is, constraints of the form l i x i u i where l i and u i are per-variable constant lower and upper bounds, respectively (for each x i, either or both bounds may be omitted). This algorithm is called L BF GS B. Remark. The L-BFGS-B algorithm is available in scipy as scipy.optimize.fmin_l_bfgs_b. 3 Methods specific to least squares 3.1 Gauss-Newton method When considering least square problems the function to minimize reads: f(x) = 1 2 f i (x) 2. Newton method can be applied to the minimization of f. The gradient and the Hessian matrix read in this particular case: f(x) = f i (x) f i (x), and 2 f(x) = f i (x) f i (x) T + f i (x) 2 f i (x). If we are close to the optimum, where the f i (x) are assumed to be small the second term can be ignored. The matrix obtained reads: H(x) = f i (x) f i (x) T. This matrix is always positive. Furthermore when m is much larger than n, this matrix is often positive definite. The Gauss-Newton method consists in using H(x) in place of the Hessian. The method reads: x 0 = fixed, H k = f i (x k ) f i (x k ) T, 3.2 Levenberg-Marquardt method x k+1 = x k H 1 k f(x k). In order to guarantee the convergence of the Gauss-Newton method, it can be combined with a line search procedure: x k+1 = x k ρ k H 1 k f(x k). However in order to guarantee that the H k stay positive definite a modified method, known as Levenberg- Marquardt, is often used. The idea is simply to replace H k by H k + λi n. One can notice that if λ is large, this method is equivalent to a simple gradient method. The Levenberg-Marquardt method reads: x 0 = fixed, H k = f i (x k ) f i (x k ) T, d k = (H k + λi n ) 1 f(x k ) x k+1 = x k + ρ k d k. Remark. The Levenberg-Marquardt method is available in scipy as scipy.optimize.leastsq. page 7

### Multilayer Perceptrons

Made wi t h OpenOf f i ce. or g 1 Multilayer Perceptrons 2 nd Order Learning Algorithms Made wi t h OpenOf f i ce. or g 2 Why 2 nd Order? Gradient descent Back-propagation used to obtain first derivatives

### t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).

1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction

### Solutions of Equations in One Variable. Fixed-Point Iteration II

Solutions of Equations in One Variable Fixed-Point Iteration II Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011

### A Perfect Example for The BFGS Method

A Perfect Example for The BFGS Method Yu-Hong Dai Abstract Consider the BFGS quasi-newton method applied to a general nonconvex function that has continuous second derivatives. This paper aims to construct

### Nonlinear Least Squares Data Fitting

Appendix D Nonlinear Least Squares Data Fitting D1 Introduction A nonlinear least squares problem is an unconstrained minimization problem of the form minimize f(x) = f i (x) 2, x where the objective function

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Lecture 8 February 4

ICS273A: Machine Learning Winter 2008 Lecture 8 February 4 Scribe: Carlos Agell (Student) Lecturer: Deva Ramanan 8.1 Neural Nets 8.1.1 Logistic Regression Recall the logistic function: g(x) = 1 1 + e θt

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### General Framework for an Iterative Solution of Ax b. Jacobi s Method

2.6 Iterative Solutions of Linear Systems 143 2.6 Iterative Solutions of Linear Systems Consistent linear systems in real life are solved in one of two ways: by direct calculation (using a matrix factorization,

### Parameter Estimation: A Deterministic Approach using the Levenburg-Marquardt Algorithm

Parameter Estimation: A Deterministic Approach using the Levenburg-Marquardt Algorithm John Bardsley Department of Mathematical Sciences University of Montana Applied Math Seminar-Feb. 2005 p.1/14 Outline

### The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem

### Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

### Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

### (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.

Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product

### Metric Spaces. Chapter 7. 7.1. Metrics

Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field

3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field 77 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field Overview: The antiderivative in one variable calculus is an important

### CSCI567 Machine Learning (Fall 2014)

CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 22, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 22, 2014 1 /

### Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem

### GLM, insurance pricing & big data: paying attention to convergence issues.

GLM, insurance pricing & big data: paying attention to convergence issues. Michaël NOACK - michael.noack@addactis.com Senior consultant & Manager of ADDACTIS Pricing Copyright 2014 ADDACTIS Worldwide.

### (57) (58) (59) (60) (61)

Module 4 : Solving Linear Algebraic Equations Section 5 : Iterative Solution Techniques 5 Iterative Solution Techniques By this approach, we start with some initial guess solution, say for solution and

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### BIG DATA PROBLEMS AND LARGE-SCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION

BIG DATA PROBLEMS AND LARGE-SCALE OPTIMIZATION: A DISTRIBUTED ALGORITHM FOR MATRIX FACTORIZATION Ş. İlker Birbil Sabancı University Ali Taylan Cemgil 1, Hazal Koptagel 1, Figen Öztoprak 2, Umut Şimşekli

### Lecture 13 Linear quadratic Lyapunov theory

EE363 Winter 28-9 Lecture 13 Linear quadratic Lyapunov theory the Lyapunov equation Lyapunov stability conditions the Lyapunov operator and integral evaluating quadratic integrals analysis of ARE discrete-time

### Reference: Introduction to Partial Differential Equations by G. Folland, 1995, Chap. 3.

5 Potential Theory Reference: Introduction to Partial Differential Equations by G. Folland, 995, Chap. 3. 5. Problems of Interest. In what follows, we consider Ω an open, bounded subset of R n with C 2

### Principles of Scientific Computing Nonlinear Equations and Optimization

Principles of Scientific Computing Nonlinear Equations and Optimization David Bindel and Jonathan Goodman last revised March 6, 2006, printed March 6, 2009 1 1 Introduction This chapter discusses two related

### Problem 1 (10 pts) Find the radius of convergence and interval of convergence of the series

1 Problem 1 (10 pts) Find the radius of convergence and interval of convergence of the series a n n=1 n(x + 2) n 5 n 1. n(x + 2)n Solution: Do the ratio test for the absolute convergence. Let a n =. Then,

### Big Data - Lecture 1 Optimization reminders

Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014 Schedule Introduction Major issues Examples Mathematics

### MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix.

MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix. Inverse matrix Definition. Let A be an n n matrix. The inverse of A is an n n matrix, denoted

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### Similar matrices and Jordan form

Similar matrices and Jordan form We ve nearly covered the entire heart of linear algebra once we ve finished singular value decompositions we ll have seen all the most central topics. A T A is positive

### Cofactor Expansion: Cramer s Rule

Cofactor Expansion: Cramer s Rule MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Introduction Today we will focus on developing: an efficient method for calculating

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Factorization Theorems

Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

### DERIVATIVES AS MATRICES; CHAIN RULE

DERIVATIVES AS MATRICES; CHAIN RULE 1. Derivatives of Real-valued Functions Let s first consider functions f : R 2 R. Recall that if the partial derivatives of f exist at the point (x 0, y 0 ), then we

### AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS

AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS Revised Edition James Epperson Mathematical Reviews BICENTENNIAL 0, 1 8 0 7 z ewiley wu 2007 r71 BICENTENNIAL WILEY-INTERSCIENCE A John Wiley & Sons, Inc.,

NORTHWESTERN UNIVERSITY Department of Electrical Engineering and Computer Science LARGE SCALE UNCONSTRAINED OPTIMIZATION by Jorge Nocedal 1 June 23, 1996 ABSTRACT This paper reviews advances in Newton,

### THE FUNDAMENTAL THEOREM OF ALGEBRA VIA PROPER MAPS

THE FUNDAMENTAL THEOREM OF ALGEBRA VIA PROPER MAPS KEITH CONRAD 1. Introduction The Fundamental Theorem of Algebra says every nonconstant polynomial with complex coefficients can be factored into linear

### Nonlinear Algebraic Equations. Lectures INF2320 p. 1/88

Nonlinear Algebraic Equations Lectures INF2320 p. 1/88 Lectures INF2320 p. 2/88 Nonlinear algebraic equations When solving the system u (t) = g(u), u(0) = u 0, (1) with an implicit Euler scheme we have

### Lecture 6. Inverse of Matrix

Lecture 6 Inverse of Matrix Recall that any linear system can be written as a matrix equation In one dimension case, ie, A is 1 1, then can be easily solved as A x b Ax b x b A 1 A b A 1 b provided that

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

### Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

### The Characteristic Polynomial

Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem

### CONTROLLABILITY. Chapter 2. 2.1 Reachable Set and Controllability. Suppose we have a linear system described by the state equation

Chapter 2 CONTROLLABILITY 2 Reachable Set and Controllability Suppose we have a linear system described by the state equation ẋ Ax + Bu (2) x() x Consider the following problem For a given vector x in

### Probabilistic Models for Big Data. Alex Davies and Roger Frigola University of Cambridge 13th February 2014

Probabilistic Models for Big Data Alex Davies and Roger Frigola University of Cambridge 13th February 2014 The State of Big Data Why probabilistic models for Big Data? 1. If you don t have to worry about

### Logistic Regression. Vibhav Gogate The University of Texas at Dallas. Some Slides from Carlos Guestrin, Luke Zettlemoyer and Dan Weld.

Logistic Regression Vibhav Gogate The University of Texas at Dallas Some Slides from Carlos Guestrin, Luke Zettlemoyer and Dan Weld. Generative vs. Discriminative Classifiers Want to Learn: h:x Y X features

### Numerical Methods for Solving Systems of Nonlinear Equations

Numerical Methods for Solving Systems of Nonlinear Equations by Courtney Remani A project submitted to the Department of Mathematical Sciences in conformity with the requirements for Math 4301 Honour s

### 1 Fixed Point Iteration and Contraction Mapping Theorem

1 Fixed Point Iteration and Contraction Mapping Theorem Notation: For two sets A,B we write A B iff x A = x B. So A A is true. Some people use the notation instead. 1.1 Introduction Consider a function

A Globally Convergent Primal-Dual Interior Point Method for Constrained Optimization Hiroshi Yamashita 3 Abstract This paper proposes a primal-dual interior point method for solving general nonlinearly

### Pricing and calibration in local volatility models via fast quantization

Pricing and calibration in local volatility models via fast quantization Parma, 29 th January 2015. Joint work with Giorgia Callegaro and Martino Grasselli Quantization: a brief history Birth: back to

### Inner Product Spaces and Orthogonality

Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,

### U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009. Notes on Algebra

U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009 Notes on Algebra These notes contain as little theory as possible, and most results are stated without proof. Any introductory

### Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Roots and Coefficients of a Quadratic Equation Summary

Roots and Coefficients of a Quadratic Equation Summary For a quadratic equation with roots α and β: Sum of roots = α + β = and Product of roots = αβ = Symmetrical functions of α and β include: x = and

### TRANSACTIONS JUNE, 1969 AN UPPER BOUND ON THE STOP-LOSS NET PREMIUM--ACTUARIAL NOTE NEWTON L. BOWERS, JR. ABSTRACT

TRANSACTIONS OF SOCIETY OF ACTUARIES 1969 VOL. 21 PT. 1 NO. 60 VOL. XXI, PART I M~TINO No. 60 TRANSACTIONS JUNE, 1969 AN UPPER BOUND ON THE STOP-LOSS NET PREMIUM--ACTUARIAL NOTE NEWTON L. BOWERS, JR. ABSTRACT

### IRREDUCIBLE OPERATOR SEMIGROUPS SUCH THAT AB AND BA ARE PROPORTIONAL. 1. Introduction

IRREDUCIBLE OPERATOR SEMIGROUPS SUCH THAT AB AND BA ARE PROPORTIONAL R. DRNOVŠEK, T. KOŠIR Dedicated to Prof. Heydar Radjavi on the occasion of his seventieth birthday. Abstract. Let S be an irreducible

### 2.1: MATRIX OPERATIONS

.: MATRIX OPERATIONS What are diagonal entries and the main diagonal of a matrix? What is a diagonal matrix? When are matrices equal? Scalar Multiplication 45 Matrix Addition Theorem (pg 0) Let A, B, and

### GenOpt (R) Generic Optimization Program User Manual Version 3.0.0β1

(R) User Manual Environmental Energy Technologies Division Berkeley, CA 94720 http://simulationresearch.lbl.gov Michael Wetter MWetter@lbl.gov February 20, 2009 Notice: This work was supported by the U.S.

### Increasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.

1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

### CITY UNIVERSITY LONDON. BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION

No: CITY UNIVERSITY LONDON BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION ENGINEERING MATHEMATICS 2 (resit) EX2005 Date: August

### Notes on Symmetric Matrices

CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.

### constraint. Let us penalize ourselves for making the constraint too big. We end up with a

Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the

### Analysis of a Production/Inventory System with Multiple Retailers

Analysis of a Production/Inventory System with Multiple Retailers Ann M. Noblesse 1, Robert N. Boute 1,2, Marc R. Lambrecht 1, Benny Van Houdt 3 1 Research Center for Operations Management, University

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Matrix Inverse and Determinants

DM554 Linear and Integer Programming Lecture 5 and Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1 2 3 4 and Cramer s rule 2 Outline 1 2 3 4 and

### 1 VECTOR SPACES AND SUBSPACES

1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such

### Vectors, Gradient, Divergence and Curl.

Vectors, Gradient, Divergence and Curl. 1 Introduction A vector is determined by its length and direction. They are usually denoted with letters with arrows on the top a or in bold letter a. We will use

### Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 )

Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 ) and Neural Networks( 類 神 經 網 路 ) 許 湘 伶 Applied Linear Regression Models (Kutner, Nachtsheim, Neter, Li) hsuhl (NUK) LR Chap 10 1 / 35 13 Examples

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J. Olver 5. Inner Products and Norms The norm of a vector is a measure of its size. Besides the familiar Euclidean norm based on the dot product, there are a number

### Math 315: Linear Algebra Solutions to Midterm Exam I

Math 35: Linear Algebra s to Midterm Exam I # Consider the following two systems of linear equations (I) ax + by = k cx + dy = l (II) ax + by = 0 cx + dy = 0 (a) Prove: If x = x, y = y and x = x 2, y =

### Optimization. J(f) := λω(f) + R emp (f) (5.1) m l(f(x i ) y i ). (5.2) i=1

5 Optimization Optimization plays an increasingly important role in machine learning. For instance, many machine learning algorithms minimize a regularized risk functional: with the empirical risk min

### We know a formula for and some properties of the determinant. Now we see how the determinant can be used.

Cramer s rule, inverse matrix, and volume We know a formula for and some properties of the determinant. Now we see how the determinant can be used. Formula for A We know: a b d b =. c d ad bc c a Can we

### G.A. Pavliotis. Department of Mathematics. Imperial College London

EE1 MATHEMATICS NUMERICAL METHODS G.A. Pavliotis Department of Mathematics Imperial College London 1. Numerical solution of nonlinear equations (iterative processes). 2. Numerical evaluation of integrals.

### Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they

### 24. The Branch and Bound Method

24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

### Lecture 15 An Arithmetic Circuit Lowerbound and Flows in Graphs

CSE599s: Extremal Combinatorics November 21, 2011 Lecture 15 An Arithmetic Circuit Lowerbound and Flows in Graphs Lecturer: Anup Rao 1 An Arithmetic Circuit Lower Bound An arithmetic circuit is just like

### Pattern Analysis. Logistic Regression. 12. Mai 2009. Joachim Hornegger. Chair of Pattern Recognition Erlangen University

Pattern Analysis Logistic Regression 12. Mai 2009 Joachim Hornegger Chair of Pattern Recognition Erlangen University Pattern Analysis 2 / 43 1 Logistic Regression Posteriors and the Logistic Function Decision

### 4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns

L. Vandenberghe EE133A (Spring 2016) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows

### Orthogonal Projections

Orthogonal Projections and Reflections (with exercises) by D. Klain Version.. Corrections and comments are welcome! Orthogonal Projections Let X,..., X k be a family of linearly independent (column) vectors

### Taylor and Maclaurin Series

Taylor and Maclaurin Series In the preceding section we were able to find power series representations for a certain restricted class of functions. Here we investigate more general problems: Which functions

### Online Learning, Stability, and Stochastic Gradient Descent

Online Learning, Stability, and Stochastic Gradient Descent arxiv:1105.4701v3 [cs.lg] 8 Sep 2011 September 9, 2011 Tomaso Poggio, Stephen Voinea, Lorenzo Rosasco CBCL, McGovern Institute, CSAIL, Brain

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### Nonlinear Algebraic Equations Example

Nonlinear Algebraic Equations Example Continuous Stirred Tank Reactor (CSTR). Look for steady state concentrations & temperature. s r (in) p,i (in) i In: N spieces with concentrations c, heat capacities

### Least-Squares Intersection of Lines

Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a

### The equivalence of logistic regression and maximum entropy models

The equivalence of logistic regression and maximum entropy models John Mount September 23, 20 Abstract As our colleague so aptly demonstrated ( http://www.win-vector.com/blog/20/09/the-simplerderivation-of-logistic-regression/

### Network Traffic Modelling

University of York Dissertation submitted for the MSc in Mathematics with Modern Applications, Department of Mathematics, University of York, UK. August 009 Network Traffic Modelling Author: David Slade

### Linear Codes. Chapter 3. 3.1 Basics

Chapter 3 Linear Codes In order to define codes that we can encode and decode efficiently, we add more structure to the codespace. We shall be mainly interested in linear codes. A linear code of length

### Introduction to Algebraic Geometry. Bézout s Theorem and Inflection Points

Introduction to Algebraic Geometry Bézout s Theorem and Inflection Points 1. The resultant. Let K be a field. Then the polynomial ring K[x] is a unique factorisation domain (UFD). Another example of a

### Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems

Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001,

### Stochastic Inventory Control

Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the

Facts About Eigenvalues By Dr David Butler Definitions Suppose A is an n n matrix An eigenvalue of A is a number λ such that Av = λv for some nonzero vector v An eigenvector of A is a nonzero vector v

### 1 Formulating The Low Degree Testing Problem

6.895 PCP and Hardness of Approximation MIT, Fall 2010 Lecture 5: Linearity Testing Lecturer: Dana Moshkovitz Scribe: Gregory Minton and Dana Moshkovitz In the last lecture, we proved a weak PCP Theorem,

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA

SECOND DERIVATIVE TEST FOR CONSTRAINED EXTREMA This handout presents the second derivative test for a local extrema of a Lagrange multiplier problem. The Section 1 presents a geometric motivation for the

### Topic 1: Matrices and Systems of Linear Equations.

Topic 1: Matrices and Systems of Linear Equations Let us start with a review of some linear algebra concepts we have already learned, such as matrices, determinants, etc Also, we shall review the method