# Big Data - Lecture 1 Optimization reminders

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014

2 Big Data - Lecture 1 Optimization reminders S. Gadat Toulouse, Octobre 2014

3 Schedule Introduction Major issues Examples Mathematics 1 Introduction Major issues Examples Mathematics 2 Convex functions Example of convex functions Gradient descent method 3 Equality constraint Inequality constraint KKT Conditions 4

4 Major issues Examples Mathematics I Introduction - Major issues in data science Data science: Extract from data some knowledge for industrial or academic exploitation. Involves: 1 Signal Processing (how to record the data and represent it?) 2 Modelization (What is the problem, what kind of mathematical model and answer?) 3 Statistics (reliability of estimation procedures?) 4 Machine Learning (what kind of efficient optimization algorithm?) 5 Implementation (software needs) 6 Visualization (how can I represent the resulting knowledge?) In its whole, this sequence of questions are at the core of Artificial Intelligence and may also be referred to as Computer Science problems. In this lecture, we will address some issues raised in red items. Each time, practical examples will be provided Most of our motivation comes from the Big Data world, encountered in image processing, finance, genetics and many other fields where knowledge extraction is needed, when facing to many observations described by many variables. n: number of observations - p: number of variables per observations p >> n >> O(1).

5 Major issues Examples Mathematics I Introduction - Spam classification - Signal Processing datasets From a set of labelled messages (spam or not), build a classification for automatic spam rejection. Select among the words meaningful elements? Automatic classification?

6 Major issues Examples Mathematics I Introduction - Micro-array analysis - Biological datasets One measures micro-array datasets built from a huge amount of profile genes expression. Number of genes p (of order thousands). Number of samples n (of order hundred). Diagnostic help: healthy or ill? Select among the genes meaningful elements? Automatic classification?

7 Major issues Examples Mathematics I Introduction - Fraud detection - Industrial datasets Set of individual electrical consumption for some people in Medellin (Colombia). Each individual provides a monthly electrical consumption. The electrical firm measures the whole consumption for important hubs (they are formed by a set of clients). Want to detect eventual fraud. Problems: Missing data: completion of the table. How? Noise in the several measurements: how does it degrages the fraud detection? Can we exhibit several monthly behaviour of the clients?

8 Introduction Major issues Examples Mathematics I Introduction - Data Completion & Recommandation systems - Advertisement and e-business datasets Recommandation problems: And more recently: What kind of database? Reliable recommandation for clients? Online strategy? S. Gadat Big Data - Lecture 1

9 Major issues Examples Mathematics I Introduction - Credit scoring - Actuaries datasets Build an indicator (Q score) from a dataset for the probability of interest in a financial product (Visa premier credit card). 1 Define a model, a question? 2 Use a supervised classification algorithm to rank the best clients. 3 Use logistic regression to provide a score.

10 Major issues Examples Mathematics I Introduction - What about maths? Various mathematical fields we will talk about: Analysis: Convex optimization, Approximation theory Statistics: Penalized procedures and their reliability Probabilistic methods: concentration inequalities, stochastic processes, stochastic approximations Famous keywords: Lasso Boosting Convex relaxation Supervised classification Support Vector Machine Aggregation rules Gradient descent Stochastic Gradient descent Sequential prediction Bandit games, minimax policies Matrix completion

11 Schedule Introduction Convex functions Example of convex functions Gradient descent method 1 Introduction Major issues Examples Mathematics 2 Convex functions Example of convex functions Gradient descent method 3 Equality constraint Inequality constraint KKT Conditions 4

12 Convex functions Example of convex functions Gradient descent method Convex functions We recall some background material that is necessary for a clear understanding of how some machine learning procedures work. We will cover some basic relationships between convexity, positive semidefiniteness, local and global minimizers. (Convex sets, convex functions) A set D is convex if for any (x 1, x 2) D 2 and all α [0, 1], x = αx 1 + (1 α)x 2 D. A function f is convex if its domain D is convex f(x) = f(αx 1 + (1 α)x 2) αf(x 1) + (1 α)f(x 2). (Positive Semi Definite matrix (PSD)) A p p matrix H is (PSD) if for all p 1 vectors z, we have z t Hz 0. There exists a strong link between SDP matrix and convex functions, given by the following proposition. Proposition A smooth C 2 (D) function f is convex if and only if D 2 f is SDP at any point of D. The proof follows easily from a second order Taylor expansion.

13 Example of convex functions Convex functions Example of convex functions Gradient descent method Exponential function: θ R exp(aθ) on R whatever a is. Affine function: θ R d a t θ + b Entropy function: θ R + θ log(θ) p-norm: θ R d θ p := d p θ i p. i=1 Quadratic form: θ R d θ t P θ + 2q t θ + r where P is symetric and positive.

14 Convex functions Example of convex functions Gradient descent method Why convex functions are useful? From external motivations: Many problems in machine learning come from the minimization of a convex criterion and provide meaningful results for the statistical initial task. Many optimization problems admit a convex reformulation (SVM classification or regression, LASSO regression, ridge regression, permutation recovery,... ). From a numerical point of view: Local minimizer = global minimizer. It is a powerful point since in general, descent methods involve f(x) (or something related to), which is a local information on f. x is a local (global) minimizer of f if and only if 0 f(x). Many fast algorithms for the optimization of convex function exist, and sometimes are independent on the dimension d of the original space.

15 Convex functions Example of convex functions Gradient descent method Why convex functions are powerful? Two kind of optimization problems: On the left: non convex optimization problem, use of Travelling Salesman type method. Greedy exploration step (simulated annealing, genetic algortihms). On the right: convex optimization problem, use local descent methods with gradients or subgradients. (Subgradient (nonsmooth functions?)) For any function f : R d R, and any x in R d, the subgradient f(x) is the set of all vectors g such that f(x) f(y) g, x y. This set of subgradients may be empty. Fortunately, it is not the case for convex functions. Proposition f : R d R is convex if and only if f(x) for any x of R d.

16 Convex functions Example of convex functions Gradient descent method Convexity and gradient: Constrained case In either constrained or unconstrained problems, descent methods are powerful with convex functions. In particular, consider constrained problems in X R d. The most famous local descent method relies on y t+1 = x t ηg t where g t f(x t), and where η > 0 is a fixed step-size parameters. x t+1 = Π X (y t+1), Théorème (Convergence of the projected gradient descent method, fixed step-size) If f is convex over X with X B(0, R) and f L, the choice η = ( ) 1 t f x s min f RL t s=1 S. Gadat Big Data - Lecture t 1 R L leads to t

17 Convex functions Example of convex functions Gradient descent method Convexity and gradient: Smooth unconstrained case Results can be seriously improved with smooth functions with bounded second derivatives. f is β smooth if f is β Lipschitz: Standard gradient descent over R d becomes f(x) f(y) β x y. x t+1 = x t η f(x t), Théorème (Convergence of the gradient descent method, β smooth function) If f is a convex and β-smooth function, then η = 1 β leads to ( ) 1 t 2β x1 x0 2 f x s min f t t 1 s=1 Remark Note that the two past results do not depend on the dimension of the state space d. The last result can be extended to the constrained situation.

18 Schedule Introduction Equality constraint Inequality constraint KKT Conditions 1 Introduction Major issues Examples Mathematics 2 Convex functions Example of convex functions Gradient descent method 3 Equality constraint Inequality constraint KKT Conditions 4

19 Equality constraint Inequality constraint KKT Conditions Constrained optimisation: Elements of the problem: θ unknown vector of R d to be recovered J : R d R function to be minimized f i and g i differentiable functions defining a set of constraints. of the problem: min θ R d J(θ) such that: f i(θ) = 0, i = 1,..., n g i(θ) 0, i = 1,..., m Set of admissible vectors: { } Ω := θ R d f i(θ) = 0, i and g j(θ) 0, j

20 Constrained optimisation: Example Equality constraint Inequality constraint KKT Conditions Typical situation: Ω: circle of radius 2 Optimal solution: θ = ( 1, 1) t and J(θ ) = 2. Important restriction: we will restrict our study to convex functions J.

21 Constrained convex optimisation Equality constraint Inequality constraint KKT Conditions A constrained optimization problem is convex if: J is a convex function f i are linear or affine functions g i are convex functions

22 Equality constraint Inequality constraint KKT Conditions Case of a unique equality constraint min J(θ) such that a t θ b = 0 θ Descent direction h: J(θ) t h < 0. Admissible direction h: a t (θ + h) b = 0 a t h = 0. Optimality θ is optimal if there is no admissible descent direction starting from θ. The only possible case is when J(θ ) and a are linearly dependent: λ R J(θ ) + λa = 0. In this situation: J(θ) = ( ) 2θ1 + θ 2 2 θ 1 + 2θ ( 1 and a = 1) Hence, we are looking for θ such that J(θ) a. Computations lead to θ 1 = θ 2. Optimal value reached for θ 1 = 1/2 (and J(θ ) = 15/4).

23 Lagrangian function Equality constraint Inequality constraint KKT Conditions min J(θ) such that f(θ) := a t θ b = 0 θ We have seen the important role of the scalar value λ above. (Lagrangian function) L(λ, θ) = J(θ) + λf(θ) λ is the Lagrange multiplier. The optimal choice of (θ, λ ) corresponds to θ L(λ, θ ) = 0 and λ L(λ, θ ) = 0. Argument: θ is optimal if there is no admissible descent directions h. Hence, J and f are linearly dependent. As a consequence, there exists λ such that θ L(λ, θ ) = J(θ) + λ f(θ) = 0 (Dual equation) Since θ must be admissible, we have θ L(λ, θ ) = f(θ ) = 0 (Primal equation)

24 Equality constraint Inequality constraint KKT Conditions Case of a unique inequality constraint min J(θ) such that g(θ) 0 θ Descent direction h: J(θ) t h < 0. Admissible direction h: g(θ) t h 0 guarantees that g(θ + αh) is decreasing with α. Optimality θ is optimal if there is no admissible descent direction starting from θ. The only possible case is when J(θ ) and g(θ ) are linearly dependent and opposite: λ R J(θ ) = µ g(θ ) with µ 0. We can check that θ = ( 1, 1).

25 Equality constraint Inequality constraint KKT Conditions We consider the minimization problem: min θ J(θ) such that g j(θ) 0, j = 1,..., m f i(θ) = 0, i = 1,..., n (Lagrangian function) We associate to this problem the Lagrange multipliers (λ, µ) = (λ 1,..., λ n, µ 1,..., µ m). n m L(θ, λ, µ) = J(θ) + λ if i(θ) + µ jg j(θ) i=1 j=1 θ primal variables (λ, µ) dual variables

26 KKT Conditions Introduction Equality constraint Inequality constraint KKT Conditions (KKT Conditions) If J and f, g are smooth, we define the Karush-Kuhn-Tucker (KKT) conditions as Stationarity: θ L(λ, µ, θ) = 0. Primal Admissibility: f(θ) = 0 and g(θ) 0. Dual admissibility µ j 0, j = 1,..., m. Theorem A convex minimization problem of J under convex constraints f and g has a solution θ if and only if there exists λ and µ such that KKT conditions hold. Example: J(θ) = 1 2 θ 2 2 s.t. θ 1 2θ We get L(θ, µ) = θ µ(θ 1 + 2θ 2 + 2) with µ 0. Stationarity: (θ 1 + µ, θ 2 2µ) = 0. We deduce that θ = ( 2/5, 4/5). θ 2 = 2θ 1 with θ 2 0.

27 Dual problems (1) Introduction Equality constraint Inequality constraint KKT Conditions We introduce the dual function: We have the following important result Theorem L(λ, µ) = min θ L(θ, λ, µ). Denote the optimal value of the constrained problem p = min {J(θ) f(θ) = 0, g(θ) 0}, then Remark: L(λ, µ) p. The dual function L is lower than p, for any (λ, µ) R n R m + We aim to make this lower bound as close as possible to p : idea to maximize w.r.t. λ, µ the function L. (Dual problem) max L(λ, µ). λ R n,µ R m + L(θ, λ, µ) affine function on λ, µ and thus convex. Hence, L is convex and almost unconstrained.

28 Dual problems (2) Introduction Equality constraint Inequality constraint KKT Conditions Dual problems are easier than primal ones (because of almost constraints omissions). Dual problems are equivalent to primal ones: maximization of the dual minimization of the primal (not shown in this lecture). Dual solutions permit to recover primal ones with KKT conditions (Lagrange multipliers). Example: Lagrangian: L(θ, µ) = θ2 1 +θ µ(θ 1 2θ 2 + 2). Dual function L(µ) = min θ L(θ, µ) = 5 2 µ2 + 2µ. Dual solution: max L(µ) such that µ 0: µ = 2/5. Primal solution: KKT = θ = ( µ, 2µ) = ( 2/5, 4/5).

29 Take home message Big Data problems arise in a large variety of fields. They are complicated for a computational reason (and also for a statistical one, see later). Many Big Data problems will be traduced in an optimization of a convex problem. Efficient algorithms are available to optimize them: independently on the dimension of the underlying space. Primal - Dual formulations are important to overcome some constraints on the optimization. Numerical convex solvers are widely and freely distributed.

### Support Vector Machine (SVM)

Support Vector Machine (SVM) CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin

### Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

### Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

### Introduction to Support Vector Machines. Colin Campbell, Bristol University

Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### ME128 Computer-Aided Mechanical Design Course Notes Introduction to Design Optimization

ME128 Computer-ided Mechanical Design Course Notes Introduction to Design Optimization 2. OPTIMIZTION Design optimization is rooted as a basic problem for design engineers. It is, of course, a rare situation

### Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1

Further Study on Strong Lagrangian Duality Property for Invex Programs via Penalty Functions 1 J. Zhang Institute of Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing

### Convex Optimization SVM s and Kernel Machines

Convex Optimization SVM s and Kernel Machines S.V.N. Vishy Vishwanathan vishy@axiom.anu.edu.au National ICT of Australia and Australian National University Thanks to Alex Smola and Stéphane Canu S.V.N.

### 10. Proximal point method

L. Vandenberghe EE236C Spring 2013-14) 10. Proximal point method proximal point method augmented Lagrangian method Moreau-Yosida smoothing 10-1 Proximal point method a conceptual algorithm for minimizing

### LAGRANGIAN RELAXATION TECHNIQUES FOR LARGE SCALE OPTIMIZATION

LAGRANGIAN RELAXATION TECHNIQUES FOR LARGE SCALE OPTIMIZATION Kartik Sivaramakrishnan Department of Mathematics NC State University kksivara@ncsu.edu http://www4.ncsu.edu/ kksivara SIAM/MGSA Brown Bag

### Nonlinear Programming Methods.S2 Quadratic Programming

Nonlinear Programming Methods.S2 Quadratic Programming Operations Research Models and Methods Paul A. Jensen and Jonathan F. Bard A linearly constrained optimization problem with a quadratic objective

### GI01/M055 Supervised Learning Proximal Methods

GI01/M055 Supervised Learning Proximal Methods Massimiliano Pontil (based on notes by Luca Baldassarre) (UCL) Proximal Methods 1 / 20 Today s Plan Problem setting Convex analysis concepts Proximal operators

### Minimize subject to. x S R

Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### Outline. Optimization scheme Linear search methods Gradient descent Conjugate gradient Newton method Quasi-Newton methods

Outline 1 Optimization without constraints Optimization scheme Linear search methods Gradient descent Conjugate gradient Newton method Quasi-Newton methods 2 Optimization under constraints Lagrange Equality

### Optimization of Design. Lecturer:Dung-An Wang Lecture 12

Optimization of Design Lecturer:Dung-An Wang Lecture 12 Lecture outline Reading: Ch12 of text Today s lecture 2 Constrained nonlinear programming problem Find x=(x1,..., xn), a design variable vector of

### Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Linear Programming Notes V Problem Transformations

Linear Programming Notes V Problem Transformations 1 Introduction Any linear programming problem can be rewritten in either of two standard forms. In the first form, the objective is to maximize, the material

### Increasing for all. Convex for all. ( ) Increasing for all (remember that the log function is only defined for ). ( ) Concave for all.

1. Differentiation The first derivative of a function measures by how much changes in reaction to an infinitesimal shift in its argument. The largest the derivative (in absolute value), the faster is evolving.

### A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Lecture 6: Logistic Regression

Lecture 6: CS 194-10, Fall 2011 Laurent El Ghaoui EECS Department UC Berkeley September 13, 2011 Outline Outline Classification task Data : X = [x 1,..., x m]: a n m matrix of data points in R n. y { 1,

### Distributed Machine Learning and Big Data

Distributed Machine Learning and Big Data Sourangshu Bhattacharya Dept. of Computer Science and Engineering, IIT Kharagpur. http://cse.iitkgp.ac.in/~sourangshu/ August 21, 2015 Sourangshu Bhattacharya

### Making Sense of the Mayhem: Machine Learning and March Madness

Making Sense of the Mayhem: Machine Learning and March Madness Alex Tran and Adam Ginzberg Stanford University atran3@stanford.edu ginzberg@stanford.edu I. Introduction III. Model The goal of our research

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Support Vector Machines

Support Vector Machines Charlie Frogner 1 MIT 2011 1 Slides mostly stolen from Ryan Rifkin (Google). Plan Regularization derivation of SVMs. Analyzing the SVM problem: optimization, duality. Geometric

### (Quasi-)Newton methods

(Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting

### Online Convex Optimization

E0 370 Statistical Learning heory Lecture 19 Oct 22, 2013 Online Convex Optimization Lecturer: Shivani Agarwal Scribe: Aadirupa 1 Introduction In this lecture we shall look at a fairly general setting

Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours MNOS 2014-2015 114 / 267 Lecture Outline 1 Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse

### Lecture 2: The SVM classifier

Lecture 2: The SVM classifier C19 Machine Learning Hilary 2015 A. Zisserman Review of linear classifiers Linear separability Perceptron Support Vector Machine (SVM) classifier Wide margin Cost function

### Interior Point Methods and Linear Programming

Interior Point Methods and Linear Programming Robert Robere University of Toronto December 13, 2012 Abstract The linear programming problem is usually solved through the use of one of two algorithms: either

### Stochastic gradient methods for machine learning

Stochastic gradient methods for machine learning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with Eric Moulines, Nicolas Le Roux and Mark Schmidt - April 2013 Context Machine

### Big Data Analytics. Lucas Rego Drumond

Big Data Analytics Lucas Rego Drumond Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Going For Large Scale Going For Large Scale 1

### Introduction to Machine Learning NPFL 054

Introduction to Machine Learning NPFL 054 http://ufal.mff.cuni.cz/course/npfl054 Barbora Hladká hladka@ufal.mff.cuni.cz Martin Holub holub@ufal.mff.cuni.cz Charles University, Faculty of Mathematics and

### Mathematical finance and linear programming (optimization)

Mathematical finance and linear programming (optimization) Geir Dahl September 15, 2009 1 Introduction The purpose of this short note is to explain how linear programming (LP) (=linear optimization) may

### On generalized gradients and optimization

On generalized gradients and optimization Erik J. Balder 1 Introduction There exists a calculus for general nondifferentiable functions that englobes a large part of the familiar subdifferential calculus

### Support Vector Machines Explained

March 1, 2009 Support Vector Machines Explained Tristan Fletcher www.cs.ucl.ac.uk/staff/t.fletcher/ Introduction This document has been written in an attempt to make the Support Vector Machines (SVM),

### Big learning: challenges and opportunities

Big learning: challenges and opportunities Francis Bach SIERRA Project-team, INRIA - Ecole Normale Supérieure December 2013 Omnipresent digital media Scientific context Big data Multimedia, sensors, indicators,

### Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

### Beyond stochastic gradient descent for large-scale machine learning

Beyond stochastic gradient descent for large-scale machine learning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with Eric Moulines, Nicolas Le Roux and Mark Schmidt - ECML-PKDD,

### Social Media Mining. Data Mining Essentials

Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems

A Lagrangian-DNN Relaxation: a Fast Method for Computing Tight Lower Bounds for a Class of Quadratic Optimization Problems Sunyoung Kim, Masakazu Kojima and Kim-Chuan Toh October 2013 Abstract. We propose

### A Simple Introduction to Support Vector Machines

A Simple Introduction to Support Vector Machines Martin Law Lecture for CSE 802 Department of Computer Science and Engineering Michigan State University Outline A brief history of SVM Large-margin linear

### A Stochastic 3MG Algorithm with Application to 2D Filter Identification

A Stochastic 3MG Algorithm with Application to 2D Filter Identification Emilie Chouzenoux 1, Jean-Christophe Pesquet 1, and Anisia Florescu 2 1 Laboratoire d Informatique Gaspard Monge - CNRS Univ. Paris-Est,

### Date: April 12, 2001. Contents

2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........

### Introduction to Online Learning Theory

Introduction to Online Learning Theory Wojciech Kot lowski Institute of Computing Science, Poznań University of Technology IDSS, 04.06.2013 1 / 53 Outline 1 Example: Online (Stochastic) Gradient Descent

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### Introduction to Machine Learning. Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011

Introduction to Machine Learning Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011 1 Outline 1. What is machine learning? 2. The basic of machine learning 3. Principles and effects of machine learning

### By W.E. Diewert. July, Linear programming problems are important for a number of reasons:

APPLIED ECONOMICS By W.E. Diewert. July, 3. Chapter : Linear Programming. Introduction The theory of linear programming provides a good introduction to the study of constrained maximization (and minimization)

### 2014-2015 The Master s Degree with Thesis Course Descriptions in Industrial Engineering

2014-2015 The Master s Degree with Thesis Course Descriptions in Industrial Engineering Compulsory Courses IENG540 Optimization Models and Algorithms In the course important deterministic optimization

### The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method

The Steepest Descent Algorithm for Unconstrained Optimization and a Bisection Line-search Method Robert M. Freund February, 004 004 Massachusetts Institute of Technology. 1 1 The Algorithm The problem

### Support Vector Machines

CS229 Lecture notes Andrew Ng Part V Support Vector Machines This set of notes presents the Support Vector Machine (SVM) learning algorithm. SVMs are among the best (and many believe are indeed the best)

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### The p-norm generalization of the LMS algorithm for adaptive filtering

The p-norm generalization of the LMS algorithm for adaptive filtering Jyrki Kivinen University of Helsinki Manfred Warmuth University of California, Santa Cruz Babak Hassibi California Institute of Technology

### FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT MINING SYSTEM

International Journal of Innovative Computing, Information and Control ICIC International c 0 ISSN 34-48 Volume 8, Number 8, August 0 pp. 4 FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT

### Department of Industrial Engineering

Department of Industrial Engineering Master of Engineering Program in Industrial Engineering (International Program) M.Eng. (Industrial Engineering) Plan A Option 2: Total credits required: minimum 39

### Principles of demand management Airline yield management Determining the booking limits. » A simple problem» Stochastic gradients for general problems

Demand Management Principles of demand management Airline yield management Determining the booking limits» A simple problem» Stochastic gradients for general problems Principles of demand management Issues:»

### The equivalence of logistic regression and maximum entropy models

The equivalence of logistic regression and maximum entropy models John Mount September 23, 20 Abstract As our colleague so aptly demonstrated ( http://www.win-vector.com/blog/20/09/the-simplerderivation-of-logistic-regression/

### Proximal mapping via network optimization

L. Vandenberghe EE236C (Spring 23-4) Proximal mapping via network optimization minimum cut and maximum flow problems parametric minimum cut problem application to proximal mapping Introduction this lecture:

### CHAPTER 9. Integer Programming

CHAPTER 9 Integer Programming An integer linear program (ILP) is, by definition, a linear program with the additional constraint that all variables take integer values: (9.1) max c T x s t Ax b and x integral

### Online Learning. 1 Online Learning and Mistake Bounds for Finite Hypothesis Classes

Advanced Course in Machine Learning Spring 2011 Online Learning Lecturer: Shai Shalev-Shwartz Scribe: Shai Shalev-Shwartz In this lecture we describe a different model of learning which is called online

### The Method of Lagrange Multipliers

The Method of Lagrange Multipliers S. Sawyer October 25, 2002 1. Lagrange s Theorem. Suppose that we want to maximize (or imize a function of n variables f(x = f(x 1, x 2,..., x n for x = (x 1, x 2,...,

### Convex Programming Tools for Disjunctive Programs

Convex Programming Tools for Disjunctive Programs João Soares, Departamento de Matemática, Universidade de Coimbra, Portugal Abstract A Disjunctive Program (DP) is a mathematical program whose feasible

### Convex analysis and profit/cost/support functions

CALIFORNIA INSTITUTE OF TECHNOLOGY Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m

### Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

### Quiz 1 Sample Questions IE406 Introduction to Mathematical Programming Dr. Ralphs

Quiz 1 Sample Questions IE406 Introduction to Mathematical Programming Dr. Ralphs These questions are from previous years and should you give you some idea of what to expect on Quiz 1. 1. Consider the

### Estimating the Inverse Covariance Matrix of Independent Multivariate Normally Distributed Random Variables

Estimating the Inverse Covariance Matrix of Independent Multivariate Normally Distributed Random Variables Dominique Brunet, Hanne Kekkonen, Vitor Nunes, Iryna Sivak Fields-MITACS Thematic Program on Inverse

### Kuhn-Tucker theorem foundations and its application in mathematical economics

MPRA Munich Personal RePEc Archive Kuhn-Tucker theorem foundations and its application in mathematical economics Dushko Josheski and Elena Gelova University Goce Delcev-Stip 12. October 2013 Online at

### Perron vector Optimization applied to search engines

Perron vector Optimization applied to search engines Olivier Fercoq INRIA Saclay and CMAP Ecole Polytechnique May 18, 2011 Web page ranking The core of search engines Semantic rankings (keywords) Hyperlink

### Variational approach to restore point-like and curve-like singularities in imaging

Variational approach to restore point-like and curve-like singularities in imaging Daniele Graziani joint work with Gilles Aubert and Laure Blanc-Féraud Roma 12/06/2012 Daniele Graziani (Roma) 12/06/2012

### Machine Learning and Pattern Recognition Logistic Regression

Machine Learning and Pattern Recognition Logistic Regression Course Lecturer:Amos J Storkey Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh Crichton Street,

### Readings. D Chapter 1. Lecture 2: Constrained Optimization. Cecilia Fieler. Example: Input Demand Functions. Consumer Problem

Economics 245 January 17, 2012 : Example Readings D Chapter 1 : Example The FOCs are max p ( x 1 + x 2 ) w 1 x 1 w 2 x 2. x 1,x 2 0 p 2 x i w i = 0 for i = 1, 2. These are two equations in two unknowns,

### Learning outcomes. Knowledge and understanding. Competence and skills

Syllabus Master s Programme in Statistics and Data Mining 120 ECTS Credits Aim The rapid growth of databases provides scientists and business people with vast new resources. This programme meets the challenges

### Machine learning and optimization for massive data

Machine learning and optimization for massive data Francis Bach INRIA - Ecole Normale Supérieure, Paris, France ÉCOLE NORMALE SUPÉRIEURE Joint work with Eric Moulines - IHES, May 2015 Big data revolution?

### i=1 In practice, the natural logarithm of the likelihood function, called the log-likelihood function and denoted by

Statistics 580 Maximum Likelihood Estimation Introduction Let y (y 1, y 2,..., y n be a vector of iid, random variables from one of a family of distributions on R n and indexed by a p-dimensional parameter

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### CSCI567 Machine Learning (Fall 2014)

CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 22, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 22, 2014 1 /

### LABEL PROPAGATION ON GRAPHS. SEMI-SUPERVISED LEARNING. ----Changsheng Liu 10-30-2014

LABEL PROPAGATION ON GRAPHS. SEMI-SUPERVISED LEARNING ----Changsheng Liu 10-30-2014 Agenda Semi Supervised Learning Topics in Semi Supervised Learning Label Propagation Local and global consistency Graph

### An Overview Of Software For Convex Optimization. Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.

An Overview Of Software For Convex Optimization Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.edu In fact, the great watershed in optimization isn t between linearity

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Online Learning, Stability, and Stochastic Gradient Descent

Online Learning, Stability, and Stochastic Gradient Descent arxiv:1105.4701v3 [cs.lg] 8 Sep 2011 September 9, 2011 Tomaso Poggio, Stephen Voinea, Lorenzo Rosasco CBCL, McGovern Institute, CSAIL, Brain

### NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice

NONLINEAR AND DYNAMIC OPTIMIZATION From Theory to Practice IC-32: Winter Semester 2006/2007 Benoît C. CHACHUAT Laboratoire d Automatique, École Polytechnique Fédérale de Lausanne CONTENTS 1 Nonlinear

### Optical Flow. Thomas Pock

Optical Flow Thomas Pock 1 Optical Flow (II) Content Global approaches (Horn-Schunck, TV-L1) Coarse-to-fine warping 2 The Horn and Schunck (HS) Method 3 The Horn and Schunck (HS) Method Global energy to

### Machine Learning. CUNY Graduate Center, Spring 2013. Professor Liang Huang. huang@cs.qc.cuny.edu

Machine Learning CUNY Graduate Center, Spring 2013 Professor Liang Huang huang@cs.qc.cuny.edu http://acl.cs.qc.edu/~lhuang/teaching/machine-learning Logistics Lectures M 9:30-11:30 am Room 4419 Personnel

### Duality of linear conic problems

Duality of linear conic problems Alexander Shapiro and Arkadi Nemirovski Abstract It is well known that the optimal values of a linear programming problem and its dual are equal to each other if at least

### Government of Russian Federation. Faculty of Computer Science School of Data Analysis and Artificial Intelligence

Government of Russian Federation Federal State Autonomous Educational Institution of High Professional Education National Research University «Higher School of Economics» Faculty of Computer Science School

### Machine learning challenges for big data

Machine learning challenges for big data Francis Bach SIERRA Project-team, INRIA - Ecole Normale Supérieure Joint work with R. Jenatton, J. Mairal, G. Obozinski, N. Le Roux, M. Schmidt - December 2012

### Big Data Analytics: Optimization and Randomization

Big Data Analytics: Optimization and Randomization Tianbao Yang, Qihang Lin, Rong Jin Tutorial@SIGKDD 2015 Sydney, Australia Department of Computer Science, The University of Iowa, IA, USA Department of

### L1 vs. L2 Regularization and feature selection.

L1 vs. L2 Regularization and feature selection. Paper by Andrew Ng (2004) Presentation by Afshin Rostami Main Topics Covering Numbers Definition Convergence Bounds L1 regularized logistic regression L1

### Problem Set 6 - Solutions

ECO573 Financial Economics Problem Set 6 - Solutions 1. Debt Restructuring CAPM. a Before refinancing the stoc the asset have the same beta: β a = β e = 1.2. After restructuring the company has the same

### Course Syllabus For Operations Management. Management Information Systems

For Operations Management and Management Information Systems Department School Year First Year First Year First Year Second year Second year Second year Third year Third year Third year Third year Third

### Fundamentals of Operations Research. Prof. G. Srinivasan. Department of Management Studies. Indian Institute of Technology, Madras. Lecture No.

Fundamentals of Operations Research Prof. G. Srinivasan Department of Management Studies Indian Institute of Technology, Madras Lecture No. # 22 Inventory Models - Discount Models, Constrained Inventory

### Foundations of Machine Learning On-Line Learning. Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu

Foundations of Machine Learning On-Line Learning Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Motivation PAC learning: distribution fixed over time (training and test). IID assumption.

### Dual Methods for Total Variation-Based Image Restoration

Dual Methods for Total Variation-Based Image Restoration Jamylle Carter Institute for Mathematics and its Applications University of Minnesota, Twin Cities Ph.D. (Mathematics), UCLA, 2001 Advisor: Tony

### Notes V General Equilibrium: Positive Theory. 1 Walrasian Equilibrium and Excess Demand

Notes V General Equilibrium: Positive Theory In this lecture we go on considering a general equilibrium model of a private ownership economy. In contrast to the Notes IV, we focus on positive issues such

### Operations Research and Financial Engineering. Courses

Operations Research and Financial Engineering Courses ORF 504/FIN 504 Financial Econometrics Professor Jianqing Fan This course covers econometric and statistical methods as applied to finance. Topics