# MATLAB and Big Data: Illustrative Example

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 MATLAB and Big Data: Illustrative Example Rick Mansfield Cornell University August 19, 2014

2 Goals Use a concrete example from my research to: Demonstrate the value of vectorization Introduce key commands/functions facilitating vectorization in MATLAB Highlight the value (and necessity) of using sparse matrices Demonstrate the uses for subscript, logical and linear indexing Demonstrate the syntax for calling optimization routines in MATLAB

3 Background Research Question: How much does the school system/neighborhood you attend/grow up in affect long run outcomes (wages, college completion, etc.)? Challenge: Schools/neighborhoods with more desirable amenities attract students and parents with potentially superior (observable and unobservable) characteristics Simple model of outcomes: Y s = X s β + X U s β U + SQ s Project Goal: Provide a lower bound estimate of the contribution of schools/neighborhoods to the cross-sectional variance in later outcomes.

4 Background If students had fully idiosyncratic preferences for schools/neighborhoods... Then school-average values of each student characteristic would converge to the population average at each school as school sizes became large. Variation in school-average student characteristics reflects the fact that these characteristics predict relative taste for particular school/neighborhood amenities.

5 Background Key hypothesis: If unobservable student characteristics only affect tastes for the same set of school/neighborhood amenities as observable characteristics... Then average values of both sets of characteristics will only differ across schools because these amenities differ across schools. School-average values of observable student characteristics can serve as a control function for school-average values of unobservable student characteristics. Y s = X s β C + ν s, cov(ν s, X U s β U ) = 0

6 Background Control function will also soak up all variation in the desired amenities But can isolate the contribution to earnings of the component of school/neighborhood quality unknown or unvalued by parents at the time school/neighborhood is chosen.

7 Goal of Sample Program Test this hypothesis... by performing a Monte Carlo simulation of a model of school choice. Since the hypothesis requires observing... A large number of student characteristics for each student A large number of students at each school A large number of schools... Computational efficiency is at a premium here! Good context for illustrating MATLAB big data techniques. Currently using North Carolina administrative data that satisfies all these requirements!

8 Model of School Choice Hedonic model of school choice Set of L observable student characteristics, {X 1i,..., X Li } Set of M unobserved student characteristics, {X U 1i,..., XU Mi } Set of K <= L underlying desired/undesired school/neighborhood-level amenities {A 1s,..., A Ks } Assumes that amenity space has an underlying factor structure All variation in school/neighborhood desirability captured by K factors.

9 Model of School Choice W T P i (s) = γ 1i A 1s + γ 2i A 2s + + γ Ki A Ks + ɛ s,i. (1) ɛ s,i reflects idiosyncratic tastes of student i for school/neighborhood s that are unrelated to the school s amenities.

10 Model of School Choice Let the taste parameters γ ki depend on all L elements of X i and all M elements of X U i γ ki = L M δ kl X li + δkm U XU mi + κ ki (2) l=1 m=1 Formulation allows each observable and unobservable student characteristic to differentially affect the relative taste for each of the K amenities κ ki is the component of i s taste for amenity k that is unpredictable given X i and X U i κ ki influences school choice but has no direct effect on student outcomes.

11 General Case: Model of School Choice Can collect the terms involving the κ ki and combine them with ɛ si to rewrite as where W T P i (s) = K L ( δ kl X li + k=1 l=1 ɛ si = M δkm U XU mi)a ks + ɛ si m=1 K κ ki A ks + ɛ si k=1 Given an inelastic housing supply, the equilibrium price function P (A 1s,..., A Ks ) adjusts to clear the housing market.

12 Simulation First step: Draw a matrix of preferences for N students choosing between S schools. Could do this by writing a nested for-loop...

13 Vectorization Or, could vectorize. Write the above equation in matrix form, and use matrix operations: W T P i (s) = K k=1 (X iδ k + X U i δu k )A ks + ɛ si X i : 1xL, δ k : Lx1 Did it speed up the computation? Try re-running the program with L = 500, M = 500 How about now? Vectorization tends to pay off when loops require many iterations.

14 Vectorization Can vectorize further... W T P i (s) = (X i δ + X U i δu )A s + ɛ si δ: LxK, A s : Kx1 Small efficiency payoff this time (despite small K)

15 Vectorization Can vectorize further... WTP = (Xδ + X U δ U + κ)a + ɛ X: IxL, A: KxS. Huge efficiency payoff from eliminating the long student loop. Note that testing with a small sample would be misleading about the time savings from vectorization!

16 Approach to Simulation One possibility: Have students/parents make school/neighborhood choices based on initial price vector... Raise the price of oversubscribed schools/neighborhoods and lower the price of undersubscribed schools/neighborhoods... Until equilibrium is reached in which house prices clear the housing market. Requires a while-loop, can be quite slow to converge.

17 Alternative approach to Simulation Use the 1st Welfare Theorem: A perfectly competitive market always reaches the efficient allocation. Can solve directly for the allocation that maximizes the total willingness of pay of all students... Subject to the constraints that No school can be oversubscribed Every child must attend exactly one school

18 Formal Statement of Problem max f:i S s.t. i W T P is i I 1(f(i) = s) = I s s s.t. s 1(f(i) = s) = 1 i (3) 1(f(i) = s) indicates that student i chooses school/neighborhood s.

19 Linear Programming Version This optimization problem can be recast as a linear programming problem: max x UT x s.t. P x = I s s s.t. Q x = 1 i s.t. x 0 s.t. x 1 (4) U is a (N S)x1 column created by stacking the rows of the NxS matrix of student W T P for schools. x consists of a (N S)x1 vector indicating potential school assignments.

20 Linear Programming Problems in MATLAB Can call MATLABs linear programming solver: linprog. linprog requires the user to supply several arguments: An objective function (the row vector U, in this case) A matrix of inequality constraints on feasible allocations (each row represents a different constraint) The values associated with these inequality constraints A matrix of equality constraints on feasible allocations The values associated with these equality constraints An initial allocation (starting point) Upper and lower bounds for each parameter Display and algorithm choice options

21 Linear Programming Problem Need U to look like: U = U 11. U N1 U 12. U N2. U 1S. U NS

22 Linear Programming Problem Need x to look like: x = x 11. x N1 x 12. x N2. x 1S. x NS

23 Linear Programming Problem Matrix of school size constraints P x = I P = I 1 I 2 I =. I S

24 Linear Programming Problem Matrix of constraints requiring that each student select one school: Qx = Q =

25 Kronecker Product Can write a loop to populate these constraint matrices or can vectorize using the Kronecker product. Kronecker product of two matrices formed by taking products of each pair of elements of the two matrices. Kronecker product of A and B denoted A B

26 Example ( ) A11 A A = 12 A 21 A 22 ( ) B11 B B = 12 B 21 B 22 ( ) A11 B A A B = 12 B A 21 B A 22 B A 11 B 11 A 11 B 12 A 12 B 11 A 12 B 12 = A 11 B 21 A 11 B 22 A 12 B 21 A 12 B 22 A 21 B 11 A 21 B 12 A 22 B 11 A 22 B 12 A 21 B 21 A 21 B 22 A 22 B 21 A 22 B 22

27 School Capacity Constraints Let A be an NxN identity matrix: A = Let B be a row vector of S ones: B = ( ) A B =

28 Individual School Attendance Requirements B A =

29 Problem Just one problem... The school size constraint matrix (P ) is Sx(N S)... and the individual attendance constraint matrix (Q) is Nx(N S) If S is 100 and N is then P is 100x10, 000, And Q is 100, 000x10, 000, 000! Matrices this big require far more memory than even the largest servers can provide to MATLAB... unless elements are stored in binary format!

30 Solution: Sparse Matrices Notice that to re-create either constraint matrix we only need to know the row and column positions of each of the 1s... And there are only N S ones in each matrix (a single one in each column). Thus, we only really need to store N S 2 pieces of information (rows and columns for each one) More generally, if the ones were replaced by an arbitrary set of real numbers... Could recreate the matrix if we knew the positions of the real numbers and their values N S 3 pieces of information.

31 Solution: Sparse Matrices Matrices that consist primarily of zeros are referred to as sparse matrices. MATLAB can store matrices in sparse format... Only the positions and values of non-zero entries are stored. Can perform operations on huge matrices... As long as there aren t too many non-zero elements.

32 The Sparse Command in MATLAB The sparse command in MATLAB creates a sparse matrix given the following arguments A vector of row indices for each non-zero element A vector of column indices for each non-zero element A vector of values for each non-zero element The number of total rows in the matrix The number of total columns in the matrix The total number of non-zero elements for which MATLAB should clear space (useful if you expect to add non-zero elements to the matrix later on in the program)

33 School size constraints in sparse format What is the appropriate vector of row indices for the school size constraints? Need the first S elements to be 1, the next S elements to be 2,... last S elements to be S Can create this vector using R = D E, where: D = S E =

34 School size constraints in sparse format Column indices are straightforward: C = N S Since all the non-zero entries are ones, the value vector simply consists of N S ones. Number of rows = S, number of columns = N S, number of nonzeros = N S

35 Matrix Indexing Three basic techniques for selecting elements of a matrix: Subscript Indexing Logical Indexing Linear Indexing Each is appropriate for different forms of selection

36 Subscript Indexing 5th student s WTP for the 8th school: WTP(5,8) 5th student s WTP for all schools: WTP(5,:) First 5 student s WTP at all schools: WTP(1:5,:) 5th and 7th student s WTP at 4th and 6th schools: WTP([5,7],[4,6]) 5th student s WTP at their chosen school: WTP(5,Assignment(5)) 5th and 7th student s WTP at their own and each other s chosen schools: WTP([5,7],Assignment([5,7]))

37 Logical Indexing 2 4 W T P = Suppose you want to identify all student-school combinations in which the student WTP is less than 0 (willing to pay to avoid the school/live in the neighborhood); Can create a logical matrix: W T P ltzero = (W T P < 0): 1 0 W T P ltzero = The logical matrix places ones at the positions of elements that satisfy the provided condition.

38 Logical Indexing Can replace negative values with zero via: W T P (W T P ltzero ) = 0: 0 4 W T P =

39 Logical Indexing In my program, I want to convert the (N S)x1 vector of 0 s and 1 s indicating school choices into a vector that provides the index number of the school chosen for each student. First, can transform the output from linprog into an NxS matrix using reshape: Assignment2 = reshape(assignment, [N, S]): Assignment = 1 0 Assignment2 =

40 Logical Indexing Can also transpose to observe, for each school, which students selected the school: Assignment3 = (Assignment2) : ( ) Assignment3 = Consider the following matrix, which replicates N times a column of school indices (1,..., S): T emp = repmat((1 : S), 1, N); ( ) T emp =

41 Logical Indexing Can generate the desired N-vector of school assignments via: Assignment3 = logical(assignment3); Schoolassign = Temp(Assignment3): Vector is created by moving down columns (left to right), selecting elements flagged by a 1. Logical indexing is generally quite useful (and efficient!) when you want to select (or replace) elements that satisfy an easily programmed condition.

42 Linear Indexing Suppose you wanted a vector with each student s WTP at their chosen schools only. IOW, you wanted a vector that looks like: [W T P (1, Schoolassign(1)),..., W T P (N, Schoolassign(N))] Can t use W T P (:, Schoolassign(:))... Will get an NxN matrix! Solution: Linear indexing

43 Linear Indexing Can reshape any matrix into a vector by stacking all the columns (moving left to right) Each original element s position in the stacked vector is its linear index. e.g. if a matrix is MxN, the (i, j) element s linear index is M (j 1) + i If a matrix is called with only one argument (e.g. WTP(25)), MATLAB will find the element whose linear index value is 25. If WTP were 10x10, W T P (25) = W T P (5, 3)

44 Linear Indexing To obtain the elements of a matrix associated with a list of subscript pairs: 1. Use the command sub2ind to convert the list to its linear index equivalent 2. Evaluate the original matrix at the linear index vector. Example: W T P _linindex = sub2ind(size(w T P ), [1 : N], Schoolassign); W T P _choice = W T P (W T P _linindex);

45 Accumarray Given the allocation of students to schools provided by the solution to the linear programming problem now need to calculate means of each observable and unobservable student characteristic... at each school. Could parallelize this loop... Or could vectorize using the accumarray command.

46 Accumarray Accumarray offers the same functionality as the by option in Stata: Performs functions within (potentially many) subgroups.

47 Arguments Passed to Accumarray A (possibly multidimensional) index vector identifying the subgroup associated with each observation A vector of values on which the function is to be performed A size for the resulting vector or matrix (necessary when not all subgroups of interest are represented in the index vector) The function being applied within each subgroup A flag for whether you want the output to be sparse A value to be filled in when no observations exist in the subgroup

48 4 student, 2 school case: Accumarray Example 1 Consider a single characteristic, X 1 : 2 X 1 = Suppose the 1st and 4th students attend school 1, 2nd and 3rd students attend school 2. Index vector: 1 IX = 2 2 1

49 Accumarray Example 1 Means = accumarray(ix, X 1, [2, ( ) 5 Means = 2

50 Accumarray Example 2 Can also provide two (or more) dimensions of subscripts to create a matrix of means Note: Vector of values must be a vector not a matrix (so must first reshape matrix to a vector) ( ) 3 IX2 = 1 1 X1 1 2 X = = 8 X

51 Accumarray Example 2 M eans2d = accumarray(ix2, X, [8, ( ) Means2D = 2 3 In my code: X s = accumarray([repmat(schoolassign, L, 1), kron((1 : L), ones(n, 1))], X O (:),

52 Results of Simulation Given school-average values of all observable and unobservable characteristics... Can regress the combined contribution of school-average unobservables on the vector of school-average observables And observe that the contribution to the average school outcome of sorting on unobservables is completely predictable using the average observables. IOW, school-average observables span the same space as school-average unobservables.

53 Results of Simulation Result requires: Large samples of students choosing schools (but not more than is realistic) Dimension of relevant school/neighborhood amenities < Dimension of observed student characteristics No neighborhood attribute whose desirability varies with unobservable but not observable characteristics Residual between-school variation thus reflects the component of school quality not known or valued at the time neighborhoods are chosen. Can evaluate the quality of information about school/neighborhood quality that parents have at their disposal.

54 Useful Commands for Vectorizing MATLAB provides a list of commands commonly used in vectorizing: typing "Vectorization" into help search box and scroll to bottom... But they omit a number of commands I have found to be absolutely essential.

55 Useful Commands for Vectorizing Code accumarray: perform functions by subgroup (equivalent of by in Stata) unique: identify unique elements and (importantly) rows of elements (and their positions in the original matrix) ismember: Determine whether elements of matrix A are also elements of matrix B (and the positions of the A elements in the B matrix, where available) kron: Kronecker product! Very useful for creating analytical Jacobians! intersect: Finds the intersection of the elements of vectors A and B.

56 Final Notes Vectorization is a substitute in many cases for parallel processing... But it is also complementary. Vectorize to the extent possible (given memory limits)... And then process the remaining (much smaller) loop on a small number of processors

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

### Solving polynomial least squares problems via semidefinite programming relaxations

Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose

### Chap 4 The Simplex Method

The Essence of the Simplex Method Recall the Wyndor problem Max Z = 3x 1 + 5x 2 S.T. x 1 4 2x 2 12 3x 1 + 2x 2 18 x 1, x 2 0 Chap 4 The Simplex Method 8 corner point solutions. 5 out of them are CPF solutions.

### General Framework for an Iterative Solution of Ax b. Jacobi s Method

2.6 Iterative Solutions of Linear Systems 143 2.6 Iterative Solutions of Linear Systems Consistent linear systems in real life are solved in one of two ways: by direct calculation (using a matrix factorization,

### Direct Methods for Solving Linear Systems. Matrix Factorization

Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011

### Solving Linear Systems of Equations. Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.

Solving Linear Systems of Equations Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab:

### LINEAR SYSTEMS. Consider the following example of a linear system:

LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x +2x 2 +3x 3 = 5 x + x 3 = 3 3x + x 2 +3x 3 = 3 x =, x 2 =0, x 3 = 2 In general we want to solve n equations in

### Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

### WEEK #3, Lecture 1: Sparse Systems, MATLAB Graphics

WEEK #3, Lecture 1: Sparse Systems, MATLAB Graphics Visualization of Matrices Good visuals anchor any presentation. MATLAB has a wide variety of ways to display data and calculation results that can be

### Chapter 6: Sensitivity Analysis

Chapter 6: Sensitivity Analysis Suppose that you have just completed a linear programming solution which will have a major impact on your company, such as determining how much to increase the overall production

### MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix.

MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix. Inverse matrix Definition. Let A be an n n matrix. The inverse of A is an n n matrix, denoted

### SYSTEMS OF REGRESSION EQUATIONS

SYSTEMS OF REGRESSION EQUATIONS 1. MULTIPLE EQUATIONS y nt = x nt n + u nt, n = 1,...,N, t = 1,...,T, x nt is 1 k, and n is k 1. This is a version of the standard regression model where the observations

### MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix.

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. Nullspace Let A = (a ij ) be an m n matrix. Definition. The nullspace of the matrix A, denoted N(A), is the set of all n-dimensional column

### Lecture 1: Systems of Linear Equations

MTH Elementary Matrix Algebra Professor Chao Huang Department of Mathematics and Statistics Wright State University Lecture 1 Systems of Linear Equations ² Systems of two linear equations with two variables

### Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

### MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### Sensitivity Analysis 3.1 AN EXAMPLE FOR ANALYSIS

Sensitivity Analysis 3 We have already been introduced to sensitivity analysis in Chapter via the geometry of a simple example. We saw that the values of the decision variables and those of the slack and

### CITY UNIVERSITY LONDON. BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION

No: CITY UNIVERSITY LONDON BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION ENGINEERING MATHEMATICS 2 (resit) EX2005 Date: August

### 24. The Branch and Bound Method

24. The Branch and Bound Method It has serious practical consequences if it is known that a combinatorial problem is NP-complete. Then one can conclude according to the present state of science that no

### 2.1: MATRIX OPERATIONS

.: MATRIX OPERATIONS What are diagonal entries and the main diagonal of a matrix? What is a diagonal matrix? When are matrices equal? Scalar Multiplication 45 Matrix Addition Theorem (pg 0) Let A, B, and

### CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES. From Exploratory Factor Analysis Ledyard R Tucker and Robert C.

CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES From Exploratory Factor Analysis Ledyard R Tucker and Robert C MacCallum 1997 180 CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES In

### 2.2/2.3 - Solving Systems of Linear Equations

c Kathryn Bollinger, August 28, 2011 1 2.2/2.3 - Solving Systems of Linear Equations A Brief Introduction to Matrices Matrices are used to organize data efficiently and will help us to solve systems of

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### 1 Introduction. Linear Programming. Questions. A general optimization problem is of the form: choose x to. max f(x) subject to x S. where.

Introduction Linear Programming Neil Laws TT 00 A general optimization problem is of the form: choose x to maximise f(x) subject to x S where x = (x,..., x n ) T, f : R n R is the objective function, S

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Introduction to Matlab

Introduction to Matlab Social Science Research Lab American University, Washington, D.C. Web. www.american.edu/provost/ctrl/pclabs.cfm Tel. x3862 Email. SSRL@American.edu Course Objective This course provides

### DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

### Discrete Convolution and the Discrete Fourier Transform

Discrete Convolution and the Discrete Fourier Transform Discrete Convolution First of all we need to introduce what we might call the wraparound convention Because the complex numbers w j e i πj N have

### IEOR 4404 Homework #2 Intro OR: Deterministic Models February 14, 2011 Prof. Jay Sethuraman Page 1 of 5. Homework #2

IEOR 4404 Homework # Intro OR: Deterministic Models February 14, 011 Prof. Jay Sethuraman Page 1 of 5 Homework #.1 (a) What is the optimal solution of this problem? Let us consider that x 1, x and x 3

### Linear Programming. April 12, 2005

Linear Programming April 1, 005 Parts of this were adapted from Chapter 9 of i Introduction to Algorithms (Second Edition) /i by Cormen, Leiserson, Rivest and Stein. 1 What is linear programming? The first

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### Duality in General Programs. Ryan Tibshirani Convex Optimization 10-725/36-725

Duality in General Programs Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: duality in linear programs Given c R n, A R m n, b R m, G R r n, h R r : min x R n c T x max u R m, v R r b T

### Generalized Neutral Portfolio

Generalized Neutral Portfolio Sandeep Bhutani bhutanister@gmail.com December 10, 2009 Abstract An asset allocation framework decomposes the universe of asset returns into factors either by Fundamental

### Reduced echelon form: Add the following conditions to conditions 1, 2, and 3 above:

Section 1.2: Row Reduction and Echelon Forms Echelon form (or row echelon form): 1. All nonzero rows are above any rows of all zeros. 2. Each leading entry (i.e. left most nonzero entry) of a row is in

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### Methods for Finding Bases

Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,

### 18.06 Problem Set 4 Solution Due Wednesday, 11 March 2009 at 4 pm in 2-106. Total: 175 points.

806 Problem Set 4 Solution Due Wednesday, March 2009 at 4 pm in 2-06 Total: 75 points Problem : A is an m n matrix of rank r Suppose there are right-hand-sides b for which A x = b has no solution (a) What

### Machine Learning and Data Mining. Regression Problem. (adapted from) Prof. Alexander Ihler

Machine Learning and Data Mining Regression Problem (adapted from) Prof. Alexander Ihler Overview Regression Problem Definition and define parameters ϴ. Prediction using ϴ as parameters Measure the error

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 10 Boundary Value Problems for Ordinary Differential Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

### ECON20310 LECTURE SYNOPSIS REAL BUSINESS CYCLE

ECON20310 LECTURE SYNOPSIS REAL BUSINESS CYCLE YUAN TIAN This synopsis is designed merely for keep a record of the materials covered in lectures. Please refer to your own lecture notes for all proofs.

### Generating Random Numbers Variance Reduction Quasi-Monte Carlo. Simulation Methods. Leonid Kogan. MIT, Sloan. 15.450, Fall 2010

Simulation Methods Leonid Kogan MIT, Sloan 15.450, Fall 2010 c Leonid Kogan ( MIT, Sloan ) Simulation Methods 15.450, Fall 2010 1 / 35 Outline 1 Generating Random Numbers 2 Variance Reduction 3 Quasi-Monte

### Equilibria and Dynamics of. Supply Chain Network Competition with Information Asymmetry. and Minimum Quality Standards

Equilibria and Dynamics of Supply Chain Network Competition with Information Asymmetry in Quality and Minimum Quality Standards Anna Nagurney John F. Smith Memorial Professor and Dong (Michelle) Li Doctoral

### x y The matrix form, the vector form, and the augmented matrix form, respectively, for the system of equations are

Solving Sstems of Linear Equations in Matri Form with rref Learning Goals Determine the solution of a sstem of equations from the augmented matri Determine the reduced row echelon form of the augmented

### Maximum Likelihood Estimation of an ARMA(p,q) Model

Maximum Likelihood Estimation of an ARMA(p,q) Model Constantino Hevia The World Bank. DECRG. October 8 This note describes the Matlab function arma_mle.m that computes the maximum likelihood estimates

### Speeding up MATLAB Applications

Speeding up MATLAB Applications Mannheim, 19. Februar 2014 Michael Glaßer Dipl.-Ing. Application Engineer 2014 The MathWorks, Inc. 1 Ihr MathWorks Team heute: Andreas Himmeldorf Senior Team Leader Educational

### (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.

Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product

### Definition of a Linear Program

Definition of a Linear Program Definition: A function f(x 1, x,..., x n ) of x 1, x,..., x n is a linear function if and only if for some set of constants c 1, c,..., c n, f(x 1, x,..., x n ) = c 1 x 1

### 3. Evaluate the objective function at each vertex. Put the vertices into a table: Vertex P=3x+2y (0, 0) 0 min (0, 5) 10 (15, 0) 45 (12, 2) 40 Max

SOLUTION OF LINEAR PROGRAMMING PROBLEMS THEOREM 1 If a linear programming problem has a solution, then it must occur at a vertex, or corner point, of the feasible set, S, associated with the problem. Furthermore,

### Matrix-vector multiplication in terms of dot-products

Matrix-vector multiplication in terms of dot-products Let M be an R C matrix. Dot-Product Definition of matrix-vector multiplication: M u is the R-vector v such that v[r] is the dot-product of row r of

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### ANALYSIS, THEORY AND DESIGN OF LOGISTIC REGRESSION CLASSIFIERS USED FOR VERY LARGE SCALE DATA MINING

ANALYSIS, THEORY AND DESIGN OF LOGISTIC REGRESSION CLASSIFIERS USED FOR VERY LARGE SCALE DATA MINING BY OMID ROUHANI-KALLEH THESIS Submitted as partial fulfillment of the requirements for the degree of

### Linear Programming for Optimization. Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc.

1. Introduction Linear Programming for Optimization Mark A. Schulze, Ph.D. Perceptive Scientific Instruments, Inc. 1.1 Definition Linear programming is the name of a branch of applied mathematics that

### APPM4720/5720: Fast algorithms for big data. Gunnar Martinsson The University of Colorado at Boulder

APPM4720/5720: Fast algorithms for big data Gunnar Martinsson The University of Colorado at Boulder Course objectives: The purpose of this course is to teach efficient algorithms for processing very large

### Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

### Linear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.

Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.

### To give it a definition, an implicit function of x and y is simply any relationship that takes the form:

2 Implicit function theorems and applications 21 Implicit functions The implicit function theorem is one of the most useful single tools you ll meet this year After a while, it will be second nature to

### Chapter 5. Linear Inequalities and Linear Programming. Linear Programming in Two Dimensions: A Geometric Approach

Chapter 5 Linear Programming in Two Dimensions: A Geometric Approach Linear Inequalities and Linear Programming Section 3 Linear Programming gin Two Dimensions: A Geometric Approach In this section, we

### Estimating the random coefficients logit model of demand using aggregate data

Estimating the random coefficients logit model of demand using aggregate data David Vincent Deloitte Economic Consulting London, UK davivincent@deloitte.co.uk September 14, 2012 Introduction Estimation

### Common factor analysis

Common factor analysis This is what people generally mean when they say "factor analysis" This family of techniques uses an estimate of common variance among the original variables to generate the factor

### Stephen du Toit Mathilda du Toit Gerhard Mels Yan Cheng. LISREL for Windows: SIMPLIS Syntax Files

Stephen du Toit Mathilda du Toit Gerhard Mels Yan Cheng LISREL for Windows: SIMPLIS Files Table of contents SIMPLIS SYNTAX FILES... 1 The structure of the SIMPLIS syntax file... 1 \$CLUSTER command... 4

### CHAPTER 11: BASIC LINEAR PROGRAMMING CONCEPTS

Linear programming is a mathematical technique for finding optimal solutions to problems that can be expressed using linear equations and inequalities. If a real-world problem can be represented accurately

### Auxiliary Variables in Mixture Modeling: 3-Step Approaches Using Mplus

Auxiliary Variables in Mixture Modeling: 3-Step Approaches Using Mplus Tihomir Asparouhov and Bengt Muthén Mplus Web Notes: No. 15 Version 8, August 5, 2014 1 Abstract This paper discusses alternatives

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### Linear Algebra Notes

Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note

### Metric Spaces. Chapter 7. 7.1. Metrics

Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some

### LOGISTIC REGRESSION. Nitin R Patel. where the dependent variable, y, is binary (for convenience we often code these values as

LOGISTIC REGRESSION Nitin R Patel Logistic regression extends the ideas of multiple linear regression to the situation where the dependent variable, y, is binary (for convenience we often code these values

### These axioms must hold for all vectors ū, v, and w in V and all scalars c and d.

DEFINITION: A vector space is a nonempty set V of objects, called vectors, on which are defined two operations, called addition and multiplication by scalars (real numbers), subject to the following axioms

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### Chapter 6. Linear Programming: The Simplex Method. Introduction to the Big M Method. Section 4 Maximization and Minimization with Problem Constraints

Chapter 6 Linear Programming: The Simplex Method Introduction to the Big M Method In this section, we will present a generalized version of the simplex method that t will solve both maximization i and

### AN INTRODUCTION TO THE FINITE ELEMENT METHOD FOR YOUNG ENGINEERS

AN INTRODUCTION TO THE FINITE ELEMENT METHOD FOR YOUNG ENGINEERS By: Eduardo DeSantiago, PhD, PE, SE Table of Contents SECTION I INTRODUCTION... 2 SECTION II 1-D EXAMPLE... 2 SECTION III DISCUSSION...

### Bedford, Fowler: Statics. Chapter 4: System of Forces and Moments, Examples via TK Solver

System of Forces and Moments Introduction The moment vector of a force vector,, with respect to a point has a magnitude equal to the product of the force magnitude, F, and the perpendicular distance from

### 2x + y = 3. Since the second equation is precisely the same as the first equation, it is enough to find x and y satisfying the system

1. Systems of linear equations We are interested in the solutions to systems of linear equations. A linear equation is of the form 3x 5y + 2z + w = 3. The key thing is that we don t multiply the variables

### Parameter Estimation: A Deterministic Approach using the Levenburg-Marquardt Algorithm

Parameter Estimation: A Deterministic Approach using the Levenburg-Marquardt Algorithm John Bardsley Department of Mathematical Sciences University of Montana Applied Math Seminar-Feb. 2005 p.1/14 Outline

### Integer Factorization using the Quadratic Sieve

Integer Factorization using the Quadratic Sieve Chad Seibert* Division of Science and Mathematics University of Minnesota, Morris Morris, MN 56567 seib0060@morris.umn.edu March 16, 2011 Abstract We give

### (Quasi-)Newton methods

(Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting

### Operation Count; Numerical Linear Algebra

10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point

### Inverses. Stephen Boyd. EE103 Stanford University. October 27, 2015

Inverses Stephen Boyd EE103 Stanford University October 27, 2015 Outline Left and right inverses Inverse Solving linear equations Examples Pseudo-inverse Left and right inverses 2 Left inverses a number

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### Data Analysis. Using Excel. Jeffrey L. Rummel. BBA Seminar. Data in Excel. Excel Calculations of Descriptive Statistics. Single Variable Graphs

Using Excel Jeffrey L. Rummel Emory University Goizueta Business School BBA Seminar Jeffrey L. Rummel BBA Seminar 1 / 54 Excel Calculations of Descriptive Statistics Single Variable Graphs Relationships

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### Notes on Factoring. MA 206 Kurt Bryan

The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor

### We shall turn our attention to solving linear systems of equations. Ax = b

59 Linear Algebra We shall turn our attention to solving linear systems of equations Ax = b where A R m n, x R n, and b R m. We already saw examples of methods that required the solution of a linear system

### 7.4 Linear Programming: The Simplex Method

7.4 Linear Programming: The Simplex Method For linear programming problems with more than two variables, the graphical method is usually impossible, so the simplex method is used. Because the simplex method

### the points are called control points approximating curve

Chapter 4 Spline Curves A spline curve is a mathematical representation for which it is easy to build an interface that will allow a user to design and control the shape of complex curves and surfaces.

### Moral Hazard. Itay Goldstein. Wharton School, University of Pennsylvania

Moral Hazard Itay Goldstein Wharton School, University of Pennsylvania 1 Principal-Agent Problem Basic problem in corporate finance: separation of ownership and control: o The owners of the firm are typically

### Nonlinear Optimization: Algorithms 3: Interior-point methods

Nonlinear Optimization: Algorithms 3: Interior-point methods INSEAD, Spring 2006 Jean-Philippe Vert Ecole des Mines de Paris Jean-Philippe.Vert@mines.org Nonlinear optimization c 2006 Jean-Philippe Vert,

### Part 2: Analysis of Relationship Between Two Variables

Part 2: Analysis of Relationship Between Two Variables Linear Regression Linear correlation Significance Tests Multiple regression Linear Regression Y = a X + b Dependent Variable Independent Variable

### Eigenvalues and eigenvectors of a matrix

Eigenvalues and eigenvectors of a matrix Definition: If A is an n n matrix and there exists a real number λ and a non-zero column vector V such that AV = λv then λ is called an eigenvalue of A and V is

### Week 5 Integral Polyhedra

Week 5 Integral Polyhedra We have seen some examples 1 of linear programming formulation that are integral, meaning that every basic feasible solution is an integral vector. This week we develop a theory

### 1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

### GenOpt (R) Generic Optimization Program User Manual Version 3.0.0β1

(R) User Manual Environmental Energy Technologies Division Berkeley, CA 94720 http://simulationresearch.lbl.gov Michael Wetter MWetter@lbl.gov February 20, 2009 Notice: This work was supported by the U.S.

### 4.6 Linear Programming duality

4.6 Linear Programming duality To any minimization (maximization) LP we can associate a closely related maximization (minimization) LP. Different spaces and objective functions but in general same optimal

### Optimal Income Taxation with Multidimensional Types

1 Optimal Income Taxation with Multidimensional Types Kenneth L. Judd Hoover Institution NBER Che-Lin Su Northwestern University (Stanford Ph. D.) PRELIMINARY AND INCOMPLETE November, 2006 2 Introduction

### Manifold Learning Examples PCA, LLE and ISOMAP

Manifold Learning Examples PCA, LLE and ISOMAP Dan Ventura October 14, 28 Abstract We try to give a helpful concrete example that demonstrates how to use PCA, LLE and Isomap, attempts to provide some intuition