# Overview. Sparse Matrices for Large Data Modeling. Acknowledgements

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Overview Sparse Matrices for Large Data Modeling Acknowledgements Martin Maechler ETH Zurich Switzerland user!2007, Ames, Iowa, USA Aug Doug Bates has introduced me to sparse matrices and the Matrix package, and the last two years have been a lot of fun in collaboration with him. Large Data not just sparse matrices (Sparse) Matrices for Large Data : Applications LMER talk by Doug Bates * Quantile Smoothing with Constraints Regression Splines for Total Variation Penalized Densities; notably in 2D (triograms) * Sparse Least Squares Regression (and Generalized, Robust,...) Sparse Matrix Graph incidence (in Network ) Sparse Covariance or Correlation Matrices and Conditional Variance via sparse arithmetic * Teaser of a case study: ETH professor evaluation by students: Who s the best in teaching? Overview of Sparse Matrices sparse matrix storage Sparse Matrices in R s Matrix: arithmetic, indexing, solve methods, possibly even for sparse RHS. Sparse Matrices factorizations: chol(), qr, Schur, LU, Bunch-Kaufmann Large Data Analysis This session Large Data will focus on one important kind of large data analysis, namely: Sparse Matrix modelling. Further considerations a user should know: 1. Think first, then read the data Read the docs! The R Data Import/Export manual (part of the R manuals that come with R and are online in PDF and HTML). Note section 2.1 Variations on read.table ; and read help(read.table), notably about the colclasses and as.is arguments. How large is large? Do I only need some variables? Should I use a database system from R (SQLite, MySQL)? Rather work with simple random samples of rows?! 2. Think again: First plot, then model! (T-shirt); or slightly generalized: First explore, then model!, i.e., first use exploratory data analysis (EDA)

2 Constrained Quantile Smoothing Splines constrained B-spline fit Pin Ng (1996) defined quantile smoothing splines, as solution of, e.g., min g n ρτ (yi g(xi)) + λ max g (x). x (1) i=1 as a nonparametric estimator for gτ (x), τ (0, 1), where for τ = 1 2, i ρτ (ri) = i ri is least absolute values (L1) regression. Solving (1) means linear optimization with linear constraints, and hence can easily be extended by further (linear) constraints such monotonicity, convexity, upper bounds, exact fit constraints, etc. The matrix X corresponding to the linear optimization problem for the constrained smoothing splines is of dimension (f n) n but has only f2 n (f2 3) non-zero entries. Fit a constrained (B-) smoothing spline to n = 100 data points constrained to be monotone increasing, and fulfull the 3 pointwise constraints (above): > library(cobs) > Phi.cnstr <- rbind(c( 1, -3, 0), ## g(-3) >= 0 + c(-1, 3, 1), ## g(+3) <= 1 + c( 0, 0, 0.5)) ## g( 0) == 0.5 > msp <- cobs(x, y, nknots = length(x) - 1, + constraint = "increase", pointwise = Phi.cnstr, + lambda = 0.1) Example: constraints on g(.) are: g() increasing, i.e., g (x) > 0, and 0 < g( 3) g(0) = 0.5 g(3) < 1, and n = 50 observations, the matrices X and X X are > ## msp <- cobs(...) > plot(msp, main = "cobs(x,y, constraint= \"increase\", pointwise > abline(h = c(0,1), lty=2, col = "olivedrab", lwd = 2) > points(0, 0.5, cex=2, col = "olivedrab") > lines(xx, pnorm(2 * xx), col="light gray")# true function

3 Sparse Least Squares Koenker and Ng (2003) were the first to provide a sparse matrix package for R, including sparse least squares, via slm.fit(x,y,...). They provide the following nice example of a model matrix (probably from a quantile smoothing context): > library(matrix) > data(knex) # Koenker-Ng ex{ample} > dim (KNex\$mm) [1] > print(image(knex\$mm, aspect = "iso", colorkey = FALSE, + main = "Koenker-Ng model matrix")) Cholesky for Sparse L.S. For sparse matrices, the Cholesky decomposition has been the most researched factorization and hence used for least squares regression modelling. Estimating β in the model y = Xβ + ɛ by solving the normal equations X Xβ = X y (2) the Cholesky decomposition of the (symmetric positive semi-definite) X X is LL or R R with lower-left upper-right triangular matrix L or R L, respectively. System solved via two triangular (back- and forward-) solves : ˆβ = (X X) 1 X y = (LL ) 1 X y = L L 1 X y (3) CHOLMOD (Tim Davis, 2006): Efficient sparse algorithms. or rather transposed, for screen display: Cholesky Fill-in The usual Cholesky decomposition works,... > X.X <- crossprod(knex\$mm) > c1 <- chol(x.x) > image(x.x, main= "X X", aspect="iso", colorkey = FALSE) > image(c1, main= "chol(x X)",...) but the resulting cholesky factor has suffered from so-called fill-in, i.e., its sparsity is quite reduced compared to X X.

4 Fill-reducing Permutation So, chol(x X) suffered from fill-in (sparsity decreased considerably). Solution: Fill-reducing techniques which permute rows and columns of X X, i.e., use P X XP for a permutation matrix P or in R syntax, X.X[pvec,pvec] where pvec is a permutation of 1:n. The permutation P is chosen such that the Cholesky factor of P X XP is as sparse as possible > image(t(c1), main= "t( chol(x X) )",...) > c2 <- Cholesky(X.X, perm = TRUE) > image(c2, main= "Cholesky(X X, perm = TRUE)",...) ((Note that such permutations are done for dense chol() when pivot=true, but there the goal is dealing with rank-deficiency.)) Timing Least Squares Solving > y <- KNex\$y > m. <- as(knex\$mm, "matrix") # traditional (dense) Matrix > system.time(cpod.sol <- solve(crossprod(m.), crossprod(m., y))) user system elapsed > ## Using sparse matrices is so fast, we have to bump the time b > system.time(for(i in 1:10) ## sparse solution without permutat + sp1.sol <- solve(c1,solve(t(c1), crossprod(knex\$mm, user system elapsed > system.time(for(i in 1:10) ## sparse Cholesky WITH fill-reduc + sp2.sol <- solve(c2, crossprod(knex\$mm, y))) user system elapsed > stopifnot(all.equal(sp1.sol, sp2.sol), + all.equal(as.vector(sp2.sol), c(cpod.sol))) Fill-reducing Permutation - Result

5 Teaser case study: Who s the best prof? Modelling the ETH teacher ratings Private donation for encouraging excellent teaching at ETH Student union of ETH Zurich organizes survey to award prizes: Best lecturer of ETH, and of each of the 14 departments. Smart Web-interface for survey: Each student sees the names of his/her professors from the last 4 semesters and all the lectures that applied. ratings in {1, 2, 3, 4, 5}. high response rate Model: The rating depends on students (s) (rating subjectively) teacher (d) main interest department (dept) service lecture or own department student, (service: 0/1). semester of student at time of rating (studage {2, 4, 6, 8}). how many semesters back was the lecture (lectage). Main question: Who s the best prof? Hence, for political reasons, want d as a fixed effect. Who s the best prof data Model for ETH teacher ratings > ## read the data; several factor assignments, such as > md\$d <- factor(md\$d) # Lecturer_ID ("d"ozentin) > str(md) Want d ( teacher ID, 1000 levels) as fixed effect. Consequently, in y = Xβ + Zb + ɛ have X as n 1000 (roughly) data.frame : obs. of 7 variables: have Z as n 5000, n \$ s : Factor w/ 2972 levels "1","2","3","4",..: \$ d : Factor w/ 1128 levels "1","6","7","8",..: \$ studage: Ord.factor w/ 4 levels "2"<"4"<"6"<"8": > fm0 <- lmer(y ~ d*dept + dept*service + studage + lectage + (1 \$ lectage: Ord.factor w/ 6 levels "1"<"2"<"3"<"4"<..: data = md) \$ service: Factor w/ 2 levels "0","1": \$ dept : Factor w/ 15 levels "1","2","3","4",..: Error in model.matrix.default(mt, mf, contrasts) : \$ y : int cannot allocate vector of length > / 2^20 ## number of Mega bytes [1] Want sparse matrices for X and Z and crossprods, etc.

6 Intro to Sparse Matrices in R package Matrix simple example 2 The R Package Matrix contains dozens of matrix classes and hundreds of method definitions. Has sub-hierarchies of densematrix and sparsematrix. Very basic intro in some of sparse matrices: > str(a) # note that *internally* 0-based indices (i,j) are used Formal class dgtmatrix [package "Matrix"] with 6 i : int [1:7] j : int [1:7] Dim : int [1:2] 10 Dimnames:List of x : num [1:7] factors : list() > A[2:7, 12:20] <- rep(c(0,0,0,(3:1)*30,0), length = 6*9) simple example Triplet form The most obvious way to store a sparse matrix is the so called Triplet form; (virtual class TsparseMatrix in Matrix): > A <- spmatrix(10,20, i = c(1,3:8), + j = c(2,9,6:10), + x = 7 * (1:7)) > A # a "dgtmatrix" 10 x 20 sparse Matrix of class "dgtmatrix" [1,] [2,] [3,] [4,] [5,] [6,] [7,] [8,] [9,] [10,] simple example 3 > A >= 20 # -> logical sparse; nice show() method 10 x 20 sparse Matrix of class "lgtmatrix" [1,] [2,] [3,] [4,] [5,] [6,] [7,] [8,] [9,] [10,]

7 sparse compressed form Conclusions Triplet representation: easy for us humbly humans, but can be both made smaller and more efficient for (column-access heavy) operations: The column compressed sparse representation, (virtual class CsparseMatrix in Matrix): > Ac <- as(t(a), "CsparseMatrix") > str(ac) Formal class dgcmatrix [package "Matrix"] with 6 i : int [1:30] p : int [1:11] Dim : int [1:2] 20 Dimnames:List of x : num [1:30] factors : list() column index slot j replaced by a column pointer slot p. Sparse Matrices: crucial for several important data modelling situations There s the R package Matrix... Many? more conclusions at the end of Doug Bates talk :-) Other R packages for large matrices biglm updating QR decompositions; storing O(p 2 ) instead of O(n p). R.Huge: Using class FileMatrix to store matrices on disk instead of RAM memory. SQLiteDF by Miguel Manese ( our Google Summer of Code project 2006). Description: Transparently stores data frames & matrices into SQLite tables.... sqldf by Gabor Grothendieck: learn SQL... R ff package (memory mapping arrays) by Adler, Nendic, Zucchini and Glaser: poster of today and winner of the user!2007 programming competition.

### 6. Cholesky factorization

6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

### 7. LU factorization. factor-solve method. LU factorization. solving Ax = b with A nonsingular. the inverse of a nonsingular matrix

7. LU factorization EE103 (Fall 2011-12) factor-solve method LU factorization solving Ax = b with A nonsingular the inverse of a nonsingular matrix LU factorization algorithm effect of rounding error sparse

### 9. Numerical linear algebra background

Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 3 Linear Least Squares Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

### Linear Algebra Review Part 2: Ax=b

Linear Algebra Review Part 2: Ax=b Edwin Olson University of Michigan The Three-Day Plan Geometry of Linear Algebra Vectors, matrices, basic operations, lines, planes, homogeneous coordinates, transformations

### APPM4720/5720: Fast algorithms for big data. Gunnar Martinsson The University of Colorado at Boulder

APPM4720/5720: Fast algorithms for big data Gunnar Martinsson The University of Colorado at Boulder Course objectives: The purpose of this course is to teach efficient algorithms for processing very large

### Lecture 6. Inverse of Matrix

Lecture 6 Inverse of Matrix Recall that any linear system can be written as a matrix equation In one dimension case, ie, A is 1 1, then can be easily solved as A x b Ax b x b A 1 A b A 1 b provided that

### Sparse direct methods

Sparse direct methods Week 6: Monday, Sep 24 Suppose A is a sparse matrix, and P A = LU. Will L and U also be sparse? The answer depends in a somewhat complicated way on the structure of the graph associated

### Linear Systems. Singular and Nonsingular Matrices. Find x 1, x 2, x 3 such that the following three equations hold:

Linear Systems Example: Find x, x, x such that the following three equations hold: x + x + x = 4x + x + x = x + x + x = 6 We can write this using matrix-vector notation as 4 {{ A x x x {{ x = 6 {{ b General

### Lecture 5 - Triangular Factorizations & Operation Counts

LU Factorization Lecture 5 - Triangular Factorizations & Operation Counts We have seen that the process of GE essentially factors a matrix A into LU Now we want to see how this factorization allows us

Your Best Next Business Solution Big Data In R 24/3/2010 Big Data In R R Works on RAM Causing Scalability issues Maximum length of an object is 2^31-1 Some packages developed to help overcome this problem

### Linear Codes. Chapter 3. 3.1 Basics

Chapter 3 Linear Codes In order to define codes that we can encode and decode efficiently, we add more structure to the codespace. We shall be mainly interested in linear codes. A linear code of length

### The ff package: Handling Large Data Sets in R with Memory Mapped Pages of Binary Flat Files

UseR! 2007, Iowa State University, Ames, August 8-108 2007 The ff package: Handling Large Data Sets in R with Memory Mapped Pages of Binary Flat Files D. Adler, O. Nenadić, W. Zucchini, C. Gläser Institute

### (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.

Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product

### Matrix Inverse and Determinants

DM554 Linear and Integer Programming Lecture 5 and Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1 2 3 4 and Cramer s rule 2 Outline 1 2 3 4 and

### LINEAR SYSTEMS. Consider the following example of a linear system:

LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x +2x 2 +3x 3 = 5 x + x 3 = 3 3x + x 2 +3x 3 = 3 x =, x 2 =0, x 3 = 2 In general we want to solve n equations in

### An Introduction to Machine Learning

An Introduction to Machine Learning L5: Novelty Detection and Regression Alexander J. Smola Statistical Machine Learning Program Canberra, ACT 0200 Australia Alex.Smola@nicta.com.au Tata Institute, Pune,

### Matrix Multiplication

Matrix Multiplication CPS343 Parallel and High Performance Computing Spring 2016 CPS343 (Parallel and HPC) Matrix Multiplication Spring 2016 1 / 32 Outline 1 Matrix operations Importance Dense and sparse

### Syntax Description Remarks and examples Also see

Title stata.com permutation An aside on permutation matrices and vectors Syntax Description Remarks and examples Also see Syntax Permutation matrix Permutation vector Action notation notation permute rows

### Description Syntax Remarks and examples Also see

Title stata.com permutation An aside on permutation matrices and vectors Description Syntax Remarks and examples Also see Description Permutation matrices are a special kind of orthogonal matrix that,

### Math 576: Quantitative Risk Management

Math 576: Quantitative Risk Management Haijun Li lih@math.wsu.edu Department of Mathematics Washington State University Week 4 Haijun Li Math 576: Quantitative Risk Management Week 4 1 / 22 Outline 1 Basics

### A Negative Result Concerning Explicit Matrices With The Restricted Isometry Property

A Negative Result Concerning Explicit Matrices With The Restricted Isometry Property Venkat Chandar March 1, 2008 Abstract In this note, we prove that matrices whose entries are all 0 or 1 cannot achieve

### Basics Inversion and related concepts Random vectors Matrix calculus. Matrix algebra. Patrick Breheny. January 20

Matrix algebra January 20 Introduction Basics The mathematics of multiple regression revolves around ordering and keeping track of large arrays of numbers and solving systems of equations The mathematical

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### Linear Systems COS 323

Linear Systems COS 323 Last time: Constrained Optimization Linear constrained optimization Linear programming (LP) Simplex method for LP General optimization With equality constraints: Lagrange multipliers

### Joint models for classification and comparison of mortality in different countries.

Joint models for classification and comparison of mortality in different countries. Viani D. Biatat 1 and Iain D. Currie 1 1 Department of Actuarial Mathematics and Statistics, and the Maxwell Institute

### MATH 3795 Lecture 7. Linear Least Squares.

MATH 3795 Lecture 7. Linear Least Squares. Dmitriy Leykekhman Fall 2008 Goals Basic properties of linear least squares problems. Normal equation. D. Leykekhman - MATH 3795 Introduction to Computational

### Functional Data Analysis with R and MATLAB

J.O. Ramsay Giles Hooker Spencer Graves Functional Data Analysis with R and MATLAB Springer Contents 2 Introduction to Functional Data Analysis. What Are Functional Data?.. Data on the Growth of Girls..2

### Arithmetic and Algebra of Matrices

Arithmetic and Algebra of Matrices Math 572: Algebra for Middle School Teachers The University of Montana 1 The Real Numbers 2 Classroom Connection: Systems of Linear Equations 3 Rational Numbers 4 Irrational

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems

Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001,

### We seek a factorization of a square matrix A into the product of two matrices which yields an

LU Decompositions We seek a factorization of a square matrix A into the product of two matrices which yields an efficient method for solving the system where A is the coefficient matrix, x is our variable

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### Using the Singular Value Decomposition

Using the Singular Value Decomposition Emmett J. Ientilucci Chester F. Carlson Center for Imaging Science Rochester Institute of Technology emmett@cis.rit.edu May 9, 003 Abstract This report introduces

### Solution based on matrix technique Rewrite. ) = 8x 2 1 4x 1x 2 + 5x x1 2x 2 2x 1 + 5x 2

8.2 Quadratic Forms Example 1 Consider the function q(x 1, x 2 ) = 8x 2 1 4x 1x 2 + 5x 2 2 Determine whether q(0, 0) is the global minimum. Solution based on matrix technique Rewrite q( x1 x 2 = x1 ) =

### Data Structure. Lecture 3

Data Structure Lecture 3 Data Structure Formally define Data structure as: DS describes not only set of objects but the ways they are related, the set of operations which may be applied to the elements

### ANALYSIS, THEORY AND DESIGN OF LOGISTIC REGRESSION CLASSIFIERS USED FOR VERY LARGE SCALE DATA MINING

ANALYSIS, THEORY AND DESIGN OF LOGISTIC REGRESSION CLASSIFIERS USED FOR VERY LARGE SCALE DATA MINING BY OMID ROUHANI-KALLEH THESIS Submitted as partial fulfillment of the requirements for the degree of

### Sections 2.11 and 5.8

Sections 211 and 58 Timothy Hanson Department of Statistics, University of South Carolina Stat 704: Data Analysis I 1/25 Gesell data Let X be the age in in months a child speaks his/her first word and

### Statistical Foundations: Measures of Location and Central Tendency and Summation and Expectation

Statistical Foundations: and Central Tendency and and Lecture 4 September 5, 2006 Psychology 790 Lecture #4-9/05/2006 Slide 1 of 26 Today s Lecture Today s Lecture Where this Fits central tendency/location

### Operation Count; Numerical Linear Algebra

10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point

### Elementary Matrices and The LU Factorization

lementary Matrices and The LU Factorization Definition: ny matrix obtained by performing a single elementary row operation (RO) on the identity (unit) matrix is called an elementary matrix. There are three

### Practical Numerical Training UKNum

Practical Numerical Training UKNum 7: Systems of linear equations C. Mordasini Max Planck Institute for Astronomy, Heidelberg Program: 1) Introduction 2) Gauss Elimination 3) Gauss with Pivoting 4) Determinants

### AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 19: SVD revisited; Software for Linear Algebra Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 9 Outline 1 Computing

### Solving polynomial least squares problems via semidefinite programming relaxations

Solving polynomial least squares problems via semidefinite programming relaxations Sunyoung Kim and Masakazu Kojima August 2007, revised in November, 2007 Abstract. A polynomial optimization problem whose

### Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

### Geostatistics Exploratory Analysis

Instituto Superior de Estatística e Gestão de Informação Universidade Nova de Lisboa Master of Science in Geospatial Technologies Geostatistics Exploratory Analysis Carlos Alberto Felgueiras cfelgueiras@isegi.unl.pt

### Direct Methods for Solving Linear Systems. Matrix Factorization

Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011

### Weighted Least Squares

Weighted Least Squares The standard linear model assumes that Var(ε i ) = σ 2 for i = 1,..., n. As we have seen, however, there are instances where Var(Y X = x i ) = Var(ε i ) = σ2 w i. Here w 1,..., w

### EC327: Advanced Econometrics, Spring 2007

EC327: Advanced Econometrics, Spring 2007 Wooldridge, Introductory Econometrics (3rd ed, 2006) Appendix D: Summary of matrix algebra Basic definitions A matrix is a rectangular array of numbers, with m

### Linear Least Squares

Linear Least Squares Suppose we are given a set of data points {(x i,f i )}, i = 1,...,n. These could be measurements from an experiment or obtained simply by evaluating a function at some points. One

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### 1 Introduction. 2 Matrices: Definition. Matrix Algebra. Hervé Abdi Lynne J. Williams

In Neil Salkind (Ed.), Encyclopedia of Research Design. Thousand Oaks, CA: Sage. 00 Matrix Algebra Hervé Abdi Lynne J. Williams Introduction Sylvester developed the modern concept of matrices in the 9th

### Least Squares Estimation

Least Squares Estimation SARA A VAN DE GEER Volume 2, pp 1041 1045 in Encyclopedia of Statistics in Behavioral Science ISBN-13: 978-0-470-86080-9 ISBN-10: 0-470-86080-4 Editors Brian S Everitt & David

### Smoothing and Non-Parametric Regression

Smoothing and Non-Parametric Regression Germán Rodríguez grodri@princeton.edu Spring, 2001 Objective: to estimate the effects of covariates X on a response y nonparametrically, letting the data suggest

### Tutorial 3: Graphics and Exploratory Data Analysis in R Jason Pienaar and Tom Miller

Tutorial 3: Graphics and Exploratory Data Analysis in R Jason Pienaar and Tom Miller Getting to know the data An important first step before performing any kind of statistical analysis is to familiarize

### Exploratory Data Analysis with MATLAB

Computer Science and Data Analysis Series Exploratory Data Analysis with MATLAB Second Edition Wendy L Martinez Angel R. Martinez Jeffrey L. Solka ( r ec) CRC Press VV J Taylor & Francis Group Boca Raton

### Principle Component Analysis and Partial Least Squares: Two Dimension Reduction Techniques for Regression

Principle Component Analysis and Partial Least Squares: Two Dimension Reduction Techniques for Regression Saikat Maitra and Jun Yan Abstract: Dimension reduction is one of the major tasks for multivariate

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### Lecture 5: Conic Optimization: Overview

EE 227A: Conve Optimization and Applications January 31, 2012 Lecture 5: Conic Optimization: Overview Lecturer: Laurent El Ghaoui Reading assignment: Chapter 4 of BV. Sections 3.1-3.6 of WTB. 5.1 Linear

### Syntax. Description. MATLAB Function Reference Arithmetic Operators + - * / \ ^ ' Matrix and array arithmetic

MATLAB Function Reference Arithmetic Operators + - * / \ ^ ' Matrix and array arithmetic Syntax A+B A-B A*B A.*B A/B A./B A\B A.\B A^B A.^B A' A.' Description MATLAB has two different types of arithmetic

### Regularized Logistic Regression for Mind Reading with Parallel Validation

Regularized Logistic Regression for Mind Reading with Parallel Validation Heikki Huttunen, Jukka-Pekka Kauppi, Jussi Tohka Tampere University of Technology Department of Signal Processing Tampere, Finland

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### The basic unit in matrix algebra is a matrix, generally expressed as: a 11 a 12. a 13 A = a 21 a 22 a 23

(copyright by Scott M Lynch, February 2003) Brief Matrix Algebra Review (Soc 504) Matrix algebra is a form of mathematics that allows compact notation for, and mathematical manipulation of, high-dimensional

### Math 407A: Linear Optimization

Math 407A: Linear Optimization Lecture 4: LP Standard Form 1 1 Author: James Burke, University of Washington LPs in Standard Form Minimization maximization Linear equations to linear inequalities Lower

### Multivariate Normal Distribution

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues

### Several Views of Support Vector Machines

Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

### MATLAB and Big Data: Illustrative Example

MATLAB and Big Data: Illustrative Example Rick Mansfield Cornell University August 19, 2014 Goals Use a concrete example from my research to: Demonstrate the value of vectorization Introduce key commands/functions

### Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### Sketching as a Tool for Numerical Linear Algebra

Sketching as a Tool for Numerical Linear Algebra (Graph Sparsification) David P. Woodruff presented by Sepehr Assadi o(n) Big Data Reading Group University of Pennsylvania April, 2015 Sepehr Assadi (Penn)

### Transportation Polytopes: a Twenty year Update

Transportation Polytopes: a Twenty year Update Jesús Antonio De Loera University of California, Davis Based on various papers joint with R. Hemmecke, E.Kim, F. Liu, U. Rothblum, F. Santos, S. Onn, R. Yoshida,

### LINEAR ALGEBRA. September 23, 2010

LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................

### Linear Models and Conjoint Analysis with Nonlinear Spline Transformations

Linear Models and Conjoint Analysis with Nonlinear Spline Transformations Warren F. Kuhfeld Mark Garratt Abstract Many common data analysis models are based on the general linear univariate model, including

### Cryptography and Network Security. Prof. D. Mukhopadhyay. Department of Computer Science and Engineering. Indian Institute of Technology, Kharagpur

Cryptography and Network Security Prof. D. Mukhopadhyay Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module No. # 01 Lecture No. # 12 Block Cipher Standards

### 1.5 Elementary Matrices and a Method for Finding the Inverse

.5 Elementary Matrices and a Method for Finding the Inverse Definition A n n matrix is called an elementary matrix if it can be obtained from I n by performing a single elementary row operation Reminder:

### Numerical Methods I Eigenvalue Problems

Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001, Fall 2010 September 30th, 2010 A. Donev (Courant Institute)

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### Lecture 5 Principal Minors and the Hessian

Lecture 5 Principal Minors and the Hessian Eivind Eriksen BI Norwegian School of Management Department of Economics October 01, 2010 Eivind Eriksen (BI Dept of Economics) Lecture 5 Principal Minors and

### HSL and its out-of-core solver

HSL and its out-of-core solver Jennifer A. Scott j.a.scott@rl.ac.uk Prague November 2006 p. 1/37 Sparse systems Problem: we wish to solve where A is Ax = b LARGE Informal definition: A is sparse if many

### Non-negative Matrix Factorization (NMF) in Semi-supervised Learning Reducing Dimension and Maintaining Meaning

Non-negative Matrix Factorization (NMF) in Semi-supervised Learning Reducing Dimension and Maintaining Meaning SAMSI 10 May 2013 Outline Introduction to NMF Applications Motivations NMF as a middle step

### Alabama Department of Postsecondary Education

Date Adopted 1998 Dates reviewed 2007, 2011, 2013 Dates revised 2004, 2008, 2011, 2013, 2015 Alabama Department of Postsecondary Education Representing Alabama s Public Two-Year College System Jefferson

### Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

### BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES

BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES 123 CHAPTER 7 BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES 7.1 Introduction Even though using SVM presents

### Component Ordering in Independent Component Analysis Based on Data Power

Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals

### Numerical Linear Algebra Software

Numerical Linear Algebra Software (based on slides written by Michael Grant) BLAS, ATLAS LAPACK sparse matrices Prof. S. Boyd, EE364b, Stanford University Numerical linear algebra in optimization most

### Doolittle Decomposition of a Matrix

Doolittle Decomposition of a Matrix It is always possible to factor a square matrix into a lower triangular matrix and an upper triangular matrix. That is, [A] = [L][U] EXAMPLE: = 2 2 2 4 Having the LU

### 3. Interpolation. Closing the Gaps of Discretization... Beyond Polynomials

3. Interpolation Closing the Gaps of Discretization... Beyond Polynomials Closing the Gaps of Discretization... Beyond Polynomials, December 19, 2012 1 3.3. Polynomial Splines Idea of Polynomial Splines

### Data Mining: Exploring Data. Lecture Notes for Chapter 3. Slides by Tan, Steinbach, Kumar adapted by Michael Hahsler

Data Mining: Exploring Data Lecture Notes for Chapter 3 Slides by Tan, Steinbach, Kumar adapted by Michael Hahsler Topics Exploratory Data Analysis Summary Statistics Visualization What is data exploration?

### Cheng Soon Ong & Christfried Webers. Canberra February June 2016

c Cheng Soon Ong & Christfried Webers Research Group and College of Engineering and Computer Science Canberra February June (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 31 c Part I

### Solving Linear Systems of Equations. Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.

Solving Linear Systems of Equations Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab:

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### Exploratory Data Analysis

Goals of EDA Relationship between mean response and covariates (including time). Variance, correlation structure, individual-level heterogeneity. Guidelines for graphical displays of longitudinal data

### How to assess the risk of a large portfolio? How to estimate a large covariance matrix?

Chapter 3 Sparse Portfolio Allocation This chapter touches some practical aspects of portfolio allocation and risk assessment from a large pool of financial assets (e.g. stocks) How to assess the risk

### Quadratic forms Cochran s theorem, degrees of freedom, and all that

Quadratic forms Cochran s theorem, degrees of freedom, and all that Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 1, Slide 1 Why We Care Cochran s theorem tells us

### Yousef Saad University of Minnesota Computer Science and Engineering. CRM Montreal - April 30, 2008

A tutorial on: Iterative methods for Sparse Matrix Problems Yousef Saad University of Minnesota Computer Science and Engineering CRM Montreal - April 30, 2008 Outline Part 1 Sparse matrices and sparsity

### MAT Solving Linear Systems Using Matrices and Row Operations

MAT 171 8.5 Solving Linear Systems Using Matrices and Row Operations A. Introduction to Matrices Identifying the Size and Entries of a Matrix B. The Augmented Matrix of a System of Equations Forming Augmented