# Matrix Multiplication

 To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video
Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Matrix Multiplication CPS343 Parallel and High Performance Computing Spring 2016 CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

2 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

3 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

4 Definition of a matrix A matrix is a rectangular two-dimensional array of numbers. We say a matrix is m n if it has m rows and n columns. These values are sometimes called the dimensions of the matrix. Note that, in contrast to Cartesian coordinates, we specify the number of rows (the vertical dimension) and then the number of columns (the horizontal dimension). In most contexts, the rows and columns are numbered starting with 1. Several programming APIs, however, index rows and columns from 0. We use a ij to refer to the entry in i th row and j th column of the matrix A. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

5 Matrices are extremely important in HPC While it s certainly not the case that high performance computing involves only computing with matrices, matrix operations are key to many important HPC applications. Many important applications can be reduced to operations on matrices, including (but not limited to) 1 searching and sorting 2 numerical simulation of physical processes 3 optimization The list of the top 500 supercomputers in the world (found at is determined by a benchmark program that performs matrix operations. Like most benchmark programs, this is just one measure, however, and does not predict the relative performance of a supercomputer on non-matrix problems, or even different matrix-based problems. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

6 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

7 Dense matrices The m n matrix A is dense if all or most of its entries are nonzero. Storing a dense matrix (sometimes called a full matrix) requires storing all mn elements of the matrix. Usually an array data structure is used to store a dense matrix. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

8 Dense matrix example F 15 E A 3 D B 6 C Find a matrix to represent this complete graph if the ij entry contains the weight of the edge connecting node corresponding to row i with the node corresponding to column j. Use the value 0 if a connection is missing CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

9 Dense matrix example Note: This is considered a dense matrix even though it contains zeros. This matrix is symmetric, meaning that a ij = a ji. What would be a good way to store this matrix? CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

10 Sparse matrices A matrix is sparse if most of its entries are zero. Here most is not usually just a simple majority, rather we expect the number of zeros to far exceed the number of nonzeros. It is often most efficient to store only the nonzero entries of a sparse matrix, but this requires that location information also be stored. Arrays and lists are most commonly used to store sparse matrices. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

11 Sparse matrix example F 15 E 5 14 A 3 D 9 B 6 C Find a matrix to represent this graph if the ij entry contains the weight of the edge connecting node corresponding to row i with the node corresponding to column j. As before, use the value 0 if a connection is missing CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

12 Sparse matrix example Sometimes its helpful to leave out the zeros to better see the structure of the matrix = This matrix is also symmetric. How could it be stored efficiently? CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

13 Banded matrices An important type of sparse matrices are banded matrices. Nonzeros are along diagonals close to main diagonal. Example: The bandwidth of this matrix is 5. = CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

14 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

15 Using a two-dimensional arrays It is natural to use a 2D array to store a dense or banded matrix. Unfortunately, there are a couple of significant issues that complicate this seemingly simple approach. 1 Row-major vs. column-major storage pattern is language depent. 2 It is not possible to dynamically allocate two-dimensional arrays in C and C++; at least not without pointer storage and manipulation overhead. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

16 Row-major storage Both C and C++ use what is often called a row-major storage pattern for 2D matrices. In C and C++, the last index in a multidimensional array indexes contiguous memory locations. Thus a[i][j] and a[i][j+1] are adjacent in memory. Example: The stride between adjacent elements in the same row is 1. The stride between adjacent elements in the same column is the row length (5 in this example). CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

17 Column-major storage In Fortran 2D matrices are stored in column-major form. The first index in a multidimensional array indexes contiguous memory locations. Thus a(i,j) and a(i+1,j) are adjacent in memory. Example: The stride between adjacent elements in the same row is the column length (2 in this example) while the stride between adjacent elements in the same column is 1. Notice that if C is used to read a matrix stored in Fortran (or vice-versa), the transpose matrix will be read. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

18 Significance of array ordering There are two main reasons why HPC programmers need to be aware of this issue: CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

19 Significance of array ordering There are two main reasons why HPC programmers need to be aware of this issue: 1 Memory access patterns can have a dramatic impact on performance, especially on modern systems with a complicated memory hierarchy. These code segments access the same elements of an array, but the order of accesses is different. Access by rows Access by columns for ( i = 0; i < 2; i ++) for ( j = 0; j < 5; j ++) a[i][j] =... for ( j = 0; j < 5; j ++) for ( i = 0; i < 2; i ++) a[i][j] =... CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

20 Significance of array ordering There are two main reasons why HPC programmers need to be aware of this issue: 1 Memory access patterns can have a dramatic impact on performance, especially on modern systems with a complicated memory hierarchy. These code segments access the same elements of an array, but the order of accesses is different. Access by rows Access by columns for ( i = 0; i < 2; i ++) for ( j = 0; j < 5; j ++) a[i][j] =... for ( j = 0; j < 5; j ++) for ( i = 0; i < 2; i ++) a[i][j] =... 2 Many important numerical libraries (e.g. LAPACK) are written in Fortran. To use them with C or C++ the programmer must often work with a transposed matrix. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

21 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

22 Row-sweep matrix-vector multiplication Row-major matrix-vector product y = A x, A is M N: for ( i = 0; i < M; i ++) { y[ i] = 0.0; for ( j = 0; j < N; j ++) { y[i] += a[i][j] * x[j]; } } matrix elements accessed in row-major order repeated consecutive updates to y[i] we can usually assume the compiler will optimize this also called inner product form since the i th entry of y is the result of an inner product between the i th column of A and the vector x. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

23 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

24 Column-sweep matrix-vector multiplication Column-major matrix-vector product y = A x, A is M N: for ( i = 0; i < M; i ++) { y[ i] = 0.0; } for ( j = 0; j < N; j ++) { for ( i = 0; i < M; i ++) { y[i] += a[i][j] * x[j]; } } matrix elements accessed in column-major order repeated updates to y[i], but each element in array is updated before any element is updated again. also called outer product form. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

25 Matrix-vector algorithm comparison Which of these two algorithms will run faster? Why? Row-Sweep Form for ( i = 0; i < M; i ++) { y[ i] = 0.0; for ( j = 0; j < N; j ++) { y[i] += a[i][j] * x[j]; } } Column-Sweep Form for ( i = 0; i < M; i ++) { y[ i] = 0.0; } for ( j = 0; j < N; j ++) { for ( i = 0; i < M; i ++) { y[i] += a[i][j] * x[j]; } } CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

26 Matrix-vector algorithm comparison Answer: it deps... Both algorithms carry out the same operations but do so in a different order. In particular, the memory access patterns are quite different. The row-sweep form will typically work better using a language like C or C++ which access 2D arrays in row-major form. Since Fortran accesses 2D arrays column-by-column, it is usually best to use the column-sweep form. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

27 Operation counts In order to compute the computation rate in MFLOPS or GFLOPS we need to know the number of floating point operations carried out and the elapsed time. Divisions are usually the most expensive of the four basic operations, followed by multiplication. Addition and subtraction are equivalent in terms of time. We typically count all four operations on floating point numbers when calculating FLOPS but ignore integer operations like array subscript calculations. In the case of a matrix-vector product, the innermost loop body contains a multiplication and an addition: y i = y i + a ij x j The inner and outer loop are done M and N times respectively, so there are a total of 2MN FLOPs. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

28 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

29 The textbook algorithm Consider the problem of multiplying two matrices: C = AB = The standard textbook algorithm to form the product C of the M P matrix A and the P N matrix B is based on the inner product. The c ij entry in the product is the inner product (or dot product) of the i th row of A and the j th column of B. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

30 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) known as the ijk-form of the product due to the loop ordering CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

31 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) known as the ijk-form of the product due to the loop ordering What is the operation count...? CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

32 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) known as the ijk-form of the product due to the loop ordering What is the operation count...? Number of FLOPs is 2MNP. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

33 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) known as the ijk-form of the product due to the loop ordering What is the operation count...? Number of FLOPs is 2MNP. For square matrices M = N = P = n so number of FLOPs is 2n 3. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

34 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) Notice that A is accessed row-by-row but B is accessed column-by-column. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

35 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) Notice that A is accessed row-by-row but B is accessed column-by-column. The column index for C varies faster than the row index, but these are constant with respect to the inner loop so is much less significant. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

36 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) Notice that A is accessed row-by-row but B is accessed column-by-column. The column index for C varies faster than the row index, but these are constant with respect to the inner loop so is much less significant. Regardless of the language we use (C or Fortran), we have an efficient access pattern for one matrix but not for the other. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

37 The textbook algorithm ijk-form Matrix-matrix product pseudocode: for i = 1 to M for j = 1 to N c(i,j) = 0 for k = 1 to P c(i,j) = c(i,j) + a(i,k) * b(k,j) Notice that A is accessed row-by-row but B is accessed column-by-column. The column index for C varies faster than the row index, but these are constant with respect to the inner loop so is much less significant. Regardless of the language we use (C or Fortran), we have an efficient access pattern for one matrix but not for the other. Could we improve things by rearranging the order of operations? CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

38 Outline 1 Matrix operations Importance Dense and sparse matrices Matrices and arrays 2 Matrix-vector multiplication Row-sweep algorithm Column-sweep algorithm 3 Matrix-matrix multiplication Standard algorithm ijk-forms CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

39 The ijk forms From linear algebra we know that column i of the matrix-matrix product AB is defined as a linear combination of the columns of A using the values in the i th column of B as weights. The pseudocode for this is: for j = 1 to N for i = 1 to M c(i,j) = 0.0 for k = 1 to P for i = 1 to M c(i,j) = c(i,j) + a(i,k) * b(k,j) the loop ordering changes but the innermost statement is unchanged the initialization of values in C is done one column at a time the operation count is still 2MNP CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

40 The ijk forms Other loop orderings are possible... How many? How many ways can i, j, and k be arranged? Recall from discrete math that this is a permutation problem. three ways to choose the first letter, two ways to choose the second, and one way to choose the third : = 6 There are six possible loop orderings. We ll work with all six on Friday. CPS343 (Parallel and HPC) Matrix Multiplication Spring / 32

### 9. Numerical linear algebra background

Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization

### Notes on Matrix Multiplication and the Transitive Closure

ICS 6D Due: Wednesday, February 25, 2015 Instructor: Sandy Irani Notes on Matrix Multiplication and the Transitive Closure An n m matrix over a set S is an array of elements from S with n rows and m columns.

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### Matrices, transposes, and inverses

Matrices, transposes, and inverses Math 40, Introduction to Linear Algebra Wednesday, February, 202 Matrix-vector multiplication: two views st perspective: A x is linear combination of columns of A 2 4

### MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix.

MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix. Inverse matrix Definition. Let A be an n n matrix. The inverse of A is an n n matrix, denoted

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### Bindel, Fall 2012 Matrix Computations (CS 6210) Week 1: Wednesday, Aug 22

Logistics Week 1: Wednesday, Aug 22 We will go over the syllabus in the first class, but in general you should look at the class web page at http://www.cs.cornell.edu/~bindel/class/cs6210-f12/ The web

### Lecture 2 Mathcad basics and Matrix Operations

Lecture 2 Mathcad basics and Matrix Operations Announcements No class or lab Wednesday, 8/29/01 I will be posting a lab worksheet on the web site on Tuesday for you to work through on your own. Operators

### 13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.

3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in three-space, we write a vector in terms

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### 6. Cholesky factorization

6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

### MATH 304 Linear Algebra Lecture 4: Matrix multiplication. Diagonal matrices. Inverse matrix.

MATH 304 Linear Algebra Lecture 4: Matrix multiplication. Diagonal matrices. Inverse matrix. Matrices Definition. An m-by-n matrix is a rectangular array of numbers that has m rows and n columns: a 11

### Basics Inversion and related concepts Random vectors Matrix calculus. Matrix algebra. Patrick Breheny. January 20

Matrix algebra January 20 Introduction Basics The mathematics of multiple regression revolves around ordering and keeping track of large arrays of numbers and solving systems of equations The mathematical

### A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form

Section 1.3 Matrix Products A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form (scalar #1)(quantity #1) + (scalar #2)(quantity #2) +...

### Math 018 Review Sheet v.3

Math 018 Review Sheet v.3 Tyrone Crisp Spring 007 1.1 - Slopes and Equations of Lines Slopes: Find slopes of lines using the slope formula m y y 1 x x 1. Positive slope the line slopes up to the right.

### Relations Graphical View

Relations Slides by Christopher M. Bourke Instructor: Berthe Y. Choueiry Introduction Recall that a relation between elements of two sets is a subset of their Cartesian product (of ordered pairs). A binary

### = [a ij ] 2 3. Square matrix A square matrix is one that has equal number of rows and columns, that is n = m. Some examples of square matrices are

This document deals with the fundamentals of matrix algebra and is adapted from B.C. Kuo, Linear Networks and Systems, McGraw Hill, 1967. It is presented here for educational purposes. 1 Introduction In

### Data Structure. Lecture 3

Data Structure Lecture 3 Data Structure Formally define Data structure as: DS describes not only set of objects but the ways they are related, the set of operations which may be applied to the elements

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### Further Maths Matrix Summary

Further Maths Matrix Summary A matrix is a rectangular array of numbers arranged in rows and columns. The numbers in a matrix are called the elements of the matrix. The order of a matrix is the number

### Introduction to Programming (in C++) Multi-dimensional vectors. Jordi Cortadella, Ricard Gavaldà, Fernando Orejas Dept. of Computer Science, UPC

Introduction to Programming (in C++) Multi-dimensional vectors Jordi Cortadella, Ricard Gavaldà, Fernando Orejas Dept. of Computer Science, UPC Matrices A matrix can be considered a two-dimensional vector,

### Chapter 4 - Systems of Equations and Inequalities

Math 233 - Spring 2009 Chapter 4 - Systems of Equations and Inequalities 4.1 Solving Systems of equations in Two Variables Definition 1. A system of linear equations is two or more linear equations to

### Optimizing matrix multiplication Amitabha Banerjee abanerjee@ucdavis.edu

Optimizing matrix multiplication Amitabha Banerjee abanerjee@ucdavis.edu Present compilers are incapable of fully harnessing the processor architecture complexity. There is a wide gap between the available

### Basic Terminology for Systems of Equations in a Nutshell. E. L. Lady. 3x 1 7x 2 +4x 3 =0 5x 1 +8x 2 12x 3 =0.

Basic Terminology for Systems of Equations in a Nutshell E L Lady A system of linear equations is something like the following: x 7x +4x =0 5x +8x x = Note that the number of equations is not required

### A DAG-based sparse Cholesky solver. architectures. Jonathan Hogg. Sparse Days at CERFACS June John Reid

for multicore architectures Jonathan Hogg j.hogg@ed.ac.uk John Reid john.reid@stfc.ac.uk Jennifer Scott jennifer.scott@stfc.ac.uk Sparse Days at CERFACS June 2009 Outline of talk How to efficiently solve

### The basic unit in matrix algebra is a matrix, generally expressed as: a 11 a 12. a 13 A = a 21 a 22 a 23

(copyright by Scott M Lynch, February 2003) Brief Matrix Algebra Review (Soc 504) Matrix algebra is a form of mathematics that allows compact notation for, and mathematical manipulation of, high-dimensional

### Vector algebra Christian Miller CS Fall 2011

Vector algebra Christian Miller CS 354 - Fall 2011 Vector algebra A system commonly used to describe space Vectors, linear operators, tensors, etc. Used to build classical physics and the vast majority

### Lecture 2 Matrix Operations

Lecture 2 Matrix Operations transpose, sum & difference, scalar multiplication matrix multiplication, matrix-vector product matrix inverse 2 1 Matrix transpose transpose of m n matrix A, denoted A T or

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.

Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product

### ( % . This matrix consists of \$ 4 5 " 5' the coefficients of the variables as they appear in the original system. The augmented 3 " 2 2 # 2 " 3 4&

Matrices define matrix We will use matrices to help us solve systems of equations. A matrix is a rectangular array of numbers enclosed in parentheses or brackets. In linear algebra, matrices are important

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Linear Algebra and TI 89

Linear Algebra and TI 89 Abdul Hassen and Jay Schiffman This short manual is a quick guide to the use of TI89 for Linear Algebra. We do this in two sections. In the first section, we will go over the editing

### row row row 4

13 Matrices The following notes came from Foundation mathematics (MATH 123) Although matrices are not part of what would normally be considered foundation mathematics, they are one of the first topics

### Image Manipulation in MATLAB Due 11/1 at 5:00 PM

Image Manipulation in MATLAB Due 11/1 at 5:00 PM 1 Introduction Digital images are just matrices of pixels, and any type of matrix operation can be applied to a matrix containing image data. In this project

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### Operation Count; Numerical Linear Algebra

10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point

### Solving linear systems. Solving linear systems p. 1

Solving linear systems Solving linear systems p. 1 Overview Chapter 12 from Michael J. Quinn, Parallel Programming in C with MPI and OpenMP We want to find vector x = (x 0,x 1,...,x n 1 ) as solution of

### Mathematics Notes for Class 12 chapter 3. Matrices

1 P a g e Mathematics Notes for Class 12 chapter 3. Matrices A matrix is a rectangular arrangement of numbers (real or complex) which may be represented as matrix is enclosed by [ ] or ( ) or Compact form

### Tensor Decompositions for Analyzing Multi-link Graphs

Tensor Decompositions for Analyzing Multi-link Graphs Danny Dunlavy, Tammy Kolda, Philip Kegelmeyer Sandia National Laboratories SIAM Parallel Processing for Scientific Computing March 13, 2008 Sandia

### Parallel Algorithm for Dense Matrix Multiplication

Parallel Algorithm for Dense Matrix Multiplication CSE633 Parallel Algorithms Fall 2012 Ortega, Patricia Outline Problem definition Assumptions Implementation Test Results Future work Conclusions Problem

### We know a formula for and some properties of the determinant. Now we see how the determinant can be used.

Cramer s rule, inverse matrix, and volume We know a formula for and some properties of the determinant. Now we see how the determinant can be used. Formula for A We know: a b d b =. c d ad bc c a Can we

### Chapter 19. General Matrices. An n m matrix is an array. a 11 a 12 a 1m a 21 a 22 a 2m A = a n1 a n2 a nm. The matrix A has n row vectors

Chapter 9. General Matrices An n m matrix is an array a a a m a a a m... = [a ij]. a n a n a nm The matrix A has n row vectors and m column vectors row i (A) = [a i, a i,..., a im ] R m a j a j a nj col

### Matrix Algebra and Applications

Matrix Algebra and Applications Dudley Cooke Trinity College Dublin Dudley Cooke (Trinity College Dublin) Matrix Algebra and Applications 1 / 49 EC2040 Topic 2 - Matrices and Matrix Algebra Reading 1 Chapters

### SECTIONS 1.5-1.6 NOTES ON GRAPH THEORY NOTATION AND ITS USE IN THE STUDY OF SPARSE SYMMETRIC MATRICES

SECIONS.5-.6 NOES ON GRPH HEORY NOION ND IS USE IN HE SUDY OF SPRSE SYMMERIC MRICES graph G ( X, E) consists of a finite set of nodes or vertices X and edges E. EXMPLE : road map of part of British Columbia

### MATH36001 Background Material 2015

MATH3600 Background Material 205 Matrix Algebra Matrices and Vectors An ordered array of mn elements a ij (i =,, m; j =,, n) written in the form a a 2 a n A = a 2 a 22 a 2n a m a m2 a mn is said to be

### Practical Numerical Training UKNum

Practical Numerical Training UKNum 7: Systems of linear equations C. Mordasini Max Planck Institute for Astronomy, Heidelberg Program: 1) Introduction 2) Gauss Elimination 3) Gauss with Pivoting 4) Determinants

### October 3rd, 2012. Linear Algebra & Properties of the Covariance Matrix

Linear Algebra & Properties of the Covariance Matrix October 3rd, 2012 Estimation of r and C Let rn 1, rn, t..., rn T be the historical return rates on the n th asset. rn 1 rṇ 2 r n =. r T n n = 1, 2,...,

### Definition A square matrix M is invertible (or nonsingular) if there exists a matrix M 1 such that

0. Inverse Matrix Definition A square matrix M is invertible (or nonsingular) if there exists a matrix M such that M M = I = M M. Inverse of a 2 2 Matrix Let M and N be the matrices: a b d b M =, N = c

### Solving Systems of Linear Equations; Row Reduction

Harvey Mudd College Math Tutorial: Solving Systems of Linear Equations; Row Reduction Systems of linear equations arise in all sorts of applications in many different fields of study The method reviewed

### A Introduction to Matrix Algebra and Principal Components Analysis

A Introduction to Matrix Algebra and Principal Components Analysis Multivariate Methods in Education ERSH 8350 Lecture #2 August 24, 2011 ERSH 8350: Lecture 2 Today s Class An introduction to matrix algebra

### Chapter 4: Systems of Equations and Ineq. Lecture notes Math 1010

Section 4.1: Systems of Equations Systems of equations A system of equations consists of two or more equations involving two or more variables { ax + by = c dx + ey = f A solution of such a system is an

### Introduction to Matrices

Introduction to Matrices Tom Davis tomrdavis@earthlinknet 1 Definitions A matrix (plural: matrices) is simply a rectangular array of things For now, we ll assume the things are numbers, but as you go on

### Helpsheet. Giblin Eunson Library MATRIX ALGEBRA. library.unimelb.edu.au/libraries/bee. Use this sheet to help you:

Helpsheet Giblin Eunson Library ATRIX ALGEBRA Use this sheet to help you: Understand the basic concepts and definitions of matrix algebra Express a set of linear equations in matrix notation Evaluate determinants

### 5. Orthogonal matrices

L Vandenberghe EE133A (Spring 2016) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal

### Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

### Introduction to Matrix Algebra I

Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model

### 1.5 Elementary Matrices and a Method for Finding the Inverse

.5 Elementary Matrices and a Method for Finding the Inverse Definition A n n matrix is called an elementary matrix if it can be obtained from I n by performing a single elementary row operation Reminder:

### Parallel and Distributed Computing Programming Assignment 1

Parallel and Distributed Computing Programming Assignment 1 Due Monday, February 7 For programming assignment 1, you should write two C programs. One should provide an estimate of the performance of ping-pong

### The Solution of Linear Simultaneous Equations

Appendix A The Solution of Linear Simultaneous Equations Circuit analysis frequently involves the solution of linear simultaneous equations. Our purpose here is to review the use of determinants to solve

### Lecture Notes 2: Matrices as Systems of Linear Equations

2: Matrices as Systems of Linear Equations 33A Linear Algebra, Puck Rombach Last updated: April 13, 2016 Systems of Linear Equations Systems of linear equations can represent many things You have probably

### 4 Solving Systems of Equations by Reducing Matrices

Math 15 Sec S0601/S060 4 Solving Systems of Equations by Reducing Matrices 4.1 Introduction One of the main applications of matrix methods is the solution of systems of linear equations. Consider for example

### CSE 6040 Computing for Data Analytics: Methods and Tools

CSE 6040 Computing for Data Analytics: Methods and Tools Lecture 12 Computer Architecture Overview and Why it Matters DA KUANG, POLO CHAU GEORGIA TECH FALL 2014 Fall 2014 CSE 6040 COMPUTING FOR DATA ANALYSIS

### CUDAMat: a CUDA-based matrix class for Python

Department of Computer Science 6 King s College Rd, Toronto University of Toronto M5S 3G4, Canada http://learning.cs.toronto.edu fax: +1 416 978 1455 November 25, 2009 UTML TR 2009 004 CUDAMat: a CUDA-based

### A Primer on Index Notation

A Primer on John Crimaldi August 28, 2006 1. Index versus Index notation (a.k.a. Cartesian notation) is a powerful tool for manipulating multidimensional equations. However, there are times when the more

### Arithmetic and Algebra of Matrices

Arithmetic and Algebra of Matrices Math 572: Algebra for Middle School Teachers The University of Montana 1 The Real Numbers 2 Classroom Connection: Systems of Linear Equations 3 Rational Numbers 4 Irrational

### Row and column operations

Row and column operations It is often very useful to apply row and column operations to a matrix. Let us list what operations we re going to be using. 3 We ll illustrate these using the example matrix

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### HSL and its out-of-core solver

HSL and its out-of-core solver Jennifer A. Scott j.a.scott@rl.ac.uk Prague November 2006 p. 1/37 Sparse systems Problem: we wish to solve where A is Ax = b LARGE Informal definition: A is sparse if many

### A Brief Primer on Matrix Algebra

A Brief Primer on Matrix Algebra A matrix is a rectangular array of numbers whose individual entries are called elements. Each horizontal array of elements is called a row, while each vertical array is

### Typical Linear Equation Set and Corresponding Matrices

EWE: Engineering With Excel Larsen Page 1 4. Matrix Operations in Excel. Matrix Manipulations: Vectors, Matrices, and Arrays. How Excel Handles Matrix Math. Basic Matrix Operations. Solving Systems of

### Direct Methods for Solving Linear Systems. Linear Systems of Equations

Direct Methods for Solving Linear Systems Linear Systems of Equations Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University

### Matrix-vector multiplication in terms of dot-products

Matrix-vector multiplication in terms of dot-products Let M be an R C matrix. Dot-Product Definition of matrix-vector multiplication: M u is the R-vector v such that v[r] is the dot-product of row r of

### Lecture 5 - Triangular Factorizations & Operation Counts

LU Factorization Lecture 5 - Triangular Factorizations & Operation Counts We have seen that the process of GE essentially factors a matrix A into LU Now we want to see how this factorization allows us

Hill Cipher Project K80TTQ1EP-??,VO.L,XU0H5BY,_71ZVPKOE678_X,N2Y-8HI4VS,,6Z28DDW5N7ADY013 Directions: Answer all numbered questions completely. Show non-trivial work in the space provided. Non-computational

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### Notes on Cholesky Factorization

Notes on Cholesky Factorization Robert A. van de Geijn Department of Computer Science Institute for Computational Engineering and Sciences The University of Texas at Austin Austin, TX 78712 rvdg@cs.utexas.edu

### Inverses and powers: Rules of Matrix Arithmetic

Contents 1 Inverses and powers: Rules of Matrix Arithmetic 1.1 What about division of matrices? 1.2 Properties of the Inverse of a Matrix 1.2.1 Theorem (Uniqueness of Inverse) 1.2.2 Inverse Test 1.2.3

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Economics 102C: Advanced Topics in Econometrics 2 - Tooling Up: The Basics

Economics 102C: Advanced Topics in Econometrics 2 - Tooling Up: The Basics Michael Best Spring 2015 Outline The Evaluation Problem Matrix Algebra Matrix Calculus OLS With Matrices The Evaluation Problem:

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### Announcements. CompSci 230 Discrete Math for Computer Science. Test 1

CompSci 230 Discrete Math for Computer Science Sep 26, 2013 Announcements Exam 1 is Tuesday, Oct. 1 No class, Oct 3, No recitation Oct 4-7 Prof. Rodger is out Sep 30-Oct 4 There is Recitation: Sept 27-30.

### Matrices, Determinants and Linear Systems

September 21, 2014 Matrices A matrix A m n is an array of numbers in rows and columns a 11 a 12 a 1n r 1 a 21 a 22 a 2n r 2....... a m1 a m2 a mn r m c 1 c 2 c n We say that the dimension of A is m n (we

### Systems of Linear Equations

A FIRST COURSE IN LINEAR ALGEBRA An Open Text by Ken Kuttler Systems of Linear Equations Lecture Notes by Karen Seyffarth Adapted by LYRYX SERVICE COURSE SOLUTION Attribution-NonCommercial-ShareAlike (CC

### 4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns

L. Vandenberghe EE133A (Spring 2016) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows

### Math 4707: Introduction to Combinatorics and Graph Theory

Math 4707: Introduction to Combinatorics and Graph Theory Lecture Addendum, November 3rd and 8th, 200 Counting Closed Walks and Spanning Trees in Graphs via Linear Algebra and Matrices Adjacency Matrices

### APPLICATIONS OF MATRICES. Adj A is nothing but the transpose of the co-factor matrix [A ij ] of A.

APPLICATIONS OF MATRICES ADJOINT: Let A = [a ij ] be a square matrix of order n. Let Aij be the co-factor of a ij. Then the n th order matrix [A ij ] T is called the adjoint of A. It is denoted by adj

### Lecture 3: Single processor architecture and memory

Lecture 3: Single processor architecture and memory David Bindel 30 Jan 2014 Logistics Raised enrollment from 75 to 94 last Friday. Current enrollment is 90; C4 and CMS should be current? HW 0 (getting

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### Linear Codes. In the V[n,q] setting, the terms word and vector are interchangeable.

Linear Codes Linear Codes In the V[n,q] setting, an important class of codes are the linear codes, these codes are the ones whose code words form a sub-vector space of V[n,q]. If the subspace of V[n,q]

### 5. A full binary tree with n leaves contains [A] n nodes. [B] log n 2 nodes. [C] 2n 1 nodes. [D] n 2 nodes.

1. The advantage of.. is that they solve the problem if sequential storage representation. But disadvantage in that is they are sequential lists. [A] Lists [B] Linked Lists [A] Trees [A] Queues 2. The

### Solution to Homework 2

Solution to Homework 2 Olena Bormashenko September 23, 2011 Section 1.4: 1(a)(b)(i)(k), 4, 5, 14; Section 1.5: 1(a)(b)(c)(d)(e)(n), 2(a)(c), 13, 16, 17, 18, 27 Section 1.4 1. Compute the following, if

### Software Pipelining by Modulo Scheduling. Philip Sweany University of North Texas

Software Pipelining by Modulo Scheduling Philip Sweany University of North Texas Overview Opportunities for Loop Optimization Software Pipelining Modulo Scheduling Resource and Dependence Constraints Scheduling

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of