# A Introduction to Matrix Algebra and Principal Components Analysis

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 A Introduction to Matrix Algebra and Principal Components Analysis Multivariate Methods in Education ERSH 8350 Lecture #2 August 24, 2011 ERSH 8350: Lecture 2

2 Today s Class An introduction to matrix algebra Scalars, vectors, and matrices Basic matrix operations Advanced matrix operations An introduction to principal components analysis ERSH 8350: Lecture 2 2

3 Introduction and Motivation Nearly all multivariate statistical techniques are described with matrix algebra It is the language of modern statistics When new methods are developed, the first published work typically involves matrices It makes technical writing more concise formulae are smaller Have you seen: Useful tip: matrix algebra is a great way to get out of conversations and other awkward moments ERSH 8350: Lecture 2 3

4 Definitions We begin this class with some general definitions (from dictionary.com): Matrix: 1. A rectangular array of numeric or algebraic quantities subject to mathematical operations 2. The substrate on or within which a fungus grows Algebra: 1. A branch of mathematics in which symbols, usually letters of the alphabet, represent numbers or members of a specified set and are used to represent quantities and to express general relationships that hold for all members of the set 2. A set together with a pair of binary operations defined on the set. Usually, the set and the operations include an identity element, and the operations are commutative or associative ERSH 8350: Lecture 2 4

5 Why Learn Matrix Algebra Matrix algebra can seem very abstract from the purposes of this class (and statistics in general) Learning matrix algebra is important for: Understanding how statistical methods work And when to use them (or not use them) Understanding what statistical methods mean Reading and writing results from new statistical methods Today s class is the first lecture of learning the language of multivariate statistics ERSH 8350: Lecture 2 5

6 DATA EXAMPLE AND SAS ERSH 8350: Lecture 2 6

7 A Guiding Example To demonstrate matrix algebra, we will make use of data Imagine that somehow I collected data SAT test scores for both the Math (SATM) and Verbal (SATV) sections of 1,000 students The descriptive statistics of this data set are given below: ERSH 8350: Lecture 2 7

8 The Data In Excel: In SAS: ERSH 8350: Lecture 2 8

9 Matrix Computing: PROC IML To help demonstrate the topics we will discuss today, I will beshowing examples in SAS proc iml The Interactive Matrix Language (IML) is a scientific computing package in SAS that typically used for complicated statistical routines Other matrix programs exist for many statistical applications MATLAB is very useful SPSS has matrix computing capabilities, but (in my opinion) it is not as efficient, user friendly, or flexible as SAS proc iml ERSH 8350: Lecture 2 9

10 PROC IML Basics Proc IML is a proc step in SAS that runs without needing to use a preliminary data step To use IML the following lines of syntax are placed in a SAS file: The reset print; line makes every result get printed in the output window The IML syntax will go between the reset print; and the quit; ERSH 8350: Lecture 2 10

11 THE BASICS: DEFINITIONS OF MATRICES, VECTORS, AND SCALARS ERSH 8350: Lecture 2 11

12 Matrices A matrix is a rectangular array of data Used for storing numbers Matrices can have unlimited dimensions For our purposes all matrices will have two dimensions: Row Columns Matrices are symbolized by boldface font in text, typically with capital letters Size (r rows x c columns) ERSH 8350: Lecture 2 12

13 Vectors A vector is a matrix where one dimension is equal to size 1 Column vector: a matrix of size Row vector: a matrix of size Vectors are typically written in boldface font text, usually with lowercase letters ERSH 8350: Lecture 2 13

14 Scalars A scalar is just a single number (as we have known before) The name scalar is important: the number scales a vector it can make a vector longer or shorter ERSH 8350: Lecture 2 14

15 Matrix Elements A matrix (or vector) is composed of a set of elements Each element is denoted by its position in the matrix (row and column) For our matrix of data (size 1000 rows and 2 columns), each element is denoted by: The first subscript is the index for the rows: i= 1,,r (= 1000) The second subscript is the index for the columns: j = 1,,c (= 2),, ERSH 8350: Lecture 2 15

16 Matrix Transpose The transpose of a matrix is a reorganization of the matrix by switching the indices for the rows and columns An element in the original matrix is now in the transposed matrix ERSH 8350: Lecture 2 16

17 Types of Matrices Square Matrix: A matrix that has the same number of rows and columns Correlation/covariance matrices are square matrices Diagonal Matrix: A diagonal matrix is a square matrix with non zero diagonal elements ( and zeros on the off diagonal elements ( ): We will use diagonal matrices to form correlation matrices Symmetric Matrix: A symmetric matrix is a square matrix where all elements are reflected across the diagonal ( Correlation and covariance matrices are symmetric matrices ERSH 8350: Lecture 2 17

18 VECTORS ERSH 8350: Lecture 2 18

19 Vectors in Space Vectors (row or column) can be represented as lines on a Cartesian coordinate system Consider the vectors: and A graph of these vectors would be: Question: how would a column vector for each of our example variables (SATM and SATV) be plotted? ERSH 8350: Lecture 2 19

20 Vector Length The length of a vector emanating from the origin is given by the Pythagorean formula This is also called the Euclidean distance between the endpoint of the vector and the origin From the last slide: ; From our data: In data: length is an analog to the standard deviation In mean centered variables, the length is the square root of the sum of mean deviations (not quite the SD, but close) ERSH 8350: Lecture 2 20

21 Vector Addition Vectors can be added together so that a new vector is formed Vector addition is done element wise, by adding each of the respective elements together: The new vector has the same number of rows and columns Geometrically, this creates a new vector along either of the previous two In Data: a new variable (say, SAT total) is the result of vector addition ERSH 8350: Lecture 2 21

22 Vector Multiplication by Scalar Vectors can be multiplied by scalars All elements are multiplied by the scalar Scalar multiplication changes the length of the vector: This is where the term scalar comes from: a scalar ends up rescaling (resizing) a vector ERSH 8350: Lecture 2 22

23 Linear Combinations Addition of a set of vectors (all multiplied by scalars) is called a linear combination: Here, is the linear combination For all k vectors, the set of all possible linear combinations is called their span In Data: linear combinations happen frequently: Linear models (i.e., Regression and ANOVA) Principal components analysis (later today) ERSH 8350: Lecture 2 23

24 Linear Dependencies A set of vectors are said to be linearly dependent if and are all not zero Example: let s make a new variable SAT Total: The new variable is linearly dependent with the others: ERSH 8350: Lecture 2 24

25 Normalizing Vectors Normalizing vectors is the process of rescaling a vector (multiplication by scalar) so that the length of the vector is equal to one: From our example vector: As such, the length of the vector is: ERSH 8350: Lecture 2 25

26 Normalizing in Statistics We will encounter normalized vectors later today in principal components analysis Generally speaking, entities that have no real space (such as principal components and latent factors) have corresponding vectors that are normalized Normalization allows for a solution to be found It s an arbitrary standard (but one that is well agreed upon) ERSH 8350: Lecture 2 26

27 Inner (Dot) Product of Vectors An important concept in vector geometry is that of the inner product of two vectors The inner product is also called the dot product The inner product is related to the angle between vectors and to the projection of one vector onto another In data: the angle between vectors is related to the correlation between variables and the projection is related to regression/anova/linear models From our example: ERSH 8350: Lecture 2 27

28 Angle Between Vectors As vectors are conceptualized geometrically, the angle between two vectors can be calculated From the example: From our data:, ERSH 8350: Lecture 2 28

29 In Data: Cosine Angle = Correlation If you have data that are: Placed into vectors Centered by the mean (subtract the mean from each observation) then the cosine of the angle between those vectors is the correlation between the variables: ERSH 8350: Lecture 2 29

30 Vector Projections A final vector property that shows up in statistical terms frequently is that of a projection The projection of a vector onto is the orthogonal projection of onto the straight line defined by The projection is the shadow of one vector onto the other: In data: linear models can be thought of as projections ERSH 8350: Lecture 2 30

31 MATRIX ALGEBRA ERSH 8350: Lecture 2 31

32 Moving from Vectors to Matrices A matrix can be thought of as a collection of vectors Matrix operations are vector operations on steroids Matrix algebra defines a set of operations and entities on matrices I will present a version meant to mirror your previous algebra experiences Definitions: Identity matrix Zero vector Ones vector Basic Operations: Addition Subtraction Multiplication Division ERSH 8350: Lecture 2 32

33 Matrix Addition and Subtraction Matrix addition and subtraction are much like vector addition/subtraction Rules: Matrices must be the same size (rows and columns) Method: The new matrix is constructed of element by element addition/subtraction of the previous matrices Order: The order of the matrices (pre and post ) does not matter ERSH 8350: Lecture 2 33

34 Matrix Addition/Subtraction ERSH 8350: Lecture 2 34

35 Matrix Multiplication Matrix multiplication is a bit more complicated The new matrix may be a different size from either of the two multiplying matrices Rules: Pre multiplying matrix must have number of columns equal to the number of rows of the post multiplying matrix Method: Order: The elements of the new matrix consist of the inner (dot) product of the row vectors of the pre multiplying matrix and the column vectors of the post multiplying matrix The order of the matrices (pre and post ) matters ERSH 8350: Lecture 2 35

36 Matrix Multiplication ERSH 8350: Lecture 2 36

37 Multiplication in Statistics Many statistical formulae with summation can be re expressed with matrices A common matrix multiplication form is: Diagonal elements: Off diagonal elements: For our SAT example: 251,797, ,928, ,928, ,862,700 ERSH 8350: Lecture 2 37

38 Identity Matrix The identity matrix is a matrix that, when pre or postmultiplied by another matrix results in the original matrix: The identity matrix is a square matrix that has: Diagonal elements = 1 Off diagonal elements = 0 ERSH 8350: Lecture 2 38

39 Zero Vector The zero vector is a column vector of zeros When pre or post multiplied the result is the zero vector: ERSH 8350: Lecture 2 39

40 Ones Vector A ones vector is a column vector of 1s: The ones vector is useful for calculating statistical terms, such as the mean vector and the covariance matrix Next class we will discuss what these matrices are, how we compute them, and what the mean ERSH 8350: Lecture 2 40

41 Matrix Division : The Inverse Matrix Division from algebra: First: Second: 1 Division in matrices serves a similar role For square and symmetric matrices, an inverse matrix is a matrix that when pre or post multiplied with another matrix produces the identity matrix: Calculation of the matrix inverse is complicated Even computers have a tough time Not all matrices can be inverted Non invertable matrices are called singular matrices In statistics, singular matrices are commonly caused by linear dependencies ERSH 8350: Lecture 2 41

42 The Inverse In data: the inverse shows up constantly in statistics Models which assume some type of (multivariate) normality need an inverse covariance matrix Using our SAT example Our data matrix was size (1000 x 2), which is not invertable However was size (2 x 2) square, and symmetric 251,797, ,928, ,928, ,862,700 The inverse is: ERSH 8350: Lecture 2 42

43 Matrix Algebra Operations For such that exists: ERSH 8350: Lecture 2 43

44 ADVANCED MATRIX OPERATIONS ERSH 8350: Lecture 2 44

45 Advanced Matrix Functions/Operations We end our matrix discussion with some advanced topics All related to multivariate statistical analysis To help us throughout, let s consider the correlation matrix of our SAT data: ERSH 8350: Lecture 2 45

46 Matrix Trace For a square matrix with p rows/columns, the trace is the sum of the diagonal elements: For our data, the trace of the correlation matrix is 2 For all correlation matrices, the trace is equal to the number of variables because all diagonal elements are 1 The trace will be considered the total variance in multivariate statistics Used as a target to recover when applying statistical models ERSH 8350: Lecture 2 46

47 Matrix Determinants A square matrix can be characterized by a scalar value called a determinant: det Calculation of the determinant is tedious Our determinant was The determinant is useful in statistics: Shows up in multivariate statistical distributions Is a measure of generalized variance of multiple variables If the determinant is positive, the matrix is called positive definite Is invertable If the determinant is not positive, the matrix is called non positive definite Not invertable ERSH 8350: Lecture 2 47

48 Matrix Orthogonality A square matrix is said to be orthogonal if: Orthogonal matrices are characterized by two properties: 1. The dot product of all row vector multiples is the zero vector Meaning vectors are orthogonal (or uncorrelated) 2. For each row vector, the sum of all elements is one Meaning vectors are normalized The matrix above is also called orthonormal Orthonormal matrices are used in principal components ERSH 8350: Lecture 2 48

49 Eigenvalues and Eigenvectors A square matrix can be decomposed into a set of eigenvalues and a set of eigenvectors Each eigenvalue has a corresponding eigenvector The number equal to the number of rows/columns of The eigenvectors are all orthogonal In many classical multivariate statistics, eigenvalues and eigenvectors are used very frequently We will see their use in principal components shortly ERSH 8350: Lecture 2 49

50 Eigenvalues and Eigenvectors Example In our SAT example, the two eigenvalues obtained were: The two eigenvectors obtained were: These terms will have much greater meaning in one moment (principal components analysis) ERSH 8350: Lecture 2 50

51 Spectral Decomposition Using the eigenvalues and eigenvectors, we can reconstruct the original matrix using a spectral decomposition: For our example, we can get back to our original matrix: ERSH 8350: Lecture 2 51

52 Additional Eigenvalue Properties The matrix trace can be found by the sum of the eigenvalues: In our example, the The matrix determinant can be found by the product of the eigenvalues In our example ERSH 8350: Lecture 2 52

53 AN INTRODUCTION TO PRINCIPAL COMPONENTS ANALYSIS ERSH 8350: Lecture 2 53

54 PCA Overview Principal Components Analysis (PCA) is a method for reexpressing the covariance (correlation) between a set of variables The re expression comes from creating a set of new variables (linear combinations) of the original variables PCA has two objectives: 1. Data reduction Moving from many original variables down to a few components 2. Interpretation Determining which original variables contribute most to the new components My objective: use PCA to demonstrate eigenvalues and eigenvectors ERSH 8350: Lecture 2 54

55 Goals of PCA The goal of PCA is to find a set of k principal components (composite variables) that: Is much smaller in number than the original set of p variables Accounts for nearly all of the total variance Total variance = trace of covariance/correlation matrix If these two goals can be accomplished, then the set of k principal components contains almost as much information as the original p variables Meaning the components can now replace the original variables in any subsequent analyses ERSH 8350: Lecture 2 55

56 PCA Features PCA often reveals relationships between variables that were not previously suspected New interpretations of data and variables often stem from PCA PCA usually serves as more of a means to an end rather than an end it itself Components (the new variables) are often used in other statistical techniques Multiple regression Cluster analysis Unfortunately, PCA is often intermixed with Exploratory Factor Analysis Don t. Please don t. Please help make it stop. ERSH 8350: Lecture 2 56

57 PCA Details Notation: let denote our new components and let be our original data matrix (with N observations and p variables) We will let i be our index for a subject The new components are linear combinations: The weights of the components come from the eigenvectors of the covariance or correlation matrix ERSH 8350: Lecture 2 57

58 Details About the Components The components are formed by the weights of the eigenvectors of the covariance or correlation matrix of the original data The variance of a component is given by the eigenvalue associated with the eigenvector for the component Using the eigenvalue and eigenvectors means: Each successive component has lower variance Var(Y 1 ) > Var(Y 2 ) > > Var(Y p ) All components are uncorrelated The sum of the variances of the principal components is equal to the total variance: ERSH 8350: Lecture 2 58

59 PCA on our Example We will now conduct a PCA on the correlation matrix of our sample data This example is given for demonstration purposes typically we will not do PCA on small numbers of variables ERSH 8350: Lecture 2 59

60 PCA in SAS The SAS procedure that does principal components is called proc princomp: The results (look familiar?): ERSH 8350: Lecture 2 60

61 Graphical Representation Plotting the components and the original data side by side reveals the nature of PCA: Shown from PCA of covariance matrix ERSH 8350: Lecture 2 61

62 PCA Summary This introduction to PCA was meant to help describe the nature of eigenvalues and eigenvectors Many classical multivariate statistics use these terms We will return to PCA in the 3 rd part of class when we discuss inferences about covariances in multivariate data Eigenvalues and eigenvectors will reappear in the next 2 3 weeks The basis for classical Multivariate ANOVA Modern versions have moved away from these terms But they still are useful ERSH 8350: Lecture 2 62

63 Wrapping Up Matrix algebra is the language of multivariate statistics Learning the basics will help you read work (both new old) Over the course of the semester, we will use matrix algebra frequently It provides for more concise formulae In practice, we will use matrix algebra very little But understanding how it works is the key to understanding how statistical methods work and are related Up next: statistical terms and statistical distributions (univariate and multivariate) The basis for modern estimation methods using maximum likelihood or Bayesian estimation ERSH 8350: Lecture 2 63

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:

### Vector algebra Christian Miller CS Fall 2011

Vector algebra Christian Miller CS 354 - Fall 2011 Vector algebra A system commonly used to describe space Vectors, linear operators, tensors, etc. Used to build classical physics and the vast majority

### The basic unit in matrix algebra is a matrix, generally expressed as: a 11 a 12. a 13 A = a 21 a 22 a 23

(copyright by Scott M Lynch, February 2003) Brief Matrix Algebra Review (Soc 504) Matrix algebra is a form of mathematics that allows compact notation for, and mathematical manipulation of, high-dimensional

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### 1 Introduction. 2 Matrices: Definition. Matrix Algebra. Hervé Abdi Lynne J. Williams

In Neil Salkind (Ed.), Encyclopedia of Research Design. Thousand Oaks, CA: Sage. 00 Matrix Algebra Hervé Abdi Lynne J. Williams Introduction Sylvester developed the modern concept of matrices in the 9th

### Orthogonal Diagonalization of Symmetric Matrices

MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding

### 13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.

3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in three-space, we write a vector in terms

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### Introduction to Matrix Algebra I

Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Dimensionality Reduction: Principal Components Analysis

Dimensionality Reduction: Principal Components Analysis In data mining one often encounters situations where there are a large number of variables in the database. In such situations it is very likely

### α = u v. In other words, Orthogonal Projection

Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

### Notes on Orthogonal and Symmetric Matrices MENU, Winter 2013

Notes on Orthogonal and Symmetric Matrices MENU, Winter 201 These notes summarize the main properties and uses of orthogonal and symmetric matrices. We covered quite a bit of material regarding these topics,

### Helpsheet. Giblin Eunson Library MATRIX ALGEBRA. library.unimelb.edu.au/libraries/bee. Use this sheet to help you:

Helpsheet Giblin Eunson Library ATRIX ALGEBRA Use this sheet to help you: Understand the basic concepts and definitions of matrix algebra Express a set of linear equations in matrix notation Evaluate determinants

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

### Mehtap Ergüven Abstract of Ph.D. Dissertation for the degree of PhD of Engineering in Informatics

INTERNATIONAL BLACK SEA UNIVERSITY COMPUTER TECHNOLOGIES AND ENGINEERING FACULTY ELABORATION OF AN ALGORITHM OF DETECTING TESTS DIMENSIONALITY Mehtap Ergüven Abstract of Ph.D. Dissertation for the degree

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### MATH 551 - APPLIED MATRIX THEORY

MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points

### Introduction to Multivariate Analysis

Introduction to Multivariate Analysis Lecture 1 August 24, 2005 Multivariate Analysis Lecture #1-8/24/2005 Slide 1 of 30 Today s Lecture Today s Lecture Syllabus and course overview Chapter 1 (a brief

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### 3 Orthogonal Vectors and Matrices

3 Orthogonal Vectors and Matrices The linear algebra portion of this course focuses on three matrix factorizations: QR factorization, singular valued decomposition (SVD), and LU factorization The first

### Linear Algebra: Vectors

A Linear Algebra: Vectors A Appendix A: LINEAR ALGEBRA: VECTORS TABLE OF CONTENTS Page A Motivation A 3 A2 Vectors A 3 A2 Notational Conventions A 4 A22 Visualization A 5 A23 Special Vectors A 5 A3 Vector

### CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES. From Exploratory Factor Analysis Ledyard R Tucker and Robert C.

CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES From Exploratory Factor Analysis Ledyard R Tucker and Robert C MacCallum 1997 180 CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES In

### Presentation 3: Eigenvalues and Eigenvectors of a Matrix

Colleen Kirksey, Beth Van Schoyck, Dennis Bowers MATH 280: Problem Solving November 18, 2011 Presentation 3: Eigenvalues and Eigenvectors of a Matrix Order of Presentation: 1. Definitions of Eigenvalues

### [1] Diagonal factorization

8.03 LA.6: Diagonalization and Orthogonal Matrices [ Diagonal factorization [2 Solving systems of first order differential equations [3 Symmetric and Orthonormal Matrices [ Diagonal factorization Recall:

### 2.1: MATRIX OPERATIONS

.: MATRIX OPERATIONS What are diagonal entries and the main diagonal of a matrix? What is a diagonal matrix? When are matrices equal? Scalar Multiplication 45 Matrix Addition Theorem (pg 0) Let A, B, and

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### 2D Geometric Transformations. COMP 770 Fall 2011

2D Geometric Transformations COMP 770 Fall 2011 1 A little quick math background Notation for sets, functions, mappings Linear transformations Matrices Matrix-vector multiplication Matrix-matrix multiplication

### Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

### Introduction to Principal Component Analysis: Stock Market Values

Chapter 10 Introduction to Principal Component Analysis: Stock Market Values The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from

### Diagonalisation. Chapter 3. Introduction. Eigenvalues and eigenvectors. Reading. Definitions

Chapter 3 Diagonalisation Eigenvalues and eigenvectors, diagonalisation of a matrix, orthogonal diagonalisation fo symmetric matrices Reading As in the previous chapter, there is no specific essential

### Similar matrices and Jordan form

Similar matrices and Jordan form We ve nearly covered the entire heart of linear algebra once we ve finished singular value decompositions we ll have seen all the most central topics. A T A is positive

### Section 1.1. Introduction to R n

The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to

### Nonlinear Iterative Partial Least Squares Method

Numerical Methods for Determining Principal Component Analysis Abstract Factors Béchu, S., Richard-Plouet, M., Fernandez, V., Walton, J., and Fairley, N. (2016) Developments in numerical treatments for

### Overview. Essential Questions. Precalculus, Quarter 3, Unit 3.4 Arithmetic Operations With Matrices

Arithmetic Operations With Matrices Overview Number of instruction days: 6 8 (1 day = 53 minutes) Content to Be Learned Use matrices to represent and manipulate data. Perform arithmetic operations with

### 1 2 3 1 1 2 x = + x 2 + x 4 1 0 1

(d) If the vector b is the sum of the four columns of A, write down the complete solution to Ax = b. 1 2 3 1 1 2 x = + x 2 + x 4 1 0 0 1 0 1 2. (11 points) This problem finds the curve y = C + D 2 t which

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.

Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry

### Vector Math Computer Graphics Scott D. Anderson

Vector Math Computer Graphics Scott D. Anderson 1 Dot Product The notation v w means the dot product or scalar product or inner product of two vectors, v and w. In abstract mathematics, we can talk about

### Lecture 2 Matrix Operations

Lecture 2 Matrix Operations transpose, sum & difference, scalar multiplication matrix multiplication, matrix-vector product matrix inverse 2 1 Matrix transpose transpose of m n matrix A, denoted A T or

### Mathematics of Cryptography Modular Arithmetic, Congruence, and Matrices. A Biswas, IT, BESU SHIBPUR

Mathematics of Cryptography Modular Arithmetic, Congruence, and Matrices A Biswas, IT, BESU SHIBPUR McGraw-Hill The McGraw-Hill Companies, Inc., 2000 Set of Integers The set of integers, denoted by Z,

### Math 550 Notes. Chapter 7. Jesse Crawford. Department of Mathematics Tarleton State University. Fall 2010

Math 550 Notes Chapter 7 Jesse Crawford Department of Mathematics Tarleton State University Fall 2010 (Tarleton State University) Math 550 Chapter 7 Fall 2010 1 / 34 Outline 1 Self-Adjoint and Normal Operators

### Vector Spaces; the Space R n

Vector Spaces; the Space R n Vector Spaces A vector space (over the real numbers) is a set V of mathematical entities, called vectors, U, V, W, etc, in which an addition operation + is defined and in which

### GRADES 7, 8, AND 9 BIG IDEAS

Table 1: Strand A: BIG IDEAS: MATH: NUMBER Introduce perfect squares, square roots, and all applications Introduce rational numbers (positive and negative) Introduce the meaning of negative exponents for

### Multiple regression - Matrices

Multiple regression - Matrices This handout will present various matrices which are substantively interesting and/or provide useful means of summarizing the data for analytical purposes. As we will see,

### Mechanics 1: Vectors

Mechanics 1: Vectors roadly speaking, mechanical systems will be described by a combination of scalar and vector quantities. scalar is just a (real) number. For example, mass or weight is characterized

### Exploratory Factor Analysis

Exploratory Factor Analysis Definition Exploratory factor analysis (EFA) is a procedure for learning the extent to which k observed variables might measure m abstract variables, wherein m is less than

### Grassmann Algebra in Game Development. Eric Lengyel, PhD Terathon Software

Grassmann Algebra in Game Development Eric Lengyel, PhD Terathon Software Math used in 3D programming Dot / cross products, scalar triple product Planes as 4D vectors Homogeneous coordinates Plücker coordinates

### Content. Chapter 4 Functions 61 4.1 Basic concepts on real functions 62. Credits 11

Content Credits 11 Chapter 1 Arithmetic Refresher 13 1.1 Algebra 14 Real Numbers 14 Real Polynomials 19 1.2 Equations in one variable 21 Linear Equations 21 Quadratic Equations 22 1.3 Exercises 28 Chapter

### Multivariate Normal Distribution

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### MATH 304 Linear Algebra Lecture 24: Scalar product.

MATH 304 Linear Algebra Lecture 24: Scalar product. Vectors: geometric approach B A B A A vector is represented by a directed segment. Directed segment is drawn as an arrow. Different arrows represent

### 1. True/False: Circle the correct answer. No justifications are needed in this exercise. (1 point each)

Math 33 AH : Solution to the Final Exam Honors Linear Algebra and Applications 1. True/False: Circle the correct answer. No justifications are needed in this exercise. (1 point each) (1) If A is an invertible

### Big Ideas in Mathematics

Big Ideas in Mathematics which are important to all mathematics learning. (Adapted from the NCTM Curriculum Focal Points, 2006) The Mathematics Big Ideas are organized using the PA Mathematics Standards

### A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 4: September

### South Carolina College- and Career-Ready (SCCCR) Pre-Calculus

South Carolina College- and Career-Ready (SCCCR) Pre-Calculus Key Concepts Arithmetic with Polynomials and Rational Expressions PC.AAPR.2 PC.AAPR.3 PC.AAPR.4 PC.AAPR.5 PC.AAPR.6 PC.AAPR.7 Standards Know

### Principal Components Analysis (PCA)

Principal Components Analysis (PCA) Janette Walde janette.walde@uibk.ac.at Department of Statistics University of Innsbruck Outline I Introduction Idea of PCA Principle of the Method Decomposing an Association

### Lecture L3 - Vectors, Matrices and Coordinate Transformations

S. Widnall 16.07 Dynamics Fall 2009 Lecture notes based on J. Peraire Version 2.0 Lecture L3 - Vectors, Matrices and Coordinate Transformations By using vectors and defining appropriate operations between

### A tutorial on Principal Components Analysis

A tutorial on Principal Components Analysis Lindsay I Smith February 26, 2002 Chapter 1 Introduction This tutorial is designed to give the reader an understanding of Principal Components Analysis (PCA).

### Situation 23: Simultaneous Equations Prepared at the University of Georgia EMAT 6500 class Date last revised: July 22 nd, 2013 Nicolina Scarpelli

Situation 23: Simultaneous Equations Prepared at the University of Georgia EMAT 6500 class Date last revised: July 22 nd, 2013 Nicolina Scarpelli Prompt: A mentor teacher and student teacher are discussing

### Inner Product Spaces and Orthogonality

Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,

### Applied Linear Algebra I Review page 1

Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### Figure 1.1 Vector A and Vector F

CHAPTER I VECTOR QUANTITIES Quantities are anything which can be measured, and stated with number. Quantities in physics are divided into two types; scalar and vector quantities. Scalar quantities have

### Linear Algebra: Matrices

B Linear Algebra: Matrices B 1 Appendix B: LINEAR ALGEBRA: MATRICES TABLE OF CONTENTS Page B.1. Matrices B 3 B.1.1. Concept................... B 3 B.1.2. Real and Complex Matrices............ B 3 B.1.3.

### Lecture 5: Singular Value Decomposition SVD (1)

EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system

### MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0

### Brief Introduction to Vectors and Matrices

CHAPTER 1 Brief Introduction to Vectors and Matrices In this chapter, we will discuss some needed concepts found in introductory course in linear algebra. We will introduce matrix, vector, vector-valued

### Typical Linear Equation Set and Corresponding Matrices

EWE: Engineering With Excel Larsen Page 1 4. Matrix Operations in Excel. Matrix Manipulations: Vectors, Matrices, and Arrays. How Excel Handles Matrix Math. Basic Matrix Operations. Solving Systems of

### QUALITY ENGINEERING PROGRAM

QUALITY ENGINEERING PROGRAM Production engineering deals with the practical engineering problems that occur in manufacturing planning, manufacturing processes and in the integration of the facilities and

### STATISTICS AND DATA ANALYSIS IN GEOLOGY, 3rd ed. Clarificationof zonationprocedure described onpp. 238-239

STATISTICS AND DATA ANALYSIS IN GEOLOGY, 3rd ed. by John C. Davis Clarificationof zonationprocedure described onpp. 38-39 Because the notation used in this section (Eqs. 4.8 through 4.84) is inconsistent

### Inner product. Definition of inner product

Math 20F Linear Algebra Lecture 25 1 Inner product Review: Definition of inner product. Slide 1 Norm and distance. Orthogonal vectors. Orthogonal complement. Orthogonal basis. Definition of inner product

### Images and Kernels in Linear Algebra By Kristi Hoshibata Mathematics 232

Images and Kernels in Linear Algebra By Kristi Hoshibata Mathematics 232 In mathematics, there are many different fields of study, including calculus, geometry, algebra and others. Mathematics has been

### CURVE FITTING LEAST SQUARES APPROXIMATION

CURVE FITTING LEAST SQUARES APPROXIMATION Data analysis and curve fitting: Imagine that we are studying a physical system involving two quantities: x and y Also suppose that we expect a linear relationship

### Recall that two vectors in are perpendicular or orthogonal provided that their dot

Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal

### Computer Graphics Prof. Sukhendu Das Dept. of Computer Science and Engineering Indian Institute of Technology, Madras Lecture 7 Transformations in 2-D

Computer Graphics Prof. Sukhendu Das Dept. of Computer Science and Engineering Indian Institute of Technology, Madras Lecture 7 Transformations in 2-D Welcome everybody. We continue the discussion on 2D

### Eigenvalues and Eigenvectors

Chapter 6 Eigenvalues and Eigenvectors 6. Introduction to Eigenvalues Linear equations Ax D b come from steady state problems. Eigenvalues have their greatest importance in dynamic problems. The solution

### NOTES on LINEAR ALGEBRA 1

School of Economics, Management and Statistics University of Bologna Academic Year 205/6 NOTES on LINEAR ALGEBRA for the students of Stats and Maths This is a modified version of the notes by Prof Laura

### ELEMENTS OF VECTOR ALGEBRA

ELEMENTS OF VECTOR ALGEBRA A.1. VECTORS AND SCALAR QUANTITIES We have now proposed sets of basic dimensions and secondary dimensions to describe certain aspects of nature, but more than just dimensions

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 3 Linear Least Squares Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

### Thinkwell s Homeschool Algebra 2 Course Lesson Plan: 34 weeks

Thinkwell s Homeschool Algebra 2 Course Lesson Plan: 34 weeks Welcome to Thinkwell s Homeschool Algebra 2! We re thrilled that you ve decided to make us part of your homeschool curriculum. This lesson

### Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances

Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances It is possible to construct a matrix X of Cartesian coordinates of points in Euclidean space when we know the Euclidean

### 6. Vectors. 1 2009-2016 Scott Surgent (surgent@asu.edu)

6. Vectors For purposes of applications in calculus and physics, a vector has both a direction and a magnitude (length), and is usually represented as an arrow. The start of the arrow is the vector s foot,

### NEW YORK STATE TEACHER CERTIFICATION EXAMINATIONS

NEW YORK STATE TEACHER CERTIFICATION EXAMINATIONS TEST DESIGN AND FRAMEWORK September 2014 Authorized for Distribution by the New York State Education Department This test design and framework document

### Manifold Learning Examples PCA, LLE and ISOMAP

Manifold Learning Examples PCA, LLE and ISOMAP Dan Ventura October 14, 28 Abstract We try to give a helpful concrete example that demonstrates how to use PCA, LLE and Isomap, attempts to provide some intuition

### Introduction to Principal Components and FactorAnalysis

Introduction to Principal Components and FactorAnalysis Multivariate Analysis often starts out with data involving a substantial number of correlated variables. Principal Component Analysis (PCA) is a

### Multidimensional data and factorial methods

Multidimensional data and factorial methods Bidimensional data x 5 4 3 4 X 3 6 X 3 5 4 3 3 3 4 5 6 x Cartesian plane Multidimensional data n X x x x n X x x x n X m x m x m x nm Factorial plane Interpretation

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### Introduction to Matrices for Engineers

Introduction to Matrices for Engineers C.T.J. Dodson, School of Mathematics, Manchester Universit 1 What is a Matrix? A matrix is a rectangular arra of elements, usuall numbers, e.g. 1 0-8 4 0-1 1 0 11

### Recall the basic property of the transpose (for any A): v A t Aw = v w, v, w R n.

ORTHOGONAL MATRICES Informally, an orthogonal n n matrix is the n-dimensional analogue of the rotation matrices R θ in R 2. When does a linear transformation of R 3 (or R n ) deserve to be called a rotation?

### Chapter 17. Orthogonal Matrices and Symmetries of Space

Chapter 17. Orthogonal Matrices and Symmetries of Space Take a random matrix, say 1 3 A = 4 5 6, 7 8 9 and compare the lengths of e 1 and Ae 1. The vector e 1 has length 1, while Ae 1 = (1, 4, 7) has length

### Solutions to Linear Algebra Practice Problems

Solutions to Linear Algebra Practice Problems. Find all solutions to the following systems of linear equations. (a) x x + x 5 x x x + x + x 5 (b) x + x + x x + x + x x + x + 8x Answer: (a) We create the

### This unit will lay the groundwork for later units where the students will extend this knowledge to quadratic and exponential functions.

Algebra I Overview View unit yearlong overview here Many of the concepts presented in Algebra I are progressions of concepts that were introduced in grades 6 through 8. The content presented in this course

### Multivariate Analysis of Variance (MANOVA)

Multivariate Analysis of Variance (MANOVA) Aaron French, Marcelo Macedo, John Poulsen, Tyler Waterson and Angela Yu Keywords: MANCOVA, special cases, assumptions, further reading, computations Introduction