Linear Algebra Review

Transcription

1 Linear Algebra Review Yang Feng Yang Feng (Columbia University) Linear Algebra Review 1 / 45

2 Definition of Matrix Rectangular array of elements arranged in rows and columns A matrix has dimensions The dimension of a matrix is its number of rows and columns It is expressed as 3 2 (in this case) Yang Feng (Columbia University) Linear Algebra Review 2 / 45

3 Indexing a Matrix Rectangular array of elements arranged in rows and columns [ ] a11 a A = 12 a 13 a 21 a 22 a 23 A matrix can also be notated A = [a ij ], i = 1, 2; j = 1, 2, 3 Yang Feng (Columbia University) Linear Algebra Review 3 / 45

4 Square Matrix and Column Vector A square matrix has equal number of rows and columns [ ] a a 12 a 13 a a 22 a 23 a 31 a 32 a 33 A column vector is a matrix with a single column c 1 c 2 c 3 c 4 c 5 All vectors (row or column) are matrices, all scalars are 1 1 matrices. Yang Feng (Columbia University) Linear Algebra Review 4 / 45

5 Transpose The transpose of a matrix is another matrix in which the rows and columns have been interchanged 2 5 A = [ ] A = Yang Feng (Columbia University) Linear Algebra Review 5 / 45

6 Equality of Matrices Two matrices are the same if they have the same dimension and all the elements are equal a 1 4 A = a 2 B = 7 a 3 3 A = B implies a 1 = 4, a 2 = 7, a 3 = 3 Yang Feng (Columbia University) Linear Algebra Review 6 / 45

7 Matrix Addition and Substraction Then A = 2 5 B = A + B = Yang Feng (Columbia University) Linear Algebra Review 7 / 45

8 Multiplication of a Matrix by a Scalar [ ] 2 7 A = 9 3 [ ] [ ] 2 7 2k 7k ka = k = 9 3 9k 3k Yang Feng (Columbia University) Linear Algebra Review 8 / 45

9 Multiplication of two Matrices where (AB) ij = for i = 1,, l and j = 1,, n. A l m B m n = (AB) l n, m A ik B jk, k=1 Yang Feng (Columbia University) Linear Algebra Review 9 / 45

10 Another Matrix Multiplication Example A = AB = [ 1 3 ] [ B = 5 2 ] [ ] = 41 2 Yang Feng (Columbia University) Linear Algebra Review 10 / 45

11 Special Matrices If A = A, then A is a symmetric matrix A = A = If the off-diagonal elements of a matrix are all zeros it is then called a diagonal matrix a A = 0 a 2 0 B = a Yang Feng (Columbia University) Linear Algebra Review 11 / 45

12 Identity Matrix A diagonal matrix whose diagonal entries are all ones is an identity matrix. Multiplication by an identity matrix leaves the pre or post multiplied matrix unchanged I = and AI = IA = A Yang Feng (Columbia University) Linear Algebra Review 12 / 45

13 Vector and matrix with all elements equal to one n =..... J n = n n n n 1 n =. [ ] n 1 = n 1 n n = J n Yang Feng (Columbia University) Linear Algebra Review 13 / 45

14 Linear Dependence and Rank of Matrix Consider A = and think of this as a matrix of a collection of column vectors. Note that the third column vector is a multiple of the first column vector. Yang Feng (Columbia University) Linear Algebra Review 14 / 45

15 Linear Dependence When m scalars k 1,..., k m not all zero, can be found such that: k 1 A k m A m = 0 where 0 denotes the zero column vector and A i is the i th column of matrix A, the m column vectors are called linearly dependent. If the only set of scalars for which the equality holds is k 1 = 0,..., k m = 0, the set of m column vectors is linearly independent. In the previous example matrix the columns are linearly dependent = Yang Feng (Columbia University) Linear Algebra Review 15 / 45

16 Rank of Matrix The rank of a matrix is defined to be the maximum number of linearly independent columns in the matrix. Rank properties include The rank of a matrix is unique The rank of a matrix can equivalently be defined as the maximum number of linearly independent rows The rank of an r c matrix cannot exceed min(r, c) The row and column rank of a matrix are equal The rank of a matrix is preserved under nonsingular transformations., i.e. Let A (n n) and C (k k) be nonsingular matrices. Then for any n k matrix B we have rank(b) = rank(ab) = rank(bc) Yang Feng (Columbia University) Linear Algebra Review 16 / 45

17 Inverse of Matrix Like a reciprocal 6 1/6 = 1/6 6 = 1 x 1 x = 1 But for matrices AA 1 = A 1 A = I Yang Feng (Columbia University) Linear Algebra Review 17 / 45

18 Example More generally, [ ] 2 4 A = 3 1 [ ] A =.3.2 [ ] A A = 0 1 [ ] a b A = c d [ ] d b where D = ad bc A 1 = 1 D c a Yang Feng (Columbia University) Linear Algebra Review 18 / 45

19 Inverses of Diagonal Matrices are Easy then A = /3 0 0 A 1 = 0 1/ /2 Yang Feng (Columbia University) Linear Algebra Review 19 / 45

20 Finding the inverse Finding an inverse takes (for general matrices with no special structure) O(n 3 ) operations (when n is the number of rows in the matrix) We will assume that numerical packages can do this for us in R: solve(a) gives the inverse of matrix A Yang Feng (Columbia University) Linear Algebra Review 20 / 45

21 Example Solving a system of simultaneous equations 2y 1 + 4y 2 = 20 3y 1 + y 2 = 10 ] [ ] ] y1 = [ [ y1 y 2 ] = y 2 [ [ ] 1 [ ] Yang Feng (Columbia University) Linear Algebra Review 21 / 45

22 List of Useful Matrix Properties A + B = B + A (A + B) + C = A + (B + C) (AB)C = A(BC) C(A + B) = CA + CB k(a + B) = ka + kb (A ) = A (A + B) = A + B (AB) = B A (ABC) = C B A (AB) 1 = B 1 A 1 (ABC) 1 = C 1 B 1 A 1 (A 1 ) 1 = A, (A ) 1 = (A 1 ) Yang Feng (Columbia University) Linear Algebra Review 22 / 45

23 Random Vectors and Matrices Let s say we have a vector consisting of three random variables Y 1 Y = Y 2 Y 3 The expectation of a random vector is defined as E(Y 1 ) E(Y) = E(Y 2 ) E(Y 3 ) Yang Feng (Columbia University) Linear Algebra Review 23 / 45

24 Expectation of a Random Matrix The expectation of a random matrix is defined similarly E(Y) = [E(Y ij )] i = 1,...n; j = 1,..., p Yang Feng (Columbia University) Linear Algebra Review 24 / 45

25 Variance-covariance Matrix of a Random Vector The variances of three random variables σ 2 (Y i ) and the covariances between any two of the three random variables σ(y i, Y j ), are assembled in the variance-covariance matrix of Y σ 2 (Y 1 ) σ(y 1, Y 2 ) σ(y 1, Y 3 ) cov(y) = σ 2 {Y} = σ(y 2, Y 1 ) σ 2 (Y 2 ) σ(y 2, Y 3 ) σ(y 3, Y 1 ) σ(y 3, Y 2 ) σ 2 (Y 3 ) remember σ(y 2, Y 1 ) = σ(y 1, Y 2 ) so the covariance matrix is symmetric Yang Feng (Columbia University) Linear Algebra Review 25 / 45

26 Derivation of Covariance Matrix In vector terms the variance-covariance matrix is defined by σ 2 {Y} = E[(Y E(Y))(Y E(Y)) ] because Y 1 E(Y 1 ) σ 2 {Y} = E( Y 2 E(Y 2 ) ( Y 1 E(Y 1 ) Y 2 E(Y 2 ) Y 3 E(Y 3 ) ) ) Y 3 E(Y 3 ) Yang Feng (Columbia University) Linear Algebra Review 26 / 45

27 Regression Example Take a regression example with n = 3 with constant error terms σ 2 (ɛ i ) and are uncorrelated so that σ 2 (ɛ i, ɛ j ) = 0 for all i j The variance-covariance matrix for the random vector ɛ is σ σ 2 (ɛ) = 0 σ σ 2 which can be written as σ 2 {ɛ} = σ 2 I Yang Feng (Columbia University) Linear Algebra Review 27 / 45

28 Basic Results If A is a constant matrix and Y is a random matrix then W = AY is a random matrix E(A) = A E(W) = E(AY) = A E(Y) σ 2 {W} = σ 2 {AY} = Aσ 2 {Y}A Yang Feng (Columbia University) Linear Algebra Review 28 / 45

29 Multivariate Normal Density Let Y be a vector of p observations Y 1 Y 2 Y =... Y p Let µ be a vector of the means of each of the p observations µ = µ p Yang Feng (Columbia University) Linear Algebra Review 29 / 45 µ 1 µ 2...

30 Multivariate Normal Density let Σ be the variance-covariance matrix of Y σ 2 1 σ σ 1p σ 21 σ σ 2p Σ =... σ p1 σ p2... σp 2 Then the multivariate normal density is given by P(Y µ, Σ) = 1 (2π) p/2 Σ 1/2 exp[ 1 2 (Y µ) Σ 1 (Y µ)] Yang Feng (Columbia University) Linear Algebra Review 30 / 45

31 Matrix Simple Linear Regression Nothing new-only matrix formalism for previous results Remember the normal error regression model Y i = β 0 + β 1 X i + ɛ i, ɛ i N(0, σ 2 ), i = 1,..., n Expanded out this looks like Y 1 = β 0 + β 1 X 1 + ɛ 1 Y 2 = β 0 + β 1 X 2 + ɛ 2... Y n = β 0 + β 1 X n + ɛ n which points towards an obvious matrix formulation. Yang Feng (Columbia University) Linear Algebra Review 31 / 45

32 Regression Matrices If we identify the following matrices Y 1 Y 2 Y =... Y n 1 X 1 1 X 2 X =... 1 X n β = ( β0 β 1 ɛ 1 ) ɛ 2 ɛ =... ɛ n We can write the linear regression equations in a compact form Y = Xβ + ɛ Yang Feng (Columbia University) Linear Algebra Review 32 / 45

33 Regression Matrices Of course, in the normal regression model the expected value of each of the ɛ s is zero, we can write E(Y) = Xβ This is because E(ɛ) = 0 E(ɛ 1 ) E(ɛ 2 )... E(ɛ n ) = Yang Feng (Columbia University) Linear Algebra Review 33 / 45

34 Error Covariance Because the error terms are independent and have constant variance σ 2 σ σ 2 {ɛ} = 0 σ σ 2 σ 2 {ɛ} = σ 2 I Yang Feng (Columbia University) Linear Algebra Review 34 / 45

35 Matrix Normal Regression Model In matrix terms the normal regression model can be written as Y = Xβ + ɛ where ɛ N(0, σ 2 I) Yang Feng (Columbia University) Linear Algebra Review 35 / 45

36 Least Square Estimation If we remember both the starting normal equations that we derived and the fact that nb 0 + b 1 Xi = Y i b 0 Xi + b 1 X 2 i = X i Y i 1 X 1 [ ] X X 2 [ ] X = X 1 X 2... X n.... = n Xi Xi X 2 i 1 X n Y 1 [ ] X Y 2 Y = X 1 X 2... X n.. = Y n [ ] Yi Xi Y i Yang Feng (Columbia University) Linear Algebra Review 36 / 45

37 Least Square Estimation Then we can see that these equations are equivalent to the following matrix operations with X X b = X Y b = ( b0 b 1 with the solution to this equation given by when (X X) 1 exists. ) b = (X X) 1 X Y Yang Feng (Columbia University) Linear Algebra Review 37 / 45

38 Fitted Value Ŷ = Xb Because: Ŷ = Ŷ 1 Ŷ 2... Ŷ n 1 X 1 1 X 2 = X n ( b0 b 1 b 0 + b 1 X 1 ) b 0 + b 1 X 2 =... b 0 + b 1 X n Yang Feng (Columbia University) Linear Algebra Review 38 / 45

39 Fitted Values, Hat Matrix plug in b = (X X) 1 X Y We have Ŷ = Xb = X(X X) 1 X Y or where Ŷ = HY H = X(X X) 1 X is called the hat matrix. Property of hat matrix H: 1 symmetric 2 idempotent: HH = H. Yang Feng (Columbia University) Linear Algebra Review 39 / 45

40 Residuals Then e = Y Ŷ = Y HY = (I H)Y e = (I H)Y The matrix I H is also symmetric and idempotent. The variance-covariance matrix of e is And we can estimate it by σ 2 {e} = σ 2 (I H) s 2 {e} = MSE(I H) Yang Feng (Columbia University) Linear Algebra Review 40 / 45

41 Analysis of Variance Results We know SSTO = (Y i Ȳ )2 = Y 2 i ( Y i ) 2 n Y Y = Y 2 i and J is the matrix with entries all equal to 1. Then we have As a result: ( Y i ) 2 n = 1 n Y JY SSTO = Y Y 1 n Y JY Yang Feng (Columbia University) Linear Algebra Review 41 / 45

42 Analysis of Variance Results Also, SSE = e 2 i = (Y i Ŷ i ) 2 can be represented as SSE = e e = Y (I H) (I H)Y = Y (I H)Y Notice that H1 = 1, then (I H)J = 0 Finally by similarly reasoning, SSR = ([H 1 n J]Y) ([H 1 n J]Y) = Y [H 1 n J]Y Easy to check that SSTO = SSE + SSR Yang Feng (Columbia University) Linear Algebra Review 42 / 45

43 Sums of Squares as Quadratic Forms When n = 2, an example of quadratic forms: 5Y Y 1 Y 2 + 4Y 2 2 can be expressed as matrix term as ( ) ( ) ( ) 5 3 Y1 Y1 Y 2 = Y AY 3 4 In general, a quadratic term is defined as : Y 2 Y AY = n n A ij Y i Y j i=1 j=1 where A ij = A ji Here, A is a symmetric n n matrix, the matrix of the quadratic form. Yang Feng (Columbia University) Linear Algebra Review 43 / 45

44 Quadratic forms for ANOVA SSTO = Y [I 1 n J]Y SSE = Y [I H]Y SSR = Y [H 1 n J]Y Yang Feng (Columbia University) Linear Algebra Review 44 / 45

45 Inference in Regression Analysis Regression Coefficients: The variance-covariance matrix of b is σ 2 {b} = σ 2 (X X) 1 Mean ( Response: ) To estimate the mean response at X h, define 1 X h = Then X h Ŷ h = X h b And the variance-covariance matrix of Ŷ h is σ 2 {Ŷ h } = X h σ2 {b}x h = σ 2 X h (X X) 1 X h Prediction of New Observation: s 2 {pred} = MSE(1 + X h (X X) 1 X h ) Yang Feng (Columbia University) Linear Algebra Review 45 / 45