5. Matrix Algebra A Prelude to Multiple Regression

Transcription

1 5 Matrix Algebra A Prelude to Multiple Regression Matrices are arrays of numbers and are denoted by boldface (capital) symbols Example: a 2 2 matrix (always #rows #columns) A = Example: a 4 2 matrix B, and a 2 3 matrix C B = , C =

2 In general, an r c matrix is given by A r c = a 11 a 12 a 1j a 1c a 21 a 22 a 2j a 2c a i1 a i2 a ij a ic a r1 a r2 a rj a rc or in abbreviated form A r c = a ij, i = 1, 2,, r, j = 1, 2,, c 1st subscript gives row#, 2nd subscript gives column# Where is a 79 or a 44? 2

3 A matrix A is called square, if it has the same # of rows and columns (r = c) Example: A 2 2 = Matrices having either 1 row (r = 1) or 1 column (c = 1) are called vectors Example: column vector A (c = 1) and row vector C (r = 1) A = , C = c 1 c 2 c 3 c 4 Row vectors always have the prime! 3

4 Transpose: A is the transpose of A where A = , A = A is obtained by interchanging columns & rows of A a ij is the typical element of A a ij is the typical element of A a ij = a ji (a 12 = a 21) 4

5 Equality of Matrices: Two matrices A and B are said to be equal if they are of the same dimension and all corresponding elements are equal A r c = B r c means a ij = b ij, i = 1,, r, j = 1,, c Addition and Subtraction: To add or subtract matrices they must be of the same dimension The result is another matrix of this dimension If A 3 2 = , B 3 2 = , then its sum and its difference is calculated elementwise C = A + B = =

6 D = A B = =

7 Regression Analysis Remember, we had (X 1, Y 1 ), (X 2, Y 2 ),, (X n, Y n ) and wrote the SLR as Y i = E(Y i ) + ɛ i, i = 1, 2,, n Now we are able to write the above model as Y n 1 = E(Y n 1 ) + ɛ n 1 with the n 1 column vectors Y = Y 1 Y 2 Y n, E(Y) = E(Y 1 ) E(Y 2 ) E(Y n ), ɛ = ɛ 1 ɛ 2 ɛ n 7

8 Matrix Multiplication: (1) by a scalar, which is a (1 1) matrix Let A = If the scalar is 3, then 3 A = A + A + A or 3 A = =

9 Generally, if λ denotes the scalar, we get λ A = 5λ 3λ λ 2λ 4λ 7λ = A λ We can also factor out a common factor, eg = (2) by a matrix: we write the product of two matrices A and B as AB For AB to exist, the #col s of A must be the same as the #rows of B A 3 2 = , B 2 3 =

10 Let C = AB You get c ij by taking the inner product of the ith row of A and the jth column of B, that is c ij = #col s in A k=1 a ik b kj Since i = 1,, #rows in A, j = 1,, #col s in B the resulting matrix C has dimension: (#rows in A) (#col s in B) For C to exist, (#col s in A)=(#rows in B) 10

11 Hence, for A 3 2 B 2 3 we get the 3 3 matrix C = ( 1) ( 1) ( 1) = Note, this is different to D 2 2 = B 2 3 A 3 2 which gives the 2 2 matrix D = = For AB we say, B is premultiplied by A or A is postmultiplied by B 11

12 Regression Analysis Remember our SLR with all means on the straight line E(Y i ) = β 0 + β 1 X i, i = 1, 2,, n With we get X n 2 = E(Y) = Xβ = Thus we rewrite the SLR as 1 X 1 1 X 2 1 X n 1 X 1 1 X 2 1 X n, β 2 1 = β0 β 1 Y = Xβ + ɛ = β0 β 1 β 0 + β 1 X 1 β 0 + β 1 X 2 β 0 + β 1 X n 12

13 Important Matrices in Regression: Y Y = Y 1 Y 2 Y n X X = X Y = Y 1 Y 2 Y n X 1 X 2 X n X 1 X 2 X n = 1 X 1 1 X 2 1 X n Y 1 Y 2 Y n n i=1 Y 2 i = n i X i = i Y i i X iy i i X i i X2 i 13

14 Special Types of Matrices: Symmetric Matrix, if A = A, A is said to be symmetric, eg A = , A = A symmetric matrix necessarily is square! Any product like Z Z is symmetric 14

15 Diagonal Matrix is a square matrix whose off-diagonal elements are all zeros A = , B = b b b b 44 Identity Matrix I is a diagonal matrix whose elements are all 1s, eg B above with b ii = 1, i = 1, 2, 3, 4 Pre- and postmultiplying by I does not change a matrix, A = IA = AI 15

16 Vector and matrix with all elements Unity A column vector with all elements 1 is denoted by 1, a square matrix with all elements 1 is denoted by J, 1 = 1 1, J = Note that for an n 1 vector 1 we obtain 1 1 = = n 16

17 and 11 = = = J n n Zero vector A column vector with all elements 0 0 =

18 Linear Dependence and Rank of Matrix Consider the following matrix A = Think of A as being made up of 4 column vectors A =

19 Notice that the third column is 5 times the first = We say the columns of A are linearly dependent (or A is singular) When no such relationships exist, A s columns are said to be linearly independent The rank of a matrix is the number of linearly independent columns (in the example, its rank is 3) 19

20 Inverse of a Matrix Q: What s the inverse of a number (6)? A: Its reciprocal (1/6)! A number multiplied by its inverse always equals 1 Generally, for the inverse 1/x of a scalar x x 1 x = 1 x x = x 1 x = xx 1 = 1 In matrix algebra, the inverse of A is the matrix A 1, for which A 1 A = AA 1 = I In order for A to have an inverse: A must be square, col s of A must be linearly independent 20

21 Example: Inverse of a matrix A 2 2 = , A = AA 1 = = A 1 A = =

22 Example: Inverse of a diagonal matrix D = d d d 3, D 1 = 1/d /d /d 3 eg, D = DD 1 = , D 1 = 1/ /4 1/ /4 =

23 Finding the Inverse: The 2 2 case A = a b c d, A 1 = 1 D d c b a where D = ad bc denotes the determinant of A If A is singular then D = 0 and no inverse would exist AA 1 a b 1 d b = c d D c a = 1 ad bc ab + ba D cd dc cb + da 1 0 = = I 0 1 = 1 D D 0 0 D 23

24 Example: A = Determinant D = ad bc = = 10 A 1 = =

25 Regression Analysis Principal inverse matrix in regression is the inverse of X X = n i X i i X i i X2 i = a b c d Its determinant is D = n ( ) ( ) 2 Xi 2 X i = n Xi 2 1 ( ) 2 nx n i i i ( ) ( ) = n Xi 2 nx 2 = n (X i X) 2 = ns XX 0 i 25 i

26 Thus (X X) 1 = 1 ns XX i X2 i i X i i X i n = 1 S XX 1 n i X2 i X X 1 Uses of Inverse Matrix In ordinary algebra, we solve an equation of the type 5y = 20 by multiplying both sides by the inverse of 5 and obtain y = (5y) =

27 System of equations: 2y 1 + 4y 2 = 20 3y 1 + y 2 = 10 With matrix algebra we rewrite this system as y1 y 2 = Thus, we have to solve AY = C Premultiplying with the inverse A 1 gives A 1 AY = A 1 C Y = A 1 C 27

28 The solution of these equations then is y1 y 2 = = =

29 Some Basic Matrix Facts 1 (AB)C = A(BC) 2 C(A + B) = CA + CB 3 (A ) = A 4 (A + B) = A + B 5 (AB) = B A 6 (AB) 1 = B 1 A 1 7 (A 1 ) 1 = A 8 (A ) 1 = (A 1 ) 9 (ABC) = C B A 29

30 Random Vectors and Matrices A random vector is a vector of random variables, eg Y = Y 1, Y 2,, Y n The expected value of Y is the vector E(Y) = E(Y 1 ), E(Y 2 ),, E(Y n ) Regression Example: ɛ = ɛ 1 ɛ 2 ɛ n ; E(ɛ) = E(ɛ 1 ) E(ɛ 2 ) E(ɛ n ) = 0 n 1 30

31 The usual rules for expectation still work: Suppose V and W are random vectors and A, B, and C are matrices of constants Then E(AV + BW + C) = AE(V) + BE(W) + C Regression Example: Find E(Y) = E(Xβ + ɛ) E(Xβ + ɛ) = E(Xβ) + E(ɛ) = Xβ + 0 = Xβ 31

32 Variance-Covariance Matrix of a Random Vector For a random vector Z n 1 define var(z) = var(z 1 ) cov(z 1, Z 2 ) cov(z 1, Z n ) cov(z 2, Z 1 ) var(z 2 ) cov(z 2, Z n ) cov(z n, Z 1 ) cov(z n, Z 2 ) var(z n ) where cov(z i, Z j ) = E (Z i E(Z i ))(Z j E(Z j )) (n n) matrix If Z i and Z j are independent, then cov(z i, Z j ) = 0 = cov(z j, Z i ) It is a symmetric 32

33 Regression Example: because we assumed n independent random errors ɛ i, each with the same variance σ 2, we have var(ɛ) = σ σ σ 2 = σ2 I n n 33

34 Rules for a Variance-Covariance Matrix Remember: if V is a rv and a, b are constant terms, then var(av + b) = var(av ) = a 2 var(v ) Suppose now that V is a random vector and A, B are matrices of constants Then var(av + B) = var(av) = Avar(V)A Regression Analysis: Find var(y) = var(xβ + ɛ) var(xβ + ɛ) = var(ɛ) = σ 2 I n n Off-diagonal elements are zero because the ɛ i s, and hence the Y i s, are independent 34

35 SLR in Matrix Terms Now we can write the SLR in matrix terms compactly as and we assume that ɛ N(0, σ 2 I) β and σ 2 are unknown parameters X is a constant matrix Y = Xβ + ɛ Consequences: E(Y) = Xβ and var(y) = σ 2 I In the next step we define the Least Squares (LS) estimators (b 0, b 1 ) using matrix notation 35

36 Normal Equations: Remember the LS criterion Q = n (Y i (β 0 + β 1 X i )) 2 = (Y Xβ) (Y Xβ) i=1 Recall that when we take derivatives of Q wrt β 0 and β 1 and set the resulting equations equal to zero, we get the normal equations nb 0 + nxb 1 = ny nxb 0 + i X 2 i b 1 = i X i Y i Let s write these equations in matrix form n nx nx i X2 i b0 b 1 = ny i X iy i 36

37 But with b 2 1 = (b 0 b 1 ), this is exactly equivalent to (X X)b = (X Y) Premultiplying with the inverse (X X) 1 gives (X X) 1 (X X)b = (X X) 1 (X Y) b = (X X) 1 X Y 37

38 Fitted Values and Residuals Remember Ŷi = b 0 + b 1 X i Because Ŷ 1 Ŷ n = 1 X 1 1 X n b0 b 1 = b 0 + b 1 X 1 b 0 + b 1 X n we write the vector of fitted values as Ŷ = Xb With b = (X X) 1 X Y we get Ŷ = Xb = X(X X) 1 X Y 38

39 We can express this by using the (n n) Hat Matrix H = X(X X) 1 X (it puts the hat on Y) as Ŷ = HY H is symmetric (H = H ) & idempotent (HH = H) Symmetric: ( H = X(X X) 1 X ) ( 9 = X (X X) 1) X ( 8 = X (X X) ) 1 ( X 5 1X = X X X) = H 39

40 Idempotent: because (X X) 1 X X = I we have HH = (X(X X) 1 X )( X(X X) 1 X ) = XI(X X) 1 X = H With these results we get (HX = X) (HIH = H) E(Ŷ) = E(HY) = HXβ = Xβ var(ŷ) = var(hy) = H σ2 I H = σ 2 H Residuals: e = Y Ŷ = IY HY = (I H)Y Like H, also I H is symmetric and idempotent E(e) = (I H)E(Y) = 0 var(e) = var((i H)Y) = σ 2 (I H) 40

41 Inferences in Regression Analysis Distribution of LS Estimates b = (X X) 1 X Y = CY = c11 c 1n Y 1 c 21 c 2n Y n with C = (X X) 1 X a 2 n matrix of constants Thus, each element of b is a linear combination of independent normals, Y i s, and therefore a normal rv ( ) E(b) = E (X X) 1 X Y = (X X) 1 X E(Y) = (X X) 1 X Xβ = Iβ = β 41

42 ( ) var(b) = var (X X) 1 X Y = (X X) 1 X var(y) ((X X) 1 X ) = (X X) 1 X σ 2 I X(X X) 1 = σ 2 (X X) 1 X X(X X) 1 = σ 2 I(X X) 1 = σ 2 (X X) 1 With the previous result we have var(b) = σ 2 (X X) 1 = σ2 S XX 1 n i X2 i X X 1 Its estimator is var(b) = MSE S XX 1 n i X2 i X X 1 42

43 As covariance/correlation between b 0 and b 1 we get cov(b 0, b 1 ) = σ2 S XX X cor(b 0, b 1 ) = cov(b 0, b 1 ) var(b0 )var(b 1 ) = 1 n X i X2 i b 0, b 1 are not independent! Together we have b N (β, σ 2 (X X) 1) This is used to construct CI s and tests regarding β as before 43

44 Estimate the Mean of the Response at X h Recall our estimate for E(Y h ) = β 0 + β 1 X h is Ŷ h = b 0 + b 1 X h = X hb, where X h = (1, X h) The fitted value is a normal rv with mean and variance E(Ŷh) = E(X hb) = X he(b) = X hβ var(ŷh) = var(x hb) = X hvar(b)x h = X hσ 2 (X X) 1 X h = σ 2 X h(x X) 1 X h Thus, Ŷ h X h β σ2 X h (X X) 1 X h N(0, 1) Ŷ h X h β MSE X h (X X) 1 X h t(n 2) 44

45 What is X h (X X) 1 X h? = X n i X2 i X 1 h S XX X 1 Xh 1 = 1 S n i X2 i XX 1 h X + X h XX X h ( ) 1 1 = Xi 2 XX h XX h + Xh 2 S XX n i ( 1 1 ( = S XX + nx 2) ) 2XX h + Xh 2 n S XX = 1 n + 1 S XX (X h X) 2 by applying S XX = i X2 i nx2 45

46 Matrix Algebra with R: Whiskey Example > one <- rep(1,10); age <- c(0,5,1,2,3,4,5,6,7,8) > y <- c(1046, 1041, 1044, 1050, 1060, , 1077, 1087, 1106, 1121) > X <- matrix(c(one, age), ncol=2) > XtX <- t(x) %*% X; XtX,1,2 1, , > solve(xtx),1,2 1, , > b <- solve(xtx) %*% t(x)%*%y; b,1 1, , > H <- X %*% solve(xtx) %*% t(x) 46

47 > e <- y - H %*% y; SSE <- t(e) %*% e; SSE,1 1, > asnumeric(sse/8) * solve(xtx),1,2 1, , > summary(lm(y ~ age)) Coefficients: Estimate StdError t value Pr(> t ) (Intercept) < 2e-16 *** age e-06 *** 47