Interpretation of coeffi cients in multiple regression

Transcription

1 Interpretation of coeffi cients in multiple regression Consider the multiple regression model with K = 2 : y i = x 1i β 1 + x 2i β 2 + u i i = 1,..., n or y = X 1 β 1 + X 2 β 2 + u = Xβ + u where y, X 1, X 2 and u are n 1 column vectors, X is an n 2 matrix and β is a 2 1 column vector Notice that X 1 = (x 11,..., x 1n ) and X 2 = (x 21,..., x 2n ) denote the two columns of X 1

2 y = X 1 β 1 + X 2 β 2 + u = Xβ + u Assume, for simplicity, that y = X 1 = X 2 = 0 (not really restrictive, since we can always express variables as deviations from their sample means) Recall β OLS = (X X) 1 X y = Ay ŷ = X β OLS = X(X X) 1 X y = P y û = y ŷ = (I X(X X) 1 X )y = (I P )y = My with AX = I, MX = 0 and X û = 0 2

3 Note that X û = X 1û X 2û = 0 0 Also note that y = ŷ + û = X β OLS + û = X 1 β1 + X 2 β2 + û How can we interpret the OLS estimates of the individual coeffi cients, β 1 and β 2, in the multiple regression model? 3

4 First consider the simple regression of y on X 1 : y = X 1 γ + r y We have γ OLS = (X 1X 1 ) 1 X 1y = A 1 y ỹ = X 1 γ OLS = X 1 (X 1X 1 ) 1 X 1y = P 1 y r y = y ỹ = (I X 1 (X 1X 1 ) 1 X 1)y = (I P 1 )y = M 1 y with A 1 X 1 = I, M 1 X 1 = 0 and X 1 r y = 0 Note that we can write γ OLS = A 1 y = A 1 [X 1 β1 + X 2 β2 + û] 4

5 Consider also the simple regression of X 2 on X 1 : We have X 2 = X 1 δ + r 2 δols = (X 1X 1 ) 1 X 1X 2 = A 1 X 2 X 2 = X 1 δols = X 1 (X 1X 1 ) 1 X 1X 2 = P 1 X 2 r 2 = X 2 X 2 = (I X 1 (X 1X 1 ) 1 X 1)X 2 = (I P 1 )X 2 = M 1 X 2 with X 1 r 2 = 0 5

6 Now write γ OLS = A 1 y = A 1 [X 1 β1 + X 2 β2 + û] = (X 1X 1 ) 1 (X 1X 1 ) β 1 + (X 1X 1 ) 1 (X 1X 2 ) β 2 + (X 1X 1 ) 1 X 1û = β 1 + δ OLS β2 + 0 = β 1 + δ OLS β2 So β 1 γ OLS, unless either β 2 = 0 or δ OLS = 0 (or both) In general, the estimated coeffi cient ( β 1 ) on X 1 in the multiple regression of y on X 1 and X 2 differs from the estimated coeffi cient ( γ OLS ) on X 1 in the simple regression of y on X 1 6

7 Now consider the simple regression of r y on r 2 : r y = r 2 b + v where r y are the OLS residuals from the simple regression of y on X 1, and r 2 are the OLS residuals from the simple regression of X 2 on X 1 bols = ( r 2 r 2 ) 1 r 2 r y Recall that r 2 = M 1 X 2 and r y = M 1 y, with M 1 a symmetric and idempotent matrix r 2 r 2 = (M 1 X 2 ) M 1 X 2 = (X 2M 1)M 1 X 2 = X 2M 1 M 1 X 2 = X 2M 1 X 2 r 2 r y = (M 1 X 2 ) M 1 y = (X 2M 1)M 1 y = X 2M 1 M 1 y = X 2M 1 y 7

8 So bols = ( r 2 r 2 ) 1 r 2 r y = (X 2M 1 X 2 ) 1 X 2M 1 y Consider M 1 y = M 1 [X 1 β1 + X 2 β2 + û] = M 1 X 1 β1 + M 1 X 2 β2 + M 1 û = 0 + M 1 X 2 β2 + M 1 û since M 1 X 1 = 0 Moreover M 1 û = (I X 1 (X 1X 1 ) 1 X 1)û = û X 1 (X 1X 1 ) 1 X 1û = û since X 1û = 0 Then we have M 1 y = M 1 X 2 β2 + û 8

9 And using M 1 y = M 1 X 2 β2 + û, we can write bols = (X 2M 1 X 2 ) 1 X 2M 1 y = (X 2M 1 X 2 ) 1 X 2[M 1 X 2 β2 + û] = (X 2M 1 X 2 ) 1 (X 2M 1 X 2 ) β 2 + (X 2M 1 X 2 ) 1 X 2û = β 2 since X 2û = 0 The estimated coeffi cient ( β 2 ) on X 2 in the multiple regression of y on X 1 and X 2 equals the estimated coeffi cient ( b OLS ) on r 2 in the simple regression of r y on r 2, i.e. where we regress the OLS residuals ( r y ) from the regression of y on X 1 on the OLS residuals ( r 2 ) from the regression of X 2 on X 1 9

10 y = X 1 β 1 + X 2 β 2 + u r y = r 2 b + v β 2 = b OLS That is, we partial out or remove (in a linear regression sense) the effect of X 1 on both y and X 2 before considering the relationship between y and X 2 In effect, the estimated coeffi cient in the multiple regression considers the relationship between y and X 2, controlling for the effect of X 1 Recall that we have X 1 r y = 0 and X 1 r 2 = 0, so both r y and r 2 are orthogonal to X 1 by construction 10

11 This is closely related to the concept of partial correlation between y and X 2, holding X 1 constant ρ yx2.x 1 = corr( r y, r 2 ) = cov( r y, r 2 ) Var( ry )Var( r 2 ) β 2 = b OLS = ( r 2 r 2 ) 1 r 2 r y = r yi r 2i r 2i 2 = 1 n r yi r 2i 1 n r 2i 2 The numerator 1 n r yi r 2i estimates cov( r y, r 2 ) 11

12 As a result: β 2 has the same sign as ρ yx2.x 1 β 2 = 0 if and only if ρ yx2.x 1 = 0 the R 2 from the simple regression of r y on r 2 estimates (ρ yx2.x 1 ) 2 The OLS estimates of coeffi cients in the multiple regression model are thus informative about patterns of partial correlations in the data 12

13 These ideas generalise to models with more than 2 explanatory variables Consider y i = β 1 x 1i + β 2 x 2i β K 1 x (K 1)i + β K x Ki + u i y i = γ 1 x 1i + γ 2 x 2i γ K 1 x (K 1)i + r yi x Ki = δ 1 x 1i + δ 2 x 2i δ K 1 x (K 1)i + r Ki r yi = b r Ki + v i Again we have that β K = b OLS = ( r K r K ) 1 r K r y = r yi r Ki r Ki 2 = 1 n r yi r Ki 1 n r Ki 2 13

14 Notice that here we have partialled out the effects of all the remaining K 1 explanatory variables in the multiple regression model, in the construction of the OLS residuals r y and r K By construction, these OLS residuals are orthogonal to (X 1, X 2,..., X K 1 ) i.e. we have X j r y = X j r K = 0 for j = 1, 2,..., K 1 The result that β K = b OLS is sometimes called the Frisch-Waugh-Lovell theorem 14

15 Again the OLS estimate ( β K ) of the coeffi cient on X K in the multiple regression of y on (X 1, X 2,..., X K 1, X K ) is related to the partial correlation between y and X K, holding (X 1, X 2,..., X K 1 ) constant ρ yxk.x 1 X 2...X K 1 = corr( r y, r K ) = cov( r y, r K ) Var( ry )V ar( r K ) The R 2 from the OLS regression of r y on r K estimates (ρ yxk.x 1 X 2...X K 1 ) 2 This provides a measure of the strength of the relationship between y and X K, holding (X 1, X 2,..., X K 1 ) constant, and is sometimes referred to as the partial R 2 measure 15