Lecture 9. The multiple Classical Linear Regression model

Transcription

1 Lecture 9. The multple Classcal Lnear Regresson model Often we want to estmate the relaton between a dependent varable Y and say K ndependent varables X,, X K K. Even wth K explanatory varables, the lnear relaton s not exact: Y = β X + β X + L + β K X K + u wth u the random error term that captures the effect of the omtted varables. Note f the relaton has an ntercept, then X and β s the ntercept of the relaton. If we have n observatons =, K,n, they satsfy Y, X, K, X, K Y = β X + β X + L+ β X + u K K for =, K, n.

2 Why do we want to nclude more than explanatory varable?. Because we are nterested on the relaton between all K varables and Y. Because we want to estmate the drect effect of one varable and nclude the other varables for that purpose

3 To see. consder the relaton ( K = 3) Y β X + β X + X + u = β3 3 Even f we are only nterested n the effect of X on Y we cannot omt X 3, f X and X 3 are related. For nstance, f X γ X + X + v 3 = γ then after substtuton we fnd Y = ( β + β3γ ) X + ( β + β3γ ) X + u + β 3v In the relaton between Y and X, X, the coeffcent of X s the sum of the drect effect β and the ndrect effect β 3 γ of X on Y. If we nclude X 3 n the relaton, the coeffcent of X s β,.e. the drect effect of X on Y.

4 The assumptons are the same as before Assumpton : u, =, K, n are random varables wth E ( ) = 0 u Assumpton : X k, =, K, n, k =, K, K are determnstc,.e. non-random, constants. Assumpton 3 (Homoskedastcty) All u ' s have the same varance,.e. for =, K,n = E( u ) Var ( u ) = σ Assumpton 4 (No seral correlaton) The random errors u and u j are not correlated for all j =, K, n Cov ( u, u ) E( u u ) = j = j 0

5 The nexact lnear relaton for =, K, n Y = β X + β X + L+ β X + u wth assumptons -4 s the multple Classcal Lnear Regresson (CLR) model (remember wth an ntercept X =, =,, n) K K As n the smple CLR model the estmators of the regresson parameters β, K, β K are found by mnmzng the sum of squared resduals, K, β ) = n K ( Y βx β X L β K X K ) = S( β Theβ ˆ, K, βˆ K that mnmze the sum of squared resduals are Ordnary Least Squares (OLS) estmators of β,, β K K K.

6 The OLS resduals are e = Y βˆ ˆ L X βx βˆ K X K The (unbased) estmator of s = n K n e = σ s Note n K = no. of observatons - no. of regresson coeffcents (ncludng ntercept) The OLS resduals have the same propertes as n the regresson model wth regressor: for k =, K, K n () = X k e = 0 In words: The sample covarance of the OLS resduals and all regressors s 0. n e = For k = we have X = and hence Note: ths holds f the regresson model has an ntercept. = 0.

7 Goodness of ft Defne the ftted value as before Yˆ = βˆ L X + + βˆ K X K By defnton Y = Yˆ + e Because of () the sample covarance of the ftted values and the OLS resduals s 0. If the model has an ntercept, then n n ( Y Y ) = ( Y Y ) + = = = ˆ Total Explaned Unexplaned Varaton Varaton Varaton Total Sum Regresson Error of Squares Sum of Sum of (TSS) Squares Squares (RSS) (ESS) n e

8 The R or Coeffcent of Determnaton s defned as R RSS = = TSS ESS TSS The R ncreases f a regressor s added to the model. Why? Hnt: Consder sum of squared resduals.

9 R Adjusted (for degrees of freedom) decreases f added varable does not explan much,.e. f ESS does not decrease much, R = ESS /( n K ) TSS /( n ) If ESS K +, ESSK are the ESS for the model wth K + (one added) and K regressors, respectvely, then ESS K + < ESSK, and the R ncreases f X K + s added, but R decreases f ESS ESS ESS K K + > n K K + K.e. f the relatve decrease s not bg enough. For nstance for n = 00, K = 0, the relatve decrease has to be at least.% to lead to an mprovement n R. Ths cutoff value looks arbtrary (and t s!) In Secton 4.3 many more crtera that balance decrease n ESS and degrees of freedom: Forget about them.

10 Applcaton: Demand for bus travel (Secton 4.6 of Ramanathan) Varables BUSTRAVL = Demand for bus travel (000 of passenger hours) FARE = Bus fare n $ GASPRICE = Prce of gallon of gasolne ($) INCOME = Average per capta ncome n $ DENSITY = Populaton densty (persons/sq. mle) LANDAREA = Area of cty (sq. mles) Data for 40 US ctes n 988 Queston: Does demand for bus decrease f average ncome ncreases? Important for examnng effect economc growth on bus system.

11 Date: 0/0/0 Tme: :09 Sample: 40 BUSTRAVL DENSITY FARE GASPRICE INCOME LANDAREA Mean Medan Maxmum Mnmum Std. Dev Skewness Kurtoss Jarque-Bera Probablty Observatons Date: 0/0/0 Tme: :09 Sample: 40 POP Mean Medan Maxmum Mnmum Std. Dev Skewness Kurtoss 9.95 Jarque-Bera Probablty Observatons 40

12 BUSTRAVL vs. INCOME 5000 BUSTRAVL INCOME

13 Dependent Varable: BUSTRAVL Method: Least Squares Date: 0/0/0 Tme: :05 Sample: 40 Included observatons: 40 Varable Coeffcent Std. Error t-statstc Prob. C INCOME R-squared Mean dependent var Adjusted R-squared S.D. dependent var S.E. of regresson Akake nfo crteron Sum squared resd.9e+08 Schwarz crteron Log lkelhood F-statstc Durbn-Watson stat.8387 Prob(F-statstc) 0.558

14 Dependent Varable: BUSTRAVL Method: Least Squares Date: 0/0/0 Tme: :07 Sample: 40 Included observatons: 40 Varable Coeffcent Std. Error t-statstc Prob. C INCOME FARE GASPRICE POP DENSITY LANDAREA R-squared Mean dependent var Adjusted R-squared S.D. dependent var S.E. of regresson Akake nfo crteron Sum squared resd 8367 Schwarz crteron 6.5 Log lkelhood F-statstc Durbn-Watson stat.0867 Prob(F-statstc)

15 Reported are Descrptve statstcs Scatterplot BUSTRVL and INCOME OLS estmates n smple lnear regresson model OLS estmates n multple lnear regresson model Note n smple model INCOME has postve effect (be t that coeffcent s not sgnfcantly dfferent from 0) If we nclude the other varables the effect s negatve: $ more average ncome per head gves about 95 fewer passenger hours. Note that the ft mproves dramatcally from R =.05 to R =. 9.

16 The samplng dstrbuton of β,, βˆ ˆ K K Under assumptons -4 the samplng dstrbuton of β ˆ, K, βˆ K has mean β, K, β K and a samplng varance that s proportonal to σ. The square root of the estmated (estmate σ by s ) varance of the ndvdual regresson coeffcents are called the standard errors of the regresson coeffcents. Regresson programs always report both βˆ k and std( β ˆ k ), the standard error of βˆ k. The OLS estmator s also the Best Lnear Unbased Estmator (BLUE) n the multple CLR model.

17 If we make the addtonal assumpton Assumpton 5. The random error terms u, =, K, n are random varables wth a normal dstrbuton. We can derve the samplng dstrbuton of β ˆ, K, ˆ and s. β K The samplng dstrbuton of β βˆ ˆ, K, K multvarate normal wth mean β, K, β K. It can be shown that for all k =, K, K T k = βˆ k β std( βˆ k k ) has a t-dstrbuton wth freedom. n K degrees of As n the smple CLR model we can use ths to fnd a confdence nterval for β k. Ths s done n the same way.

18 Hypothess testng n the multple CLR model We consder two cases. Hypotheses nvolvng coeffcent. Hypotheses nvolvng or more coeffcents. Hypotheses nvolvng coeffcent For regresson coeffcent β k the hypothess s H β = β 0 : k k0 wth alternatve hypothess or H β β two-sded alternatve : k k0 H β > β one-sded alternatve : k k0 In these hypotheses β k0 s a hypotheszed value, e.g. β k0 = 0 f the null hypothess s no effect of X on Y. k

19 The test s based on the statstc T k = βˆ k β std( βˆ k0 k ) If H 0 s not true, then T k s lkely ether large negatve or large postve. For two-sded alternatve, H : βk βk0, we reject f T k > c and for a one-sded alternatve, H : β k > βk0, f T k > d. For a test wth a 5% confdence level, the cutoff values c, d are chosen such that or Pr( > c) =.05 T k,.e. Pr( T k ) > c) =. 05 (two-sded) Pr( > d) =.05 T k (one-sded) where T k has a t-dstrbuton wth n K d.f.

20 In example, = 40, K = 7 n, c. 035, d. 695 Note that the coeffcents of INCOME, POP are sgnfcantly dfferent from 0 at the 5% level. The coeffcent of DENSITY s sgnfcantly dfferent from 0 at the 0% level or f we test wth a one-sded alternatve.

21 . Hypothess nvolvng or more coeffcent As example consder the queston whether the model wth only INCOME as regressor s adequate. In the more general model wth INCOME, FARE, GASPRICE, POP, DENSITY, LANDAREA ths corresponds to the hypothess that the coeffcents of FARE, GASPRICE, POP, DENSITY, LANDAREA, β 3, β 4, β5, β6, β7 are all 0,.e. H 0 : β3 = 0, β4 = 0, β5 = 0, β6 = 0, β7 = 0 wth alternatve hypothess H : One of these coeffcents s not 0 How do we test ths? Idea: If 0 H s true then the model wth only INCOME should ft as well as the model wth all regressors.

22 Measure of ft s sum of squared OLS resduals n e. = Denote sum of squared resduals f H 0 s true by ESS 0. Ths s ESS f only INCOME s ncluded. The ESS f H s true s denoted by ESS. Note: ESS 0 ESS. Why? We consder F ESS0 ESS = ESS Numerator: Dfference of restrcted ESS ( ESS 0) and unrestrcted ESS ( ESS ) dvded by the number of restrctons. Denomnator: Estmator of wth all varables. σ n the model

23 We reject f F s large,.e. F > c. If H 0 s true then F has an F-dstrbuton wth degrees of freedom 5 (numerator) and 35 (denomnator). From Table n back cover Ramanathan, we fnd that e.g. f c. 49. In example: Pr( F > c) =.05 ESS = ESS = ESS0 ESS = F = =

24 Concluson: We reject H 0 at the 5% level (and at the % level; check ths) and the model wth only INCOME s not adequate. In general, we consder F = ESS 0 ESS ESS n K m wth m the number of restrctons and we reject H 0 f F s (too) large. Ths s the F test or Wald test (Wald test s really F test f n s large)

25 Specal case: Test of overall sgnfcance H 0 : β = L = β K = 0.e. all regresson coeffcents except ntercept are 0. Hence, m = K In example, the F statstc for ths hypothess s 64.4 wth under H 0, m = 6 and n K = 33 d.f. Hence, we reject H 0 and we conclude that the regressors are needed to explan BUSTRAVL.