ECON 43 arald Goldste, Nov. 3 Itroducto to F-testg lear regso models (Lecture ote to lecture Frday 5..3) Itroducto A F-test usually s a test where several parameters are volved at oce the ull hypothess cotrast to a T-test that cocers oly oe parameter. The F-test ca ofte be cosde a refemet of the more geeral lelhood rato test (LR) cosde as a large sample ch-square test. The F-test ca (e.g.) be used the specal case that the error term a regso model s ormally dstrbuted. Ths s the same way as the T-test for a sgle parameter a model wth ormally dstrbuted data s a refemet of a more geeral large sample Z-test. The F-test (as the T-test) ca be used also for small data sets cotrast to the large sample ch-square tests (ad large sample Z-tests), but requre addtoal assumptos of ormally dstrbuted data (or error terms). Note also that, f the ull-hypothess cossts of oly oe parameter, the the F ad T test statstcs satsfy F T exactly, so that a two-sded T-test wth d degrees of freedom s equvalet to a F-test wth ad d degrees of freedom. Example from o-semar exercse wee 38 (og Kog cosumer data). Y Cosumpto: housg, cludg fuel ad lght. X Icome (.e., we use total expedture as a proxy).,,, where cosumers. Lower c. (< 5) gher c. (> 5) Y =cos. X=c. Y=cos. X=c. 497 53 585 658 839 448 64 65 3 798 3358 98 537 4 89 46 746 6748 5 755 385 865 973 6 388 49 54 5637 7 67 97 8 48 773 9 8 44 69 66 53 738 66 659
Exp. Commodty group Males 5 5 3 38 864 4 99 899 ousehold expedtu me 4 6 8 XM Testg of structural brea as a example of F-testg Ths s a typcal F-test type of problem a regso model. The example: Full model (cludg the possblty of a structural brea betwee lower ad hgher comes) Suppose ( X, Y ),( X, Y ),,( X, Y ) are d pars as ( X, Y) ~ f ( x, y) f ( y x) f X ( x) (where f ( x, y ) deotes the jot populato pdf of ( XY, ). As dscussed before, whe all parameters of tet are cotaed the codtoal pdf f ( y x ), we do ot eed to say aythg about the margal pdf f ( x ), ad we ca cosde all X as fxed equal to ther observed values, x. Let D be a dummy for hgher come, Note that D s a fucto of X. f X 5 D f X 5 For usg the F-test we eed to postulate a ormal ad homoscedastc pdf for f ( y x ),.e., ( Y X x) ~ N E( Y x),, where X ( ) ( 3) x f d E( Y x) x d 3dx x f d dcatg a structural brea f at least oe of, 3 s dfferet from zero. Cosderg the observed X s as fxed, we may exps the model smpler as
3 Y x d d x e where e, e,, e ~ d wth () 3 e N. ~ (, ) We wat to test the ull hypothess of o structural brea as expsed by the Reduced model () Y x e where e, e,, e ~ d wth e N. ~ (, ) whch s the same as testg : ad 3 agast : At least oe of, 3 (.e.) the model. We see that here cotas two trctos o the betas so a F-test s proper here.. The F-test has a smple recpe, but to uderstad ths we eed to defe the F-dstrbuto ad 4 smple facts about the multple regso model wth d ad ormally dstrbuted error terms. Frst the F-dstrbuto: Itroducto to the F-dstrbuto (see Rce, secto 6.) Defto. If Z, Z are depedet ad ch-square dstrbuted wth r, r degrees of freedom (df) pectvely ( short Z r F Z r Z j ~ r j, j, ), the has a dstrbuto called the F-dstrbuto wth freedom ( short F ~ F( r, r ) ). r ad r degrees of Notes The F-dstrbuto s a oe-topped o-symmetrc dstrbuto o the postve axs cocetrated aroud (ote that, sce E( Z ) df r j, the j E Z r ). If F ~ F( r, r ), the F ~ F( r, r ) (follows drectly from defto). Table 5 the bac of Rce gves oly upper percetles for varous F-dstrbutos. If you eed lower percetles, use the prevous property (a lower percetle of F s a upper percetle of F ). j j
4 The basc tool for performg a F-test s the Source table a Stata-output, whch summarzes varous measu of varato relevat to the aalyss. The bass for uderstadg ths table s gve secto 3 whch you may sp at frst f you just wsh to ow the recpe for performg the F-test secto 4 ( that case, just read Fact,,3, ad the Source table (8) secto 3, ad come bac to the explaato secto 3 later f eeded). 3 Some basc facts about the regso model ad the source table Frst a summary of OLS Model. () Y x x e,,, where the { x ;,,, ad j,,, } are cosde fxed umbers ad repet j observatos of explaatory varables, X, X,, X (see justfcato the appedx of the lecture ote o cto). For the error terms we assume, e, e,, e are d ad ormally dstrbuted, e N. ~ (, ) The error terms (beg o observable sce the beta s are uow) ca be wrtte () e Y x x Y E( Y ) The OLS estmators (equal to the mle estmators ths model) are determed as mmzg (3) Q( ) Y x x e wth pect to (,,, ). The soluto to ths mmzato problem (whch s always uque uless there s a exact lear relatoshp the data betwee some of the X- varables) are the OLS estmators, ˆ ˆ ˆ,,,, satsfyg the so called ormal equatos : (4) Q( ˆ ), j,,,, j We defe the cted Y s ad duals as pectvely Yˆ ˆ ˆ x ˆ x, ad eˆ Y Yˆ,,,, The ormal equatos (4) ca be expsed terms of the duals as (defg, for coveece, a costat term varable, x ), Other programs call ths Aova table. Aova stads for aalyss of varace.
5 (5) eˆ x for j,,,, j I partcular, the frst ormal equato (5) shows that eˆ ˆ e x, ad, therefore that the mea of the Y s must be equal to the mea of the cted Y s, ) (6) Y Yˆ. (Notce Yˆ Yˆ ( Y eˆ ) Y Y We ow troduce the relevat sums of squa ( s) whch satsfy the same (fudametal) relatoshp (fact ) as the smple regso wth oe explaatory varable: Defe Total sum of squa, tot Y Y Resdual sum of squa, ˆ eˆ Y Y Q( ˆ ) Model sum of squa, ˆ ˆ ˆ Y Y Y Y (6) Wrtg Y Y Y Yˆ Yˆ Y, squarg, ad usg a lttle bt of smple (matrx) OLS algebra, we get the fudametal (ad bass for the Source table) Fact : tot Note. Ofte the pose Y, ad R tot s terpreted as measurg the varato of the explaed part ( Y ˆ ) of as the varato of the uexplaed part of Y. Itroducg we get the so called coeffcet of determato terpreted as the percetage (.e., R ) of the total varato of Y explaed by the regsors, X, X,, X, the data. It ca also be show that, defg R as the sample correlato betwee, Y ad Y ˆ (called the (sample) multple correlato betwee Y ad X, X,, X ), the R s exactly equal to the defto gve. I the Stata output of the Source table. R s reported to the rght To do ferece we also eed to ow the dstrbutoal propertes of the s. Frst of all, they ca be used to estmate the error varace,, uder varous crcumstaces. Notce frst that e N e N e ~ (, ) ~ (,) ~ (as show the lectu). Sce a Wheever the regso fucto has a costat term,, ad oly the.
6 sum of depedet ch-square varables s tself ch-square wth degrees of freedom equal to the sum of degrees of freedom for each varable (recall also that the expected value of chsquare varable s equal to the degree of freedom), we have e ~ E e E e ece, f we could observe the e s, we could use e as a ud estmator of. The e s beg o observable, we use the duals, e ˆ s, stead. The ormal equatos (5) show that the duals must satsfy trctos, so oly ca vary freely (hece the term degree of freedom, beg df for the duals). Now the matrx OLS algebra (detals omtted) gves us fact showg that degrees of freedom, s ch-square dstrbuted wth Fact ˆ ~ e E ( df ) E df ece, defg the mea sum of squa duals as MS df ( ), we have obtaed a ud estmator of, (7) MS ( ˆ df Q ) df (Note cotrast that the mle estmator s ˆ (show the appedx).) Fact 3 ad are depedet rv s. Also has smlar propertes as, amely that ~ mplyg that MS s a ud estmator of. But ths s true oly (!) f the hypothess that all the regso coeffcets (excludg the costat term) s zero (.e., : ( trctos), whch s the same as sayg that oe of the explaatory varables have explaatory power). If s ot true, the the OLS algebra (detals omtted) shows that E( MS ). Thus, comparg wth MS MS gves formato o - leadg to a F-test. The test statstc s F MS MS whch, uder, s F-dstrbuted wth df ad df degrees of freedom, ad we reject f F s suffcetly large. Ths test s always reported to the rght of the Source table ad s tae as a gree lght for dog the regso f t leads to rejecto of. If the test
7 does ot reject, t s terpreted as that there s too lttle formato (too small ) the data to fd ay effects amog the X s o the pose, Y. All the formato facts,,3 s summarzed the Source table 3 costructed as follows, (8) The Source table Source df MS=/df Model Resdual Total df MS df MS ( ) tot Y Y MS tot The Source table for the model () the example - together wth the dagostc formato to the rght - became (9) The Source table for the model () Source df MS Number of obs = -------------+------------------------------ F( 3, 6) = 68.9 Model 578488.74 3 9869.58 Prob > F =. Resdual 447637.457 6 7977.34 R-squa =.98 -------------+------------------------------ Adj R-squa =.947 Total 63446. 9 383.484 Root MSE = 67.6 Accordg to ths, the estmate of the error varace,, s 7 977.484. The square root of ths (67.6) s the estmate of ad s gve as Root MSE to the rght. The F-test for the (cosstg of 3 trctos) s at the rght ad has a p-value., dcatg that the (3) explaatory varables have explaatory power, so t maes sese to cotue the aalyss. R-squa s smply tot ad shows that 9.8% of the varato the data of Y s explaed by the 3 X s the model. Also the adjusted R-square s a dagostc tool. If the dfferece betwee the two R- squa s substatal, ths s a sg that too may explaatory varables have bee cluded the model relato to the umber of observatos (). (I the extreme case, for example, that we clude X s the model, we get ad R- squa =, ad the regso aalyss collapses completely,.e., there s o formato at all the data for such a model.) I the pet example there s o dager of such a possblty sce both values are qute close. 3 Ths source table repet a regso model wth a costat term ( ). If the regso fucto cotas X s oly wthout a costat term, the source table s slghtly dfferet. The ( ) tot Y, df, df, ad df. Otherwse, the same. tot
8 4 The recpe for F-testg of regso coeffcets The Model s as () () Y x x e,,, where the { x ;,,, ad j,,, } are cosde fxed umbers ad repet j observatos of explaatory varables, X, X,, X (see justfcato the appedx of the lecture ote o cto). For the error terms we assume, e, e,, e are d ad ormally dstrbuted, e N. ~ (, ) The uced Model We wat to test a ull hypothess cosstg of s (lear) trctos o,,,. Whe the trctos are lear, the model uder ca be expsed as a regso model (called the uced model ) wth p regsor varables some of whch may be dfferet from the X s (see the extra exercse the semar wee 47 for a example) ad p regso parameters, (,,, p ), (wth a costat term f pet), where p. Let, deote the dual sum of squa ( ) for the model ad the uced model pectvely ad the corpodg degrees of freedom ( the case that a costat occurs both the ad the uced model otherwse, see footote 3), df - - ad df p. The lelhood rato prcple tells us (see the appedx) that we should compare ad to test the uced model agast the model. Ths s exactly what the F-test does. The matrx OLS algebra (detals omtted) gves us what we eed for the F-test fact 4: Fact 4 The rv s true, the ad are depedet, ad, f (the uced model) s ( ) s ch-square dstrbuted wth degree of freedom (equal to the expected value) equal to s df df (vald geeral wth or wthout costat terms the two models). ece, ( ) s s a ud estmator of f s true, ad, as ca be prove, has expectato of freedom f s wrog. Sce, ay case, df, we get our F test statstc s ch-square wth degree ( ) / s ( ) / ( s) / df / ( df ) F, whch, accordg to the costructo secto, s F dstrbuted wth s df df ad
9 df degrees of freedom f s true. If s wrog, the F teds to get larger, so we reject f F s suffcetly large. I other words, the recpe of the F-test s as follows: () Recpe for the F-test of the uced model agast the model Ru two regsos, oe for the model ad oe for the uced. Pc out the dual sums of squa ( ad ) from the two source tables. Pc out the dual degrees of freedom ( df ad df ) from the two source tables ad calculate the umber of trctos to be tested, s df df. Calculate the F statstc, ( ) / s F, ad reject f F s larger tha the / df upper percetle the F( s, df ) dstrbuto (corpodg to the level of sgfcace, ). Or calculate the p-value, P ( ) F F obs (usg e.g., the F.DIST fucto Excel or a smlar fucto Stata). Example of testg structural brea descrbed the troducto. Full model Y x d 3dx e where e, e,, e ~ d wth e N ~ (, ) Stata output model Source df MS Number of obs = -------------+------------------------------ F( 3, 6) = 68.9 Model 578488.74 3 9869.58 Prob > F =. Resdual 447637.457 6 7977.34 R-squa =.98 -------------+------------------------------ Adj R-squa =.947 Total 63446. 9 383.484 Root MSE = 67.6 M Coef. Std. Err. t P> t [95% Cof. Iterval] -------------+---------------------------------------------------------------- D 639.755 83.3 5.79. 39.33 4.78 DX -.745789.5758-4.8. -.3958499 -.5338 XM.74643.459396 5.97..768768.37658 _cos 86.55 5.384.8.45-37.493 39.6594 Reduced model ( ) Y x e where e, e,, e ~ d wth : 3 e N ~ (, )
Stata output uced model Source df MS Number of obs = -------------+------------------------------ F(, 8) = 59.65 Model 478763.87 478763.87 Prob > F =. Resdual 44485.33 8 867.585 R-squa =.768 -------------+------------------------------ Adj R-squa =.7553 Total 63446. 9 383.484 Root MSE = 83.3 M Coef. Std. Err. t P> t [95% Cof. Iterval] -------------+---------------------------------------------------------------- D 67.667 38.437 7.7. 777.75 358.6 _cos 656 75.798 8.66. 496.999 85.8 The relevat quattes are 447 637.457 df 6 444 85.33 df 8 No. of trctos uder : s df df ( ) / s ( 444 85.33 447 637.457) / F 7.8 / df 447 637.457 /6 F ~ F (,6) uder. P-value (usg F.Dst Excel): P F F P F 5 ( ) ( 7.8) 8.49. obs so the evdece for a structural brea as defed s strog,.e., the uced model s rejected. 5. Specfcato test of same varace the two come groups The F-test secto 4 assumes costat error varace,, both groups. If ths assumpto s wrog, the F-test secto 4 s valdated. It s therefore atural to as f there s ay evdece the data for doubtg the costat varace assumpto. For ths purpose we ca use aother F test whch ofte ca be used to compare the varaces two depedet groups. Let, be the error term varaces for the d group ad d group pectvely. We wat to test : agast : The F test s well suted for ths: Ru two regsos, oe for each group. Pc out the two MS, called MS ad MS pectvely, from the two rus ad form / the F statstc, F MS df, where df MS / df, df are the dual degrees of freedom the two groups. Note that MS ad MS must be depedet sce they come from two depedet groups.
Sce / ( df ) F V, where V ~ F( df, df ), t follows that / ( df) F ~ F( df, df ) f s true. The problem s two-sded, so we reject f F c or F c, where the crtcal values, c, c for level of sgfcace, are determed by P ( F c ) ad P ( F c ). Or calculate the p-value: the smallest of P ( F Fobs ) ad P ( F Fobs ). Stata output for the example Group D = Source df MS Number of obs = 4 -------------+------------------------------ F(, ) = 4.56 Model 99775.494 99775.494 Prob > F =. Resdual 956.56 4584.788 R-squa =.777 -------------+------------------------------ Adj R-squa =.757 Total 99 3 99399.3846 Root MSE = 56.8 M Coef. Std. Err. t P> t [95% Cof. Iterval] -------------+---------------------------------------------------------------- XM.74643.4364 6.37..84356.36893 _cos 86.55 98.7886.87.4-8.9857 3.4957 ---------------- ---- Group D = Source df MS Number of obs = 6 -------------+------------------------------ F(, 4) =. Model.389347.389347 Prob > F =.994 Resdual 56.95 4 3855.376 R-squa =. -------------+------------------------------ Adj R-squa = -.5 Total 563.333 5 354.6667 Root MSE = 95.33 M Coef. Std. Err. t P> t [95% Cof. Iterval] -------------+---------------------------------------------------------------- XM -.346.39897 -..994 -.844.48 _cos 76. 37.34 5.6.5 873.639 578.45 MS Test: F ~ F(, 4) uder. MS The crtcal values at the 5% level from table 5 bac Rce : P ( F c ).5 P ( F c ).975 c 8.75 P ( F c ).5 P.5.975 P F c F c 4. c.4 c 4. so we reject f F.4 or F 8.75.
MS 4584.788 Observed: Fobs.64 MS 3855.376 Cocluso: Do t reject. I other words: Our () model secto 4 passed the specfcato test, whch creases ts cblty. 6 Appedx The F-test as a lelhood rato test (optoal readg) Cosder the model () () Y E( Y ) e x x e,,,, where e, e,, e are d ad e N. Ths mples that Y, Y,, Y are depedet ad ~ (, ) Y N E Y. ~ ( ( ), ) for,,, The lelhood s (wrtg (,,, ) ) ( ( )) ( ) ye Y Q L(, ) f ( y, y,, y ;, ) e e ( ) ( ) Sce h( x) e x s a decreasg fucto, the, whatever the value of, the maxmum of L over s obtaed by mmzg Q( ),.e., whe s equal to the OLS ˆ. ece the mle ˆ s equal to the OLS estmator. We the fd the mle of by maxmzg ˆ l L(, ) l( ) l Q( ˆ ) wth pect to. ˆ l (, ) ( ˆ L Q ) gves the mle ˆ Q( ˆ ) 3. Substtutg ths the lelhood, we get the maxmum value (3) Q( ˆ ) ( ˆ Q ) ˆ ˆ L(, ˆ) e e e ( ) ( ) ( ) Q( ˆ ) Q( ˆ ) Now let deote the parameter set, ( ),, uder the model (), ad the parameter set, ( ),, uder the uced model secto 4. Let over ad pectvely. The lelhood rato (LR) the becomes L ad L be the maxmum lelhoods
3 ( ) e ˆ L Q( ˆ ) Q( ) L ( ˆ) Q e ( ) ˆ Q( ) The LR test tells us to reject the uced model ( ) f W l l suffcetly large, whch s the same as sayg that should be rejected f suffcetly large (sce the l-fucto s creasg), or f s suffcetly large. Ths s equvalet to rejectg f the F statstc, F s suffcetly large. The dstrbuto of F s ow exactly (as a s F-dstrbuto) uder o matter sample sze - cotrast to the geeral LR test whch s oly approxmately a Ch-square test (wth degree of freedom s) for large samples. s s