Chapter 15 Multiple Regression

Transcription

1 Chapter 5 Multple Regresson In chapter 9, we consdered one dependent varable (Y) and one predctor (regressor or ndependent varable) (X) and predcted Y based on X only, whch also known as the smple lnear regresson. In ths chapter we wll consder one dependent varable (Y) but a set of predctors (regressors or ndependent varables) (say p varables) and we want to predct Y based on the smultaneous knowledge of all p predctors, also known as multple lnear regresson. 5. Multple Lnear Regresson Regresson analyss s a statstcal tool for evaluatng the relatonshp of one or more ndependent varables X,, X n to a sngle or multple contnuous dependent varable Y. Here we wll consder Y =response varable, and x, x,, x p are p ndependent varables (also called the regressors or predctors). We want to predct Y based on the smultaneous knowledge of all p predctors. Example: Suppose we want to predct the success of a student n a graduate school (Y) based on the undergraduate GPA ( X ), GRE scores ( X ), number of courses taken n the maor dscplne ( X 3 ) and ratngs of the recommendaton leters ( X 4 ) etc. The Regresson Equaton Y = p p n β + β x + β x + + β x + e, =,,, () where Y =response (or dependent) varable and x, x,, x p are ndependent varables, β s the ntercept, the value of Y when x equals zero and β, β, β p, are the regresson coeffcents, e 's are random errors and are ndependently and dentcally (d ) dstrbuted wth E e ) = and ( ) = σ ( V. e

2 The (p+) parameters, β, β, β,, β p and σ are unknown and have to be estmated from data. Least Squares Estmaton of the Regresson Coeffcents th The ftted values = ˆ β ˆ ˆ ˆ + βx + βx + + β px p =,, n, yˆ th The resdual values are eˆ = y yˆ Prncple: ``Dfferences (squared dfferences) between the observed values y and the ftted (predcted) values, ŷ, should be mnmzed". The least squares estmators are those values ˆβ of β and mnmze the resdual sum of squares (RSS): That s n βˆ of β that ˆ ˆ ˆ ˆ RSS = [ y ( β + βx + βx + + β px p )]. () = when evaluated at ˆβ and βˆ ( =,,...p). Here responses. y s are observed One method of mnmzng () s to dfferentate wth respect to β and β, sets the dervatves equal to zero, and solve the resultng equatons. The calculatons requred to estmate the β become more cumbersome as the number of predctors (regressors) ncrease, we wll not dscuss the estmaton technques here. Instead, we wll consder the real data and use computer software SPSS to estmate them. After estmatng the regresson coeffcents (parameters), the ftted regresson lne wll be = ˆ β ˆ ˆ ˆ + βx + βx + + β pxp =,, n, = b + b x + b x + + b x =,,, yˆ yˆ p p n

3 Example (page 57): There has been an ongong debate n ths coutry about what we can do to mporved the qualty of prmary and secondary eduaton. It s generalry assumed that spendng more money on eduacton wl lead to better prepared students. However, that s ust as assumpton. Gruber (999) addressed that queston by collectng data for each of the 5 states n USA. She recorded the amount of educaton, the people/teacher raton (PTrato), average teachers salary, the percentage of stdents n that state take the SAT exams (PctSAT) and the combned SAT score. Dr. Howell () has added the percentage of students n each state talkng the ACT (PctACT) and the mean ACT score for that state. The combned SAT score wll be used as dependent varable and remanng varables are consdered as ndependent varables or regressors. The data are gven n the followng Table 5.. Table 5. Data on performance versus expendture on educaton state Expend PTrato Salary PctSAT SAT PctACT ACT Alabama Alaska Arzona Ark Calf Col Conn Del Florda Georga Hawa Idaho Illno Indana Iowa Kansas Ken Lousa Mane Marylan Mass Mch

4 Mnn Mssss Mssour Montana Neb Nev NH NJ NM NY NC ND Oho Ok Oregon Penn RI SC SD Tenn Texas Utah Vermont Va Wash WV Wsc Wyomng

5 Usng SPSS, some plots of the data (hstograms and qq plots) are gven below. 5

6 6

7 Two VarableRelatonshps Table 5.: Correlatons between varables Correlatons Expend PTrato Salary PctSAT SAT LogPctSAT ACT PctACT Expend Pearson Correlaton -.37 **.87 **.593 ** -.38 **.56 ** -.5 **.38 ** Sg. (-taled) N PTrato Pearson Correlaton -.37 ** Sg. (-taled) N Salary Pearson Correlaton.87 ** ** -.44 **.63 ** **.355 * Sg. (-taled) N PctSAT Pearson Correlaton.593 ** ** **.96 ** **.6 Sg. (-taled) N SAT Pearson Correlaton -.38 ** ** ** -.96 **.877 **.69 LogPctS AT Sg. (-taled) N Pearson Correlaton.56 ** **.96 ** -.96 ** **.3 Sg. (-taled) N ACT Pearson Correlaton -.5 ** ** **.877 ** ** -.43 Sg. (-taled) N PctACT Pearson Correlaton.38 ** * Sg. (-taled) N **. Correlaton s sgnfcant at the. level (-taled). And 7

8 Correlatons Expend PTrato Salary PctSAT SAT LogPctSAT ACT PctACT Expend Pearson Correlaton -.37 **.87 **.593 ** -.38 **.56 ** -.5 **.38 ** Sg. (-taled) N PTrato Pearson Correlaton -.37 ** Sg. (-taled) N Salary Pearson Correlaton.87 ** ** -.44 **.63 ** **.355 * Sg. (-taled) N PctSAT Pearson Correlaton.593 ** ** **.96 ** **.6 Sg. (-taled) N SAT Pearson Correlaton -.38 ** ** ** -.96 **.877 **.69 LogPctS AT Sg. (-taled) N Pearson Correlaton.56 ** **.96 ** -.96 ** **.3 Sg. (-taled) N ACT Pearson Correlaton -.5 ** ** **.877 ** ** -.43 Sg. (-taled) N PctACT Pearson Correlaton.38 ** * Sg. (-taled) N **. Correlaton s sgnfcant at the. level (-taled). And *. Correlaton s sgnfcant at the.5 level (-taled). Sg.(-taled) mples p-value for the test :ρ = vs H : ρ. The nterestng fact H that the expend and SAT varables are negatvely related. 8

9 The Multple Regresson Equaton One predctor: (SAT on Expend) Descrptve Statstcs Mean Std. Devaton N SAT Expend \ Correlatons SAT Expend Pearson Correlaton SAT Expend Sg. (-taled) SAT..3 Expend.3. N SAT 5 5 Expend 5 5 Model R R Square Model Summary Adusted R Square Std. Error of the Estmate.38 a a. Predctors: (Constant), Expend ANOVA b Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total a. Predctors: (Constant), Expend b. Dependent Varable: SAT Coeffcents a Unstandardzed Coeffcents Standardzed Coeffcents 95.% Confdence Interval for B Model B Std. Error Beta t Sg. Lower Bound Upper Bound (Constant) Expend

10 Two predctors: (SAT on Expend and LogPctSAT) Descrptve Statstcs Mean Std. Devaton N SAT Expend LogPctSAT Correlatons SAT Expend LogPctSAT Pearson Correlaton SAT Expend LogPctSAT Sg. (-taled) SAT..3. Expend.3.. LogPctSAT... N SAT Expend LogPctSAT Model Summary Model R R Square Adusted R Square Std. Error of the Estmate.94 a a. Predctors: (Constant), LogPctSAT, Expend ANOVA b Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total a. Predctors: (Constant), LogPctSAT, Expend

11 Unstandardzed Coeffcents Coeffcents a Standardze d Coeffcents 95.% Confdence Interval for B Lower Upper Model B Std. Error Beta t Sg. Bound Bound (Constant) Expend LogPctSAT a. Dependent Varable: SAT The regresson coeffcents are b = 47.3, b =.3, b = 78.5 The ftted regresson model (or lne) s Ŷ = (Expend) 78.5(LogPctSAT) The value of 47.3 s the ntercept, denoted by b (also known as constant). Ths s the predcted value of SAT f both Expand and LogPctSAT were. Ths ftted model can be used to obtan a predcted value of Y (SAT) for any specfed values of the regressors/predctors. For example, for the state Colorado as an example, our predcted mean SAT score would be Ŷ = (Expend) 78.5(LogPctSAT) = (5.443) 78.5(Log(9)) = The actual mean for Colrado was 98, that means, we have under estmated the mean and resdual error s =35.648

12 Interpretaton of estmated regresson coeffcents: b =.3, mples, f LogPctSAT helds constant, the predcted value of Y would be hgher by.3 unt for every one unt ncrease n Expend. b = 78.5, mples, f expend helds constant, the predcted value of Y would be lower by 78.5 unt for every one unt ncrease n LogPctSAT. 5. Usng Addtonal Predctors Regresson Model wth three predctors: Expend, and LogPctSAT & PTraro ANOVA b Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total a. Predctors: (Constant), PTrato, LogPctSAT, Expend b. Dependent Varable: SAT Model Unstandardzed Coeffcents Standardzed Coeffcents Coeffcents a 95.% Confdence Interval for B Collnearty Statstcs B Std. Error Beta t Sg. Lower Bound Upper Bound Tolerance VIF (Constant) Expend LogPctSAT PTrato a. Dependent Varable: SAT The ftted regresson model s Ŷ = (Expend) (LogPctSAT) -.74 PTrato

13 We note that Expand and LogPctSAT are sll sgnfcant at % LOS. However, PTrato s not sgnfcant. VIF and Tolerence: VIF and Tolerance are defned respectvley as VIF = and Tolerence = R Snce all VIFs are less then, we assume that there s no any multcollnearty n the data set. VIF Standardzed regresson coeffcents (SRC) One common mstake s to treat the relatve magntudes of the b as an ndex of the relatve mportance of the ndvdual predctors. By ths logc (mstake), one mght conclude that LocPctSAT s a more mportant predctor than s Expend or PTrato, because ts coeffcent s apprecable larger. Although t mght actually be the case that Expend s a more mportant predctor than LogPctSAT or PTrato, one can not make such a comparson based on the regresson coeffcents. The relatve magntudes of the coeffcents are n part a functon of the standard devatons of the correspondng varables. The standardzed regresson coeffcents are obtaned based on the standardzed varables. The standardzed varable have mean zero and varable. Moreover, they are unt free. The standardzed regresson coeffcents are denoted by beta and obtaned by SPSS are as follows: β =., β =.4, β 3 =. About the nterpretaton of the SRC, please read the last paragraph of page 58. The relatonshp between unstandardzed and standardzed regresson coeffcents are as follows β = b s s 3

14 where s s the standard devaton of Y varable and s s the standard devaton of the correspondng regressor. Descrptve Statstcs N Mnmum Maxmum Mean Std. Devaton Expend LogPctSAT PTrato SAT Vald N (lstwse) 5 For nstance, the SRC for Expend β =.665 Smlalry, the SRC for LogPctSAT β = = 74.8 = Standard errors and Tests of regresson coeffcents After estmatng the regresson coeffcents t s mportant to test the statstcal sgnfcance. If the regresson coeffcent relatng Expend to SAT s not sgnfcantly dfferent from zero, then Expend wll serve no useful purpose n the predcton of SAT. Testng Regresson Coeffcents: Suppose we want to test the followng hypothess H : β = β vs H : β β The test statstc s t = b β s b where b s the estmated values of the regresson parameters β, standadrd error of b e. SE( b )= s. b s b s the 4

15 Reect the null hypothess, f t > tα /, n p Example: Consder three predctors model and suppose we want to test the hypothess s that Epend has no effect on SAT. That s, The test statstc s Reect the null hypothess, f H : β = vs H : β t = b s b.665 = t > t.5,46 = 3.3 =. Snce, t = 3.3 >., we do reect the null hypothess at 5% LOS (The p-value s. whch s less then.5). Thus the regressor Expend has sgnfcant effect on Y (SAT). In other words, the predcted value of Y ncreases wth ncreasng Expend, and ths Expend makes a sgnfcant contrbuton to the predcton of SAT. The SPSS output have both tests or confdence ntervals for the regresson coeffcents. Confdence Interval: The (-α)% CI for β are obtaned as follows: = ± α /, n p CI b t SE( b ) Example: Consder three predctors model and suppose we want to fnd 95% CI for β. The 95% CI for β s obtaned as follows: CI = b ± t.5,46 SE( b ) = ± = ± 9.66 = [ , 69.37] From SPSS output, we observed that the 95% confdence nterval for β s [-87.57, ]. Thus we are 95% confdent that the populaton regresson coeffcent β wll fall between and (My calculatons are dfferent than SPSS due to roundng error) Snce ths nterval does not cover 5

16 , we reect the null hypothess that H : β = at 5% level of sgnfcance (LOS) (From SPSS, the p-value=.). 5.4 Resdual Varance The resduals are eˆ = y yˆ =,, n The resdual varance s defned as ˆ σ = MS resdual = MS error = n = ( y yˆ ) n p The square root of the resdual varance s known as standard error of estmate. For three predctors regresson model, the resdual varance s ˆ σ = = ˆ σ = = 6. 6

17 5.5 Dstrbuton Assumptons Some necessary Assumptons: The mean of Y s a lnear functon of x 's. The Y have a common varance σ, for all x 's. The e d N(, σ ). Normal P-P Plot of Regresson Standardzed Resdual. Dependent Varable: Overall.8 Expected Cum Prob Observed Cum Prob.8. 7

18 5.6 The multple correlaton coeffcent R SS = y SS SS For three predctors regresson model y resdual SS = SS resdual y R = =.34 =.8866 Note that SS y = ( y y). The nterpretaton of R -square s extremely mportant. If R =.8866, then about 89% of the varablty n Y has been explaned by predctors (three ndependent varables). A relable unbased estmator to adusted R-square estr estr ( = ( R )( n ) = n p.8866)(49) = Ths value s close to ``Adusted R-square" prnted by SPSS. 8

19 Testng the sgnfcance of R Suppose we are testng: H R = vs H : R : The test statstc s F and s defned as = R / p ( n p ) R F = ( R ) /( n p ) p( R ) Under the null hypothess, F s dstrbuted as F wth p and ( n p ) degrees of freedoms. Test the hypothess that, H R = vs H : R For ths data, n=5, p=3, =.8866 : R. F = (5 3 )(.8866) 3( (.8866)) = 9.88 RR: We wll reect the null hypothess f F > F.5,3, 46 =.79 (approxmate) Snce F = 9.88 >.79, we wll reect the null hypothess at 5% LOS. Thus conclude that we can predct at better-than-chance levels. Model Summary b Std. Error Change Statstcs Mode R Adusted R of the R Square F Sg. F l R Square Square Estmate Change Change df df Change.94 a a. Predctors: (Constant), PTrato, LogPctSAT, Expend and Dependent Varable=SAT For proper sample sze, please read page 533 There are several suggessons: () n () n p+4 () More s better 9

20 5.8 Partal and Sempartal Correlaton Partal Correlaton: Partal correlaton ( r. ) between two varables wth one or more varables partalled out of both X and Y. It s the correlaton between two sets of resduals formed from the predcton of the orgnal varable by one or more varables. From SPSS output, we have partal correlaton between Y and X s.438, whch represents the ndependent contrbuton of X towards the predcton of Y partalled out X, X. 3 Sempartal Correlaton: Sempartal correlaton ( r (.) ) s the correlaton between the crteron (dependent varable) and a partalled predctor varable. In ths case the sempartal correlaton between Y and the resdual ( X ˆ X = Xr ) of X predcted on X. Please see page 535 to 536 for partal and sempartal correlaton (also called part correlaton). SPSS output for partal and sempartal correlaton Model Unstandardzed Coeffcents Coeffcents a Standardzed Coeffcents B Std. Error Beta t Sg. Zero-order Partal Part (Constant) Correlatons Expend LogPctSAT PTrato a. Dependent Varable: SAT 5. Regresson Dagnostcs Dstance: The resdual, y yˆ, whch s useful to fnd potental outlers n the dependent varable ( y ) Leverage ( h ): Leverage s useful n dentfyng potental outlers n the ndependent varables X, X,, X p. The leverage s denoted by ``hat dag" and measures the degree to whch a case s unusual wth respect to the

21 predctor varable n (that s h ) wth a mean of predctors. X. The possble value of leverage vary between n and ( p + ), where p s the number of n Influence: Influence combnes dstance and leverage to dentfy unusually nfluental observatons. An observaton s nfluental f the locaton of the regresson surface would change markedly dependng on the presence or absence of that observaton. The most common measure nfluence s known as Cook's D (Cook's dstance). It s a functon of the sum of squared changes n b that would occur f the th observaton were removed from the data and the analyss rerun. If the data met the underlyng assumptons, we would expect the values of Y to be normally dstrbuted about the regresson lne. For more on regresson dagnostcs, please read page 539 to 544. Consder the followng data (page 54) Y X PRED Rstudent Leverage Cook's

22 Regresson model of SAT on Expend & LogPctSAT Resdual Rstudent Cook's D Leverage

23 Dagnostc Plots Regresson model of SAT on Log(PctSAT) & Expend 3

24 Regresson model of SAT on PctSAT)& Expend 4

25 Subset selecton Consder the full model Y β + β x + + β x + e, =,,, (3) = p p n and the partal model (reduced model) Y = + p q, p q n β β x + + β x + e, =,,, (4) The last q( q p ) terms are dropped n the partal or reduced model by settng the correspondng coeffcents equal to zero. To determne whether the ft of the full model s sgnfcantly better than that of the partal model we test H : β p q+ = = β p = vs H : At least one of β p q+,, β p (5) Reect the null hypothess n ( ) f ( SSE p q SSE p )/ q F = > Fq, n p, α, (6) SSE /[ n ( p + )] p where SSE p q s the error sums of squares for the partal (reduced) model and SSE p s the error sums of squares of the full model. Example: Suppose we want to ft a model that contans Expend, LogPctSAT, PTRato and Salary and have the followng ANOVA. ANOVA b Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total a. Predctors: (Constant), Salary, PTrato, LogPctSAT, Expend b. Dependent Varable: SAT 5

26 Now we want to ft a model that contans Expend and LogPctSAT and have the followng ANOVA. ANOVA b Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total a. Predctors: (Constant), LogPctSAT, Expend b. Dependent Varable: SAT Now we want to test the followng hypotheses: H β = β = vs H : At least one of β β (7) : SSE(reduced or partal model)= SSE(full model) = Reect the null hypothess n (7) f F = ( )/ 368.4/45 = =.456. Snce the calculated F value s less than F,45,.5 = 3. (approx.), we do not reect the Null Hypothess n ( 7 ). We dd not do a better ob for predctng SAT scores wth addtonal two predctors. For the fnal model, we may consder only Expend and LogPctSAT as explanatory varables. 5.: Constructng a Regresson Equaton The maor problem n multple regresson analyss s to ft a proper regresson lne or to fnd a set of useful predctors to be ncluded n the fnal model. To fnd a better model or to ft a fnal model to predct the future response, one mght need to understand the followng mportant terms/tools: 6

27 Cross correlaton: The cross correlaton s denoted by R X, whch measures the correlaton between one predctor and all other predctors n the model. Tolerance: Tolerance refers to the degree to whch one predctor can tself be predcted by the other predctors n the model. The tolerance s defned as Tolerance R = X where R X s the correlaton (also called cross correlaton) between one predctor wth other predctors. In the SPSS output, SAT s the crteron (dependent) and Expend, LogPactSAT, Ptrato and Salary as the predctors. The tolerance s equal to.6. Thus R X = (.6) =.894 The squared multple correlaton predctng the Exam predctor from other three predctors s.894. Sngular: In the extreme case where one predctor can be perfectly predcted from others, we wll have what s called a sngular covarance matrx and most program wll stop wthout generatng model. Varance nflaton factor (VIF): The recprocal of tolerance s called VIF. That s, VIF = Tolerance The VIF refers to the degree to whch the standard error of b s ncreased because X s correlated wth other predctors. We want to have stable regresson coeffcents, and therefore we want varable wth low VIF and hgh tolerance. 7

28 Model Unstandardzed Coeffcents Coeffcents a Standardzed Coeffcents Collnearty Statstcs B Std. Error Beta t Sg. Tolerance VIF (Constant) Expend LogPctSAT PTrato Salary a. Dependent Varable: SAT Model Unstandardzed Coeffcents Coeffcents a Standardzed Coeffcents Collnearty Statstcs B Std. Error Beta t Sg. Tolerance VIF (Constant) LogPctSAT PTrato Salary a. Dependent Varable: Expend Model R R Square Model Summary Adusted R Square Std. Error of the Estmate.946 a a. Predctors: (Constant), Salary, PTrato, LogPctSAT Selecton Methods There are many ways to construct some sort of ``optmal" regresson equaton from a large set of varables. Ths secton s mportant and wll dscuss several of these approaches. Best subset selecton The best model can be selected n several ways. () The magntude of R : Predctors wth hghest R () The magntude of MS resdual : Predctors wth lowest MS resdual RSS p (3) Mallow's C p : Predctors wth smallest C p, where C p = ( n p) and MSE RSS denotes the resdual sum of squares from a model contanng the p p 8

29 parameters. n (4) PRESS (predcted resdual sum of squares)= ( y ˆ ) = y, where ŷ s the predcted value from a data set that all ncludes except th observaton. Backward Elmnaton (page 548) Stepwse regresson (page 549) Forward selecton (page 549) Usng stepwse (SPSS), we have the followng fnal model when we consder all regressors Model Model R R Square.96 a b a. Predctors: (Constant), LogPctSAT b. Predctors: (Constant), LogPctSAT, PctACT ANOVA c Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total Regresson b Resdual Total a. Predctors: (Constant), LogPctSAT b. Predctors: (Constant), LogPctSAT, PctACT c. Dependent Varable: SAT Coeffcents a Unstandardzed Coeffcents Model Summary Standardzed Coeffcents Adusted R Square Std. Error of the Estmate Collnearty Statstcs B Std. Error Beta t Sg. Tolerance VIF (Constant) LogPctSAT (Constant) LogPctSAT PctACT

30 ANOVA c Model Sum of Squares df Mean Square F Sg. Regresson a Resdual Total Regresson b Resdual Total a. Predctors: (Constant), LogPctSAT b. Predctors: (Constant), LogPctSAT, PctACT a. Dependent Varable: SAT Cross-Valdaton (page 549) Mssng Observatons (page 55) 5.5 Logstc Regresson Logstc regresson s a technque for fttng a regresson surface to data n whch the dependent varable s a dchotomy. For the hosptal patents, we want to predct response to treatment, where we mght code survvors as and don't survve as. In psychology we mght class clents as Improved () or not-mproved (). Usng SPSS, we have analyzed the data n Exercse 5.3 to descrbe the logstc regresson. Exercse 5.3, page 578: The data set Harass.dat contans slghtly modfed data on 343 cases created to replcate the results of a study of sexual harassment by Brooks and Perot (99). The dependent varable s whether or not the subects reported ncdents of sexual harassment, and the ndependent varables are, n order, Age, Martal Status, Femnst deology, Frequency of the behavor, Offensveness of the behavor and whether or not t was reported. Usng SPSS examne the lkelhood that a subect wll report sexual harassment on the bass of the ndependent varables. The SPSS output are gven n the followng page/ 3

31 Step Constant Logstc Regresson Varables n the Equaton B S.E. Wald df Sg. Exp(B) Model Summary Step - Log Cox & Snell Nagelkerke lkelhood R Square R Square a.98.3 a. Estmaton termnated at teraton number 4 because parameter estmates changed by less than.. Hosmer and Lemeshow Test Step Ch-square df Sg Step a age marstat fem freq offensuv Constant Varables n the Equaton B S.E. Wald df Sg. Exp(B) a. Varable(s) entered on step : age, marstat, fem, freq, offensuv. Correlaton Matrx Step Constant age marstat fem freq offensuv Constant age marstat fem freq offensuv

32 age marstat fem freq offensuv reported PRE_ COO_ LEV_ RES_ LRE_

33