428 CHAPTER 12 MULTIPLE LINEAR REGRESSION

Transcription

1 48 CHAPTER 1 MULTIPLE LINEAR REGRESSION Table 1-8 Team Wis Pts GF GA PPG PPcT SHG PPGA PKPcT SHGA Chicago Miesota Toroto St. Louis Detroit Edmoto Calgary Vacouver Wiipeg Los Ageles Philadelphia NY Isladers Washigto NY Ragers New Jersey Pittsburgh Bosto Motreal Buffalo Quebec Hartford SHG PPGA PKPcT SHGA Short-haded goals scored durig the seaso. Power play goals agaist. Pealty killig percetage. Measures a team s ability to prevet goals while its oppoet is o a power play. Oppoet power play goals divided by oppoet s opportuities. Short-haded goals agaist. Fit a multiple liear regressio model relatig wis to the other variables. Estimate ad fid the stadard errors of the regressio coefficiets Cosider the liear regressio model Y i 0 1 1x i1 x 1 1x i x i where x 1 g x i1 ad x g x i. (a) Write out the least squares ormal equatios for this model. (b) Verify that the least squares estimate of the itercept i this model is ˆ 0 g y i y. (c) Suppose that we use y i y as the respose variable i the model above. What effect will this have o the least squares estimate of the itercept? 1- HYPOTHESIS TESTS IN MULTIPLE LINEAR REGRESSION I multiple liear regressio problems, certai tests of hypotheses about the model parameters are useful i measurig model adequacy. I this sectio, we describe several importat hypothesis-testig procedures. As i the simple liear regressio case, hypothesis testig requires that the error terms i i the regressio model are ormally ad idepedetly distributed with mea zero ad variace Test for Sigificace of Regressio The test for sigificace of regressio is a test to determie whether a liear relatioship exists betwee the respose variable y ad a subset of the regressor variables x 1, x, p, x k. The

2 1- HYPOTHESIS TESTS IN MULTIPLE LINEAR REGRESSION 49 appropriate hypotheses are H 0 : 1 ### k 0 H 1 : j Z 0 for at least oe j (1-17) Rejectio of H 0 : 1 p k 0 implies that at least oe of the regressor variables x 1, x, p, x k cotributes sigificatly to the model. The test for sigificace of regressio is a geeralizatio of the procedure used i simple liear regressio. The total sum of squares SS T is partitioed ito a sum of squares due to regressio ad a sum of squares due to error, say, SS T SS R SS E Now if H is true, SS R 0 : 1 p k 0 is a chi-square radom variable with k degrees of freedom. Note that the umber of degrees of freedom for this chi-square radom variable is equal to the umber of regressor variables i the model. We clso show the SS E is a chi-square radom variable with p degrees of freedom, ad that SS E ad SS R are idepedet. The test statistic for H 0 : 1 p k 0 is F 0 SS R k SS E 1 p MS R MS E (1-18) We should reject H 0 if the computed value of the test statistic i Equatio 1-18, f 0, is greater tha f,k, p. The procedure is usually summarized i alysis of variace table such as Table 1-9. We ca fid a computig formula for SS E as follows: SS E a 1 y i ŷ i a e i e e Substitutig e y ŷ y X ˆ ito the above, we obtai SS E y y ˆ X y (1-19) Table 1-9 Aalysis of Variace for Testig Sigificace of Regressio i Multiple Regressio Source of Degrees of Variatio Sum of Squares Freedom Mea Square F 0 Regressio SS R k MS R MS R MS E Error or residual SS E p MS E Total SS T 1

3 430 CHAPTER 1 MULTIPLE LINEAR REGRESSION A computatioal formula for SS R may be foud easily. Now sice SS T g 1g y i y y 1 g y i y i, we may rewrite Equatio 1-19 as or SS E y y Therefore, the regressio sum of squares is ˆ y SS E SS T SS R SS R ˆ y (1-0) EXAMPLE 1-3 We will test for sigificace of regressio (with 0.05) usig the wire bod pull stregth data from Example 1-1. The total sum of squares is SS T y y 7, The regressio sum of squares is computed from Equatio 1-0 as follows: SS R ˆ X y 7, ad by subtractio SS E SS T SS R y y ˆ X y The aalysis of variace is show i Table To test H 0 : 1 0, we calculate the statistic f 0 MS R MS E 5.35 Sice f 0 f 0.05,, 3.44 (or sice the P-value is cosiderably smaller tha = 0.05), we reject the ull hypothesis ad coclude that pull stregth is liearly related to either wire legth or die height, or both. However, we ote that this does ot ecessarily imply that the

4 1- HYPOTHESIS TESTS IN MULTIPLE LINEAR REGRESSION 431 Table 1-10 Test for Sigificace of Regressio for Example 1-3 Source of Degrees of Variatio Sum of Squares Freedom Mea Square f 0 P-value Regressio E-19 Error or residual Total relatioship foud is ppropriate model for predictig pull stregth as a fuctio of wire legth ad die height. Further tests of model adequacy are required before we ca be comfortable usig this model i practice. Most multiple regressio computer programs provide the test for sigificace of regressio i their output display. The middle portio of Table 1-4 is the Miitab output for this example. Compare Tables 1-4 ad 1-10 ad ote their equivalece apart from roudig. The P-value is rouded to zero i the computer output. R ad Adjusted R We may also use the coefficiet of multiple determiatio R as a global statistic to assess the fit of the model. Computatioally, R SS R SS T 1 SS E SS T (1-1) For the wire bod pull stregth data, we fid that R SS R SS T Thus the model accouts for about 98% of the variability i the pull stregth respose (refer to the Miitab output i Table 1-4). The R statistic is somewhat problematic as a measure of the quality of the fit for a multiple regressio model because it always icreases whe a variable is added to a model. To illustrate, cosider the model fit to wire bod pull stregth data i Example This was a simple liear regressio model with x 1 wire legth as the regressor. The value of R for this model is R Therefore, addig x y die height to the model icreases R by , a very small amout. Sice R always icreases whe a regressor is added, it ca be difficult to judge whether the icrease is tellig us aythig useful about the ew regressor. It is particularly hard to iterpret a small icrease, such as observed i the pull stregth data. May regressio users prefer to use djusted R statistic: R adj 1 SS E 1 p SS T 1 1 (1-) Because SS E 1 p is the error or residual mea square ad SS T 1 1 is a costat, R adj will oly icrease whe a variable is added to the model if the ew variable reduces the error mea square. Note that for the multiple regressio model for the pull stregth data R adj (see the Miitab output i Table 1-4), whereas i Example 11-8 the adjusted R for the oe-variable model is R adj Therefore, we would coclude that addig x die height to the model does result i a meaigful reductio i uexplaied variability i the respose.

5 43 CHAPTER 1 MULTIPLE LINEAR REGRESSION The adjusted R statistic essetially pealizes the aalyst for addig terms to the model. It is a easy way to guard agaist overfittig, that is, icludig regressors that are ot really useful. Cosequetly, it is very useful i comparig ad evaluatig competig regressio models. We will use R adj for this whe we discuss variable selectio i regressio i Sectio Tests o Idividual Regressio Coefficiets ad Subsets of Coefficiets We are frequetly iterested i testig hypotheses o the idividual regressio coefficiets. Such tests would be useful i determiig the potetial value of each of the regressor variables i the regressio model. For example, the model might be more effective with the iclusio of additioal variables or perhaps with the deletio of oe or more of the regressors presetly i the model. Addig a variable to a regressio model always causes the sum of squares for regressio to icrease ad the error sum of squares to decrease (this is why R always icreases whe a variable is added). We must decide whether the icrease i the regressio sum of squares is large eough to justify usig the additioal variable i the model. Furthermore, addig a uimportat variable to the model cctually icrease the error mea square, idicatig that addig such a variable has actually made the model a poorer fit to the data (this is why R adj is a better measure of global model fit the the ordiary R ). The hypotheses for testig the sigificace of ay idividual regressio coefficiet, say j, are H 0 : j 0 H 1 : j 0 (1-3) If H 0 : j 0 is ot rejected, this idicates that the regressor x j ca be deleted from the model. The test statistic for this hypothesis is T 0 ˆ ˆ j j ˆ C jj se1 ˆ j (1-4) where C jj is the diagoal elemet of 1X X 1 correspodig to ˆ j. Notice that the deomiator of Equatio 1-4 is the stadard error of the regressio coefficiet ˆ j. The ull hypothesis H 0 : j 0 is rejected if 0 t 0 0 t, p. This is called a partial or margial test because the regressio coefficiet ˆ j depeds o all the other regressor variables x i (i j) that are i the model. More will be said about this i the followig example. EXAMPLE 1-4 Cosider the wire bod pull stregth data, ad suppose that we wat to test the hypothesis that the regressio coefficiet for x (die height) is zero. The hypotheses are H 0 : 0 H 1 : 0

6 1- HYPOTHESIS TESTS IN MULTIPLE LINEAR REGRESSION 433 The mai diagoal elemet of the 1X X 1 matrix correspodig to ˆ is C , so the t-statistic i Equatio 1-4 is t 0 ˆ ˆ C Note that we have used the estimate of reported to four decimal places i Table Sice t 0.05,.074, we reject H 0 : 0 ad coclude that the variable x (die height) cotributes sigificatly to the model. We could also have used a P-value to draw coclusios. The P-value for t is P 0.000, so with = 0.05 we would reject the ull hypothesis. Note that this test measures the margial or partial cotributio of x give that x 1 is i the model. That is, the t-test measures the cotributio of addig the variable x die height to a model that already cotais x 1 wire legth. Table 1-4 shows the value of the t-test computed by Miitab. The Miitab t-test statistic is reported to two decimal places. Note that the computer produces a t-test for each regressio coefficiet i the model. These t-tests idicate that both regressors cotribute to the model. There is aother way to test the cotributio of a idividual regressor variable to the model. This approach determies the icrease i the regressio sum of squares obtaied by addig a variable x j (say) to the model, give that other variables x i (i j) are already icluded i the regressio equatio. The procedure used to do this is called the geeral regressio sigificace test, or the extra sum of squares method. This procedure clso be used to ivestigate the cotributio of a subset of the regressor variables to the model. Cosider the regressio model with k regressor variables y X (1-5) where y is ( 1), X is ( p), is (p 1), is ( 1), ad p k 1. We would like to determie if the subset of regressor variables x 1, x,..., x r (r k) as a whole cotributes sigificatly to the regressio model. Let the vector of regressio coefficiets be partitioed as follows: c 1 d (1-6) where 1 is (r 1) ad is [(p r) 1]. We wish to test the hypotheses H 0 : 1 0 H 1 : 1 0 (1-7) where 0 deotes a vector of zeroes. The model may be writte as y X X 1 1 X (1-8) where X 1 represets the colums of X associated with 1 ad X represets the colums of X associated with.

7 434 CHAPTER 1 MULTIPLE LINEAR REGRESSION For the full model (icludig both 1 ad ), we kow that ˆ 1X X 1 X y. additio, the regressio sum of squares for all variables icludig the itercept is I SS R 1 ˆ X y 1 p k 1 degrees of freedom ad MS E y y ˆ X y p SS R ( ) is called the regressio sum of squares due to. To fid the cotributio of the terms i 1 to the regressio, fit the model assumig the ull hypothesis H 0 : 1 0 to be true. The reduced model is foud from Equatio 1-8 as y X (1-9) The least squares estimate of is ˆ 1X X 1 X y, ad SS R 1 ˆ X y 1p r degrees of freedom (1-30) The regressio sum of squares due to 1 give that is already i the model is SS R SS R 1 SS R 1 (1-31) This sum of squares has r degrees of freedom. It is sometimes called the extra sum of squares due to 1. Note that SS R is the icrease i the regressio sum of squares due to icludig the variables x 1, x, p, x r i the model. Now SS R is idepedet of MS E, ad the ull hypothesis 1 0 may be tested by the statistic F 0 SS R1 1 r MS E (1-3) If the computed value of the test statistic f 0 f,r, p, we reject H 0, cocludig that at least oe of the parameters i 1 is ot zero ad, cosequetly, at least oe of the variables x 1, x, p, x r i X 1 cotributes sigificatly to the regressio model. Some authors call the test i Equatio 1-3 a partial F-test. The partial F-test is very useful. We ca use it to measure the cotributio of each idividual regressor x j as if it were the last variable added to the model by computig SS R 1 j 0 0, 1, p, j 1, j 1, p, k, j 1,, p, k This is the icrease i the regressio sum of squares due to addig x j to a model that already icludes x 1,..., x j 1, x j 1,..., x k. The partial F-test is a more geeral procedure i that we ca measure the effect of sets of variables. I Sectio we show how the partial F-test plays a major role i model buildig that is, i searchig for the best set of regressor variables to use i the model.

8 1- HYPOTHESIS TESTS IN MULTIPLE LINEAR REGRESSION 435 EXAMPLE 1-5 Cosider the wire bod pull stregth data i Example 1-1. We will ivestigate the cotributio of the variable x (die height) to the model usig the partial F-test approach. That is, we wish to test H 0 : 0 H 1 : 0 To test this hypothesis, we eed the extra sum of squares due to, or I Example 1-3 we have calculated ad from Example 11-8, where we fit the model Y 0 1 x 1, we ca calculate Therefore, SS R 1 1, 0 0 ˆ X y SS R 1 0 1, 0 SS R 1 1,, 0 SS R 1 1, 0 SS R 1 1, 0 0 SS R SS R ˆ 1S xy oe degree of freedom SS R 1 0 1, oe degree of freedom This is the icrease i the regressio sum of squares due to addig x to a model already cotaiig x 1. To test H 0 : 0, calculate the test statistic f 0 SS R1 0 1, 0 1 MS E two degrees of freedom Note that the MS E from the full model, usig both x 1 ad x, is used i the deomiator of the test statistic. Sice f 0.05,1, 4.30, we reject H 0 : 0 ad coclude that the regressor die height (x ) cotributes sigificatly to the model. Table 1-4 shows the Miitab regressio output for the wire bod pull stregth data. Just below the aalysis of variace summary i this table the quatity labeled Seq SS shows the sum of squares obtaied by fittig x 1 aloe (5885.9) ad the sum of squares obtaied by fittig x after x 1. Notatioally, these are referred to above as SS R ad SS R 1 0 1, 0. Sice the partial F-test i the above example ivolves a sigle variable, it is equivalet to the t-test. To see this, recall from Example 1-5 that the t-test o H 0 : 0 resulted i the test statistic t Furthermore, the square of a t-radom variable with degrees of freedom is a F-radom variable with oe ad degrees of freedom, ad we ote that t 0 (4.4767) 0.04 f More About the Extra Sum of Squares Method (CD Oly)