Lecture 5 Hypothesis Testing in Multiple Linear Regression

Transcription

1 Lecture 5 Hypothesis Testing in Multiple Linear Regression BIOST 515 January 20, 2004

2 Types of tests 1 Overall test Test for addition of a single variable Test for addition of a group of variables

3 Overall test 2 y i = β 0 + x i1 β x ip β p + ɛ i Does the entire set of independent variables contribute significantly to the prediction of y?

4 Test for an addition of a single variable 3 Does the addition of one particular variable of interest add significantly to the prediction of y acheived by the other independent variables already in the model? y i = β 0 + x i1 β x ip β p + ɛ i

5 Test for addition of a group of variables 4 Does the addition of some group of independent variables of interest add significantly to the prediction of y obtained through other independent variables already in the model? y i = β 0 + x i1 β x i,p 1 β p 1 + x ip β p + ɛ i

6 The ANOVA table 5 Source of Sums of squares Degrees of Mean E[Mean square] variation freedom square Regression SSR = ˆβ X y nȳ 2 SSR p p pσ 2 + β R X C X Cβ R Error SSE = y y ˆβ X SSE y n (p + 1) n (p+1) σ 2 Total SST O = y y nȳ 2 n 1 X C is the matrix of centered predictors: X C = 0 x 11 x 1 x 12 x 2 x 1p x p x 21 x 1. x 22 x 2. x 2p x p. x n1 x 1 x n2 x 2 x np x p 1 C A and β R = (β 1,, β p ).

7 The ANOVA table for 6 y i = β 0 + x i1 β1 + x i2 β2 + + x ip β p + ɛ i is often provided in the output from statistical software as Source of Sums of squares Degrees of F variation freedom Regression x 1 1 x 2 x 1. 1 x p x p 1, x p 2,, x 1 1 Error SSE n (p + 1) Total SST O n 1 where SSR = SSR(x 1 ) + SSR(x 2 x 1 ) + + SSR(x p x p 1, x p 2,..., x 1 ) and has p degrees of freedom.

8 Overall test 7 H 0 : β 1 = β 2 = = β p = 0 H 1 : β j 0 for at least one j, j = 1,..., p Rejection of H 0 implies that at least one of the regressors, x 1, x 2,..., x p, contributes significantly to the model. We will use a generalization of the F-test in simple linear regression to test this hypothesis.

9 Under the null hypothesis, SSR/σ 2 χ 2 p and SSE/σ 2 χ 2 n (p+1) are independent. Therefore, we have 8 F 0 = SSR/p SSE/(n p 1) = MSR MSE F p,n p 1 Note: as in simple linear regression, we are assuming that ɛ i N(0, σ 2 ) or relying on large sample theory.

10 CHS example, cont. 9 > anova(lmwtht) Analysis of Variance Table y i = β 0 + weight i β 1 + height i β 2 + ɛ i Response: DIABP Df Sum Sq Mean Sq F value Pr(>F) WEIGHT ** HEIGHT Residuals Signif. codes: 0 *** ** 0.01 * ( )/2 F 0 = = 5.59 > F 2,495,.95 = /495 We reject the null hypothesis at α =.05 and conclude that at least one of β 1 or β 2 is not equal to 0.

11 The overall F statistic is also available from the output of summary(). 10 > summary(lmwtht) Call: lm(formula = DIABP ~ WEIGHT + HEIGHT, data = chs) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-10 *** WEIGHT * HEIGHT Signif. codes: 0 *** ** 0.01 * Residual standard error: on 495 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 2 and 495 DF, p-value:

12 Tests on individual regression coefficients 11 Once we have determined that at least one of the regressors is important, a natural next question might be which one(s)? Important considerations: Is the increase in the regression sums of squares sufficient to warrant an additional predictor in the model? Additional predictors will increase the variance of ŷ - include only predictors that explain the response (note: we may not know this through hypothesis testing as confounders may not test significant but would still be necessary in the regression model). Adding an unimportant predictor may increase the residual mean square thereby reducing the usefulness of the model.

13 12 y i = β 0 + x i1 β x ij β j + + x ip β p + ɛ i H 0 : β j = 0 H 1 : β j 0 As in simple linear regression, under the null hypothesis t 0 = ˆβ j ŝe( ˆβ j ) t n p 1. We reject H 0 if t 0 > t n p 1,1 α/2. This is a partial test because ˆβ j depends on all of the other predictors x i, i j that are in the model. Thus, this is a test of the contribution of x j given the other predictors in the model.

14 CHS example, cont. 13 y i = β 0 + weight i β 1 + height i β 2 + ɛ i H 0 : β 2 = 0 vs H 1 : β 2 0, given that weight is in the model. From the ANOVA table, ˆσ2 = C = (X X) 1 = t 0 = / = < t 495,.975 = 1.96 Therefore, we fail to reject the null hypothesis.

15 Tests for groups of predictors 14 Often it is of interest to determine whether a group of predictors contribute to predicting y given another predictor or group of predictors are in the model. In CHS example, we may want to know if age, height and sex are important predictors given weight is in the model when predicting blood pressure. We may want to know if additional powers of some predictor are important in the model given the linear term is already in the model. Given a predictor of interest, are interactions with other confounders of interest as well?

16 Using sums of squares to test for groups of predictors 15 Determine the contribution of a predictor or group of predictors to SSR given that the other regressors are in the model using the extra-sums-of-squares method. Consider the regression model with p predictors y = Xβ + ɛ. We would like to determine if some subset of r < p predictors contributes significantly to the regression model.

17 Partition the vector of regression coefficients as β = [ ] β 1 β 2 16 where β 1 is (p + 1 r) 1 and β 2 is r 1. We want to test the hypothesis H 0 : β 2 = 0 Rewrite the model as where X = [X 1 X 2 ]. H 1 : β 2 0 y = Xβ + ɛ = X 1 β 1 + X 2 β 2 + ɛ, (1)

18 Equation (1) is the full model with SSR expressed as 17 SSR(X) = ˆβ X y (p+1 degrees of freedom) and MSE = y y ˆβ X y n p 1. To find the contribution of the predictors in X 2, fit the model assuming H 0 is true. This reduced model is y = X 1 β 1 + ɛ, where ˆβ 1 = (X 1 X 1 ) ( 1) X 1 y

19 and 18 SSR(X 1 ) = ˆβ 1 X 1 y (p+1-r degrees of freedom). The regression sums of squares due to X 2 when X 1 is already in the model is SSR(X 2 X 1 ) = SSR(X) SSR(X 1 ) with r degrees of freedom. This is also known as the extra sum of squares due to X 2. SSR(X 2 X 1 ) is independent of MSE. We can test H 0 : β 2 = 0 with the statistic F 0 = SSR(X2 X 1 )/r MSE F r,n p 1.

20 CHS example, cont. 19 Full model: y i = β 0 + weight i β 1 + height i β 2 H 0 : β 2 = 0 Df Sum Sq Mean Sq F value Pr(>F) WEIGHT HEIGHT Residuals F 0 = / = 0.95 < F 1,495,0.95 = 3.86 This should look very similar to the t-test for H 0.

21 20 BP i = β 0 + weight i β 1 + height i β 2 + age i β 3 + gender i β 4 + ɛ > summary(lm(diabp~weight+height+age+gender,data=chs)) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-08 *** WEIGHT HEIGHT AGE *** GENDER Signif. codes: 0 *** ** 0.01 * Residual standard error: on 493 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 4 and 493 DF, p-value:

22 H 0 : β 2 = β 3 = β 4 = 0 vs H 1 : β j, j = 2, 3, 4 21 Df Sum Sq Mean Sq F value Pr(>F) WEIGHT HEIGHT AGE GENDER Residuals SSR(intercept, weight, height, age, gender) = = SSR(intercept, weight) = = SSR(height, age, gender intercept, weight) = = 1670 Notice we can also get this from the ANOVA table above SSR(height, age, gender intercept,weight) = = 1670

23 The observed F statistic is 22 F 0 = 1670/3/ = 13.5 > F 3,493,.95 = 2.62, and we reject the null hypothesis, concluding that at least one of β 2, β 3 or β 4 is not equal to 0. This should look very similar to the overall F test if we considered the intercept to be a predictor and all the covariates to be the additional variables under consideration.

24 What if we had put the predictors in the model in a different order? 23 diabp i = β 0 + height i β 2 + age i β 3 + weight i β 1 + gender i β 4 + ɛ Df Sum Sq Mean Sq F value Pr(>F) HEIGHT AGE WEIGHT GENDER Residuals Could we use this table to test H 0 : β 2 = β 3 = β 4 = 0?

25 What if we had the ANOVA table for the reduced model? Df Sum Sq Mean Sq F value Pr(>F) WEIGHT Residuals Given that SSR = SSR(x 2 ) + SSR(x 3 x 2 ) + SSR(x 1 x 2, x 3 ) + SSR(x 4 x 3, x 2, x 1 ) and then SSR(x 2, x 3, x 4 x 1 ) = SSR SSR(x 1 ) SSR(x 2, x 3, x 4 x 1 ) = = 1680.

26 One other question we might be interested in asking is if there are any significant interactions in the model? 25 lm(diabp~weight*height*age*gender,data=chs) Estimate Std. Error t value Pr(> t ) (Intercept) WEIGHT HEIGHT AGE GENDER WEIGHT:HEIGHT WEIGHT:AGE HEIGHT:AGE WEIGHT:GENDER HEIGHT:GENDER AGE:GENDER WEIGHT:HEIGHT:AGE WEIGHT:HEIGHT:GENDER WEIGHT:AGE:GENDER HEIGHT:AGE:GENDER WEIGHT:HEIGHT:AGE:GENDER

27 ANOVA table 26 Df Sum Sq Mean Sq F value Pr(>F) WEIGHT HEIGHT AGE GENDER WEIGHT:HEIGHT WEIGHT:AGE HEIGHT:AGE WEIGHT:GENDER HEIGHT:GENDER AGE:GENDER WEIGHT:HEIGHT:AGE WEIGHT:HEIGHT:GENDER WEIGHT:AGE:GENDER HEIGHT:AGE:GENDER WEIGHT:HEIGHT:AGE:GENDER Residuals

28 We can simplify the ANOVA table to 27 Df Sum Sq Mean Sq F value Pr(>F) Main effects Interactions Residuals How do we fill in the rest of this table?