4.5.1 Model Utility Test To test if any of the predictors has an effect test the null hypothesis that all slopes are 0

Transcription

1 4. Hypothesis Tests After estimating the parameters of the model, we should check if the model is supported by the data. The first questions to be answered will be if any of the predictors has an effect on the mean response. If evidence shows this to be true, then we should answer the question, which of predictors has an effect. 4.. Model Utility Test To test if any of the predictors has an effect test the null hypothesis that all slopes are 0 H 0 : β β β k 0 versus H a : at least on slope is not 0 The test is based on an ANOVA approach, where we analyse the total sum of squares in Y, and find how much of the variation is due to the variables in the model and how much of the variation remains unexplained. The Total Sum of Squares: Let J n be the n n matrix with all entries being. SS T Y Y The Residual Sum of Squares (see above): n Y J n Y nx i (Y i Ȳ ) SS Res e e Y (I n X(X X) X ) Y Y Y ˆβ X Y nx The Sum of Squares for Regression: SS R SS T SS Res ˆβ X Y n Y J n Y nx i i (Ŷi Ȳ ) (Y i Ŷi) The F-score relating the Sum of Squares for Regression with the Sum of Squares for Residuals is used for the test SS R /k F SS Res /(n k ) MS R MS Res Under the assumptions of the MLRM and if β β β k 0 this F-score is F-distributed with df n k and df d n k. The proof of this result uses the fact that SS R is also χ distributed with df k and that SS R and SS Res are independent. The information for this test is usually summarized in an ANOVA table. Source SS df MS F Regression SS R k MS R MS R /MS Res Residual SS Res n k MS Res Total SS T n

2 ANOVA F-test. Hyp.: Choose α. H 0 : β β β k 0 versus H a : at least on slope is not 0. Assumptions: Random samples, the MLRM is a proper description of the population, including homoscedasticity and normality of the error.. Test statistic: F 0 4. P-value: P (F > F 0 ) SS R /k SS Res /(n k ) MS R MS Res, df n k p, df d n k n p Continue Example.?? Continue the example on the effect of age and weight on blood pressure and the analysis of the model: bp β 0 + β (weight) + β (age) + ε, ε N (0, σ ) First find the ANOVA table. We already calculated SS Res.8. It is SS T y y n y J y (0, 4, 4,,, 9) Therefore SS R SS T SS Res (Use that n y J y ( P n i y ) /n) ANOVA table C A () 4.8 Source SS df MS F Regression Residual.8.9 Total 4.8 ANOVA F-test. Hyp.: Choose α 0.0. H 0 : β β 0 versus H a : at least one slope is not 0. Assumptions: Random samples, the MLRM is a proper description of the population, including homoscedacity and normality of the error.

3 . Test statistic: F , df n, df d 4. P-value: P (F > F 0 ), use F-distribution table from my website, since F 0 is greater than the 0.00 critical value of the F distribution with df n and df d, the P-value is smaller than Therefore the P-value is smaller than α, reject H 0.. Context: At significance level of % the data provide sufficient evidence that at least one of the slopes is different from zero, i.e. that at least one of age or weight has an effect on the mean blood pressure. From here arises the question which of the slopes is not zero. The model utility test does not indicate that the model is a good description for the population, only that if it is a good fit then at least on of the regressors has an effect on the mean response. A residual analysis will help to judge if the model is appropriate for the population under consideration. Adjusted coefficient of determination The adjusted coefficient of determination, R adj, indicates how well the data is represented by the estimated model. In comparison to the coefficient of determination, R, it is adjusted for the number of variables in the model. When adding an additional variable to an existing model the coefficient of determination will always increase, regardless of the relevance of the variable for the response. The adjusted coefficient of determination includes an adjustment for the number of variables in the model to correct for this effect, but otherwise is to be interpreted in the same way as R. R adj MS Res MS T Continue Example.?? The adjusted coefficient of determination for the blood pressure example is R adj.9 4.8/ 0.9 9% of the sample variation in blood pressure can be explained by the changes in weight and age. The adjusted coefficient of determination is often used in the model building process. 4.. Test for individual slopes In order to find out if a variable x i has a significant linear effect on the mean response when correcting for the other variables in the model, a test for H 0 : β i 0, i k should be conducted. The test can be based on the t-score standardizing ˆβ i : Theorem 4.. In the MLRM t ˆβ i β i se( ˆβ i ) ˆβ i β i ˆσ C ii, where C ii is the i th diagonal entry in (X X), is t-distributed with df n p.

4 Conclusion: Let i {,,..., k}. A ( α) 00% confidence interval for slope β i is given by ˆβ i ± t n p α/ Ȉσ C ii, Test for individual slopes correcting for the other variables in the model:. Hypotheses: Choose α. Type Hypotheses upper tail H 0 : β i β i0 versus H a : β i > β i0 lower tail H 0 : β i β i0 versus H a : β i < β i0 tail H 0 : β i β i0 versus H a : β i β i0. Assumptions: Random samples, the MLRM is a proper description of the population, including homoscedacity and normality of the error.. Test statistic: t 0 ˆβ i β i0 ˆσ C ii, df n p 4. P-value: Type P-value upper tail P (t > t 0 ) lower tail P (t < t 0 ) tail P (t > t 0 ) Continue Example.?? Test if the mean blood pressure increases with increasing weight(x ) within the proposed model bp β 0 + β (weight) + β (age) + ε, ε N (0, σ ). Hypotheses: H 0 : β 0 versus H a : β > 0, choose α Assumptions: Random samples, the MLRM is a proper description of the population, including homoscedasticity and normality of the error.. Test statistic: t È.9(0.08).8, df 4. P-value: P-value P (t >.8), using the t-table with df, find that the P-value< Decision: Reject H 0.. Context: At significance level of % the data provide sufficient evidence that on average the blood pressure increases with weight when correcting for age. 4

5 4.. Test for a subset of slopes In some applications it is of interest to test if in the presence of some (usually confounding) variables at least one in a set of slopes for different variables is not zero. Such a question would for example arise, when it is of interest to find the effect of the amount of humour, violence, and drama on the popularity of a movie, after correcting for the effects of age and sex (the confounding variables). For the development of the test statistic the extra sum of squares principle is applied. It is based on the comparison of the residual sum of squares for two models:. The full model: model which includes all variables.. The reduced model: the model only including the variables to be corrected for. Essentially it is tested if the inclusion of the extra variables results in a significant decrease in the residual sum of squares.. The full model with k predictors, therefore p k + parameters (including the intercept) Y X β + ε where Y and ε are n dimensional random vectors, the design matrix X R n p, and the parameter vector β R p. To test for the effect of r < k predictors partition the design matrix and the parameter vector such that Y [X X ] " β β # + ε X β + X β + ε with partitioned design matrix where X R n (p r) and X R n r, and partitioned parameter vector where β R (p r) and β R r. For the full model with ˆβ (X X) X Y with df n p. We wish to test H 0 : β 0. SS Res ( β) Y Y ˆβ X Y. The reduced model, where we assume β 0, with k r predictors and k r + p r parameters: Y X β + ε and with ˆβ (X X ) X Y with df n (p r) n p + r. SS Res ( β ) Y Y ˆβ X Y

6 The difference in the Sum of Squares for Residuals when adding variables x k r+,..., x k to the model with variables x,..., x k r is SS Res ( β β ) SS Res ( β ) SS Res ( β) with df n p + r (n p) r. These are called the extra sum of squares due to β (extra sum of squares for regression, since the decrease in the residual sum of squares implies an increase by the same amount in the regression sum of squares). One can prove that SS Res ( β β ) is independent of MS Res therefore is F distributed with df n r and df d n p. ANOVA Extra Sum of Squares Test:. Hypotheses: Choose α. F SS Res( β β )/r MS Res H 0 : β k r+ β k 0( β 0) versus H a : at least one slope is not 0. Assumptions: Random samples, the MLRM is a proper description of the population, including homoscedasticity and normality of the error.. Test statistic: F 0 SS Res( β β )/r MS Res, df n r, df d n k 4. P-value: P (F > F 0 ) Continue Example.?? The model is bp β 0 + β (weight) + β (age) + ε, ε N (0, σ ) Use the extra sum of squares test to see if age has a significant effect on the mean blood pressure when correcting for weight. The full model: The reduced model:. Hypotheses: Choose α. bp β 0 + β (weight) + β (age) + ε, ε N (0, σ ) bp β 0 + β (weight) + ε, ε N (0, σ ) H 0 : β 0 versus H a : β 0

7 . Assumptions: Random samples, the MLRM is a proper description of the population, including homoscedasticity and normality of the error.. Test statistic: r. From previous work SS Res ( β).8 and MS Res.9. For finding SS Res (β ) X C A with X X Œ and (X X ) Œ and X y 0 Œ gives SS Res (β ) y y (X y) (X X ) (X y) 8. F 0 (8.8.8)/.9 4.4, df n r, df d n p 4. P-value: 0.00 < P (F > F 0 ) < 0.0 according to the F-distribution table.. Decision: Reject H 0.. Context: At significance level of % the data provide sufficient evidence that age has a significant effect on the blood pressure when correcting for weight. R reduced<-lm(bp~weight) full<-lm(bp~weight+age) anova(reduced, full) 4. Orthogonal Columns in the Design Matrix Definition 4.. Matrices A R n m and B R n k are orthogonal, iff A B 0 m k. Theorem 4.. If the design matric can be partitioned, so that X [X X ] with X X 0 then the least squares estimator for β ( β, β ) can be partitioned into ˆβ ( ˆβ, ˆβ ), so that ˆβ (X X ) X Y, and ˆβ (X X ) X Y

8 Proof: It is ˆβ (X X) X Y. First find (X X) : X X X [X X X ] X X X X X X X X X X 0 (p r) r 0 r (p r) X X Then is the inverse of X X, because X XA A 4 X X 0 (p r) r 0 r (p r) X X (X X ) 0 (p r) r 0 r (p r) (X X ) 4 (X X ) 0 (p r) r 0 r (p r) (X X ) 4 (X X )(X X ) + 0 (p r) r 0 r (p r) (X X )0 (p r) r + 0 r (p r) (X X ) 0 r (p r) (X X ) + (X X )0 r (p r) 0 r (p r) 0 (p r) r + (X X )(X X ) 4 I p r 0 (p r) r 0 r (p r) I r With this result now find ˆβ I p ˆβ (X X) X Y 4 (X X ) 0 (p r) r 0 r (p r) (X X ) 4 (X X ) 0 (p r) r 0 r (p r) (X X ) 4 (X X ) X Y (X X ) X Y 4 ˆβ ˆβ 4 X X 4 X Y X Y Y q.e.d. Conclusion: Because of the theorem, we find in the case of orthogonal columns in the design matrix, that the parameter vector can be partitioned, so that the parts are independent from each other, and can be found independently. The extra sum of squares test for H 0 : β 0 is in the case of orthogonal columns the same as the model utility test in the partitioned model Y X β + ε. For the full model, it is now easy to prove that SS R ( β) SS R ( β ) + SS R ( β ) 8

9 I.e. the sum of squares for regression for the complete parameter vector equals the sum of squares for regression for the partitioned parameter vector, and therefore according to the extra sum of squares principle the two are independent, and tests for one partitioned vector is independent from the other. 9