SELECTING THE BEST MODEL FOR MULTIPLE LINEAR REGRESSION

Transcription

1 SELECTING THE BEST MODEL FOR MULTIPLE LINEAR REGRESSION Introduction In multiple regression a common goal is to determine which independent variables contribute significantly to explaining the variability in the dependent variable. A goal in determining the best model is to minimize the residual mean square, which would intern maximize the multiple correlation value, R 2. The model that contains all independent variables will give the maximum R 2 value, but not all variables may contribute significantly to explaining the variability in the dependent variable. Another statistic provided in determining the best model is the C P criterion. The C P values will decrease as the number of independent variables in the model increases. C! = N P 1!"# 1 + (P + 1), where!! o N = number of observations o P = number of independent variables in the model o RMS = residual mean square of the P selected variables o σ! = is the residual mean square when all independent variables are included in the model. Stepwise Regression A variable selection method where various combinations of variables are tested together. The first step will identify the best one-variable model. Subsequent steps will identify the best two-variable, three-variable, etc. models. An F-test on each independent variable in the model The best models are typically identified as those that maximize R 2, C P, or both. Variations of stepwise regression include Forward Selection Method and the Backward Elimination Method. o Forward selection: a method of stepwise regression where one independent variable is added at a time that increases the R 2 value. Addition of variables to the model stops when the minimum F-to-enter exceeds a specified probability level. The default minimum F-to-enter in SAS is o Backward elimination: a method of stepwise regression where all independent variables begin in the model and subsequent variables are eliminated.

2 The variables eliminated first are those that contribute the least to the model. Elimination continues until the minimum F-to-remove drops below a specified probability level. The default minimum F-to-remove in SAS is Stepwise Regression Using SAS In this example, the lung function data will be used again, with two separate analyses. o Analysis 1: Determining which independent variables for the father (fage, fheight, fweight) significantly contribute to the variability in the father s (ffev1)? o SAS commands are: Proc STEPWISE; Model ffev1=fage fheight fweight; Title 'Stepwise regression of father data'; Run; o Analysis 2: Determining which independent variables for the youngest child (ycage, ycheight, ycweight) significantly contribute to the variability in the father s (ycfev1)? o SAS commands are: Proc STEPWISE; Model ycfev1=ycage ycheight ycweight; Title 'Stepwise regression of youngest child s data'; Run;

3 Stepwise regression of father data Dependent : ffev1 Number of Observations Read 150 Number of Observations Used 150 Stepwise Selection: Step 1 fheight Entered: R-Square = and C(p) = Model <.0001 Error Corrected Total Intercept fheight <.0001 Bounds on condition number: 1, 1 Stepwise Selection: Step 2 fage Entered: R-Square = and C(p) =

4 Stepwise regression of father data Dependent : ffev1 Stepwise Selection: Step 2 Model <.0001 Error Corrected Total Intercept fage <.0001 fheight <.0001 Bounds on condition number: , Stepwise Selection: Step 3 fweight Entered: R-Square = and C(p) = Model <.0001 Error Corrected Total

5 Stepwise regression of father data Dependent : ffev1 Stepwise Selection: Step 3 Intercept fage <.0001 fheight <.0001 fweight Bounds on condition number: , All variables left in the model are significant at the level. All variables have been entered into the model. Prediction Equation is Found in Step 3 ffev1! = (fage) (fheight) 0.005(fweight) 35.6% of the variation in ffev1 is explained by having fage, fheight, and fweight in the model. Step Entered Removed Summary of Stepwise Selection Number Vars In Partial R-Square Model R-Square C(p) F Value Pr > F 1 fheight < fage < fweight Collective R 2 values

6 Stepwise regression of youngest child data Dependent : ycfev1 Number of Observations Read 150 Number of Observations Used 24 Number of Observations with Missing Values 126 Forward Selection: Step 1 ycheight Entered: R-Square = and C(p) = Model <.0001 Error Corrected Total Intercept <.0001 ycheight <.0001 Bounds on condition number: 1, 1 Forward Selection: Step 2 ycage Entered: R-Square = and C(p) =

7 Stepwise regression of youngest child data Dependent : ycfev1 Forward Selection: Step 2 Model <.0001 Error Corrected Total Intercept ycage ycheight Bounds on condition number: , ycage is non- significant Forward Selection: Step 3 ycweight Entered: R-Square = and C(p) = Model <.0001 Error Corrected Total

8 Stepwise regression of youngest child data Dependent : ycfev1 Forward Selection: Step 3 Intercept ycage ycheight ycweight ycage and ycweight are non- significant Bounds on condition number: , All variables have been entered into the model. Prediction Equation is Found in Step 1 ycfev1! = (ycheight) Over 73% of the variation of ycfev1 is explained by having ycheight in the model.

9 Stepwise regression of youngest child data Dependent : ycfev1 Forward Selection: Step 3 Step Entered Summary of Forward Selection Number Vars In Partial R-Square Model R-Square C(p) F Value Pr > F 1 ycheight < ycage ycweight

10 Summary of Identifying the Best Model The F-test for each independent variable is testing to determine if that variable contributes significantly to the model given that the other independent variables in the step are included in the model. For example, in step 2 in the analysis of the father s data, the null hypothesis being tested on the F-test for fage is H o :fage = 0 given fheight is already in the model. For example, in step 3 in the analysis of the father s data, the null hypothesis being tested on the F-test for fage is H o :fage = 0 given fheight and fweight are already in the model. To determine which model best explains the variation in the dependent variable, find the model where for the first time you detect that one of the independent variables does not significantly contribute to the model. The model prior to this model is the one that should be used. The cumulative R 2 *100 for this model tells you the percent of the variation in the dependent variable that is explained by having the identified independent variables in the model. Considerations When Conducting Stepwise Regression The selection of the best model is as good as the independent variables used in the analyses. If important independent variables are not considered or are left out of the analyses, the results obtained may have biased regression coefficients, low R 2 values or both. The tests should be considered a screening method, not tests of significance since the F- values calculated don t necessarily match up with values in an F-table. Like multiple linear regression, results from stepwise regression are sensitive to violations of the assumptions underlying regression or problematic data. To test the robustness of the independent variables identified to be important, analyze subsets of the data to determine if the identified independent variables continue to be detected as significant.