Model Selection in Minitab

Transcription

1 Model Selection in Minitab Variable Selection The basic procedure is obtained by Stat Regression Best Subsets Necessary input: The response variable is specified in the Responses: box. The pool of explanatory variables of which subsets are to be evaluated is entered in the Free predictors: box. No other specifications are necessary, though some may be desired, as follows. Commonly useful options: If there are some explanatory variables which are to be kept in all models, they are entered in the Predictors in all models: box in the main Best Subsets Regression window, and not in the Free predictors: box. The default output lists the two best models of each size. To change this number (I prefer to see at least 4), click the Options button, and entered the desired value in the Models of each size to print: box. To restrict the minimum or maximum sizes of the models to be evaluated, click the Options button, and entered the desired value(s) in the appropriate box under Free Predictor(s) in Each Model.

2 Output On the next page is an excerpt of the output from the analysis specified in the preceding screen shots. Each line in the table represents a particular model; as requested, four models of each size are reported. The variables in a particular model are indicated by the Xs in the columns under the variable names (which read downwards). For example, the first line is for the best one-variable model, with only TotPersInc. This model has R 2 = 44.4, R 2 a = 44.3, C p = 357.5, and s (= /MSE) = As another example, the best three-variable model is shown in the row starting This model contains the variables TotPop, PctPoverty, and PerCapInc. By both the C p and the R 2 a (=s) criteria, the best model in this example is the first 8-variable model, with all the explanatory variables except LandArea and PctBach. The best two 9-variable models, which add one or the other of the variables not in the preceding model, are nearly as good by both criteria. Using the C p criterion can be facilitated by plotting the C p values against the number of variables in the model. This can be done by cutting-and-pasting the output table into another Minitab worksheet (or Excel, etc.). The reference line can be added by creating two new columns, one with the values 0 and the maximum number of variables added, and the other with the corresponding p (i.e. 1 and the maximum number variables plus 1). After making the scatterplot of C p, these columns can be used to add a Calculated line to the scatterplot. The other criteria can be plotted similarly. Examples are on the next page. Model Selection in Minitab 2

3 Best Subsets Regression: logphys versus LandArea, TotPop,... Response is logphys P T P P P c P o L c c c t P e t a t t t P P c r P n T 1 O H c o t C e d o 8 v S t v U a r A t t e G B e n p s r P o r r a r e I I Mallows e o 3 6 a c t m n n Vars R-Sq R-Sq(adj) C-p S a p 4 5 d h y p c c X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Model Selection in Minitab 3

4 Cp number of variables (= p-1) adjusted R^ s (= square-root of MSE) number of variables (= p-1) number of variables (= p-1) 8 10 When one or more variables are forced to be in all models, the Vars column in the best-subsets output specifies how many of the free predictors are included; it does not count the predictors forced into the models. In such a case the number of parameters, p, will be the sum of the number of free predictors included (as given in the Vars column), the number of variables forced in, plus 1 for the intercept. On the next page is an excerpt of the output resulting from moving TotPop from the Free predictors: box to the Predictors in all models: box (bolding added to show these points). Model Selection in Minitab 4

5 Best Subsets Regression: logphys versus LandArea, Pct18to34,... Response is logphys The following variables are included in all models: TotPop P T P P P c P o L c c c t P e t a t t t P P c r P n 1 O H c o t C e d 8 v S t v U a r A t e G B e n p s r o r r a r e I I Mallows e 3 6 a c t m n n Vars R-Sq R-Sq(adj) C-p S a 4 5 d h y p c c X X X X X X Other Criteria PRESS The only other model-selection criterion available in Minitab is PRESS p. This can be gotten only for one model at a time, using the usual regression procedure. Stat Regression Regression Click on the Options button, then check the box for PRESS and predicted R- square under the Display part of the options window. (Predicted R 2 is the fraction of the Model Selection in Minitab 5

6 variation in the response variable explained by the leave-one-out predictions used to calculate PRESS. These two statistics are essentially redundant, but the predicted R 2 can be compared directly to the regular R 2 for the model, to judge whether the latter accurately reflects the predictive value of the model or is inflated by over-fitting.) Output The PRESS and predicted R 2 statistics are printed just below the regular R 2 : In this Regression Analysis: logphys versus TotPop, Pct18to34,... The regression equation is logphys = TotPop Pct18to PctOver PctHSGrad PctPoverty PctUnemp PerCapInc TotPersInc Predictor Coef SE Coef T P TotPersInc S = R-Sq = 69.9% R-Sq(adj) = 69.3% PRESS = R-Sq(pred) = 59.14% Analysis of Variance Source DF SS MS F P Regression Residual Error Total example the PRESS statistics is reasonably close to SSE, and the predicted R 2 is reasonably close to the regular R 2. These findings indicate that the model is at least not substantially overfit. Model Selection in Minitab 6

7 Validation Internal validation, using PRESS and predicted R 2, is done as explained above. MSPR with New Data The predicted values for the observations in the validation data set, predicted based on the selected model fit to the model-building data set, can be calculated in two ways. First, the fitted model equation can be entered into the Calculator, creating a column of predicted values in a worksheet containing only the validation data set. Alternatively, the validation data can be added (as new columns) to the modelbuilding data set, and Prediction intervals for new observations: can be requested in the Options window of the regression procedure, with the regression being done on the original data set. For example, if ntotpop is the Total Population variable in the new data set, and so on for the other variables, the following would be used: The predicted values will be stored in a new column, named PFIT1. Once the column of predicted values is created, the MSPR can be calculated in the Calculator, using an expression like SSQ ( nlgphys - PFIT1 ) / N( PFIT1), where nlgphys is the column of observed values of the response variable in the new data set, PFIT1 is the predicted values, SSQ is a function computing the sum of squared values for an entire column (or in this case, of squared differences between two columns), and N is a function returning the number of non-missing values in a column. The result will be a column with only one entry, which is the MSPR. Model Selection in Minitab 7

8 Data Splitting Various methods can be used to split a data set into model-building and validation sub-sets. A relatively easy method is to first create a column distinguishing which observations are to go into which sub-set, then use Data Split Worksheet to divide the data set based on the column just created. For example, to separate odd and even observations, the calculation shown in the window on the next page will create a column (c21) containing 1s for all observations with odd IDNum and 0s for all even observations. (The function MOD(x,y) returns the remainder after dividing x by y, and the entire expression is a logical comparison, evaluating to 1 (true) when the equality is true, and 0 (false) otherwise.) This column can then be used to split the data, as in the following window (invoked by Data Split Worksheet ). To simply select a random subset of observations, an easier method uses Calc Random Data Sample From Columns Using either of the preceding methods to create separate subsets, the MSPR can be calculated as described under MSPR with New Data above, either using the calculator to compute predicted values for the validation data set, or copying the validation data (as new columns) into the data-building subset and using the Prediction intervals for new observations: method. Yet another approach to data-splitting is to not actually divide the data set, but to separate the response variable into different columns for the model-building and Model Selection in Minitab 8

9 validation subsets. For instance, creating a new column as logphys / c21 (with c21 the 1/0 column created above), will have missing values for all even-numbered observations (for which c21 = 0). This new column, of observed values for the model-building data subset and missing values for the validation data subset, can then be used as the response variable in the regression procedure, and fits can be stored (click the Storage button in the main regression window, and check the box for Fits). Fits will be stored for all observations, including those with missing values of the response variable. The fits for the validation observations can then be separated from those for the model-building observations by copying the fits column to another column but selecting only observations in which the subsetting column (c21 here) equals 0. MSPR then can be calculated using this new column of the predicted values for the validation observations. Model Selection in Minitab 9