3 Lack of Fit and Multicollinearity in Regression Models

Transcription

1 3 Lack of Fit and Multicollinearity in Regression Models 3. Lack of Fit in Regression When data is collected and a regression model is fit, we want to assess if the fitted regression model adequately models the response variable. The ability to assess the quality of the fitted models is possible when replications are taken at several combinations of the predictor variables. One simple check for model adequacy is to perform a lack-of-fit test. To perform the lackof-fit test, the error sum of squares, SSE, is partitioned into two components: pure error and lack-of-fit error: SSE = The pure error sum of squares, SSE P, is computed using only the responses from replicated observations: SSE P = where m is the number of unique combinations of predictor variables with multiple observations, and n i is the number of replicates for the i th combination of variables. SSE P can also be written as SSE P = where s i is the sample variance of the n i observations for the i th combination of variables. MSE P = SSE m P q m where q = n + n n m. Note q m = (n i ), the total i= degrees of freedom from pooling across m sets of repeated observations. The lack-of-fit sum of squares: SSE LOF = SSE SSE P, and the degrees of freedom are d.f.(lof) = d.f.(error) d.f.(pure Error) = N p q + m. The ANOVA table for a model with p terms (not including the intercept): Source of Variation d.f. Sum of Squares Mean Squares F-statistic Regression p SS R MSR F = Error N p SS E MSE Lack of Fit N p q + m SSE LOF MSE LOF F = Pure Error q m SSE P MSE P Total N SS T A statistically significant lack-of-fit F -statistic implies that the terms in the model do not account for all of the assignable cause variation in the response variable. Because the pure error mean square, MSE P, measures the variation associated with replicate observations, a large F -ratio indicates that the MS LOF is a measure of the variation that exceeds the variation due to pure (replication) error. When a large F -ratio occurs, alternative model specifications should be considered (such as a transformation of the response or regressor variables, or inclusion of additional polynomial terms). 8

2 3.. Lack of Fit Example From Statistical Design and Analysis of Experiments by Mason, Gunst, and Hess. For a particular brand of automobile tire, an experiment was run to examine the relative tire tread wear. Two factors (temperature T and miles M) were believed to affect tread wear (y). Temperature T is in degrees Fahrenheit. Miles M is the number of miles the tires were tested on a wet road surface. There are m = 8 unique combinations of (T, M), and n i = 4 replicates taken for each of the m = 8 combinations. The experimental data is given in the following table. We will perform lack-of-fit tests for several regression models. Temp Miles Rep Pure Wear Sum of squares for (T ) (M) (j) Error df y ij (T, M) combination j= (y j y ) = i= (y j y ) = i= (y 3j y 3 ) = i= (y 4j y 4 ) = i= (y 5j y 5 ) = i= (y 6j y 6 ) = i= (y 7j y 7 ) = i= (y 8j y 8 ) = df P = 4 SS P = Pure error mean square MS P =.88373/4 =.785 9

3 Let x = T and x = M. Perform a lack-of-fit test for the following models: Model () y ij = β + β x + β x + ɛ ij Model () y ij = β + β x + β x + β x x + β x + β x + ɛ ij Model (3) y ij = β + β x + β x + β x x + β x + ɛ ij We know that SS P =.88373, df P = 4, and MS P =.785. These values associated with pure error do not depend on the choice of model. The next step is to fit each model, and calculate SS LOF = SS E SS P, df LOF = df E df P, and MS LOF = SS LOF /df LOF for each model. For Model (): Source of Variation d.f. Sum of Squares Mean Squares F-statistic p-value Regression F = 7.6. Error Lack of Fit F =3.56 <. Pure Error Total There is a significant lack-of-fit for Model (p-value <.). We should consider adding additional terms to the model (such as an interaction term (β x x ) or squared terms (β x or β x )). If we add these three terms, we now have Model. For Model (): Source of Variation d.f. Sum of Squares Mean Squares F-statistic p-value Regression F = 4.3 <. Error Lack of Fit.7.35 F = Pure Error Total There is no significant lack-of-fit for Model (p-value =.645). We do not need to add any additional terms to the model. It may be possible to remove terms and still not reject the lack-of-fit test. If we remove β x, then we get Model 3. For Model (3): Source of Variation d.f. Sum of Squares Mean Squares F-statistic p-value Regression F =.4 <. Error Lack of Fit F = Pure Error Total There is marginal evidence (p-value =.538) for lack-of-fit at α =.5 significance level. 3

4 3.. Using PROC REG in SAS PROC REG Output for Model : y ij = β + β x + β x + ɛ ij. LOF Test and Collinearity Check for MODEL Fit Diagnostics for wear wear.6. LOF Test and Collinearity Check for MODEL.4 Cook's D Leverage Number.of Observations Read Number of Observations Used 3 3 Quantile Observation Fit Mean Percent 3 Source Analysis of Variance. DF. -. Sum of Squares Observations 3 Parameters 3 Error DF 9 MSE.48 R-Square.3439 Adj R-Square.987 Mean Square F Value Pr > F Model Error Lack of Fit Proportion.64 Less 3.56 <. Pure Error Corrected Total Root MSE.576 R-Square.3439 Dependent Mean.3446 Adj R-Sq.987 Coeff Var Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Variance Inflation Intercept <. T M

5 PROC REG Output for Model : y ij = β + β x + β x + β x x + β x + β x + ɛ ij. LOF Test and Collinearity Check for MODEL Fit Diagnostics for wear Percent Quantile wear LOF Test and Collinearity Check for MODEL Fit Mean 3 Observation.3. Sum of Mean Observations 3 Parameters 6. Source DF Squares Square. Error DF F Value Pr > F MSE 6.75 Model R-Square Adj <. R-Square Error Lack of Fit Proportion Less Cook's D Number.of Observations Read 3. Number of Observations Used 3 Analysis of Variance Leverage Pure Error Corrected Total Root MSE.8669 R-Square.8 Dependent Mean.3446 Adj R-Sq.7879 Coeff Var Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Variance Inflation Intercept <. T M TM T M

6 SAS PROG REG Output for Model 3: y ij = β + β x + β x + β x x + β x + ɛ ij. LOF Test and Collinearity Check for MODEL 3 Fit Diagnostics for wear Percent Quantile wear LOF Test and Collinearity Check for MODEL 3 Source Fit Mean Error Observations 3 Parameters 5 Error DF 7 MSE.95 R-Square.7655 Adj R-Square Lack of Fit Proportion Less Pure Error Cook's D Number of Observations Read 3 Number of Observations Used 3 Analysis of Variance DF Sum of Squares Mean Square F Value Pr > F Model < Leverage 3 Observation Corrected Total Root MSE.9765 R-Square.7655 Dependent Mean.3446 Adj R-Sq.738 Coeff Var Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Variance Inflation Intercept <. T < M TM < M

7 Original and Coded Data Obs wear temp T miles M T

8 PROC REG SAS Code for Models,, and 3: DM LOG; CLEAR; OUT; CLEAR; ; ODS PRINTER PDF file= C:\COURSES\ST578\SAS\LOF_WEAR.pdf ; ODS LISTING; OPTIONS PS=54 LS=7 NODATE NONUMBER; ***********************************************; *** LACK OF FIT ANALYSIS FOR TREADWEAR DATA ***; ***********************************************; DATA in; INPUT wear temp LINES; ; DATA in; SET in; *** CODE THE VARIABLE LEVELS WITH MIN = - and MAX = + ***; T = (temp-7.4)/8.; M = (miles-.5)/5.5; TM = T*M; T = T**; M = M**; PROC PRINT DATA = in; VAR wear temp T miles M T; TITLE Original and Coded Data ; PROC REG DATA=in; MODEL wear = T M / LACKFIT VIF; TITLE LOF Test and Collinearity Check for MODEL ; PROC REG DATA=in; MODEL wear = T M TM T M / LACKFIT VIF; TITLE LOF Test and Collinearity Check for MODEL ; PROC REG DATA=in; MODEL wear = T M TM M / LACKFIT VIF; TITLE LOF Test and Collinearity Check for MODEL 3 ; RUN; 3. Multicollinearity in Regression In regression problems, strong linear dependencies may exist among the regressor (predictor) variables in the regression model. If strong linear dependencies exist, then multicollinearity exists among the regressors. Matrix interpretation: Let X = [ x, x,..., x k ] be the design matrix where x j is the j th column of X. Multicollinearity occurs when there are strong linear dependencies among the k columns of X. That is, there exits a set of constants c j (not all ) for which c j x j. j= 35

9 Multicollinearity can seriously affect the estimates of the model parameters, and, therefore, affect the interpretation and applicability of the final model. When a set of regressors are conditioned in such a way that the regressors are highly collinear, the least squares procedure can not detect the structure of the multicollinearity in the data. With multicollinearity, the least squares method cannot separate the individual effects of collinear variables. There are numerous methods for detecting the presence of multicollinearity. I will discuss several common methods:. Look at sample correlation coefficients r ij for each pair of regressor variables x i and x j. If r ij is close to, then x i and x j are strongly collinear. This method, however, will only detect collinearity among pairs of variables. If more than two variables are multicollinear, this method will not always enable the detection of the multicollinearity.. If the F -test for the fit of the regression model is significant, but the F -tests for the individual regression coefficients are not significant, then multicollinearity may be present. 3. The eigenvalues of the X X matrix for design matrix X provide a measure of multicollinearity. One or more eigenvalues near imply that multicollinearity is present. If λ max and λ min are the largest and smallest eigenvalues of X X, then the condition number κ = Rule of thumb: If κ <, there is little problem with multicollinearity. Also, the number of small eigenvalues is directly related to the number of multicollinearities. 4. The variance inflation factor (VIF) is defined to be: VIF(b j ) = j =,,..., k where Rj is the coefficient of multiple determination resulting from the regression of x j on the remaining k regressors (b j is the model coefficient associated with x j ). A strong multicollinearity between x j and the other regressors variables is reflected in Rj close to. Thus, the VIF will increase as Rj. 5. We say that the variance of b j is inflated by the quantity That is, the VIFs ( Rj ). represent the inflation that each regression coefficient experiences above the case when the correlation matrix of the regressor variables is an identity matrix. Some authors have suggested that if any VIFs exceed (R j =.9), then multicollinearity is a problem. Other authors consider this value is too liberal and suggest that the VIFs should not exceed 4 or 5 (R j =.75,.8). Some computing packages provide tolerances. The tolerance of b j = VIF(b j ) = Thus, we do not want tolerances to be too small. For example if a VIF >, then the tolerance is <.. 36

10 37