Diagnostic for Multiple Linear Regression

Transcription

1 Diagnostic for Multiple Linear Regression Yang Feng

2 Diagnostics Added-Variable Plots (Model Adequacy for a Predictor Variable) Studentized Deleted Residuals (Identifying outlying Y ) Hat Matrix Leverage Values (Identifying outlying X ) DFFITS, Cook s Distance, DFBETAS (Identifying Influential Cases) Variance Inflation Factor (Multicollinearity Diagnostic)

3 Added-Variable Plots Measures the marginal role of X k given other variables are already in the model. Links with Coefficient of Partial Determination Suppose only X 1 and X 2, we want to measure the role of X 1 given X 2 is already in the model. 1 Regress Y on X 2 and obtain the residuals e i (Y X 2 ) = Y i Ŷi(X 2 ) 2 Regress X 1 on X 2 and obtain the residuals e i (X 1 X 2 ) = X i1 ˆX i1 (X 2 ) 3 The scatter plot of e i (Y X 2 ) and e i (X 1 X 2 ) provides a graphical representation of the strength of the relationship between Y and X 1, adjusted for X 2. The plot is called Added-Variable Plots.

4 Added-Variable Plots (Some prototypes)

5 Studentized Deleted Residuals Figure :

6 Studentized Deleted Residuals (Cont ) Let d i be the deleted residual for the i th case d i = Y i Ŷ i(i) The studentized deleted residual, denoted by t i is where t i = d i s{d i } s 2 {d i } = MSE (i) d i t(n p 1) 1 h ii s{d i } It can be shown that [ n p 1 t i = e i SSE(1 h ii ei 2) ] 1/2 These t i s can be used to formally test (e.g. using a Bonferroni test procedure) whether the largest absolute studentized deleted residual is an outlier.

7 Hat Matrix Leverage Values Figure :

8 Hat Matrix Leverage Values For hat matrix H, we have 0 h ii 1, n i=1 h ii = p. h ii is called the leverage of the ith case. Large h ii indicates that it has substantial leverage in determining the fitted value Ŷi. Reasons: The fitted value is a linear combination of the observed Y values, where h ii is the weight of observation Y i. The large is h ii, the smaller the variance of the residual e i. If h ii > 2p/n, we say it is large.

9 Identifying Influential Cases After identifying cases as outlying, we would like to ascertain that these cases are influential, i.e., whether its exclusion causes major changes in the fitted regression function. Three measures: Influence on Single Fitted Value (DFFITS). Case i has on the fitted value Ŷi is given by (DFFITS) i = Ŷ i Ŷ i(i) MSE(i) h ii (1) ( ) 1/2 hii = t i (2) 1 h ii Influential if larger than 1 for small to medium data sets, or larger than 2 p/n for large data sets.

10 Identifying Influential Cases Influence on all fitted values (Cook s Distance). n j=1 (Ŷj Ŷj(i)) 2 D i = pmse (3) Influential if near 50 percentile of F (p, n p) = e2 i h ii pmse (1 h ii ) 2 (4) Influence on the Regression Coefficients (DFBETAS). (DFBETAS) k(i) = b k b k(i) MSE(i) c kk, (5) where c kk is the kth diagonal element of (X X) 1. Influential if larger than 1 for small to medium data sets, or larger than 2/ n for large data sets.

11 Variance Inflation Factor A formal way of detecting the presence of multcollinearity. Considering the standardized regression model, we have σ 2 {b } = (σ ) 2 r 1 XX Define (VIF ) k as the kth diagonal element of r 1 XX. VIF value measure how large is the variance of bk relative to what the variance would be if the predictor variables were uncorrelated. (VIF ) k = (1 Rk 2) 1, for k = 1, 2,, p 1, where Rk 2 is the coefficient of determination when X k is regressed on the p 2 other X variables. Think about what happens when R k = 0 and when Rk 2 is close to 1. max k (VIF ) k > 10 indicates that there is serious multicollinearity problem.