For more information about how to cite these materials visit

Transcription

1 Author(s): Kerby Shedden, Ph.D., 010 License: Unless otherwise noted, this material is made available under the terms of the Creative Commons Attribution Share Alike 3.0 License: We have reviewed this material in accordance with U.S. Copyright Law and have tried to maximize your ability to use, share, and adapt it. The citation key on the following slide provides information about how you may share and adapt this material. Copyright holders of content included in this material should contact with any questions, corrections, or clarification regarding the use of content. For more information about how to cite these materials visit Any medical information in this material is intended to inform and educate and is not a tool for self-diagnosis or a replacement for medical evaluation, advice, diagnosis or treatment by a healthcare professional. Please speak to your physician if you have questions about your medical condition. Viewer discretion is advised: Some medical content is graphic and may not be suitable for all viewers. 1 / 8

2 Regression diagnostics Kerby Shedden Department of Statistics, University of Michigan December 7, 015 / 8

3 Motivation When working with a linear model with design matrix X, the conventional linear modeling assumptions can be expressed as E[Y X ] col(x ) and var[y X ] = σ I. Least squares point estimates and inferences depend on these assumptions approximately holding. Inferences for small sample sizes may also depend on the distribution of Y E[Y X ] being approximately multivariate Gaussian, but for moderate or large sample sizes this is not critical. Regression diagnostics are approaches for assessing how well the key linear modeling assumptions hold in a particular data set. 3 / 8

4 Residuals Linear models can be expressed in two equivalent ways: Expression based on moments: E[Y X ] col(x ) and var[y X ] = σ I. Expression based on an additive error model: Y = X β + ɛ, where ɛ is random with E[ɛ X ] = 0, and cov[ɛ X ] I. Since the residuals can be viewed as predictions of the errors, it turns out that regression model diagnostics can often be developed using the residuals. Recall that the residuals can be expressed R (I P)Y where P is the projection onto col(x ). / 8

5 Residuals The residuals have two key mathematical properties regardless of the correctness of the model specification: The residuals sum to zero, since (I P)1 = 0 and hence 1 R = 1 (I P)Y = 0. The residuals and fitted values are orthogonal (they have zero sample covariance): ĉov(r, Ŷ X) (R R) Ŷ = R Ŷ = Y (I P)PY = 0. These properties hold as long as an intercept is included in the model (so P 1 = 1, where 1 is a vector of 1 s). 5 / 8

6 Residuals If the basic linear model assumptions hold, these two properties have population counterparts: The expected value of each residual is zero: E[R X] = (I P)E[Y X] = 0 R n. The population covariance between any residual and any fitted value is zero: cov(r, Ŷ X) = ERŶ = (I P)cov(Y X)P = σ (I P)P = 0 R n n. 6 / 8

7 Residuals If the model is correctly specified, there is a simple formula for the variances and covariances of the residuals: cov(r X) = (I P) (EYY ) (I P) = (I P) ( X ββ X + σ I ) (I P) = σ (I P). If the model is correctly specified, the standardized residuals and the Studentized residuals Y i Ŷi ˆσ Y i Ŷi ˆσ(1 P ii ) 1/ approximately have mean zero and variance one. 7 / 8

8 External standardization of residuals Let ˆσ i be the estimate of σ obtained by fitting a regression model omitting the i th case. It turns out that we can calculate this value without actually refitting the model: ˆσ i = (n p 1)ˆσ r i /(1 P ii) n p where r i is the residual for the model fit to all data. The externally standardized residuals are Y i Ŷi ˆσ i, The externally Studentized residuals are Y i Ŷ i ˆσ i (1 P ii ) 1/. 8 / 8

9 Outliers and masking In some settings, residuals can be used to identify outliers. However, in a small data set, a large outlier will increase the value of ˆσ, and hence may mask itself. Externally Studentized residuals solve the problem of a single large outlier masking itself. But masking may still occur if multiple large outliers are present. 9 / 8

10 Outliers and masking If multiple large outliers may be present we may use alternate estimates of the scale parameter σ: Interquartile range (IQR): this is the difference between the 75 th percentile and the 5 th percentile of the distribution or data. The IQR of the standard normal distribution is 1.35, so IQR/1.35 can be used to estimate σ. Median Absolute Deviation (MAD): this is the median value of the absolute deviations from the median of the distribution or data, i.e. median( Z median(z) ). The MAD of the standard normal distribution is 0.65, so MAD/0.65 can be used to estimate σ. These alternative estimates of σ can be used in place of the usual ˆσ for standardizing or Studentizing residuals. 10 / 8

11 PRESS residuals If case i is deleted and a prediction of Y i is made from the remaining data, we can compare the observed and predicted values to get the prediction residual: R (i) Y i Ŷ (i)i. A simple formula for the prediction residual is given by R (i) = Y i X i: ˆβ (i) = Y i X i: ( ˆβ R i (X X ) 1 X i /(1 P ii )) = R i /(1 P ii ). The sum of squares of the prediction residuals is called PRESS (prediction error sum of squares). It is equivalent to using leave-one-out cross validation to estimate the generalization error rate. 11 / 8

12 Leverage Leverage is a measure of how strongly the data for case i determine the fitted value Ŷi. Since Ŷ = PY, and Ŷ i = j P ij Y j, it is natural to define the leverage for case i as P ii, where P is the projection matrix onto col(x ). This is related to the fact that the variance of the i th residual is σ (1 P ii ). Since the residuals have mean zero, when P ii is close to 1, the residual will likely be close to zero. This means that fitted line will usually pass close to (X i, Y i ) if it is a high leverage point. 1 / 8

13 Leverage These are the coefficients P ij plotted against X j (for a specific value of i), in a simple linear regression: X j Ŷ k = i S + n(x i X )(X k X ) Y i ns 13 / 8

14 P ij Leverage If we use basis functions, the coefficients in each row of P are much more local. X j 1 / 8

15 Leverage What is a big leverage? The average leverage is trace(p)/n = (p + 1)/n. If the leverage for a particular case is two or more times greater than the average leverage, it may be considered to have high leverage. In simple linear regression, it is easy to show that var(y i ˆα ˆβX i ) = (n 1)σ /n σ (X i X ) / j (X j X ). This implies that when p = 1, P ii = 1/n + (X i X ) / j (X j X ). 15 / 8

16 Leverage Leverage values in a simple linear regression: Y X Leverage X 16 / 8

17 Leverage Leverage values in a linear regression with two independent variables: X X1 17 / 8

18 Leverage In general, P ii = X i (X X ) 1 X i = X i (X X /n) 1 X i /n where X i is the i th row of X (including the intercept). Let X i be row i of X without the intercept, let µ be the sample mean of the X i, and let Σ X be the sample covariance matrix of the X i (scaled by n rather than n 1). It is a fact that and therefore X i (X X /n) 1 X i = ( X i µ)σ 1 X ( X i µ) + 1 P ii = Note that this implies that P ii 1/n. ( ) ( X i µ X )Σ 1 X ( X i µ X ) + 1 /n. 18 / 8

19 Leverage The expression ( X i µ X )Σ 1 X ( X i µ X ) is the Mahalanobis distance between X i and µ X. Thus there is a direct relationship between the Mahalanobis distance of a point relative to the center of the covariate set, and its leverage. 19 / 8

20 Influence Influence measures the degree to which deletion of a case changes the fitted model. We will see that this is different from leverage a high leverage point has the potential to be influential, but is not always influential. The deleted slope for case i is the fitted slope vector that obtained upon deleting case i. The following identity allows the deleted slopes to be calculated efficiently ˆβ (i) = ˆβ R i 1 P ii (X X ) 1 X i:, where R i is the i th residual, and X i: is row i of the design matrix. 0 / 8

21 Influence The deleted fitted values Ŷ(i) are Ŷ (i) = X ˆβ (i) = Ŷ Influence can be measured by Cook s distance: R i 1 P ii X (X X ) 1 X i:. D i = = 1 (p + 1)ˆσ (Ŷ Ŷ(i)) (Ŷ Ŷ(i)) R i (1 P ii ) (p + 1)ˆσ X i:(x X ) 1 X i: P ii R s i (1 P ii )(p + 1), where R i is the residual and R s i is the studentized residual. 1 / 8

22 Influence Cook s distance approximately captures the average squared change in fitted values due to deleting case i, in error variance units. Cook s distance is large only if both the leverage P ii is high, and the studentized residual for the i th case is large. As a general rule, D i values from 1/ to 1 are high, and values greater than 1 are considered to be a possible problem. / 8

23 Influence Cook s distances in a simple linear regression: 0.10 Cook's distance X 3 / 8

24 Influence Cook s distances in a linear regression with two variables: X X / 8

25 Regression graphics Quite a few graphical techniques have been proposed to aid in visualizing regression relationships. We will discuss the following plots: 1. Scatterplots of Y against individual X variables.. Scatterplots of X variables against each other. 3. Residuals versus fitted values plot.. Added variable plots. 5. Partial residual plots. 6. Residual quantile plots. 5 / 8

26 Scatterplots of Y against individual X variables E[Y X ] = X 1 X + X 3, var[y X ] = 1, var(x j ) = 1, cor(x j, X k ) = 0.3 Y 0 Y 0 0 X 1 0 X Y 0 Y 0 0 X 3 0 X 1 X +X 3 6 / 8

27 Scatterplots of X variables against each other E[Y X ] = X 1 X + X 3, var[y X ] = 1, var(x j ) = 1, cor(x j, X k ) = 0.3 X 0 0 X 1 X 3 0 X X 0 X 1 7 / 8

28 Residuals against fitted values plot E[Y X ] = X 1 X + X 3, var[y X ] = 1, var(x j ) = 1, cor(x j, X k ) = 0.3 Residuals 0 0 Fitted values 8 / 8

29 Residuals against fitted values plots Heteroscedastic errors: E[Y X ] = X 1 + X 3, var[y X ] = + X 1 + X 3, var(x j ) = 1, cor(x j, X k ) = Residuals Fitted values 9 / 8

30 Residuals against fitted values plots Nonlinear mean structure: E[Y X ] = X 1, var[y X ] = 1, var(x j) = 1, cor(x j, X k ) = 0.3 Residuals 0 0 Fitted values 30 / 8

31 Added variable plots Suppose P j is the projection onto the span of all covariates except X j, and define Ŷ j = P j Y, Xj = P j X j. The added variable plot is a scatterplot of Y Ŷ j against X Xj. The squared correlation coefficient of the points in the added variable plot is the partial R for variable j. Added variable plots are also called partial regression plots. 31 / 8

32 Added variable plots E[Y X ] = X 1 X + X 3, var[y X ] = 1, var(x j ) = 1, cor(x j, X k ) = 0.3 Ŷ X 1 Ŷ 3 0 Ŷ 0 0 X 3 0 X 3 / 8

33 Partial residual plot Suppose we fit the model Ŷ i = ˆβ X i = ˆβ 0 + ˆβ 1 X i1 + ˆβ p X ip. The partial residual plot for covariate j is a plot of ˆβ j X ij + R i against X ij, where R i is the residual. The partial residual plot attempts to show how covariate j is related to Y, if we control for the effects of all other covariates. 33 / 8

34 Partial residual plot E[Y X ] = X 1, var[y X ] = 1, var(x j) = 1, cor(x j, X k ) = 0.3 ˆβ1 X 1 +R 0 ˆβ X +R 0 0 X 1 0 X ˆβ3 X 3 +R 0 0 X 3 3 / 8

35 Residual quantile plots E[Y X ] = X 1, var[y X ] = 1, var(x j) = 1, cor(x j, X k ) = 0.3 t distributed errors Residual quantiles (standardized) 0 0 Standard normal quantiles 35 / 8

36 Transformations If it appears that the linear model assumptions do not hold, it may be possible to continuously transform either Y or X so that the linear model becomes more consistent with the data. 36 / 8

37 Variance stabilizing transformations A common violation of the linear model assumptions is a mean/variance relationship, where EY i and var(y i ) are related. Suppose that var Y i = g(ey i )σ, and let f ( ) be a transform to be applied to the Y i. The goal is to find a transform such that the variances of the transformed responses are constant. Using a Taylor expansion, f (Y i ) f (EY i ) + f (EY i )(Y i EY i ). 37 / 8

38 Variance stabilizing transformations Therefore var f (Y i ) f (EY i ) var(y i ) = f (EY i ) g(ey i )σ. The goal is to find f such that f = 1/ g. Example: Suppose g(z) = z λ. This includes the Poisson regression case λ = 1, where the variance is proportional to the mean, and the case λ = where the standard deviation is proportional to the mean. When λ = 1, f solves f (z) = 1/ z, so f is the square root function. When λ =, f solves f (z) = 1/z, so f is the logarithm function. 38 / 8

39 Log/log regression Suppose we fit a simple linear regression of the form E(log(Y ) log(x )) = α + β log(x ). Suppose the logarithms are base 10. Let X z = X 10 z. Under the model, E(log(Y ) X z ) E(log(Y ) X ) = βz Using the crude approximation log E(Y X ) E(log(Y ) X ), we conclude E(Y X ) is approximately scaled by a factor of 10 βz when X is scaled by a factor of 10 z. This holds for relatively small values of z where the crude approximation holds. Thus in a log/log model, we may say that a f % change in X is approximately associated with a f β % change in the expected response. 39 / 8

40 Maximum likelihood estimation of a data transformation The Box-Cox family of transforms is y y λ 1, λ which makes sense only when all Y i are positive. The Box-Cox family includes the identity (λ = 1), all power transformations such as the square root (λ = 1/) and reciprocal (λ = 1), and the logarithm in the limiting case λ 0. 0 / 8

41 Maximum likelihood estimation of a data transformation Suppose we assume that for some value of λ, the transformed data follow a linear model with Gaussian errors. We can then set out to estimate λ. The joint log-likelihood of the transformed data is n log(π) n log σ 1 σ i (Y (λ) i X i β). Next we transform this back to a likelihood in terms of Y i = g 1 λ This joint log-likelihood is (Y (λ) i ). n log(π) n log σ 1 σ (g λ (Y i ) X i β) + i i log J i where the Jacobian is log J i = log g λ(y i ) = (λ 1) log Y i. 1 / 8

42 Maximum likelihood estimation of a data transformation The joint log likelihood for the Y i is n log(π) n log σ 1 σ (g λ (Y i ) X i β) + (λ 1) i i log Y i. This likelihood is maximized with respect to λ, β, and σ to identify the MLE. / 8

43 Maximum likelihood estimation of a data transformation To do the maximization, let Y (λ) g λ (Y ) denote the transformed observed responses, and let Ŷ (λ) denote the fitted values from regressing Y (λ) on X. Since σ does not appear in the Jacobian, ˆσ λ n 1 Y (λ) Ŷ (λ) will be the maximizing value of σ. Therefore the MLE of β and λ will maximize n log ˆσ λ + (λ 1) i log Y i. 3 / 8

44 Collinearity Diagnostics Collinearity inflates the sampling variances of covariate effect estimates. To understand the effect of collinearity on var ˆβ j, reorder the columns and partition the design matrix X as X = ( X j X 0 ) = ( Xj X j + X j X 0 ) where X 0 is the n p matrix consisting of all columns in X except X j, and Xj is the projection of X j onto col(x 0 ). Therefore ( H X X = X j X j (X j X j ) X 0 X 0 (X j X j ) X 0 X 0 ). var ˆβ j = σ H 1 11, so we want a simple expression for H / 8

45 Collinearity Diagnostics A symmetric block matrix can be inverted using: ( A B C B ) 1 = ( S 1 C 1 B S 1 S 1 BC 1 C 1 + C 1 B S 1 BC 1 ), where S = A BC 1 B. Therefore H 1 1,1 = 1 X j (X j Xj ) P 0 (X j Xj ), where P 0 = X 0 (X 0 X 0) 1 X 0 is the projection matrix onto col(x 0 ). 5 / 8

46 Collinearity Diagnostics Since X j X j and since X j so col(x 0 ), we can write H 1 1,1 = 1 X j X j X j, (X j Xj ) = 0, it follows that X j = X j X j + X j = X j Xj + Xj, H 1 1,1 = 1 Xj. This makes sense, since smaller values of Xj correspond to greater collinearity. 6 / 8

47 Collinearity Diagnostics Let R jx be the coefficient of determination (multiple R ) for the regression of X j on the other covariates. R jx = 1 X j (X j X j ) X j X j = 1 X j X j X j. Combining the two equations yields H 1 11 = 1 X j X j 1 1 Rjx. 7 / 8

48 Collinearity Diagnostics The two factors in the expression H 1 11 = 1 X j X j reflect two different sources of variance of ˆβ j : 1 1 Rjx. 1/ X j X j = 1/ ((n 1) var(x j )) reflects the scaling of X j The variance inflation factor (VIF) 1/(1 Rjx ) is scale-free. It is always greater than or equal to 1, and is equal to 1 only if X j is orthogonal to the other covariates. Large values of the VIF indicate that parameter estimation is strongly affected by collinearity. 8 / 8