Appendix D Methods for Multiple Linear Regression Analysis

Transcription

1 Appendix D Methods for Multiple Linear Regression Analysis D.1. Introduction and Notation. The EPOLLS model equations were derived from a multiple linear regression (MLR) analysis as described in Chapters 9 through 11. This analysis was performed using the SAS System (version 6.08) statistical software from the SAS Institute Inc. MLR methods and associated statistical tests used to develop the EPOLLS model are documented in this appendix. This brief discussion of statistical regression methods is based on more thorough treatments given by Montgomery and Peck (1992) and Freund and Littell (1991). Notation and basic definitions associated with regression analyses include the following: i = index denoting different observations or case studies in a data set j = index denoting different regressor variables in a model n = number of observations in a data sample k = number of regressor variables in a model p = k + 1 = number of coefficients in a regression model u = a random variable µ u = mean of the population of u u = Σu i / n = mean of a sample of u, giving an estimate of population mean (^µ u ) σ 2 u = Σ (u i - µ u ) 2 / n = variance of the population of u, indicating variability about the mean 2 s u = Σ (u i - u) 2 / (n - 1) = variance of a sample of u, giving an estimate of population variance (^σ u ) y i = system response (dependent), which is random and observable, from i th observation in sample x ij = j th regressor (independent) variable, which is constant and known, from i th observation in sample ε i = error (random) between a regression model and the i th observation in sample = coefficients (parameters) of a regression model β j ^θ = an estimator of the true value of the parameter θ ^θ (i) = a parameter calculated with a model fit to all available data except the i th observation ^θ (j) = a parameter calculated with a model using all regressor variables except the j th regressor 306

2 307 D.2. Simple and Multiple Linear Regression. The familiar linear regression analysis involves two coefficients, which define the slope and intercept of a line. A simple linear regression (SLR) model can be written as: y = β 0 + β 1 x + ε (D.1) In fitting a line to a sample data set using SLR, the goal is to find estimates of the coefficients (^β 0 and ^β 1 ) that minimize the error ε. Mathematically, this is done by minimizing the square of the errors between the observed and predicted values of y (hence the name least squares regression). With the fitted model, the predicted value of y for a given x is: ^y = ^β 0 + ^β 1 x (D.2) Multiple linear regression (MLR) is used to fit a model with more than one regressor variable. Analogous to Equation D.1 for SLR, an MLR model can be written using matrix notation: (D.3) where the matrix of the regressor variables [X] is called the design matrix. The first model coefficient (β 0 ) in the vector {β} is the intercept of the model. Hence, the design matrix must be specified as: (D.4) In fitting an MLR model, the goal is to find the "best" estimates of the coefficients {β} that minimize the differences between all of the observed responses (y i ) and the corresponding model predictions (^y i ). In the same manner as for SLR, the coefficients are found by minimizing the sum of the squares of the errors (that is, minimize Σε 2 i = Σ(y i - ^y i ) 2 ) in a least squares regression analysis. The solution for least squares estimators of the model coefficients can be written: (D.5) A related calculation defines the "hat" matrix [H]:

3 308 (D.6) The [H] matrix is used in several model diagnostics described later in this appendix. A multiple linear regression model is called "linear" because only linear coefficients {β} are used. Transforms of the regressor variables (such as 1/x, x 2, x 0.5, e x, ln(x), etc.) are all permitted in an MLR model. A basic assumption in a regression analysis is that the functional form of the model is appropriate and includes all important variables. In addition, four assumptions about the errors {ε} are fundamental to a regression analysis: (1) the mean of the errors is zero, (2) the variances of the errors for all observations are constant, (3) the errors are independent of each other (uncorrelated), and (4) for some statistical tests, the errors are normally distributed. Gross violations of these basic assumptions will yield a poor or biased model. However, if the variances of the errors {ε} are unequal and can be estimated, weighted regression schemes can sometimes be used to obtain a better model. D.3. Category Variables. The regressors in an MLR model are usually quantitative variables that can take on any real value. Frequently, however, the sample data can be sorted into categories. For example, data on the performance of a group of students can be sorted by gender. To consider the effect of this division in an MLR model, a category variable can be introduced into the design matrix [X]. The category variable x 1 might be defined as x 1 =0 for females and x 1 =1 for males. Instead of developing separate models for each group, a category variable allows for the fitting of a single MLR model that is simpler to use and applicable to both groups. Moreover, because all of the model coefficients are determined using the entire data set, a single model with a category variable is much better overall than separate models fit to subsets of the data. However, adding just the category variable to the design matrix will mean, in essence, that only the zero-intercept of the model will change with the category. That is, all other regressors will have the same effect on the system response regardless of the category to which the observation belongs. To consider the effect of a category on other regressors, product or interaction terms should be added to the model. For example, to add the category variable x 1 to a model with the quantitative regressor x j, the regression model should be: y = β 0 + β 1 x 1 + β 2 x j + β 3 x 1 x j + ε (D.7) More than one category variable can be included in the design matrix. To represent categories with more than three levels (like eye color of brown, blue, or green), two category

4 309 variables are needed. For example: x 1 =0, x 2 =0 for brown eyes, x 1 =0, x 2 =1 for blue eyes, and x 1 =1, x 2 =1 for green eyes. A four level category would require three category variables, and so forth. However, to get a valid model, the design matrix must include all possible combinations of these category variables. D.4. Model Quality and Significance of Regressors. The squares of the errors between the observed and predicted observations are used to evaluate how well a regression model fits the sample data. A basic identity in regression analyses is obtained by partitioning the sum of the squares of the variation in the sample data: SST = SSE + SSR (D.8) In words, Equation D.8 means that the total variability in the observed responses [SST=Σ(y i - y) 2 ] is equal to the random variability not explained by the model or model error [SSE=Σ(y i -^y i ) 2 ] plus the systematic variability that is explained by the regression model [SSR=Σ(^y i - y) 2 ]. When adjusted for the associated degrees-of-freedom, the three terms in Equation D.8 lead to the following definitions: MST = SST / (n-1) = Σ( y i - y ) 2 / (n-1) = total mean square of variation in observations MSE = SSE / (n-p) = Σ( y i - ^y i ) 2 / (n-p) = error mean square MSR = SSR / 1 = Σ( ^y i - y ) 2 / 1 = regression mean square Significantly, the MSE value gives an unbiased estimate of the variance in the errors ε i. To express the quality of fit between a regression model and the sample data, the coefficient of multiple determination (R 2 ) is typically used. Ranging in value from 0.0 to 1.0, the coefficient of multiple determination is defined as: (D.9) Higher values of R 2 (from a smaller SSE) indicate a better fit of the model to the sample observations. However, adding any regressor variable to an MLR model, even an irrelevant regressor, yields a smaller SSE and greater R 2. For this reason, R 2 by itself is not a good measure of the quality of fit.

5 310 To overcome this deficiency in R 2, an adjusted value can be used. The adjusted coefficient of multiple determination ( R 2 ) is defined as: (D.10) Because the number of model coefficients (p) is used in computing R 2, the value will not necessarily increase with the addition of any regressor. Hence, R 2 is a more reliable indicator of model quality. The global F-test is used to assess the overall ability of a model to explain at least some of the observed variability in the sample responses. Giving a statistical test for the significance of the regression, the global F-test is performed in the following steps: (1) Null hypothesis: β 1 = β 2 =... = β k = 0. (2) Compute the test statistic F 0 with an analysis-of-variance table: Source Degrees of Freedom Sum of Squares Mean Square F o Regression k SSR MSR F 0, k, n-p = MSR / MSE Error n - p SSE MSE Total n - 1 SST (4) From the F distribution, find F α, k, n-p corresponding to the desired level of significance (α). (5) If F 0, k, n-p > F α, k, n-p, reject the null hypothesis and conclude that at least one β j 0 and at least one model regressor explains some of the response variation. A global F-test only indicates that at least one regressor is significant, but does not indicate which regressors are significant and which are not. To study the significance of each regressor variable, partial F-tests are used. These tests look at the significance of a given regressor in the presence of all other regressor variables in the model. Hence, partial F-tests can be performed for each of the "k" regressors: (1) Null hypothesis: β j = 0. (2) Compute the test statistic F j with: F j, 1, n-p = [ SSR(full model) - SSR(full model without x j ) ] / MSE = (SSR - SSR (j) )/MSE (4) From the F distribution, find F α, 1, n-p corresponding to the desired level of significance (α). (5) If F j, 1, n-p > F α, 1, n-p, reject the null hypothesis and conclude that β j 0 and that the variable x j is a significant regressor in the full model. If a regression model is poorly specified, either because important regressor variables are

6 311 missing or unnecessary variables have been included, the fitted model may be biased. An indication of the model bias is given by the Mallows statistic (C p ), which attempts to measure the overall bias or mean square error in the estimated model parameters. The Mallows statistic is defined with: (D.11) Here, MSE is computed using all available regressors and SSE(p) is computed from a model with only "p" coefficients. Note that when p includes all available regressors, SSE(p) = SSE and C p = p. A low C p value indicates good model predictions but, more importantly, a p-term model with little bias will yield C p p. The C p statistic is most useful in evaluating candidate regression models as discussed in the next section. D.5. Selection of Regressors for Candidate Models. The real challenge in performing a multiple linear regression analysis is to find the "best" set of regressor variables that explain the variation in the observed system responses. A model is desired that not only fits well to the observations, but also yields good predictions of future responses and only includes regressor variables that contribute significantly to the model. This process of finding the best set of regressors for an MLR model is known as variable selection or model building. Given M regressor variables that could be included in a simple MLR model, there are 2 M possible equations that could be written with different combinations of the regressors. With a typically large pool of potential regressor variables, examination of all possible models is simply not practical. Instead, systematic methods are employed to find a subset of the regressor variables that will form an appropriate, "best" model. Actually, different variable selection procedures usually suggest different "best" models so additional analyses and judgment are required to obtain a good, reliable MLR model. Final selection of the best model can be based on the evaluations discussed in Section D.8. Perhaps the simplest variable selection procedure involves an attempt to find a model with maximum R 2 or R 2. This procedure does require fitting all possible models, but the results are ranked to easily identify the best model. All possible models with "k" regressors are evaluated and the one model giving the greatest R 2 or R 2 is tabulated. The maximum R 2 or R 2 found from models with one, two, three, etc. regressors are then plotted as shown in Figure D.1. Recall that R 2 always increases with the addition of more regressor variables, while R 2 may eventually decrease as shown in Figure D.1. A good candidate model can be selected where R 2 reaches a maximum or where the R 2 curve begins to flatten out.

7 312 A stepwise selection procedure relies on partial F-tests to find a group of significant regressor variables. The "best" model is found by adding or eliminating regressors in steps. First, partial F-statistics are computed for every potential regressor and the one variable giving the highest F j is inserted into the model. Next, partial F-statistics are computed for all of the remaining regressors and the one yielding the highest F j, in the presence of the first-selected regressor, is added to the model. However, no regressor is added to the model at this step unless F j exceeds a specified threshold value. Next, all variables in the model are evaluated with partial F-tests to see if each one is still significant. In this step, any regressor that is no longer significant, according to the specified threshold value of F j, is dropped from the model. The selection procedure continues in steps as new regressors are added to the model and then any variables that are no longer significant are dropped. The stepwise selection procedure stops when no other potential regressor yields a partial F greater than the threshold and all regressors in the model remain significant. The threshold or cut-off partial F values, for addition to or elimination from the model, are specified in terms of a level of significance (α). One disadvantage of the stepwise selection procedure is that not all possible combinations of regressor variables are considered for the model. Also, since one final equation is produced in a stepwise procedure, other equally good models may go unrecognized. The Mallows statistic, defined in the previous section, can also be used to find a good set of regressors for an MLR model. As done in the selection procedure based on maximum R 2 or R 2, all possible models with "k" regressors are evaluated. Here, from all possible models with one, two, three, etc. regressors, the one model giving the lowest C p is tabulated. The results are then plotted as shown in Figure D.2. Recall that C p = p when all available regressor variables are used in the model; therefore, the trend line in Figure D.2 converges to the C p = p line. A good candidate model can be selected from the C p plot by remembering that good model predictions are indicated by a low C p value and low model bias is indicated by C p p. D.6. Tests for Multicollinearity. The problem of multicollinearity exists when two or more model regressors are strongly correlated or linearly dependent. In building a regression model, the regressor variables in the design matrix are assumed to be independent of one another. However, when the system behavior is poorly understood, the selected MLR model might include several regressors that each measure, to some degree, similar components of the system response. In this situation, columns of the design matrix may be linearly correlated to a sufficient degree as to create a multicollinearity problem. When significant multicollinearity exists in the data, the mathematical solution used to fit the regression model is unstable. That is, small changes in the regressor values can cause large changes in the parameter estimates and yield an unrealistic model. Multicollinearity is especially problematic if the fitted MLR model is then used to make future predictions. Note that the regressor variables do not have to be totally independent of one

8 313 another, and some degree of correlation within the design matrix is tolerable. Techniques for ascertaining potential problems with multicollinearity are outlined in this section. When serious multicollinearity is detected, the problem can often be eliminated by discarding one or more regressor variables from the model. Possible problems due to multicollinearity may be detected during fitting of a regression model. Common indications of multicollinearity include: (1) Parameter estimates (^β j ) with signs that defy prior knowledge (i.e., a model coefficient with a negative sign when a positive sign is expected). (2) Models with large R 2, or high significance in a global F-test, but in which none of the model variables are significant in partial F-tests. (3) Different model selection procedures yield very different models. (4) Standard errors of the regression coefficients that are large, with respect to the parameter estimates, indicating poor precision in the estimates. The standard error of a coefficient is calculated as se(^β j ) = (MSE C jj ) 0.5 where C jj is the j th diagonal element of ([X] T [X]) -1. A simple form of multicollinearity is caused by pair-wise correlation between any two regressor variables. This can be detected by inspection of the correlation matrix [r] for the regressor values in the design matrix. The empirical correlation between regressors x j and x m, giving one element of [r], is computed with: (D.12) The greater the linear dependence between x j and x m, the closer r jm will be to one (obviously, the diagonals of the correlation matrix (r jj ) are equal to one). As a general rule, multicollinearity may be a problem if, for the off-diagonal terms of the correlation matrix, r jm 0.9. However, the pair-wise correlation coefficients will not indicate multicollinearity problems arising from linear dependencies between combinations of more than two regressors. Multicollinearity can also be detected from the eigenvalues or characteristic roots of the correlation matrix [r]. For a model with k regressors, there will be k eigenvalues, λ j. The ratio of the maximum over the minimum eigenvalues of [r] defines the model condition number:

9 314 (D.13) As a general rule, κ < 100 indicates no serious problem, 100 < κ < 1000 indicates moderate to strong multicollinearity, and κ > 1000 indicates a severe problem with multicollinearity. In addition, the condition index associated with each regressor variable x j is defined as: (D.14) The number of κ j > 1000 indicates the number of linear dependencies in the design matrix. Another indication of potential multicollinearity is obtained from variance inflation factors (VIFs). The VIF associated with regressor x j is computed with: (D.15) where R 2 (j) is the coefficient of multiple determination (R 2 ) from a regression of x j on all other k-1 regressors in the model. Hence, as more of the variation in x j can be explained by the other regressor variables, R 2 (j) will approach one and the VIF j will increase. Large values of VIF j indicate possible multicollinearity associated with regressor x j. In general, a VIF j 5 indicates a possible multicollinearity problem, while a VIF j 10 indicates that multicollinearity is almost certainly a problem. D.7. Tests for Influential Observations. In addition to multicollinearity, another common problem in regression analyses are models adversely affected by influential observations. Three types of influential observations are illustrated in Figure D.3 for a simple linear regression model. Outliers, defined as observations outside the general trend of the data, are a familiar type of influential observation. When an observation does fall within the trend of the data, but is found beyond the range of the other regressors, the resulting influential observation is called a high leverage point. When a high leverage point is also an outlier, that single data point can have a large impact on the regression model and is called a highly influential observation. In this section, methods are presented for detecting influential observations. The presence of influential observations can be detected by computing the PRESS (prediction error sum of squares) statistic defined as:

10 315 (D.16) where ^y (i) is a prediction of the i th observed response made from a model regressed on all of the available data except the i th observation. The PRESS statistic is then compared with the sum of the square of the errors, SSE = Σ( y i - ^y i ) 2. If PRESS is much larger than SSE, influential observations may exist. with: Outliers can be detected from studentized residuals that are defined for the i th observation (D.17) where h ii is the i th diagonal element of the hat matrix [H] defined in Equation D.6. When working with a sufficient number of observations, such that (n-p-1) > 20, a r i > 2.0 indicates that the i th observation might be an outlier. Similarly, a r i > 2.5 is a strong indicator of a likely outlier. In addition, high leverage points can be detected directly from the diagonals of the hat matrix, h ii. As a general rule, an h ii > 2p/n indicates that the i th observation is a possible high leverage point. One diagnostic test for highly influential observations uses the DFFITS statistic. For the i th observation, this statistic is defined as: (D.18) where ^y (i) and MSE (i) are based on a model regressed on all of the available data except the i th observation. A possible highly influential observation is indicated by DFFITS i > 2(p/n) 0.5. Another test for highly influential observations is based on the effects of the i th observation on the estimated model coefficients. The DFBETAS statistic, which is calculated for each j th regressor variable and each i th observation, is defined as:

11 316 (D.19) where ^β j(i) and MSE (i) are computed from a regression on all available data except the i th observation. As a general rule, a possible highly influential observation is indicated by DFBETAS ij > 2/n 0.5. In practice, a highly influential observation will cause consistently high DFBETAS for most of the regressor variables. A third test for highly influential observations is based on Cook's Distance defined as: (D.20) where r i is the studentized residual defined in Equation D.17. Values of D i much larger than all others indicates that the i th observation may be highly influential. All influential observations should be investigated for correctness and accuracy. When using the statistical tests described here, the cutoff values for indicating an influential observation should be used only as a guideline. In practice, those observations giving the strongest indications in these tests should be investigated first. Moreover, a data point should not be discarded simply because it is an influential observation. A single influential data point can sometimes illuminate an important trend in the system response. D.8. Evaluation of Final Model. In addition to the diagnostics for multicollinearity and influence data, graphical methods are useful for evaluating the performance of a regression model. In a simple linear regression model, the adequacy of a linear equation can be easily visualized with a scatter plot of the observed data (x i, y i ). However, with a multiple linear regression model, plots of the observed responses (y i ) versus each regressor variable (x ij ) are of little value. Because the system response is a function of multiple regressors, plots of the observed response versus individual variables often fail to indicate a linear relationship and can be very misleading in evaluating an otherwise good MLR model. Other plots that are more useful in visualizing the performance of an MLR model are described in this section.

12 317 A scatter plot of the predicted response versus the observed response (^y versus y), as shown in Figure D.4, gives a simple indication of model performance. Any model that can explain most of the variation in the observed responses will produce a plot with points clustered around a 45 line. Better models yield less scatter about this ^y=y line. Moreover, the scatter of points about the ^y=y line should remain roughly constant with magnitude. That is, a poor model that is less accurate at larger values of ^y will produce increasing scatter with larger values of y. A scatter plot of the residuals, as shown in Figure D.5, is also useful in evaluating a regression model. Here, the model residuals or errors (e = ^y - y) are plotted against the model predictions (^y). Residual plots are used to visually verify some of the basic assumptions underlying an MLR analysis, as discussed previously in Section D.2. Namely, the residuals (errors) between the model predictions and observed responses should have a mean of zero and a constant variance. Hence, the scatter in the residuals should be fairly uniform and centered about e = 0. A good regression model will produce a scatter in the residuals that is roughly constant with ^y as shown in Figure D.5a. Unsatisfactory models yield a scatter in the residuals that changes with ^y; three common examples are shown in (b), (c), and (d) of Figure D.5. Models producing an unsatisfactory scatter in the residuals can often be improved by transforming y to stabilize the variance in the residuals. For example, a model might be re-defined in terms of ln(y), y 0.5, y -0.5, or 1/y. However, such transformations necessitate bias-reducing adjustments when the model predictions are de-transformed, as discussed further in the next section. A third graphical method for evaluating a multiple linear regression model is based on the idea of partial residuals. Designated with e y i(j), the partial residual of y for x j is defined as: (D.21) where ^y i(j) is a prediction of y i from a regression model using all of the regressors except x j. Similarly, the partial residual of x j is designated with e x i(j) and defined as: (D.22) where ^x i(j) is a prediction of the regressor x ij from a regression of x j on all the other regressor variables. Hence, the partial residual e y i(j) represents the variation in y i not explained by a model that excludes the regressor x j, and the partial residual e x i(j) represents the variation in x j that can not be explained by the other regressor variables. Plotting e y i(j) against e x i(j) in a partial regression plot more clearly shows the influence of x j on y in the presence of all other regressors in the model. Partial regression plots, shown in Figure D.6, can be generated for every regressor variable

13 318 in a model. If the regressor x j is linearly related to the model response, a plot of the partial residuals e y i(j) and e x i(j) will cluster about a line that passes through the origin at a slope equal to β j (the coefficient corresponding to x j in the full model). Moreover, a stronger linear relationship between y and x j will be evidenced by a narrower clustering of the partial residuals. For example, less scatter is seen in the partial regression plot in Figure D.6a than the plot in Figure D.6b; this indicates a stronger relationship with the regressor x 1 than for the regressor x 2. Influence data and their effect on the fit of the model can also be spotted on a partial regression plot as shown in Figure D.6c. Even more useful, the need for linear transformations of a given regressor variable (such as ln(x), 1/x, x 0.5, x 2, etc.) may be suggested by a partial regression plot like that in Figure D.6d. Finally, it is often important to evaluate the ability of a fitted regression model to predict future events. The best way to do this is to gather additional, new data and compare these observed responses with predictions from the model. However, when this is not possible, the data used to fit the model can be split and a cross validation performed. A good regression model can be fit to part of the original data set and still accurately predict the other observations. To perform a double cross-validation: (1) Partition the data into two subsets (say, A and B) with an equal number of observations in each. Assigning individual observations to subset A or B must be done randomly. (2) Using the same model form, fit the model using the data from subset A. Use this model to predict the observations in subset B. (3) Compute the prediction R 2 p,a for the model fit to subset A, as defined in Equation D.23 below. (4) Similarly, fit the model to subset B and use this to predict the observations in subset A. (5) Compute the prediction R 2 p,b for the model fit to subset B. A good model will produce high values of R 2 p for both subsets and these values will be approximately equal (R 2 p,a R 2 p,b). The prediction R 2 p for a model fit to subset A (R 2 p,a) is computed with: (D.23) where n B and y B are the number and mean of the observed responses (y ib ) in the random subset B. Using the model fit to subset A, predictions of the observations in subset B are made to give ^y ia. The prediction R 2 p for a model fit to subset B (R 2 p,b) is computed the same way, with the use of subsets A and B reversed.

14 319 D.9. Predictions from a Regression Model. Often, the purpose of developing a regression model is to allow predictions of future events. In equation form, the predicted system response (^y), for a given set of regressor values (x), is computed with: (D.24) where ^β denotes the fitted regression coefficients. As mentioned in the previous section, the observed system responses are sometimes transformed to stabilize the variance of the model errors, based on an examination of a residuals plot. For example, suppose a model for predicting some response "θ" is desired, and the regression analysis indicates the need to transform θ by taking the square root and fitting the model to θ 0.5. Predictions of ^θ would then be obtained by squaring the model prediction. Unfortunately, the de-transformed prediction is substantially biased and will consistently underpredict the value of θ. To alleviate this problem, Miller (1984) suggests bias-reducing adjustment factors for logarithmic, square root, and inverse transformations of the dependent variable. For a square-root transformation, where a regression model is fit to y i 0.5, the low-bias prediction of ^y given by Miller can be written as: (D.25) where MSE is used as an estimate of the variance of the errors in the fitted model. Because a regression model is simply an equation fit to a database of observed responses, a regression model should not be trusted to make predictions outside the range of the regressor variables used in fitting the model. Hence, the first step in using a regression model should be to verify that the prediction does not require extrapolation beyond the range of the regressor variables in the original data set. This can be done most simply by referring to histograms of the original regressor variables. However, with more than two or three model variables, it is possible to be within the range of each regressor yet still extrapolate beyond the combined range of the variables. Referred to as hidden extrapolation, this problem is illustrated in Figure D.7 for a model with two regressor variables. Hidden extrapolation is especially problematic if the regression model is unstable due to problems of multicollinearity. An indication of possible hidden extrapolation can be made by computing h 0 : (D.26) where [X] is the design matrix used to fit the model and [x] = [ 1 x 1 x 2... x k ] is the row matrix

15 320 of values at which the model prediction is to be made. The computed value of h 0 is then compared with h max, which is the maximum of the diagonal elements of the hat matrix [H] from Equation D.6. As a general rule, extrapolation is indicated when h 0 > h max. Finally, based on how well the model fits the available data, it is possible to construct a prediction interval. For a given set of the regressor values, the actual response is believed to fall within the prediction interval (1-α)% of the time. The upper and lower bounds of the prediction interval are defined by: (D.27) where t α/2, n-p is the tail area of the t-distribution at α/2 for n-p degrees of freedom. The value ^y' is the predicted response at [x], but without the bias-reducing adjustment used with detransformations of y (see Equation D.25). For a model specified with a square-root transformation, ^y' = ( [x]{^β} ) 2. Note that the prediction interval given in Equation D.27 applies to the model prediction of a single system response. The confidence interval, for the predicted mean of multiple responses for a given set of regressor values, is narrower.

16 321 Maximum R 2 Maximum R 2 or R 2 for any k-regressor model Maximum R 2 Candidate model k = number of regressors in model Figure D.1. Selection of a candidate model based on maximum R 2 or R 2. Candidate model with low C p value that is approximately equal to p Minimum C p for any p-term model Trend Line C p = p Full Model p = k +1 = number of model coefficients Figure D.2. Selection of a candidate model from a C p plot.

17 322 Response Variable (y) Outlier High Leverage Point Correct model Model fit with influence data Highly Influential Observation Regressor Variable (x) Figure D.3. Definition of influential observations in a simple linear regression model. Predicted Response ( y ^ ) Perfect Predictions ^y = y 45 Observed Response (y) Figure D.4. Evaluation of final model with a y-y ^ scatter plot.

18 323 Residual, e = y - y^ y^ Residual, e = y - y^ y^ (a) satisfactory (b) funnel Residual, e = y - y^ y^ Residual, e = y - y^ y^ (c) double bow (d) nonlinear Figure D.5. Common patterns in residual plots used to evaluate MLR models (after Montgomery and Peck 1992).

19 324 e y (1) = partial residual of y for x e x (1) = partial residual of x 1 e y (2) = partial residual of y for x 2 Slope = β 1 Slope = β e x (2) = partial residual of x 2 (a) (b) e y (3) = partial residual of y for x 3 0 Slope = β 3 0 e x (3) = partial residual of x 3 Outlier e y (4) = partial residual of y for x 4 0 Slope = β 4 0 (c) e x (4) = partial residual of x 4 (d) Figure D.6. Partial regression plots used to evaluate MLR models.

20 325 x 2 Region of data used to fit model Prediction involving hidden extrapolation Range of regressor x 2 used in fitting model x 1 Range of regressor x 1 used in fitting model Figure D.7. Illustration of hidden extrapolation in a model with two regressor variables (after Montgomery and Peck 1992).