Regression Add a line to the plot that fits the data well. Don t do any calculations, just add the line.

Transcription

1 Regression Regression 9.1 Simple Linear Regression The Least Squares Method Example. Consider the following small data set. somedata <- data.frame( x=1:5, y=c(1,3,2,4,4) ) somedata 4 x y y x 1. Add a line to the plot that fits the data well. Don t do any calculations, just add the line. 2. Estimate the slope and intercept of your line by reading them off of the graph 3. Now estimate the residuals for each point relative to your line residual = observed response predicted response 4. Compute the sum of the squared residuals, SSE. Square each residual and add them up.

2 138 Regression For example, suppose we we select a line that passes through (0,1) and (5,4). the equation for this line is y =1+.6x, and it looks like a pretty good fit: my.y <- makefun( * x x) xyplot( y x, data=somedata, xlim=c(0,6), ylim=c(0,5) ) + plotfun( my.y(x) x, col="gray50" ) 4 y x The residuals for this function are resids <- with(somedata, y - my.y(x)) ; resids [1] and SSE is sum(residsˆ2) [1] 2 If your line is a good fit, then SSE will be small. The least squares regression line is the line that has the smallest possible SSE. 1 Thelm() function will find this best fitting line for us. model1 <- lm( y x, data=somedata ); model1 lm(formula = y x, data = somedata) (Intercept) x This says that the equation of the best fit line is ŷ = x 1 Usingcalculus,itiseasytoderiveformulasfortheslopeandinterceptofthisline. Butwewillusesoftwaretodothesecomputations. All statistical packages can perform these calculations for you.

3 Regression 139 xyplot( y x, data=somedata, type=c('p','r') ) + plotfun( my.y(x) x, col="gray50" ) # let's add our previous attempt, too y x We can compute SSE using the resid() function. SSE <- sum ( resid(model1)ˆ2 ); SSE [1] 1.9 As we see, this is a better fit than our first attempt at least according to the least squares criterion. It will be better than any other attempt it is the least squares regression line Properties of the Least Squares Regression Line For a line with equation y = ˆβ 0 + ˆβ 1 x, the residuals are and the sum of the squares of the residuals is e i =y i (ˆβ 0 + ˆβ 1 x) SSE = ei 2 = (y i (ˆβ 0 + ˆβ 1 x)) 2 Simple calculus (which we won t do here) allows us to compute the best ˆβ 0 and ˆβ 1 possible. These best values define the least squares regression line. We always compute these values using software, but it is good to note that the least squares line satisfies two very nice properties. 1. The point (x,y) is on the line. This means that y = ˆβ 0 + ˆβ 1 x (and ˆβ 0 =y ˆβ 1 x) 2. The slope of the line is b =r s y s x where r is the correlationcoefficient: r = 1 xi x yi y n 1 s x s y Since we have a point and the slope, it is easy to compute the equation for the line if we know x, s x, y, s y, and r.

4 140 Regression Explanatory and Response Variables Matter It is important that the explanatory variable be the x variable and the response variable be the y variable when doing regression. If we reverse the roles of y and x we do not get the same model. This is because the residuals are measured vertically (in the y direction) Example: Florida Lakes Example Does the amount of mercury found in fish depend on the ph level of the lake? Fish were captured and ph measured in a number of Florida lakes. We can use this data to explore this question. xyplot(avgmercury ph, data = FloridaLakes, type = c("p", "r")) lm(avgmercury ph, data = FloridaLakes) lm(formula = AvgMercury ph, data = FloridaLakes) (Intercept) ph AvgMercury ph You can get terser output with coef(lm(avgmercury ph, data = FloridaLakes)) # just show me the coefficients (Intercept) ph From these coefficients, we see that our regression equation is AvgMercury=1.531+( 0.152) ph So for example, this suggests that the average average mercury level (yes, that s two averages 2 ) for lake with a ph of 6 is approximately AvgMercury=1.531+( 0.152) 6.0= For each lake, the average mercury level is calculated. Different lakes will have different average mercury levels. Our regression line is estimating the average of these averages for lakes with a certain ph.

5 Regression 141 Using makefun(), we can automate computing the estimated response: Mercury.model <- lm(avgmercury ph, data = FloridaLakes) estimated.avgmercury <- makefun(mercury.model) estimated.avgmercury(6) Example: Inkjet Printers Here s another example in which we want to predict the price of an inkjet printer from the number of pages it prints per minute (ppm). xyplot(price PPM, data = InkjetPrinters, type = c("p", "r")) lm(price PPM, data = InkjetPrinters) lm(formula = Price PPM, data = InkjetPrinters) (Intercept) PPM Price PPM You can get terser output with coef(lm(price PPM, data = InkjetPrinters)) (Intercept) PPM So our regression equation is Price= PPM For example, this suggests that the average price for inkjet printers that print 3 pages per minute is Price= =

6 142 Regression 9.2 Parameter Estimates Interpreting the Coefficients Thecoefficientsofthelinearmodeltellushowtoconstructthelinearfunctionthatweusetoestimateresponse values, but they can be interesting in their own rite as well. The intercept β 0 is the mean response value when the explanatory variable is 0. This may or may not be interesting. Often β 0 is not interesting because we are not interested in the value of the response variable whenthe predictoris 0. (That mightnot even be a possible value for the predictor.) Furthermore, if we do not collect data with values of the explanatory variable near 0, then we will be extrapolating from our data when we talk about the intercept. The estimate for β 1, on the other hand, is nearly always of interest. The slope coefficient β 1 tells us how quickly the response variable changes per unit change in the predictor. This is an interesting value in many moresituations. Furthermore,whenβ 1 =0,thenourmodelsaysthattheaverageresponsedoesnotdependon thepredictoratall. Sowhen0iscontainedintheconfidenceintervalforβ 1 orwecannotrejecth 0 : β 1 =0,then we do not have sufficient evidence to be convinced that our predictor is of any use in predicting the response. Since ˆβ 1 =r s y s x, testing whether β 1 =0 is equivalent to testing whether correlation coefficient ρ = Estimating σ There is one more parameter in our model that we have been mostly ignoring so far: σ (or equivalently σ 2 ). This is the parameter that describes how tightly things should cluster around the regression line. We can estimate σ 2 from our residuals: ˆσ 2 =MSE = i e2 i n 2 ˆσ =RMSE = MSE = i e2 i n 2 The acronyms MSE and RMSE stand for Mean Squared Error and Root Mean Squared Error. The numerator in these expressions is the sum of the squares of the residuals SSE = ei 2. This is precisely the quantity that we were minimizing to get our least squares fit. MSE = SSE DFE where DFE = n 2 is the degrees of freedom associated with the estimation of σ 2 in a simple linear model. We lose two degrees of freedom when we estimate β 0 and β 1, just like we lost 1 degree of freedom when we had to estimate µ in order to compute a sample variance. RMSE = MSE is listed in the summary output for the linear model as the residual standard error because it is the estimated standard deviation of the error terms in the model. summary(mercury.model) i

7 Regression 143 lm(formula = AvgMercury ph, data = FloridaLakes) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) e-10 ph e-06 Residual standard error: on 51 degrees of freedom Multiple R-squared: 0.331,Adjusted R-squared: F-statistic: 25.2 on 1 and 51 DF, p-value: 6.57e-06 We will learn about other parts of this summary output shortly. Much is known about the estimator σ 2, including ˆσ 2 is unbiased (on average it is σ 2 ), and the sampling distribution is related to a Chi-Squared distribution with n 2 degrees of freedom ANOVA for regression and the Correlation Coefficient There is another connection between the correlation coefficient and the least squares regression line. We can think about regression as a way to analyze the variability in the response. anova(lm(avgmercury ph, data = FloridaLakes)) Analysis of Variance Table Response: AvgMercury Df Sum Sq Mean Sq F value Pr(>F) ph e-06 Residuals This is a lot like the ANOVA tables we have seen before. This time: SST = (y y) 2 SSE = (y ŷ) 2 SSM = (ŷ y) 2 SST =SSM+SSE As before, when SSM is large and SSE is small, then the model (ŷ = ˆβ 0 + ˆβ 1 x) explains a lot of the variability and little is left unexplained (SSE). On the other hand, if SSM is small and SSE is large, then the model explains only a little of the variability and most of it is due to things not explained by the model. The percentage of explained variability is denoted r 2 or R 2 :

8 144 Regression For our the Florida lakes study, we see that R 2 = SSM SST = SSM SSM+SSE SSM =2.00 SSE =4.05 SST = =6.05 R 2 = SSM SST = =0.331 This number is listed as Multiple R-squared on the summary output. So ph explains roughly 1/3 of the variability in mercury levels. The other two thirds of the variability in mercury levels is due to other things. (We can think of many things that might matter: size of the lake,depthofthelake,typesoffishinthelake,typesofplantsinthelake,proximitytoindustrialization highways, streets, manufacturing plants, etc.) More complex studies might investigate the effects of several such factors simultaneously. The correlation coefficient The square root of R 2 (with a sign to indicate whether the association between explanatory and response variables is positive or negative) is the correlation coefficient, R (or r). As a reminder, here are some important facts about R: 1. R is always between -1 and 1 2. R is 1 or -1 only if all the dots fall exactly on a line. 3. If the relationship between the explanatory and response variables is not roughly linear, then R is not a very useful number. (And simple linear regression is not very useful either). 4. For linear relationships, R is a measure of the strength of the relationship. If R is close to 1 or -1, the linearassociationisstrong. Ifitiscloserto0,thelinearassociationisweak(withlotsofscatteraboutthe best fit line). 5. R is unitless if we change the units of our measurements (from English to metric, for example) it will not affect the value of R. 9.3 Confidence Intervals and Hypothesis Tests Bootstrap So how good are these estimates? We would like have interval estimates rather than just point estimates. One way to get interval estimates for the coefficients is to use the bootstrap. Florida Lakes boot.lakes <- do(1000) * lm(avgmercury ph, data = resample(floridalakes)) head(boot.lakes, 2)

9 Regression 145 Intercept ph sigma r.squared dotplot( ph, data = boot.lakes, width = 03) dotplot( Intercept, data = boot.lakes, width = 2) histogram( ph, data = boot.lakes, width = 1) histogram( Intercept, data = boot.lakes, width = 0.1) ph Count Intercept Count ph Density Intercept Density cdata(0.95, ph, boot.lakes) low hi central.p cdata(0.95, Intercept, boot.lakes) low hi central.p Inkjet Printers boot.printers <- do(1000) * lm(price PPM, data = resample(inkjetprinters)) head(boot.printers, 2) Intercept PPM sigma r.squared

10 146 Regression histogram( PPM, data = boot.printers) histogram( Intercept, data = boot.printers) cdata(0.95, PPM, boot.printers) low hi central.p cdata(0.95, Intercept, boot.printers) low hi central.p Density Density PPM Intercept Using Standard Errors We can also compute confidence intervals using estimate±t SE For t we use n 2 degrees of freedom. (The other two degrees of freedom go for estimating the intercept and the slope). This (and much of the regression analysis) is based on the assumptions that 1. The mean values of y (in the population) for each value of x lie along a line. 2. Individual values of y (in the population) for each value of x are normally distributed. 3. The standard deviations of these normal distributions are the same no matter what x is. As before, we have two ways we can estimate the standard errors. 1. Compute the standard deviation of the appropriate bootstrap distribution. This should work well provided our bootstrap distribution is something resembling a normal distribution.

11 Regression Use formulas to compute the standard errors from summary statistics. The formulas for SE are a bit more complicated in this case, but R will standard error estimates for us, so we don t need to know the formulas. Florida Lakes The t value is based on DFE, the degrees of freedom for the errors (residuals). For simple linear regression, the error degrees of freedom is n 2=51. For a 95% confidence interval, we first compute t : t.star <- qt(0.975, df = 51) t.star [1] 2.01 Using the bootstrap distribution. To get the standard errors from or bootstrap distribution, we can use sd(). sd( Intercept, data = boot.lakes) [1] sd( ph, data = boot.lakes) [1] 257 The confint() function can be applied to bootstrap distributions to make this even simpler. We even have a choice between (a) using the standard error as estimated by taking the standard deviation of the bootstrap distribution or (b) using the percentile method: confint(boot.lakes) # 95% CIs for each parameter name lower upper level method estimate margin.of.error 1 Intercept stderr ph stderr sigma stderr r.squared stderr confint(boot.lakes, method = "perc") # 95% CIs for each parameter; percentile method name lower upper level method 1 Intercept quantile 2 ph quantile 3 sigma quantile 4 r.squared quantile confint(boot.lakes, "ph", level = 0.98, method = c("stderr", "perc")) # 98% CI just for ph, both methods

12 148 Regression name lower upper level method estimate margin.of.error 1 ph stderr ph quantile NA NA Using formulas for standard error. The summary output for a linear model includes the formula-based standard error estimates for each parameter. summary(lm(avgmercury ph, data = resample(floridalakes))) lm(formula = AvgMercury ph, data = resample(floridalakes)) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) e-08 ph Residual standard error: on 51 degrees of freedom Multiple R-squared: 0.248,Adjusted R-squared: F-statistic: 16.8 on 1 and 51 DF, p-value: 0015 So we get the following confidence intervals for intercept 1.63±t SE 1.63± ±0.425 and the slope 0.153±t SE ± 64 The confint() function can also be used to simplify these calculations. confint(lm(avgmercury ph, data = resample(floridalakes))) # 95% CI 2.5 % 97.5 % (Intercept) ph confint(lm(avgmercury ph, data = resample(floridalakes)), level = 0.99) # 99% CI % 99.5 % (Intercept) ph

13 Regression 149 Inkjet Printers summary(lm(price PPM, data = resample(inkjetprinters))) lm(formula = Price PPM, data = resample(inkjetprinters)) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) PPM e-07 Residual standard error: 50.7 on 18 degrees of freedom Multiple R-squared: 0.799,Adjusted R-squared: F-statistic: 71.4 on 1 and 18 DF, p-value: 1.11e-07 confint(lm(price PPM, data = resample(inkjetprinters)), "PPM") 2.5 % 97.5 % PPM confint(boot.printers, "PPM") name lower upper level method estimate margin.of.error 1 PPM stderr Hypothesis Tests The summary of linear models includes the results of some hypothesis tests: summary(lm(avgmercury ph, data = FloridaLakes)) lm(formula = AvgMercury ph, data = FloridaLakes) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) e-10 ph e-06

14 150 Regression Residual standard error: on 51 degrees of freedom Multiple R-squared: 0.331,Adjusted R-squared: F-statistic: 25.2 on 1 and 51 DF, p-value: 6.57e-06 Of these the most interesting is the one in the row labeledph. This is a test of H 0 :β 1 =0 H a :β 1 0 The test statistic t = ˆβ 1 0 SE is converted to a p-value using a t-distribution with DFE =n 2 degrees of freedom. t < / 303; t [1] * pt( t, df = 51 ) # p-value [1] 6.52e-06 We could also estimate this p-value using randomization. If β 1 =0, then the model equation becomes response=β 0 +ε so the explanatory variable doesn t matter for determining the response. This means we can simulate a world in which the null hypothesis is true by shuffling the explanatory variable: rand.lakes <- do(1000) * lm(avgmercury shuffle(ph), data = FloridaLakes) histogram( ph, data = rand.lakes, v = 0) 2 * prop( (ph <= ), data = rand.lakes) # p-value from randomization distribution target level: TRUE; other levels: FALSE TRUE 0 Density ph In this case, none of our 1000 resamples produced such a small value for ˆβ 1. This is consistent with the small p-value computed previously.

15 Regression Making Predictions Point Estimates for Response It may be very interesting to make predictions when the explanatory variable has some other value, however. There are two ways to do this in R. One uses the predict() function. It is simpler, however, to use the makefun() function in the mosaic package, so that s the approach we will use here. First, let s build our linear model and store it. lakes.model <- lm(avgmercury ph, data = FloridaLakes) coef(lakes.model) (Intercept) ph Now let s create a function that will estimate values ofavgmercury for a given value ofph: mercury <- makefun(lakes.model) We can now input a ph value and see what our least squares regression line predicts for the average mercury level in the fish: mercury(ph = 5) # estimate AvgMercury when ph is mercury(ph = 7) # estimate AvgMercury when ph is Interval Estimates for the Mean and Individual Response R can compute two kinds of confidence intervals for the response for a given value 1. A confidence interval for the mean response for a given explanatory value can be computed by adding interval='confidence'. mercury(ph = 5, interval = "confidence") fit lwr upr An interval for an individual response(called a prediction interval to avoid confusion with the confidence interval above) can be computed by adding interval='prediction' instead.

16 152 Regression mercury(ph = 5, interval = "prediction") fit lwr upr Prediction intervals (a) are much wider than confidence intervals (b) are very sensitive to the assumption that the population normal for each value of the predictor. (c) are (for a 95% confidence level) a little bit wider than ŷ±2se where SE is the residual standard error reported in the summary output. The prediction interval is a little wider because it takes into account the uncertainty in our estimated slope and intercept as well as the variability of responses around the true regression line. The figure below shows the confidence (dotted) and prediction (dashed) intervals as bands around the regression line. require(fastr) xyplot(avgmercury ph, data = FloridaLakes, panel = panel.lmbands, cex = 0.6, alpha = ) AvgMercury ph As the graph illustrates, the intervals are narrow near the center of the data and wider near the edges of the data. It is not safe to extrapolate beyond the data(without additional information), since there is no data to let us know whether the pattern of the data extends. 9.5 Regression Cautions Don t Fit a Line If a Line Doesn t Fit Whendoingregressionyoushouldalwayslookatthedatatoseeifalineisagoodfit. Ifitisnot,itmaybethat a suitable transformation of one or both of the variables may improve things. Or perhaps some other method is required.

17 Regression 153 Anscombe s Data Anscombe illustrated the importance of looking at the data by concocting an interesting data set. Notice how similar the numerical summaries are for these for pairs of variables summary(lm(y1 x1, anscombe)) lm(formula = y1 x1, data = anscombe) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) x Residual standard error: 1.24 on 9 degrees of freedom Multiple R-squared: 0.667,Adjusted R-squared: F-statistic: 18 on 1 and 9 DF, p-value: 0217 summary(lm(y2 x2, anscombe)) lm(formula = y2 x2, data = anscombe) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) x Residual standard error: 1.24 on 9 degrees of freedom Multiple R-squared: 0.666,Adjusted R-squared: F-statistic: 18 on 1 and 9 DF, p-value: 0218 summary(lm(y3 x3, anscombe)) lm(formula = y3 x3, data = anscombe) Residuals: Min 1Q Median 3Q Max

18 154 Regression Estimate Std. Error t value Pr(> t ) (Intercept) x Residual standard error: 1.24 on 9 degrees of freedom Multiple R-squared: 0.666,Adjusted R-squared: F-statistic: 18 on 1 and 9 DF, p-value: 0218 summary(lm(y4 x4, anscombe)) lm(formula = y4 x4, data = anscombe) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) x Residual standard error: 1.24 on 9 degrees of freedom Multiple R-squared: 0.667,Adjusted R-squared: 0.63 F-statistic: 18 on 1 and 9 DF, p-value: 0216 But the plots reveal that very different things are going on y x Outliers in Regression Outliers can be very influential in regression, especially in small data sets, and especially if they occur for extreme values of the explanatory variable. Outliers cannot be removed just because we don t like them, but they should be explored to see what is going on (data entry error? special case? etc.) Some researchers will do leave-one-out analysis, or leave some out analysis where they refit the regression with each data point left out once. If the regression summary changes very little when we do this, this means that the regression line is summarizing information that is shared among all the points relatively equally. But

19 Regression 155 ifremovingoneorasmallnumberofvaluesmakesadramaticchange,thenweknowthatthatpointisexerting a lot of influence over the resulting analysis (a cause for caution) Residual Plots In addition to scatter plots of the response vs. the explanatory variable, we can also create plots of the residuals of the model vs either the explanatory variable or the fitted values (ŷ). The latter works in a wider variety of settings (including multiple regression and two-way ANOVA). model1 <- lm(y1 x1, data = anscombe) model2 <- lm(y2 x2, data = anscombe) model3 <- lm(y3 x3, data = anscombe) model4 <- lm(y4 x4, data = anscombe) xyplot(resid(model1) x1, data = anscombe) xyplot(resid(model1) fitted(model1), data = anscombe) resid(model1) resid(model1) x1 fitted(model1) xyplot(resid(model2) x2, data = anscombe) xyplot(resid(model2) fitted(model2), data = anscombe) resid(model2) resid(model2) x2 fitted(model2) You can make similar plots for models 3 and 4. The main advantage of these plots is that they use the vertical space in the plot more efficiently. This is especially important when the size of the residuals is small relative to the range of the response variable. Returningto our Florida lakes,wesee that thingslook reasonablefor themodel wehavebeen fitting(but stay tuned for the next section).

20 156 Regression lake.model <- lm(avgmercury ph, data = FloridaLakes) xyplot(avgmercury ph, data = FloridaLakes, type = c("p", "r")) xyplot(resid(lake.model) fitted(lake.model), data = FloridaLakes) AvgMercury resid(lake.model) ph fitted(lake.model) We are hoping not to see any strong patterns in these residual plots Checking the Distribution of the Residuals for Normality Residuals should be checked to see that the distribution looks approximately normal and that that standard deviation remains consistent across the range of our data (and across time). histogram( resid(lakes.model)) xqqmath( resid(lakes.model)) Density 1.5 resid(lakes.model) resid(lakes.model) qnorm The normal-quantile plot shown above is designed so that the points will fall along a straight line when the underlying distribution is exactly normal. As the distribution becomes less and less normal, the normalquantile will look less and less like a straight line. Similar plots (and some others as well) can also be made with mplot(lakes.model) In this case things don t look quite as good as we would like on the normality front. The residuals are a bit too skewed (too many large positive residuals). Using a log transformation on the response (see below) might improve things.

21 Regression Tranformations Transformations of one or both variables can change the shape of the relationship (from non-linear to linear, we hope) and also the distribution of the residuals. In biological applications, a logarithmic transformation is often useful. lakes.model2 <- lm(log(avgmercury) ph, data = FloridaLakes) xyplot(log(avgmercury) ph, data = FloridaLakes, type = c("p", "r")) summary(lakes.model2) lm(formula = log(avgmercury) ph, data = FloridaLakes) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) e-04 ph e-07 Residual standard error: on 51 degrees of freedom Multiple R-squared: 0.381,Adjusted R-squared: F-statistic: 31.4 on 1 and 51 DF, p-value: 8.54e-07 log(avgmercury) ph If we like, we can show the new model fit overlaid on the original data: xyplot(avgmercury ph, data = FloridaLakes, main = "untransformed model", type = c("p", "r")) xyplot(avgmercury ph, data = FloridaLakes, main = "log transformed model") Hg <- makefun(lakes.model2) # turn model into a function plotfun(exp(hg(ph)) ph, add = TRUE) # add this function to the plot

22 158 Regression untransformed model log transformed model AvgMercury AvgMercury ph ph log transformed model AvgMercury ph A logarithmic transformation of AvgMercury improves the normality of the residuals. histogram( resid(lakes.model2)) qqmath( resid(lakes.model2)) xyplot(resid(lakes.model2) ph, data = FloridaLakes) xyplot(resid(lakes.model2) fitted(lakes.model2)) Density resid(lakes.model2) resid(lakes.model2) qnorm

23 Regression 159 resid(lakes.model2) resid(lakes.model2) ph fitted(lakes.model2) The absolute values of the residuals are perhaps a bit larger when the ph is higher (and fits are smaller), although this is exagerated somewhat in the plots because there is so little data with very small ph values. If we look at square roots of standardized residuals this effect is not as pronounced: mplot(lakes.model2, w = 3) [[1]] Scale Location Standardized residuals Fitted Value On balance, the log transformation seems to improve the situation and is to be preferred over the original model.