Simple Linear Regression Chapter 11

Transcription

1 Simple Linear Regression Chapter 11 Rationale Frequently decision-making situations require modeling of relationships among business variables. For instance, the amount of sale of a product may be related to its advertising expenditures for marketing, the health of the economy as measured by the stock market performance, and the number of sales people etc? The regression analysis provides tools for modeling a "response" variable with the help of a number of "predictor" variables for predicting and managing risks. The simple linear regression accounts for only one predictor in modeling a response variable. Though of limited practical use, it provides an understanding of the basic tools of the methodology, which can be easily extended to realistic models involving more than one predictor. Objectives Understand: Scatter plot and the simple linear regression model Estimation and significance testing of the slope and intercept parameters Coefficient of determination and coefficient of correlation Confidence and prediction intervals for the mean and individual responses Error analysis for testing normality and homogeneity assumptions Key Terms Scatter plot Correlation Coefficient, coefficient of determination Confidence interval, prediction interval Sum of squares, Anova

2 Regression Analysis A methodology for Model building Introduction The regression methodology models the distribution of a variable, called response, with the help of one or more predictor variables. If only one predictor is studied for its relationship with the response variable, then it is a simple regression analysis. In multiple regression more than one predictor is studied for their relationship with the response. All crucial concepts of the regression methodology follow easily from an understanding of the simple regression analysis. A component of the simple linear regression model is a hypothesized relationship between Y and X or some transform of X. For statistical modeling, the pair (X, Y) is observed for n units to yield a sample of n pairs (x 1, y 1 ), (x 2, y 2 ),...,(x n, y n ). A scatter plot of n points in two dimensions represents the sample graphically. The scatter plot provides a clue to a possible relationship between variables Y and X. In many situations a linear function of X, or a suitably transformed X, will be a good first approximation of the true relationship. The deviation of a hypothesized relationship from the true plus an inherent sampling error is symbolized by ε; usually it is called the random error. If Y is a linear function of X, then Y = α + βx, which is graphically a straight line. When X = 0, Y =α, the Y-intercept. The coefficient β of X represents the amount of change in Y corresponding to one unit change in X. It is called the slope of the line. The linear relationship is hypothesized for investigation should the scatter plot indicate a linear trend. If, for example, the scatter plot indicates a quadratic relationship between Y and X, then a linear relationship may be hypothesized between Y and Z =X 2. A theoretical model represents investigator's beliefs through an algebraic expression in X plus a random error

3 component ε. Once a suitable model that fits the sample is found, it can then be used in decision making process. Examples in Section 11.1 Simple linear regression To understand the basic concepts of regression analysis, consider the following simple linear regression model for investigating a relationship of a response variable Y with a predictor X. Y = α + βx + ε The random error ε is assumed to be normally distributed with mean 0 and variance σ 2. 2 This assumption implies that Y is also normally distributed with µ y = α + βx, and σ y = σ 2. Of this model, α, β and σ 2 are unknown constants, which are to be estimated. If it turns out that β = 0, then X is not a significant predictor of Y. If σ 2 = 0 (merely a wish), then there is a perfect line relationship between Y and X and the parameters β and α can be estimated by just two observations on the pair (Y, X). Estimation and significance testing A simple linear regression analysis proceeds to estimate µ y = α + βx and then to judge its significance. Estimate of µ y, for predictor X at x., is ^ y = a + b x, where the constants a and b are estimates of α and β respectively. Algebraic expressions for computing a and b are found in any text. The standard deviation of points in the scatter plot around the estimated regression line a+bx is S = ( ^ y - y) 2 / (n-2) The standard errors of a, b denoted by S a and S b respectively, are standard feature of any software for the regression analysis. The same is true of the standard error of ^ y, denoted

4 by S ( ^ y ), which depends upon x for which prediction ^ y is desired. It is also computed by software for any specified X value for which the mean of Y or prediction of Y is desired. To test a null hypothesis H 0 : β = β 0 against an alternative H a : β β 0 an b β is computed along with its p value for decision making. To test if X 0 t statistic = S b is a significant predictor of, one would test if the coefficient β were 0. To accomplish this β 0 in the t-statistics is replaced by 0. Likewise, one may proceed to test hypotheses about the intercept paramter α. Coefficient of Determination The coefficient of determination R 2 quantities the significance of a model ^ y = a + bx for estimating the mean and predicting individual Y scores for specified X values. Unless R 2 is high, say 85 % or more; the estimated model may be perceived inadequate. Low R 2 values will suggest the need for investigating additional predictors. The correlation coefficient is a square root of R 2. Its sign is positive if the relationship is increasing and negative if it is decreasing. The correlation coefficient and the slope estimate b must have the same sign. The correlation, expressed as a proportion, is a number between -1 and 1. Its absolute amount quantifies the strength of the linear relationship. If it is close to 1 or -1, the linear relationship is considered highly significant and insignificant if it is close to zero. Confidence and prediction intervals If a model is judged adequate by R 2, then it may be used for estimating the mean µ y and predicting individual response values. For example, if the analysis of a simple linear model

5 finds R 2 sufficiently large, then one may proceed to estimate µ y = α + βx by a confidence interval, and also predict Y response corresponding to a specified predictor value at x. For these tasks, first compute ^ y = a + bx. Then confidence and prediction intervals are given by: ^ y ± S ( ^ y ), where the expression for computing of S ( ^ y ), will vary some what depending upon whether it is for finding a confidence or a prediction interval. Analysis of residuals To verify the normality and homogeneity of variance assumptions of random errors ε, one may compute n residuals e i = y i - y^ i, i = 1, 2,..., n. These residuals are estimates of the random component ε. Draw a histogram or the box plot of the residuals to judge if it suggests a normal distribution for the residuals. Further, the residual e i can be plotted against observed predictor values x i to verify if the assumption of the homogeneity of variance is valid. A resulting plot having random appearance without any bias for different values of predictor X would suggest that the homogeneity assumption is valid. A nonrandom trend would be indicative that the homogeneity assumption does not hold. Examples in section 11.2