Chapter 9. Multiple Linear Regression Analysis

Transcription

1 Chapter 9 Multiple Linear Regression Analysis In Chapter 8, we studied simple linear regression. In simple regression we deal with one independent variable and one dependent variable. In this chapter, we will study multiple regression in which we deal with more than one independent variable. We still only have one dependent variable. As an example, think about sales revenue as a dependent variable. It depends not just on advertising expense but other factors as well such as the product price, product quality, price of competing products, quality of competing products, brand loyalty etc. When we try to predict sales revenue using several factors (or variables) we make use of multiple regression models. This is just one example. In general, when we try to predict any dependent variable using several factors, we make use of multiple regression. Scatter Plots: Remember in simple regression we first drew a scatterplot of the data to give us a visual representation of the relationship between two variables? When we have two independent variables and one dependent variable, the scatterplot becomes a three-dimensional graph. The dependent variable is on the vertical axis and the two independent variables are on the two horizontal axes at the base. You can imagine a cube with a swarm of points in the space within the cube. But when we have three or more independent variables and a dependent variable, we cannot imagine a scatterplot because we would have to imagine a four-dimensional or a higher dimensional box, something you and I as human beings simply cannot do. But it is not absolutely necessary to have scatterplots. If you can draw scatterplots, it s great, but if we cannot, it s no big deal. We will not get a visual impression of the relationship, but we can still get the mathematical representation of the relationship using multiple regression techniques. The Population Model: Remember in simple regression we had a population model that looked like this: y = α + βx + ε. In multiple regression since there are several independent variables, the model is a little bit more complex. Suppose there are k independent variables x 1, x 2,, x k. The population model will look like this: y = α + β 1 x 1 + β 2 x β k.x k + ε The Expected value of y is given by E(y) = α + β 1 x 1 + β 2 x β k.x k Just like in simple regression, these models have the y-intercept (α), but instead of one slope coefficient, we have k slope coefficients, one corresponding to each of the k independent variables. Our job is to estimate α and the k slope coefficients. We do this using sample data. The Sample Model: The sample model looks like this:, where a, b 1, b 2, b k are estimates of α, β 1, β 2, β k respectively. As in simple regression, a is called the y-intercept and b 1, b 2, b k are the slope coefficients of the k independent variables. Interpretation of slope coefficients: The interpretation of the slope coefficients in multiple regression is similar to the interpretation in simple regression with one important difference. The difference is that we have to mention that other independent variables are constant. I will explain the difference with the help of an example. Here is how we would interpret b 1 : As x 1 increases by one unit, the estimated value of y increases by an average of b 1 units while other variables stay constant. It is important to mention while other variables stay constant. Only stating that y increases by an average of b 1 as x 1 increases by one unit would be incomplete. How to perform Multiple Regression in Excel? If you know how to perform Simple Regression in Excel (which you do), you already know how to perform multiple regression. The only difference is that when specifying the X-range, you specify multiple columns 1

2 instead of one column of data. The output also looks very much like that of a simple regression. The only difference is that in the third panel, you will see extra rows. In simple regression, you see two rows, one for the y-intercept and one for the independent variable. In multiple regression, if you have k independent variables, you will see k+1 rows. One row for the y-intercept and k rows for the k independent variables. Let s look at an example: Adv. Expense ('000s) Price Sales (millions) Figure 1: Data on Sales, Adv. Expense and Price Say we have collected some sample data on Sales, Advertising Expense and Prices as shown in Figure 1. We want to predict Sales using two independent variables Adv. Expense and Price. As you might guess, as price increases, sales will tend to decrease, and as advertising expense increases, sales will tend to increase. So we want to predict sales for a particular price and a particular level of advertising expense. For this type of problem, we use multiple regression. In this example, we have one dependent variable (Sales) and two independent variables (Adv. Expense and Price). Anytime we have two or more independent variable, we use multiple regression. Luckily, in Excel, it is just as easy to run multiple regression as it is to run simple regression. In fact, Excel does not even distinguish between the two. It just calls it Regression. If we select only one column of data for the X-Range, then we are running simple regression. If we select two or more columns of data for the X-Range, then we are running multiple regression. Let s look at the regression output of the above data, shown in Figure 2. Figure 2: Regression Output using Excel for data in Figure 1. 2

3 Using the regression output in Figure 2, we can write the sample model as follows: y-hat = X X 2, where y-hat is the estimated average Sales (in millions of dollars) and X 1 is the Advertising expense (in thousands of dollars) and X 2 is Price (in dollars). Please note the coefficient values. The coefficient of X 1 is How do we interpret this coefficient? For every extra one thousand dollars spent in advertising will increase sales by million or 7.11 thousand dollars at a given price level. The coefficient of X 2 is How to interpret this coefficient? For every extra one dollar increase in price, the total sales will decrease by million at a given level of advertising expense. Please note that since the sign of the coefficient for the price variable is negative, we should be careful in saying that an increase in the price will decrease sales. It is also important to state that the advertising expense stays constant at some level. Notice that in the regression output, the third panel had three rows of information - the first row for the y- intercept and two for the two independent variables. R-Square in Multiple Regression: The R-Square value has the same interpretation in multiple regression as in simple regression i.e. it represents the proportion of variation in the dependent variable that is explained by the independent variables. As we add more independent variables, the R-Square always gets better, i.e. adding an extra independent variable can only increase the proportion of the variation explained by the independent variables. Can R-Square ever decrease by adding an extra independent variable? No. R-Square can never decrease by adding an extra independent variable. Even if an independent variable has no correlation whatsoever with the dependent variable, even then the R-square value will not decrease by including this variable. It will only go up, even if very slightly. It is important to understand the consequences of this phenomenon. Because the R-square value keeps increasing as you keep adding independent variables and since the R-square value is often used as a benchmark to evaluate a model s explanatory power, there is a tendency for model builders to keep adding more independent variables in an effort to improve the R-square value. But this practice is not a very good one from the decision maker s point of view. To resolve this problem, statisticians have invented another measure called the Adjusted R-square. The Adjusted R-Square The adjusted R-square is adjusted for the number of independent variables. So if an unrelated independent variable is added to a model, it might increase the R-square value a little bit but it will decrease the adjusted R-square value. It is for this reason that when evaluating a multiple regression model that we look at the adjusted R-square value instead of the R-square value. In our example, the R-square value is and the adjusted R-square is The adjusted R-square value will always be smaller than the R-square value. Hypotheses for the Regression Coefficients In regression, we are interested in the statistical significance of the slope coefficients of the variables. We test the significance of the coefficients using the hypothesis testing procedure we have learnt in previous chapters. Let me explain this procedure with the help of the following examples: 3

4 Example 1: Test whether the variable advertising expense is significant. Null Hypothesis: H 0 : β 1 = 0 Alt. Hypothesis: H a : β 1 0 The Test Statistic: t-value = b 1 /s b1 = / = Critical t-value: two tailed t value for alpha of 0.05 for degree of freedom 5 = +/ The deg. Of freedom is n-3 = 8-3 = 5. If there are k independent variables, the degree of freedom is n (k+1). The p-value: corresponding to the test statistic is Decision: Since the test statistic is less than the critical value and correspondingly, since the p-value is larger than 0.05, we fail to reject the null hypothesis. Conclusion: At a significance level of 0.05, there is not sufficient evidence that the variable advertising expense is making any difference on Sales. Example 2: Test whether the variable Price is significant Null Hypothesis: H 0 : β 2 = 0 Alt. Hypothesis: H a : β 2 0 The Test Statistic: t-value = b 2 /s b2 = / = Critical t-value: two tailed t value for alpha of 0.05 for degree of freedom 5 = +/ The p-value: corresponding to the test statistic is Decision: Since the p-value is much smaller than 0.05, we reject the null hypothesis. Note that the t-statistic value is which is in the rejection region because the rejection region is below and above Conclusion: At a significance level of 0.05, there is sufficient evidence that the variable price is making a difference on Sales. An important insight: In the above two examples, we found one variable (price) to be significant while the other variable (advertising expense) to be not significant. I want you to note that for the significant variable, in this case, price, both the lower and the upper confidence limits have the same sign (in this case, negative). For the nonsignificant variable, in this case, advertising expense, the lower confidence limit is negative while the upper confidence limit is positive. You will always see that for non-significant variables, the lower and the upper confidence limits will have different signs. Because the lower limit is negative and the upper limit is positive, zero lies within the confidence interval. It implies that slope could be zero, which implies that the independent variable has no effect on the dependent variable. Which is why we say that the slope is not significant. 4

5 Hypotheses for the Overall Model: Besides performing hypotheses testing for individual variables, we can also perform hypothesis testing for testing the significance of the overall model. Let me explain with the help of an example: Null Hypothesis: H 0 : β 1 = β 2 = 0 Alt. Hypothesis: H a : At least one of the βs is not zero The Test Statistic: F-value = The p-value: corresponding to the test statistic is Decision: Since the since the p-value is smaller than 0.05, we reject the null hypothesis. Conclusion: At a significance level of 0.05, there is sufficient evidence that the model is significant i.e., advertising expense and price together have an effect on Sales. Note: For testing the significance of the overall model, the test statistic used is the F-statistic. For testing the significance of individual coefficients, the test statistic used is the t-statistic. You may recall that in ANOVA, we test hypotheses about several means. In ANOVA we use the F statistic. Similarly here, for the overall model, we are testing the significance of several betas and therefore we use the F statistic found in the ANOVA panel of the Regression output. Which Independent Variable is more significant? Whenever we run a multiple regression model involving several independent variables, an obvious question is which one of the many independent variables is the most statistically significant. The answer is rather straightforward. You basically compare the p-values of the coefficients of each variable. The variable whose slope coefficient has the least p-value is the most statistically significant. In our example, the p-value for Advertising expense is whereas the p-value for Price is Clearly Price is the more significant variable here. Please note that the absolute value of the slope coefficient does not tell us anything about the significance of the variable. Practical vs. Statistical Significance If the magnitude of the coefficient of the variable is very small, it implies that the dependent variable does not increase (or decrease) much if we increase the independent variable by one unit. Therefore there may be a tendency to treat that variable as insignificant from a practical point of view. However, one has to be cautious. Even if the magnitude of the coefficient is small, it may be practically significant because the units of the dependent variable may be a large unit. In our example, the unit of our dependent variable (Sales) is millions of dollars. So even though the magnitude of the coefficient is rather small (0.0071), since it is in millions, if we convert it to dollars, the coefficient value is really 7,112, which is fairly large and therefore practically significant because spending an extra one thousand dollars in advertising expense will increase sales by 7,112. But suppose the unit of the dependent variable is small and the magnitude of the coefficient is small, then there may not be a high practical significance for this variable even but as long as its p-value is smaller than the significance level (alpha) the coefficient is considered statistically significant. 5

6 Summary of the Chapter 1. In Multiple Regression, we have one dependent variable and two or more independent variables. 2. In Linear Multiple Regression, the population model is: y = α + β 1 x 1 + β 2 x β k.x k + ε 3. The sample regression model is: 4. In multiple regression, the Adjusted R-square is a better measure for the coefficient of determination than R-square because the adjusted R-square penalizes addition of more explanatory variable. 5. The independent variable with the least p-value is considered the most significant variable. 6. A high F-value of a model signifies a significant model. 7. A variable may be statistically very significant, but in practice, it may or may not have a high coefficient value to be of any practical use. 6