Chapter 8. Simple Linear Regression Analysis (Part-I)

Transcription

1 Chapter 8 Simple Linear Regression Analysis (Part-I) In Chapter 7, we looked at whether two categorical variables were dependent on each other. You can think of dependence as being similar to a relationship i.e. if one variable changes, the other also changes. In this chapter we will look at relationships between two numeric variables instead of two categorical variables. For example, we will be concerned with questions like - is there a relationship between mortgage interest rates and demand for housing? You might guess that as interest rates decrease, the affordability of a house increases and therefore demand for housing increases. Is there a relationship between new housing demand and demand for furniture? You might guess that as more new houses are built, there will be need for more furniture and hence demand for furniture will rise. Is there a relationship between advertising expense and sales? Clearly, the more you advertise, the more likely you are to sell. In all these examples, we noticed that as one variable increased (or decreased) the other either increased or decreased. In business decision making, it helps if the decision maker knows the relationships between some variables over which he or she has control. For example, if I am the marketing manager, it will help me in my decisions if I know by how much will sales revenue increase if I spend an extra $1,000 in advertising. After all, if sales only increase by $500 when I spend an extra $1,000 in advertising, then I will be better off not spending the extra money in advertising. In regression analysis we try to learn underlying relationships between two (or more) numerical variables. Before we get into regression analysis, let us first learn two very important terms that have to do with relationships between two numeric variables. The two terms are Covariance and Correlation. Both these are essentially measures of relationship between two numeric variables. Just like the mean is a measure of central tendency and standard deviation is a measure of variation, covariance and correlation are measures of relationships between two variables. If I have some data involving two variables, I can easily calculate these measures. I say we can easily calculate these measures because Excel makes it easy. In the old days (when I was first learning Statistics), things were different. Covariance and correlation actually involve huge formulas and my Statistics teacher could not claim that they could be easily calculated. But as far as we are concerned we just need to know the name of the Excel function to calculate these two measures. The Excel function for covariance is =COVAR() and for correlation, it is =CORREL(). Computing these measures becomes as simple as computing the mean or the standard deviation. Let s look at an example in Excel. Let s say we have the following data on advertising expense (in thousands of dollars) and sales (in millions of dollars). Advertising Expense (in thousands) Sales (in millions) Figure 1: Data on Advertising vs. Sales Using =COVAR() and =CORREL() functions we can easily calculate the covariance and correlation. Please see Figure 2. 1

2 Figure 2: the COVAR and CORREL functions of Excel In Figure 2, you can see the =COVAR() and =CORREL() functions in action. Let s discuss these measures now. Note that both covariance and correlation in Figure 2 are positive. Remember this - whenever covariance is positive, correlation is also positive. Similarly whenever covariance is negative, correlation is also negative. A positive value of covariance (and hence of correlation) signifies that the relationship is positive, which means that if the value of one variable increases, the value of the other also increases. Similarly, a negative value of covariance (and hence of correlation) signifies a negative relationship, i.e., when one variable increases, the other decreases or vice versa. Note that if one value decreases, the other also decreases, we have a positive relationship. To have a negative relationship, if one value increases, the other must decrease. What about the magnitudes of covariance and correlation? In our example, the covariance is and the correlation is What information do these magnitudes convey? The value of covariance really depends on the units of measurement chosen. In our example, for instance, we are displaying advertising expense in thousands and sales in millions. Suppose we were displaying both advertising expense and sales in thousands, then the Sales column would have values like 11,000, 17,000 and so on. Consequently, the covariance value will become much larger. I want you to try this out. Try replacing the values in the Sales column so they are displayed in thousands. You will find that the covariance becomes 151,944.44, but the correlation stays the same at So the lesson to be learnt is this: the value of covariance depends on the units of measurement chosen for the variables, but the value of correlation does not. The magnitude of covariance doesn t tell us much. But the magnitude of correlation tells us a lot. How to interpret the magnitude of the correlation coefficient? Note that I used the term correlation coefficient this time. Some people like to say correlation coefficient while others just like to say correlation. It is OK to say the correlation between these two variables is 0.96 as it is to say that the correlation coefficient between these two variables is So what does a value of 0.96 tell us? I will tell you that the correlation coefficient varies between -1 and +1. If your correlation coefficient ever comes out to be either less than -1 or greater than +1, please know that you made a mistake in your calculations. Making mistakes in calculations used to be fairly common in the pre-excel days. With Excel, as long as you specified your ranges correctly, there is no chance of making a calculation error. At any rate, you should know that the correlation coefficient can never exceed 1 and can never ever be less than -1. A correlation coefficient of +1 or -1 implies a perfect correlation. It is rare to find two variables in 2

3 business or social sciences that give a perfect correlation. In the physical sciences you can find examples of perfect correlation. For example the correlation between the area of a square of some width and the area of a circle with the radius equal to the width of the square will give a perfect correlation of +1. A correlation close to +1, such as 0.96 in our example, is supposed to be a very strong correlation. It s a step below perfect correlation but any correlation coefficient value of 0.9 or above is considered very strong. A correlation coefficient of 0.7 to 0.9 is considered strong. A correlation value close to 0, such as less than 0.1 is considered a very weak correlation. What about a correlation value of -0.9 or below? Any value close to -1 is also considered very strong. The relationship just happens to be negative, i.e. when one variable decreases, the other increases, or vice versa. A negative relationship is not the same as weak relationship. A correlation coefficient close to -1 implies a very strong negative relationship. Note that these cutoffs of 0.1 or 0.7 or 0.9 that I am using are fairly arbitrary. You really have to look at the context of the problem to evaluate whether the relationship is strong or weak. Now that we understand covariance and correlation, we are almost ready to learn about regression analysis. But before we jump into regression analysis, let me talk about scatter plots. Scatter plots: A scatter plot is nothing but a graph of the data points of the two numerical variables that we are interested in. Excel makes it very easy to draw scatter plots. For the data in Figure 1 (or 2), a scatter plot looks like in Figure 3. Figure 3: A scatter plot of Advertising expense vs. Sales A scatter plot is basically a visual aid to give us an insight into the relationship between two variables. Just by looking at this graph, you can tell that there is a strong positive relationship between advertising expense and sales because the points tend to be showing an upward trend. Had the points been tending downwards, you could tell that the relationship would be negative. Also you can imagine a line running though these points. An imaginary line running through these points is called a trendline. In Excel, you can draw a trendline in a snap. Just right click on one of the points on the scatter plot and amongst the options that show up, one of the options is Add Trendline. Figure 4 shows a trendline added to the scatter plot of Figure 3. 3

4 Figure 4: Trendline on a scatter plot If the points on the scatter plot are very close around the trend line, we can say that the relationship is strong. The closer the points get to the trendline, the stronger the relationship gets and higher the value of the correlation coefficient. In Figure 4, you can see that all the points are very close to the trendline, which is why the correlation coefficient is very high (0.96). If all the points lie exactly on the trendline, then you will get a perfect correlation of 1.0. So, the slope of the trendline tells us if the relationship is positive or negative. If the slope is positive, the relationship is positive. If the slope is negative, the relationship is negative. If the slope is zero, i.e. the trendline is horizontal, there is no relationship. Think about the last statement about no relationship. When will you have a horizontal trendline in our example? When the sales are constant no matter what the advertising expense is we will have a horizontal trendline. Clearly if sales are not going to change as we change advertising expense, it would be an indication of no relationship. How close the points are to the trendline gives us an indication of how strong the relationship is. The slope of the line does not have to be very high for the relationship to be considered strong. You can have a small slope, but as long as the points on the scatterplot are gathered tightly around the trendline, you will get a high value of correlation. You may have a high value of slope, but if the points are spread far from the trendline, the correlation will be small and the relationship will be considered weak. For example, look at Figure 5. Notice that the slopes of the trendlines between Figures 4 and 5 are about the same, but in Figure 5, the points are more scattered around the trendline than in Figure 4. Notice that as a result of this scattering, the correlation coefficient has fallen to 0.71 from So now we know why this is called a scatter plot. 4

5 Figure 5: A weaker relationship For an example of a smaller slope but a strong relationship, please look at Figure 6. The slope of the trendline is much smaller compared to that in Figures 4 and 5, but because the points are very close to the trendline, the correlation coefficient is very high (0.97) Figure 6: A strong relationship but smaller slope A summary of what has been discussed so far: 1. Covariance and Correlation are two measures of relationship between two numeric variables. 2. Whenever covariance is positive, correlation is also positive, and vice versa. 3. The sign of covariance and correlation tell us the direction of the relationship. i.e. a positive covariance and correlation tell us that the relationship is positive and vice versa. 4. The magnitude of covariance doesn t tell us much story because covariance varies depending on the units of the variable chosen. 5. The magnitude of correlation tells us a lot about the strength of the relationship because a correlation coefficient can only range from -1 and +1. A value of -1 implies a perfect negative relationship and a value of +1 implies a perfect positive relationship. A value of close to either +1 5

6 or -1, such as 0.9 or -0.9 implies a very strong relationship. A value close to 0 implies a very weak relationship. 6. A scatter plot gives a good visual cue about the direction and the strength of a relationship. 7. A positive slope of trendline on a scatterplot implies a positive relationship and vice versa. 8. The strength of a relationship can be determined visually by looking at how tightly the points are gathered around the trendline. Tighter the points around the trendline, the stronger the relationship and vice versa. Now we are ready to discuss Regression Analysis. We have done enough groundwork to understand Regression Analysis. Usually, when we talk about Regression Analysis, we first talk about simple regression and then we talk about multiple regression. Also, we first talk about linear regression and then nonlinear regression. So really, we first talk about a simple linear regression, then multiple linear regression, then simple nonlinear regression and finally multiple nonlinear regression. But in this course, we will not talk about nonlinear regression. So we will first talk about simple linear regression and then multiple linear regression. Simple vs. Multiple Regression In simple regression we are interested in relationships between two variables, an independent variable and a dependent variable. In our example of advertising expense and sales, please convince yourself that advertising expense is the independent variable because it happens first and sales is the dependent variable because its value depends on the amount of advertising. In multiple regression, we deal with several independent variables and one dependent variable. For example, sales revenue, which can be considered a dependent variable, does not merely depend on the advertising expense. It depends on many other factors, such as the product price, product quality, price of competing products and quality of competing products, brand loyalty etc. In multiple regression we study the effect of several independent variables on the dependent variable. Please note that in multiple regression, there are multiple independent variables but only one dependent variable. Intuitively, many causes for a single effect. Linear vs Non-Linear Regression In linear regression, the trendline is a straight line. In nonlinear regression, the trendline may be nonlinear such as U-shaped or S-shaped or some other non-linear shape. There are many variables in which the relationship may be non-linear. For instance, in our example of advertising expense and sales revenue, you can imagine that beyond a certain expenditure in advertising, the sales will not grow any further. So while for certain range of advertising expense the sales may be linearly related, beyond a certain point, the linearity of the relationship may not hold any more. But since we will not be covering any non-linear regression, we will not discuss it any more. I just wanted to give you a feel for what is involved in non-linear regression. Simple Linear Regression So let s finally talk about Simple Linear Regression. Whenever we consider two variables of interest that are related to each other, we identify one of them as being independent and the other as being dependent. This is important. It is impossible to do regression analysis unless we first establish 6

7 our independent and dependent variables. Once we establish this, we are interested in quantifying their relationship. So we obtain a sample of data points and using this sample, we try to estimate or infer the relationship between the two variables. Please note that the relationship we are interested in is for the population of all possible data points for the two variables. But since it is virtually impossible to obtain all possible data points for the two variables (that would be the whole possible population), we work with a sample and obtain the relationship for the sample data and infer the relationship for the population. Note that this is similar to what we learnt earlier about making inferences about a population using sample data. For example when we wanted to estimate the mean of a population, we obtained a sample and then we computed a point estimate and an interval estimate for the population mean. Similarly in regression analysis, we will first obtain the sample data and calculate the point and interval estimates of the slope. The difference is, now our data items will be pairs of numbers, (x,y)s. Regression and Trendline: We have already seen the concept of a scatterplot and a trendline. They give us a visual clue about the linear relationship between two variables. In simple linear regression we basically try to find the mathematical equation of the trendline. Before we talk any further, let me give you a quick review of mathematical equation of a line. A review of mathematical equation of a line: Y 5 Y= X X Figure 7: A straight line between X and Y variables Let s say our independent variable is X and the dependent variable is Y. We always show the independent variable on the horizontal or the X-axis and the dependent variable on the vertical or the Y- axis. Suppose the red line, above is the line for which we want the mathematical equation. This line has two characteristics. The first characteristic is that it has a slope. I.e. if x increases by one unit, the y increases (or decreases) by a certain amount. In Figure 7, we can tell that if x increases by one unit, that y increases by about half a unit. So we say that the slope of this line is roughly ½ (or half). The other characteristic is that it crosses (or intercepts) the y-axis at a certain point. In Figure 7, we can see that the red line intercepts the y-axis at about 2.5. Once we know these two characteristics of a line, we can completely describe a line. The equation of the red line in Figure 7 is: Y = X The equation of a straight line is of the form Y = a + bx, where a is the y-intercept and b is the slope. Knowing a and b about any line will completely describe the line. In other words, if I gave you 7

8 the a and b values for a line that I have in mind, you can uniquely draw the line that I have in mind. There is only one way that a line with a given a and b values can be drawn. Let s suppose I give you a = 4 and b = -1. The line for these two parameters is shown in Figure 8. Y 5 Y= 4 - X X Figure 8: A straight line between X and Y variables where the y-intercept is 4 and slope is -1 Trendlines and regression line: In Figures 4, 5 and 6, we looked at trendlines. These trendlines were drawn using Excel. Please note that if I have a scatter plot and if I ask you to draw a line through the scatter points such that the line represents the trend of the scatter plot, you will draw one of many possible lines. If I asked two people to draw a trendline (without using Excel of course), it is very unlikely that the y-intercept and the slope of both these lines are exactly the same. They will be close, but not exactly the same. If I asked ten different people to draw a trendline, I will get ten different lines. Theoretically there are an infinite number of such lines, one for every possible combination of y-intercept and slope. But given any two such lines drawn by two random people, one will be better than the other. So if I have ten lines, one of them will be better than the other nine. When Excel draws the trendline, it draws a line that cannot be beaten by anyone. It uses a complex algorithm to come up with a trendline which is the best amongst an infinite number of possible lines. How do we define best? For any line, there are deviations from the line to each of the points. Some deviations are positive and some are negative. If we square these deviations, the squared deviations become positive. The sum of all the squared deviations can be considered a measure of fit of the trend line to the scatterplot. Using this measure, we can compare two lines. Smaller this measure, the better the fit. For any line the sum of the squared deviation can be computed using a formula. Definition of "best-fit" for a trendline: Given two lines, whichever line has a smaller sum of the squared deviations is considered better than the other. If for a line, the sum of the squared deviations is the least of all possible lines, that line would be the best of all possible lines. Such a line is called the least squared deviation line or simply least squares line. When Excel draws a trendline on a scatter plot, it draws the least squares line. The least squares line is also called the Regression line. 8

9 In fact, in Excel, after you add a trendline to a scatter plot, you can even request Excel to display the equation of the trendline. Just double click on the trendline and a new dialog box shows up ; towards the bottom of this dialog box you will see the option to Display Equation on Chart. Figure 9 shows the same graph as in Figure 3, except that the trendline and the equation are also displayed. Figure 9: Equation of trendline displayed on the chart (y = x ) Note that when Excel displays the equation of the trendline, it shows the y-intercept as the second term. So in Figure 9, the y-intercept is and the slope is Since the trendline drawn by Excel does not extend to the y-axis, I have superimposed a line over the trendline so that you can see that intercepts the y-axis at about 3.6. Also I have superimposed an oval around the equation to highlight it. Excel does not draw this oval. The slope of implies that if we spend one extra unit of advertising expense (in other words, 1 thousand dollars), that the sales revenue will increase by units or 82,300 dollars because recall that one unit of sales is a million dollars so million dollars is 82,300 dollars. A Summary of what has been discussed since the last summary: 1. Given a scatter plot of points, many lines can be drawn through them. 2. For each such line the sum of the squared deviations from each point can be calculated. The line with the least Sum of the squared deviations is called the least squares line or the regression line. 3. The least square line can be described by the y-intercept and the slope. 4. The trendline created by Excel, is the least squares line. 5. Excel can display the equation of the trendline which also happens to be the regression or the least squares line. 9