Practical Regression: Log vs. Linear Specification

Transcription

1 DAVID DRANOVE Practical Regression: Log vs. Linear Specification This is one in a series of notes entitled Practical Regression. These notes supplement the theoretical content of most statistics texts with practical advice on solving real world empirical problems through regression analysis. The technical notes Building Your Model 1 and Omitted Variable Bias 2 explain how to distinguish between variables worth including in your regression and variables that are better left out. Once you ve decided what variables belong in your model, you have to think about what functional form those variables should take. The technical note Regression Basics 3 discusses dummy variables for categorical measures and interactions for variables whose effect is dependent on the value of another variable. In this note, we will discuss nonlinear relationships between Y and X, specifically relationships in which we take the natural log of X and/or Y. There are two reasons why it is important to use the correct functional form. First, the functional form has a profound impact on the economic interpretation of the model. Second, models with the incorrect functional form can be heteroskedastic. Logging Variables The Basics Suppose you have the following variables: If you type this command in Stata: regress sales adverts sales: Monthly sales for firm i in year t adverts: Advertising expense for firm i in year t 1 David Dranove, Practical Regression: Building Your Model, Case # (Kellogg School of Management, 2012). 2 David Dranove, Practical Regression: Introduction to Endogeneity: Omitted Variable Bias, Case # (Kellogg School of Management, 2012). 3 David Dranove, Practical Regression: Regression Basics, Case # (Kellogg School of Management, 2012) by the Kellogg School of Management, Northwestern University. This technical note was prepared by Professor David Dranove. Technical notes are developed solely as the basis for class discussion. Technical notes are not intended to serve as endorsements, sources of primary data, or illustrations of effective or ineffective management. To order copies or request permission to reproduce materials, call or cases@kellogg.northwestern.edu. No part of this publication may be reproduced, stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by any means electronic, mechanical, photocopying, recording, or otherwise without the permission of the Kellogg School of Management.

2 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION Stata estimates the following model: sales = β 0 + β 1 adverts + ε By estimating this model you are implicitly assuming that there is a linear relationship between adverts and sales, meaning that: If you plotted the predicted relationship between adverts and sales, you would get a straight line. Each additional unit of adverts is predicted to cause a constant increase of β 1 in sales. Assuming a linear relationship is often a good starting point for analysis. In fact, if your predictors vary over only a narrow range of values, then the linear assumption is very good, as any departure from linearity is probably small. There are two main justifications for departing from linearity on the right hand side (RHS): 1. (a) Your RHS variable shows considerable variation and (b) You have reason to suspect that the effect of the RHS variable is nonlinear. There are many situations in which the effect of a predictor variable is likely to be nonlinear. (For example, you might be studying cost curves that display economies of scale, or you might be examining a dose-response relationship in medicine that shows diminishing returns). or 2. You are performing robustness checks and want to see if your key conclusions change as you change the model specification. The most common ways to specify a nonlinear RHS variable are by taking exponents or by taking the natural log of the variable. 4 This note will focus on the latter. It turns out that it is equally, if not more, important to consider departures from linearity on the left hand side (LHS). Usually this involves taking the natural log. But be careful choosing to log the dependent variable does a lot more than change the shape of a relationship on a graph; it can profoundly affect how you interpret your results. It is therefore essential to spend some time reviewing the mathematical interpretation of log specifications. Models with Logs Although there are many possible functional forms, empirical researchers often use one of the following: 5 4 Henceforth when I mention logging a variable, I am referring to the natural log. Econometricians (i.e., economists who perform statistical analyses) work with natural logs because of the simplicity of the resulting model interpretations, as you will shortly discover. 5 Linear-log is possible but rarely used in practice. 2 KELLOGG SCHOOL OF MANAGEMENT

3 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION Linear: The dependent variable and independent variables are untransformed. Log-log: All continuous variables on the LHS and RHS are logged (using natural logs). Indicator variables on the RHS keep their 0/1 values. Log-linear or semi-log: The dependent variable is logged (using natural logs). All predictor variables remain untransformed. Each specification has its own unique economic interpretation. You must learn the underlying economics if you are to choose and interpret your models correctly. Why Take Logs? Some researchers log a variable because it has a long tail (skewed to the right) whereas its logged value looks normal. Recall the yogurt data described in the technical note Heteroskedasticity. 6 This data contains information about the weekly sales, prices, and promotions of four different brands of yogurt. The following graph shows the distribution of weekly sales of the store brand of yogurt (yogurt 3): Density 0 1.0e e e e e Number of containers of yogurt 3 sold Here you can see the long right hand tail, which implies that sales3 is much higher in some weeks than others. (If you examined the data further, you would discover that sales3 is much higher when the yogurt goes on sale.) Although it is tempting to log sales3 at this point, there is no statistical reason to do so; your dependent variable does not need to appear normal. The only requirement for ordinary least squares (OLS) regression is that the residuals are normal. 6 David Dranove, Practical Regression: Noise, Heteroskedasticity, and Grouped Data, Case # (Kellogg School of Management, 2012). KELLOGG SCHOOL OF MANAGEMENT 3

4 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION There are two ways to assess whether you need to log the dependent variable; both involve the interocular (eyeball) approach. In the first, you regress Y on the predictor variables and log Y on the predictor variables. You then examine the distribution of the residuals from both regressions to see which looks more normal. Alternatively, you can create a two-way plot of Y on X and another of log Y on X and see which looks more linear. Unfortunately, these interocular approaches often leave you staring at clouds of data. Fortunately, there is a formal statistical test to determine whether taking logs really does improve the fit of your model. We will discuss all of these approaches in more detail, but first there is a more important matter to consider. When you log the dependent variable, you are transforming the economic relationships among all the variables in your model. Thus, if you choose to log the dependent variable, you must be sure that the transformed economic relationships make sense otherwise, you may fix a statistical problem but create an even bigger economic problem. So before you worry about assessing the log specification on statistical grounds, let s be certain you understand the economics of the log specification. We can better grasp how taking logs affects economic interpretations by considering the simple economic relationship between advertising and sales. If you specify a linear (i.e., constant slope) relationship, then you are implicitly assuming that a one-unit increase in A causes a B 1 unit increase in S. It seems just as plausible to suppose that a 1 percent increase in A causes a B 1 percent change in S. You will soon see that if the latter supposition appeals to you, you will need to estimate a log-log model. You may recall from microeconomics that this implies that the elasticity of sales with respect to advertising is a constant B 1. To reiterate, if you run a linear model you assume that there is a constant additive relationship between A and S. If you run a log-log model, you assume that there is a constant elasticity relationship between A and S. By choosing the specification (log or linear), you are determining how you will interpret the results. You should be sure you are comfortable with the implied interpretation. Let s illustrate this idea using a simple marketing model. Suppose your sample consists of firms of different sizes and scopes. Small firms (with, say, small product lines and limited geographic scope) find that spending $A on advertising has a small impact on sales. Larger firms may experience a larger increase in sales for the same $A in spending. The following table captures these effects: Firm Advertising ($) Sales A A A B 10 2,000 B 11 2,100 B 12 2,195 C 10 6,000 C 11 6,300 C 12 6,586 4 KELLOGG SCHOOL OF MANAGEMENT

5 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION From this table, it is clear that a one-unit increase in advertising does not generate a constant increase in sales. However, careful inspection reveals that a 10 percent increase in advertising generates a constant 5 percent increase in sales. (Thus, the elasticity of sales with respect to advertising is constant and equals 0.50.) We will soon see that if the real world data-generating process displays this type of constant elasticity relationship, then you should estimate a log-log specification. 7 Any other specification will fail to generate this constant elasticity relationship and therefore the model would be economically incorrect. How to Interpret Models with Logged Variables You may have already guessed that if you take logs, you should think in percentage terms. You can use a bit of math to confirm this. Start with the log-log specification. Suppose you have monthly data on sales (S), advertising (A), and price (P). You specify a log-log model: (1) log S = log B 0 + αlog A + γlog P You can exponentiate this to obtain another equation: (2) S = B 0 A α P γ The coefficients α and γ in equations (1) and (2) are identical by construction. Thus, if you insist on estimating the log-log specification of equation (1), then you must believe that the relationship expressed in equation (2) is correct. Equation (2) tells you that the effects of advertising and price on sales are multiplicative, where the degree of the multiplicative effect is captured by α and γ. For example, regardless of the level of sales, if advertising increases by 10 percent, sales will increase by approximately 10α percent. It bears repeating: if you select a log-log specification, then you are accepting that the effects of RHS variables are multiplicative. Alternatively, if you think the effects of predictors are multiplicative, you should use a log-log specification. We mentioned previously that the coefficients in a log-log specification can be interpreted as elasticities. Here is the proof. First, differentiate S with respect to A (or P). It is easier to do this with equation (2) (if you don t know how to do this, don t worry about it): (3a) (3b) S/A = B 0 α A α 1 P γ = S (α/a) Equation (3b) implies that α = (S/A) (A/S), which is the expression for the elasticity of S with respect to A. This bit of calculus implies that if you use linear regression to estimate regress logsales logadverts logprice 7 The Stata command for taking logs is gen logvar=log(var). Remember, this is in natural logs. This is equivalent to gen logvar=ln(var). KELLOGG SCHOOL OF MANAGEMENT 5

6 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION the resulting coefficients are elasticities and you are assuming that the relationships between advertising, price, and sales can be expressed as constant elasticities. Suppose instead that you have a log-linear regression specification: (4) log S = B 0 + αa + γp To see how to interpret the results, exponentiate equation (4) to obtain: (5) S = e (B 0 + αa + γp) where e represents the base of the natural log. Differentiating S with respect to A yields: (6) S/A = α e (B 0 + αa + γp) = α S The effect of a one-unit change in A on the value of S is again multiplicative. For example, if α = 0.1, then a one-unit increase in A causes S to change by 0.1S. That is, S increases by 10 percent. One can use similar methods to show that in a linear-log model the effect of a one-unit change in the raw (unlogged) RHS variable on the LHS is approximated by S = α (A/A). In other words, a 1 percent change in A causes an α-unit change in S. The following table summarizes how to interpret coefficients from different regression models. Note that in all cases involving logs, the results come from calculus, so they are correct only for small changes. Model Nature of change in X Resulting change in Y Linear One-unit change in X B x -unit change in Y Log-linear One-unit change in X 100B x percent change in Y Linear-log (rarely used) 1 percent change in X B x -unit change in Y Log-log 1 percent change in X B x percent change in Y When in doubt, you can always rely on algebra to help you interpret the coefficients in your model. This is especially helpful for the log-linear specification. Suppose you have the following log-linear model: (7) log S = P A If P increases by one unit, then log S will decrease by 0.06 units but you will typically want to know the effect on S, not its log. This requires a bit of algebra. We have learned that when P increases by 1 unit, say from P 0 to P 0 + 1, then log S changes by In other words: (8) log S (P = P 0 + 1) log S (P = P 0 ) = where log S (P = P 0 ) refers to log S when price is P 0. Now, if you exponentiate both sides of equation (8), you get: 6 KELLOGG SCHOOL OF MANAGEMENT

7 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION (9) [S (P = P 0 + 1)]/S (P = P 0 ) = exp(-0.06) That is, the ratio of sales after the price change to sales before the price change is exp(-0.06), which your calculator will tell you is approximately Thus, sales fall by -6 percent. To put it in more general terms, if the coefficient in a log-linear model is B x, then the predicted effect of X on Y is as follows: A one-unit change in X causes Y to change by 100 (e Bx 1)%. If you play around with your calculator a bit, you will find that if B x is small (say, less than 0.20 in magnitude), then you can use the approximation that a one-unit change in X will cause a 100B x percent change in Y. 8 As B x gets bigger, this approximation is no longer valid and you must use your calculator to compute the effect. The following table shows how to interpret a range of coefficients in log-linear specifications. This table is especially helpful for interpreting coefficients on dummy variables, where a one-unit change is equivalent to going from a value of 0 to a value of 1. You should confirm some of these values for yourself as practice for interpreting the coefficients in your own log-linear models. Guide to Interpreting Coefficients in Log-Linear Specifications Effect of One-Unit Change in X Coefficient on X (%) One final note here: Be careful logging variables with values close to zero. The logged values will be large and negative and can overwhelm the regression. Model Specification: Letting the Data Help You Decide When possible you should pick model specification based on the underlying economics. Sometimes you may have plausible economic arguments for various specifications. In these 8 As long as B x * ΔX is small, you can state that changing X by ΔX will cause a 100 * B x * ΔX percent change in Y. KELLOGG SCHOOL OF MANAGEMENT 7

8 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION situations it is entirely appropriate to let the data help you choose among them. One common approach is to plot the (unlogged) dependent variable against the key RHS variable. If the plot looks linear, choose a linear model. Next, plot the log dependent variable against the RHS variable. If this looks linear, try log-linear. Finally, plot log against log. If this looks linear, run a log-log model. Here are the three plots using sales3 and price3 and their various transformations: Number of containers of yogurt 3 sold Price of yogurt 3 in pennies per ounce logsales Price of yogurt 3 in pennies per ounce 8 KELLOGG SCHOOL OF MANAGEMENT

9 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION logsales logprice3 That doesn t help very much, does it? None look particularly linear or particularly nonlinear. Unfortunately, one often gets ambiguous results when performing interocular tests. Fortunately, there is a better way. Before we discuss it, let us dismiss one possible approach: Do not compare the R 2 of alternative specifications. As discussed in the Regression Basics technical note cited above, when choosing a model specification, you can compare R 2 among different models, but only if they have exactly the same LHS variables and exactly the same observations. In this case, by logging the LHS you no longer have exactly the same LHS variables. As a result, you change the potential for a good fit. Consider that a linear regression can get a terrific R 2 by fitting a handful of extreme outliers. By logging the LHS variable, the outliers become less extreme. This handicaps the log model s ability to generate an equally terrific R 2, even though the log model may generate superior predictions. Besides, the purpose of any statistical analysis is to get the economics and statistics correct, not to maximize the R 2. The Box-Cox Transformation and Test Here is the better way: the Box-Cox test provides a comparison of log versus linear specifications by performing a Box-Cox transformation. The Box-Cox transformation plugs your original dependent variable Y into the following formula to obtain a transformed variable Y*: (10) Y* = (Y θ 1)/θ You can perform an OLS regression using any transformed dependent variable Y*. By using a goodness-of-fit measure called the likelihood score (more in a moment), it is possible to compare the goodness of fit across different OLS regressions that use different values of Y*. In this way, it is possible to find the θ that generates the best-fitting regression. For example, suppose you run two regressions, one when θ = 0 and one when θ = 1. Suppose the likelihood score of the regression when θ = 0 is better than the likelihood score of the regression when θ = 1. In that case, we will say that the model fits better when θ = 0. KELLOGG SCHOOL OF MANAGEMENT 9

10 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION Although θ can take on any value, three particular values are of greatest interest: θ = 1 is the equivalent of a regression in which the dependent variable is untransformed. This is a linear specification. θ = 0 is the equivalent of a regression in which Y* = log(y). (The proof is beyond the scope of this note.) This is a log specification. θ = -1 is the equivalent of a regression in which Y* = 1/Y. This is an inverse specification. Thus, if θ = 1 in the best-fitting model, use a linear specification. If θ = 0, go with a log model, and if θ = -1, go with an inverse. We mentioned that the goodness-of-fit measure is called the likelihood score. The algorithm for fitting regressions that use the likelihood score is known as maximum likelihood estimation (MLE). We revisit MLE at length in the Discrete Dependent Variables 9 and Maximum Likelihood Estimation 10 technical notes. For now, it is sufficient to note that MLE is considered the gold standard for goodness of fit and has considerable advantages when compared to OLS. You implement the Box-Cox test by specifying a regression model whose RHS includes all the main predictors that you believe belong in the model. Using the yogurt data and focusing on yogurt 3, you might want to include prices and promotions of all four brands of yogurt. In Stata, you then type: boxcox sales3 price1 price2 price3 price4 promo1 promo2 promo3 promo4 (or you can use the shortcut boxcox sales3 pr*). The Stata output is lengthy and requires some explanation (see next page). 9 David Dranove, Practical Regression: Discrete Dependent Variables, Case # (Kellogg School of Management, 2012). 10 David Dranove, Practical Regression: Maximum Likelihood Estimation, Case # (Kellogg School of Management, 2012). 10 KELLOGG SCHOOL OF MANAGEMENT

11 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION We call your attention to three separate parts of the Stata output, outlined in red. MLE is an iterative process. The log likelihood is the measure of model fit. MLE reports the log of the likelihood score because the actual likelihood score tends to be a very small number with lots of zeroes after the decimal point. Notice how the fit improves with each iteration as the log likelihood gets closer to zero. Eventually the model cannot fit any better, the log likelihood score is maximized, and the iterations stop; the estimation is complete. Stata reports the value of θ that best fits the model. In this case, θ = Thus, the model fits best when the dependent variable Y* = (sales )/ This is a cumbersome expression and makes no economic sense, so we will directly test the three transformations that do make sense. KELLOGG SCHOOL OF MANAGEMENT 11

12 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION Stata tests whether you can reject the null hypotheses that θ = 1, θ = 0, and θ = -1. o In this case, you can reject the hypotheses that θ = -1 and θ = 1. (Both tests are significant at p = ) o You cannot reject the null that θ = 0; this test is significant at p = o Thus, you cannot reject the null hypothesis that the best-fitting transformation is the log of sales3. o In plain English, the log transformation provides the best fit for the data. As long as this does not conflict with your economic interpretation of the model, you are best off taking the log of sales3. In many cases, the best-fitting θ is some value other than 0 or 1 and you are able to reject all three null hypotheses. If you insist on maximizing the log likelihood score, you have to cope with some ugly computations that lack economic sense. We will not do so. Instead, we will use the Box-Cox test to get a qualitative feel for whether log is preferred to linear (or to inverse). What About the RHS? That is an easy one. Using the same dependent variable and observations, compare the R 2 of the regressions with linear and log RHS variables; the bigger R 2 wins. Analysts usually either log all continuous RHS variables, or none of them. Variables expressed as percentages are, for all intents and purposes, already logged. Do not log dummy variables. Summarizing Approaches to Selecting a Log or Linear Specification 1. Begin with the underlying economic logic. Do you want to think in linear or percentage terms? 2. If there is no dominant economic logic, let the data speak to you. 3. Select your core group of RHS variables be sure there are enough to make it unlikely for you to improve R 2 very much as you build the model further. 4. Use the Box-Cox test to determine whether to transform the LHS by taking logs. 5. Now that you have settled on the LHS, build the RHS of your model. You may use R 2 to determine whether to use log or linear RHS variables. (Remember that linear-log is rarely used because it is hard to interpret.) You might go back and forth between adding some RHS variables, logging them, taking them out again, and so on. Such experimentation is normal. Let robustness be your guide to variable inclusion. 6. Once you have your final model specification, redo the Box-Cox test to confirm the specification. Do not feel obligated to do Box-Cox every step of the way, however. This is a waste of time, and if you started with a good core model, totally unnecessary. 12 KELLOGG SCHOOL OF MANAGEMENT

13 TECHNICAL NOTE: LOG VS. LINEAR SPECIFICATION Making Predictions in a Model With Logs The following is a general representation of a log-linear model (complete with error term): (11) log(y) = BX + ε If you exponentiate both sides of equation (11) you get: (12) Y = e BX e ε If you want to predict the value of Y for any given X, you must take the expected value of the RHS. Letting E[ ] denote the expected value, you obtain: (13) E(Y) = e BX E[e ε ] In other words, the predicted value of Y is the exponent of BX times the expected value of the exponent of the error term. Because the expected value of the errors is 0 and e 0 = 1, you might think that E[e ε ] = 1. Were that the case, you would conclude that E(Y) = e BX. To predict Y, you would simply have to exponentiate BX. Unfortunately, if the errors are skewed, then E[e ε ] 1. Thus, in order to make unbiased predictions, you will need to compute E[e ε ] and plug it into equation (13). Here is how to make predictions from regressions where the LHS variable is expressed in logs and the errors are potentially skewed: 1. Run the regression and recover the predicted values and residuals. regress logy x predict pred predict resid, residual 2. Exponentiate each residual. gen expresid=exp(resid) 3. Find the mean of the exponentiated residuals. sum experesid The resulting mean is the adjustment factor. gen adjfactor = xxx (where xxx is the mean you obtained above) 4. Multiply the predicted values by the adjustment factor. gen prediction = pred*adjfactor If the adjustment factor is less than 1.05, you should probably go ahead and ignore the adjustment altogether on the grounds of simplicity. You would need to make the same adjustment in a log-log model. KELLOGG SCHOOL OF MANAGEMENT 13