Projections and Regressions

Transcription

1 Projections and Regressions Simple example - no intercept term Start with a standard cost estimation problem from cost accounting. Suppose we have collected the following data on direct labor hours and manufacturing overhead. The problem is to estimate a linear relationship to predict the cost of manufacturing overhead given the number of direct labor hours worked. Period Manufacturing overhead dollars (MOH) Direct labor hours (DLH) Usually in a problem like this, we would allow for a fixed cost, or intercept, component. But to start with here, assume there is no fixed cost; we wish only to find the variable cost of a direct labor hour, call it α. The cost equation we wish to estimate, then, will be MOH = α DLH, where MOH and DLH are the data vectors from the table. And we are looking for an α. The nice thing about setting the problem up this way (without a fixed cost component) is that is it can be solved using the techniques in the separating hyperplane note. The MOH vector is comparable to the x vector from before, so we are looking for the nearest point on the

2 extended DLH vector to MOH. As before, let λ be the difference vector between MOH and the extended DLH, so λ = α DLH - MOH. As before, we wish to minimize the length (λ T λ) of the λ vector. And, also as before, we can write down the orthogonality condition which ensures λ T λ is minimized. DLH T λ = DLH T (α DLH - MOH) = 0. Solving for α: α = (DLH T DLH) -1 DLH T MOH α= = = And we can also get the difference vector λ and its length λ T λ. λ = αdlh MOH = = λλ T = = =

3 The point of the exercise is that, using nothing more than the orthogonality condition, we have effectively solved the estimation problem by projecting the MOH vector into the extended DLH vector. Or, to use alternative terminology, we have regressed MOH on DLH. It is useful at this stage to compare our calculations with a standard computer regression package. The following is output from Excel's regression command. For this regression run, the intercept was suppressed, which can be seen in the output as the coefficient on the intercept is zero. 3

4 MOH DLH SUMMARY OUTPUT Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 5 ANOVA df SS MS F Significance F Regression Residual Total 5.8 Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept 0 #N/A #N/A #N/A #N/A #N/A X Variable We notice immediately that the coefficient on the x variable (DLH) is equal to what we calculated for α. In other words, the weight on the independent variable is equal to where we position the projection on the DLH vector. Furthermore, the length of the difference vector, λ T λ, shows up in the regression output as the residual sum of squares. 4

5 Several of the other numbers in the output can be calculated directly from these two. The standard error of the regression is a measure of how much uncertainty remains in MOH after using the information in DLH. It is calculated as the square root of the residual sum of squares divided by its degrees of freedom. (Here the degrees of freedom is four, since there are five data points less one degree of freedom used for the independent variable, DLH.) The standard error of the estimate is a measure of dispersion of the estimate α, that is, how confident we are in the estimate. We can use our expression for α to write down the variance of α. Var α = Var (DLH T DLH) -1 DLH T MOH We already have an estimate for the variance of MOH: the standard error of the regression. Using the definition of variance, the coefficient of the random variable MOH is brought out in front of the variance operator and squared. Var α = (DLH T DLH) - DLH T DLH Var MOH = (DLH T DLH) -1 Var MOH We already know DLH T DLH to be 9. The standard error of the estimate is the standard error of the regression divided by 9. Std error of estimate = = R-squared is a measure of the power of DLH to explain MOH; it compares the amount of the variation in MOH, itself, with the amount of variation in MOH remaining when DLH is used 5

6 for prediction. The latter number is measured by the length of the difference vector, λ. The unconditional variation in MOH is sum of the squared deviations of MOH from its mean: MOH. ( ) = ( ) MOH MOH MOH = MOH MOH T MOH n ( MOH ) ( ) = = n 8. The unconditional variation in MOH is reported as the total sum of squares. To normalize R between zero and one, it is reported as one minus the ratio of residual to total sums of squares. Res SS 593 R = 1 = 1 9 = Tot SS 8. One more thing: if we are interested in the predicted MOH numbers given the DLH data (usually denoted MOH ), we can multiply the DLH vector by the computed α. T 1 MOH T = DLH( DLH DLH) ( DLH MOH)= DLHα = = The predicted MOH numbers are referred to as the projection of MOH on DLH. Same example - intercept term included 6

7 Now consider the problem the way it is usually presented in a cost accounting setting: allow for the existence of a fixed cost component in MOH. This will allow a better fit of the data and we would expect better goodness of fit measures, such as R. Even though the calculations are slightly more complicated, the basic idea remains the same, and the sequence of events is identical. We have one more vector to project the MOH numbers into (to allow for the intercept), but we start with the same orthogonality condition. Orthogonality allows us to calculate the weights on the two vectors we are projecting into (the regression coefficients), and then calculate the length of the difference vector, λ T λ. Various goodness of fit statistics can be calculated as before, paying some attention to degree of freedom issues. Period Manufacturing overhead dollars (MOH) Fixed Cost Direct labor hours (DLH) In the preceding table, a column has been added for the fixed cost term: a column of ones. The fixed cost is the same each period, so what we need to do is find a weight to attach to the vector of ones; that will be the fixed cost number. The variable cost is found as before: the weight on the DLH vector. The orthogonality condition is the same as before, except that we have two vectors that must be orthogonal to the difference vector. 7

8 α + α = The difference vector, λ, is the second term in square brackets. It consists of a weighted combination of the fixed cost and DLH vectors, using weights α 1 and α, and then the MOH vector subtracted out. The difference vector must be orthogonal to both the fixed cost vector and DLH, and that, of course, is what the equation says, since there is a zero vector on the right hand side. After doing the arithmetic, the above matrix equation is reduced to the following system of two equations in two unknowns. 5α + 31α = α + 9α = 95 1 Solving two linear equtions for two unkowns is a relatively simple matter, and so we have the regression weights. α α 1 61 = = = = And we can substitute back in to find λ. 8

9 λ= ( ) + ( ) = The length of the difference vector: λλ T = = = The preceding discussion can be simplified by using matrix notation. Let X be a matrix whose columns are the vector of ones (for the fixed cost) and the DLH vector. Similarly, let β be a vector whose elements are α 1 and α X = = α 1 β α Now the orthogonality condition can be written as X T (Xβ - MOH) = 0. Similarly, the difference vector and the regression weights are written as follows. λ = Xβ MOH T β = ( XX) 1 T XMOH Notice these are the same expressions as for the previous - no intercept - case except that the DLH vector is replaced by the X matrix, and the scalar α is now the vector β. We should note that X T X is now a matrix and (X T X) -1 is now a matrix inverse instead of just the reciprocal of a 9

10 scalar. The two expressions are presented for completeness and because they are used in the computer regression output. T T XX= 5 31 XX 1 ( ) = SUMMARY OUTPUT MOH FC DLH Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 5 ANOVA df SS MS F Significance F Regression Residual Total 4.8 Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept X Variable Now there are two regression coefficients, and we calculated those above. The standard error of the regression is the square root of λ T λ divided by the degrees of freedom; notice there are now only three degrees of freedom, since two are being used for the parameters of the regression. 10

11 179 Std err regression = 184 = Recall that in the previous regression the standard error of the coefficient was the regression standard error times the square root of (DLH T DLH) -1. Now (X T X) -1 is a matrix and the diagonal elements are used in the calculation of the coefficient standard errors. Std err intercept 179 = = Std err variable 1 = = There is one more thing worth noting before concluding this section: the elements of λ sum to zero. This is sometimes referred to as the zero mean error property. It holds here since λ is constrained to be orthogonal to the fixed cost vector of all ones; multiplying λ by a vector of ones merely sums the elements in λ, hence, they must sum to zero. It is of interest to us that the logic works the other way, as well. That is, when the elements of a vector sum to zero, the vector must be orthogonal to a vector of ones. In the accounting problem we deal with an x vector which sums to zero; also every column of the A matrix does, as well. It turns out to be useful on occasion that x and the columns of A are all orthogonal to a vector of ones. It is, in fact, one way to state the nature of the double entry system: the balances in the accounts (and any individual journal entry) are orthogonal to the same (unit) vector. 11

12 Projecting into the nullspace In this section we briefly present an alternative way to calculate the difference vector λ. It relies on the fact that λ is orthogonal to the columns of X. Therefore, if we took vectors orthogonal to all the columns in X, and projected MOH into them, we would reproduce λ. The logic is the same as the simple two dimensional example in the nearest distance note. (0, b) (a, b) (a, 0) If we are interested in the difference vector between the vector (a, b) and the nearest point on the vertical axis, we could project (a, b) into the vertical axis and then subtract to get the difference vector: (a, b) - (0, b) = (a, 0). Alternatively, we could find the difference vector by projecting directly into the horizontal axis: (a, 0). Either technique works, since the vertical and horizontal axes are orthogonal. For the problem at hand, the following three vectors, denoted as the columns of N, are orthogonal to the columns of X. It's easy to verify that the inner product of any column of N with any column of X is zero. Because they multiply to zero, the matrix N is sometimes referred to as a nullspace matrix relative to X. 1

13 N = The projection of MOH into N is accomplished using the relationships from above T 1 T 1 NNN ( ) NMOH = The sign is different from l, since this is the vector to be subtracted. See the picture. In this problem there isn't any advantage to using the nullspace. However, in the accounting problem there are three things which make the nullspace approach a good way to go. 1. The number of vectors in the nullspace tends to be small.. The nullspace vectors are easy to work with since they are already orthogonal to each other. 3. It is easy to see by inspection what the nullspace vectors are. (Here we take advantage of the fact that in double entry, the vector of ones is orthogonal to the columns of A.) 13