Restricted Least Squares, Hypothesis Testing, and Prediction in the Classical Linear Regression Model

Transcription

1 Restrited Least Squares, Hypothesis Testing, and Predition in the Classial Linear Regression Model A. Introdution and assumptions The lassial linear regression model an be written as (1) or (2) where xt is the tth row of the matrix X or simply as (3) where it is impliit that x t is a row vetor ontaining the regressors for the tth time period. The lassial assumptions on the model an be summarized as (4) Assumption V as written implies II and III. With normally distributed disturbanes, the joint density (and therefore likelihood funtion) of y is (5) The natural log of the likelihood funtion is given by 1

2 2 (6) Maximum likelihood estimators are obtained by setting the derivatives of (6) equal to zero and solving 2 the resulting k+1 equations for the k s and. These first order onditions for the M.L estimators are (7) Solving we obtain (8) The ordinary least squares estimator is obtained be minimizing the sum of squared errors whih is defined by (9) The neessary ondition for to be a minimum is that

3 3 (10) This gives the normal equations whih an then be solved to obtain the least squares estimator (11) The maximum likelihood estimator of is the same as the least squares estimator. B. Restrited least squares 1. Linear restritions on Consider a set of m linear onstraints on the oeffiients denoted by (12) Restrited least squares estimation or restrited maximum likelihood estimation onsists of minimizing the objetive funtion in (9) or maximizing the objetive funtion in (6) subjet to the onstraint in (12). 2. Constrained maximum likelihood estimates 2 2 Given that there is no onstraint on, we an differentiate equation 6 with respet to to get an 2 estimator of as a funtion of the restrited estimator of. Doing so we obtain (13) where is the onstrained maximum likelihood estimator. Now substitute this estimator for 2 bak into the log likelihood equation (6) and simplify to obtain (14)

4 4 Note that the onentrated likelihood funtion (as opposed to the onentrated log likelihood funtion) is given by (15) The maximization problem defining the restrited estimator an then be stated as (16) Clearly we maximize this likelihood funtion by minimizing the sum of squared errors (y - X ) (y - X ). To maximize this subjet to the onstraint we form the Lagrangian funtion where is an m 1 vetor of Lagrangian multipliers (17) Differentiation with respet to and yields the onditions (18) -1 Now multiply the first equation in (18) by R(X X) to obtain (19) Now solve this equation for substituting as appropriate (20) The last step follows beause R = r. Now substitute this bak into the first equation in (18) to obtain

5 5 (21) With normally distributed errors in the model, the maximum likelihood and least squares estimates of the onstrained model are the same. We an rearrange (21) in the following useful fashion (22) Now multiply both sides of (22) by to obtain (23) We an rearrange equation 21 in another useful fashion by multiplying both sides by X and then subtrating both sides from y. Doing so we obtain (24) where u is the estimated residual from the onstrained regression. Consider also u u whih is the sum of squared errors from the onstrained regression. (25) where e is the estimated residual vetor from the unonstrained model. Now remember that in ordinary least squares X e = 0 as an be seen by rewriting equation 10 as follows

6 6 (26) Using this information in equation 25 we obtain (27) Thus the differene in the sum of squared errors in the onstrained and unonstrained models an be written as a quadrati form in the differene between and r where is the unonstrained ordinary least squares estimate. Equation 21 an be rearranged in yet another fashion that will be useful in finding the variane of the onstrained estimator. First write the ordinary least square estimator as a funtion of and as follows (28) Then substitute this expression for in equation 21 as follows (29) Now define the matrix M as follows (30) We an then write - as

7 7 (31) 3. Statistial properties of the restrited least squares estimates a. expeted value of (32) b. variane of (33) The matrix M is not symmetri, but it is idempotent as an be seen by multiplying it by itself. (34). Now onsider the expression for the variane of We an write it out and simplify to obtain

8 8 (35) We an also write this is another useful form (36) The variane of the restrited least squares estimator is thus the variane of the ordinary least squares estimator minus a positive semi-definite matrix, implying that the restrited least squares estimator has a lower variane that the OLS estimator. 4. Testing the restritions on the model using estimated residuals We showed previously (equation 109 in the setion on statistial inferene) that (37) Consider the numerator in equation 37. It an be written in terms of the residuals from the restrited and unrestrited models using equation 27 (38) Denoting the sum of squared residuals from a partiular model by SSE( ) we obtain

9 9 (39) -1 Rather than performing the hypothesis test by inverting the matrix [R(X X) R ] and then pre and post multiplying by, we simply run two different regressions and ompute the F statisti from the onstrained and unonstrained residuals. The form of the test statisti in equation 39 is referred to as a test based on the hange in the objetive funtion. A number of tests fall in this ategory. The idea is to ompare the sum of squares in the onstrained and unonstrained models. If the restrition auses SSE to be signifiantly larger than otherwise, this is evidene that the data to not satisfy the restrition and we rejet the hypothesis that the restrition holds. The general proedure for suh tests is to run two regressions as follows. 1) Estimate the regression model without imposing any onstraints on the vetor. Let the assoiated sum of squared errors (SSE) and degrees of freedom be denoted by SSE and (n - k), respetively. 2) Estimate the same regression model where the is onstrained as speified by the hypothesis. Let the assoiated sum of squared errors (SSE) and degrees of freedom be denoted by SSE( ) and (n - k), respetively. 3) Perform test using the following statisti (40) where m = (n-k) - (n-k) is the number of independent restritions imposed on by the hypothesis. For example, if the hypothesis was H o: =4, 4 = 0, then the numerator degrees of freedom (q) is equal to 2. If the hypothesis is valid, then SSE( ) and (SSE) should not be signifiantly different from eah other. Thus, we rejet the onstraint if the F value is large. Two useful referenes on this type of test are Chow and Fisher. 5. Testing the restritions on the model using a likelihood ratio (LR) test a. idea and definition The likelihood ratio (LR) test is a ommon method of statistial inferene in lassial statistis. The LR test statisti reflets the ompatibility between a sample of data and the null hypothesis through a omparison of the onstrained and unonstrained likelihood funtions. It is based on determining whether there as been a signifiant redution in the value of the likelihood (or loglikelihood) value as a result of imposing the hypothesized onstraints on the parameters in the estimation proess. Formally let the random sample (X 1, X 2,..., X n) have the joint probability density funtion f(x 1, x 2,..., x n; ) and the assoiated likelihood funtion L( ; x 1, x 2,..., x n ). The generalized likelihood ratio (GLR) is defined as

10 10 (41) where sup denotes the supremum of the funtion over the set of parameters satisfying the null hypothesis (H 0), or the set of parameters that would satisfy either the null or alternative (H a) hypothesis. A generalized likelihood ratio test for testing H o against the alternative H a is given by the following test rule: (42) where is the ritial value from a yet to be determined distribution. For an test the onstant is hosen to satisfy (43) where ( ) is the power funtion of the statistial test. We use the term generalized likelihood ratio as ompared to likelihood ratio to indiate that the two likelihood funtions in the ratio are optimized with respet to over the two different domains. The literature often refers to this as a likelihood ratio test without the modifier generalized and we will often follow that onvention. b. likelihood ratio test for the lassial normal linear regression model Consider the null hypothesis in the lassial normal linear regression model R = r. The likelihood funtion evaluated at the restrited least squares estimates from equation 15 is (44) In an analogous manner we an write the likelihood funtion evaluated at the OLS estimates as (45) The generalized likelihood ratio statisti is then

11 11 (46) We rejet the null hypothesis for small values of or small values of the right hand side of equation 46. The diffiulty is that we not know the distribution of the right hand side of equation 46. Note that we an write it in terms of estimated residuals as (47) This an then be written as (48) So we rejet the null hypothesis that R = r if (49)

12 12 Now subtrat from both sides of equation 49 and simplify (50) Now multiply by to obtain (51) We rejet the null hypothesis if the value in equation 51 is greater than some arbitrary value. The question is then finding the distribution of the test statisti in equation 51. We an also write the test statisti as (52) We have already shown that the numerator in equation 52 is the same as the numerator in equations Therefore this statisti is equivalent to the those statistis and is distributed as an F. Speifially, (53) Therefore the likelihood ratio test and the F test for a set of linear restritions R = r in the lassial normal linear regression model are equivalent.

13 13. asymptoti distribution of the likelihood ratio statisti We show in setion on non-linear estimation that (54) where is the value of the log-likelihood funtion for the model estimated subjet to the null hypothesis, is the value of the log-likelihood funtion for the unonstrained model, and there are m restritions on the parameters in the form R( ) = r. 6. Some examples a. two linear onstraints Consider the unonstrained model (55) with the usual assumptions. Consider the null hypothesis H o: 2 = 3 = 0 whih onsists of two restritions on the oeffiients. We an test this hypothesis by running two regressions and then forming the test statisti. 1) Estimate (56) 2 2 and obtain SSE = (n - 4)s where s is the estimated variane from the unrestrited model. 2) Estimate (57) 2 2 and obtain SSE = (n - 2)s where s is the estimated variane from the unrestrited model. 3) Construt the test statisti (58)

14 14 b. equality of oeffiients in two separate regressions (possibly different time periods) We sometimes want to test the equality of the full set of regression oeffiients in two regressions. Consider the two models below (59) 1 2 We may want to test the hypothesis H o: = where there are k oeffiients and thus k restritions. Rewrite the model in staked form as (60) and estimate as usual to obtain SSE (no restrition). Note that (n - k) = n 1 + n 2-2k. Then 1 2 impose the restrition (hypothesis that = = ) by writing the model in equation 60 as (61) Estimate equation 61 using least squares to obtain the onstrained sum of squared errors SSE with degrees of freedom (n - k) = n + n - k. Then onstrut the test statistis 1 2 (62)

15 15 C. Foreasting 1. introdution Let (63) denote the stohasti relationship between the variable y t and the vetor of variables x t = (x t1,..., x tk). represents a vetor of parameters. Foreasts are generally made by estimating the vetor of parameters, determining the appropriate vetor x t (sometimes foreasted by )and then evaluating (64) The foreast error is given by (65) There are at least four fators whih ontribute to foreast error. a. inorret funtional form

16 16 b. the existene of the random disturbane t Even if the "appropriate" value of x t were known with ertainty and the parameters were known, the foreast error would not be zero beause of the presene of t. The foreast error is given by (66) Confidene intervals for y t would be obtained from (67) We an see this graphially as. unertainty about Consider a set of n 0 observations not inluded in the original sample of n observations. Let X0 denote the n 0 observations on the regressors and y 0 the observations on y. The relevant model for this out of sample foreast is (68) where E( 0) = 0 and and 0 is independent of from the sample period.

17 17 Now onsider a foreast for y 0 given by (69) where is the OLS estimator obtained using the initial n observations. Then let v 0 be the set of foreast errors defined by (70) We an show that v 0 has an expetation of zero as follows (71) Now onsider the variane of the foreast error. We an derive it as (72) Now note that 0 and are independent by the independene of 0 and and the independene of and whih we proved earlier in the setion on statistial inferene (equations 59-62). As a result the middle two terms in (72) will have an expetation of zero. We then obtain (73) This indiates the sampling variability is omposed of two parts, that due to the equation error, 0, and that due to the error in estimating the unknown parameters. We should note that an be viewed as an estimator of E(y ) and as a preditor of y. In other words 0 0 is the best preditor we have of the regression line, and of an individual y 0. The least squares estimator of E(y 0) is whih has expeted value. Now onsider the ovariane

18 18 matrix for the random vetor. To simplify the derivation write it as Then we an ompute this ovariane as (74) 2 This is less than the ovariane in equation 73 by the variane of the equation error I. Now onsider the ase where n 0 = 1 and we are foreasting a given y 0 for a given x 0, where x0 is a row vetor. Then the predited value of y for a given value of x is given by 0 0 (75) The predition error is given by (76) The variane of the predition error is given by (77) The variane of E(y 0 x 0) is (78) Based on these varianes we an onsider onfidene intervals for y 0 and E(y 0 x 0) where we 2 2 estimate with s. The onfidene interval for E(y x )is 0 0 (79) where by is the square root of the variane in equation 78. The onfidene interval for y is given 0 (80) where is the square root of the variane in equation 77. Graphially the onfidene bounds in prediting an individual y are wider than in prediting the expeted value of y. 0 o

19 19 d. unertainty about X. In many situations the value of the independent variable also needs to be predited along with the value of y. A "poor" estimate of x t will likely result in a poor foreast for y. e. preditive tests One way to onsider the auray of a model is to estimate it using only part of the data. Then use the remaining observations as a test of the model. 1) Compute using observations 1 to n 1. 2) Compute using the observations on the x's from n 1 to n. 3) Compare the predited values of y with the atual ones from the sample for observations n to n. 1

20 20 Literature Cited Chow, G. C., "Tests of Equality Between Subsets of Coeffiients in Two Linear Regressions," Eonometria, 28(1960), Fisher, F. M., "Tests of Equality Between Sets of Coeffiients in Two Linear Regressions: An Expository Note," Eonometria, 38(1970),

21 21 That equation 38 is distributed as an F random variable an also be seen by remembering that an F is the ratio of two hi-square random variable, eah divided by its degrees of freedom. We have previously shown (equation 177 in the setion on Statistial Inferene) that is distributed as 2 a (n-k) random variable. Note that the vetor Its variane is given by is distributed normally with a mean of zero. (81)