3. Hypothesis tests and confidence intervals in multiple regression

3. Hypothesis tests and confidence intervals in multiple regression Contents of previous section: Definition of the multiple regression model OLS estimation of the coefficients Measures-of-fit (based on estimation results) Some problems in the regression model (omitted-variable bias, multicollinearity) Now: Statistical inference based on OLS estimation (hypothesis tests, confidence intervals) 39

3.1. Standard errors for the OLS estimators Recall: OLS estimators are subject to sampling uncertainty Given the OLS assumptions on Slide 18 the OLS estimators are normally distributed in large samples, that is ˆβ j N(β j, σ 2ˆβ j ) for j = 0,..., k Now: How can we estimate the (unknown) OLS estimator s variance σ 2ˆβ and its standard deviation σ 2ˆβ j j σˆβ j 40

Definition 3.1: (Standard error) We call an appropriately defined estimator of the standard deviation σˆβ j the standard error of ˆβ j and denote it by SE(ˆβ j ). Natural question: What constitutes a good estimator of σˆβ j? Answer: The analytical formula of a good estimator crucially hinges on whether the errors u i are homoskedastic or heteroskedastic (see Definition 2.2 on Slide 8) 41

3.1.1. Homoskedasticity / heteroskedasticity Important notes: The way we defined the terms homoskedasticity and heteroskedasticity in Definition 2.2 on Slide 8 implies that homoskedasticity is a special case of heteroskedasticity ( heteroskedasticity is more general than homoskedasticity ) Since the OLS assumptions on Slide 18 place no restrictions on the conditional variance of the error terms u i, they apply to both the general case of heteroskedasticity and the special case of homoskedasticity Theorem 2.4 on Slide 19 is valid under both concepts 42

Corollary 3.2: (To Theorem 2.4, Slide 19) Given the OLS assumptions on Slide 18, the OLS estimators are unbiased, consistent, and normally distributed in large samples (asymptotically normal) irrespective of whether the error terms are heteroskedastic or homoskedastic. Classical econometrics: In classical econometrics the default assumption is that the error terms are homoskedastic Given our OLS assumptions plus homoskedasticity, the OLS estimators are efficient (optimal) among all alternative linear and unbiased estimators of the regression coefficients β 0,..., β k (Gauss-Markov theorem) 43

Classical econometrics: [continued] Under heteroskedasticity there are more efficient estimators than OLS, namely the so-called (feasible) Generalized Least Squares (GLS) estimators (see the lectures Econometrics I+II) Mathematical aspects: There exist specific formulas for the standard errors SE(ˆβ j ) both under heteroskedasticity and homoskedasticity Since homoskedasticity is a special case of heteroskedasticity the standard errors under homoskedasticity have a simpler structural form (homoskedasticity-only standard errors) 44

Mathematical aspects: [continued] The homoskedasticity-only standard errors are valid only under homoskedasticity, but lead to invalid statistical inference under heteroskedasticity The more general standard errors under heteroskedasticity were proposed by Eicker (1967), Huber (1967), and White (1980) (Eicker-Huber-White standard errors) The Eicker-Huber-White standard errors produce valid statistical inference irrespective of whether the error terms are heteroskedastic or homoskedastic (heteroskedasticity-robust standard errors) 45

Homoskedasticity-only and heteroskedasticity-robust standard errors for the house-prices dataset Dependent Variable: SALEPRICE Method: Least Squares Date: 07/02/12 Time: 16:50 Sample: 1 546 Included observations: 546 Variable Coefficient Std. Error t-statistic Prob. C -4009.550 3603.109-1.112803 0.2663 LOTSIZE 5.429174 0.369250 14.70325 0.0000 BEDROOMS 2824.614 1214.808 2.325153 0.0204 BATHROOMS 17105.17 1734.434 9.862107 0.0000 STOREYS 7634.897 1007.974 7.574494 0.0000 R-squared 0.535547 Mean dependent var 68121.60 Adjusted R-squared 0.532113 S.D. dependent var 26702.67 S.E. of regression 18265.23 Akaike info criterion 22.47250 Sum squared resid 1.80E+11 Schwarz criterion 22.51190 Log likelihood -6129.993 Hannan-Quinn criter. 22.48790 F-statistic 155.9529 Durbin-Watson stat 1.482942 Prob(F-statistic) 0.000000 Dependent Variable: SALEPRICE Method: Least Squares Date: 28/02/12 Time: 09:43 Sample: 1 546 Included observations: 546 White heteroskedasticity-consistent standard errors & covariance Variable Coefficient Std. Error t-statistic Prob. C -4009.550 3668.048-1.093102 0.2748 LOTSIZE 5.429174 0.459424 11.81735 0.0000 BEDROOMS 2824.614 1262.593 2.237153 0.0257 BATHROOMS 17105.17 2263.281 7.557690 0.0000 STOREYS 7634.897 917.6527 8.320029 0.0000 R-squared 0.535547 Mean dependent var 68121.60 Adjusted R-squared 0.532113 S.D. dependent var 26702.67 S.E. of regression 18265.23 Akaike info criterion 22.47250 Sum squared resid 1.80E+11 Schwarz criterion 22.51190 Log likelihood -6129.993 Hannan-Quinn criter. 22.48790 F-statistic 155.9529 Durbin-Watson stat 1.482942 Prob(F-statistic) 0.000000 46

Practical issues: Heteroskedasticity arises in many econometric applications It is prudent to assume heteroskedastic errors unless you have compelling reasons to believe otherwise Rule of thumb: to be on the safe side, always use heteroskedasticity-robust standard errors Many software packages (like EViews) report homoskedasticity-only standard errors as their default setting It is up to the user to activate the option of heteroskedasticity-robust standard errors (in EViews: use the command ls(cov=white)) 47

3.1.2. Autocorrelated errors Problem: Particularly in time-series regressions (that is when the index i represents distinct points in time) we often encounter autocorrelated error terms: Corr(u i, u j ) 0 for some i j Under autocorrelation the OLS coefficient estimators are still consistent, but the usual OLS standard errors become inconsistent Statistical inference based on the usual OLS standard errors becomes invalid 48

Solution: Standard errors should be computed using a heteroskedasticity- and autocorrelation-consistent (HAC) estimator of the variance Such HAC standard errors become relevant in the Sections 6 and 9 A well-known (special) HAC estimator is the so-called Newey- West variance estimator (see Newey and West, 1987) For a more formal discussion of HAC estimators see Stock and Watson (2011, Section 15.4) 49

3.2. Hypothesis tests and confidence intervals for a single coefficient Testing problem: Consider one of the k regressors, say X j, and the corresponding regression coefficient β j We aim at testing the two-sided problem that the unknown β j takes on some specific value β j,0 In technical terms: H 0 : β j = β j,0 vs. H 1 : β j β j,0 50

Testing procedure: Compute the standard error of ˆβ j, SE(ˆβ j ) Compute the so-called t-statistic: t = ˆβ j β j,0 SE(ˆβ j ) (3.1) Compute the p-value: p-value = 2 Φ ( t act ), (3.2) where t act is the value of the t-statistic actually computed and Φ( ) is the cdf of the standard normal distribution Reject H 0 at the 5% significance level if p-value < 0.05 (or, equivalently, if t act > 1.96) 51

Remarks: Our testing procedure makes use of the result that the sampling distribution of the OLS estimator ˆβ j is approximately normal for moderate and large sample sizes Under H 0 the mean of this distribution is β j,0 The t-statistic (3.1) is approximately N(0, 1) distributed The phrasing t-statistic stems from the fact that for finite sample sizes and under some additional (classical) assumptions on the multiple regression model the t-statistic (3.1) follows the t-distribution with n k 1 degrees of freedom Given these restrictive assumptions the p-values should be computed from the quantiles of the t n k 1 -distribution 52

Remarks: [continued] Since these additional assumptions are rarely met in realworld applications and since sample sizes are typically moderate or even large, we base inference on the p-values (3.2) computed from the normal distribution Attention: Many software packages (like EViews) assume the validity of the classical assumptions and report p-values based on the t n k 1 -distribution in the default setting p-values should be corrected manually (see the following example) 53

p-values for the house-prices dataset based on the t- and the normal distribution, respectively Dependent Variable: SALEPRICE Method: Least Squares Date: 28/02/12 Time: 09:43 Sample: 1 546 Included observations: 546 White heteroskedasticity-consistent standard errors & covariance Variable Coefficient Std. Error t-statistic Prob. C -4009.550 3668.048-1.093102 0.2748 LOTSIZE 5.429174 0.459424 11.81735 0.0000 BEDROOMS 2824.614 1262.593 2.237153 0.0257 BATHROOMS 17105.17 2263.281 7.557690 0.0000 STOREYS 7634.897 917.6527 8.320029 0.0000 R-squared 0.535547 Mean dependent var 68121.60 Adjusted R-squared 0.532113 S.D. dependent var 26702.67 S.E. of regression 18265.23 Akaike info criterion 22.47250 Sum squared resid 1.80E+11 Schwarz criterion 22.51190 Log likelihood -6129.993 Hannan-Quinn criter. 22.48790 F-statistic 155.9529 Durbin-Watson stat 1.482942 Prob(F-statistic) 0.000000 Dependent Variable: SALEPRICE Method: Least Squares Date: 28/02/12 Time: 09:43 Sample: 1 546 Included observations: 546 White heteroskedasticity-consistent standard errors & covariance Variable Coefficient Std. Error t-statistic Prob. C -4009.550 3668.048-1.093102 0.2743 LOTSIZE 5.429174 0.459424 11.81735 0.0000 BEDROOMS 2824.614 1262.593 2.237153 0.0253 BATHROOMS 17105.17 2263.281 7.557690 0.0000 STOREYS 7634.897 917.6527 8.320029 0.0000 R-squared 0.535547 Mean dependent var 68121.60 Adjusted R-squared 0.532113 S.D. dependent var 26702.67 S.E. of regression 18265.23 Akaike info criterion 22.47250 Sum squared resid 1.80E+11 Schwarz criterion 22.51190 Log likelihood -6129.993 Hannan-Quinn criter. 22.48790 F-statistic 155.9529 Durbin-Watson stat 1.482942 Prob(F-statistic) 0.000000 54

Remarks: [continued] A (1 α) two-sided confidence interval for the coefficient β j is an interval that contains the true value of β j with a (1 α) probability It contains the true value of β j in 100 (1 α)% of all possible randomly drawn samples Equivalently, it is the set of values of β j,0 that cannot be rejected by an α-level hypothesis test H 0 : β j = β j,0 vs. H 1 : β j β j,0 When the sample size is large, we approximate the (1 α) confidence interval for β j by [ˆβ j u 1 α/2 SE(ˆβ j ), ˆβ j + u 1 α/2 SE(ˆβ j ) ], (3.3) where u α denotes the α-quantile of the N(0, 1)-distribution 55

3.3. Tests of joint hypotheses Now: Testing hypotheses on two or more regression coefficients (joint hypotheses) Example: H 0 : β 1 = 0 and β 2 = 0 vs. H 1 : β 1 0 and/or β 2 0 (two restrictions) 56

General form of joint hypothesis: Consider the k + 1 regression coefficients β 0, β 1,..., β k and k + 1 prespecified real numbers β 0,0, β 1,0,..., β k,0 For q out of the k + 1 coefficients we consider joint null and alternative hypotheses of the form H 0 : β j = β j,0, β m = β m,0,..., for a total of q restrictions, H 1 : one or more of the q restrictions under H 0 does or do not hold 57

Remarks: A special case is given by considering the k restrictions with β 1,0 = 0, β 2,0 = 0,..., β k,0 = 0, that is H 0 : β 1 = 0,..., β k = 0 H 1 : at least one of the β m is nonzero for m = 1,..., k (overall-significance test of the regression model) It is tempting to conduct the test by using the usual t- statistics to test the k restrictions one at a time: Test #1: H 0 : β 1 = 0 vs. H 1 : β 1 0 Test #2: H 0 : β 2 = 0 vs. H 1 : β 2 = 0......... Test #k: H 0 : β k = 0 vs. H 1 : β k 0 58

Remarks: [continued] This approach is unreliable:... testing a series of single hypotheses is not equivalent to testing those same hypotheses jointly. The intuitive reason for this is that in a joint test of several hypotheses any single hypothesis is affected by the information in the other hypotheses. (Gujarati and Porter, 2009, p. 238) Solution: Joint-hypotheses testing on the basis of the F -statistic EViews-example: overall-significance test for the house-prices dataset 59

F -test (overall-significance test) for the house-prices dataset Dependent Variable: SALEPRICE Method: Least Squares Date: 28/02/12 Time: 09:43 Sample: 1 546 Included observations: 546 White heteroskedasticity-consistent standard errors & covariance Variable Coefficient Std. Error t-statistic Prob. C -4009.550 3668.048-1.093102 0.2743 LOTSIZE 5.429174 0.459424 11.81735 0.0000 BEDROOMS 2824.614 1262.593 2.237153 0.0253 BATHROOMS 17105.17 2263.281 7.557690 0.0000 STOREYS 7634.897 917.6527 8.320029 0.0000 R-squared 0.535547 Mean dependent var 68121.60 Adjusted R-squared 0.532113 S.D. dependent var 26702.67 S.E. of regression 18265.23 Akaike info criterion 22.47250 Sum squared resid 1.80E+11 Schwarz criterion 22.51190 Log likelihood -6129.993 Hannan-Quinn criter. 22.48790 F-statistic 155.9529 Durbin-Watson stat 1.482942 Prob(F-statistic) 0.000000 60

Form and null-distribution of the F -statistic: The exact formula of the F -statistic used for testing the (general) problem of q restrictions given on Slide 57 depends on the specific assumptions imposed on the multiple regression model Also, the exact null-distribution of the F -statistic (F -distribution with exactly specified degree-of-freedom parameters n 1 and n 2 ) also depends on the these assumptions In EViews, the F -statistic and its null-distribution are computed under the classical assumptions of (1) normally distributed and (2) homoskedastic error terms u i In contrast, Stock and Watson (2011) do not assume normally distributed errors u i and consider a heteroskedasticityrobust F -statistic (Stock and Watson, 2011, pp. 748-749) 61

3.4. Testing single restrictions involving multiple coefficients General F -testing: Consider again the multiple regression model: Y i = β 0 + β 1 X 1i + β 2 X 2i +... + β k X ki + u i (3.4) We aim at testing hypotheses involving some linear restrictions on the parameters of the k-variable model such as H 0 : β 2 = β 3 vs. H 1 : β 2 β 3 H 0 : β 3 + β 4 + β 5 = 3 vs. H 1 : β 3 + β 4 + β 5 3 H 0 : β 3 = β 4 = β 5 = 0 vs. H 1 : β 3 0 and/or β 4 0 and/or β 5 0 (the regressors X 3, X 4, X 5 are absent from the model) 62

General F -testing: [continued] All these hypotheses can be tested using a general F -statistic This general testing strategy distinguishes sharply between the so-called unrestricted regression model (3.4) and the restricted regression obtained from plugging the restriction specified under H 0 into the unrestricted regression (3.4) The general F -statistic then compares the sum of squared residuals obtained from the unrestricted regression (3.4), denoted by SSR UR, with the sum of squared residuals obtained from the restricted regression, denoted by SSR R (see Gujarati and Porter, 2009, pp. 249-254) 63

General F -testing: [continued] As in Section 3.3., the exact null-distribution of this general F -statistic (F -distribution with exactly specified degree-offreedom parameters) again depends on the assumptions imposed on the multiple regression model (in particular on the normality/nonnormality and the homoskedasticity/heteroskedasticity of the error terms u i ) EViews provides a fully-fledged framework for performing these general F -tests under the classical assumptions of normally distributed and homoskedastic error terms A thorough discussion will be given in the class 64

3.5. Confidence sets for multiple coefficients Definition 3.3: (Confidence set) A 95% confidence set for two or more coefficients is the set of numbers that contains the true population values of these coefficients in 95% of randomly drawn samples. Remarks: A confidence set is the generalization to two or more coefficients of a confidence interval for a single coefficient Recall Formula (3.3) on Slide 55 for constructing a confidence interval for the single coefficient β j 65

Remarks: [continued] Instead of using Formula (3.3), an equivalent way of constructing a say 95% confidence interval for the single coefficient β j consists in determining the set of all values β j,0 that cannot be rejected by a two-sided hypothesis test H 0 : β j = β j,0 vs. H 1 : β j β j,0 at the 5% significance level based on the t-statistic (3.1) on Slide 51 This approach can be extended to the case of multiple coefficients using the general F -testing approach described in Section 3.3 66

Example: [continued] Suppose you are interested in constructing a confidence set for the two coefficients β j and β m (for j, m = 0,..., k, j m) In line with Slide 57, consider testing a joint null hypothesis with the 2 restrictions H 0 : β j = β j,0, β m = β m,0 at the 5% level using the appropriate F -statistic The set of all pairs (β j,0, β m,0 ) for which you cannot reject H 0 at the 5% level constitutes a 95% confidence set for β j and β m 67

Remarks: In line with the confidence-interval formula (3.3), there are also analytical formulas for constructing confidence sets for multiple coefficients (not to be discussed here) EViews provides a fully-fledged framework for constructing confidence intervals and confidence sets (see class for details) Example: Confidence intervals and two-dimensional confidence sets for the house-prices dataset in EViews 68

Single-coefficient confidence intervals for the house-prices dataset Coefficient Confidence Intervals Date: 20/03/12 Time: 12:39 Sample: 1 546 Included observations: 546 90% CI 95% CI 99% CI Variable Coefficient Low High Low High Low High C -4009.550-10053.30 2034.202-11214.91 3195.812-13491.26 5472.162 LOTSIZE 5.429174 4.672192 6.186155 4.526700 6.331647 4.241586 6.616761 BEDROOMS 2824.614 744.2712 4904.956 344.4288 5304.799-439.1222 6088.350 BATHROOMS 17105.17 13376.02 20834.33 12659.28 21551.07 11254.71 22955.64 STOREYS 7634.897 6122.903 9146.891 5832.298 9437.496 5262.813 10006.98 69

95% two-dimensional confidence sets for the house-prices dataset 6.5 6.0 LOTSIZE 5.5 5.0 4.5 BEDROOMS 4,000 2,000 0 22,000 BATHROOMS 20,000 18,000 16,000 14,000 12,000 9,000 STOREYS 8,000 7,000 6,000 70-10,000-5,000 0 4.5 5.0 5.5 6.0 6.5 0 2,000 4,000 12,000 16,000 20,000 C LOTSIZE BEDROOMS BATHROOMS

Remarks: The vertical and horizontal dotted lines show the corresponding 95% confidence intervals for the single coefficients β j, β m The orientation of the ellipse indicates the estimated correlation between the OLS estimators ˆβ j and ˆβ m If the OLS estimators ˆβ j and ˆβ m were independent, the ellipses would be exact circles 71