Chapter 4: Constrained estimators and tests in the multiple linear regression model (Part II)

Transcription

1 Chapter 4: Constrained estimators and tests in the multiple linear regression model (Part II) Florian Pelgrin HEC September-December 2010 Florian Pelgrin (HEC) Constrained estimators September-December / 34

2 1 Introduction 2 Hypothesis testing Overview Different tests... Outcomes of hypothesis testing Level of significance General methodology 3 The Student test Framework Student test-statistic Decision rules 4 The Fisher test Framework Fisher test-statistic Decision rule 5 Summary 6 Appendix Florian Pelgrin (HEC) Constrained estimators September-December / 34

3 1. Introduction Introduction Aim: Testing parameters in the multiple linear regression model: Assume normality of the error terms exact distribution of estimators; Different tests... Testing hypothesis about individual parameters; Testing single hypothesis involving more than one parameter; Testing multiple restrictions....using the Student and Fisher test statistics Florian Pelgrin (HEC) Constrained estimators September-December / 34

4 Introduction The main assumption is: u X N ( ) 0, σ0 2 I n i.e. the population error u is independent of X and normally distributed with zero mean and variance σ 2 0 I n. This could be a strong requirement (especially, in finite samples). This assumption implies the zero conditional mean assumption (Assumption 3, Chapter 2) and the spherical error terms assumption (Assumption 4, Chapter 2) other assumptions are assumed to hold. Florian Pelgrin (HEC) Constrained estimators September-December / 34

5 2. Hypothesis testing 2.1. Overview Hypothesis testing Overview A hypothesis is a statement about a population parameter. A standard hypothesis testing problem involves two complementary hypothesis: 1 The null hypothesis: H 0 2 The alternative hypothesis: H 1 or H a. If θ 0 is the parameter (vector) of interest: 1 H 0 : θ 0 Θ 0 Θ 2 H 1 : θ 0 Θ 1 Θ, with Θ 1 = Θ Θ 0. Florian Pelgrin (HEC) Constrained estimators September-December / 34

6 Hypothesis testing Overview The null hypothesis is tested by finding an appropriate subset of outcomes W Θ called the rejection region or critical region. For example, a standard rejection region W is of the form: W = {z : T S (z) > c} where T S is a test statistic and c is a critical value. The complement of the rejection region is the acceptance region. A (hypothesis) test is thus a rule that specified: 1 For which sample values the decision is made to fail to reject H 0 as true; 2 For which sample values the decision is made to reject H 0. Florian Pelgrin (HEC) Constrained estimators September-December / 34

7 2.2. Different tests... Hypothesis testing Different tests... Definition A hypothesis of the form H 0 : θ 0 = θ is called a simple hypothesis. A hypothesis of the form H 0 : θ 0 θ or H 0 : θ 0 θ is called a composite hypothesis. Florian Pelgrin (HEC) Constrained estimators September-December / 34

8 Hypothesis testing Different tests... Definition A test of the form: { H0 : θ 0 = θ H a : θ 0 θ is called a two-sided test. Example: { H0 : β 0,j = 0 H a : β 0,j 0 The null hypothesis means x j has no effect on y once the other x s are controlled for ; β 0,j 0 is a natural alternative hypothesis when there is no prior knowledge about the sign... Florian Pelgrin (HEC) Constrained estimators September-December / 34

9 Hypothesis testing Different tests... Definition A test of the form: { H0 : θ 0 θ H a : θ 0 > θ or { H0 : θ 0 θ H a : θ 0 < θ is called a one-sided test. Florian Pelgrin (HEC) Constrained estimators September-December / 34

10 Hypothesis testing Outcomes of hypothesis testing 2.3. Outcomes of hypothesis testing Truth Decision Fail to reject H 0 Reject H 0 H 0 Correct decision Type I error H a Type II error Correct decision 1 The probability of Type I error: P(reject H 0 H 0 is true). 2 The probability of Type II error: P(fail to reject H 0 H 1 is true). Florian Pelgrin (HEC) Constrained estimators September-December / 34

11 Hypothesis testing Outcomes of hypothesis testing Definition The power function of a test is defined by: β(θ 0 ) = 1 P(fail to reject H 0 H 1 ) 1 η(θ 0 ) The size of a test is defined to be: α = sup θ 0 P(reject H 0 H 0 ) A test is said to have level α if its size is less than or equal to α. Remark: One seeks to determine the critical region such that the power is maximum and with minimal size distortion. Florian Pelgrin (HEC) Constrained estimators September-December / 34

12 Hypothesis testing 2.4. Level of significance Level of significance The decision Reject H 0 or fail to reject H 0 is not so informative! Indeed, there is some arbitrariness to the choice of α (level). Another strategy is to ask, for every α, whether the test rejects at that level. Another alternative is to use the so-called p-value the smallest level of significance at which H 0 would be rejected given the value of the test-statistic. Florian Pelgrin (HEC) Constrained estimators September-December / 34

13 Hypothesis testing Level of significance More formally... Definition Suppose that for every α (0, 1), one has a size α test with rejection region W α. Then, the p-value is defined to be: p-value = inf{α : T S (Y ) W α }. The p-value is the smallest level at which one can reject H 0. The p-value is a measure of evidence against H 0 : p-value evidence <.01 Very strong evidence against H Strong evidence against H Weak evidence against H 0 >.10 Little or no evidence against H 0 Florian Pelgrin (HEC) Constrained estimators September-December / 34

14 Hypothesis testing Level of significance Remarks: 1. A large p-value does not mean strong evidence in favor of H A large p-value can occur for two reasons: 1 H 0 is true; 2 H 0 is false but the test has low power. 3. The p-value is not the probability that the null hypothesis is true! Florian Pelgrin (HEC) Constrained estimators September-December / 34

15 Hypothesis testing 2.5. General methodology General methodology Hypothesis testing is defined by the following general procedure Step 1: State the relevant null and alternative hypotheses (mis-stating the hypotheses muddies the rest of the procedure!); Step 2: Consider the statistical assumptions being made about the sample in doing the test (independence, distributions, etc) incorrect assumptions mean that the test is invalid! Step 3: Choose the appropriate test (exact or asymptotic tests) and thus state the relevant test statistic (say, T ). Step 4: Derive the distribution of the test statistic under the null hypothesis (sometimes it is well-known, sometimes it is more tedious!) for example, the Student t-distribution or the Fisher distribution. Florian Pelgrin (HEC) Constrained estimators September-December / 34

16 Hypothesis testing General methodology 2.5. General methodology (continued) Step 5: Determine the critical region (and thus the acceptance region). Step 6: Compute (using the observations!) the observed value of the test statistic T, say t obs. Step 7: Decide to either fail to reject the null hypothesis or reject in favor of the alternative assumption the decision rule is to reject the null hypothesis H 0 if the observed value of the test statistic, t obs is in the critical region, and to fail to reject the null hypothesis otherwise. Florian Pelgrin (HEC) Constrained estimators September-December / 34

17 3. The Student test 3.1. Framework The Student test Framework Consider the (population) model Y = Xβ 0 + u that satisfies the assumptions H1 H6 (Chapter 2). The β j s are unknown features of the population, but: One can formulate a hypothesis about their value; One can construct a test statistic with a known distribution under the maintained hypothesis; One can take a decision meaning reject H 0 if the value of the test statistic is too unlikely. Florian Pelgrin (HEC) Constrained estimators September-December / 34

18 The Student test Framework Three tests of interest: { H0 : β 0,j = a j H a : β 0,j > a j or < a j { H0 : β 0,j = a j H a : β 0,j a j and { H0 : β 0,j = β l H a : not H 0 where a j = 0 or a j 0. Florian Pelgrin (HEC) Constrained estimators September-December / 34

19 The Student test Framework All of these tests are based on the following results: 1. The exact distribution of the ordinary least squares estimator of β: ( ˆβ OLS X N β 0, σ0 2 (X X) 1). 2. Any linear combination of the ˆβ j,ols s is also normally distributed: A ˆβ OLS X N ( Aβ 0, σ 2 0A (X X) 1 A ). 3. Any subset of ˆβ OLS has a joint normal distribution. Florian Pelgrin (HEC) Constrained estimators September-December / 34

20 The Student test 3.2. Student test-statistic Student test-statistic Definition The Student test-statistic is defined to be: T s = ˆβ j,ols a ( ) j Γ(n k) se ˆβ j,ols where n is the number of observations, k is the number of explanatory variables (including the constant term), Γ is the Student t-distribution, and: ( ) se ˆβ j,ols = ˆσ OLS mjj with m jj is the j th diagonal element of (X X) 1. Proof: See Appendix 1. Florian Pelgrin (HEC) Constrained estimators September-December / 34

21 The Student test Decision rules 3.3. Decision rules Testing H 0 : β 0,j = a j against H a : β 0,j > a j Definition The critical region is that H 0 is rejected in favor of β 0,j > a j at the α% (say, 5%) significance level if: t obs > c 1 α where c 1 α is the α% (say, 5%) critical value or the percentile of order 1 α of a Student t-distribution with n k degrees of freedom, i.e. the solution of: α = P (reject H 0 H 0 is true) = P (T s > c 1 α ). Florian Pelgrin (HEC) Constrained estimators September-December / 34

22 The Student test Decision rules Figure 1: 5% rejection rule for H a : β 0,j > 0 (df=28) Area = t Rejection region 5% rejection rule for (df=18) Florian Pelgrin (HEC) Constrained estimators September-December / 34

23 The Student test Decision rules Testing H 0 : β 0,j = a j against H a : β 0,j < a j Definition The critical region is that H 0 is rejected in favor of β 0,j < a j at the α% (say, 5%) significance level if: t obs < c α where c α is the α% (say, 5%) critical value or the percentile of order α of a Student t-distribution with n k degrees of freedom, i.e. the solution of: α = P (reject H 0 H 0 is true) = P (T s < c α ). Florian Pelgrin (HEC) Constrained estimators September-December / 34

24 The Student test Decision rules Figure 2: 5% rejection rule for H a : β 0,j < 0 (df=18) Area = Rejection region t Florian Pelgrin (HEC) Constrained estimators September-December / 34

25 The Student test Decision rules Testing H 0 : β 0,j = a j against H a : β 0,j a j Definition The critical region is that H 0 is rejected in favor of β 0,j a j at the α% (say, 5%) significance level if: t obs > c 1 α 2 where c 1 α is the α 2 2 % (say, 2.5%) critical value or the percentile of order 1 α 2 of a Student t-distribution with n k degrees of freedom, i.e. the solution of: α ( ) 2 = P (reject H 0 H 0 is true) = P T s > 1 c 1 α. 2 Remark: c 1 α is chosen such that the probability mass in each tail of 2 the t-distribution equals α 2. Florian Pelgrin (HEC) Constrained estimators September-December / 34

26 The Student test Decision rules Figure 3: 5% rejection rule for H a : β 0,j 0 (df=25) Area = Area = Rejection region t Rejection region Other hypotheses about Florian Pelgrin (HEC) Constrained estimators September-December / 34

27 The Student test Decision rules Testing a linear combination of the parameters Suppose the test is: { H0 : β 0,j = β 0,l H a : β 0,j β 0,l The Student test statistic is given by: T s = ˆβ j,ols ˆβ ( l,ols ) se ˆβ j,ols ˆβ l,ols where ( se ˆβj,OLS ˆβ ) l,ols = V ( ) ( ) ( ˆβj,OLS + V ˆβl,OLS 2Cov ˆβj,OLS, ˆβ ) l,ols. The procedure is then the same as a standard two-sided test. Florian Pelgrin (HEC) Constrained estimators September-December / 34

28 The Fisher test Framework 4. The Fisher test 4.1. Framework Consider the linear Gaussian regression model (with suitable regularity conditions) and suppose that one seeks to test: H 0 : Rβ 0 = q constrained model H a : Rβ 0 q unconstrained model. Florian Pelgrin (HEC) Constrained estimators September-December / 34

29 The Fisher test Fisher test-statistic 4.2. Fisher test-statistic Definition The Fisher test statistic is given by: where T F = SSR 0 SSR a SSR a dl a dl 0 dl a F(dl 0 dl a, dl a ). dl 0 is the number of degrees of freedom under the null hypothesis (i.e., dl 0 = n (k p)); dl a is the number of degrees of freedom under the alternative hypothesis (i.e., dl a = n k); SSR 0 is the sum of squared (estimated) residuals under the null hypothesis; SSR a is the sum of squared (estimated) residuals under the alternative hypothesis. Florian Pelgrin (HEC) Constrained estimators September-December / 34

30 The Fisher test Fisher test-statistic Definition Consider the Gaussian multiple linear regression model: Y = Xβ 0 + u. The Fisher test statistic corresponding to { H0 : Rβ 0 = q H a : Rβ 0 q is given by: T F = û COLS 2 û OLS 2 û OLS 2 n k p F(p, n k). Florian Pelgrin (HEC) Constrained estimators September-December / 34

31 The Fisher test Fisher test-statistic A second derivation of the Fisher test statistic Definition Under suitable regularity conditions, the Fisher test statistic can also be written as: T F = 1 p σ 2 OLS ( R β ) OLS q [R(X X) 1 R ] 1 ( R β ) OLS q. Remark: An equivalent test statistic is also: T F = 1 ( βols p σ OLS 2 β ) COLS (X X) ( βols β ) COLS. Exercise: Show these two results. Florian Pelgrin (HEC) Constrained estimators September-December / 34

32 4.3. Decision rule The Fisher test Decision rule 1 Determine SSR 0 (constrained model) and SSR a (unconstrained model); 2 Determine the realization of the Fisher test statistic: F obs = (SSR 0 SSR a ) /p SSR a / (n k) where p is the number of linearly independent constraints. 3 Determine the critical region: W = {T F > F 1 α } and especially the critical value which corresponds to the (1 α) percentile of a Fisher distributed random variable F(p, n k). 4 Reject the null assumption H 0 (at the level α) if F obs > F 1 α. Florian Pelgrin (HEC) Constrained estimators September-December / 34

33 5. Summary Summary What is a simple hypothesis? A composite hypothesis? What is a one-sided test? A two-sided test? What is Type-I error? Type-II error? What is the p-value? What is the critical value? Decision rule? What is a Student test? In which cases can we use it? Derive the Student test statistic in standard cases. What is a Fisher test? In which sense(s) is it different from the Student test? Derive the different Fisher test statistics. Interpretation of results! Florian Pelgrin (HEC) Constrained estimators September-December / 34

34 Appendix Appendix 1: Student test statistic The result is based on the following theorem: Theorem Let U and V denote two independent random variables: Then: Using U = ˆβ j,ols a j σ jj U N (0, 1) Z χ 2 (n 1 ). U Z n 1 Γ(n 1 ). (with σ jj = σ 0 mjj ), Z = (n k) ˆσ2 OLS, n σ0 2 1 = n k, and the result that U are V are independent in probability (see Chapter 2), one gets the result. Florian Pelgrin (HEC) Constrained estimators September-December / 34