The Two Variable Linear Model

Transcription

1 Chapter 1 The Two Varable Lnear Model 1.1 The Basc Lnear Model The goal of ths secton s to buld a smple model for the non-exact relatonshp between two varables Y and X, related by some economc theory. For example, consumpton and ncome, quantty consumed and prce, etc. The proposed model: Y = α + βx + u, = 1,..., n (1.1) where α and β are unknown parameters whch are the purpose of the estmaton. What we wll call data are the n realzatons of (X, Y ). We are abusng notaton a bt by usng the same letters to refer to random varables and ther realzatons. u s an unobserved random varable whch represents the fact that the relatonshp between Y and X s not exactly lnear. We wll momentarly assumet that u has expected value zero. Note that f u = 0, then the relatonshp between Y and X would be exactly lnear, so t s the presence of u what breaks ths exact nature of the relatonshp. Y s usually reffered to as the explaned or dependent varable, X s the explanatory or ndependent varable. We wll refer to u as the error term, whch s a termnology more approprate n the expermental scences, where a cause x (say the dose of a drug) s admnstered to dfferent subjects and then an effect y s measured (say, body temperature). In ths case u mght be a measurement error due to the erratc behavor of a measurement nstrument (for example, a thermometer). In a socal scence lke economcs, u represents a broader noton of gnorance that represents whatever s not observed (by gnorance, ommson, etc.) that affects y besdes x. [ FIGURE 1: SCATTER DIAGRAM ] The frst goal wll be to fnd reasonable estmates for α and β based solely on the data, that s (X, Y ), = 1,..., n. 1

2 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 1. The Least Squares Method Let us denote wth ˆα and ˆβ the estmates of α and β n the smple lnear model. Let us also defne the followng quanttes. The frst one s an estmate of Y : Ŷ ˆα + ˆβX Intutvely, we have replaced α and β by ts estmates, and treated u as f the relatonshp were exactly lnear,.e., as f u were zero. Ths wll be undersood as an estmate of Y. Then t s natural to defne a noton of estmaton error as follows: e Y Ŷ whch measures the dfference between Y and ts estmate. A natural goal s to fnd ˆα and ˆβ so as e s are small n some sense. It s nterestng to see how the problem works from a graphcal perspectve. Data wll correspond to n ponts scattered n a (X, Y ) plane. The presence of a lnear relatonshp lke (1.1) s consstent wth ponts scatterd around an magnary straght lne. Note that f u where ndeed zero, all ponts wll le along the same lne, consstent wth an exact lnear relatonshp. As mentoned above, t s the presence of u what breaks ths exact relatonshp. Now note that for any gven values of ˆα and ˆβ, the ponts determned by the ftted model: Ŷ ˆα + ˆβX correspond to a lne n the (X, Y ) plane. Hence dfferent values of ˆα and ˆβ correspond to dfferent estmated lnes, whch mples that choosng partcular values s equvalent to choosng a specfc lne on the plane. For the -th observaton, the estmaton errors e can be seen graphcally as the vertcal dstance between the ponts (X, Y ) and (X, Ŷ), that s, between (X, Y ) and the ftted lne. So, ntutvely, we want values of ˆα and ˆβ so as the ftted lne they nduce passes as close as possble to all the ponts n the scatter so errors are as small as possble. [ FIGURE : SCATTER DIAGRAM WITH CANDIDATE LINE] Note that f we had only two observatons, the problem has a very smple soluton, and reduces to fndng the only two values of ˆα and ˆβ that make estmaton errors exactly equal to zero. Graphcally, ths s possble snce ths s equvalent to fndng the only straght lne that passes through the two observatons avalable. Trvally, n ths extreme case all estmaton errors wll be zero. The more realstc case appears when we have more than two observatons, not all of them lyng on a sngle lne. Obvously, a lne cannot pass through more than two nonalgned ponts, so we cannot make all errors equal to zero. So now the problem s to fnd values of ˆα and ˆβ that determne a lne that passes the closest as posble to all the ponts, so estmaton errors are, n the aggregate, small. For ths we need to ntroduce a crteron

3 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 3 of what do we mean by the lne beng close or far from the ponts. Let us defne a penalty functon, whch conssts n addng all the estmaton errors squared, so as postve and negatve errors matter alke. For any ˆα and ˆβ, ths wll gve us an dea of how large s the aggregate estmaton error: SSR(ˆα, ˆβ) n = e = (Y ˆα ˆβX ) =1 SSR stands for sum of squared resduals. Note that gven the observatons Y and X, ths s a functon that depends on ˆα and ˆβ, that s, dfferent values of ˆα and ˆβ correspond to dfferent lnes that pass through the data ponts, mplyng dfferent estmaton errors. It s now natural to look for ˆα and ˆβ so as to make ths aggregate error as small as possble. The values of ˆα and ˆβ that mnmze the sum of squared resduals are: ˆβ = X Y nȳ X X n X and ˆα = Ȳ ˆβ X whch are known as the least squares estmators of β and α. Dervaton of the Least Squares Estmators The next paragraphs show how to obtan these estmators. Fortunately, t s easy to show that SRC(ˆα, ˆβ) s globally concave and dfferentable, so frst order condtons for a local mnmum are: SRC(ˆα, ˆβ) ˆα SRC(ˆα, ˆβ) ˆβ = 0 = 0 The frst order condton s: e ˆα = (Y ˆα ˆβX ) = 0 (1.) Dvdng by mnus and dstrbutng the summatons: Y = nˆα + ˆβ X (1.3) Ths last expresson s very mportant, and we wll return to t frequently. From the second frst order condton:

4 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 4 e ˆβ = X (Y ˆα ˆβX ) = 0 (1.4) Dvdng by - and dstrbutng the summatons: X Y = ˆα X + ˆβ X (1.5) (1.3) and (1.5) form a system of two lnear equatons wth two unknowns (ˆα y ˆβ) known as the normal equatons. Dvdng (1.3) by n and solvng for ˆα we get: Replacng n (1.5): ˆα = Ȳ ˆβ X (1.6) X Y = (Ȳ ˆβ X) X + ˆβ X X Y = Ȳ X ˆβ X X + ˆβ X X Y Ȳ X = ˆβ ( X X X ) ˆβ = X Y Ȳ X X X X Note that: X = X /n then Z = Zn. Replacng, we get: ˆβ = X Y nȳ X X n X (1.7) It wll be useful to adopt the followng notaton. x = X X, and y = Y Ȳ, so lowercase letters denote the observatons as devatons from ther sample means. Usng ths notaton: x y = (X X)(Y Ȳ ) = (X Y X Ȳ XY I + XȲ ) = X Y Ȳ X X Y + n XȲ = X Y nȳ X n XȲ + n XȲ = X Y nȳ X

5 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 5 corresponds to the numerator of (1.7). Makng a smlar operaton n the denomnator of (1.7) we get the followng alternatve expresson for the least squares estmate of β: ˆβ = x y x [ FIGURE 3: SCATTER DIAGRAM AND OLS LINE ] 1.3 Algebrac Propertes of Least Squares Estmators By algebrac propertes of the estmator we mean those that are a drect consequence of the mnmzacon process, stressng the dfference wth statstcal propertes, whch wll be studed n the next secton. Property 1: e = 0 From the frst normal equaton (1.), dvdng by mnus and replacng by the defnton of e we easly verfy that as a consequence of mnmzng the sum of squared resduals, the sum of the resduals, and consequently ther average, s equal to zero. Property : X e = 0. Ths can be checked by dvdng by mnus n the second normal equaton (1.4). The covarance between X and e s gven by: Cov(X, e) = = = 1 (X n 1 X)(e ē) 1 [ X e ē X n 1 X e + ] Xē 1 X e n 1 snce from the prevous property e and hence ē are equal to zero. Then, ths property says that as a consequence of usng the method of least squares the sample covarance between the explanatory varable X and the error term e s zero, or, whch s the same, the resduals are lnearly unrelated to the explanatory varable. Property 3: The estmated regresson lne corresponds to the functon Ŷ (X) = ˆα+ ˆβX where e take ˆα and ˆβ as parameters, so as Ŷ s a functon that depends on X. Consder what happens when we evaluate ths functon at X, the mean of X: But from (1.6): Ŷ ( X) = ˆα + ˆβ X ˆα + ˆβ X = Ȳ Then Ŷ ( X) = Ȳ, ths s, the estmated regresson lne by the method of least squares passes through the pont of means.

6 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 6 Property 4: Relatonshp between regresson and correlaton: Remember that the sample correlaton coeffcent between X and Y for a sample of n observatons (X, Y ), = 1,,..., n s defned as: r XY = Cov(X, Y ) S X S Y The followng result establshes the relatonshp between r XY and ˆβ. ˆβ = x y x = x y x x = x y x x y y = x y x y y / n x / n ˆβ = r S Y S X If r = 0 then ˆβ = 0.Note that f both varables have the same sample varance, then the correlaton coeffcent s equal to the regresson coeffcent ˆβ. We can also see that, unlke the correlaton coeffcent, ˆβ s not nvarant to changes n scales or unt of measurement. Property 5: The sample means of Y and Ŷ are the same. By defnton, Y = Ŷ + e for = 1,..., n. Then, summng for every : Y = Ŷ + e and dvdng by n: Y Ŷ n = n snce e = 0 from the frst order condtons. Then: whch s the desred result. Ȳ = Ŷ

7 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 7 Property 6: ˆβ s a lnear functon of the Y s. Ths s, ˆβ can be wrtten as ˆβ = w Y, where the w s are real numbers not all of them equal to zero. Ths s easy to prove. Let us start by wrtng ˆβ as follows: ˆβ = ( ) x x y and call w = x / x. Note that: x = (X X) = X n X = 0 whch mples w = 0. From the prevous result: whch gves the desred result. ˆβ = w y = w (Y Ȳ ) = w Y Ȳ w = w Y Ths does not have much ntutve meanng so far, but t wll be a useful for later results. 1.4 The Two-Varable Lnear Model under the Classcal Assumptons Y = α + βx + u, = 1,..., n In addton the the lnear relatonhps beteween Y and X we wll assume: 1. E(u ) = 0, = 1,,..., n. On average the relatonshp between Y and X s lnear.. V ar(u ) = E[(u E(u )) ] = Eu = σ = 1,,..., n. The varance of the error term s constant for all observatons. We wll say that the error term s homoskedastc. 3. Cov(u, u j ) = 0 j. The error term for an observaton s not lnearly related to the error term of any other dfferent observaton j. If varables are measured over tme,.e., = 1980, , 1997 we wll say that there s no autocorrelaton. In general, we wll say that there s no seral correlaton. Note that snce E(u ) = 0, assumng Cov(u, u j ) = 0 s equvalent to assumng E(u u j ) = The values of X are non-stochastc and not all of them equal.

8 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 8 The classcal assumptons provde a basc probablstc structure to study the lnear model. Most assumptons are of a pedagogc nature and we wll study later on how they can be relaxed. Nevertheless, they provde a smple framework to explore the nature of least squares estmator. 1.5 Statstcal Propertes of Least Squares Estmators Actually, the problem s to fnd good estmates of α, β and σ. The prevous secton presents estmates of the frst two based on the prncple of least squares so, trvally, these estmates are good n the sense that they mnmze certan noton of ft: they make the sum of squared resduals as small as possble. It s relevant to remark that n obtanng the least squares estmators we have made no use of the classcal assumptons descrbed above. Hence, the natural step s to explore whether we can deduce addtonal propertes satsfed by the least squares estmator, so we can say that t s good n a sense that goes beyond that mplct n the least squares crteron. The followng are called statstcal propertes snce they arse as a consequence of the statstcal structure of the model. We wll use repeatedly the followng expressons for the LS estmators: ˆβ = x y x ˆα = Ȳ ˆβ X We wll frst explore the man propertes of ˆβ n detal, and leave the analyss of ˆα as exercses. The startng conceptual pont s to see that ˆβ depends explctely on the Y s whch, n turn, depend on the u s whch are, by constructon, random varables. Then ˆβ s a random varable and then t makes sense to talk about ts moments (mean and varance, for example) and ts dstrbuton. It s easy to verfy that: y = x β + u where u = u ū, and, accordng to the classcal assumptons, E(u ) = 0 and, consequently, E(y ) = x β. Ths s known as the classcal two-varables lnear model n devatons form the means. ˆβ s an unbased estmator, that s: E( ˆβ) = β To prove the result, from the lnearty property of the prevous secton ˆβ = w y E( ˆβ) = w E(y ) (w s are non-stochastc) = w x β

9 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 9 = β w x = β x /( x ) = β The varance of ˆβ s σ / x From the lnearty property, ˆβ = w Y, then V ( ˆβ) ( ) = V w Y Now note two thngs. Frst: V (Y ) = V (α + βx + u ) = V (u ) = σ snce X s non-stochastc. Second, note that E(Y ) = α + βx, so Cov(Y, Y j ) = E [(Y E(Y ))(Y j E(Y j ))] = E(u u j ) = 0 by the no seral correlaton assumpton. Then V ( w Y ) s the varance of (weghted) sum of uncorrelated terms. Hence V ( ˆβ) ( ) = V w Y = w V (Y ) = σ w = σ [ ] (x )/ x = σ / x Gauss-Markov Theorem: under the classcal assumptons, ˆβ, the LS estmator of β, has the smallest varance among the class of lnear and unbased estmators. More formally, f β s any lnear and unbased estmator of β then: V (β ) V ( ˆβ) The proof of a more general verson of ths result wll be postponed untl Chapter 3. Dscusson: BLUE, best does not mean good, we want mnmum varance unbased (wthout lnear ), lnear s not an nterestng class, etc. If we drop any assumpton, the OLS estmate s no longer BLUE. Ths justfes the use of OLS when all the asumptons are correct.

10 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 10 Estmaton of σ So far we have concentrated the analyss on α and β. As an estmate for σ we wll propose: S = e n We wll later show that S provdes and unbased estmator for σ. 1.6 Goodness of ft After estmatng the parameters of the regresson lne, t s nterestng to check how well does the estmated model ft the data. We want a measure of how well does the ftted lne represent the observatons of the varables of the model. To look for such measure of goodness of ft, we start from the defnton of ftted value e = Y Ŷ, solve for Y and substract n both members the sample mean of Y to obtan: Y Ȳ = Ŷ Ȳ + e y = ŷ + e usng the notaton defned before and notng that from Property 4, Ȳ = Ŷ. Takng the square of both sdes and summng over all the observatons: y = (ŷ + e ) = ŷ + e + ŷ e y = ŷ + e + ŷ e The next step s to show that ŷ e = 0: ŷ e = (ˆα + ˆβX )e = ˆα e + ˆβ X e = from the frst order condtons. Then we get the followng mportant decomposton: y = ŷ + e T SS = ESS + RSS Ths s a key result that ndcates that when the we use the least squares method, the total varablty of the dependent varable (TSS) around ts sample mean can be decomposed as the sum of two factors. The frst one corresponds to the varablty of Ŷ (ESS) and represents the varablty explaned by the ftted model. The second term represents the varablty not explaned by the model (RSS), assocated to the error term.

11 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 11 For a gven model, the best stuaton arses when errors are all zero, n whch case the total varablty (TSS) conncdes wth the explaned varablty (ESS). The worst case corresponds to the stuaton n whch the ftted model does not explan anythng of the total varablty, n whch case TSS concdes wth RSS. From ths observaton, t s natural to suggest the followng goodness of ft measure, known as R, or coeffcent of determnaton: R = SCE SCT = 1 SCR SCT It can be shown (we wll do t n the exercses) that R = r. Consequently, 0 R 1. When R = 1 r = 1, whch corresponds to the case n whch the relatonshp between Y and X s exactly lnear. On the other hand, R = 0 s equvalent to r = 0, whch corresponds to the case n whch Y and X are lnearly unrelated. It s nterestng to note that T SS does not depend on the estmated model, that s, t does not depend on ˆβ nor ˆα. Then, f ˆβ and ˆα are choosen so as to mnmze SSR then they automatcally maxmze R. Ths mples that, for a gven model, the least squares estmate maxmzes R. The R s, arguably, the most used and abused measure of qualty of a regresson model. A detaled analyss of the extent to whch a hgh R can be taken as representatve of a good model wll be undertaken n Chapter Inference n the two-varable lnear model The methods dscussed so far provde reasonably good pont estmates of the parameters of nterest α, β and σ but usually we wll be nterested n evaluatng hypotheses nvolvng the parameters, or constructng confdence ntervals for them. For example, consder the case of a smple consumpton functon where consumpton s specfed as a smple lnear functon of ncome. We could be nterested n evaluatng whether the margnal propensty to consume s equal to, say, 0.75, or that autonomous consumpton s equal to zero. In general terms, a hypothess about a parameter of the model s a conjecture about t, that can be ether false or true. The central problem s that n order to check whether such statement s true or false we do not have the chance to observe such a parameter. Instead, based on the avalable data, we have an estmate of t. As an example, suppose we are nterested n evaluatng the, rather strong, null hypothess that ncome s not an explanatory factor of consumpton, aganst the hypothess that t s a relevant factor. In our smple setup ths corresponds to H 0 : β = 0 aganst H A : β 0. The logc we wll use s the followng: f the null hypothess were n fact true β would be exactly zero. Realzatons of ˆβ can potentally take any value, snce ˆβ s, by constructon, a random varable. But f ˆβ s a good estmator of β, when the null hypothess s true t should take values close to zero. On the other hand, f the null hypothess were false, the realzatons of ˆβ should be sgnfcantly dfferent from zero. Then, the procedure conssts n computng ˆβ from the data, and reject the null f the obtaned value s sgnfcantly dfferent from zero, or accept otherwse.

12 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 1 Of course, the central concept behnd ths procedure les n specfyng what do we mean by very close or very far, gven that ˆβ s a random varable. More specfcally, we need to know the dstrbuton of ˆβ under the null hypothess so we can defne precsely the noton of sgnfcantly dfferent from zero. In ths context such a statement s necessarly probablstc, that s, we wll take as the rejecton regon a set of values that le far away from zero, or, a set of values that under the null hypothess appear wth very low probablty. The propertes dscussed n the prevous secton are nformatve about certan moments of ˆβ or ˆα (for example, ther means and varances) but they are not enough for the purposes of knowng ther dstrubutons. Consequently, we need to ntroduce an addtonal assumpton. We wll assume that u s normally dstrbuted, for = 1,..., n. Gven that we have already assumed that u has zero mean and constant varance equal to σ, we have: u N(0, σ ) Gven that Y = α + βx + u and that the X s are non-stochastc, we mmedately see that the Y s are also normally dstrbuted snce lnear transformatons of normal random varables are also normal. In partcular, gven that the normal dstbuton can be characterzed by ts mean and varance only, we get: Y N(α + βx, σ ), for every = 1..., n. In a smlar fashon ˆβ s also normally dstrbuted snce by Property 1 t s a lnear combnaton of the Y s, that s: ˆβ N(β, σ / x ) If σ were known we could use ths result to test smple hypothess lke: H o : β = β o vs. H A : β β o Substractng from ˆβ ts expected value and dvdng by ts standard devaton we get: z = ˆβ β o N(0, 1) σ/ x Hence, f the null hypothess s true, z should take values that are small n absolute value, and large otherwse. As you should remember from a basc statstcs course, ths s acomplshed by defnng a rejecton regon and an acceptance regon as follows. The acceptance regon ncludes values that le close to the one correspondng to the null hypothess. Let c < 1 and z c be a number such that: Replacng z by ts defnton: P r( z c z z c ) = 1 c

13 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 13 ( ) ( P r [β o z c σ/ x ˆβ )] β o + z c σ/ x = 1 c Then the acceptance regon s gven by the nterval: β o ± z c (σ/ x ) so we accept the null hypothess f the observed realzaton of ˆβ les wthn ths nterval and reject otherwse. The number c s specfed n advance and t s usually a small number. It s called the sgnfcance of the test. Note that t gves the probablty that we reject the null hypohtess when t s correct. Under the normalty assumptons, the value z c can be easly obtaned from a table of percentles of the standard normal dstrbuton. As you should also remember from a basc statstcs class, a smlar logc can be appled to construct a confdence nterval for β 0. Note that: [ ] P r ˆβ z c (σ/ x ) β o ˆβ + z c (σ/ x ) = 1 c Then a 1 c confdence nterval for β 0 wll be gven by: ) ˆβ ± z c (σ/ x The practcal problem wth the prevous procedures s that they requre that we know σ, whch s usually not avalable. Instead, we can compute ts estmated verson S. Defne t as: t = ˆβ β S/ x t s smply z where we have replaced σ by S. A very mportant result s that by dong ths replacement we have: t t n that s, the t-statstc has the so-called t-dstrbuton wth n degrees of freedom. Hence, when we use the estmated verson of the varance we obtan a dfferent dstrbuton for the statstc used to test smple hypotheses and construct confdence ntervals. Consequently, applyng once agan the same logc, n order to test the null hypothess H o : β = β o aganst H A : β β o we use the t-statstc: t = ˆβ β o S/ x t n

14 CHAPTER 1. THE TWO VARIABLE LINEAR MODEL 14 and a 1 c confdence nterval for β 0 wll be gven by: ˆβ ± t c (S/ x ) where now t c s a percentle of the t dstrbuton wth n degrees of freedom, whch s usually tabulated n basc statstcs and econometrcs textbooks. An mportant partcular case s the nsgnfcance hypothess, that s H o : β o = 0 aganst H A : β 0 0. Under the null X does not help explan Y, and under the alternatve, X s lnearly related to Y. Replacng β o by 0 above we get: t I = ˆβ S/ x t n whch s usually reported as a standard outcome n most regresson packages. Another alternatve to check for the sgnfcance of the lnear relatonshp s to look at how large s the explaned sum of squares ESS. Recall that f the model has an ntercept we have that: T SS = ESS + RSS If there s no lnear relatonshp between Y and X, ESS should be very close to zero. Consder the followng statstc, whch s just a standardzed verson of the ESS: F = ESS RSS/(n ) It can be shown that under the normalty assumpton, F has the F dstrbuton wth 1 degree of freedom n the numerator, and n degrees of freedom n the denomnator, whch s usually labeled as F (1, n ). Note that f X does not help explan Y n a lnear sense, ESS should be very small, whch would make F very small. Then, we should reject the null hypothess that X does not help explan Y s the F statstc computed from the data takes a large value, and accept otherwse. Note that by defnton R = ESS/T SS = 1 RSS/T SS. Dvde both the numerator of the F statstc by T SS. Solvng for ESS and RSS and replacng above we can wrte the F statstc n terms of the R coeffcent as: F = R (1 R )/(n ) Then, the F test s actually lookng at whether the R s sgnfcantly hgh. As t s expected, there s a close relatonshp between the F statstc and the t statstc for the nsgnfcance hypothess (t I ). In fact, when there s no lnear relatonshp between Y and X, ESS s zero, or β 0 = 0. In fact, t can be easly shown that: We wll leave the proof as an excercse. F = t I