Econometrics Deriving the OLS estimator in the univariate and bivariate cases

Transcription

1 Econometrcs Dervng the OLS estmator n the unvarate and bvarate cases athanel Hggns nhggns@jhu.edu Prmer otaton Here are some bts of notaton that you wll need to understand the dervatons below. They are standard n ths class, Wooldrdge, and (for the most part) econometrcs generally. Referrng to varables When we wrte x we usually mean to refer to a varable n general. When we wrte x we mean the th observaton of the varable x. You can thnk of x as the frst observaton of x, e.g. Summaton notaton The equaton n = s a common example of summaton notaton. Ths s just a compact way of wrtng x x + x x n. When we wrte x we mean the average of x. x = x, where s the number of observatons n our sample.

2 Regresson notaton The equaton you wll see the most as you learn econometrcs s ths one: y = β 0 + β x + u. Varables y: The dependent varable (the varable that we are tryng to explan) x: The ndependent varable (the varable that we thnk causes y or s assocated wth changes n y) u: The unobservables n the model (the stuff that changes y that s not part of our data) Operators The one you need to know about for ths document s the expectaton operator. The expectaton of a random varable s, well, what we expect to get when we observe a draw of that random varable. So f x s some varable, I expect to observe E(x) when I draw an observaton of x. We have a specal way of wrtng the expectaton of a random varable: E(x) = µ x. Bascally, µ x s just a compact way of representng the fact that the expectaton of a random varable s a fxed value. When we wrte µ x we refer to the true mean of x. That s, the expectaton of x, E(x), s some fxed number µ x. The true mean of x s not to be confused wth the mean of x n any gven sample of data. We denote the sample mean of x by x. We stll need to use the expectaton operator even though we have the specalzed µ notaton, snce sometmes we wll be takng the expectaton of more than just x, e.g. E(x y) (the expectaton of x tmes y). For the purposes of tryng to understand what the expectaton operator does, t may help to thnk of the expectaton as the average. (The average usually refers to the average of a sample of data, however, whereas the expectaton refers to a true parameter. But t doesn t hurt to substtute average for expectaton when you are tryng to grasp the concept.) Dervng the OLS estmator. Method of moments Ths s the way that Wooldrdge does t. I ll do t the same way here, but I ll fll n some of the blanks that Wooldrdge leaves for the appendx. The method of moments s an appealng way to derve estmators. It doesn t use a shred of calculus! (A lttle bt of algebra s absolutely unavodable) The method of moments s based on the dea that for every constrant that we mpose on the data, we can dentfy one parameter. We make an assumpton about our model. Then we use ths assumpton as a rule. Instead of just assumng t s true, we make t true n our data. That s, we make the

3 data that we have (our sample) obey ths rule. Each rule that we mpose gves us the ablty to say what one parameter must be for the rule to hold... Unvarate model Start wth the smplest possble model: what I call the unvarate regresson model, or a model wth no x-varable. A model where y s always equal to a constant: y = β 0 + u. There s one unknown parameter n ths model: β 0. Therefore we need one assumpton one rule to dentfy β 0. We wll mpose ths rule on the data, and ths wll gve us the formula for our best estmate of β 0, whch we wll denote β 0. The standard assumpton we wll use s E(u) = 0, the assumpton that the model s correct, on average. (ot to be confused wth the unreasonable assumpton that the model s always correct) The method of moments works by mposng ths assumpton on the data. So we start wth E(u) = 0, and we make ths true n the sample of data that we have. ote that so that u = y β 0, E(u) = E(y β 0 ). () What does t mean to make the assumpton true n our data? It means takng the concept of expectaton, whch s a concept about the unverse of data, and applyng the analogous concept to our sample of data. What s the sample analog of the expectaton? The mean. The average. The sample analog of the expected value of somethng s the mean of that same thng. So f the theoretcal expected value of a varable x s µ x, then the sample analog of µ x s x. Makng E(u) = 0 true n our data translates to forcng the average of u n our data to equal zero. Applyng ths noton to equaton (), we get E(u) = 0 = E(y β 0 ) = E(y) E(β 0 ) = y β 0 () = ȳ β 0 = ȳ β 0. (3) I don t thnk anybody else n the world calls t ths, so don t go throwng the term around n ntellgent econometrc conversaton unless you want people to look at you funny. 3

4 We now have a sngle, smple equaton wth a sngle unknown. We choose our estmate of β 0, whch we call β 0, to be the estmate that solves equaton (3). 0 = ȳ β 0 (4) β 0 = ȳ. (5) The best estmate of a varable y that we can manage when we model y as a constant s β 0 = ȳ, the mean of y... Bvarate model What I call the bvarate model s a model relatng the dependent varable y to an ndependent (or explanatory) varable x. The model looks lke ths: y = β 0 + β x + u. We can use the method of moments to derve estmates of β0 and β, just as we dd n the unvarate model above. The only dfference s that now we have two unknown parameters that we would lke to estmate, nstead of one. Ths means that we need two assumptons (two rules; two constrants) that we can place on the data n order to dentfy the two unknown parameters. Just as before, the assumpton E(u) = 0 wll serve as one of the constrants we place on the data. The other that Wooldrdge uses s the assumpton that Cov(x, u) = 0. Ths condton follows from the assumpton that x and u are ndependent. If two random varables are ndependent, then they must also have a covarance of zero. We wll mpose these two condtons on the data. E(u) = 0 = E(y β 0 β x) = E(y) E(β 0 ) E(β x) = y β 0 = ȳ β 0 β x β 0 = ȳ β x. β x (6) Why s the assumpton Cov(x, u) = 0 a good assumpton? Why s t somethng that we would lke to be true? Thnk of t ths way. We want to model the relatonshp between x and y. If we were runnng an experment where we had complete control over x, we could measure the relatonshp between x and y perfectly. We manpulate x, then smply watch how much y moves. All the movement n y that we observe s caused by movement n x. ow consder an alternatve realty. Suppose every tme you manpulated x, u moved too. So x s movng, u s movng, and of course, y s movng. We can no longer attrbute observed movements n y to x alone we don t know f t s x or u (or a combnaton of the two) that s causng y to move. What s the moral of the story? It would be really nce f when x moves, we know that other stuff sn t movng wth t. 4

5 Once agan we have dentfed the estmator that we wll use to estmate β 0. β 0 = ȳ β x (7) Only ths tme the unknown parameter β s part of the equaton. We wll get the formula for β usng the second condton (Cov(x, u) = 0), then plug ths estmate back nto (7). The second condton s Cov(x, u) = 0. Ths condton follows from the assumpton that x and u are statstcally ndependent. If they are ndependent, then ther covarance s zero. So we mpose zero covarance on the data n our sample. Here s the defnton of covarance between x and u (.e. what follows s smply an expanson of the defnton of covarance): Cov(x, u) = E[(x E(x))(u E(u))] = E[(x µ x )(u µ u )] = E[x u x µ u µ x u + µ x µ u ] = E(x u) E(x µ u ) E(µ x u) + E(µ x µ u ). (8) The frst lne of (8) expresses the defnton of covarance between two varables (see p of Wooldrdge). The second lne smply substtutes µ x for E(x) and µ u for E(u). Recall that E(x) s a theoretcal construct the expected value of the random varable x s equal to some fxed value, whch we call µ x. Lkewse wth u. The thrd lne expands the expresson usng the FOIL method. The fourth lne uses the fact that the expectaton operator (whch you could thnk of roughly as the average operator ) s lnear. 3 Let s pck up where we left off, wth the fnal lne of (8). ow we wll apply the populaton analog of the expectaton operator, just lke we dd n (6) above. E(x u) E(x µ u ) E(µ x u) + E(µ x µ u ) = E(x u) µ u E(x) µ x E(u) + µ x µ u = E(x u) µ u µ x µ x µ u + µ x µ u (9) = E(x u) µ x µ u = E(x u) The second lne of (9) uses the fact that µ u and µ x are constants. The expectaton of a constant tmes a varable s the same as the constant tmes the expectaton of the varable (f that seems lke math-ey jargon to you, just read the prevous sentence agan, substtutng the word average for expectaton ). Havng appled ths rule to obtan lne, we calculate E(x) and E(u) to obtan lne 3. At ths pont, t s straghtforward to collect terms and see that the fourth lne of (9) s the smplest way to express Cov(x, u). 3 Ths s just lke sayng that the average of q + r, q + r, s equal to the sum of the averages, q + r. Try t wth a few numbers f you need to convnce yourself. otce! The expectaton operator s lnear, so you can expand t over the four terms n (8) above, but t s not true that E(q r) = E(q) E(r). Just a note to be sure you don t msunderstand 5

6 Fnally, snce the expectaton of u s zero by assumpton (that was our frst condton we used n (7), remember?), the whole thng smplfes to just E(x u). ow we take ths smple expresson, and substtute n our regresson equaton, whch ncludes the two unknown parameters β 0 and β. We do ths n order to use the condton E(x u) = 0 to dentfy the two parameters. E(x u) = E(x (y β 0 β x)) = E(x y β 0 x β x ) = E(x y) E(β 0 x) E(β x ) = x y β 0 x β Usng only the rules we have already dscussed above, we now have an expresson for the sample value of E(x u). We now set ths equal to zero (recall that the condton we are mposng on the data s Cov(x, u) = 0) and solve for β. x y β 0 x β x x = 0 (0) But notce that the unknown term β 0 s part of the expresson n (0). Before we solve for β, then, we need to substtute n our estmator for β 0 (the expresson we got n (7) above). That s, when we see β 0 n (0), we wll substtute n the dentty β 0 = ȳ β x. 0 = x y β 0 x β x = x y (ȳ β () x) x β x ow we dstrbute terms by multplyng (ȳ β x) and x. 0 = x y (ȳ β x) x β x = x y ȳ x + β x x β x Fnally, we do two thng: () we dvde everythng n the equaton by / to elmnate t from the expresson (snce / currently multples everythng n the equaton, t has no effect, so we can get rd of t); () factor out β so that we can solve the equaton for β. 0 = x y ȳ x + β x x β x = x y ȳ x + β ( x β = x y ȳ x x x x 6 x x ) ()

7 We are now done. Ths expresson for β gves us our method-of-moments estmator of β, whch we call β. β = x y ȳ x x x x (3) You wll notce, however, that ths expresson s dfferent than expresson.9 on p. 9 of Wooldrdge. Both expressons are fully correct. Wooldrdge has smply re-arranged the terms above n (3) to make them more... nterpretable. By takng what you see n (3) and makng t look lke expresson.9 on page 9, you can see more clearly exactly what β s n terms of the data. For your own edfcaton, I ll show you below how (3) and expresson.9 are completely dentcal. I ll do ths n two parts. Frst I ll show that the numerator of (3) s equal to the numerator n expresson.9, then the denomnator. I ll work backwards from Wooldrdge s expresson of the numerator. (x x)(y ȳ) = x y x ȳ x y + xȳ = x y xȳ xȳ + xȳ = x y xȳ = x y ȳ x All I have done s use the fact that x = / x, whch mples that x = x (and lkewse for y). ow the denomnator. (x x) = = x x x x x x + x xx + x = = x x x + x x x = x x x As you can see, the Wooldrdge expresson and the expresson that we derved, workng step-by-step, are totally dentcal. Wooldrdge s expresson contans some ntutve nformaton: our estmate of the slope parameter β s equal to the sample covarance between x and y, dvded by the sample varance of x. Sad another way, our estmate of the relatonshp between x and y s equal to the covarance between x and y, normalzed by the varance of x. What does t mean to normalze, and why does ths make sense? 7

8 Why does t help us to thnk ths way? The covarance of any two varables (here x and y) tells us somethng about what happens to one varable when the other moves. But covarance s a term wthout scale. That s, there are no sensble unts of covarance. Ths s because the covarance of two varables depends on the unts n whch these two varables were measured. In the case of β, we can nterpret the estmate as the amount of covarance between x and y, measured n unts of varaton n x.. Method of least-squares ow we wll derve the exact same estmators, β0 and β, usng the least-squares procedure. You should see ths at least once. After all, the least-squares procedure s how the OLS estmator got ts name! We have some data, x and y. We have a model y = β 0 + β x + u. Our goal s to choose estmates β 0 and β that ft the data well. There are many possble ways to measure ft, but one that s partcularly appealng (and t s the one that s most wdely used n practce) s to mnmze a measure of how wrong our predctons of y are. Thnk of t ths way. If the true model s y = β 0 + β x + u, and we pck some estmates β 0 and β, then we can use these estmates to predct y: ŷ = β 0 + β x. But these predctons of y that we call ŷ are not always gong to be rght. If our model s good, they should be close, but they won t be exactly correct. They wll be off by some amount ŷ y. Ths amount by whch our predctons wll be off s a measure of the unobserved determnants of y. That s, t s a measure of the unexplaned part of y the varaton n y that our model β0 + β x does not explan. So we call t û. We want to choose the estmates β 0 and β that make the amount by whch our predctons are off, û, small. We could make û small my mnmzng û. That s, we could choose β 0 and β to mnmze û = y β 0 β x. But thnk of what ths wll do. If we want to make somethng really small, all we have to do s make t really negatve. To make the expresson for û really negatve, all we need to do s under-predct y all the tme. Ths sn t very nformatve, and t sn t n keepng wth our goal to fnd estmators that ft the data well. So we shouldn t try to mnmze û. We want to mnmze a measure that weghts ms-predctons above y and ms-predctons below y equally. There are two obvous canddate measures. We could mnmze the absolute value of û, or we could mnmze the square of û. Both are legtmate. As the name least-squares suggests, the standard estmator of β 0 and β 8

9 s the estmator that mnmzes û. There end up beng some nce propertes that result from usng the square of û rather than the absolute value of û. We wll explore these n the future. But for now, f you need a justfcaton an dea of what usng û nstead of û gets us realze that mnmzng the square of ŷ y penalzes large errors more. That s, when our predctons ŷ are close to y, the squared dfference û s small. When our predctons ŷ are far from y, the squared dfference s much larger. Sad another way, the measure û weghts predcton errors more the bgger they are. So choosng our estmates of β0 and β to mnmze û should result n estmates that mnmze really bg predcton errors. ot a bad deal. ow for the dervaton of β 0 and β. We want to choose β 0 and β to mnmze the sum of the squared dfferences between ŷ and y û = (y β 0 β x ). (4) To do ths, we need to employ some calculus (don t worry, t s not hard-core calculus, just the regular (soft-core?) varety). We take the dervatve of (4) wth respect to β 0 and set the expresson equal to zero. We want to fnd the β 0 where changng β 0 a lttle bt doesn t change the value of û (f the dervatve of û were negatve nstead of zero, ths would mean that makng β 0 a lttle bgger would decrease û, tellng us that our choce of β 0 was too small; lkewse, f the dervatve of û were postve, ths would mean that makng β 0 a lttle smaller would decrease û, tellng us that our choce of β 0 was too bg). We then take the dervatve of (4) wth respect to β and set the expresson equal to zero (for the same reason gven above for β 0 ). d d β û = d 0 d β (y β 0 β x ) 0 = d d β (y β 0 y + β 0 β x y + β 0 β x + β x ) 0 = ( y + β 0 + β x ) Because everythng n the expresson s multpled by, we can dvde the whole expresson by and elmnate that term. We then set the expresson equal to zero and solve 9

10 for β 0. ( y + β 0 + β x ) = 0 = ( y + β 0 + β x ) = y + β 0 + β x = y + β 0 + β x β 0 = y β β 0 = ȳ β x Look famlar? Ths s exactly the expresson we got for β 0 when we used the method of moments. We now only need to take the dervatve of (4) wth respect to β, set t equal to zero, and solve for β. I wll skp the explanaton of some of the steps that are dentcal to the case above for β 0. d d β û = 0 = d d β (y β 0 y + β 0 β x y + β 0 β x + β x ) = = = ( x y + β 0 x + β x ) ( x y + β 0 x + β x ) x y + β 0 x + β x x Substtute n β 0 = ȳ β x. 0 = = = x y + β 0 x + β x x y + (ȳ β x) x + β x x y + ȳ x β x x + β x (5) otce what we have here. The last lne of (5) s exactly the frst lne of (). When we use the logc of least-squares, we quckly end up wth the same estmator that we 0

11 obtaned by usng the logc of the method of moments. The rest of the math needed to derve the expressons for β 0 and β s lterally exactly the same as the math used above n the method-of-moments secton.