Chapter 7: Estimating the Variance of an Estimate s Probability Distribution

Transcription

1 Chaper 7: Esimaing he Variance of an Esimae s Probabiliy Disribuion Chaper 7 Ouline Review o Clin s Assignmen o General Properies of he Ordinary Leas Squares (OLS) Esimaion Procedure o Imporance of he Coefficien Esimae s Probabiliy Disribuion Mean (Cener) of he Coefficien Esimae s Probabiliy Disribuion Variance (Spread) of he Coefficien Esimae s Probabiliy Disribuion Sraegy o Esimae he Variance of he Coefficien Esimae s Probabiliy Disribuion Sep : Esimae he Variance of he Error erm s Probabiliy Disribuion o Firs Aemp: Variance of he Error erm s Numerical Values o Second Aemp: Variance of he Residual s Numerical Values o hird Aemp: Adjused Variance of he Residual s Numerical Values Sep : Use he Esimaed Variance of he Error erm s Probabiliy Disribuion o Esimae he Variance of he Coefficien Esimae s Probabiliy Disribuion ying Up a Loose End: Degrees of Freedom o Reviewing our Second and hird Aemps o Esimae he Variance of he Error erm s Probabiliy Disribuion o How Do We Calculae an Average? Summary: he Ordinary Leas Squares (OLS) Esimaion Procedure o hree Imporan Pars Esimaing he Value of he Regression Parameers Esimaing he Variance of he Error erm s Probabiliy Disribuion Esimaing he Variance of he Coefficien Esimae s Probabiliy Disribuion o Regression Resuls

2 Chaper 7 Prep Quesions. Consider an esimae s probabiliy disribuion: a. Why is he mean of he probabiliy disribuion imporan? b. Why is he variance of he probabiliy disribuion imporan?. Consider Professor Lord s firs quiz. a. Suppose ha we know he acual value of he consan and coefficien. More specifically, suppose ha he acual value of he consan is 50 and he acual value of he coefficien is. Fill in he blanks below o calculae each suden s error erm and he error erm squared. hen, compue he sum of he squared error erms. β Cons = 50 β = e = y (β Cons + β ) Suden y 50 + e s Quiz e s Quiz = = = Sum = b. In realiy, we do no know he acual value of he consan and coefficien. We used he ordinary leas squares (OLS) esimaion procedure o esimae heir values. he esimaed consan was 63 and he esimaed value of he coefficien was 6/5. Fill in he blanks below o calculae each suden s residual and he residual squared. hen, compue he sum of he squared residuals. b Cons = 63 Suden y b = 6 5 =. Esy = Res = y (b Cons + b ) 5 Res s Quiz Res s Quiz = = = 5 Sum = c. Compare he sum of squared errors wih he sum of squared residuals. d. In general, when applying he ordinary leas squares (OLS) esimaion procedure could he sum of squared residuals ever eceed he sum of squared errors? Eplain.

3 3 3. Suppose ha Suden had missed Professor Lord s quiz. Suden y 5 66 = minues sudied y = quiz score a. Plo a scaer diagram of he daa. b. Wha is he equaion for he bes fiing line? c. Wha are he residuals for each observaion? d. Suppose ha he quiz scores were differen. For eample, suppose ha Suden received a 70 insead of 66. ) Wha is he equaion for he bes fiing line now? ) Wha are he residuals for each observaion? e. Again, suppose ha he quiz scores were differen. For eample, suppose ha Suden received an 86 insead of 66 of 70. ) Wha is he equaion for he bes fiing line now? ) Wha are he residuals for each observaion? f. In general, when here are only wo observaions wha will he residuals for he bes fiing line equal? Eplain. Review Clin s Assignmen: Assess he Effec of Sudying on Quiz Scores Clin s assignmen is o assess he effec of sudying on quiz scores: Projec: Use daa from Professor Lord s firs quiz o assess he effec of sudying on quiz scores. General Properies of he Ordinary Leas Squares (OLS) Esimaion Procedure An esimae s probabiliy disribuion describes he general properies of he esimaion procedure. In he las chaper we showed ha when he sandard ordinary leas squares (OLS) premises are me, he mean of he coefficien esimae s probabiliy disribuion equals he acual value, β, and he variance equals he variance of he error erm s probabiliy disribuion divided by he sum Var[ e] of he squared deviaions, : ( ) Mean[ b ] = β Var[ b ] = Var[ e] ( )

4 4 Imporance of he Coefficien Esimae s Probabiliy Disribuion Le us now review he imporance of he mean and variance. In general, he mean and variance of he coefficien esimae s probabiliy disribuion play imporan roles: Mean: When he mean of he esimae s probabiliy disribuion equals he acual value he esimaion procedure is unbiased. An unbiased esimaion procedure does no sysemaically underesimae or overesimae he acual value. Variance: When he esimaion procedure is unbiased, he variance of he esimae s probabiliy disribuion deermines he reliabiliy of he esimae. As he variance decreases, he probabiliy disribuion becomes more ighly cropped around he acual value making i more likely for he coefficien esimae o be close o he acual value: Mean of Esimae s Probabiliy Variance of Esimae s Disribuion Equals Acual Value Probabiliy Disribuion Esimaion Procedure Is Unbiased Deermines he Reliabiliy of he Esimae We can apply hese general conceps o he ordinary leas squares (OLS) esimaion procedure. he mean of he coefficien esimae s probabiliy disribuion, Mean[b ], equals he acual value of he coefficien, β ; consequenly, he ordinary leas squares (OLS) esimaion procedure is unbiased. he variance of he coefficien esimae s probabiliy disribuion is now imporan; he variance deermines he reliabiliy of he esimae. Wha does he variance equal? We derived he equaion for he variance: Var[ e] Var[ b ] =. ( ) Bu, neiher Clin nor we know he variance of he error erm s probabiliy disribuion, Var[e]. How hen can he variance of he variance of he coefficien esimae s probabiliy disribuion be calculaed? How can of Clin proceed? When Clin was faced wih a similar problem before, wha did he do? Clin used he economerician s philosophy: Economerician s Philosophy: If you lack he informaion o deermine he value direcly, esimae he value o he bes of your abiliy using he informaion you do have. As Variance Decreases Reliabiliy Increases

5 5 Wha informaion does Clin have? Clin has he daa from Professor Lord s firs quiz: Suden Minues Sudied () Quiz Score (y) able 7.: Firs Quiz Resuls How can Clin use his informaion o esimae he variance of he coefficien esimae s probabiliy disribuion? Sraegy o Esimae he Variance of he Coefficien Esimae s Probabiliy Disribuion Clin needs a procedure o esimae he variance of he coefficien esimae s probabiliy disribuion. Ideally, his procedure should be unbiased. ha is, i should no sysemaically underesimae or overesimae he acual variance. His approach will be based on he relaionship beween he variance of he coefficien esimae s probabiliy disribuion and he variance of he error erm s probabiliy disribuion ha we derived in Chaper 6: Var[ e] Var[ b ] =. ( ) Clin s sraegy is o replace he acual variances in his equaion wih esimaed variances: EsVar[ e] EsVar[ b ] = ( ) where EsVar[ b ] = Esimaed Variance of he Coefficien Esimae's Probabiliy Disribuion EsVar[ e] = Esimaed Variance of he Error erm's Probabiliy Disribuion

6 6 Clin adops a wo sep sraegy. Sep : Clin esimaes he variance of he error erm s probabiliy disribuion. Sep : Clin uses he esimae for he variance of he error erm s probabiliy disribuion o esimae he variance for he coefficien esimae s probabiliy disribuion. Sep : Esimae he variance of he error erm s probabiliy disribuion from he available informaion daa from he firs quiz EsVar[e] é EsVar[ b ] = Sep : Apply he relaionship beween he variances of coefficien esimae s and error erm s probabiliy disribuions Var[ e] Var[ b ] = ( ) EsVar[ e] ( ) Sep : Esimaing he Variance of he Error erm s Probabiliy Disribuion he daa from Professor Lord s firs quiz is he only available informaion ha he can use o esimae he variance of he error erm s probabiliy disribuion, Var[e]. We shall now describe hree aemps o esimae he variance using he resuls of Professor Lord s firs quiz by calculaing he:. Variance of he error erm s numerical values from he firs quiz.. Variance of he residual s numerical values from he firs quiz 3. Adjused variance of he residual s numerical values from he firs quiz. In each case, we will us simulaions o assess hese aemps by eploiing he relaive frequency inerpreaion of probabiliy: Relaive Frequency Inerpreaion of Probabiliy: Afer many, many repeiions of he eperimen, he disribuion of he numerical values from he eperimens mirrors he random variable s probabiliy disribuion; he wo disribuions are idenical: Disribuion of he Numerical Values Probabiliy Disribuion Afer many, many repeiions ã

7 7 he firs wo aemps fail. Neverheless, hey provide he moivaion for he hird aemp which succeeds. Even hough he firs wo aemps fail, i is insrucive o eplore hem. Firs Aemp o Esimae he Variance of he Error erm s Probabiliy Disribuion: Variance of he Error erm s Numerical Values from he Firs Quiz In realiy, Clin canno observe he acual parameers, β Cons and β ; bu, for he momen, assume ha we know hem. If we were privy o he acual parameers, we would be able o calculae he acual numerical values of he error erms for each of our hree sudens from he firs quiz: Suden s error erm Suden s error erm Suden 3 s error erm e = y (β Cons + β ) e = y (β Cons + β ) e 3 = y 3 (β Cons + β 3 ) How could we use hese hree numerical values for he error erms from he firs quiz o esimae he variance of he error erm s probabiliy disribuion? Why no calculae he variance of he numerical values of he hree error erms and hen use ha variance o esimae he variance of he error erm s probabiliy disribuion? ha is, EsVar[e] = Var[e, e, and e 3 for s Quiz] Recall ha he variance is he average of he squared deviaions from he mean: EsVar[ e] = Var[ e, e, and e3 for s Quiz] ( e Mean[]) e + ( e Mean[]) e + ( e3 Mean[]) e = 3 Since he error erms represen random influences, he mean of he error erm s probabiliy disribuion mus equal 0. herefore, he deviaions from he mean are jus e, e, and e 3. he variance is he sum of he squared errors divided by 3: EsVar[ e] = Var[ e, e, and e3 for s Quiz] e + e + e3 fors Quiz SSE fors Quiz = = 3 3

8 8 In our simulaion, we can specify he value of he acual parameers; by defaul, β Cons equals 50 and β equals. Using hese values, calculae he numerical values of he error erms: Firs Quiz: β Cons =50 β = e = y (β Cons + β ) Suden y β Cons + β = 50 + e = y (50 + ) = = 6 6 = = = 7 7 = = = 0 0 = 00 Sum = 85 EsVar[ e] = Var[ e, e, and e3 for s Quiz] e + e + e3 fors Quiz SSE fors Quiz 85 = = = = Can we epec variance of he numerical values, 6 /3, o equal he acual variance of he coefficien esimae s probabiliy disribuion? Absoluely no, random influences are presen. In fac, we can be all bu cerain ha he acual variance of he coefficien esimae s probabiliy disribuion will no equal 6 /3. Wha hen can we hope for? We can hope ha his procedure is unbiased. We can hope ha he procedure does no sysemaically overesimae or underesimae he acual value. Bu in fac, is he esimaion procedure unbiased? e

9 9 Economerics Lab 7.: Is he Firs Aemp Esimaion Procedure Unbiased? We address his quesion by using he Esimaing Variances simulaion in our Economerics Lab: Figure 7.: Variance of he Error erm s Probabiliy Disribuion Simulaion [Link o MI-Lab 7. goes here.] Some new boes now appear: In he Use line, Err is seleced indicaing ha he simulaion will calculae he numerical value of each suden s error erm and hen square he error erms o esimae he variance of he error erm s probabiliy disribuion. In he Divide by line, is seleced. equals he sample size, 3 in his case. he simulaion will divide he sum of he squared errors by 3 o esimae he variance of he error erm s probabiliy disribuion. By selecing Err in he Use line and in he Divide by line, he simulaion mimics he procedure ha we jus described o esimae he variance of he error erm s probabiliy disribuion. Also noe ha he acual variance of he error erm s probabiliy disribuion equals 500 by defaul.

10 0 Be cerain ha he Pause checkbo is checked and click Sar. he simulaion repors he sum of squared errors (SSE) and he esimae for variance of he error erm s probabiliy disribuion (Error Var Es) based on he daa for he firs repeiion: EsVar[ e] = Var[ e, e, and e3 for s Repeiion] Sum of Squared Errors fors Repeiion = Convince yourself ha he simulaion is calculaing EsVar[e] correcly by applying he procedure we jus oulined. hen, click Coninue o simulae a second quiz. he simulaion now repors on he esimae for variance of he error erm s probabiliy disribuion (Error Var Es) based on he daa for he second repeiion: EsVar[ e] = Var[ e, e, and e3 for nd Repeiion] Sum of Squared Errors for nd Repeiion = Again, convince yourself ha he simulaion is calculaing EsVar[e] by applying he procedure we oulined. Also, he simulaion calculaes he mean (average) of he wo variance esimaes; he mean of he variance esimaes is repored in he Mean line direcly below Error Var Es. Convince yourself ha he simulaion is calculaing he mean of he variance esimaes correcly. Click Coninue a few more imes. Noe ha for some repeiions he esimaed variance is less han he acual variance and someimes he esimae is greaer han he acual. Does his esimaion procedure for he variance sysemaically underesimae or overesimae he acual variance or is he esimaion procedure unbiased? We can apply he relaive frequency inerpreaion of probabiliy o address his quesion by comparing he mean (average) of he variance esimaes wih he acual variance afer many, many repeiions. If he esimaion procedure is unbiased, he mean of he variance esimaes will equal he acual variance of he error erm s probabiliy disribuion, 500 in his case, afer many, many repeiions: Afer many, many repeiions Mean (average) of he esimaes = Acual value Esimaion procedure is unbiased

11 Clear he Pause checkbo and click Coninue; afer many, many repeiions click Sop. he mean of he esimaes for he error erm s variance equals abou 500, he acual variance. Ne, change he acual variance o 00; click Sar and hen afer many, many repeiions click Sop. Again he mean of he esimaes approimaely equals he acual value. Finally, change he acual variance o 50 and repea he procedure: Mean (Average) of he Esimaes for he Variance of he Error erm s Acual Value Simulaion Probabiliy Disribuion Var[e] Repeiions SSE Divided by 500 >,000, >,000, >,000, able 7.: Error erm Variance Simulaion Resuls Firs Aemp he simulaion illusraes ha his esimaion procedure does no sysemaically underesimae or overesimae he acual variance; ha is, his esimaion procedure for he variance of he error erm s probabiliy disribuion is unbiased. Bu does his help Clin? Unforunaely, i does no. o calculae he error erms we mus know ha acual value of he consan, β Cons, and he acual value of he coefficien, β. In a simulaion, we can specify he acual values of he parameers, β Cons and β, bu neiher Clin nor we know he acual values for Professor Lord s quiz. Afer all, if Clin knew he acual value of he coefficien, he would no need o go hrough he rouble of esimaing i, would he? he whole problem is ha Clin will never know wha he acual value equals, ha is why he mus esimae i. Consequenly, his esimaion procedure does no help Clin; he lacks he informaion o perform he calculaions. So, wha should he do?

12 Second Aemp o Esimae he Variance of he Error erm s Probabiliy Disribuion: Variance of he Residual s Numerical Values from he Firs Quiz Clin canno calculae he acual values of he error erms because he does no know he acual values of he parameers, β Cons and β. So, he decides o do he ne bes hing. He has already used he daa from he firs quiz o esimae he values of β Cons and β. Firs Quiz Suden Minues Sudied () Quiz Score (y) ( y y)( ) b = = = =. b = y b = 8 5 = 8 8 = Cons ( ) Clin s esimae of β Cons is 63 and β is.. Consequenly, why no use hese esimaed values for he consan and coefficien o esimae he numerical values of error erms for he hree sudens? In oher words, jus use he residuals o esimae he error erms: Res = y (b Cons + b ) Res = y (b Cons + b ) Res 3 = y 3 (b Cons + b 3 ) hen, use he variance of he hree numerical values of he residuals o esimae he variance of he error erm s probabiliy disribuion: EsVar[e] = Var[Res, Res, and Res 3 for s Quiz] Recall ha he variance equals he average of he squared deviaions from he mean: EsVar[ e] = Var[ Res, Res, and Res3 for s Quiz] ( Res Mean[ Res]) + ( Res Mean[ Res]) + ( Res3 Mean[ Res]) = 3

13 3 Clin has all he informaion needed o perform hese calculaions: Firs Quiz: b Cons =63 b = 6 5 =. Suden y Es = b Cons + b = Res = y Esy Res = = 3 3 = = = 6 6 = = = 3 3 = 9 Sum = 0 Sum = 54 Wha is he mean of he residuals? Clearly, he mean is 0: Mean[ Res] = Mean[ Res, Res, and Res3 for s Quiz] Res+ Res + Res = + = Clin can easily calculae he variance of he esimaed errors when using he residuals o do so: EsVar[ e] = Var[ Res, Res, and Res3 for s Quiz] ( Res Mean[ Res]) + ( Res Mean[ Res]) + ( Res3 Mean[ Res]) = 3 Res + Res + Res3 = 3 Sum of Squared Residuals for s Quiz SSR for s Quiz 54 = = = = he good news is ha Clin can indeed perform hese calculaions. He can calculae he residuals and herefore can esimae he variance of he error erm s probabiliy disribuion using his procedure. Unforunaely, here is also some bad news. his esimaion procedure is biased; i sysemaically underesimaes he variance of he error erm s probabiliy disribuion.

14 4 Economerics Lab 7.: Is he Second Aemp Esimaion Procedure Unbiased? o illusrae his, reurn o he Esimaing Variances simulaion: [Link o MI-Lab 7. goes here.] Noe he Res is seleced in he Use line indicaing ha he variance of he residuals raher han he error erms will be used o esimae he variance of he error erm s probabiliy disribuion. As before, he acual variance of he error erm s probabiliy disribuion is specified as 500 by defaul. Be cerain ha he Pause checkbo is cleared; click Sar and afer many, many repeiions click Sop. he mean (average) of he esimaes for he variance equals abou 67 while he acual variance of he error erm is 500. Ne, selec a variance of 00 and hen 50 and repea he process. Convince yourself ha his procedure consisenly underesimaes he variance. Mean (Average) of he Esimaes for he Variance of he Error erm s Acual Value Simulaion Probabiliy Disribuion Var[e] Repeiions SSR Divided by 500 >,000, >,000, >,000,000 7 able 7.3: Error erm Variance Simulaion Resuls Second Aemp he mean of he esimaes is less han he acual values; his esimaion procedure is biased downward. his esimaion procedure sysemaically underesimaes he variance of he error erm s probabiliy disribuion.

15 5 Economerics Lab 7.3: Comparing he Sum of Squared Residuals and he Sum of Squared Errors o undersand why his esimaion procedure is biased downward, we shall reurn o he Esimaing Variances simulaion. [Link o MI-Lab 7.3 goes here.] his ime be cerain ha he Pause checkbo is checked and hen click Sar. Noe ha boh he sum of squared errors and he sum of squared residuals are repored. Which is less in he firs repeiion? Click he Coninue buon o run he second repeiion. Which sum is less in he second repeiion? Coninue o do his unil you recognize he paern ha is emerging. he sum of squared residuals is always less han he sum of squared errors. Why? Recall how b Cons and b were chosen. hey were chosen so as o minimize he sum of squared residuals: SSR = Res + Res + Res = ( y b b ) + ( y b b ) + ( y b b ) 3 Cons Cons 3 Cons 3 and compare i o he sum of squared errors: SSE = e + e + e = ( y β β ) + ( y β β ) + ( y β β ) 3 Cons Cons 3 Cons 3 Res + Res + Res3, would equal he acual sum e + e + e3, only if b Cons equaled β Cons and b equaled β : he sum of squared residuals, of squared errors, Only if b Cons = β Cons and b = β Res + Res + Res3 = e + e + e3 As a consequence of random influences, we can never epec he esimaes o equal he acual values, however. Consequenly, we mus epec he sum of squared residuals o be less han he sum of squared errors: ypically, b Cons β Cons and b β Res + Res + Res < 3 e + e + e 3

16 6 Divide boh sides of he inequaliy by 3 o compare he variance of he Res s and e s: Res + Res + Res3 e + e + e3 < 3 3 Var[Res, Res, and Res 3 ] < Var[e, e, and e 3 ] he variance of he residuals will be less han he variance of he acual errors. Now, recall our firs aemp o esimae he variance of he error erm s probabiliy disribuion. When we used he variance of he acual errors, he procedure was unbiased: Res + Res + Res3 e + e + e3 < 3 3 Var[Res, Res, and Res 3 ] < Var[e, e, and e 3 ] Sysemaically Unbiased esimaion underesimaes variance procedure Using he variance of he residuals leads o bias because i sysemaically underesimaes he variance of he error erm s numerical values. So now, wha can Clin do?

17 7 hird Aemp o Esimae he Variance of he Error erm s Probabiliy Disribuion: Adjused Variance of he Residual s Numerical Values from he Firs Quiz While we shall no provide a mahemaical proof, Clin can correc for his bias by calculaing wha we shall call he adjused variance of he residuals. Insead of dividing he sum of squared residuals by he sample size, Clin can calculae he adjused variance by dividing by wha are called he degrees of freedom: EsVar[ e] = AdjVar[ Res, Res, and Res3 for s Quiz] Sum of Squared Residuals for s Quiz = Degrees of Freedom where Degrees of Freedom = Sample Size Number of Esimaed Parameers he degrees of freedom equal he sample size less he number of esimaed parameers. For he ime being, do no worry abou precisely wha he degrees of freedom represen and why hey solve he problem of bias. We shall moivae he raionale laer in his chaper. We do no wish o be disraced from Clin s effors o esimae he variance of he error erm s probabiliy disribuion a his ime. So le us pospone he raionalizaion for now. For he momen please accep ha fac ha he degrees of freedom equal in his case: Degrees of Freedom = Sample Size Number of Esimaed Parameers = 3 = We subrac because we are esimaing he values of parameers: he consan, β Cons, and he coefficien, β. Clin has he informaion necessary o perform he calculaions for he adjused variance of he residuals. Recall ha we have already calculaed he sum of squared residuals: Firs Quiz: b Cons =63 b = 6 5 =. Suden y Es = b Cons + b = Res = y Esy Res = = 3 3 = = = 6 6 = = = 3 3 = 9 Sum = 0 Sum = 54

18 8 So, we need only divide he sum, 54, by he degrees of freedom o use he adjused variance o esimae he variance of he error erm s probabiliy disribuion: EsVar[ e] = AdjVar[ Res, Res, and Res3 for s Quiz] Sum of Squared Residuals for s Quiz 54 = = = 54 Degrees of Freedom Economerics Lab 7.4: Is he hird Aemp Esimaion Procedure Unbiased? We shall use he Esimaing Variances simulaion o illusrae ha his hird esimaion procedure is unbiased. [Link o MI-Lab 7.4 goes here.] In he Divide by line, selec insead of. Since we are esimaing wo parameers, he simulaion will be dividing by he degrees of freedom insead of he sample size. Iniially, he variance of he error erm s probabiliy disribuion is specified as 500. Be cerain ha he Pause checkbo is cleared; click Sar and hen afer many, many repeiions click Sop. he mean (average) of he variance esimaes equals abou 500, he acual variance. Ne, repea he process by selecing a variance of 00 and hen 50. able 7.4 he resuls. In each case, he mean of he esimaes equals he acual value afer many, many repeiions. his esimaion procedure proves o be unbiased: Mean (Average) of he Esimaes for he Variance of he Error erm s Acual Value Simulaion Probabiliy Disribuion Var[e] Repeiions SSR Divided by 500 >,000, >,000, >,000, able 7.4: Error erm Variance Simulaion Resuls hird Aemp

19 9 Sep : Use he Esimae for he Variance of he Error erm s Probabiliy Disribuion o Esimae he Variance of he Coefficien Esimae s Probabiliy Disribuion A las Clin has found an unbiased esimaion procedure for he variance of he error erm s probabiliy disribuion: EsVar[e] = AdjVar[Res, Res, and Res 3 for s Quiz] = 54 Bu why did he need his esimae in he firs place? He needs i o esimae he variance of he coefficien esimae s probabiliy disribuion in order o assess he reliabiliy of he coefficien esimae. Recall his wo sep sraegy: Sep : Esimae he variance of he error erm s probabiliy disribuion from he available informaion daa from he firs quiz EsVar[e] é EsVar[ b ] = Sep : Apply he relaionship beween he variances of coefficien esimae s and error erm s probabiliy disribuions EsVar[ e] ( ) Var[ b ] = ã Var[ e] ( ) A lile arihmeic allows Clin o esimae he variance of he coefficien esimae s probabiliy disribuion: EsVar[ e] EsVar[ b ] = ( ) + ( ) + ( 3 ) 54 = (5 5) + (5 5) + (5 5) = = = =.7 ( 0) + (0) + (

20 0 Recall ha he sandard deviaion is he square roo of he variance; hence, we can calculae he esimaed sandard deviaion by compuing he square roo of esimaed variance: EsSD[ b] = EsVar[ b ] =.7 =.596 he esimaed sandard deviaion is called he sandard error: SE[ b] = EsSD[ b] = Esimaed sandard deviaion of b s probabiliy disribuion = EsVar[ b ] =.7 =.596 he sandard error equals he square roo of he esimaed variance. Le us summarize Clin s wo sep sraegy. Sep : Clin esimaes he variance of he error erm s probabiliy disribuion. Sep : Clin uses he esimae for he variance of he error erm s probabiliy disribuion o esimae he variance for he coefficien esimae s probabiliy disribuion. Sep : Esimae he variance of he error erm s Sep : Apply he relaionship beween he probabiliy disribuion from he available variances of coefficien esimae s and informaion daa from he firs quiz error erm s probabiliy disribuions EsVar[ e] = AdjVar[ Res's] Var[ e] Var[ b ] = SSR 54 = = = 54 ( ) Degrees of Freedom é ã EsVar[ e] 54 EsVar[ b ] = = =.7 00 ( ) EsSD[ b ] = EsVar[ b ] =.7 =.596

21 We have already used a simulaion o show ha Sep is jusified; ha is, we have shown ha he esimaion procedure for he variance of he error erm s probabiliy disribuion is unbiased. Now, we shall jusify he Sep by showing ha he esimaion procedure for he variance of he coefficien esimae s probabiliy disribuion is also unbiased. o do so, we shall once again eploi he relaive frequency inerpreaion of probabiliy: Disribuion of he Numerical Values Afer many, many repeiions Probabiliy Disribuion An esimaion procedure is unbiased whenever he mean (average) of he esimaed numerical values equals he acual value afer many, many repeiions: Afer many, many repeiions Mean (average) of he esimaes = Acual value Esimaion procedure is unbiased

22 Economerics Lab 7.5: Is Esimaion Procedure for he Variance of he Coefficien Esimae s Probabiliy Disribuion Unbiased? We will use our Esimaing Variance simulaion in he Economerics Lab o show ha his wo sep esimaion procedure for he variance of he coefficien esimae s probabiliy disribuion is unbiased. [Link o MI-Lab 7.5 goes here.] Figure 7.: Variance of he Coefficien Esimae s Probabiliy Disribuion Simulaion By defaul, he acual variance of he error erm s probabiliy disribuion is 500 and he sample size is 3. We can now calculae he variance of he coefficien esimae s probabiliy disribuion: Var[ e] Var[ b ] = ( )

23 3 From before recall ha he sum of squared deviaions equals 00: ( ) = 00 and ha when he variance of he error erm s probabiliy disribuion is specified as 500 and he sum of he squared deviaions equals 00, he variance of he coefficien esimaes probabiliy disribuion equals.50: Var[ e] 500 Var[ b ] = = = ( ) Le us begin by confirming ha he simulaion is performing he calculaions correcly. Be cerain ha he Pause buon is checked and hen click Sar. he sum of squared residuals and he sum of squared deviaions are repored for he firs repeiion. Use his informaion along wih a pocke calculaor o compue he esimae for he variance of he coefficien esimae s probabiliy disribuion, EsVar[b ]: EsVar[ e] SSR EsVar[ b ] = where EsVar[ e ] = Degrees of Freedom ( ) Compare your calculaion wih he simulaion s esimae. You will discover ha hey are idenical. Ne, click Coninue and perform he same calculaion for he second repeiion. Again, you will discover ha he simulaion has calculaed he esimae for he variance of he coefficien esimae s probabiliy disribuion correcly. Also, confirm ha he simulaion is compuing he mean of he variance esimaes correcly by aking he average of he coefficien variance esimaes from he firs wo repeiions. Click Coninue a few more imes. he variance esimae should be less han he acual value,.50, in some of he repeiions and greaer han he acual value in ohers. Now, he criical quesion: Criical Quesion: Afer many, many repeiions, will he mean (average) of he variance esimaes equal he acual variance of he coefficien esimae s probabiliy disribuion? If he answer is yes, he variance esimaion procedure is unbiased; he procedure is no sysemaically overesimaing or underesimaing he acual variance. On he oher hand, if he answer is no, he variance esimaion procedure is biased. o answer his quesion, clear he Pause checkbo and click Coninue. Afer many, many repeiions, click Sop. Wha do you observe? Afer many, many repeiions he average of he coefficien s variance esimaes indeed equals abou.50.

24 4 Repea his process afer you change he error erm variance o 00 and hen o 50. As able 7.5 repors, he answer o he criical quesion is yes in all cases. he esimaion procedure for he variance of he coefficien esimae s probabiliy disribuion is unbiased. Is he esimaion procedure for he variance of he coefficien esimae s probabiliy disribuion unbiased? ã é Variance of he Mean (Average) of he Esimaes Acual Coefficien Esimae s for he Variance of he Coefficien Var[e ] Probabiliy Disribuion Esimae s Probabiliy Disribuion able 7.5: Esimaing Variance Simulaion Resuls ying Up a Loose End: Degrees of Freedom Reviewing our Second and hird Aemps o Esimae he Variance of he Error erm s Probabiliy Disribuion Earlier in his chaper, we posponed our eplanaion of degrees of freedom because i would have inerruped he flow of our discussion. We shall now reurn o he opic by reviewing Clin s effors o esimae he variance of he error erm s probabiliy disribuion. Since Clin can never observe he acual consan, β Cons, and he acual coefficien, β, he canno calculae he acual values of he error erms. He can, however, use his esimaes for he consan, b Cons, and coefficien, b, o esimae he errors by calculaing he residuals: Error erms Residuals e = y (β Cons + β ) Res = y (b Cons + b ) We can hink of he residuals as he esimaed error erms. Now le us briefly review our second and hird aemps o esimae he variance of he error erm s probabiliy disribuion.

25 5 In our second aemp we used he variance of he residuals ( esimaed errors ) o esimae he variance of he error erm s probabiliy disribuion. he variance is he average of he squared deviaions from he mean: Var[ Res, Res, and Res3 for s Quiz] ( Res Mean[ Res]) + ( Res Mean[ Res]) + ( Res3 Mean[ Res]) = Sample Size Since he residuals are he esimaed errors, i seemed naural o divide he sum of squared residuals by he sample size, 3 in Clin s case. Furhermore, since he Mean[Res] = 0: Res + Res + Res3 SSR SSR EsVar[ Res, Res, and Res3 for s Quiz] = = = Sample Size Sample Size 3 Bu we showed ha his procedure was biased; he Esimaing Variance simulaion revealed ha i sysemaically underesimaed he error erm s variance. We hen modified he procedure; insead of dividing by he sample size, we divided by he degrees of freedom, he sample size less he number of esimaed parameers: AdjVar[ Res, Res, and Res3 for s Quiz] Res + Res + Res3 SSR SSR = = = Degrees of Freedom Degrees of Freedom Degrees of Freedom = Sample Size Number of Esimaed Parameers = 3 = he Esimaing Variances simulaion illusraed ha his modified procedure was unbiased.

26 6 How Do We Calculae an Average? Why does dividing by raher han 3 work? ha is, why do we subrac from he sample size when calculaing he average of he squared residuals ( esimaed errors )? o provide some inuiion, we shall briefly revisi Amhers precipiaion in he wenieh cenury: Year Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec able 7.6: Monhly Precipiaion in Amhers, MA during he wenieh Cenury Calculaing he mean for June: Mean (Average) for June = = = Each of he 00 Junes in he wenieh cenury provides one piece of informaion ha we use o calculae he average. o calculae an average we divide he sum by he number of pieces of informaion. Key Principle: o calculae a mean (an average) we divide he sum by he number of pieces of informaion: Sum Mean (Average)= Number of Pieces of Informaion Hence, o calculae he average of he squared deviaions, he variance, we mus divide by he number of pieces of informaion. Now, le us reurn o our effors o esimae he variance of he error erm s probabiliy disribuion: Claim: he degrees of freedom equal he number of pieces of informaion ha are available o esimae he variance of he error erm s probabiliy disribuion. o jusify his claim, suppose ha he sample size were. Plo he scaer diagram: Wih only wo observaions, we only have wo poins. he bes fiing line passes direcly hrough each of he wo poins on he scaer diagram. Consequenly, he wo residuals, he wo esimaed errors, for each observaion mus always equal 0 when he sample size is regardless of wha he acual variance of he error erm s probabiliy disribuion equals: Res = 0 and Res = 0 regardless of wha he variance acually equals.

27 7 y Figure 7.3: Degrees of Freedom wo Observaions he firs wo residuals, he firs wo esimaed errors, provide no informaion abou he acual variance of he error erm s probabiliy disribuion because he line fis he daa perfecly boh residuals equal 0. Only wih he inroducion of a hird observaion do we ge some sense of he error erm s variance. o summarize: he firs wo observaions provide no informaion abou he error erm; saed differenly, he firs wo observaions provide zero informaion abou he error erm s variance. he hird observaion provides he firs piece of informaion abou he error erm s variance. y y Res 3 Res Res Suggess large error erm variance Suggess small error erm variance Figure 7.4: Degrees of Freedom hree Observaions

28 8 his eplains why Clin should divide by o calculae he average of he squared deviaions. In general, he degrees of freedom equal he number of pieces of informaion ha we have o esimae he variance of he error erm s probabiliy disribuion: Degrees of Freedom = Sample Size Number of Esimaed Parameers o calculae he average of he sum of squared residuals, we should divide he sum of squared residuals by he degrees of freedom, he number of pieces of informaion. Summary: he Ordinary Leas Squares (OLS) Esimaion hree Imporan Pars hree Imporan Pars he ordinary leas squares (OLS) esimaion procedure acually includes hree procedures; a procedure o esimae he: Regression Parameers. Variance of he Error erm s Probabiliy Disribuion. Variance of he Coefficien Esimae s Probabiliy Disribuion. All hree esimaion procedures are unbiased. (Recall ha we are assuming ha he sandard ordinary leas squares (OLS) premises are me. We shall address he imporance of hese premises in Par 4 of he ebook.) We shall now review he calculaions and hen show ha saisical sofware performs all hese calculaions for us. Esimaing he Value of he Regression Parameers We calculaed he ordinary leas squares (OLS) esimaes, b and b Cons, by using he appropriae equaions; hese esimaes minimize he sum of squared residuals: ( y y)( ) b = = = =. b = y b = 8 5 = 8 8 = Cons ( )

29 9 Esimaing he Variance of he Error erm s Probabiliy Disribuion Once we calculae he esimaes, i is easy o calculae he sum of squared residuals, SSR: Cons SSR = Res = ( y Esy ) == ( y b b ) = 54 We esimaed he variance of he error erm s probabiliy disribuion, EsVar[e], by dividing he sum of squared residuals by he degrees of freedom: SSR 54 EsVar[ e ] = AdjVar[ Res+ Res + Res3] = = = 54 Degrees of Freedom he square roo of his esimaed variance is ypically called he sandard error of he regression: SE of Regression = EsVar[ e ] = 54 = Noe ha he erm sandard error always refers o he square roo of an esimaed variance. Esimaing he Variance of he Coefficien Esimae s Probabiliy Disribuion he esimaed value of he error erm s probabiliy disribuion allowed us o esimae he variance of he coefficien esimae s probabiliy disribuion, EsVar[b ]: EsVar[ e] 54 EsVar[ b ] = = =.7 00 ( ) he square roo of his esimaed variance of he coefficien esimae s probabiliy disribuion is called he sandard error of he coefficien esimae, SE[b ]: SE[ b] = EsVar[ b ] =.7 =.596 We illusraed ha hese esimaion procedure have nice properies. When he sandard ordinary leas squares (OLS) premises are saisfied: Each of hese procedures is unbiased. he procedure o esimae he value of he parameers is he bes linear unbiased esimaion procedure (BLUE).

30 30 Regression Resuls In realiy, we did no have o make all hese laborious calculaions. Saisical sofware performs hese calculaions for us hereby saving us he ask of performing he arihmeic: Professor Lord s Firs Quiz Daa: Cross secion daa of minues sudied and quiz scores in he firs quiz for he 3 sudens enrolled in Professor Lord s class. Minues sudied by suden Quiz score received by suden y [Link o MI-Quiz.wf goes here.] Ordinary Leas Squares (OLS) Dependen Variable: y Eplanaory Variable(s): Esimae SE -Saisic Prob Cons Number of Observaions 3 Sum Squared Residuals SE of Regression Esimaed Equaion: Esy = able 7.7: Quiz Scores Regression Resuls We previously noed he regression resuls repor he parameer esimaes and he sum of squared residuals. While saisical sofware ypically does no repor he esimaed variance of he error erm s probabiliy disribuion, i does repor he sandard error of he regression, SE of Regression, which is jus he square roo of he esimaed variance of he error erm s probabiliy disribuion. We can easily calculae he esimaed variance of he error erm s probabiliy disribuion from he regression resuls by squaring he sandard error of he regression: EsVar[e] = = 54 Similarly, while he saisical sofware does no repor he esimaed variance of he coefficien esimae s probabiliy disribuion, i does repor is sandard error. We can easily calculae he esimaed variance of he coefficien esimaes probabiliy disribuion from he regression resuls by squaring he sandard error of he coefficien esimae: EsVar[b ] =.5965 =.7

31 In fac, using a lile algebra, we can show ha he mean of he residuals mus always equal 0 when we use he ordinary leas squares (OLS) esimaion procedure. 3