Maximum Likelihood Estimation L. Magee January, 2010

Transcription

1 Maxmum Lkelhood Estmaton L. Magee January, Notaton Let (f y (y x, θ be the probablty densty functon (pdf of y gven a vector of exogenous explanatory varables x (treated as non-random here and an unknown parameter vector θ. If y s a dscrete random varable, then these results stll apply, wth some mnor adjustments where noted. Assume that the y s are ndependently dstrbuted across. The log lkelhood functon for observaton s defned as l (θ = ln(f y (y x, θ Expectatons and varances wll nvolve ntegrals over the condtonal (on x dstrbuton of y, as n El = ln(f y (y x, θf y (y x, θdy If y s dscrete, then let (f y (y x, θ represent the probablty of y gven x and θ, and the correspondng expectaton would then be El = ln(f y (y x, θf y (y x, θ Dependng on the context, y and l can refer to random varables or ther realzatons, and f y (y x, θ can refer to a pdf or a probablty. 2 ML and M-Estmaton The ML estmates satsfy the frst-order condtons l (ˆθ = 0 Ths ˆθ s an M-estmator because t can be shown that E l (θ/ = 0 at the true value of θ f the densty assumpton s correct. Here s a proof. 1

2 2.1 Proof of E l (θ/ = 0 Droppng the subscrpt to reduce notaton, then E l(θ ( ln(fy (y x, θ = E ( = E 1 f y (y x, θ f y (y x, θ Snce the expectaton of some functon of y, say h(y, s h(yf y (y x, θdy, then E l(θ ( l(θ = f y (y x, θdy ( 1 f y (y x, θ = f y (y x, θdy f y (y x, θ ( fy (y x, θ = dy = ( f y (y x, θdy = (1 = 0 These steps nvolve a cancellaton, a swtchng of the order of the dervatve and ntegraton, and usng the fact that the area under the densty curve equals one at every value of θ. As requred n M-estmaton, ths expectaton result holds only at the true value of θ n general. Otherwse, the cancellaton gong from lne two to lne three would not be possble, snce the two functons to be cancelled would be evaluated at dfferent values of θ and therefore would not be equal. The proof s smlar for dscrete y. Summng over below s analogous to ntegratng over the support of y n the above proof: E l (θ = = ( l (θ f y (y x, θ ( 1 f y (y x, θ f y (y x, θ f y (y x, θ ( fy (y x, θ = = ( f y(y x, θ = (1 = 0 snce f y(y x, θ = 1 for any x. 2

3 3 The Informaton Matrx: Defnton Consder the k 1 vector of partal dervatves, l (θ/, and the k k matrx of second dervatves 2 l (θ/. Snce we are assumng the observatons are ndependent, then l(θ = l (θ so that l(θ = l (θ and 2 l(θ = 2 l (θ The nformaton matrx s defned as I(θ = E 2 l(θ (A note about evaluatng I(θ at θ = ˆθ: It s evaluated by frst takng the expectaton of 2 l (θ/ wth respect to the assumed densty of y. Then evaluate the resultng expresson, whch s a functon of the x s and θ, at θ = ˆθ. That s, the computaton of E 2 l (ˆθ/ does not requre the much more dffcult procedure of frst substtutng out a closed form expresson of ˆθ to obtan a complcated functon of θ, x, and y, and then takng the expectaton over y. 4 The Informaton Matrx Equalty and Estmators of the Varance of ML The E l (θ/ = 0 result allows us to use the general results for M-estmators, wth and ψ (θ = l (θ ψ = 2 l (θ whch s a symmetrc k k matrx. Asymptotcally, ˆθ s normal wth mean θ and a varance estmator obtaned by substtutng the above quanttes nto the general formula (1 for the estmatng the 3

4 varance of an M-estmator wth ndependent data. ˆV (ˆθ = ( 1 ( (( ψ (ˆθ ψ (ˆθψ (ˆθ 1 ψ (ˆθ (1 In ths ML specal case, the varance estmator becomes ˆV (ˆθ = ( 1 ( 2 l (ˆθ l (ˆθ ( l (ˆθ A number of smpler varance estmators often are used, ncludng 1 2 l (ˆθ (2 nformaton matrx estmator: negatve Hessan varance estmator: outer product of gradent (OPG: I(ˆθ 1 ( ( 1 2 l (ˆθ l (ˆθ 1 l (ˆθ These estmators follow from (2 usng the nformaton matrx equalty ( l (θ E l (θ = E ( 2 l (θ when f y s the correct densty functon of y (3 whch mples E ( l (θ l (θ = E ( 2 l (θ = E 2 l(θ = I(θ The proof of (3 s omtted here, but follows smlar steps to those used n subsecton 2.1. We then have three O p (n matrces that are asymptotcally equvalent when the model s correctly specfed, I(ˆθ = 2 l (ˆθ + o p (n = l (ˆθ l (ˆθ + o p (n The frst one, the nformaton matrx, requres that we can derve a closed-form expresson for the 4

5 expected value of the second dervatve matrx. The second one, the negatve Hessan, requres the matrx of second dervatves of the log lkelhood functon. The thrd one, the OPG, requres only the frst dervatves of the log lkelhood. By substtutng these matrces for one another n (2, one can obtan the varance estmators lsted below (2. The varance estmator (2 was developed more recently than the others, despte the order n whch they are presented here. (2 s robust to msspecfcaton, whereas the others are not. To see ths, note that (2 was obtaned wthout requrng that the assumed lkelhood functon was correct, whereas ts asymptotc equvalence to the other varance estmators reles on the nformaton matrx equalty, whch holds n general only when the model s correctly specfed. Despte the robustness of (2, t s not used as often as the other ˆV s for ML estmators. Although t s a bt more computatonally demandng, a more mportant reason s that for many knds of msspecfcaton, ths robustness s not a very relevant property. The msspecfcaton to whch (2 s robust may also make the ML estmator tself nconsstent, whch of course reduces the theoretcal appeal of the entre estmaton procedure. Freedman (2006 makes ths pont, concludng, It remans unclear why appled workers should care about the varance of an estmator for the wrong parameter. The leadng specal case of the use of (2 s the HCCME estmator for OLS. The OLS estmator s the ML estmator for a lnear regresson model wth normally dstrbuted homoskedastc errors. If the source of msspecfcaton we wsh to be protected aganst s heteroskedastcty, then b stll s consstent. In that sense, the msspecfcaton handled by ths specal OLS case s relatvely bengn. The fact that the nformaton matrx equalty (3 fals n general when the model s msspecfed underles the class of model specfcaton tests known as Informaton Matrx (IM tests proposed by Whte (1982 and dscussed n Davdson and MacKnnon (1993, secton These tests check for the valdty of the restrctons mpled by (3or of a subset of these. 5 The Cramér-Rao Lower Bound If the model s correctly specfed along wth some regularty condtons, then no unbased estmator of θ has a varance smaller than I(θ 1. That s, Var(ˆθ I(θ 1 s postve semdefnte for any unbased estmator ˆθ. Ths has mportant mplcatons for ML estmaton, snce our asymptotc results show that ML s asymptotcally unbased and has an asymptotc varance equal to I(θ 1. Therefore ML s asymptotcally effcent, meanng that ts varance approaches the Cramér-Rao lower bound. A bref dscusson of ths result and further references are gven n Greene (2003, p

6 6 LR, W, and LM tests Three commonly used hypothess tests n maxmum lkelhood estmaton are the LR (lkelhood rato, W (Wald and LM (Lagrange Multpler tests. Let ˆθ U be the unrestrcted ML estmator, and let ˆθ R be the value of θ that maxmzes the lkelhood subject to some restrcton(s on θ that we wsh to test, H 0 : Rθ r = 0, where R and r are known constants. The test statstcs are LR = 2(l(ˆθ U l(ˆθ R W = ( (Rˆθ U r R ˆV (ˆθR 1 (RˆθU r LM = ( ( l (ˆθ R 1 ( 2 l (ˆθ R l (ˆθ R The LR statstc requres computng both the restrcted and unrestrcted ML estmates, W requres only the unrestrcted ML estmates, and LM requres only the restrcted ML estmates. Under H 0, all three statstcs have a χ 2 dstrbuton asymptotcally wth degrees-of-freedom equal to the number of restrctons. Moreover, the dfference between them converges to zero under H 0 as n. 7 References Davdson, R. and J.G. MacKnnon (1993, Estmaton and Inference n Econometrcs, Oxford Unversty Press, Oxford. Freedman, D.A. (2006, On the So-Called Huber Sandwch Estmator and Robust Standard Errors, The Amercan Statstcan, 60(4, Greene, W.H. (2003, Econometrc Analyss, 5 th edton, Prentce Hall, New Jersey. Whte, H. (1982, Maxmum Lkelhood Estmaton of Msspecfed Models, Econometrca, 50,