Outline. Outline. 1 Maximum Likelihood Estimation in a Nutshell. 2 MLE of Independent Data

Transcription

1 Ouline Ouline Ouline Maximum Likelihood Esimaion in a Nushell Maximum Likelihood: An Inroducion 2 of Independen Daa Example: esimaing mean and variance Example: OLS as Chrisian Julliard Deparmen of Economics and FMG London School of Economics 3 for ime Series Ergodic heorem 4 ML Asympoics 5 he Dela Mehod 6 Examples of esimaion he linear sandard regression model of he AR) process of Nonlinear leas squares models of he MA) process Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion in a Nushell of Independen Daa he likelihood funcion ofen simply he likelihood) is a funcion of he parameers, ψ, of a saisical model If he daa are independen, idenically disribued iid) we have L x ψ) = f x,..., x n ψ) where x,..., x n is he sample of daa, f. ψ) is a known probabiliy densiy funcion pdf) parameerized by he unknown vecor of parameers ψ he maximum likelihood esimaor ) is ˆψ = arg maxl x ψ) = arg max log L x ψ) ψ ψ ha is, he maximizes a condiional probabiliy funcion considered as a funcion of is second argumen, wih is firs argumen he daa held fixed. answers he quesion: Wha is he mos likely value of ψ given he sample we have observed? f x,..., x n ψ) = f x ψ) f x 2 ψ)... f x n ψ) in he same way as P A, B) = P A) P B) if and only if A and B are independen. he likelihood can han be wrien as he produc of n probabiliy densiies L x ψ) = n f x i ψ) log L x ψ) = i= n log f x i ψ) i=

2 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Example Example: esimaing mean and variance Recall he Normal Gaussian) disribuion N µ, σ 2) has pdf { f x i µ, σ 2) = exp x i µ) 2 } 2πσ 2 2 he corresponding pdf for a sample of n idd Normal random variables he likelihood is L x µ, σ 2) n { } = exp x i µ) 2 i= 2πσ 2 2 σ 2 = 2πσ 2) { n 2 exp n x } i= i µ) 2 2 σ 2 log L x µ, σ 2) = n 2 log 2πσ 2) ni= x i µ) 2 2 σ 2 where ψ = µ, σ 2 ] are he unknown parameers we wan o esimae. σ 2 Example aking he FOC for a maximum we have x µ, σ 2) ni= x i µ) = µ σ 2 = 0 ˆµ = n x i n x µ, σ 2) σ 2 i= = n + 2 ˆσ 2 = n x i ˆµ) 2 n i= ni= x i µ) 2 σ 4 = 0 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Example Example: OLS as for ime Series Models Consider he linear sandard regression model y i = x i β + ε i N 0, σ 2) ; i =,..., n; E x i ε s] = 0 s, i ) Since y i x i β = ε i N 0, σ 2) i we have L y, x β, σ 2) n { = exp 2πσ 2 i β ) } 2 i= log L y, x β, σ) = n 2 log 2πσ 2) n i= i β ) 2 he sandard approach o we have seen so far is o obain he likelihood funcion by wriing he densiy for each observaion and hen 2 since he observaions are independen, wrie he likelihood as he produc of hese densiies. his sandard approach will no work in ime series since he observaions are generally dependen. Bu: a join densiy can be always facored ino a condiional imes a marginal. Noe ha by definiion ˆβ = arg max log L y, x β, σ) = arg max = arg min n n= i β ) 2 = ˆβOLS n n= i β ) 2

3 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Example: if you have hree observaions f y 3, y 2, y ) = f y 3 y 2, y ) f y 2, y ) = f y 3 y 2, y ) f y 2 y ) f y ). Hence he likelihood for observaions is Ly; ψ) = f y y,..., y ) f y ) = f y I ) f y ) =2 =2 where I denoes all he informaion available a ime. aking logs hen yields log Ly; ψ) = log f y I ) + log f y ). =2 Noe: f y ) can be eiher modeled direcly or y can be assumed o be a consan more on his laer) Ergodic heorem An) Ergodic heorem If a sochasic process y, =, 2,... is ergodic wih mean µ < hen p lim y = µ. = Ergodiciy is a sufficien condiion for sample means o converge o heir expecaions. his definiion exends o vecor valued sochasic processes. Moreover, funcions of vecor valued ergodic processes are ergodic. Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Asympoics for ime Series Models Even if observaions are dependen, for ergodic processes, he ML esimaor of a vecor of parameers ψ is generally consisen. Moreover, he asympoic normaliy resuls derived for he in he iid seing carry over for ergodic processes. ha is, he ML esimaed parameers will be efficien and have an asympoic Gaussian disribuion. ML Asympoics For a vecor of parameers ψ and ergodic daa, we have he sandard asympoic resul ) ) ˆψ ψ 0 ) D N 0, Iψ 0) where Iψ 0 ) is he informaion marix defined as 2 ] log Lψ 0 ) ψ0 ) Iψ 0 ) := E = E ψ 0 ) ] 2) where he las ideniy is he so called informaion marix ideniy.

4 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Obviously, I ψ 0) is in general no observed. So o make he asympoic normaliy resul operaional we need a consisen esimaor of I ψ 0) wo commonly used esimaors are: Asympoic Variance Esimaors he Hessian based esimaor ] 2 log L ˆψ ). ) 2 he empirical informaion marix I ˆψ based more on his shorly). he Dela Mehod Suppose we know ha ˆψ ψ 0 ) D N 0, V ) ) and we are ineresed in making inference abou g ˆψ, where g.) is some differeniable funcion wih coninuous firs derivaive). ) Wha is he disribuion of g ˆψ? hese are boh consisen since ˆψ ψ 0 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Consider he aylor expansion around ψ 0 ) ) g ˆψ g ψ 0 ) G ψ 0 ) ˆψ ψ 0 where G ψ) := gψ). his implies ha ) ) Var g ˆψ g ψ 0 ) )) Var G ψ 0 ) ˆψ ψ 0 )) = G ψ 0 ) Var ˆψ ψ 0 G ψ 0 ) = G ψ 0 ) VG ψ 0 ) We can herefore apply a CL argumen o ge ) ) D ) g ˆψ g ψ 0 ) N 0, G ψ 0 ) VG ψ 0 ). he linear sandard regression model he linear sandard regression model Consider he sandard model ) Recall: a consisen esimaor of he asympoic variance is ] 2 log L ˆψ ). 3) Noe ha y, z β, σ 2) β 2 log L y, z β, σ 2) β σ 2 2 log L y, z β, σ 2) β β = σ 2 = σ 4 = = = x y x β ) = 0 x y x β ) = 0 = xx σ 2 = = xx ˆσ 2 his resul is he so called Dela mehod. we have he usual resul for he variance of he OLS = ) coefficiens x x ˆσ 2.

5 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion of he AR) process of he AR) process Consider he AR) y = φy + ε ε iid N0, σ 2 ), φ <. hen y y is Nφy, σ 2 ), herefore f y I ) = f y y ) = exp 2πσ 2 y φy }{{} And he log likelihood is simply, log Ly; φ, σ 2 ) ) = log 2π 2 =2 ) 2 ε log σ 2 ) 2 of he AR) process Wha do we do abou he iniial condiion? One possibiliy is o condiion on y, i.e. ake i as fixed. In his case he final erm can be dropped and he likelihood becomes he likelihood for he linear regression of y on y for observaions = 2,...,. hus we have, a he maximum, φ = y σ 2 φy ) y = 0 y y ˆφ = ˆφ = ˆφ y 2 OLS y φy ) 2 + log f y ). Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion of he AR) process of he AR) process Alernaively, you can use he uncondiional disribuion for y, Recall: in he AR), he uncondiional mean, Ey ) = 0, and he uncondiional variance, vary ) = σ2 φ 2 ). ) σ so he uncondiional disribuion is N 0, 2. φ 2 ) his assumpion for y is sensible if he process has been going on for a long ime a =. Under his assumpion log f y ) = 2 log 2π 2 log σ2 + 2 log φ2 ) φ2 )y 2 And his gives he log likelihood log Ly; φ, σ 2 ) = 2 log 2π 2 log σ2 y φy ) 2 =2 + 2 log φ2 ) φ2 )y 2. Noe: hese resuls can be exended o: he saionary ARp) model 2 he regression model wih boh process independen regressors and lagged dependen variables.

6 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion of Nonlinear leas squares of Nonlinear leas squares models An imporan sub class of is ha of nonlinear regression models, y = gx ; β) + ε ε iid N0, σ 2 ), =,...,, x process independen. Noe ha ε β) = y gx ; β) { f ε β)) = exp ε β) 2 } 2πσ 2. Hence, log Lβ, σ 2 ) = 2 log 2π 2 log σ2 ε β) 2, = So, maximizing log L wr β is equivalen o minimizing he residual sum of squares wih respec o β. of Nonlinear leas squares Differeniaing he log likelihood, = ε β) β σ 2 β ε β) = σ 2 z ε = 0 where σ 2 = + ) 2 ε β) 2 = 0 z = ε β = gx ; β). β Noe: he firs order condiions wih respec o β are nonlinear and he ML esimaes of β have o be obained by numerical maximizaion. he firs order condiions wih respec o σ 2 yield he usual ML esimaor for σ 2, ˆσ 2 = ε ˆβ) 2. Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion of Nonlinear leas squares of Nonlinear leas squares Recall ha we consruced an esimae of he variance-covariance marix of our esimaes based on he empirical informaion marix Iψ), 2 ] log Lψ) Iψ) = E. In he presen case ψ = β, σ 2 ). So, looking a he componens of Iψ), we have ] ] 2 log L E = E 2 ε β β σ 2 β β ε + E ε ε β β E E ] 2 L σ 2 ) 2 ] 2 log L β σ 2 = E 2 ε σ 2 β β Eε) + E ε ε β β = σ E z z 2 since Eε ) = 0. = ) + 2 Eε 2 2 ) 3 ) = ) ) 3 σ2 = = ) 2 σ 2 ) E z ε 2 = Ez )Eε ) σ 2 ) 2 = 0 since x is indipenden of ε) ]

7 Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion of Nonlinear leas squares Hence, he informaion marix is Iψ) = E z z 0 σ 2 0 Invering, and subsiuing he consisen ML esimaes of β and σ 2 for unknown parameers, and he sample momen z z for E z z, we approximae he disribuion of ˆβ, ˆσ 2 ) by ] ] ) ˆβ β 0 ˆσ 2 σ 2 N ; ˆσ2 z z 0 0 2ˆσ 0 4. of he MA) process of he MA) process Consider he MA) y = ε + ψε ε iid N0, σ 2 ) y ε N ψε, σ 2) Assume we sar from ε 0 = 0, hen we may define ε ψ) by using he recursive equaion Since ε 0 = 0, ε ψ) = y ψε ψ), =, 2,...,. ε ψ) = y ε 2 ψ) = y 2 ψy ha is equivalen o 2) ε 3 ψ) = y 3 ψy 2 + ψ 2 y ε ψ) = y ψy + ψ 2 y ψ) y. Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion Independen Daa ime Series ML Asympoics Dela Mehod Examples of esimaion of he MA) process Since y ε N ψε, σ 2), hen f y I ) = 2πσ 2 ) 2 So he log likelihood is exp y ψε ψ)) 2. log Lψ, σ 2 ) = 2 log 2π 2 log σ2 = 2 log 2π 2 log σ2 y ψε ψ)) 2 = = ε ψ) 2. As before we have = σ 2 z ψ)ε ψ) where z ψ) = ε ψ). of he MA) process Furhermore, using he empirical I ψ) we can show as before ha he variance of ˆψ is given by where var ˆψ) = ˆσ 2 ˆσ 2 = z 2 ˆψ) ε 2 ˆψ). = ) So he ˆψ saisfies z ψ)ε ψ) = 0