Lecture 4: Estimation of ARIMA models

Transcription

1 Lecture 4: Estimation of ARIMA models Florian Pelgrin University of Lausanne, École des HEC Department of mathematics (IMEA-Nice) Sept Dec Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

2 Road map 1 Introduction Overview Framework 2 Sample moments 3 (Nonlinear) Least squares method Least squares estimation Nonlinear least squares estimation Discussion 4 (Generalized) Method of moments Methods of moments and Yule-Walker estimation Generalized method of moments 5 Maximum likelihood estimation Overview Estimation Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

3 Introduction Overview 1. Introduction 1.1. Overview Provide an overview of some estimation methods for linear time series models : Sample moments estimation ; (Nonlinear) Linear least squares method ; Maximum likelihood estimation (Generalized) Method of moments Other methods are available (e.g., Bayesian estimation or Kalman filtering through state-space models)! Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

4 Introduction Overview Broadly speaking, these methods consist in estimating the parameters of interest (autoregressive coefficients, moving average coefficients, and variance of the innovation process) as follows : Optimize Q T (δ) δ s.t. (nonlinear) linear equality and inequality constraints where Q T is the objective function and δ is the vector of parameters of interest. Remark : (Nonlinear) Linear equality and inequality constraints may represent the fundamentalness conditions. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

5 Introduction Overview Examples (Nonlinear) Linear least squares methods aims at minimizing the sum of squared residuals ; Maximum likelihood estimation aims at maximizing the (log-) likelihood function ; Generalized method of moments aims at minimizing the distance between the theoretical moments and zero (using a weighting matrix). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

6 Introduction Overview These methods differ in terms of : Assumptions ; Nature of the model (e.g., MA versus AR) ; Finite and large sample properties ; Etc. High quality software programs (Eviews, SAS, Splus, Stata, etc) are available! However, it is important to know the estimation options (default procedure, optimization algorithm, choice of initial conditions) and to keep in mind that all these estimation techniques do not perform equally and do depend on the nature of the model... Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

7 1.2. Framework Introduction Framework Suppose that (X t ) is correctly specified as an ARIMA(p,d,q) model Φ(L) d X t = Θ(L)ɛ t where ɛ t is a weak white noise (0, σɛ 2 ) and Φ(L) = 1 φ 1 L φ p L p with φ p 0 Θ(L) = 1 + θ 1 L + + θ q L q with θ q 0. Assume that 1 The model order (p,d, and q) is known ; 2 The data has zero mean ; 3 The series is weakly stationary (e.g., after d-difference transformation, etc). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

8 Sample moments 2. Sample moments Let (X 1,, X T ) represent a sample of size T from a covariance-stationary process with E [X t ] = m X for all t Cov [X t, X t h ] = γ X (h) for all t Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

9 Sample moments If nothing is known about the distribution of the time series, (non-parametric) estimation can be provided by : Mean Autocovariance or ˆγ X (h) = 1 T ˆγ X (h) = Autocorrelation T h ˆm X X T = 1 T T X t (X t X T )(X t+ h X T ) T h 1 (X t X T )(X t+ h X T ). T h ˆρ X (h) = ˆγ X (h) ˆγ X (0). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

10 (Nonlinear) Least squares method Least squares estimation 3. (Nonlinear) Least squares method 3.1. Least squares estimation Consider the linear regression model (t = 1,, T ) : y t = x tb 0 + ɛ t where (y t, x t) are i.i.d. (H1), the regressors are stochastic (H2), x t is the t th -row of the X matrix, and the following assumptions hold : 1 Exogeneity (H3) 2 Spherical error terms (H4) 3 Rank condition (H5) E [ɛ t x t ] = 0 for all t; V [ɛ t x t ] = σ 2 0 for all t; Cov [ɛ t, ɛ t x t, x t ] = 0 for t t rank [E (x t x t)] = k. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

11 (Nonlinear) Least squares method Least squares estimation Definition The ordinary least squares estimation of b 0, b ols, defined from the following minimization program ˆb ols = argmin b 0 = argmin b 0 T ɛ 2 t T ( yt x tb ) 2 0 is given by ( T ) 1 ( T ) ˆb ols = x t x t x t y t. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

12 (Nonlinear) Least squares method Least squares estimation Proposition Under H1-H5, the ordinary least squares estimator of b is weakly consistent ˆb ols p T b 0 and is asymptotically normally distributed ) ( l T (ˆbols b 0 N 0 k 1, σ0e 2 [ x t x t ] ) 1. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

13 (Nonlinear) Least squares method Least squares estimation Example : AR(1) estimation Let (X t ) be a covariance-stationary process defined by the fundamental representation ( φ < 1) : X t = φx t 1 + ɛ t where (ɛ t ) is the innovation process of (X t ). The ordinary least squares estimation of φ is defined to be : Moreover ˆφ ols = ( T t=2 x 2 t 1 ) 1 ( T ) x t 1 x t t=2 ( ) l T ˆφ ols φ N ( 0, 1 φ 2). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

14 (Nonlinear) Least squares method Nonlinear least squares estimation 3.2. Nonlinear least squares estimation Definition Consider the nonlinear regression model defined by (t = 1,, T ) : y t = h(x t; b 0 ) + ɛ t where h is a nonlinear function (w.r.t. b 0 ). The nonlinear least squares estimator of b 0, ˆb nls, satisfies the following minimization program : ˆb nls = argmin b 0 = argmin b 0 T ɛ 2 t T ( yt h(x t; b 0 ) ) 2. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

15 (Nonlinear) Least squares method Nonlinear least squares estimation Example : MA(1) estimation Let (X t ) be a covariance-stationary process defined by the fundamental representation ( θ < 1) : X t = ɛ t + θɛ t 1 where (ɛ t ) is the innovation process of (X t ). The ordinary least squares estimation of θ, which is defined by ˆθ ols = argmin θ T (x t θɛ t 1 ) 2, t=2 is not feasible (since ɛ t 1 is not observable!). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

16 (Nonlinear) Least squares method Example : MA(1) estimation (cont d) Conditionally on ɛ 0, x 1 = ɛ 1 + θɛ 0 ɛ 1 = x 1 θɛ 0 Nonlinear least squares estimation x 2 = ɛ 2 + θɛ 1 ɛ 2 = x 2 θɛ 1 ɛ 2 = x 2 θ(x 1 θɛ 0 ). t 2 x t 1 = ɛ t 1 θɛ t 2 ɛ t 1 = ( θ) k x t 1 k + ( θ) t 1 ɛ 0. The (conditional) nonlinear least squares estimation of θ is defined by (assuming that ɛ 0 is negligible) ( ) T t 2 2 ˆθ nls = argmin x t θ ( θ) k x t 1 k θ t=2 k=0 and ) l T (ˆθ nls θ N ( 0, 1 θ 2). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50 k=0

17 (Nonlinear) Least squares method Nonlinear least squares estimation Example : Least squares estimation using backcasting procedure Consider the fundamental representation of a MA(q) : X t = u t Given some initial values δ (0) = u t = ɛ t + θ 1 ɛ t θ q ɛ t q ( ), θ (0) 1,, θ(0) q Compute the unconditional residuals û t û t = X t! Backcast values of ɛ t for t = (q 1),, T using the backward recursion ɛ t = û t θ (0) 1 ɛ t+1 θ q (0) ɛ t+q where the q values for ɛ t beyond the estimation sample are set to zero : ɛ T +1 = = ɛ T +q = 0, and û t = 0 for t < 1. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

18 (Nonlinear) Least squares method Nonlinear least squares estimation Example : Least squares estimation using backcasting procedure (cont d) Estimate the values of ɛ t using a forward recursion ˆɛ t = û t θ (0) 1 ɛ t 1 θ q (0) ɛ t q Minimize the sum of squared residuals using the fitted values ˆɛ t : and find ˆδ (1) = T (X t ˆɛ t θ 1ˆɛ t 1 θ qˆɛ t q ) 2 q+1 ( ). θ (1) 1,, θ(1) q Repeat the backcast step, forward recursion, and minimization procedures until the estimates of the moving average part converge. Remark : The backcast step can be turned off ( ɛ (q 1) = = ɛ 0 = 0) and the forward recursion is only used. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

19 3.3. Discussion (Nonlinear) Least squares method Discussion (Linear and Nonlinear) Least squares estimation are often used in practise. These methods lead to simple algorithms (e.g., backcasting procedure for ARMA models) and can often be used to initialize more sophisticated estimation methods. These methods are opposed to exact methods (see likelihood estimation). Nonlinear estimation requires numerical optimization algorithms : Initial conditions ; Number of iterations and stopping criteria ; Local and global convergence ; Fundamental representation and constraints. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

20 (Generalized) Method of moments Methods of moments and Yule-Walker estimation 4. (Generalized) Method of moments 4.1. Methods of moments and Yule-Walker estimation Definition Suppose there is a set of k conditions S T g (δ) = 0 k 1 where S T R k denotes a vector of theoretical moments, δ R k is a vector of parameters, and g : R k R k defines a (bijective) mapping between S T and δ. The method of moments estimation of δ, ˆδ mm, is defined to be the value of δ such that ) Ŝ T g (ˆδmm = 0 k 1 where Ŝ T is the estimation (empirical counterpart) of S T. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

21 (Generalized) Method of moments Methods of moments and Yule-Walker estimation Example 1 : MA(1) estimation Let (X t ) be a covariance-stationary process defined by the fundamental representation ( θ < 1) : X t = ɛ t + θɛ t 1 where (ɛ t ) is the innovation process of (X t ). One has S T g(δ) = ( γx (0) (1 + θ 2 )σ 2 ɛ γ X (1) θσ 2 ɛ ) = where δ = (θ, σɛ 2 ) and S T = (γ X (0), γ X (1)) = (E [ Xt 2 ], E [Xt X t 1 ]). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

22 (Generalized) Method of moments Methods of moments and Yule-Walker estimation The method of moment estimation of δ, ˆδ mm, solves ( ) ˆγ Ŝ T g(ˆδ mm ) = X (0) (1 + ˆθ mm)ˆσ 2 ɛ,mm 2 ˆγ X (1) ˆθ mmˆσ ɛ,mm 2 = Therefore (using the constraint θ < 1) ˆφ mm = 1 1 4ˆρ 2 X (1) and 2ˆρ X (1) ˆσ 2 ɛ,mm = 2γ X (0) ˆρ 2X (1) Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

23 (Generalized) Method of moments Methods of moments and Yule-Walker estimation Example 2 : Yule-Walker estimation Let (X t ) be a causal AR(p). The Yule-Walker equations write p E X t k X t φ j X t j = 0 for all k = 0,, p j=1 or S T g(δ) = 0 (p+1) 1 { γx (0) φ γ p = σ 2 ɛ γ p Γ p φ = 0 p 1 where γ p = (γ X (1),, γ X (p)) = (E [X t X t 1 ],, E [X t X t p ]), S T = (γ X (0), γ p), φ = (φ 1,, φ p ), δ = (φ, σ 2 ɛ ). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

24 (Generalized) Method of moments Methods of moments and Yule-Walker estimation The method of moment estimation of (φ, σ 2 ɛ ) solves and ˆφ mm = 1 ˆΓ p ˆγ p ˆσ 2 ɛ,mm = ˆγ X (0) ˆφ mmˆγ p. Question : Difference(s) with ordinary least squares estimation of an AR(p)? Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

25 (Generalized) Method of moments Methods of moments and Yule-Walker estimation Discussion It is necessary to use theoretical moments that can be well-estimated from the data (or given the sample size). Method of moment estimation depends on the mapping g 1 and especially the smoothness of the inverse g 1 (need to avoid that the inverse map has a large derivative ). Summarizing the data through the sample moments incurs a loss of information the main exception is when the sample moments are sufficient (in a statistical sense). Instead of reducing a sample of size T to a sample of empirical moments of size k, one may reduce the sample to more empirical moments than there are parameters...the so-called generalized method of moments. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

26 (Generalized) Method of moments 4.2. Generalized method of moments Generalized method of moments Intuition : Consider the linear regression model in an i.i.d. context y t = x tb 0 + ɛ t The exogeneity assumption E [ɛ t X ] implies the k moments conditions (for t = 1,, T ) E [ x t (y t x tb 0 ) ] = 0 k 1. Using the empirical counterpart (analogy principle) 1 T x t (y t x T tb 0 ) = 0 k 1 This is a system of k equations with k unknowns (just-identified) ( T ) 1 ( T ) ˆb ols = x t x t x t y t. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

27 (Generalized) Method of moments Generalized method of moments More generally, suppose that there exists a set of instruments z t R p (with p k) the instruments being (i) uncorrelated with the error terms and (ii) correlated with the explanatory variables such that E [ z t (y t x tb 0 ) ] = 0 p 1 E [h(z t ; δ 0 )] = 0 p 1. Using the empirical counterpart (analogy principle) 1 T T z t (y t x tb 0 ) = 0 p 1 1 T T h(z t ; δ 0 ) = 0 p 1. This system is overidentified for p > k. How can we find an estimation of b 0? Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

28 (Generalized) Method of moments Generalized method of moments The answer is provided by the generalized method of moments, i.e. an estimation method based on estimating equations that imposes the nullity of the expectation of a vector function of the observations and the parameters of interest b 0. The basic idea is to choose a value for δ such that the empirical counterpart is as close as possible to zero. As close as possible to zero means that one minimizes the T distance between 1 T z t (y t x tb 0 ) and 0 p 1 using a weighting matrix (say, W T ). Two issues 1 Choice of instruments ; 2 Choice of the (optimal) weighting matrix. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

29 (Generalized) Method of moments Generalized method of moments Definition Consider the set of moment conditions E [h(z t, δ 0 )] = 0 p 1 where δ 0 R k, p k, and h is a p-dimensional (nonlinear) linear function of the parameters of interest and depends on (i.i.d.) observations. The generalized method of moments estimator of δ 0 associated with W T (symmetric positive definite matrix) is a solution to the problem min δ [ 1 T ] [ ] T 1 T h(z t ; δ) W T h(z t ; δ). T Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

30 (Generalized) Method of moments Generalized method of moments Definition The generalized method of moments estimator of δ 0 solves the first-order conditions [ 1 T T h(z t ; ˆδ ] [ gmm ) 1 W T δ T ] T h(z t ; ˆδ gmm ) = 0 k 1. This estimator is consistent and asymptotically normally distributed. Remark : The first-order conditions correspond to k linear combinations of the moments conditions. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

31 (Generalized) Method of moments Generalized method of moments This estimator depends on W t. There exists a best GMM estimator, i.e. an optimal weighting matrix. The optimal weighting matrix is given by (i.i.d. context) W T = V 1 [h(z t, δ 0 )] E 1 [ h(z t, δ 0 )h(z t, δ 0 ) ] and the sample analog is [ ] 1 1 T h(z t, δ 0 )h(z t, δ 0 ). T It does depend on δ 0! How can we proceed in practise? Two-step GMM or iterated GMM method ; One-step method (Continuously updating estimator, exponential tilting estimator, empirical likelihood estimator, etc). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

32 (Generalized) Method of moments Generalized method of moments Example : AR(1) estimation Let (X t ) be a covariance-stationary process defined by the fundamental representation ( φ < 1) : X t = φx t 1 + ɛ t. Since (ɛ t ) is the innovation process of (X t ), one has the following moment conditions E [x t k ɛ t ] = 0 for k > 0 Past values x t 1,, x t h can be used as instruments Let z t denote the set of instruments z t = (x t 1,, x t h ). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

33 (Generalized) Method of moments Generalized method of moments The moment conditions write where E [h(z t, δ 0 )] = 0 h 1 h(z t, δ 0 ) = z t (x t φ 1 x t 1 ). The empirical counterpart is defined to be x 1 T t 1 T. (x t φ 1 x t 1 ) = 0 h 1. t=h+1 x t h Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

34 (Generalized) Method of moments Generalized method of moments The 2S-GMM estimator of φ is a solution to the problem [ T t=h+1 h(z t ; ˆδ gmm ) δ ] ˆΩ 1 T [ T t=h+1 h(z t ; ˆδ gmm ) ] = 0 h 1. where ˆΩ 1 T is a first-step consistent estimate of the inverse of the variance-covariance matrix of the moments conditions and it accounts for the dependence of the moment conditions. Remark : In a first step, one solves [ T t=h+1 ] [ T ] h(z t ; ˆδ 1,gmm ) I T h(z t ; δ ˆδ 1,gmm ) = 0 h 1. t=h+1 and then compute ˆΩ 1 T (ˆδ 1,gmm ) ˆΩ 1 T. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

35 (Generalized) Method of moments Generalized method of moments Discussion Choice of instruments (optimal instruments?) and of the weighting matrix ; Bias-efficiency trade-off ; Numerical procedures can be cumbersome! Other methods can be written as a GMM estimation. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

36 Maximum likelihood estimation Overview 5. Maximum likelihood estimation 5.1. Overview Consider the linear regression model (t = 1,, T ) : y t = x tb 0 + ɛ t where (y t, x t) are i.i.d., the regressors are stochastic, x t is the t th -row of the X matrix, and It follows that ɛ t x t i.i.d.n (0, σ 2 ɛ ). y t x t i.i.d.n (x tb 0, σ 2 ɛ ). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

37 Maximum likelihood estimation Overview One observes a sample of (conditionally) i.i.d. observations (y t, x t). These observations are realizations of random variables with a known distribution but unknown parameters. The basic idea is to maximize the probability of observing these realizations (given a known parametric distribution with unknown parameters). The probability of observing these realizations is provided by the joint density (or pdf) of the observations (or evaluated at the observations). This joint density is the likelihood function, with the exception that it is a function of the unknown parameters (and not the observations!). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

38 5.2. Estimation Maximum likelihood estimation Estimation Definition The exact likelihood function of the linear regression model is defined to be L(y, x; δ 0 ) = T L t (y t, x t ; δ 0 ) where L t is the (exact) likelihood function for observation t L t (y t, x t ; δ 0 ) = f Yt X t (y t x t ; δ 0 ) f Xt (x t ). where f Yt Xt (respectively, f Xt ) is the (conditional) probability density function of Y t X t (respectively, of X t ). Accordingly, the exact log-likelihood function is defined to be l(y, x; δ 0 ) = T l t (y t, x t ; δ 0 ) T logl t (y t, x t ; δ 0 ). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

39 Maximum likelihood estimation Estimation The exact likelihood function is given by T T L(y, x; δ 0 ) = f Yt X t (y t x t ; δ 0 ) f Xt (x t ) The second right-hand side term is often neglected (especially if f X does not depend on δ 0 ) and one considers the conditional likelihood function T L(y x; δ 0 ) = f Yt X t (y t x t ; δ 0 ) T ( = σ 2 ɛ 2π ) 1/2 exp ( 1 ) 2σ 2 (y t x tb 0 ) 2 ɛ = ( σɛ2π 2 ) ( ) T /2 exp 1 T 2σɛ 2 (y t x tb 0 ) 2 and the conditional log-likelihood function is given by l(y x; δ 0 ) = T 2 log(σ2 ɛ) T 2 log(2π) 1 T 2σɛ 2 (y t x tb 0 ) 2. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

40 Maximum likelihood estimation Estimation Definition The maximum likelihood estimator of δ 0 in the linear regression model, which is the solution of the problem max δ L(y x; δ) maxl(y x; δ), is asymptotically unbiased and efficient, and normally distributed ) l T (ˆδml δ 0 N ( 0 k 1, I 1 ) F where I F is the Fisher information matrix. δ Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

41 Maximum likelihood estimation Example : AR(1) estimation Estimation Consider the following AR(1) representation ( φ < 1) X t = φx t 1 + ɛ t. where ɛ t or ɛ t X t 1 i.i.d.n ( 0, σɛ 2 ) Therefore Accordingly, X t X t 1 = x t 1 N ( φx t 1, σɛ 2 ) ) X 1 N ( 0, σ 2 ɛ 1 φ 2 f Xt X t 1 (x t x t 1 ; δ 0 ) = ( σ 2 ɛ 2π ) 1/2 exp ( 1 f X1 (x 1 ; δ 0 ) = 2σ 2 ɛ (x t φx t 1 ) 2 ) ( ) σ 2 1/2 ɛ 1 φ 2 2π exp ( 1 ) φ2 2σɛ 2 x1 2. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

42 Maximum likelihood estimation Exact (log-) likelihood function Estimation The exact likelihood function is given by T L(x; δ 0 ) = L 1 (x 1 ; δ 0 ) L τ (x τ x τ 1 ; δ 0 ) = f X1 (x 1 ; δ 0 ) = τ=2 T f Xτ X τ 1 (x τ x τ 1 ; δ 0 ) τ=2 ( ) σ 2 1/2 ɛ 1 φ 2 2π exp ( 1 φ2 ( σ 2 ɛ2π ) T 1 2 exp ( 1 2σ 2 ɛ 2σ 2 ɛ x 2 1 ) ) T (x t φx t 1 ) 2. The exact log-likelihood is l(x; δ 0 ) = T 2 log(2π) 1 ( ) σ 2 2 log ɛ 1 φ 2 1 φ2 T 1 log(σ 2 2 ɛ) 1 2σɛ 2 t=2 2σ 2 ɛ T (x t φx t 1 ) 2. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50 t=2 x 2 1

43 Maximum likelihood estimation Estimation Conditional (log-) likelihood function The conditional likelihood function is given by L(x x 1 ; δ 0 ) = = T L τ (x τ x τ 1 ; δ 0 ) τ=2 T f Xτ X τ 1 (x τ x τ 1 ; δ 0 ) τ=2 = ( σ 2 ɛ 2π ) T 1 2 exp The conditional log-likelihood is ( 1 2σ 2 ɛ ) T (x t φx t 1 ) 2. t=2 l(x x 1 ; δ 0 ) = T 1 log(2π) T 1 log(σɛ 2 ) T 2σɛ 2 (x t φx t 1 ) 2. t=2 Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

44 Maximum likelihood estimation Estimation The two methods lead to different maximum likelihood estimates ˆδ cml ˆδ eml The conditional maximum likelihood estimator of φ is also the ordinary least squares estimator of φ This also holds for any AR(p). ˆφ cml = ˆφ ols The Yule-Walker, conditional maximum likelihood and (ordinary) least squares estimator are asymptotically equivalent. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

45 Discussion Maximum likelihood estimation Estimation Generally the maximum likelihood estimates are built as if the observations are Gaussian, though it is not necessary that (X t ) is Gaussian when doing the estimation. If the model is misspecified, the only difference (to some extent...) is that estimates may be less efficient. In this case, one can use the pseudo- or quasi-maximum likelihood estimation. In the case of linear models, there might be not so much gain of using exact maximum likelihood against conditional likelihood method (or least squares method). Maximum likelihood estimation often requires nonlinear numerical optimization procedures. Estimation of the Fisher information matrix can be cumbersome (especially in finite samples). Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

46 Appendix Appendix : ML estimation of a MA(q) One has Step 1 : Writing ɛ = (ɛ 1,, ɛ T ) ɛ 1 q = ɛ 1 q. =. ɛ 0 = ɛ 0 ɛ 1 = x 1 θ 1 ɛ 1 θ q ɛ 1 q ɛ 1 = x 2 θ 1 ɛ 2 θ q ɛ 2 q. =. ɛ T = x T θ 1 ɛ T θ q ɛ T q Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

47 Appendix Therefore ɛ = NX + Zɛ = ( Z N ) ( ɛ X ) with X = (x 1,, x T ), ɛ = (ɛ 1 q, ɛ 2 q,, ɛ 1, ɛ 0 ), and N = ( 0q T A 1 (θ) ) ( Iq, Z = A 2 (θ) where A 1 (θ) is a lower triangular matrix (T T ), A 2 (θ) is a T q matrix, and I q is the identity matrix of order q. ) Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

48 Appendix Step 2 : Determination of the joint density of ɛ The joint probability density function of ɛ is defined to be [ ( ) 2πσ 2 T +q 2 ɛ exp 1 ] 2σɛ 2 (NX + Zɛ ) (NX + Zɛ ). Step 3 : Determination of the marginal density of X The marginal density of X is defined to be ( ) 2πσ 2 T ( 2 ɛ detx X ) [ 1 2 exp 1 ] 2σɛ 2 (NX + Z ˆɛ ) (NX + Z ˆɛ ) with ˆɛ = (Z Z) 1 Z NX. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

49 Appendix Step 4 : Determination of the log-likelihood funcion The log-likelihood function is defined to be l (x; δ) = T 2 log(2π) T 2 log(σ2 ɛ ) T 2 log ( detx X ) S(θ) 2σ 2 ɛ where δ = (θ, σ 2 ɛ ) and S(θ) = (NX + Z ˆɛ ) (NX + Z ˆɛ ). Step 5 : Determination of ˆσ ɛ,ml 2 Using the first-order condition w.r.t. σɛ 2, one has ) S (ˆθ ml ˆσ ɛ,ml 2 =. T Florian Pelgrin (HEC) Univariate time series Sept Dec / 50

50 Appendix Step 6 : Determination of the concentrated log-likelihood funcion The concentrated log-likelihood function is defined to be l (x; θ) = T log [S(θ)] log [ detx X ]. Step 7 : Determination of ˆθ ml ˆθ ml solves Remarks : min θ { T log [S(θ)] + log [ detx X ]}. 1 Least squares methods only focus on the first right-hand side term of the log-likelihood function. Exact methods account for both right-hand side terms. 2 This method generalizes to ARMA(p,q) models. Florian Pelgrin (HEC) Univariate time series Sept Dec / 50