Stationary models. MA, AR and ARMA. Matthieu Stigler. November 14, Version 1.1

Transcription

1 Stationary models MA, AR and ARMA Matthieu Stigler November 14, 2008 Version 1.1 This document is released under the Creative Commons Attribution-Noncommercial 2.5 India license. Matthieu Stigler () Stationary models November 14, / 65

2 Lectures list 1 Stationarity 2 ARMA models for stationary variables 3 Seasonality 4 Non-stationarity 5 Non-linearities 6 Multivariate models 7 Structural VAR models 8 Cointegration the Engle and Granger approach 9 Cointegration 2: The Johansen Methodology 10 Multivariate Nonlinearities in VAR models 11 Multivariate Nonlinearities in VECM models Matthieu Stigler () Stationary models November 14, / 65

3 Outline 1 Last Lecture 2 AR(p) models Autocorrelation of AR(1) Stationarity Conditions Estimation 3 MA models ARMA(p,q) The Box-Jenkins approach 4 Forecasting Matthieu Stigler () Stationary models November 14, / 65

4 Recall: auto-covariance Definition (autocovariance) Cov(X t, X t k ) γ k (t) E [(X t µ)(x t k µ)] Definition (Autocorrelation) Corr(X t, X t k ) ρ k (t) Cov(Xt,X t k) Var(X t) Proposition Corr(X t, X t 0 ) = Var(X t ) Corr(X t, X t j ) = φ j depend on the lage: plot its values at each lag. Matthieu Stigler () Stationary models November 14, / 65

5 Recall: stationarity The stationarity is an essential property to define a time series process: Definition A process is said to be covariance-stationary, or weakly stationary, if its first and second moments are time invariant. E(Y t ) = E[Y t 1 ] = µ Var(Y t ) = γ 0 < Cov(Y t, Y t k ) = γ k t t t, k Matthieu Stigler () Stationary models November 14, / 65

6 Recall: The AR(1) The AR(1): Y t = c + ϕy t 1 + ε t ε t iid(0, σ 2 ) with ϕ < 1, it can be can be written as: Y t = c t 1 1 ϕ + ϕ i ε t i i=0 Its moments do not depend on the time: : E(X t ) = Var(X t ) = c 1 ϕ σ2 1 ϕ 2 Cov(X t, X t j ) = Corr(X t, X t j ) = φ j ϕj σ 2 1 ϕ 2 Matthieu Stigler () Stationary models November 14, / 65

9 Autocorrelation function A usefull plot to understand the dynamic of a process is the autocorrelation function: Plot the autocorrelation value for different lags. Matthieu Stigler () Stationary models November 14, / 65

10 φ = 0 φ = 0.5 ρ k ρ k Lag k Lag k φ = 0.9 φ = 1 ρ k ρ k Lag k Lag k Matthieu Stigler () Stationary models November 14, / 65

11 AR(1) with 1 < φ < 0 in the AR(1): Y t = c + ϕy t 1 + ε t ε t iid(0, σ 2 ) with 1 < φ < 0 we have negative autocorrelation. Matthieu Stigler () Stationary models November 14, / 65

12 phi= 0.2 phi= 0.7 ar ar Time Time φ = 0.2 φ = 0.7 ρ k ρ k Lag k Lag k Matthieu Stigler () Stationary models November 14, / 65

13 Definition (AR(p)) y t = c + φ 1 y t 1 + φ 2 y t φ p y t p + ε t Expectation? Variance? Auto-covariance? Stationary conditions? Matthieu Stigler () Stationary models November 14, / 65

14 Lag operator Definition (Backshift /Lag operator) LX t = X t 1 Proposition See that: L 2 X t = X t 2 Proposition (Generalisation) L k X t = X t k Matthieu Stigler () Stationary models November 14, / 65

15 Lag polynomial We can thus rewrite: Example (AR(2)) X t = c + ϕ 1 X t 1 + ϕ 2 X t 2 + ε t (1 ϕ 1 L ϕ 2 L 2 )X t = c + ε t Definition (lag polynomial) We call lag polynomial: Φ(L) = (1 ϕ 1 L ϕ 2 L 2... φ p L p ) So we write compactly: Example (AR(2)) Φ(L)X t = c + ε t Matthieu Stigler () Stationary models November 14, / 65

17 Definition (Characteristic polynomial) (1 ϕ 1 z ϕ 2 z 2... φ p z p ) Stability condition: Proposition The AR(p) process is stable if the roots of the lag polynomial lie outside the unit circle. Example (AR(1)) The AR(1): X t = ϕx t 1 + ε t can be written as: (1 ϕl)x t = ε t Solving it gives: 1 ϕx = 0 x = 1 ϕ And finally: 1 ϕ > 1 ϕ < 1 Matthieu Stigler () Stationary models November 14, / 65

18 Proof. 1 Write an AR(p) as AR(1) 2 Show conditions for the augmented AR(1) 3 Transpose the result to the AR(p) Matthieu Stigler () Stationary models November 14, / 65

19 Proof. The AR(p): y t = φ 1 y t 1 + φ 2 y t φ p y t p + ε t can be recast as the AR(1) model: ξ t = F ξ t 1 + ε t y t y t 1 y t 2. y t p+1 y t y t 1... φ 1 φ 2 φ 3... φ p 1 φ p = y t p+1 y t 1 y t 2 y t 3. y t p ε t = c + φ 1 y t 1 + φ 2 y t φ p y t p + ε t = y t 1 = y t p+1 Matthieu Stigler () Stationary models November 14, / 65

20 Proof. Starting from the augmented AR(1) notation: ξ t = F ξ t 1 + ε t Similarly as in the simple case, we can write the AR model recursively: ξ t = F t ξ 0 + ε t + F ε t 1 + F 2 ε t F t 1 ε 1 + F t ε 0 Remember the eigenvalue decomposition: F = T ΛT 1 and the propriety that: F j = T Λ j T 1 with λ j Λ j 0 λ j = λ j 3 So the AR(1) model is stable if λ i < 1 i Matthieu Stigler () Stationary models November 14, / 65

21 Proof. So the condition on F is that all λ from F λi = 0 are < 1. One can show that the eigenvalues of F are: Proposition λ p φ 1 λ p 1 φ 2 λ p 2... φ p 1 λ φ p = 0 But the λ are the reciprocal of the values z that solve the characteristic polynomial of the AR(p): (1 ϕ 1 z ϕ 2 z 2... φ p z p ) = 0 So the roots of the polynomial should be > 1, or, with complex values, outside the unit circle. Matthieu Stigler () Stationary models November 14, / 65

22 Stationarity conditions The conditions of roots outside the unit circle lead to: AR(1): φ < 1 AR(2): φ1 + φ 2 < 1 φ1 φ 2 < 1 φ 2 < 1 Matthieu Stigler () Stationary models November 14, / 65

23 Example Consider the AR(2) model: Its AR(1) representation is: [ ] yt = Y t = 0.8Y t Y t 2 + ε t y t 1 [ Hence its eigenvalues are taken from: 0.8 λ λ = λ2 0.8λ 0.09 = 0 And the eigenvalues are smaller than one: > Re(polyroot(c(-0.09, -0.8, 1))) [1] ] [ ] yt 1 + y t 2 [ ] εt 0 Matthieu Stigler () Stationary models November 14, / 65

24 Example Y t = 0.8Y t Y t 2 + ε t Its lag polynomial representation is: (1 0.8L 0.09L 2 )X t = ε t Its characteristic polynomial is hence: (1 0.8x 0.09x 2 ) = 0 whose solutions lie outside the unit circle: > Re(polyroot(c(1, -0.8, -0.09))) [1] And it is the inverse of the previous solutions: > all.equal(sort(1/re(polyroot(c(1, -0.8, -0.09)))), Re(polyroot(c(-0.09, , 1)))) [1] TRUE Matthieu Stigler () Stationary models November 14, / 65

25 Unit root and integration order Definition A process is said to be integrated of order d if it becomes stationary after being differenced d times. Proposition An AR(p) process with k unit roots (or eigenvalues) is integrated of order k. Example Take the random walk: X t = X t 1 + ε t Its polynomial is (1-L), and the roots is 1 x = 0 x = 1 The eigenvalue of the trivial AR(1) is 1 λ = 0 λ = 1 So the random walk is integrated of order 1 (or difference stationary). Matthieu Stigler () Stationary models November 14, / 65

26 Integrated process Take an AR(p): With the lag polynomial: y t = φ 1 y t 1 + φ 2 y t φ p y t p + ε t Φ(L)X t = ε t If one of its p (not necessarily distinct) eigenvalues is equal to 1, it can be rewritten: (1 L)Φ (L)X t = ε t Equivalently: Φ (L) X t = ε t Matthieu Stigler () Stationary models November 14, / 65

27 The AR(p) in detail Moments of a stationary AR(p) E(X t ) = c 1 ϕ 1 ϕ 2... ϕ p Var(X t ) = ϕ 1 γ 1 + ϕ 2 γ ϕ p γ p + σ 2 Cov(X t, X t j ) = ϕ 1 γ j 1 + ϕ 2 γ j ϕ p γ j p Note that γ j Cov(X t, X t j ) so we can rewrite both last equations as: { γ 0 = ϕ 1 γ 1 + ϕ 2 γ ϕ p γ p + σ 2 γ j = ϕ 1 γ j 1 + ϕ 2 γ j ϕ p γ j p They are known under the name of Yule-Walker equations. Matthieu Stigler () Stationary models November 14, / 65

28 Yule-Walker equations Dividing by γ 0 gives: { ρ 0 = ϕ 1 ρ 1 + ϕ 2 ρ ϕ p ρ p + σ 2 ρ j = ϕ 1 ρ j 1 + ϕ 2 ρ j ϕ p ρ j p Example (AR(1)) We saw that: Var(X t ) = σ2 1 ϕ 2 Cov(X t, X t j ) = Corr(X t, X t j ) = φ j ϕj σ 2 1 ϕ 2 And we have effectively: ρ 1 = φρ 0 = φ and ρ 2 = φρ 1 = φ 2 Utility: Determination of autocorrelation function Estimation Matthieu Stigler () Stationary models November 14, / 65

30 To estimate a AR(p) model from a sample of T we take t=t-p Methods of moments: estimate sample moments (the γ i ), and find parameters (the φ) correspondly Unconditional ML: assume y p,..., y 1 N (0, σ 2 ). Need numerical optimisation methods. Conditional Maximum likelihood (=OLS): estimate f (y T, t T 1,..., y p+1 y p,..., y 1 ; θ) and assume ε t N (0, σ 2 ) and that y p,..., y 1 are given What if errors are not normally distributed? Quasi-maximum likelihood estimator, is still consistent (in this case) but standard erros need to be corrected. Matthieu Stigler () Stationary models November 14, / 65

32 Moving average models Two significations! regression model Smoothing technique! Matthieu Stigler () Stationary models November 14, / 65

33 MA(1) Definition (MA(1)) Y t = c + ε t + θɛ t 1 E(Y t ) = c Var(Y t ) = (1 + θ 2 )σ 2 { θσ 2 if j = 1 Cov(X t, X t j ) = 0 if j > 1 { θ if j = 1 (1+θ Corr(X t, X t j ) = 2 ) 0 if j > 1 Proposition A MA(1) is stationnary for every θ Matthieu Stigler () Stationary models November 14, / 65

34 θ = 0.5 θ = Time Time θ = 0.5 θ = Time Time Matthieu Stigler () Stationary models November 14, / 65

35 θ = 0.5 θ = 3 ρ k ρ k Lag k Lag k θ = 0.5 θ = 3 ρ k ρ k Lag k Lag k Matthieu Stigler () Stationary models November 14, / 65

36 MA(q) The MA(q) is given by: Y t = c + ε t + θ 1 ɛ t 1 + θ 2 ɛ t θ 1 ɛ t q E(Y t ) = c Var(Y t ) = (1 + θ θ θ2 q)σ 2 Cov(X t, X t j ) = { σ 2 (θ j + θ j+1 θ 1 + θ j+2 θ θ q θ q 1 ) if j = 1 0 if j > 1 { θ if j = 1 (1+θ Corr(X t, X t j ) = 2 ) 0 if j > 1 Proposition A MA(q) is stationary for every sequence {θ 1, θ 2,..., θ q } Matthieu Stigler () Stationary models November 14, / 65

37 Θ = c(0.5, 1.5) Θ = c( 0.5, 1.5) ρ k ρ k Lag k Lag k Θ = c( 0.6, 0.3, 0.5, 0.5) Θ = c( 0.6, 0.3, 0.5, 0.5, 3, 2, 1) ρ k ρ k Lag k Lag k Matthieu Stigler () Stationary models November 14, / 65

38 The MA( ) Take now the MA( ): Y t = ε t + θ 1 ɛ t 1 + θ 2 ɛ t θ ɛ = θ j ε t j j=0 Definition (Absolute summability) A sequence is absolute summable if i=0 α i < 0 Proposition The MA( ) is stationary if the coefficients are absolute summable. Matthieu Stigler () Stationary models November 14, / 65

39 Back to AR(p) Recall: Proposition If the characteristic polynomial of a AR(p) has roots =1, it is not stationary. See that: (1 φ 1 y t 1 φ 2 y t 2... φ p y t p )y t = (1 α 1 L)(1 α 2 L)... (1 α p L)y t = ε t It has a MA( ) representation if: α 1 1: 1 y t = (1 α 1 L)(1 α 2 L)...(1 α ε pl) t Furthermore, if the α i (the eigenvalues of the augmented AR(1)) are smaller than 1, we can write it: y t = β i ε t i=0 Matthieu Stigler () Stationary models November 14, / 65

40 Estimation of a MA(1) We do not observe neither ε t nor ε t 1 But if we know ε 0, we know ε 1 = Y t θε 0 So obtain them recursively and minimize the conditional SSR: S(θ) = T t=1 (y t ε t 1 ) 2 This recquires numerical optimization and works only if θ < 1. Matthieu Stigler () Stationary models November 14, / 65

42 ARMA models The ARMA model is a composite of AR and MA: Definition (ARMA(p,q)) X t = c+φ 1 X t 1 +φ 2 X t φ p X t p +ε t 1 +θ 1 ε t 1 +θ 2 ε t θε t q It can be rewritten properly as: Theorem Φ(L)Y t = c + Θ(L)ε t The ARMA(p,q) model is stationary provided the roots of the Φ(L) polynomial lie outside the unit circle. So only the AR part is involved! Matthieu Stigler () Stationary models November 14, / 65

43 Autocorrelation function of a ARMA(p,q) Proposition After q lags, the autocorrelation function follows the pattern of the AR component. Remember: this is then given by the Yule-Walker equations. Matthieu Stigler () Stationary models November 14, / 65

44 phi(1)=0.5, theta(1)=0.5 phi(1)= 0.5, theta(1)=0.5 ρ k ρ k Lag k Lag k phi(1)=0.5, theta(1:3)=c(0.5,0.9, 0.3 phi(1)= 0.5, theta(1:3)=c(0.5,0.9, 0.3 ρ k ρ k Lag k Lag k Matthieu Stigler () Stationary models November 14, / 65

45 ARIMA(p,d,q) Now we add a parameter d representing the order of integration (so the I in ARIMA) Definition (ARIMA(p,d,q)) ARIMA(p,d,q): Φ(L) d Y t = Θ(L)ε t Example (Special cases) White noise: ARIMA(0,0,0) X t = ε t Random walk : ARIMA(0,1,0): X t = ε t X t = X t 1 + ε t Matthieu Stigler () Stationary models November 14, / 65

46 Estimation and inference The MLE estimator has to be found numerically. Provided the errors are normaly distributed, the estimator has the usual asymptotical properties: Consistent Asymptotically efficients Normally distributed If we take into account that the variance had to be estimated, one can rather use the T distribution in small samples. Matthieu Stigler () Stationary models November 14, / 65

48 The Box-Jenkins approach 1 Transform data to achieve stationarity 2 Identify the model, i.e. the parameters of ARMA(p,d,q) 3 Estimation 4 Diagnostic analysis: test residuals Matthieu Stigler () Stationary models November 14, / 65

49 Step 1 Transformations: Log Square root Differenciation Box-Cox transformation: Y (λ) t Is log legitimate? { Y λ t 1 λ for λ 0 log(y t) for λ = 0 Process is: y t e δt Then z t = log(y t ) = δt and remove trend Process is y t = y t 1 + ε t Then (by log(1 + x) = x) log(y t ) = yt y t 1 y t Matthieu Stigler () Stationary models November 14, / 65

50 Step 2 Identification of p,q (d should now be 0 after convenient transformation) Principle of parsimony: prefer small models Recall that incoporating variables increases fit (R 2 ) but reduces the degrees of freedom and hence precision of estimation and tests. A AR(1) has a MA( ) representation If the MA(q) and AR(p) polynomials have a common root, the ARMA(p,q) is similar to ARMA(p-1,q-1). Usual techniques recquire that the MA polynomial has roots outside the unit circle (i.e. is invertible) Matthieu Stigler () Stationary models November 14, / 65

51 Step 2: identification How can we determine the parameters p,q? Look at ACF and PACF with confidence interval Use information criteria Akaike Criterion (AIC) Schwarz criterion (BIC) Definition (IC) AIC(p) = n log ˆσ 2 + 2p BIC(p) = n log ˆσ 2 + p log n Matthieu Stigler () Stationary models November 14, / 65

52 Step 3: estimation Estimate the model... R function: arima() argument: order=c(p,d,q) Matthieu Stigler () Stationary models November 14, / 65

53 Step 4: diagnostic checks Test if the residuals are white noise: 1 Autocorrelation 2 Heteroscedasticity 3 Normality Matthieu Stigler () Stationary models November 14, / 65

54 CPI CPI Time Matthieu Stigler () Stationary models November 14, / 65

55 original diff(cpi) CPI diff(cpi) Time Time log(cpi) diff(log(cpi)) log(cpi) diff(log(cpi)) Time Time Matthieu Stigler () Stationary models November 14, / 65

56 linear trend CPI detrend Time Time Smooth trend CPI CPI smo$y Time Time Quadratic trend CPI detrend Time Time Matthieu Stigler () Stationary models November 14, / 65

57 Diff2 diff Time Diff2 of log diff2log Time Matthieu Stigler () Stationary models November 14, / 65

58 Series CPI2 CPI Time Series CPI2 ACF Partial ACF Lag Lag Matthieu Stigler () Stationary models November 14, / 65

59 > library(forecast) This is forecast 1.17 > fit <- auto.arima(cpi2, start.p = 1, start.q = 1) > fit Series: CPI2 ARIMA(2,0,1)(2,0,2)[12] with zero mean Coefficients: ar1 ar2 ma1 sar1 sar2 sma1 sma s.e sigma^2 estimated as 4.031e-05: log likelihood = AIC = AICc = BIC = > res <- residuals(fit) Matthieu Stigler () Stationary models November 14, / 65

60 Series res res Partial ACF Time Lag Series res ACF Lag Matthieu Stigler () Stationary models November 14, / 65

61 > Box.test(res) Box-Pierce test data: res X-squared = , df = 1, p-value = Matthieu Stigler () Stationary models November 14, / 65

62 Normal Q Q Plot Theoretical Quantiles Sample Quantiles density.default(x = res) N = 315 Bandwidth = Density Matthieu Stigler () Stationary models November 14, / 65

64 Notation (Forecast) ŷ t+j E t (y t+j ) = E(y t+j y t, y t 1,..., ε t, ε t 1,...) is the conditional expectation of y t+j given the information available at t. Definition (J-step-ahead forecast error) e t (j) y t+j ŷ t+j Definition (Mean square prediction error) MSPE 1 H H i=1 e2 i Matthieu Stigler () Stationary models November 14, / 65

65 R implementation To run this file you will need: R Package forecast R Package TSA Data file AjaySeries2.csv put it in a folder called Datasets in the same level than your.rnw file (Optional) File Sweave.sty which change output style: result is in blue, R commands are smaller. Also in same folder as.rnw file. Matthieu Stigler () Stationary models November 14, / 65