State Space Time Series Analysis p. 1 State Space Time Series Analysis Siem Jan Koopman http://staff.feweb.vu.nl/koopman Department of Econometrics VU University Amsterdam Tinbergen Institute 2011
State Space Time Series Analysis p. 2 Classical Decomposition A basic model for representing a time series is the additive model y t = µ t +γ t +ε t, t = 1,...,n, also known as the Classical Decomposition. y t = observation, µ t = slowly changing component (trend), γ t = periodic component (seasonal), ε t = irregular component (disturbance). In a Structural Time Series Model (STSM) or Unobserved Components Model (UCM), the RHS components are modelled explicitly as stochastic processes.
State Space Time Series Analysis p. 3 Local Level Model Components can be deterministic functions of time (e.g. polynomials), or stochastic processes; Deterministic example: y t = µ+ε t with ε t NID(0,σ 2 ε). Stochastic example: the Random Walk plus Noise, or Local Level model: y t = µ t +ε t, ε t NID(0,σ 2 ε) µ t+1 = µ t +η t, η t NID(0,σ 2 η), The disturbances ε t, η s are independent for all s,t; The model is incomplete without a specification for µ 1 (note the non-stationarity): µ 1 N(a,P)
State Space Time Series Analysis p. 4 Local Level Model y t = µ t +ε t, ε t NID(0,σ 2 ε) µ t+1 = µ t +η t, η t NID(0,σ 2 η), µ 1 N(a,P) The level µ t and the irregular ε t are unobserved; Parameters: σ 2 ε,σ 2 η; Trivial special cases: σ 2 η = 0 = y t NID(µ 1,σ 2 ε) (WN with constant level); σ 2 ε = 0 = y t+1 = y t +η t (pure RW); Local Level is a model representation for EWMA forecasting.
State Space Time Series Analysis p. 5 Local Linear Trend Model The LLT model extends the LL model with a slope: y t = µ t +ε t, ε t NID(0,σ 2 ε), µ t+1 = β t +µ t +η t, η t NID(0,σ 2 η), β t+1 = β t +ξ t, ξ t NID(0,σ 2 ξ). All disturbances are independent at all lags and leads; Initial distributions β 1,µ 1 need to specified; If σξ 2 = 0 the trend is a random walk with constant drift β 1; (For β 1 = 0 the model reduces to a LL model.) If additionally ση 2 = 0 the trend is a straight line with slope β 1 and intercept µ 1 ; If σξ 2 > 0 but σ2 η = 0, the trend is a smooth curve, or an Integrated Random Walk;
State Space Time Series Analysis p. 6 Trend and Slope in LLT Model 5.0 µ 2.5 0.0 2.5 0 10 20 30 40 50 60 70 80 90 100 0.75 β 0.50 0.25 0.00 0.25 0 10 20 30 40 50 60 70 80 90 100
State Space Time Series Analysis p. 7 Trend and Slope in Integrated Random Walk Model 10 µ 5 0 0 10 20 30 40 50 60 70 80 90 100 0.75 β 0.50 0.25 0.00 0.25 0 10 20 30 40 50 60 70 80 90 100
State Space Time Series Analysis p. 8 Local Linear Trend Model Reduced form of LLT is ARIMA(0,2,2); LLT provides a model for Holt-Winters forecasting; Smooth LLT provides a model for spline-fitting; Smoother trends: higher order Random Walks d µ t = η t
State Space Time Series Analysis p. 9 Seasonal Effects We have seen specifications for µ t in the basic model y t = µ t +γ t +ε t. Now we will consider the seasonal term γ t. Let s denote the number of seasons in the data: s = 12 for monthly data, s = 4 for quarterly data, s = 7 for daily data when modelling a weekly pattern.
State Space Time Series Analysis p. 10 Dummy Seasonal The simplest way to model seasonal effects is by using dummy variables. The effect summed over the seasons should equal zero: γ t+1 = s 1 j=1 γ t+1 j. To allow the pattern to change over time, we introduce a new disturbance term: γ t+1 = s 1 j=1 γ t+1 j +ω t, ω t NID(0,σ 2 ω). The expectation of the sum of the seasonal effects is zero.
State Space Time Series Analysis p. 11 Trigonometric Seasonal Defining γ jt as the effect of season j at time t, an alternative specification for the seasonal pattern is γ t = [s/2] j=1 γ jt, γ j,t+1 = γ jt cosλ j +γ jtsinλ j +ω jt, γ j,t+1 = γ jt sinλ j +γ jtcosλ j +ω jt, ω jt,ω jt NID(0,σ 2 ω), λ j = 2πj/s. Without the disturbance, the trigonometric specification is identical to the deterministic dummy specification. The autocorrelation in the trigonometric specification lasts through more lags: changes occur in a smoother way;
State Space Time Series Analysis p. 12 Seatbelt Law 7.9 7.8 7.7 7.6 7.5 7.4 7.3 7.2 7.1 7.0 70 75 80 85
State Space Time Series Analysis p. 13 State Space Model Linear Gaussian state space model (LGSSM) is defined in three parts: State equation: Observation equation: α t+1 = T t α t +R t ζ t, ζ t NID(0,Q t ), y t = Z t α t +ε t, ε t NID(0,H t ), Initial state distribution α 1 N(a 1,P 1 ). Notice that ζ t and ε s independent for all t,s, and independent from α 1 ; observation y t can be multivariate; state vector α t is unobserved; matrices T t,z t,r t,q t,h t determine structure of model.
State Space Time Series Analysis p. 14 State Space Model state space model is linear and Gaussian: therefore properties and results of multivariate normal distribution apply; state vector α t evolves as a VAR(1) process; system matrices usually contain unknown parameters; estimation has therefore two aspects: measuring the unobservable state (prediction, filtering and smoothing); estimation of unknown parameters (maximum likelihood estimation); state space methods offer a unified approach to a wide range of models and techniques: dynamic regression, ARIMA, UC models, latent variable models, spline-fitting and many ad-hoc filters; next, some well-known model specifications in state space form...
State Space Time Series Analysis p. 15 Regression with Time Varying Coefficients General state space model: α t+1 = T t α t +R t ζ t, ζ t NID(0,Q t ), y t = Z t α t +ε t, ε t NID(0,H t ). Put regressors in Z t, T t = I, R t = I, Result is regression model with coefficient α t following a random walk.
State Space Time Series Analysis p. 16 ARMA in State Space Form Example: AR(2) model y t+1 = φ 1 y t +φ 2 y t 1 +ζ t, in state space: α t+1 = T t α t +R t ζ t, ζ t NID(0,Q t ), y t = Z t α t +ε t, ε t NID(0,H t ). with 2 1 state vector α t and system matrices: [ ] Z t = 1 0, H t = 0 [ ] [ ] φ 1 1 1 T t =, R t =, Q t = σ 2 φ 2 0 0 Z t and H t = 0 imply that α 1t = y t ; First state equation implies y t+1 = φ 1 y t +α 2t +ζ t with ζ t NID(0,σ 2 ); Second state equation implies α 2,t+1 = φ 2 y t ;
State Space Time Series Analysis p. 17 ARMA in State Space Form Example: MA(1) model y t+1 = ζ t +θζ t 1, in state space: α t+1 = T t α t +R t ζ t, ζ t NID(0,Q t ), y t = Z t α t +ε t, ε t NID(0,H t ). with 2 1 state vector α t and system matrices: [ ] Z t = 1 0, H t = 0 [ ] [ ] 0 1 1 T t =, R t =, Q t = σ 2 0 0 θ Z t and H t = 0 imply that α 1t = y t ; First state equation implies y t+1 = α 2t +ζ t with ζ t NID(0,σ 2 ); Second state equation implies α 2,t+1 = θζ t ;
State Space Time Series Analysis p. 18 ARMA in State Space Form Example: ARMA(2,1) model in state space form y t = φ 1 y t 1 +φ 2 y t 2 +ζ t +θζ t 1 [ y t ] α t = φ 2 y t 1 +θζ t [ ] Z t = 1 0, H t = 0, [ ] [ ] φ 1 1 1 T t =, R t =, Q t = σ 2 φ 2 0 θ All ARIMA(p, d, q) models have a (non-unique) state space representation.
State Space Time Series Analysis p. 19 UC models in State Space Form State space model: α t+1 = T t α t +R t ζ t, y t = Z t α t +ε t. LL model µ t+1 = η t and y t = µ t +ε t : α t = µ t, T t = 1, R t = 1, Q t = σ 2 η, Z t = 1, H t = σ 2 ε. LLT model µ t+1 = β t +η t, β t+1 = ξ t and y t = µ t +ε t : α t = [ µ t β t ] [ ] [ 1 1, T t =, R t = 0 1 [ ] Z t = 1 0, H t = σε. 2 1 0 0 1 ], Q t = [ ] ση 2 0 0 σξ 2,
State Space Time Series Analysis p. 20 UC models in State Space Form State space model: α t+1 = T t α t +R t ζ t, y t = Z t α t +ε t. LLT model with season: µ t+1 = β t +η t, β t+1 = ξ t, S(L)γ t+1 = ω t and y t = µ t +γ t +ε t : α t = [ µ t β t γ t γ t 1 γ t 2 ], 1 1 0 0 0 1 0 0 0 1 0 0 0 ση 2 0 0 0 1 0 T t = 0 0 1 1 1, Q t = 0 σξ 2 0, R t = 0 0 1, 0 0 1 0 0 0 0 σω 2 0 0 0 0 0 0 1 0 0 0 0 [ ] Z t = 1 0 1 0 0, H t = σε. 2
State Space Time Series Analysis p. 21 Kalman Filter The Kalman filter calculates the mean and variance of the unobserved state, given the observations. The state is Gaussian: the complete distribution is characterized by the mean and variance. The filter is a recursive algorithm; the current best estimate is updated whenever a new observation is obtained. To start the recursion, we need a 1 and P 1, which we assumed given. There are various ways to initialize when a 1 and P 1 are unknown, which we will not discuss here.
State Space Time Series Analysis p. 22 Kalman Filter The unobserved state α t can be estimated from the observations with the Kalman filter: v t = y t Z t a t, F t = Z t P t Z t +H t, K t = T t P t Z tf 1 t, a t+1 = T t a t +K t v t, P t+1 = T t P t T t +R t Q t R t K t F t K t, for t = 1,...,n and starting with given values for a 1 and P 1. Writing Y t = {y 1,...,y t }, a t+1 = E(α t+1 Y t ), P t+1 = var(α t+1 Y t ).
State Space Time Series Analysis p. 23 Kalman Filter State space model: α t+1 = T t α t +R t ζ t, y t = Z t α t +ε t. Writing Y t = {y 1,...,y t }, define a t+1 = E(α t+1 Y t ), P t+1 = var(α t+1 Y t ); The prediction error is v t = y t E(y t Y t 1 ) = y t E(Z t α t +ε t Y t 1 ) = y t Z t E(α t Y t 1 ) = y t Z t a t ; It follows that v t = Z t (α t a t )+ε t and E(v t ) = 0; The prediction error variance is F t = var(v t ) = Z t P t Z t +H t.
State Space Time Series Analysis p. 24 Lemma The proof of the Kalman filter uses a lemma from multivariate Normal regression theory. Lemma Suppose x, y and z are jointly Normally distributed vectors with E(z) = 0 and Σ yz = 0. Then E(x y,z) = E(x y)+σ xz Σ 1 zz z, var(x y,z) = var(x y) Σ xz Σ 1 zz Σ xz,
State Space Time Series Analysis p. 25 Multivariate local level model Seemingly Unrelated Time Series Equations model: y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,Σ η ). Observations are p 1 vectors; The disturbances ε t, η s are independent for all s,t; The p different time series are related through correlations in the disturbances. For a full discussion, see Harvey and Koopman (1997). A pdf version (scanned) at http://staff.feweb.vu.nl/koopman under section Publications and subsection Published articles as contributions to books.
State Space Time Series Analysis p. 26 Multivariate LL Model The multivariate LL model is given by y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,Σ η ). First difference y t = η t 1 + ε t is stationary; Reduced form: y t is VMA(1) or VAR( );
State Space Time Series Analysis p. 27 Multivariate LL Model Stochastic properties are multivariate analogous of univariate case: Γ 0 = E( y t y t) = Σ η +2Σ ε Γ 1 = E( y t y t 1) = Σ ε Γ τ = E( y t y t τ) = 0, τ 2, The unrestricted vector MA(1) process has p 2 +p(p+1)/2 parameters, the SUTSE has p (p+1); Such multivariate reduced form representations can also be established for general models.
State Space Time Series Analysis p. 28 Homogeneous Multivariate LL Model The homogeneous multivariate LL model is given by y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,qΣ ε ), where q is a non-negative scalar. This implies that Σ η = qσ ε. The model is restricted, all series in y t have the same dynamic properties (the same acf). Not so relevant in practical work apart from forecasting. It is the model representation for exponentially weighted moving average (EWMA) forecasting of multiple time series. This can be generalised to more general components models. Easy to estimate, only a set of univariate Kalman filters are required.
State Space Time Series Analysis p. 29 Common Levels The common local level model is given by y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,Σ η ), where rank(σ η ) = r < p. the model can be described by r underlying level components, the common levels; Σ η = AΣ c A, A is p r, Σ c is r r of full rank; interpretation of A: factor loading matrix.
State Space Time Series Analysis p. 30 Common Levels The common local level model y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,AΣ c A ), can be rewritten in terms of underlying levels: y t = a+aµ c t +ε t, µ c t+1 = µ c t +η c t, η c t NID(0,Σ c ), so that µ t = a+aµ c t, η t = Aη c t.
State Space Time Series Analysis p. 31 Common Levels For the common local level model y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,AΣ c A ), notice that decomposition Σ η = AΣ c A is not unique; identification restrictions: Σ c is diagonal, Choleski decomposition, principal compoments (based on economic theory); more interesting interpretation can be obtained by factor rotations; can be interpreted as dynamic factor analysis, see later.
State Space Time Series Analysis p. 32 Common components Common dynamic factors: are useful for interpretation cointegration; have consequence for inference and forecasting (dimension of parameter space reduces as a result). common local level model can be generally represented as a VAR( ) or VECM models, details can be provided upon request.
State Space Time Series Analysis p. 33 Multivariate components So far, we have concentrated on multivariate variants of the local level model; Similar considerations can be applied to other components such as the slope of the trend, seasonal and cycle components and other time-varying features in the multiple time series. Harvey and Koopman (1997) review such extensions. In particular, they define the similar cycle component, see Exercises.
State Space Time Series Analysis p. 34 Common and idiosyncratic factors Multiple trends can also be decomposed into a one common factor and multiple idiosyncratic factors: y t = µ t +ε t, ε t NID(0,Σ ε ), µ t+1 = µ t +η t, η t NID(0,Σ η ), where Σ η = δδ +D η with vector δ and diagonal matrix D η. This implies that the level can be represented by µ t = δµ c t +µ t, η t = δη c t +η t with common level (scalar) µ c t and independent level µ it generated by µ c t+1 = η c t NID(0,1), µ t+1 = η t NID(0,D η ).
State Space Time Series Analysis p. 35 Mulitvariate Kalman filter The Kalman filter is valid for the general multivariate state space model. Computationally it is not convenient when p becomes large, very large. Each step of the Kalman filter requires the inversion of the p p matrix F t. This is no problem when p = 1 (univariate) but when p > 20, say, it will slow down the Kalman filter considerably. However, we can treat each element in the p 1 observation vector y t as a single realisation. In other words, we can "update" each single element of y t within the Kalman filter. The arguments are given in DK book 6.4. The same applies to smoothing.
State Space Time Series Analysis p. 36 Univariate treatment of Kalman filter Consider standard model: y t = Z t α t +ε t and α t+1 = T t α t +R t η t where Var(ε t ) = H t is diagonal. Observation vector y t = (y t,1,...,y t,pt ) is treated and we view observation model as a set of p t separate equations. We then have, y t,i = Z t,i α t,i +ε t,i with α t,i = α t for i = 1,...,p t. The associated transition equations become α t,i+1 = α t,i for i = 1,...,p t and α t+1,1 = T t α t,pt +R t η t for t = 1,...,n. This disentangled model can be treated by the Kalman filter and smoother equations straightforwardly. Innovations are now relative to the past and the previous observations inside y t,pt! Non-diagonal matrix H t can be treated by data-transformation or by including ε t in the state vector α t. More details in DK book 6.4.
State Space Time Series Analysis p. 37 Exercise 1 Consider the common trends model of Harvey and Koopman (1997, 9.4.1 and 9.4.2). 1. Put the common trends model with (possibly common) stochastic slopes and based on equations (21)-(23) in state space form. 2. Put the common trends model with (possibly common) stochastic slopes and based on equations (24)-(26) in state space form. Define all vectors and matrices precisely. 3. Discuss the generalisation of Σ η 0 and the consequences for the state space formulation of the model as in 2.
State Space Time Series Analysis p. 38 Exercise 2 Consider the multivariate trend model of Harvey and Koopman (1997). 1. Consider a multiple data set of N time series y t. The aim is to decompose the time series into trend and stationary components. It is further required that the multiple trend can be decomposed into a common single trend (common to all N time series) and idiosyncratic trends (specific to the individual time series). Formulate a model for such a decomposition. Discuss the identification of the different trends. Express the model in state space form. 2. Once multiple trend models are expressed in state space form, we need to estimate the parameter coefficients of the model. Please describe shortly some relevant issues of maximum likelihood estimation. Is it feasible? What problems can you expect? Any recommendations for a successful implementation?
State Space Time Series Analysis p. 39 Exercise 3 Consider the similar cycle model of Harvey and Koopman (1997) with observation equation y t = ψ t +ε t, ε t N(0,Σ ε ), where y t is a 3 1 observation vector. Cycle ψ t represents a common similar cycle component of rank 2. 1. Please provide the state space representation of this model. 2. Comment on the restrictive nature of the similar cycle model. 3. How would you modify the similar cycle model so that each time series in y t has a different cycle frequency λ. 4. Can you apply the univariate Kalman filter of DK 6.4 in case Σ ε is diagonal? What if Σ ε is not diagonal? Give details.