DEPwmßswTF warn* snwmmmsm STÜÄFR.CAUFÖ««0DS AND APPLICATINS F TIME SERIES ANALYSIS PART II: LINEAR STCHASTIC MDELS TECHNICAL REPRT N. 2 T. W. ANDERSN AND N. D. SINGPURWALLA CTBER 984 U. S. ARMY RESEARCH FFICE NTRACT DAAG29-82-K-056 THEDRE W. ANDERSN, PRJECT DIRECTR DEPARTMENT F STATISTICS STANFRD UNIVERSITY STANFRD, CALIFRNIA *Ä 3 tt APPRVED FR PUBLIC RELEASE; DISTRIBUTIN UNLIMITED.
METHDS AND APPLICATINS F TIME SERIES ANALYSIS PART II: LINEAR STCHASTIC MDELS TECHNICAL REPRT N. 2 T. W. ANDERSN STANFRD UNIVERSITY and N. D. SINGPURWALLA* THE GERGE WASHINGTN UNIVERSITY CTBER 984 U. S. ARMY RESEARCH FFICE NTRACT DAAG29-82-K-056 Als issued as GWU/IRRA/T-84/, The Institute fr Reliability and Risk Analysis, Schl f Engineering and Applied Science, Gerge Washingtn University, Washingtn D.C. 20052. Research Supprted by Grant DAAG29-84-K-060, U.S. Army Research ffice, and Cntract N0004-77-C-0263, Prject NR042-372, ffice f Naval Research, with The Gerge Washingtn University. DEPARTMENT F STATISTICS STANFRD UNIVERSITY STANFRD, CALIFRNIA APPRVED FR PUBLIC RELEASE; DISTRIBUTIN UNLIMITED.
THE VIEW, PININS, AND/R FINDINGS NTAINED IN THIS REPRT ARE THSE F THE AUTHR(S) AND SHULD NT BE NSTRUED AS AN FFICIAL DEPARTMENT F THE ARMY PSITIN, PLICY, R DECISIN, UNLESS S DESIGNATED BY THER DCUMEN- TATIN.
NTENTS Methds and Applicatins f Time Series Analysis Part II: Linear Stchastic Mdels 5. Intrductin t Autregressive Mdels 5. Statinary Stchastic Prcesses 2 5... Examples f Statinary Stchastic Prcesses 6 6. Basic Ntins f Multivariate Nrmal Distributins 9 7. Estimatin f the Crrelatin Functin 2 8. Autregressive Prcesses 6 8. Representatin as an Infinite Mving Average 7 8... Cnditins fr Cnvergence in the Mean f Autregressive Prcesses 20 8.2 Evaluatin f the Cefficients 6 and their Behavir 24 8.2.. Special Cases Describing the Evaluatin and the Behavir f <5 's 26 8.3 The Cvariance Functin f an Autregressive Prcess 32 8.3.. Special Cases Describing the Behavir f the Autcvariance Functin f an Autregressive Prcess 34 8.3.2. Behavir f the Estimated Autcrrelatin Functin f Sme Simulated Autregressive Prcesses 4 8.4 Expressing the Parameters f an Autregressive Prcess in Terms f Autcrrelatins 49 8.5 The Partial Autcrrelatin Functin f an Autregressive Prcess 50 8.5l. Relatinship between Partial Autcrrelatin and the Last Cefficient f an Autregressive Prcess 53
8.5.2, Behavir f the Estimated Partial Autcrrelatin Functin f Sme Simulated Autregressive Prcesses 56 8.6 An Explanatin f the Fluctuatins in Autregressive Prcesses 64 8.7 Autregressive Prcesses with Independent Variables 65 8.8 Statinary Autregressive Prcesses Whse Assciated Plynmial Equatin Have at Least ne Rt Equal t 69 8.9 Sme Linear Nnstatinary Prcesses 7 8.9.. Behavir f the Cvariance Functin f 73 Integrated Autregressive Prcesses 8.9.2. The Cvariance Functin f Sme Prcesses with an Underlying Trend 77 8.9.3. Behavir f Estimated Autcrrelatin Functin f a Real Life Nnstatinary Time Series 86 8.0 Frecasting (Predictin) fr Statinary Autregressive Prcesses 93 8. Examples f Sme Real Life Time Series Described by Autregressive Prcesses 96 8... The Weekly Rtary Rigs in Use Data 96 8..2. The Landsat 2 Satellite Data 07 n
-- 5. Intrductin t Autregressive Mdels In the previus sectins we cnsidered mdels fr time series in which the characteristic and useful prperties apprpriate t the time sequence were embdied in the mean functin f(t) ; f(t) culd be a plynmial r a trignmetric functin. In astrnmy, fr example, it is reasnable t suppse that the effect f time is mainly in f(t) and thus predictin is reasnable. In ecnmics and weather, fr example, the randm part u, is als time dependent, and thus predictin is mre difficult. When the effect f time is embdied in u., we are led t a "stchastic prcess" whse characteristic prperties are described by the underlying prbabilistic structure. In these cases, fr example, there are nt regular peridic cycles but mre r less irregular fluctuatins that have statistical prperties f variability. A prcess whse prbability structure des nt change with time is called statinary. In Sectin 5 we are mainly interested in prcesses that are statinary r almst statinary r such that at least the prbability aspect (as distinguished frm a deterministic mean value functin) is rughly statinary. T illustrate these ideas, let us cnsider an autregressive prcess f rder ne, which is described by the relatinship y t = py t-l + u t ' t = > 2 "--» where the y 's are bserved values f a randm variable, and the u 's are sme unbserved randm variables, called innvatins. The innvatin u t is assumed independent f y.,»y+p" f r a^ values f t.
2- The distributin f y, and y 2 is given by the distributin f y, and py.+u^, and similarly the distributin f y,, y 2, and y 3 is given by the distributin f y,, py-, + u 2, and p(py.+u 2 )+u 3. Thus y- depends n y 2, which in turn depends upn y«, and s n. If p <, then the further apart the y's, the less they are related. An innvatin u? is absrbed int y,, y»,..., and thus the randmness perpetuates in time. We therefre say that the effect f time is embdied in the u.'s. The abve prcess is pictrially described in Figure 5.. In Sectin 5. we discuss briefly sme basic prperties f stchastic prcesses and intrduce sme ntins which are used subsequently. 5. Statinary Stchastic Prcesses The sequence f T bservatins which cnstitute an bserved time series may ften be cnsidered as a sample at T cnsecutive equally spaced time pints f a much lnger sequence f randm variables. It is cnvenient t treat this lnger sequence as infinite, extending indefinitely int the future, and pssibly ging indefinitely int the past. Such a sequence f randm variables y,, y?,..., r..., -y_» ~y_l' yn' Yi» y»"-» S knwn as a stchastic prcess with a discrete time parameter. An bjective f statistical inference may be t determine the prbability structure f the lnger infinite sequence«in a stchastic prcess thse variables that are clse tgether in time generally behave mre similarly than thse that are far apart
Time t Figure 5.. An illustratin f the structure f an autregressive prcess f rder.
in time. Usually sme simplificatins are impsed n the prbability structure f the larger series, with the result that the finite set f bservatins has implicatins fr the infinite sequence. ne simplifying prperty is that f statinarity, behind which is the subjective idea that the behavir f a set f randm variables at ne pint in time is prbabilistically the same as the behavir f a set at anther pint in time. Thus fr example, if the underlying prbability structure is assumed t be Gaussian (nrmal) and statinary, then there is ne mean, ne variance, and an infinite number f cvariances. We are interested in finding ut what infrmatin abut these can be gleaned frm a finite number f bservatins. A stchastic prcess y(t) f a cntinuus time parameter t can be defined fr t > 0 r -c < t <. A sample frm such a prcess can cnsist f bservatins n the prcess at a finite number f time pints, r it can cnsist f a cntinuus bservatin n the prcess ver an interval f time. Fr example, the sample culd be a sequence f cnsecutive hurly readings f the temperature at sme lcatin, r it might be a graph f a cntinuus reading. ften a stchastic prcess with a discrete time parameter can be thught f as a sampling at equally spaced time pints f a stchastic prcess f a cntinuus time parameter. A discrete time parameter stchastic prcess is said t be statinary, r strictly statinary, if the distributin f y +,..,y. is the same as the distributin f y t +. y. +. fr every finite set f integers {t,,...,t } and fr every integer t.
We shall dente the mean r the first rder mment y. by m(t), and the cvariance r the secnd rder mment g(y. -m(t))(y -m(s)) = Cv(y t,y s ) by a(t,s). The sequence m(t) is arbitrary, but the secnd rder mment a(t,s) = a(s,t) fr every pair s,t, and the matrix [(t.,t.)] t i,j = l,...,n, must be psitive semidefinite fr every n. If the first rder mments exist, then statinarity implies that (5.) ey s = ey t+s 5 s,t =...,-l,0,+l,..., r that m(s) = m(s+t) = m, say, fr all s and t. Statinarity als implies that fr all t>0 (y,,y. ) has the same distributin as (y. +., y. +.), s that if the secnd mments exist, then Cv(y t, y t ) = a(t,t 2 ) = Cv(y t +t,y t +t ) = ait^+t, t 2 +t). If we set t = -t 9, then (5.2) a(t,t 2 ) = a(t -t 2, 0) = a^-tg), say Thus fr a statinary prcess the cvariance between any tw vari- ables y. and y. depends upn s, their distance apart in time. The functin a(s) as a functin f s, is called the cvariance func- tin r the autcvariance functin, and the functin f s C V(y t' y t+s }. g(s) _ a(s) /Var(y t )/Var(y t+s) v^tjj" ^JßJ CT(0) is called the crrelatin functin r the autcrrelatin functin.
-6- A stchastic prcess is said t be statinary in the wide sense r weakly statinary if the mean functin and the cvariance functin exist and satisfy (5.) and (5.2). In the case f the nrmal distributin, weakly statinary implies stricly statinary and vice versa. In the general case, strictly statinary implies weakly statinary, if the secnd rder mments exist. 5.. Examples f Statinary Stchastic Prcesses Example : Suppse that the y.'s are independent and identically distributed with (5.3) ey t = m, and Var(y t ) = a ; then (5.4) a(t,s) = a 2, S = t, = 0, s^t. This prcess is strictly statinary; hwever, if we drp the requirement f identical distributins, but retain (5.3) and (5.4), then the resulting prcess is statinary in the wide sense. Example 2. Suppse that the y.'s are identically equal t a randm variable y with 2 2 y t = m and Var y. = a(t,t) = a Then, this prcess is strictly statinary.
-7- fllws: Example 3: Define a sequence f randm variables {y.} as (5.5) y. = I (A. cs X.t + B. sin X.t), t-...,-l,0,+l,... where the X.'s are cnstants such that 0<x. <TT, and A,,...,A. B,,...,B are 2q randm variables such that CAj = W. = 0, j=l q, A j = B j = a j ' =i»...q, A i A j = B i B j = ' ^J" ' i,j ' = q ' and SAiB. = 0, i.j=l q Then ey t =, and q q ty+y. = I I (A. cs x.t+b. sin x,t)(a, cs x.s+b. sin x.s) j q 2 2 = 5! [SA^ cs x^ cs ^i s + B^ s i n x n-t sin ^_ij J J J J M] J q = y a.[cs x.t cs x.s + sin x.t sin x.s] jij_ J J J J J q = I a. 2 cs X.(t-s). j=l J J
Since the cvariance f (y f,y ) depends nly n (t-s), the distance between the tw bservatins, and since ey. = 0 fr all t, the sequence {y t > is statinary in the wide sense. If, hwever, the A.'s and the B.'s are als nrmally distributed, then the y t 's will als be nrmally distributed, and then the prcess will be statinary in the strict sense. The pint f this example is that every weakly statinary prcess can be apprximated by a linear cmbinatin f the type indicated by (5.5). Example 4: Let...,v_,, v Q, v.,... be a sequence f indepen- dent and identically distributed randm variables, and let a*, a,,..., a, be q+ cefficients. Then (5.5) y t = a Q v t + ^^t_i +... + a q v t-a» t=...,-l, 0,,..., 2 is a statinary stchastic prcess. If v. = y, and Var v. = a, then y t = Y («Q +a^ +... +a ) and 2 Cv(y t, y t+s ) = a (a Q a s +... a q_ s a q )» S=0,...,q, = 0, S =q+l,..., and s {y.} is weakly statinary. Thus, fr {y.} t be weakly statinary, all we need is that the v.'s have the same mean, the same variance, and that they be uncrrelated. The prcess (5.6) is knwn as a finite mving average.
The infinite mving average (5.7) y t = I v t s=0 s z s means that the randm variable y., when it exists, is such that (5.8) me(y t - I av ) =0 n-s- s=0 A sufficient cnditin fr the existence f y t is that the v t 's be uncrrelated with a cmmn mean (=0) and variance, and (5.9) s=0 s see Andersn (97), p.377. When (5.8) hlds, the infinite sum a v. s=0 s t_s cnverge in the mean r in the quadratic mean. is said t 6. Basic Ntins f Multivariate Nrmal Distributins Tw randm variables X and Y with means u x and u y and variances a 2 and a 2 x, respectively, are said t have a bivariate y nrmal distributin, r a bivariate Gaussian distributin, if their jint density functin is given by f(x,y)= * "xy a a 2ir/l-p' x y exp^ 2 <Ky' ( x-u y X)2 y-u v x-u y-y +( y_)2. 2 ( x )( y_j x G xy y G x ü y
-0- e(x-y x )(Y-u ) then p = ^ is the crrelatin between X and Y ; xy J a a x y -<» < X <, - <y< # The marginal density f X is given by f(x) = - exi-i( -) t, -» <x<-. /27 a ' I 2 a x i A This is the nrmal density functin, which we will hencefrth dente by n(x u x, a x ). Similarly, the density functin f Y is als nrmal, n(y 'v V ' We can shw (Andersn (984), p.37), that f(x y*), the cnditinal density functin f X, given Y=y* is als nrmal, but with a mean CT x 2 2 y x + p xy ~ ( y *~ y and variance y) ' CT x( -p Xy) That s ' f(x y*) = n(x M x+pxy ^ (y*-u y ), a2(l-p2 y) ). Thus, the variance f X given that Y=y* des nt depend n y*, and its mean is a linear functin f y*. The mean value f a variate in a cnditinal distributin, when regarded as a functin f the fixed variate, is called a regressin. Thus, the regressin f X in the situatin abve is a u +p (y*-u ). ^x H KJ xy a ^y' The trivariate nrmal distributin f three randm variables X,Y, and Z is defined in a manner akin t the bivariate nrmal distributin,
- 2 2 2 nce the means y, y, and y, the variances a, a, and a, x y z x y z respectively, and the crrelatins between their pairs p, p, and p, are specified. Let f(x,y,z) dente the jint density functin f this trivariate nrmal distributin; let f(x,y z) dente the jint density functin f X and Y cnditinal n Z = z. Then f(x, y z ) = U^, f(z) where f(z)>0 is the marginal density functin f Z, which is again nrmal. A prperty f the nrmal distributins is that f(x,y z) is a bivariate nrmal density (Andersn (984), p.37). Let f(x z) and f(y z) dente the marginal densities f X and Y cnditinal n z, respectively; these densities can be btained via f(x,y z). Let fi(x z) and e(y z) dente the expected values f X and Y cnditinal n z, respectively. Frm ur previus discussins n the bivariate case, we recall that f(x z) and f(y z) are als nrmal, and that e(x z) and e(y z) can be written as (X z) = a + 3z, and (Y z) = Y +<5z. The crrelatin between X and Y cnditinal n z, dented by p is called the partial crrelatin between X and Y when Z is held cnstant. Thus, we have - e(x-(g+ßz))e(y-( Y +6z)) p xyz - ~~._.. - /e(x-(a+ez)) 2 e(y-( Y +6z)) 2
2- Small values f P XV. Z imply that there is little relatinship between X and Y that is nt explained by Z. We can als verify [Andersn (984), p.4) that p xy-z _ p xy" p xz p yz /l-p xz /- Pyz T discuss the idea f the "multiple crrelatin" between X and the pair (Y,Z), let us dente by e(x y,z) the expected value f X cnditinal upn Y=y, and Z = z. Again, frm ur discussin f the bivariate case, we nte that e(x y,z) can be written as e(x y,z) = a + ey +YZ where a, e, and Y are cnstants. Nw let us cnsider the crrelatin between X and an arbitrary linear cmbinatin f Y and Z, say by+cz, where b and c are arbitrary cnstants. Then, the multiple crrelatin between X and (Y,Z), say R is R 2 = max[crrelatin(x, (by+cz))] 2. b,c It turns ut that the values f b and c are e and Y respectively. Thus, the multiple crrelatin is the crrelatin between X and Y+YZ 7. Estimatin f the Crrelatin Functin ne f the first steps in analyzing a time series is t decide whether the bservatins y-,, y?,...,y T are frm a prcess f
3- independent randm variables r frm ne in which the successive variables are crrelated. If the prcess is assumed statinary, then r(h), an estimate f the crrelatin functin, enables us t infer the nature f the jint distributin that generates the T bservatins. T see this, cnsider a pair f randm variables Y. and Y. +., separated by sme lag k, where k=l,2,.... The nature f their jint prbability distributin can be inferred by pltting a "scatter diagram" using the pair f values y. and y.., fr t=l,2,...,t-k. In Figure 7. we shw a scatter diagram fr Y. and Y t+^ ; this diagram indicates that a large value f Y. tends t lead us t a large value f Y. +k, and vice versa. When this happens, we say that Y. and Y. +. are psitively crrelated. In Figure 7.2 the scatter diagram shws that a large value f Y. leads us t a small value f Y.. and vice versa; in this case, we say that Y. and Y. +. are negatively crrelated. A key requirement underlying ur ability t plt and interpret the scatter diagram is the assumptin f statinarity. Because f this assumptin the jint distributin f Y. and Y. +. is the same as the jint distributin f any ther pair f randm variables separated by a lag f k, say Y. and Y. +., fr sme s/0. A frmal way f describing the impressins cnveyed by a scatter plt is via an estimate f the crrelatin functin; this estimate is als knwn as the serial crrelatin. If the bservatins y^y?.--.» y T are assumed t be generated by a prcess with mean 0, then r*(l), the first rder serial crrelatin cefficient is defined as
4- Yfk Psitive Crrelatin * Y. Figure 7.. Scatter plt f Y. and Y. +. shwing a psitive crrelatin between the variables, Yf Negative Crrelatin * Y, Figure 7.2. Scatter plt f Y t and Y t+k shwing a negative crrelatin between the variables.
5- T-l (7.) r*(l) - t= I y t y t+l I y\ t=i r If the mean f the prcess is nt knwn, (7.) is mdified by replacing y. and y._, by the deviatin f these frm the sample T mean y, where y = y./t. Thus we have z t=l (7.2) r(l) = I (y t -y)(y t+l -y)/ I (y f -y) 2. z z L r t=l t=l Higher rder serial crrelatins are similarly defined; fr example, r*(h), the h-th rder serial crrelatin is T-h L y t y t + h (7.3) r*(h) = t= T? t=i z r in analgy with (7.2) it is (7.4) r(h) = T-h Uy t -y)(y t+h -y) t=i z
-6-8. Autregressive Prcesses ne f the simplest, and perhaps the mst useful, stchastic prcess which is used t mdel a time series is the autregressive prcess. A sequence f randm variables y,,y?,... is said t be an autregressive prcess f rder p, abbreviated as AR(p), if fr sme cnstant p and integer p (8.) (y t -y) +3 (y t. -y) + -...+ ßp(y t -p- y) = u t ' t = P + >P +2 >---» with u,, u 2"*" being independent and identically distributed with mean 0 and variance a, and u. independent f y._,, y. 2».... We shall set p = 0 in the fllwing discussin. The randm variable u. is called an innvatin r a disturbance. We shall refer t the sequence {u.} as an innvatin prcess. It is cnvenient t generalize (8.) t a dubly infinite sequence...» y_i> yr>j y-,,..., resulting in a dubly infinite sequence...,u_-,, UQ,U,,.... Such prcesses are als knwn as autregressive prcesses. k def If we use the freward lag peratr p, where p u. = u t+. fr any integer k, then (8.) can als be written as (8.2) (p P +3 P P_ +...+3 p P )y t _ p = u t. Since Ay t =y t+ - y t = R/t" y t = ^^yt ' we have the result that A = p- ; recall that A is the freward difference peratr intrduced in Sectin 3.3. Thus we may say that the peratr acting n y. can als be written as a plynmial in A f degree p. If u-p (3 0, then the left hand side f (8.2) can be written as a linear cmbinatin f y t _, Ay t _, A y t _,...» AP y t _ p and is therefre called a stchastic difference equatin f degree.
-7- Unless therwise stated (see fr instance Sectin 8.9), we shall assume that the stchastic prcess described by (8.2) is statinary. In Sectin 8. we shall determine the cnditins under which u f is independent f y,,, y+."'" ' The mdel (8.) can be used t generate ther prcesses. Fr example, shuld we want t incrprate the effect f a trend in (8.), then we add t the left hand side f (8.) the term y.z.., where the z it 's are knwn functins f time; this matter is discussed further in Sectin 8.7. Autregressive prcesses were suggested by Yule (927), and were applied by him t study sunspt data. Gilbert Walker (93) extended the thery and applied it t atmstpheric data. In what fllws we shall study the structure f autregressive prcesses, and address the related questins f inference and predictin. 8. Representatin as an Infinite Mving Average If we inspect (8.), we see that y. is expressed as a linear cmbinatin f the previus y.'s and u.. We shall nw study the cnditins under which y. can be written as an infinite linear cmbinatin f u. and the earlier u 's. T see the idea, we cnsider an AR() prcess y t = py t-i + u t * and nte that since y., = py t _ 2 + u t,, we have y t = u t + pu t-l + p2y t-2 '
-8- Successive substitutin f the type indicated abve leads us t wri te (3.3) y t = u t + PU t _ + P 2 u t _ 2 +... + P S u t _ s + P S+ y t _(s+l) s that (8.4) y t -(u t + PU t _ +...+P u t _ s ) = P y t. (s+) The difference between y. and a linear cmbinatin f the s+ (s+) u 's is therefre P ^t-fs+l} ' anc * t ' " s becmes sma n when P < and s is large. In particular (8.5) e[y t -(u t + P u t. +... + P s u t _ s )] 2 = p 2(s+) eyt (s+) will nt depend n t, if we assume that {y.} is a dubly infinite statinary prcess. As s increases, (8.5) will g t 0, and s we can write y t = z I r=0 pr u u. t- c r and say that the infinite sum n the right f the abve equatin cnverges in the mean t y t. (See Sectin 5.7.) Let us nw cnsider an AR(p), P 4 3^-r = U t ' 3 0 = l s that y t = U t " ß l y t-l " ß 2 y t-2"---" ß P y t-p '
9- Replacement f t by t- yields h-i - u t-r Vt-2 - HH-z - " Vt-p-i which upn substitutin gives y t = u t -ß (u t _ -3 y t _ 2-0 2 y t _ 3 -... - y^p^-ß^t-z" "Vt-p = u t - B i u t-i- (B 2- B i, yt-z- + 0 ivt-i-p Cntinuing in the abve manner s times, we arrive at (8.6) y t = V6* U> +... + <u t _ s+ a sl y t _ s _ + a s2 y t _ s _ 2 +... +^t_ s _ p We nte that each substitutin leaves us with p cnsecutive y 's n the right-hand side f the abve. Since y t _ s _i = u t_s-l " g l y t-s-2 " "Yt-s-p-l- we have y t = V 5 l u t- + ' + Vt-s + a sl (u t-s-rt y t-s-2 '' -Vt-s-p-l J + a szh-s-2 + -" + a sp y t-s-p - u t + Vt- + * + 6 s u t-s + a slvs-l + (a s2 " a sl ß l )y t-s-2 +... + (a* p - a*ibp-l)yt-s-p " 4 Vt-S-p- Thus ö s+l = a sl ' a s+l,j = (a s,j+l- a sl ß j ) ' j = P- ' * * n a,,, = -a,p s+l,p sip
20- is a set f recursin relatinships fr the cefficients. Cntinuatin * f this prcedure leads us t write, fr 5 Q =, (8.7) y = I 6* u. t i=0 l t i if the infinite sum n the right-hand side f (8.7) cnverges in the mean t y.. We shall next see the cnditins fr this cnvergence. 8.. Cnditins fr Cnvergence in the Mean f Autreqressive Prcesses The material f Sectin 8. can be frmalized by using the backward lag peratr z, where sy. = y.,, and writing the prcess (8.) as P I 3 r y t = u. r=0 r z z Then, frmally we can write ur AR(p) prcess as where P y t = ( i (v/r u., z r=0 r z ( I e/r = I 6 / j r=0 r r=0 r the 6 's are the cefficients in the equality
-2- (8.8) ( l ßzT = I 6z r r=0 r r=0 r n the basis that the abve equality can be s written meaningfully. It can be verified (Andersn (97), p.69) that the 6 's f (8.8) are indeed the same as the ö 's f (8.7), which we recall * were btained by successive substitutin; thus we write 6=6. In rder t see the cnditins under which it is meaningful t write (8.8), we cnsider (8.9) ß 0 x p + 3 x p- +... + ß x the assciated plynmial equatin f the stchastic difference equa tin (8.) (ur AR(p) prcess). Fr 3 D^0, let x,,...,x be the p rts f (8.9). If x- j <, fr i = l,...,p, then it is clear that z-, 7 z -, the rts P f P I 3/ = 0, r r=0 are such that z. =l/x. and that z. >. Nw, fr any z such that z < min z-, the series (8.0) I = -i = n I (f) = I 6/ r!. {.i, --0 ^ r=0 r^ 3^ i= *»
-22- cnverges abslutely. Thus we see that when x.,...,x, the rts f the assciated plynmial equatin f an AR(p) prcess, are less P than in abslute value, we can write ( ß z r ) = I 5 z r. r=0 r r=0 r T argue cnvergence in the mean f the AR(p) prcess, we cn- P sider the expressin ( ß z r )~ r=0 r hand divisin l+ß,z h...+ß n z P l ß zfß 2 z 2 +...+ß z p l+ß z+ß 2 z 2 +..-+ß D z P r -L r and nte that by a frmal lng = - ß,z (ß 2-ß 2 )z 2 +...+(ßp-ß ß p. )z p -ß ß p z P+ +ßjZ+ßgZ +...+ß z p If we cntinue in the abve manner, we see that 7 S+ X X, S+ P a,z +...+a Z = l+6,z*-6 0 Z,:: +...+6 Z + ^_ l+ßjz +.-.+ß Z P l S l+ßjz +.^+ß Z P where the 6 's and the ct -' s satisfy the same recurrence relatin- ships r as the 6 r 's and the a si.'s f Sectin 8.. Thus 6=6 r r and a. = a.. (See 8.6.) In view f (8.0), we nw see that a-jz 7 S+ 4- +...+a 4. z 7 S + P P - must cnl+ß zf...+ßzp verge t 0 fr z <min z., and in particular fr z =. This i i implies that the a. -> 0 (as s-> ) fr each i. Thus, if {y t > is
23- a statinary prcess 3 (y t~j 0 W/ = (a sl y t-s-l + "- +a sp y t-s-p )2 will nt depend n t and will cnverge t 0 as s -* We therefre have y t = J '- u t-r in the sense f cnvergence in the mean. We have prved P Therem 8.; If the rts f the plynmial equatin &J(. r=0 r = 0 P assciated with a statinary AR(p) prcess Y 3 y. = u. are r=0 r z ~ r z less than in abslute value, then y, can be written as an infinite linear cmbinatin f u., u.,, u.p»...». Nte that whenever y. can be written as an infinite linear cmbi- natin f u., u.,,..., y f will be independent f the future innva- tins u. +, u. +2...s ; this fllws frm ur assumptin that the sequence f innvatins {u,} is mutually independent. We thus have as a crllary t Therem 8. Crllary 8.2; If the rts f the plynmial equatin assciated with a statinary AR(p) prcess are less than in abslute value, y t is independent f u t+,, u. +2,...,.
-24-8.2 Evaluatin f the Cefficients 6 and their Behavir r Suppse that the rts f the assciated plynmial equatin P I 3 x p ~ r = 0, are less than in abslute value. Then, by r=0 r Therem 8., we can write y t = rl Q Vt-r > where the 6 's are t be viewed as weights assciated with the present and past innvatins u., u._,,.... ur gal is t determine a prcedure by which the & 's can be expressed in terms f the knwn 3 's, and als t see if there is any discernable pattern in the 6 's. Such a pattern will enable us t interpret the behavir f ur sequence {y t >. Frm (8.0) we nte that since _ V * S P r=0 n r I 3 z r r=0 r ' P P P ( I 3 zt I e.z s = ( I V P) l r=0 r s=0 s r=0 r s=0 ß s zs = ' r that > l w r+s = i r=0 s=0 s r Replacing r by (t-s) and by suitable re-arrangements, we have P- t - P I ( I 3 s S t _s )zt + H I 3 s 5 t _ s )z t =, t=0 s=0 s z s t=p s=0 s z s which is an identity in z (the series cnverging abslutely fr z <). An inspectin f the abve reveals that the cefficient f
-25- z n the left hand side is and the cefficients f the ther pwers f z are zer; thus we have the fllwing set f relatinships between the 6's and the ß's : W = 8 Q = l (8.) Vi + 3 i ö = 6 i + ß i =0 ' { 3 Vi + "- + Vi 5 = 0 ' and (8.2) Vt + - + Vt-P = 0 t = p, p+,...,. We nte that (8.2) is a hmgeneus difference equatin which crrespnds t the (stchastic) difference equatin that describes the AR(p) prcess Vt + ß l y t-l + "' + Vt-p = U t' ß n = l J 0 If the rts f (8.9), the plynmial equatin assciated with an AR(p) prcess are distinct, then the general slutin f (8.2) is f the frm P (8.3) V = I k i x r = 0,,..., wh ere k,,...,k are cefficients p
26- If a rt x. is real, then the cefficient k. is als real. If a pair f rts x. and x. +, are cnjugate cmplex, then k. and k. +, are als cnjugate cmplex and k.xc + k- +,xr +. is real, r = 0, Equatins (8.) give us the bundary cnditins fr slving (8.2). The p equatins (8.) enable us t determine the p cnstants k,,...,k by substituting (8.3) in (8.). The abve material can be better appreciated via sme special cases; these are discussed belw. 8.2. Special Cases Describing the Evaluatin and Behavir f ys We shall cnsider here tw examples, an autregressive prcess f rder and an autregressive prcess f rder 2. is An Autregressive Prcess f rder Suppse that in (8.) p = l (and y=0), s that an AR() prcess ß 0 y t +f3 l y t-l = u t ' fr t = 2 > 3 "-- The assciated plynmial equatin fr the abve prcess is BQX + S^ = 0, and s with 3 0 =» x =-ß, is the nly rt.
27- Frm (8.) we have 6«= and 6, =-ß,, s that the cefficient k, in (8.3) is. Thus, fr ur AR() prcess the cefficients <5 are such that (8.4) ö r = k i x i = (" 0 i) r ' Nw, if we assume that the prcess is statinary, then in rder t be able t write y. as an infinite linear cmbinatin f u., u. _,,..., we need t have, by Therem 8., x < r equivalently 3-, <. Thus, when ß. <, we can write (8.5) y f = I 5 u.. z r=0 We nte frm (8.4), that the weights s expnentially decay in r when ß, <. The decay is smth if 3.. < 0, and the decay alternates in sign if 3, >0. This behavir f the 6 's implies that in (8.5) the remte innvatins receive smaller weights than the mre recent nes. Such results are useful fr explaining the behavir f the series y., y._,,..., and als interpreting frecasts in autregressive prcesses r rder. An Autregressive Prcess f rder 2 prcess is Nw suppse that in (8.2) p = 2 (and u=0), s that an AR(2) ß 0 y t + ß l y t-l + ß 2 y t-2 = u t» t = 3,4,....
-28- With 3Q= the assciated plynmial equatin fr ur AR(2) prcess becmes 2 0 x + gjx + ß 2 x = 0. If x, and Xp are the rts f the abve equatin, then x. = (-3-, A\ - 4ß 2 )/2, i =,2. If the rts x, and x 2 are real and distinct, that is ß, >4ß 2 then (8.) and (8.2) give = M? + k 2 x 2 = k. + k 2 and k^ + k 2 x 2 = -ß. = x,+x 2. The slutin is x i _x? k, = and k 9 = - - Then xr+l_ x r+l (8.6) 6 r = ^ -x 2 ' ^ = 0,,2,. If we assume that ur AR(2) prcess is statinary, then in rder t be able t write y, in the frm (8.5), that is, as an infinite linear cmbinatin f u t, u._,,..., we need t have (by Therem 8.) x. <, i =,2. This in turn implies that the cefficients ß, and ß 2 will have t satisfy the fllwing cnditins: ß + ß 2 > -, (8.7) g _ 3 2 <, and - < 3 2 <.
-29- Th e abve cnditins define a triangualr regin, shwn in Figure 8., in which the cefficients ß, and ß 2 must lie^ als see Bx and Jenkins, (976), p.59. When x. <, i =,2, and x and x 2 are real, that is, when ß. and ß 2 li e utside the parablic regin f Figure 8., then frm (8.6) it is clear that the weights 5 are a linear cmbinatin f r+ r+ tw expnentially decaying functins f r, x, and x 2 When x. <, i =,2, and when x, and x? are cmplex, that 2 is ß, < 4ß 2 s that ß, and ß 2 lie in the parablic regin f i ft i ft Figure 8., x» and x? may be written as x., = ae and x 2 = ae, where i = /^T ; since x, < and xj <, a <. Thus s that J 6 " k, = --I ^r and k = " e e -e e -e (8.8) 6 r = k xj + k^ = J 5 ^JL_ e - e r sin(8(r+l)) " a sin 0 ft ' since e = cs e + i sin e. Thus S is a damped sine functin f r, whse nature is illus- trated in Figure 8.2. Such a damped sinusidal behavir f the weights ffers an explanatin f an scillatry pattern f the y.'s ften bserved in therwise nnperidic statinary time series. (Als see Sectin 8.6.)
-30- T+ F - * Figure 8.. Regin defining admissible values f 3, and 3 2
3- Figure 8.2. Behavir f the weights 6 as a functin f r, fr an autregressive prcess f rder 2 whse assciated plynmial equatin has cmplex rts.
32- In cnclusin, we nte that fr a statinary autregressive prcess f rder 2, the remte innvatins in a (8.5) type representatin f the series receive a smaller weight than the mre recent nes, regardless f whether the rts f the assciated plynmial equatin are real r cmplex. The nature f the rts determines whether the weights decay expnentially r sinusidally. 8.3 The Cvariance Functin f an Autregressive Prcess If the jint distributins f the y,'s are nrmal, then the prcess is cmpletely determined by its first and secnd rder mments, 2 y t, y t, and Sy+yt+s» s = l»2, If the jint distributins are nt nrmal, the abve mments still give us sme infrmatin abut the prcess. Fr example, &y.y f+ //&yz&y^, the crrelatin between y t and y t+. (assuming that y. =0 fr all t), is a measure f the relatin- ship between the tw variables y. and y fr t = l,2,... If the prcess is statinary, then all the variances are the same, and the cvariances depend nly n the difference between the tw indices. Thus ey t y t+s = a^ =cr (~ s ) ' s =» _ '» + "" Recall that a(s) is als called the autcvariance functin and that a(s)/a(0) is als called the autcrrelatin functin; it will be dented by p(s) and abbreviated as ACF. We shall nw lk at the prperties f the cvariance functin a(s).
33- If we replace t by t-s in y. = l öu and multiply it by n M ** H q=0 l r=0 ß r y f r = u., we have r ^ r r (8-9) rl 0 VtVt-s q i 0 Vt-s-A Nw ey t _ r y t _ s = a(s-r), eu 2 = a 2, fiu^ = 0, t^s, and s the expected value f (8.9) satisfies the fllwing equatins: P (8.20) I ß a(s-r) = a 2, s=0 r r=0 (8.2) ß a(s-r) = 0, s = l,2,... r r=0 The abve equatins are knwn as the Yule-Walker equatins; these will be discussed further in Sectin 8.4. Frm (8.2) we bserve that the sequence a(l-p), a(2-p),..., a(0), a(l),... satisfies a hmgeneus difference equatin, which is the same as the hmgeneus difference equatin (8.2). Thus, if x,, P...,x, the rts f the plynmial equatin ß x^~ = 0, are p r=0 r distinct and ß f 0, the slutin t (8.2) is f the frm P (8.22) <r(h) = I c.xj, h = l-p, 2-p 0, i=l where c,,...,c are cefficients,
34- There are p- bundary cnditins f the frm cr(h) = a(-h), h = l,...,p-l, and the ther bundary cnditin is given by (8.20) with a(-p) replaced by a(p). Thus the behavir f the autcvariance functin f an AR(p) prcess is determined by the general nature f (8.22). We study this by cnsidering sme special cases. 8.3. Special Cases Describing the Behavir f the Autcvariance Functin f an Autregressive Prcess Fllwing Sectin 8.2., we cnsider here an autregressive prcess f rder and an autregressive prcess f rder 2. An Autregressive Prcess f rder Suppse that in (8.) p = l (and y=0), s that y t + ß l y t-l = u t ' t = 2,3,...,. The assciated plynmial equatin x+ß-x = 0 has ne rt x, =-ß,. The general slutin is (frm (8.22)) a(h)=c,(-ß,), h=0,l,.... Frm (8.20) we have a 2 = a(0) + 8^(). = CjCl + ß (-ß )] =C [ - ß 2 ].
35-2 2 Hence c. = a /(-ß.), s that a(h) = (- ßl ) h a 2 /(l-ß2) s h=0,i» J Frm p(h) = a(h)/(0), the autcrrelatin functin is (8.23) P (h) = (-3J h, h=0,l If 3-, <, then we have the imprtant useful result that the theretical autcrrelatin functin f an autregressive prcess f rder decays expnentially in the lag h. The decay is smth if 3, < 0, and it alternates in sign if 3-, > 0. In Figure 8.3, we illustrate this behavir f p(h) fr nnnegative values f h. We als remark that the behavir f p(h) is analgus t the behavir f the weights S discussed in Sectin 8.2. - see (8.4) and Figure 8.2. An Autregressive Prcess f rder 2 Nw suppse that in (8.) p = 2 (and y=0), s that y t + 3 l y t-l + e 2 y t-2 = u t' t = 3,4,...,. 2 0 The assciated plynmial equatin x + 3-,x + 3X = 0 has the rts x n - = [-ßj ±/3j-43 2 ]/2, i =,2. If the rts x, and x 2 are distinct, then a(h) =c,x!? + c 2 x!], h = -l,0,l,.... Then (8.20) and a(l)=a(-l) can be slved fr c. and c 2, yielding
36- (L )9 ujidiejjdinv < c r- +J (J c 3 M- C I +-> ns t i r - c» t- s- 0) S- T3 5-4-> 4-3 03 0 ^~ c res CD U r +-> s- <u Q. S- a> a > ^~ r 4J t t CD a -C s_ 4-> D-. ai M- S- +-) S- 3 f 'r > c rö JE a> M- (M) u!;d]ajjinv
37-2 x h+ x h+ (8.24) a(h)= 2 (J 2 )? h=q^ (x -x 2 )(l-x x 2 ) -Xj l-x If we require that x.. <, i=l,2, then 3. and 3 2 must lie in the triangular regin described by Figure 8.; that is, they must satisfy the inequalities (8.7). Furthermre, if x, and x 2 are real, that is 3, and 3 2 d nt lie in the parablic regin f Figure 8., s that 3?>43 2, then by (8.24) we have the result that a(h) is a linear cmbinatin f tw expnentially decaying functins f h, x.. and x~. Depending n whether the dminant rt is psitive r negative, a(h) will remain psitive r alternate in sign as it damps ut. This behavir f a(h) as a functin f h>0, is shwn in Figure 8.4. When x. <, T =,2, and x. and x 2 are cmplex, that is, 3 < 43 2, then x, and x 2 can be written as x, = ae ft x 2 = ae", where a <, and nw (8.24) becmes and (8.25) CT(n)=^a h [sine(h + l)-a 2 sin e(h-l)]? (-a )sin e[l-2a cs 2e+a ] a a CQS(eh-(j)) (l-a 2 )sin 0vl-2a 2 cs 2e+a 4 h =0,,..., 2 2 where tan 4> = (-a )ct e/(l+a )
-38- t/> a. z < z - 4 Q Ul > Ü. dududa;nv /IS E t ( T- E 4- E >> E r i- Q. -t-> - E CU 3 +-> M- «3 i CL> CJ E Ul t t r- t S- n3 c/5 > +J - E +-> 0) 3 -E t 3 l~~ ** t r- S- +-> <u CU T3 S- S- CD.E M- S-> t 0) t +->.E t +-> cu s- *-» s- x: a. T t *^^ CD ai s- > r- 0 4- t t t -E <u s- s- E en i- cu t > s- +J t t -E +-> 3 QJ 3 " CQ t 0) c CD S- 3 C7> / (4)0 dduuda;nv
-39- Thus a(h) is a damped csine functin f h ; the behavir f a(h) as a functin f h = 0, ±, ±2,..., is illustrated in Figure 8.5. Since a(h) is a linear cmbinatin f the hth pwers f the rts x, and x?, bth f which are less than in abslute value, a(h) is bunded. We remark that the behavir f a(h) as a functin f h is analgus t the behavir f the weights 6 as a functin f r, discussed in Sectin 8.2. - see (8.6), (8.8), and Figure 8.2. Thus t cnclude, we have the imprtant practical result, that when Si and ß, the parameters f an AR(2) prcess, lie in the triangular regin described by Figure 8., the theretical autcrrelatin functin decays either expnentially r sinusidally. The expnential decay culd be either smth r alternating in sign, depending n the values that e, and ß~ take. In Sectin 8., we shw the behavir f the estimated autcrrelatin functin f sme real life data which we claim can be reasnably well described by autregressive prcesses. Hwever, in rder t be able t use the behavir f the autcrrelatin functin as a means f identifying autregressive prcesses, we need t have sme idea abut the behavir f the estimated autcrrelatin functin f sme knwn autregressive prcesses. This we d next, and als make sme ther cmments which have sme practical implicatins.
-40- ) C\J c (Ö S_ i- a) i- -a t s- > M- 3 l/l'r tc (/) +J 0) ( i 3 «3 cr s- 0) i- Q. 4J r ai tu is s-' M- E ) </> -G t/5 c +-> <u S- i <L> en a> D. S- -a =5 4-> <4-5- c 4-> t /5 t +-> t <a > 4-> i (0 C Q. r CUE cu 3 -c ccit-s u dj
-4-8.3.2 Behavir f the Estimated Autcrrelatin Functin f Sme Simulated Autregressive Prcesses The results f Sectin 8.3. can be generalized in a straightfrward manner t shw that the autcrrelatin functin f autregressive prcesses must decay expnentially r sinusidally. Even thugh this result is true in thery, it is unreasnable t expect such a behavir f the estimated autcrrelatin functin. Such a lack f cnfrmance between the thery and its applicatin is mainly due t the sampling variability in ur estimate f the autcrrelatin functin (see Sectin 7), and is particularly acute when we are dealing with series f shrt lengths, wherein ur estimate f the autcrrelatin functin is based n few bservatins. Thus a gd deal f cautin and insight has t be used in rder t identify the nature f an underlying stchastic prcess by examining the behavir f its estimated autcrrelatin functin. In Table 8. we give r(h), the values f the estimated autcrrelatin functin, h =0,,...,25, based n 250 cmputer generated bservatins frm an AR() prcess y t - -5y t _i = u t, t = 2,3,...,250, with y = u. A plt f r(h) versus h is given in Figure 8.6. Barring the slight aberratins at h=7, 8, 9, 3, 9, and 23, this plt reveals the expnential decay pattern expected f an AR() prcess with ß, < 0, and ß,j <. In Table 8.2 we give r(h), the values f the estimated autcrrelatin functin, h=0,,...,25, based n 250 cmputer generated bservatins frm an AR(2) prcess
-42- <3- X5 «3- X3 E x3 c r-~ «E '. -E >- ' +J > ai c Ln E CU t r- J2 -t-> CT3U5 3 0) «*- 4-> I <C c S- II CÜ >- E i -(-> CD. ta en X5 ^- <X3 CD.a (8 CD S- -E s_ CD +-> t- +-> < =s 2 u a. E </i +-> 3 CD IB - LT3 S_ CÜ a. +-> <T5 E E -^ r < +J T3- t CD C CD t <C f CD -Q -E E 4-3 (0 ft <+- LT3 ft n * CD *3- CTt X3 I-H X3 X3 00 X3 03 c c c II 4- CD CD CD CT. t
-43- ^ C a V) UJ I.0 0.8 0.6 0.4 0.2 Y 0.0 I I I I I I i i» I i I t i i I i i i 4-0 0 5 20 25 LAG h Figure 8.6. A plt f the estimated autcrrelatin functin r(h) versus h, h = 0,,,25,. based n 250 cmputer generated bservatins frm an AR() prcess with 3-, = -.5.
-44-. r IX) «XI «I. i-h LT) i_ 4- </) c - > x: -I- * - -4-> <d" S- (B C CD t» a i- _Q +-> " c -a c =s a) re 4- (->,.. *.. s- a cu cr> r- E +J a) D) ^" a s- i-h S- CD ca cn S- +-> 3 <u.x: i E +J X) -P -r- 3 2 I "C IX) CD CU P ( E E &- t- Q. PTI CU a ^ X)-^- - = a; P ca: A 4- IX) C «. cu. I-H CT) t < "=3- r-l IX) t-h I-H ' IX) r~» 00 X) IX) C3- ' l I t c <X> «tf- IX) i i I ) ix> =3- I-H v I H II t _ ^ x: x: S- %. 4- li CS JC cu x: CU 3 3 CD i cn r _l > _j >
-45- y t",9y t-l + - 4y t-2 = U t ' t = 3,4,...,250, with y =.9y +u 2 and y. = u,. A plt f r(h) versus h is given in Figure 8.7. Since ß,=-.9 and ß 2 = - 4, ßj < 4ß 2, and s the rts f the assciated plynmial equatin are cmplex. ( a = SÄ =.63, Xp x =.45 ±.44i, 9^45 ) Thus the theretical autcrrelatin functin must decay sinusidally; this feature is als revealed by the estimated autcrrelatin functin shwn in Figure 8.7. In Table 8.3 we give r(h), the values f the estimated autcrrelatin functin, fr h=0,,...,25, based n 250 cmputer generated bservatins frm an AR(2) prcess y t + - 5 y t _i " -^-2 = u t ' t = 3,4,...,. A plt f r(h) versus h is given in Figure 8.8. Since ß, =.5, 2 and ß 2 = -.2, ß, > 4ß?, and hence the rts f the assciated plynm- ial equatin are real. These rts being (-.5 ± /.25+.8)/2, it is clear - 5-025 that the dminant rt is negative, its value is : j- = -.763. Thus accrding t the material in Sectin 3.3., the autcrrelatin functin must decay, and alternate in sign as it des s - see Figure 8.4. The estimated autcrrelatin functin f Figure 8.8 reveals this tendency, at least in the earlier stages, up t lag 0 r s. Later n, the estimated autcrrelatin functin des alternate in sign, but des nt decay. We attribute ur reasns fr this t the sampling variability f the estimates f the autcrrelatin at the varius lags.
-46- C J u 4-* < 'S Ixl.0 0.8 0.6 (U Q2 0.0 0.2 \ t]7 ill K I t. I I I I I -^ 0 5 20 25 LAG h -0.4 t Figure 8.7. A plt f the estimated autcrrelatin functin r(h) versus h, h = 0,,...,25, based n 250 cmputer generated bervatins frm an AR(2) prcess with ß, = -.9 and =.4
-47- i t i t i-h <* S- +- I ) r 4 c a; r - rt3»i c,. v E -C - r- S- 03 > E s- a> r c II +-> -Q E ca 3 - < - J +-> T3 E res E c S- 03 J +-> E 03 J IT) CD J 5- S- S_ CU i -P. CJ 3 Q. +J t- -E 3 +-> 03 (J r s -c J L -p c? cu E E +-> s- in T3 Q- J J J re.e-q +J C 4- fl =r L c A E J 03 3 II sz CX> r~~. t-h I i-h i «3 ai i-h L =3- i-h CT) i i c L i i-h -E c i i i-h i-h L i f i-h I I-H L I i-h CTl i-h I-H i ^ I-H CTl -E S- S- 4- C -.E cu JE CU CD 03 _l 03 > 03 _l 03 >
20-48-.0 i "3» i_ k. u *» < «4-* -» v» u 0l8 - \ V \ 06 V ' \ 0A \ < 02 ">^^^ f\t\ 5.. 5 U.U " 0 02 / / 0.A - f / / 0.6 f / 0.8 / ' LAG h Figure 8.8. A plt f the estimated autcrrelatin functin r(h) versus h, h=0,,...,25, based n 250 cmputer generated bservatins frm an AR(2) prcess, with 3-, =.5 and = -.2
-49- Th e behavir f the estimated autcrrelatin functin f sme real life data which we feel can be reasnably well described by autregressive prcesses is shwn at the end f this sectin, in 8.. 8.4 Expressing the Parameters f an Autreqressive Prcess in Terms f its Autcrrelatins The Yule-Walker equatins (8.2) enable us t express the autregressive parameters ß,,...,ß in terms f the autcrrelatins P(S), s=l,...,. T see this, we set s = l,...,p in (8.2), divide thrughut by a(0), and bserve that r P() = -3j_ - ß 2 P(l).. -3 p(p-l), (8.26) p(2) = -ß lp (l) - ß 2..-y(p-2), p(p) = -ß p(p-l)-ß 2 P(p-2) -...- Mp 3. If we dente ß = [3,,...,3 ], p = [p(l),...,p(p)], and V X [J «, P(l) P(P-) P = P() P(P-2) )(p-l) p(p-2) then p = - P3 frm which we have (since Pis psitive definite) (8.27) 3 = - P - p The matrix P is unknwn as the autcrrelatin matrix.
50- Th us, the p autregressive parameters can be expressed in terms f the p autcrrelatins p(l),..., p(p). This feature can be used t estimate ß, using an estimate f P 2 We btain a, the variance f the disturbance, by setting a(-r)=a(r) in the Yule-Walker equatin (8.20) t btain (8.28) ß Q a(0) + 8^() +... + S p a(p) = a 2 8.5 The Partial Autcrrelatin Functin f an Autregressive Prcess In Sectin 8.3 we have shwn that a(h), the autcvariance func- tin f an autregressive prcess f rder p, is infinite in extent. Thus frm {a(h)} it is hard t determine the rder f an autregressive prcess. The partial autcrrelatin functin, t be discussed here, will help us in determing the rder f an autregressive prcess. T be specific, let us cnsider a statinary autregressive prcess f rder p y t = u t - ß i y t-i "Vt-P' t = p+ ' p+2 '--- Recall that in rder t predict y we need cnsider nly the p lagged variables y. i»---»yt_ D» since the ther variables y t _ D _i» y-t-d-2'"" ' iave n0 e^eci n y+
5- The partial autcrrelatin between y. and y._, t be dented by -rr(p) is the crrelatin between y. and y. when the interme- diate p- variables y t _,, y,_~ ^t-d+ are is, TT(P) is the crrelatin between y. and y. " ne^ fixed." That when the interme- diate variables are nt allwed t vary and exert their influence n the relatinship between y. and y.. Clearly, TT(), the partial autcrrelatin between y. and y._., is p(l), the (rdinary) autcrrelatin between y, and y._., whereas TT(0) the partial autcrrelatin between y. and itself is. Thus, by its very nature, since y,., y^n-?""' ^ave n effect n y, the partial autcrrelatin functin f an autregressive prcess f rder p, TT(J') f 0, fr j =0, p, and ir(j)=0, fr j>p. The fact ir(j) vanished fr j > p+, can be used t identify the rder p f an autregressive prcess, prvided that ir(j) can be cmputed. In ur discussin f the partial autcrrelatin functin -n-(p) we had mentined the fact that the intermediate values y._,,...,y t + -, had t be "held fixed". In rder t frmalize this ntin we shall use sme results which are standard in multivariate analysis. Let Y = [y., y._,,...>y t _ D ] dente the vectr f p+ bservatins, and let z dente the variance-cvariance matrix f these p+ bservatins. Suppse that Y has a multivariate nrmal distributin with mean vectr 0 and cvariance matrix z, where
-52- a(0) a(l) a(p) E = a(l) a(0) a(p-l) a(p) a(p-l)... a(0) Let us rearrange the elements f Y, and partitin it int tw cmpnent sub-vectrs Y^ ' = [y t, y t ] and Y^ ' = [y t _, y t _2"-" y^.n+il Let E,,, E 2 2> an d 2 ^e ^e var i ance-cvar i ance matrices f Y^, Y^, and Y^ and Y^ respectively. That is, i n, E 22, and Ej 2 is a partitin f the rearrangement f E. (2) ~ (?) Let y v ' be a particular value taken by the vectr Y v '. Then, it can be shwn [Andersn (984), p.28] that the cnditinal distribu- tin f Y^ ' given qiven y.\r ' is a multivariate nrmal with mean E, 0 E" 2 i^ " and cvariance matrix - E " E 2 E 22 E 2 = s -2 ' say * Thl ' S is a 9 enera ", '~ zatin f the results mentined in Sectin 6. - (2) The vectr i.^l^ y x ' is called the regressin functin f the regressin f Y^ ' n y. The matrix E,, «"" s a 2x2 matrix whse elements are indicated belw: Ell-2 a tt.(t-l),...,(t-p+l) a (t-p)t.(t-l),...,(t-p+l) a t(t-p)-(t-l),...,(t-p+l) a (t-p)(t-p).(t-l),...,(t-p+l) The partial crrelatin between y t and y (t-p+) fixed at y^ is hlding (t-),...,
-53- :(P) a t(t-p)-(t-l),...,(t-p+l) / CT tt.(t-l)...(t-p+l) / a (t-p)(t-p)-(t-l),...,(t-p+l) nte that ir(p) is independent f y. As an example, if Y = (y t> y^, y t _ 2 )\ and if Y^ - (y t» y t _ 2 5' (?) and Y v ' = y.,, then the partial crrelatin between y. and y. 2 TT(2) 5 turns ut t be *(2) = (p(2) - p 2 (l))/(l-p 2 (l)). 8.5. Relatinship between Partial Autcrrelatin and the Last Cefficient f an Autregressive Prcess An interesting relatinship between ir(p.), the partial autcrrelatin f y, and y., and 3, the last cefficient f an autregressive prcess f rder p, can be bserved. This relatinship simplifies ur calculatin f ir(p), since 3 can be easily btained frm the Yule-Walker equatins via equatin (8.28) In rder t see a relatinship between ir(p) and 3, cnsider an AR(2) prcess *t = U t " 3 l y t-l - ß 2^t-2 and slve the resulting Yule-Walker equatins t btain
54- p(l) p(l) P(2) _ P(2)-P 2 () P(D -P 2 (D p(l) 2 2 Hwever [(p(2) - p ())/( -p ())] is indeed the partial autcrrela- tin between y. and y._ 2 ; thus TT(2) = -ß«. In a similar manner, if we cnsider an AR(3) prcess y t = u t - ß l y t-l " 3 2 y t-2 - ß 3 y t-3 and slve the resulting Yule-Walker equatins, we bserve that P(D P(2) P(D P(2) P(D P(D P(D P(D P() P(2) P(3) P(2) P(D which again can be verified as the negative f the partial autcrrelatin between y. and y. -. In general, we bserve [Andersn (97), pages 88 and 222] that fr an autregressive prcess r rder p, ir(p) the partial autcrrelatin between y, J and y, is -ß, where t t-ü p '
55- P(D P(2) P(D P(l) P() P(2) (8.29) ß p = p(p-l) p(p-2) p(p-3) P(l) P(2) p(p) p(p-d P(D P() p(p-2) (p-) p(p-2) -(p-3) It is helpful t remark that the determinant in the denminatr is simply the determinant f the autcrrelatin matrix fr an AR(p) prcess P (Sectin 8.4), whereas the matrix in the numeratr is P with the last clumn replaced by p(l),...,p(p). An expressin fr TT(j) the partial autcrrelatin between y, and y.., can be btained if we write the Yule-Walker equatins fr j, and set TT(j) = g., where s. is given by equatin (8.29); recall that ir(0) =, and that ir(l) = P (). The partial autcrrelatin functin is a plt f -n-(h) versus h, h = l,2 ; the partial autcrrelatin functin is abbreviated as PACF. We estimate TT() by r(l), and estimate TT(j) by ir(j), where ir(j) is btained by replacing the p(«)'s in (8.29) by their estimates r(-)'s.
56-8.5.2 Behavir f the Estimated Partial Autcrrelatin Functin f Sme Simulated Autregressive Prcesses Even thugh the partial autcrrelatin functin f an autregressive prcess f rder p must theretically vanish at lags p+, P+2,..., it is unreasnable t expect such a behavir f the estimated partial autcrrelatin functin. The reasns fr this are analgus t thse given fr the behavir f the estimated autcrrelatin functin - see Sectin 8.3.2. Thus cautin and insight must be used when identifying the rder f an autregressive prcess by examining its estimated partial autcrrelatin functin. In Table 8.4 we give Tr(h), the values f the partial autcrrelatin functin fr h=0,,...,25 based n 250 cmputer generated bservatins frm the AR() prcess y t -.5y t _ = u t, t = 2,3,..., discussed in Secitn 8.3.2. A plt f T?(h) versus h is given in Figure 8.9. Barring sme slight aberratins at a few lags, this plt reveals the behavir that we expect frm the PACF f an AR() prcess, namely that ir(l) must be significantly different frm 0, and that TT(J), must be clse t 0 fr j = 2,3,...,. An examinatin f Figures 8.6 and 8.9 reveals the desired result that fr an AR() prcess, the autcrrelatin functin decays expnentially, and that the partial autcrrelatin functin vanishes after lag.