. Time Series Analysis.. Preliminaries.. Various Tyes of Sochasic Processes..3 Parameers of Univariae and Bivariae Time Series..4 Esimaing Covariance Funcions and Secra
. Time Series Analysis The cenral saisical model used in ime series analysis is sochasic rocesses Definiion A sochasic rocess is an ordered se of random variables, {, } indexed wih an ineger, which usually reresens ime. A ime series is realizaion of a sochasic rocess
. Time Series Analysis Examle: Whie Noise A simles sochasic rocess is a whie noise, which is an infinie sequence of zero mean iid normal random variable A whie noise has no memory,i.e. for any nonzeroτ P( 0 0 0 0.5 0 s + τ > 0 ( x, y dxdy f ( x dx f ( x dx s f s 0 0 f ( x dx > 0 f ( y dy
. Time Series Analysis Proeries of a Sochasic Process A sochasic rocess {, } is said o be saionary if all sochasic roeries are indeenden of index. I follows has he same disribuion funcion for all and s, he arameers of he oin disribuion funcion of and s deend only -s A sochasic rocess is weakly saionary, if he mean E( is indeenden of ime and he second momens E(, s are funcions of -s only The assumion of week saionary is less resricive han ha of saionary and is ofen sufficien for he mehods used in climae research A sochasic rocess is weakly cyclo-saionary, if he mean is a funcion of he ime wihin a deerminisic cycle and he cenral second momens are funcions of -s and he hase of he cycle
. Time Series Analysis Examle of Non-saionary 958-977 ime series of monhly mean amosheric CO concenraion measured a he Mauna Loa Observaory in Hawaii. The ime series can be considered as a suerosiion of a saionary rocess, a linear rend α and an oscillaion wih eriod of monhs
. Time Series Analysis Examle: Random Walk Given a whie noise, is a random walk. is non-saionary in variance:, ( ( ( 0 ( σ E E E Var E E k K If an ensemble of random walks is considered, he cener of graviy will no move, bu he scaer increases coninuously
. Time Series Analysis Ergodiciy Ergodiciy has o be assumed, since saionary or weak saionary alone is no enough o ensure ha he momens of a rocess can be esimaed from a single ime series Definiion (loose A ime series is ergodic, if i varies quickly enough in ime ha increasing amouns of informaion abou rocess arameers can be obained by exending he ime series Examle of a Non-Ergodic Process x a, where a is a realizaion of a random variable A wih mean µ n ime average x a ensemble average E( A µ n
. Time Series Analysis Various Tyes of Sochasic Processes We will concenrae on he auo-regressive (AR rocesses as he mos relevan ye of sochasic rocesses for climae research, and discuss briefly heir relaion o a more general class of shor memory rocesses, he auo-regressive moving average (ARMA rocesses, and he relaion o a class of long memory rocesses, he fracional auoregressive inegraed moving average (ARIMA rocesses.
. Time Series Analysis Definiion: AR-Processes {, } is an AR-rocess of order, if here exis real consans α k, k0,,, wih 0 and a whie noise rocess {, } such ha α α 0 + α k k k + AR-rocesses are imoran, since given any weakly saionary ergodic rocess { }, i is ossible o find an AR-rocess {Y }, ha aroximaes { } arbirarily closely AR-rocesses are oular, since hey reresen discreized ordinary differenial equaions The mean and variance are αo µ E(, α k where ρ (k is he auo-correlaion funcion Y -µ is a rocess wih zero mean Var( α ρ( k Var( k k k
. Time Series Analysis AR(-Process: α - + AR(-Processes ( have only one degree of freedom unable o oscillae A non-zero value of x a ime ends o be damed wih an average daming rae of α er ime se An AR(-rocess wih a negaive coefficien will fli around zero. Such a rocess is considered o be inaroriae for he descriion of a climae series One finds σ z ρ α, Var( ( α Thus, he variance of he rocess is a linear funcion of he variance of he inu whie noise and a non-linear funcion of he arameer α.
. Time Series Analysis AR(-Process: α - +α - + AR(-rocesses ( have only wo degrees of freedom and can oscillae wih one referred frequency The AR(-rocess wih α 0.9 and α -0.8 exhibis quasi-eriodic behavior wih a eriod of abou 6 ime ses The AR(-rocess wih α 0.3 and α 0.3 has behavior comarable o ha of an AR(- rocess wih long memory
. Time Series Analysis Saionariy of AR-Processes AR-rocesses can be non-saionary. An AR( rocess wih α and µ0 is saionary wih resec o he mean bu non-saionary wih resec o he variance, since one has for saring from 0 + + i i i i i Var Var Var E E 4 3 4 ( 4 ( 4 (, ( ( 0 0 0 An AR(-rocess wih AR coefficiens α k, k,, is saionary if and only if all roos of he characerisic olynomial lie ouside he circle y k k k y y ( α The characerisic olynomial has roos, y i, i,, They can be real or aear in comlex conugae airs
. Time Series Analysis Condiion for a saionary AR( rocess ( y α y α y 0, / An AR(-rocess is saionary, if α < Condiion for a saionary AR( rocess ( y α α 0, y y y, α ± α + 4α α An AR(-rocess is saionary, if α +α <, α -α <, α < AR( coefficiens ha saisfy hese condiions lie in he riangle deiced in he figure Region where he characerisic olynomials have wo real soluions α +4α 0 Region where he characerisic olynomials have a air of conugae roos
. Time Series Analysis More abou he roos of he characerisic olynomial: The roos of a characerisic olynomial describe he yical emoral behavior of he corresonding rocess Le y i, i,, be he roos of characerisic olynomial (y. Given a fixed i, se k k, i yi, k, L. subsiue hese values ino he corresonding rocess α k k + k disregard he noise yields Each roo y i idenifies a se of yical iniial condiions ha lead o when he noise is disregarded. Since hese iniial condiions are linearly indeenden, any se of saes ( -,, - can be reresened as a linear combinaion of he iniial saes. In he absence of noise, he fuure evoluion of hese will be + τ β y i τ i Examle I An AR( rocess has only one soluion y /a, + τ τ β α Examle II The AR( rocess wih (a,a (0.3,0.3 has soluions y.39 and y -.39 τ τ + τ βy + β y Examle III The AR( rocess wih (a,a (0.9,-0.9 has soluions y r ex( ± iφ, r., φπ/3 + τ y τ β y r τ τ + β y τ ex( iφτ β y τ * + β y τ
. Time Series Analysis The yical emoral behavior of an AR rocess in he absence of he noise is characerized by damed modes wih or wihou oscillaions The daming is necessary, since he resence of he noise makes he rocess non-saionary (random walk The rae of daming is deermined by he amliude of he roos of he corresonding characerisic olynomial comlex roos lead o oscillaions
. Time Series Analysis Definiion: Moving Average (MA Processes A rocess {, } is said o be a moving average rocess of order q (MA(q, if where. µ. β, L, β are consans such ha β 0 3.{ µ + + βl l l is he mean of he rocess q q ; }is a whie noise rocess q An MA rocess is saionary wih mean µ x and variance q + Var( Var( βl l
. Time Series Analysis Definiion: Auo-regressive Moving Average (ARMA Processes A rocess {, } is said o be an ARMA rocess of order (,q, if where. µ 3.{ µ. α, L, α i α i is he mean of he rocess i and β, L, β are consans such ha α 0, β 0 l ; }is a whie noise rocess + q q β l l q There is a subsanial overla beween he class of MA, AR, and ARMA models Any weakly saionary ergodic rocess can be aroximaed arbirarily closely by any of he hree yes of models The ARMA models can aroximae he behavior of a given weakly saionary ergodic rocess o a secific level of accuracy wih fewer arameers han a ure AR or MA model does.
. Time Series Analysis Backward shif oeraor B B acs on he ime index of he sochasic rocess. I is defined by [ ] and saisfies B ( B AR, MA and ARMA can all formally be wrien in erms of B. Secifically define he AR oeraor and he MA oeraor φ( B αib q i θ ( B + β B i AR, MA and ARMA rocesses are hen formally sochasic rocesses saisfying φ( B φ( B θ ( B θ ( B (AR (MA (ARMA Why B? rovide he ool needed o exlore he connecions beween AR and MA models inroduce oher classes of models
. Time Series Analysis Definiion: Auo-regressive-inegraed Moving Average (ARIMA Processes A rocess {, } is said o be an ARIMA rocess of order (,q,d, if he dh difference of, (-B d, saisfies he ARMA oeraor of order (,q, I.e. φ ( θ d B ( B ( B If /<d</, is called a fracional ARIMA rocess An ARIMA rocess wih a osiive ineger d is generally no saionary, whereas a fracional ARIMA rocess can be saionary Fracional ARIMA rocesses are also known as long-memory rocesses
. Time Series Analysis Parameers of Time Series: The Auo-covariance Funcion Le be a real or comlex-valued saionary rocess wih mean µ. Then * γ ( τ E(( µ ( + τ µ Cov(, + is he auo-covariance funcion of, and he normalized funcion γ ( τ ρ ( τ γ (0 is he auo-correlaion funcion of. The argumen τ is he lag. An auocorrelaion funcion has he roeries ρ( τ ρ( τ, ρ( τ τ The auo-covariance funcion and he auo-correlaion funcion have he same shae, bu differ in heir unis. The former is in unis of, he laer is dimensionless Examle: Auo-covariance funcion of a whie noise, τ 0 ρ( τ 0, oherwise
. Time Series Analysis The Yule-Walker equaions for an AR( rocess If we mulily a zero mean AR( rocess by -τ for τ,,, τ αi i τ + i and ake execaions, we obain a sysem of equaions r r Σ α γ ha are known as he Yule-Walker equaions. The equaion relaes he auo-covariances r γ ( γ (, γ (, L, γ ( a lag τ,, o he rocess arameers r α T ( α, α, L, α T τ and he auo-covariances γ(τ a lags τ0,,- hrough he marix Σ γ (0 γ ( M γ ( γ ( γ (0 M γ ( L L O L γ ( γ ( M γ (0
. Time Series Analysis The Yule-Walker equaions can be used o build an AR model If covariances γ(τ,τ0,, are known (e.g. have been esimaed from daa, he arameers of he AR( rocess can be deermined by solving he Yule-Walker equaions for α,,α. The Yule-Walker equaions can be used o deermine auocovariance funcions If r α is known, he Yule-Walker equaions can be solved for γ(,,γ(, given he variance of he rocess, γ(0. The full auo-covariance funcion can be derived by recursively exending he Yule-Walker equaions. This is done by mulilying he original AR rocess by -τ for τ o obain γ ( τ α kγ ( k τ k
. Time Series Analysis Uniqueness of he AR( Aroximaion o an Arbirary Saionary Process The following heorem is useful when fiing an AR( rocess o an observed ime series Le be a saionary rocess wih auo-correlaion funcion ρ. For each >0 here is a unique AR( rocess wih auo-correlaion funcion ρ such ha ρ ( τ ρ( τ for τ T The arameers α ( α,, L, α, of he aroximaing rocess of order are recursively relaed o hose of he aroximaing rocess of order - by where r α, k α(, k α, α (,( k k, L, α, ρ( k k α α (, k (,( k ρ( k ρ( k saring from α, ρ(.
. Time Series Analysis Auo-covariance and auo-correlaion funcions of some AR( rocesses : The Yule-Walker equaion is α γ (0 γ ( ρ( α, ρ( τ α τ Auo-correlaion funcions of wo AR( rocesses wih α 0.3 (hached and 0.9 (solid : The Yule-Walker equaion is γ (0 γ ( α γ ( γ ( γ (0 α γ ( α ρ( -α α α + α ρ( α Auo-correlaion funcions of wo AR( rocesses wih (α α (0.3,0.3 (hached and (0.9,-0.8 (solid
. Time Series Analysis The General Form of he Auo-correlaion Funcion of an AR( Process The auo-correlaion funcion of a weakly saionary AR( rocess can be exressed as ρ( τ k i a k a y i τ i y τ k + k a k cos( τφ ψ k + k τ rk yk τ where y k,k,,, are he roos of he characerisic olynomial and are eiher real or aear in comlex conugae airs. a k can be derived from he rocess arameers r α. When y k is real (comlex, he corresonding a k is comlex. The weak saionary assumion ensures ha y k > for all k. Thus, each real roo conribues a comonen o he auo-correlaion funcion ha decays exonenially and each air of comlex conugae roos conribues an exonenially damed oscillaion The auo-covariance funcion is a sum of auo-covariance funcions of AR( and AR( rocesses
. Time Series Analysis Parameer of Time Series: Secrum Definiion I (H. von Sorch & F. wiers Le be an ergodic weakly saionary sochasic rocess wih auocovariance funcion γ(τ,τ,-,0,,. The he secrum (or ower secrum Γ of is he Fourier ransform F of he auo-covariance funcion γ, i.e. Γ F{ γ } τ for all ω [ /,/ ]. γ ( τ e πiτω The Fourier ransform F{} is a maing from a se of discree summable series o he se of real funcions defined on he inerval [-/,/]. If s is a summable discree series, is Fourier ransform F{s} is a funcion ha akes, for all F{ s } ω [ /,/ ] s e πiω, he value The Fourier ransform maing is inverible, / πiω s F{ s}( ω e dω /
. Time Series Analysis Proeries of Γ(ω Γ(ω of a real-valued rocess is symmeric: Γ Γ( ω ω [ /,/ ] Γ(ω is coninuous and differeniable for dγ( ω ω 0 0 dω γ(τ can be obained using he inverse Fourier ransform: γ ( τ / / Γ( ω e iπωτ dω Γ(ω describes he disribuion of variance across ime scales Var( (0 Γ γ 7 0 Γ(ω is a linear funcion of he auo-covariance funcion. Tha is, if γ(τa γ (τ+a γ (τ, hen Γ aγ + aγ
. Time Series Analysis Parameer of Time Series: Secrum Definiion II (Koomans, Brockwell & Davis A sochasic rocess is reresened as he inverse Fourier ransform of a random comlex valued funcion (or measure ω ha is defined in he frequency domain e iπωτ ω dω (i.e. a sochasic rocess is a sum of random oscillaions The secrum is defined as he execaion of he squared modulus of he random secral measure ω The auo-covariance funcion is shown o be he inverse Fourier ransform of he secrum
. Time Series Analysis Consider a eriodic weakly saionary rocess where ω /T, i-n,,n, T, is comlex random number saisfying - *. i n n e πω The firs and second momens of he rocess are Thus, is weakly saionary, if,,,n have zero mean is uncorrelaed wih k wih k i n n i k k n n i n n i k e e E e E E e E E ( * ( ( ( ( ( ( ω ω π τ πω τ πω τ πω τ γ µ + + Under he condiion of weak saionary, one has + τ τ τ πω τ γ πω τ τ γ ( lim cos( ( ( ( ( n o i n n E E e E The secrum is a line secrum. I is discree raher han coninuous! A long memory rocess
. Time Series Analysis Definiion I is more suiable for shor memory rocess γ(τ decays sufficienly fas wih increasing, so ha lim τ τ γ ( τ << variance aribues o a coninuous range of frequency, raher han a few discree frequencies, since oherwise we would have a rocess wih a eriodic covariance funcion and hence infinie memory. Definiion II is more suiable for long memory rocesses or rocesses wih disinc oscillaions γ(τ decays no sufficienly fas wih increasing τ, so ha γ(τ is no summable lim τ τ γ ( τ Variance aribues o a few discree frequencies
. Time Series Analysis The Secrum of AR( and MA(q Processes The secrum of an AR( rocess wih rocess arameers {α,,α } and noise variance Var( σ is Γ( ω σ k α πikω ke The secrum of an MA(q rocess wih rocess arameers {β,,β q } and noise variance Var( σ is Γ( ω σ + q πilω β l le
. Time Series Analysis The Secrum of a Whie Noise Process: Γ ( ω σ The Secrum of an AR( Process: σ Γ( ω α e σ + α α cos(πω ( α σ ( α σ α(πω πiω σ + α (πω for (πω for (πω for (πω << ( α << α ( α >> α Power secra of AR( rocesses wih α 0.3(L and α 0.9(R No exremes in he inerior of [0,/], since d Γ( ω α Γ( ω sin(πω 0 dω When α >0, he secrum seak is locaed a frequency ω0. Such rocesses are referred o as red noise rocesses
. Time Series Analysis Ploing Formas Power secra of AR( rocesses wih α 0.3(L and α 0.9(R The same secra loed in log-log forma, ha Emhasizes low-frequency variaions emhasizes cerain ower law bu is no variance conserving
. Time Series Analysis Secrum of an AR( Process Power secrum of an AR( rocess wih arameers (α,α is where ( σ Γ ω + α + α g( ω g( ω α( αcos(πω + α cos(4πω The secrum has an exreme for ω (0,/ when α ( α < 4 α Power secra of AR( rocesses wih (α,α (0.3,0.3 (L and (α,α (0.9,-0.8 (R. The former has a minimum a ω~0.8, while he laer has a maximum a ω~0.7 I is a maximum when α <0 and a minimum when α >0
. Time Series Analysis The general form of he secra of AR rocesses Since he auo-covariance funcion is a sum of auo-covariance funcions of AR( and AR( rocesses, and since he Fourier ransform is linear, he secrum of an AR( rocess is he Fourier ransform of he sum of auocovariance funcions of AR( and AR( rocesses and hence he sum of auosecra of AR( and AR( rocesses Inerreaion of secra of AR rocesses The exonenial decay of auo-covariance funcions of AR( rocesses imlies ha he secra of AR( rocesses are coninuous. A eak in he secra canno reflec he resence of an oscillaory comonen, even hough i may indicae he resence of damed eigen-oscillaions wih eigen-frequencies close o ha of he eak
. Time Series Analysis Parameers of Time Series: The Cross-covariance Funcion Le (,Y reresen a air of sochasic rocesses ha are oinly weakly saionary. Then he cross-covariance funcion γ is given by γ ( τ + τ E (( µ ( Y µ Y * Where µ x is he mean of and µ y is he mean of Y, and he cross-correlaion funcion ρ is given by ρ ( τ γ ( τ / σ σ Y Where σ x and σ y are he sandard deviaions of rocesses and Y, resecively To ensure ha he cross-correlaion funcion exiss and is absoluely summable, one needs o assure The mean µ x and µ y are indeenden of ime E(( E(( τ τ µ ( µ ( Y γ ( τ < for ab s s µ γ ( s, E(( Y µ ( Y µ γ ( s, E(( Y µ ( ab xx,, yy xx Y Y s s µ γ ( s Y µ γ ( s Y yy yx Proeries. γ ( τ γ. γ ( τ 3. γ yx yx αx, βy+ z * ( τ γ (0 γ (0 xx * ( τ αβ γ ( τ + αγ ( τ yy xz
. Time Series Analysis Examles: Cross-correlaion beween and Y α γ ( τ αγ xx( τ symmeric! Cross-correlaion beween and Y α + wih being indeenden whie noise α γ xx(0 + σ z, γ yy ( τ α γ xx( τ, γ ( τ αγ ( τ xx if τ 0 if τ 0 Cross-correlaion beween and Y + - γ ( τ γ ( τ γ ( τ γ ( τ xx d d xx γ xx( τ τ
. Time Series Analysis Examles: Cross-correlaion beween an AR( rocess and is driving noise γ γ τ ( τ α σ ( τ γ z ( τ 0 for τ 0 for τ < 0 Highly non-symmeric! Esimaed cross-correlaion funcions beween wo monhly indices of SST and SLP over he Norh Pacific : one from daa (hin line and he oher from a sochasic climae model (heavy line x leads z leads
. Time Series Analysis The Effec of Feedbacks on Cross-correlaion Funcions The coninuous version of an AR( rocess is a firs-order differenial equaion λ + Where he forcing acs on wihou feedback. A sysem wih feedback can be wrien as λ λ a + + N + N z x λ a 0: no feedback, ρ xz 0 when leads λ a >0: negaive feedback, ρ xz is ani-symmeric λ a <0: osiive feedback, ρ xz is osiive everywhere wih a maximum near lag zero
. Time Series Analysis The Effec of Feedbacks on Cross-correlaion Funcions Prediced correlaion beween (monhly mean urbulen hea flux and (monhly mean sea surface emeraure for differen feedbacks (Frankignoul 985 Esimaed correlaion beween (monhly mean urbulen hea flux and (monhly mean sea surface emeraure, averaged over differen laiudinal bands in he Alanic ocean
. Time Series Analysis Parameers of Time Series: The Cross-secrum Definiion: Le and Y be wo weakly saionary sochasic rocesses wih covariance funcions γ xx and γ yy, and a cross-covariance funcion γ. Then he crosssecrum Γ is defined as he Fourier ransform of γ : Γ F{ γ } τ γ ( τ e τ πiτω, ω [-/,/] The cross-secrum is generally a comlex-values funcion, since he cross-covariance funcion is neiher sricly symmeric nor ani symmeric.
. Time Series Analysis The cross-secrum can be reresened in differen ways. The cross-secrum can be decomosed ino is real and imaginary ars as Γ Λ Λ + iψ : he co -secrum, : he quadraure secrum. The cross-secrum can be wrien in olar coordinaes as Γ A A Φ Φ Φ A : ( Λ an e ( Ψ / Λ 0 if Λ ( ω > 0 ± π if Λ < 0 π / if Ψ( ω > 0 π / if Ψ( ω < 0 Ψ he amliude secrum, iφ ( ω + Ψ / Φ : when Ψ when Ψ when Λ he hase secrum 0, Λ 0 0 0 3. The (squared coherence secrum as dimensionless amliude secrum xx A κ Γ Γ yy
. Time Series Analysis Proeries of he Cross-secrum For oinly weakly saionary rocesses, Y and. Γ. γ αx, βy+ z ( τ * αβ Γ / Γ / 3.0 κ e + αγ iπτω dω xz
. Time Series Analysis Proeries of he Cross-secrum of Real Weakly Saionary Processes. The co-secrum is he Fourier ransform of he symmeric ar, γ s (τ, and he quadraure secrum is he Fourier ransform of he ani-symmeric ar of he cross-covariance secrum, γ a (τ. Λ wih γ (0 + τ s γ ( τ cos(πτω, Ψ τ s a γ ( τ ( γ ( τ + γ ( τ, γ ( τ ( γ ( τ γ ( τ Λ Λ( ω, Ψ( ω Ψ( ω a γ ( τ sin(πτω 3. The amliude secrum is osiive and symmeric, and he hase secrum is ani-symmeric 4. The coherence secrum is symmeric 5. I is sufficien o consider secra for osiive ω
. Time Series Analysis Examles: Cross-secrum beween and Y α Γ Γ Λ Ψ Α Φ yy αγ α Γ αγ 0 αγ 0 κ xx xx xx xx Cross-secrum beween and Y α + being an indeenden whie noise Γ α Γ + σ yy α Γxx( ω κ < σ + α Γ z xx xx z Cross-secrum beween and Y + - Γ Γ Λ Ψ Α yy ( e Γ ( cos(πω Γ πiω ( cos(πω Γ sin(πω Γ ( cos(πω Γ Γ sin(πω Φ an cos(πω an (co( πω π ω, for ω 0 κ, for ω 0 xx xx xx xx xx Γ xx yy
. Time Series Analysis Proeries of he Cross-secrum of Real Weakly Saionary Processes. The co-secrum is he Fourier ransform of he symmeric ar, γ s (τ, and he quadraure secrum is he Fourier ransform of he ani-symmeric ar of he cross-covariance secrum, γ a (τ. Λ wih γ (0 + τ s γ ( τ cos(πτω, Ψ s a γ ( τ ( γ ( τ + γ ( τ, γ ( τ ( γ ( τ γ ( τ Λ Λ( ω, Ψ( ω Ψ( ω 3. When he cross-covariance funcion is symmeric, he quadraure and hase secra are zero for all ω. When he cross-covariance funcion is ani-symmeric, he co-secrum vanishes and he hase secrum is Φ 4. The amliude secrum is osiive and symmeric, and he hase secrum is ani-symmeric 5. The coherence secrum is symmeric π sin( Ψ 6. I is sufficien o consider secra for osiive ω τ a γ ( τ sin(πτω