Stationary distributions of continuous time Markov chains

Transcription

1 Saionary disribuions of coninuous ime Markov chains Jonahon Peerson April 3, 22 The following are some noes conaining he saemen and proof of some heorems I covered in class regarding explici formulas for he saionary disribuion and inerpreaions of he saionary disribuion as he iing fracion of ime spen in saes. Saionary measures in coninuous ime The following heorem is an analog of he explici formula for saionary measures for discree ime Markov chains (Theorem.2). Theorem. If X is an irreducible coninuous ime Markov process and all saes are recurren, hen for any x I he measure µ x defined by is a saionary measure. µ x (y) = P x (X = y, < T x ) d Remark 2. Noe ha we can also wrie µ x (y) = E[ X=y, <T x d [ = E X=y, <T x d = E X=y d () Tha is, µ x (y) has he inerpreaion of he amoun of ime spen in y before he firs reurn o x when saring originally from x.

2 Proof of Theorem. To show ha µ x is a saionary disribuion, i is enough o check ha µ x Q =. To his end, we firs use he formula for µ x (y) o wrie he z-h enry of µ x Q as (µ x Q)(z) = µ x (y)q(y, z) y I = µ x (y)q(y, z) λ z µ x (z) = = ( ) P x (X = y, < T x ) d q(y, z) λ z P x (X = z, < T x ) d P x (X = y, < T x )q(y, z) λ z P x (X = z, < T x ) d (2) The erm inside he braces looks similar o he Kolmogorov forward differenial equaion, bu he presence of he condiion < T x inside he probabiliies makes i differen. To recify his we will creae a new Markov process ˆX on sae space Î = I {ζ} (where ζ is a new sae no originally in I). The new Markov chain will have jump raes ˆq ha are given by q(y, z) y = x or z I\{x} q(y, x) y I\{x} and z = ζ ˆq(y, z) = y I\{x} and z = x and ˆλ y = { λ y y I ˆq(y, z) = y = ζ y = x and z = ζ z Î\{y} y = ζ and z I, Thus, he new Markov process ˆX behaves he same as he old Markov process excep all jumps ino x from anoher sie are redireced o be jumps o a new absorbing sae ζ. Wih ˆX defined in his way, we obain ha P x (X = y, < T x ) = P x ( ˆX = y), y I. We ll coninue our analysis of (µ x Q)(z) depending on wheher or no z = x. Case I: z x. If z x hen he erms inside he braces in (2) are P x ( ˆX = y)q(y, z) λ z P x ( ˆX = z) = P x ( ˆX = y)ˆq(y, z) ˆλ z P x ( ˆX = z) y Î\{z} = d d P x( ˆX = z), 2

3 where he las equaliy follows from he Kolmogorov forward equaions for he Markov process ˆX. Noe also ha he in he firs equaliy above we were able o exend he sum from y I\{x} o y Î\{x} since ˆq(ζ, z) =. Therefore, puing his back ino (2) we obain ha (µ x Q)(z) = d d P x( ˆX = z) d = s P x ( ˆX s = z) P x ( ˆX = z) =, where he las equaliy holds because P x ( ˆX = z) = (since z x) and P x ( ˆX s = z) P x (T x > s) as s (since X is recurren). Thus, we ve shown ha (µ x Q)(z) = whenever z x and i remains o show his is also rue when z = x. Case I: z = x. If z = x hen he erms inside he braces in (2) are P x ( ˆX = y)q(y, x) λ x P x ( ˆX = x) y I\{x} = y I\{x} P x ( ˆX = y)ˆq(y, ζ) λ x P x ( ˆX = x) = P x ( ˆX = y)ˆq(y, ζ) ˆλ ζ P x ( ˆX = ζ) λ x P x ( ˆX = x) y I = d { P x ( d ˆX } = ζ) λ x P x ( ˆX = x). where in he second equaliy is rue because ˆq(x, ζ) = and ˆλ ζ = and he las equaliy follows from he Kolmogorov forward equaion for ˆX. Also, noe ha since ˆX can never reurn o x once i leaves, P x ( ˆX = x) = e λx since his is jus he probabiliy ha he Markov process hasn lef x ye by ime. Puing all of his back ino (2) we obain ha (µ x Q)(x) = d d { P x ( ˆX = ζ) } d = s P x ( ˆX s = ζ) P x ( ˆX = ζ), =. λ x e λx d where he las equaliy holds since P x ( ˆX s = ζ) = P x (T x < s) as s since X is recurren. Thus, we have shown ha (µ x Q)(z) = for all z I and so µ x is a saionary measure. We should also check ha he definiion of µ x isn rivial. Tha is, µ x (y) (, ). Firs, noe 3

4 ha µ x (x) = /λ x since he holding ime a x is Exp(λ x ). Secondly, since µ x is saionary we know ha µ x p = µ x for any > and so λ x = µ x (x) = (µ x p )(x) µ x (y)p (y, x). Since X is irreducible we know ha p (y, x) > for any > and so µ x (y) λ xp (y,x) <. To show ha µ x (y) >, noe ha since () shows ha µ x (y) is he expeced amoun of ime spen in y beween visis o x, he srong Markov propery implies ha his is a leas he probabiliy of reaching y before reurning o x imes he expeced amoun of ime spen in y before he firs jump ou of y. Tha is, µ x (y) = E x [ Tx X=y d = E x X=y d V y = P x (V y < )E y P x (V y < )E y [ minz y T z = P x (V y < ) λ y. Since X is irreducible we have ha P x (V y < ) > and so µ x (y) >. X=y d X=y d As in he case of discree ime Markov chains we can normalize µ x o obain a saionary disribuion (a leas when E x [T x < ). Corollary 2.. If X is irreducible and posiive recurren, hen a saionary disribuion can be defined by π(y) = µ x(y) E x [T x for any x I. Proof. This follows Theorem and he fac ha µ x (y) = P x (X = y, < T x ) d = y y P x ( < T x ) d = E x [T x. 4

5 2 Inerpreaions of he iing disribuion If we know ha a iing disribuion π exiss hen π(j) is approximaely he probabiliy ha he Markov process will be in sae j a some very large ime. We will show in his secion ha π(j) also is equal o he iing fracion of ime ha he Markov process spends in sae j. To make his precise, define L (y) = Xs=yds. Theorem 3. If X is irreducible and posiive recurren, hen L (y) = µ x(y) E x [T x, where he above i holds wih probabiliy one, for any x I, and irrespecive of he saring locaion. Moreover, since he i is he same for any x I we have ha L (y) = λ y E y [T y. To relae his i o saionary disribuions we need he following heorem. Theorem 4. If X is irreducible and posiive recurren, hen here is a unique saionary disribuion π. Moreover, π has he formula π(y) = λ y E y [T y. Proof of Theorem 3. Suppose ha he Markov chain sars a X = x. Le T x = τ < τ 2 < τ 3 <... be he imes of successive reurns o x and le r i = τi τ i Xs=y ds be he amoun of ime spen in sae y beween imes τ i and τ i (here we le τ = by convenion). Also, le = max{n : τ n } be he number of reurns o x by ime. The law of large numbers from renewal heory (Theorem 3. in he book) implies ha = E[τ = E x [T x, (3) 5

6 wih probabiliy one. Also, since he r i are i.i.d., he law of large numbers implies ha n n n r i = E[r = E x i= X=y d = µ x (y). (4) Now, since he definiion of implies ha τ < τ + we have ha Therefore, τ Xs=y ds = r i L (y) i= i= r i L (y) + i= + r i = τ+ + + i= Xs=y ds and so wih probabiliy one L (y)/ is sandwiched beween wo erms ha boh converge o µ x (y)/e x [T x. We sill need o show ha he i is he same even if he Markov process doesn sar a X = x. To his end, suppose he Markov process sars a X = z x. Then define τ = T x o be he firs visi o x and as above τ 2 < τ 3 <... are he successive visis o x. In his case, i is sill rue ha (3) holds (his is he law of large numbers for a delayed renewal process) and ha (4) holds as well (his is because r < and r 2, r 3,... are i.i.d.). The final claim of he Theorem ha he i is equal o /(λ y E y [T y ) follows by choosing x = y and noicing ha µ y (y) = /λ y. r i, Proof of Theorem 4. Theorem 3 implies ha L (y)/ λ ye y[t y for any iniial disribuion of X. In paricular, i holds if X has an iniial disribuion π ha is saionary. Therefore, he bounded convergence heorem implies ha E π [ L (y) = λ y E y [T y. On he oher hand, he definiion of L (y) implies ha [ L (y) E π = [ E π Xs=y ds = P π (X s = y) ds = π(y) = π(y), where we used ha π was a saionary disribuion in he second o las equaliy. Thus, we have shown ha any saionary disribuion π mus saisfy π(y) = λ ye y[t y, and so π is unique. 6