Recurrence. 1 Definitions and main statements

Transcription

1 Recurrence 1 Defntons and man statements Let X n, n = 0, 1, 2,... be a MC wth the state space S = (1, 2,...), transton probabltes p j = P {X n+1 = j X n = }, and the transton matrx P = (p j ),j S def. Defnton 1.1 A state S s sad to be recurrent f P {X n = for some n 1 X 0 = } = 1. In words: S s recurrent f the probablty that the MC startng from wll return to s 1. A state s sad to be transent f t s non-recurrent. In other words, s sad to be transent f the probablty that the Markov chan startng from wll never to return to s strctly postve. As usual, we would lke to be able to tell whether or not a MC s recurrent n terms of propertes of transton probabltes of ths chan. Remember that def j = P {X n = j X 0 = } and that these probabltes can be found, at least n prncple, n terms of P, namely ( j ) = Pn. Theorem 1.2 A state S s recurrent f and only f =. (1) Defnton 1.3 We say that the MC X n s recurrent f all states of ths MC are recurrent. Theorem 1.2 wll be proved n the next secton. But we start wth two examples of concrete Markov chans for whch ths theorem allows one to establsh recurrence or transence (whchever s approprate). Example 1. Let X n be a fnte regular MC. Then ths chan s recurrent. Indeed, snce regular chans have the property that lm n k = w > 0, t follows that p(n) j =. A dfferent and, n a sense, much more natural proof of ths statement wll be gven n Secton 3. Example 2. To defne a smple one-dmensonal random walk magne a partcle movng along a one-dmensonal lattce Z = {... 2, 1, 0, 1, 2,...}. The partcle s observed at equdstant tme moments n = 0, 1, 2... Let X n be the coordnate of the partcle at tme n. If at tme n the partcle s at ste k, then at tme n + 1 t wll 1

2 jump to the rght wth probablty p or to the left wth probablty q def = 1 p and the of length of the jump s one; by defnton, each next move of the partcle does not depend on the hstory of ts movements (or, one could say, on ts past postons). It s the clear that the sequence X n forms a MC wth transton probabltes P {X n+1 = k + 1 X n = k} = p, P {X n+1 = k 1 X n = k} = q. We shall suppose that X 0 = 0. Remark. Note that ths MC has an nfnte state space - the set of all nteger numbers. In the context of the theory of random walks, t s common to call ths space state the one-dmensonal lattce. Statement. The state 0 s recurrent f p = q = 1 and s transent f p q. 2 To see ths, note frst that we can compute the probabltes of return to 0 explctly. Namely, the frst and easy remark s that p (2n+1) 00 = 0 because the random walk has to make an even number of steps n order to return to the startng pont. Next, p (2n) 00 = ( 2n n )p n q n (prove ths or see the explanaton gven n the lecture). The Statement wll follow f we show that the seres ( ) 2n j p n q n n converges when p q and dverges when p = q = 0.5. The case p q s smple because the rato test can be appled. Namely, t s easy to check that ( 2n+2 ) p n+1 q n+1 lm n n+1 2(2n + 1)pq ( 2n ) = lm = 4pq < 1 f p q. pn q n n n n + 1 (We use here the nequalty p(1 p) < 1 f p 0.5). Hence the seres converges and 4 the random walk s transent. The rato test does not work when p = q = 0.5 because then 4pq = 1. We shall prove that n ths case the seres dverges usng the famous Strlng s formula: n! 2πnn n e n. You can check that ths formula mples ( ) p (2n) 2n 00 = p n q n = (2n)! n (n!) 2 pn q n (4pq)n. πn When p = q = 0.5 we obtan p (2n) 00 1 πn. The seres thus dverges whch, accordng to the above theorem, mples the recurrence of the random walk. Remark. We say that a n b n f lm n a n bn = 1. 2 Proof of Theorem 1.2 In order to prove the theorem we need to ntroduce the followng random varable: R = the tme of the frst return to. (2) 2

3 The probablty mass functon of R shall be denoted f (n), n 1. Obvously, f (n) = P {R = n X 0 = } = P {X n =, X k for k = 1,..., n 1 X 0 = }. (3) Clearly f (1) p (k) = p. All other probabltes f (n) because of the followng Lemma 2.1 For n 1 can be calculated recursvely terms of = f (1) p (n 1) + f (2) Indeed, f (4) holds then f (n) p (n 2) = (f (1) p (n 1) + f (2) f (n 1) p (1) + f (n) p (n 2) n f (n 1) p (1) ) n 1 p(n) f (k) p (n k). (4) f (k) p (n k) and hence we can calculate f (2) usng f (1), then we can calculate f (3) usng f (1), and so on. f (2) Proof of Lemma 2.1. The event {X n = } can take place only together wth one of mutually exclusve events {R = k}, k = 1, 2,..., n. Hence, by the Total Probablty Law, But P {X n = X 0 = } = = n P {R = k X 0 = }P {X n = R = k, X 0 = } n f (k) P {X n = R = k, X 0 = }. P {X n = R = k, X 0 = } =P {X n = X k =, X l when 1 l k 1, X 0 = } =P {X n = X k = }} = p (n k), where the last equalty follows from the Markov property. The Lemma s proved. Next, set β def = =1 f (n). Obvously, β = P {X n = for at least one n 1 X 0 = } P { MC startng at would return to }. We can now say that the state S s recurrent f and only f β = 1 and Theorem 1.2 can be stated as follows: Theorem 2.2 and = f and only f β = 1 (5) 3

4 or, equvalently, < f and only f β < 1 (6) Proof of Theorem 2.2. Suppose frst that U def = p(n) < ; we shall now prove that then β < 1. To ths end, let us wrte equatons (4) as follows: p (1) =f (1) p (2) =f (1) p (1) + f (2) p (3) =f (1) p (2) + f (2) p (1) + f (3). =f (1) p (n 1) + f (2) p (n 2) f (n 1) p (1) + f (n) (7) Defne U N def = N p(n). Let us add up the frst N equatons n (7). We obtan U N = f (1) (1 + U N 1 ) + f (2) (1 + U N 2 ) f (N 1) (1 + U 1 ) + f (N). (8) Snce lm N U N = U <, we can pass to the lmt n (8) and obtan U = f (1) (1 + U) + f (2) (1 + U) f (n) (1 + U) +... = β (1 + U). (9) Hence β = U 1 + U < 1 (10) (whch means that the random walk s transent). Let us now consder the case U = whch by defnton means that lm N U N =. We shall agan make use of (8). Namely, let us replace all the factors of the form 1 + U n n the rght hand sde of (8) by 1 + U N. Snce 1 + U n 1 + U N for all n N, we obtan: U N f (1) (1 + U N ) + f (2) (1 + U N ) f (N 1) (1 + U N ) + f (N) (1 + U N ) =(f (1) + f (2) f (N) )(1 + U N ) β (1 + U N ), (11) where the last step s due to the fact that β = and therefore β lm N β U N 1 + U N U N 1 + U N = lm N f (n). Hence 1 1 U N + 1 = 1. But β 1 because ths number s probablty of some event. Hence β = 1 whch means that the random walk s recurrent. Theorem 2.2 s proved and thus also Theorem 1.2 s proved.. 4

5 3 Recurrence and communcaton classes Let, as n secton 2, X n be a MC wth a state space S (S may be an nfnte set). Defnton 3.1 We say that two states, j S ntercommuncate f there are k 1 and l 1 such that p (k) j > 0 and p (l) j > 0. In other words the states and j ntercommuncate f the two probabltes, to reach j startng from and to reach startng from j, are strctly postve. We shall rght j nstead of sayng ntercommuncates wth j. By conventon,. Note next that f j and j k then k (explan ths statement!). Defnton 3.2 We say that a subset C S forms a communcaton class f any, j C ntercommuncate wth each other. Each state S belongs to exactly one communcaton class, namely the one formed by all states from S whch ntercommuncate wth (and therefore also ntercommuncate wth each other). Hence the state space can be parttoned nto (non-ntersectng) communcaton classes. If we call them C 1, C 2,... then S = C 1 C 2... C k, wth C C j = f j. Remarks. 1. If S s a fnte set then of course the number of communcaton classes s fnte. If S s an nfnte set the number of communcaton classes can be nfnte too; However, we shall not consder such chans. 2. A communcaton class may conssts of just one state; for nstance, ths happens f s an absorbng state or f cannot be reached from any other state. 3. In the lterature communcaton classes are often called equvalence classes. They ndeed are equvalence classes wth respect to the ntercommuncaton property. Let us frst consder the smplest case, k = 1, and thus S = C 1. Snce all states ntercommuncate wth each other ths smply means that the Markov chan s rreducble. (Remember the defnton of an rreducble Markov chan!) Next, let k = 2, that s S = C 1 C 2. Then there are two further possbltes: (a) No state from from C 2 can be reached from any state from C 1 and vse versa, no state from C 1 can be reached from any state from C 2. (b) No state from C 2 can be reached from any state from C 1, but some sates from C 1 can be reached from some states from C 2 (formally, there s a thrd possblty whch n fact s the same as the second one; what s t?). Exercses. 1. Consder Markov chans wth transton matrces and (12)

6 What are the communcaton classes of these chans? Whch of the two cases (a) and (b) lsted above descrbe the behavor of each chan? 2. Suppose that S = C 1 C 2 C 3. What are the dfferent types of behavour, n terms of communcatons between the classes, whch one may observe? We shall now prove a smple but mportant theorem whch, as wll be seen, descrbes the recurrence propertes of communcatons classes. Theorem 3.3 If and j are ntercommuncatng states and s recurrent then j s recurrent. Proof. Snce and j ntercommuncate, there s a k such that p (k) j such that p (l) j > 0. But then p (k+n+l) jj p (l) j p(n) p (k) j. > 0 and and an l Ths nequalty holds because one of the possbltes to reach j startng from j n k + n + l transtons would be to frst reach n l transtons (wth probablty p (l) then to return to after n transtons (wth probablty ) and fnally to reach j from n k transtons (wth probablty p (k) j ). Hence jj p (k+n+l) jj p (l) j p(k) j because s recurrent and ths means that p(n) = j ), =. Theorem 3.3 s proved. Obvously, Theorem 3.3 mples that both recurrence and transence are propertes of a communcaton class: ether all states n the class are recurrent or all of them are transent. Examples. 1. Consder a fnte rreducble Markov chan. It follows from Theorem 3.3 that t s recurrent. Indeed, snce S t s a fnte set, there must be at least one state whch s vsted by the chan nfntely many tmes as n and hence s recurrent. But ths state ntercommuncates wth all other states. Hence the chan s recurrent. In partcular, every fnte regular Markov chan s recurrent (because regularty mples rreducblty). In example 1, Secton 1, the exstence of a more natural prof of recurrence was mentoned. So, here t s. 2. Consder the one-dmensonal random walk dscussed n Secton 1. Snce 0 ntercommuncates wth each state of the lattce, we conclude that all states are recurrent f p = q = 1 and are transent otherwse. 2 Exercse. For each of the two Marcov chans whose transton matrces are gven by (12) establsh whch states are recurrent and whch are transent? 6