Markov Chains P P P P P P P P P { } for all i o,i n-1, i, j, P ij, 0 i M, 0 j N, are called transition probabilities of the Markov chain M

Transcription

1 Markov Chans { X j X, X, X,..., X, X } n n n n n n o j for all o, n-,, j, j, M, j N, are called transton probabltes of the Markov chan,,,... M : j M j j M M M M MM {X n n, X n- n-,,x,x o } {X n n X n- n-,,x }{X n- n-,,x }, n {Xn n,...,x n- o} X,,,, { } n n n n o

2 Example: Suppose that whether or not t rans tomorrow depends on prevous weather condtons only through whether or not t s ranng today. Suppose further tha f t s ranng today, then t wll ran tomorrow wth probablty α, and f t s not rasnng today, then t wll ran tomorrow wth probablty β. If we say that the system s n state ) when t rans and state when t does not, then the above s a two-state Markov chan havng transton probablty matrx α α β β That s, α -, β - Example: Consder a gambler who, at each play of the game, ether wns unt wth probablty p or loses unt wth probablty -p. If we suppose that the gambler wll qut playng when hs fortune hts ether or M, then the gambler's sequence of fortunes s a Markov chan havng transton probabltes

3 Example: The physcsts. and T. Ehrenfest consdered a conceptual model for the movement of molecules n whch M molecules are dstrbuted among urns. At each tme pont one of the molecules s chosen at random, and s removed from ts urn and placed n the other one. If we let Xn denote the number of molecules n the frst urn mmedately after the nth exchange, then {X, X, } s a Markov chan wth transton probabltes M, M M, M M f j- > j Thus, for a Markov chan, j represents the probablty that a system n state I wll enter state j at the next transton. We can also defne the two-stage transton ( ) probablty, j, that a system, presently n state I, wll be n state j after two addtonal transton. That s, ( ) { X j X } j m m

4 ( ) The j can be computed from the j as follows: ( ) { X j X } j M { X j, X k X } k M { X j, X kx } { X k X } k M k kj k ( n) In general, we defne the n-stage transton probabltes, denoted as j, by ( n) { X j X } j n m m ( n) Ths leads us to the Chapman-Kolmogorov equatons, whch shows how the j can be computed

5 Chapman-Kolmogorov Equatons. M ( n) ( r) ( n r ) j k kj For all < r < n k roof: ( n ) { X j X } j n { X j, X k X } k { X j, X kx } { X k X } k M k n n ( n r) ( r) kj k r r r

6 Example A Random Walk. An example of a Markov chan havng a countable nfnte state space s the so-called random walk, whch tracks a partcle as t moves along a one dmensonal axs. Suppose that at each pont n tme the partcle wll ether move one step to the rght or one step to the left wth respectve probabltes p and -p. That s, suppose the partcle's path follows a Markov chan wth transton probabltes, p,, ±,... If the partcle s at state, then the probablty t wll be at state j after n transtons s th probablty that (n-j)/ of these steps are to the rght and n-[(n-j/](n-j)/ are to left. As each step wll be to the rght, ndependently of the other steps, wth probablty follows that the above s just the bnomal probablty. F n I n ( n j) / n j n j j p p HG KJ ( )/ ( ) Fn where J s taken to equal when x s not a nonnegatve nteger less than or equal to n. x HG I K

7 F HG I n n k p ( ) p ( ) k, ±,... ± n KJ F n I p ( p) k, ±,..., ± n,-(n ) HG n k KJ n n k n k, k n n k n k, k- { X j} { X j X } { X } n n o ( n) { X } j A suffcent condton for a Markov chan to possess ths property s that for some n> ( n ) j > for all I,j,,,.,M

8 Markov chans that satsfy ths equaton are sad to be ergodc. Snce the Chapman-Kolmogorov equatons yeld M ( n ) ( n) j k kj k t follows, by lettng n, that for egodc chans M M j j kj k ( n) Furthermore, smce j, we also obtan, by lettng n,, j M j j In fact, t can be shown that the j, j M, are the unque nonnegatve solutons of the above last two equatons

9 Theorem For an ergodc Markov chan j n lm ( ) n j Exsts, and the j, j M are the unque solutons of M j M j k kj k j

10 Example: Consder the example from above, n whch we assume that f t rans today, then t wll ran tomorrow wth probablty α, and f t does not ran today, then t wll ran tomorrow wth probablty β. From the above theorem t follow that the lmtng probabltes of ran and of no ran, o and are gven by α β ( α) ( β) o o whch yelds β α β α β α

11 For nstance, f α.6,β., then the lmtng probablty of ran on the nth day s /7. The quantty j s also equal to the long run proporton of tme that the Markov chan s n state j, j,.,m. To ntutvely see why ths mght be so, let j denote the long run proporton of tme the chan s n state j. (It can be proven, usng the strng law of large numbers, that for an ergodc chan such long run proportons exst and are constant). Now, snce the proporton of tme the chan s n state k s k and snce, when n state k, the chan goes to state j wth probablty kj, t follows that the proporton of tme the Markov chan s enterng state j from state k s equal to k kj. Summng over all k shows that j, the proporton of tme the Markov chan s enterng state j, satsfes Snce clearly t s also true that j j j k k are the unque soluton of the above, that pj j j,,m. kj t thus follows, snce by our theorem the j j,,m

12 Example Suppose, n the example above that we are nterested n the proporton of tme there are j molecules n urn, j-,,m. By the Chaoman-Kolmolorov theorem, these quanttes wll be the unque soluton of M M j j j j j M M M M M M j j j,..., M. However, as t t easly checked that j M F M H G I K J ( ), j j,..., M.

13 Revew of Markov Chans When the process s n state there s a fxed probablty, j, that t wll next be n state j..e. { X j X, X,..., X t } j n n n n o for all states,,... n., j and all n > " A Markov chan can be descrbed by the condtonal dstrbuton of any future state, X n, gven the past states X, X,,X n-, and the present state X n. Ths condtonal probablty s ndependent of the past states and depends only on the present state"., j j j j,,...

14 Assume a sequence of experments wth the followng propertes: Outcome of each experment s one of a fnte number of possble outcomes a, a, a n. robablty of outcome aj of any gven experment s not necessarly ndependent of prevous outcomes of experments, but depends at most on outcome of mmedately precedng experment. Assume we can assgn number, j, whch represents probablty of outcome aj, gven a occurred. Note: outcome a, a,,a, a r states, and p j transton probabltes. 4 If the process s assumed to begn n some partcular state, we have enough nformaton to determne the tree measure (flow of events) of the process and calculate the probablty of statement relatng to the experments

15 or ) Square array (transton matrx) F G H p p p p p p p p p ) Transton dagram..e. gven possble state: I J K a a ½ ½ a / / The matrx assocated wth dagram s; a a a a a a F G H I J K

16 Example Suppose that the chance of ran tomorrow depends only f t s ranng today (not on past weather And,f t s ranng today, then t wll ran tomorrow wth probablty α. If t does not ran today, t wll ran tomorrow wth probablty β. We say that the process s n state when t rans, and n state when t does not ran. Ths s a two-state Markov chan whose transton probablty matrx s L NM α β O α Q robablty vector β robablty vector

17 Example Gary s ether cheerful, C, so-so, S or glum, G. ) If he s cheerful today then he wll be C, S, or G tomorrow wth probablty.5,.4, or., respectvely. ) If he s so-so today then he wll be C, S, or G tomorrow wth probablty.,.4. or., respectvely. ) If he s glum today then he wll be C, S, or G tomorrow wth probablty.,.. or.5, respectvely. If X n Gary's mood on the nth day, {X n, n } s a stage Markov chan. If we desgnate C, S, and G the probablty matrx s: F G H p p p p p p p p p I J K J F G H... 5 I J K

18 ( n) The n transtonal probablty matrx j. Gven So, F p p p ( ) G p p p j n a a a HG ( n) ( n) ( n) ( n) ( n) ( n) ( n) ( n) ( n) p p p a a a F H G I K J I KJ ½ a ½ a ½ a a a ½ / a a / a

19 .e Smlarly, ( ) probablty of gong to state from state after steps sum of the probabltes assgned to all paths thru tree whch end at state a We also can represent ths as follows, ( ) ( ) ( ) F ( ) H G I K J 7

20 a ½ ½ a a a ½ ½ / / a a a ½ ½ / / / / a a a a a a a ( ) 7 6 ( ) ( ) (,, ) ( ) 7

21 a / a a a a a a / / a ( ) 4 7 / ( ) 4 7 ( ) / / / / a a a 7 8 ( ) 5 54 (,, ) ( ) a a a F G H a a a I J K

22 Example: Suppose that we are nterested n studyng the way n whch a gven state votes n a seres of natonal electons. We shall base our predcton only on past hstory of the outcomes of the electons (Republcan or Democrat). As a frst approxmaton we assume that knowledge of the past, beyond the last electon, would not cause us to change the probablty for the outcome on the next electon. So we have a markov process, or chan wth states, Republcan ( R ) and Democrtat ( D ). R D R D F HG a b a b I KJ where a can be estmated from past results and represents the fracton of years n whch there was a change from R to D., and b represents the fracton of years n whch there was a change from D to R. We can obtan a better approxmaton from past results of two prevous electons. Our states now are RR, RD,DR,DD, and f next electon s a Democrat, RR -> RD, etc. The frst letter of the new state must be the second letter of the old state. So, f electon outcomes for a seres of electons s DDDRDRR our process moves DD DD DR RD DR RR etc Hence RR DR RD DD F G HG RR DR RD DD a a b b c d d where a,b,c,d, must be estmated. c I J KJ

23 Regular Transton Matrx ropertes: F HG ) n W ) Each row of W s same probablty vector w. ) Components of w are postve v) If p s any probabolty vector, p n w v) w s the unque fxed pont probablty vector of. (.e. a probablty vector w, s a fxed pont of the matrx f ww) The fxed probablty vestor s bd ad bd ad ca bd ad ca bd ad ca bd ad ca ad ca I K J probablty of beng n state RR and DR bd ad bd ad ca

24 Example: Consder n F H G F w H G I K J and f (. 6. 4) we see that F HG I (. 6,. 4) (. 6,. 4) w KJ F 6 89 H G I K J F H G I K J F H G.. I K J.. F H G I K J F.. H G I K J F H G.. I K J.... I K J F HG w w ww ( w, w ) w w w w w w w I KJ w L N M O Q

25 Defne n j n j { X j X } n,, j n m m m n j n m { X j X } { X j, X k X } j k k n Note: m { X j X k, X } { X k X } n m kj k Formally, If (n) denotes the matrx of n-step transton probablolty, n, then j ( n m) ( n) ( m).e. m kj probablty that startng n the process wll go to state j n n m steps through a path that takes t nto state k at the nth transton. ( ) ( ) ( ) In general, n m n m n o n m n n k k ( n) ( n ) ( n ) ( ) n kj n k

26 Example: Consder a two stage Markov Chan, wth transton probablty matrx as α.7, and β.4, L N M O Q Calculate the probablty that t wll ran four days from today, gven that t raned today. F H G I K J F H G I K J F H G I K J ( ) F ( 4) ( ) H G I K J F H G I K J F H G I K J 4 probablty of gong from state (ran) to state (ran) n 4 steps 5749 n j probablty that state at tme n s j, gven that the ntal state at tme s. If we want the uncondtonal dstrbuton of the state at tme n, we need to specfy t he probablty dstrbuton of the ntal state. α { X } where and α

27 Example: Let X { X n; n N} be a Markov Chan wth states a,b,c. (state space, E{a,b,c}) and transton matrx Then L NM / / 4 / 4 / / / 5 / 5 [ X b, X c, X a, X c, X a, X c, X b X ( c, b) ( b, c) ( c, a) ( a, c) ( c, a) ( a, c) ( c, b) O Q 5 The two step transton probabltes are So that, for example { X c X b} ( b, c) / 6 n L NM / / 4 / 4 / / / 5 / 5 n O L Q NM / / 4 / 4 / / / 5 / 5 O L Q NM 7 8 /

28 We can wrte { X } { X j X c} { X c} n 5 n For example, consderng the Markov Chan from the prevous problem, gven α uncondtonal probablty that t wll ran 4 days from now s n j α [ X } (. 5749). 6(. 5668) ; α. 6, then the

29 Let X be a Markov Chan wth state space E[,], ntal dstrbuton (/,/) and transton L matrx N M. 5. 5O Q L N M O Q L N M O Q L N M O Q In general, t can be shown that L MM N m m m 5 (. ) 5 5 (. ) m 5 m 8 8 (. ) 8 8 (. ) O Q

30 And [ X, X, X, X X ] [ X, X, X X ] [ X, X 5 (, ) (, ) (, ) (, ) (. 5)(. 7)(. 4)( 8 8 (. ) ) (, ) (, ) (, ) (. 4)( 8 8 (. ) )(. 64) π( ) (, ) (, ) π( ) (, ) (, ) / (. 4)( (. ) / (. 6)( (.. 6 ] { X } { X, X X }

31 Accessble States State j s accessble from state I f j n > for some n. Two states that are accessble to each other are sad to be communcate, j. Any state communcates wth tself snce [ X X ] Communcaton satsfes the followng three propertes:. State I communcates wth state I, for all.. If state I communcates wth state j, then state j communcates wth state.. If state I communcates wth state j, and state j communcates wth state k, then state I communcates wth state k. Two states that communcate are of the same class. A chan s rreducble f there s only one class (.e. all states communcate wth each other).

32 Example: Gambler's Run. You have A dollars, and the gambler has B dollars. You bet $ on each game and play untl one of you s runed. p > /. Fnd the probablty that you wll be runed. Let N AB - total amount of dollars n the game. As states for the chan we choose,,.,n. At any one moment the poston of the chan s the amount of money you have. The ntal poston s: Your money. Hs money. O A B At any step there s a probablty p of movng one step to rght and probablty (-p)q of movng step to the left. We have a: Markov Chan! If the chan reaches or N we stop. These are called ABSORBING STATES, snce once they are entered they are never left. We are nterested n the probablty of your run (reachng ). Fnd the run-probablty for every possble startng poston. Let X be the probablty of your run when you start at poston We solve for N5, so we have unknown X, X, X, X 4, X 5, we start at poston, the chan moves to wth probablty p, or to wth probablty q.

33 {run/start at }{run/start at }*p {run/start at }*q But once t reaches stage, a Markov Chan behaves lke t had been started there. So, {run/start at } X {run/start at } X Therefore we can wrte X px qx Substtutng pq X (pq)x px qx (X -X ) q(x -X ) Hence, X - X r(x - X ) Where r p/q, and hence, r<. We can wrte ths for each of the four ordnal postons We have our absorbng states X and X 5. If we substtute X 5 nto the above X X r( X X ) X X r( X X ) X X r( X X ) X X r( X X ) X X r( X X ) 4 X X r( X X ) 4 4 5

34 X X X X r X X X r X X X r X 4 X X r X X X - X 4 rx 4 r ( r) X r But X,and X ( r r X X 4 ( r r r r r r r 5 5 r r 4 r ) X X 4 ) X 4 ( r 5 ) ( r) 4 r 5 r r r 4 5

35

36 Run robablty X A r r B N Suppose, for example, that a gambler wants to have probablty.999 of runng you. He must make sure that r B <.. For example, f p.495, the gambler needs $46 to have a probablty.999 of runng you, even f you are a mllonare. If p.48, he needs only $87. And even for the almost far game wth p.499, $77 wll suffce.. There are two ways that gamblers acheve ths goal. Small gamblng houses wll fx the odds qute a bt n ther favor, makng r much less than. Then even a relatvely small bank of B dollars suffces to assure them of wnnng. Larger houses, wth B qute szable, can afford to let you play nearly far games

37 SUMMARIZING Markov Chan Consder a random process or stochastc process, { X n, n,,.e. { X j X, X... X } n n n n j for all states,,...,, j, and alln > j j n, j j Absorbng State Markov Chan. A state, S, s absorbng f and j for j. Consder 5 states, absorbng L M M NM O Q L I N M O R Q Q

38 Let m j the mean number of tmes the system s n state S j, gven t started from S I, where S and S j are non absorbng states. m δ m ; δ j j k k kj j R S T j j δ j accounts for the fact that f chans goes to absorbng state n one step, t was n state S one tme. We can wrte ths for all states; m M I QM M I Q mk - over all non - absorbng states k Consder now the problem of absorpton nto varous absorbng states. Let a j robablty of system absorbed n state S j, gven t started n state S (S non-absorbng state and S j absorbng state) The system could move to Sj n one step, or t could move to a non-absorbng state, Sk, and be absorbed from there a p p a j j k k A R QA or kj non -absorbng states - A ( I - Q) R MR

39 Example: The manager of a plant has employees workng at level I and level II. New employees mat enter ether level.. At the end of each year the performance of employees are evaluated. They can be ether reassgned to level I or II jobs, termnated, or promoted to level III, n whch case they may never come back to levels I or II. We can treat ths as a Markovan process, wth two absorbng states Employment at level I - S Employment at level II - S4 Records over long perods of tme ndocate the followng probabltes as vald L NM A A A A O Q L I N M O R Q Q Thus, f employee enters levek I, the probablty s.5 that she wll jump to level II and. that she s termnated. Fnd: The expected number of evaluatons an employee must go through n ths plant.e. we want to fnd the expected number of steps to absorbton from non-absorbng state, then sum over all... M ( I Q) and from, we have R and Q.. L N M O Q L N M.. 5O Q L N M O Q L. 8. N M 7 7 O Q

40 Hence, m m m /7 m m m 8/7 Whch concludes that new employees can expect to reman there through /7 evaluatons whle n level I. Also L A MR N M O L Q N M O Q L A N M O 7 7 Q That s, an employee enterng at level I has a probablty 4/7 of reachng level II, whereas, at level II rases the probablty to 5/7.

41 . { X S X S } k j jk L N M m m Let X be startng state of the system wth probablty gven by ( ) k ( X S ) k ( n Let {beng n S k after n steps} ) k. These are convently dsplayed as; mm O Q ( ) ( ) ( ) ( ) m [,... ] ( n) ( n) ( n) ( m n ) [,... ] The relatonshp between these probabltes can be ascertan by consderng the followng: () ( ) ( ) ( ) ( ) probablty of beng n varous states after step [,... m ]. If we multply ths vector by the transton probablty matrx, () [probablty of beng n state k after step][probablty of gong to a next state] ( n) ( n ) Hence, we get.

42 Defnton: If some power of ( n ) has all postve entres, s sad to be regular. So, one can get from state S j to state S k eventually, for any (j,k) Defnton: If s regular, the Markov chan has a statonary dstrbuton that gves the probablty of beng n the respectve states after many transtons..e. j (n) must have a lmt, j as n, whch satsfes

43 Example: Market stocks brands of coffee, A, B, C, and t has been notced that customers swtch from brand to brand accordng to the followng transton matrx: L NM / 4 / 4 / / / 4 / 4 / where S purchase of A S purchase of B S purchase of C So, /4 of A today swtches to A tomorrow (stays wth A), /4 of A today swtches to B tomorrow, and of A toad swtches to C tomorrow (no one swtches). (a) Let's fnd the probablty a customer swtches to A today wll swthc to A two weeks from now, assumng she buys coffee once/week Customer starts wth purchase of A:. then ; (,, )[ D] ( ) ( ) ( ) ( ) 4 4 L F NM H G I K J ( ) O Q / 4 / 4 / / / 4 / 4 / O Q

44 (b)in the long run, what fracton of customers wll buy the respectve brands? We need the staonary dstrbuton, F HG I J K J b L NM L g NM / 4 / 4 / / / 4 / 4 / Solve ths, along wth the constrant that (must be somewhere). From () equaton ( / 4) / 4 Usng () or () / / / Substtutng ths nto, we have / 7 / And / 7, / 7, / 7 O Q We therefor conclude that the store should stock more B than A or C coffee O Q

45 Dscrete-Tme Markov Chans The Markov chan Assumptons Let Xt denote the number n stock at the end of the t th month. If Xt falls below a fxed crtcal level, more dsks are ordered. Xt depends on two quanttes: Xt and the number sold between tme t and tme t. Snce the number sold s random, X t does not completely determne Xt. But the basc assumpton s that gven the value of Xt, n tryng to predct what X t wll be, all the past records X t-,x t-,,x are rrelevant; Xt depends only on the number Xt now n stock at tme t and the number sold durng ths month. Ths s the Markov assumpton that the future s ndependent of the past gven the state of affars n the present. Stochastc rocess { X } t. The locaton X t t s measured only at the dscrete tmes t,,,. X s the startng locaton at tme.

46 Markov Chan Assumptons Let, j, t, S.Then for any tme t,, ( X j X, X,, X ) ( X j X ) t t t t t t That s, the future (tme t), gven the present (tme t), s ndependent of the past (tmes t-,,). The probablty above s the transton or jump probablty from state to state j. ( X t j X t ) j, Ths assumpton s called homogenety n tme. A (homogeneous) Markov chan n dscrete tme conssts of a partcle that jumps at each unt of tme between states n a state space S. X t denotes the state occuped at tme t for t,,,. If the partcle s n state at tme t, t wll be n state j at tme t regardless of the states occuped before tme t wth probablty ( X j X ), j t t

47 Andre Andreevch Markov (856-9) was Chebyshev's most outstandng student. Markov was most nvolved n provng lmt theorem and the law of large numbers of Chapter. Although Markov orgnally used Markov chans n abstract settngs to prove varous theorems, they were quckly appled. Markov hmself llustrated the deas by examnng vowel-consonant nterchanges n, letters of ushkn's poetry.

48 Example Let S (, ) and the transton probabltes be 4,,,, Fgure, below, depcts ths. 4 / /4.. / /4 Fgure

49 A table or matrx that dsplays the jump probabltes s / / 4 / / 4 There s a standard way of wrtng the jump probabltes, j as a table. Ths s called the transton matrx. The element n the th row of and jth column s, j - the probablty to that the partcle jumps from to j.,,,,,,, N, N N, N, N, N, N

50 Usng matrx termnology, the j entry of, s, j. Note that the th row of dsplays the jump probabltes from state ; the jth column dsplays the jump probabltes to state j. For example, f the thrd column conssted of all 's, that s, f, for all states n the state space S, the partcle could never enter state, then., > Let be a Markov chan transton matrx. Then., j for all,j n the state space S.. has row sums : N j N, j t t j ( X j X )

51 Example In ths chan there are three states; S{,,}. From state the partcle jumps to states or wth equal probablty /. From state the partcle must next jump to state. State s absorbng; that s, once the partcle enters state, t cannot leave. Draw the dagram and wrte down the transton matrx.. /.. / Fgure

52 Soluton: Fg. depcts the transton probabltes. The zeroth row of conssts of the jump probabltes from state and smlarly for the other two rows. Check that State s absorbng f,.

53 Example: A Random Walk on S {,,,,N} For N., p, q, j for j ± Case. Both absorbng,, N, N q p q p q p r -r q p p s -s N- N Fgure

54 Case. Reflectng,,, N N Case. artally reflectng, as depcted n Fg. s s r r N N N N,,,, s -s p q p q p q -r r

55 Example A Renewal rocess. The state space s S{,,, } and the state of the system s the age of the component presently nstalled. q p... q p... q p Note that the state space S, whle dscrete, has nfntely many states. (Ths model s also called a brth or dsaster model

56 THE T TH -ORDER TRANSITION MATRIX ( X j X ) t t. The one-step probabltes, j are the entres of the matrx. From these, how can one fnd the three-step probabltes, and more generally the t-step probabltes? Defnton The t th t -order transton matrx s, whose j th entry s ( X j X ) t, j t whch s the probablty of jumpng from to j n t steps.

57 Homogenety n tme mples that regardless of the tme u, ( ) t j u u t X j X, To fnd the (t) st -order transton matrx from the t th, use the basc Markov assumptons: ( ) ( ) ( ) ( ) ( ) ( ) N k j k t k N k t k j k t N k t t t N k t t N k t t t t j X k X k X j X X k X X k X j X X k X j X X j X,,,,, and and ( ) ( ) ( ) C B C B A C B A

58 t t Generalzng: t ( ) t t t Chapman-Kolmogorov Equaton Let tmes t, s. Then for all states,j t s, j N k t, k s k, j In terms of matrx multplcaton, the (ts)thorder transton matrx s the product of the tth and the sth: t s t s

59 Example Convert the jump probablty dagram of Fg. 4 nto the correspondng Markov chan and fnd the probablty that the partcle wll be state after three jumps gven t started n state Fgure 4

60 Soluton Consequently, ( X X ).,

61 The robablty Vector t ρ Let the ntal probablty vector be defned ρ ρ, ρ,, ( ) Note that ρ for all states n the state space S and ρ ρ ρ ρ N N t t t ρ, ρ, ρ t N ( ) ρ, t where ρ ( X j ntal probablty vector ρ ) That s, and j t t ρ j s the chance that the partcle wll be found n state j gven that at tme t started n ρ ρ That s, for each t, N ρ t j N ( X j) t j j t ρ s a probablty vector.

62 Defnton A probablty vector ( ) ρ N ρ ρ ρ,,, satsfes. ρ for each,,,n. N ρ ρ ρ N t j t j, ρ ρ Expressed as multplcaton between a vector and a matrx, the probablty vector at tme t s t t t ρ ρ ρ ( ) ( ) ( ) N t j N t t t j X X j X j X, ρ ρ

63 Example For the Markov chan n Fg. 5 fnd the chance that the partcle wll be n state at tme f t started n state wth probablty / and n state wth probablty / at tme. Soluton and Consequently 4 ρ 4 ( /, / )

64 ρ ρ /4, ( X ) ρ 9, /4 Fgure 5

65 ρ (,,,,,,) th entry th entry ρ t (,,,,,,) t t t t (,,,,,),,, t, N t t ( X t j) ρ j, j

66 The Steady State robablty Vector N j j n n j, N j j j n n n n, N j j j, that s, f. and n general, t t

67 rocedure for Fndng the Steady-State robablty Vector. Set up and solve these equatons: j N, j for j,,,,n or alternatvely, n matrx notaton. Normalze by nsstng that N

68 Example Fnd the steady-state probablty vector of the Markov chan n Fg. 6 Soluton / / / ( ) (, ) (, ), /.. Fgure 6 whch yelds two equatons n the two unknowns and :

69 Thus. A smulaton was run n whch 66 partcles were started n state and n state. Each jumped accordng to the transton probabltes. n Fracton n state

70 Example Consder the Markov chan n Fg. 7 Fnd the steady-state probablty vector. /.. / / / / /. Fgure 7

71 Soluton ( ) ( ) / / / / / /,,,, / / / / / / results n these equatons: Clearng the fractons and combnng factors of each : 6 4

72 Addng the second and thrd equatons results n 6 whch s essentally the same as the frst equaton. Hence the thrd equaton s redundant. From the frst two equatons 5 ( 4 ) Now use the normalzaton condton 5 Consequently, 5 5 6

73 Example Consder the cyclc process depcted n Fg. 8. There are N states,,,,n. For each state, <q <. The partcle remans n state wth probablty q. Otherwse, t jumps to the next state wth probablty p -q. If N, then s taken to be state. That s, there s a "wraparound" from state N to state. q q q q N p p p N Fgure 8 Solve for the steady-state probablty vector.

74 Soluton p N q N p q p q p q mples that ( ) ( ) N N,,,,,, q p q p p q N N or p p p p p p N N

75 N N p p p p p p p p p p p p Normalzng p p p p p p p p p p N N N Let p N p p p C Then Cp, whch determnes.

76 For the cyclc process, Cp where C N j p j

77 Example Let / / / / / / / Here are two dstnct steady-state probablty vectors:,,,, ψ (,,,, ) These can be verfed smply by checkng that and ψ ψ.

78 Example Consder a random walk on the ntegers S{,,, } n whch transtons only to the rght can occur as n Fg. 9. Assume <p< and q-p. q q q p p p Fgure 9

79 p q p q p q mples that q p q p q

80 Example Wth / / s a steady-state probablty vector, and n fact t s the only one. For a large number of partcles, there s a steady state: If n/ start n state and n/4 n each of states and, there wll be approxmately ths many n each of the states at all subsequent tmes. But for one partcle, there s no steady state. If t starts n state, t must jump to {,}, from whch. At even tmes t wll be n state ; at odd tmes n ether or. There s a perodcty problem. t s straghtforward verfcaton to check that ( /, / 4, / 4)

81 Brth and Death rocesses n Dscrete Tme: Random Walks Defnton A brth and death process n dscrete tme s a Markov chan wth ths property: From any state S transtons n one jump to -,, or only can occur. Set b r d The transton matrx s b, r, d, r b d r b d r b d N rn

82

83 r r r N. b d b d N.. - Fgure N- N. The number of these that wll jump next to state s therefore ( n ) b. But n the steady state ths must be compensated for an equal number of partcles movng down from. Ths s ( n ) d. Otherwse, partcles would "bunch up" to the left or to the rght of state and the system would not be n the steady state. Consequently (after cancelng the factor n), d b

84 rocedure for fndng the Steady-State robablty Vector Use the equaton b d to solve for each normalzaton n terms of. Then use N

85 Example Solve for for the Markov chan n Fg. / / /4 / / 4/ Fgure Soluton Usng the procedure / 4 / / / / / d b d b d b / 5/ /4

86 Use normalzaton Thus

87 Example: The Random Walk on {,,N} wth artally Reflectng Boundares For,,N-, b b, r r, and d d are constant as n Fg.. b b and d N d. -b r -d b d b d..... N Fgure Note that b b for,,,n- and d d for,,,n. Thus b b d d for,,,n-. Usng ths relaton recursvely, b b b d d d

88 Use normalzaton N N d b d b d b There are two cases: Case : The asymmetrc random walks, d b. Then ths normalzaton sum s a geometrc seres whose sum we know from Secton 4.. ( ) / / d b d b N Wth constant ( ) N C d b d b d b C N,,,, for / /

89 Case : The symmetrc random walk, bd. Then the normalzaton sum s ( ) ( ) N N Consequently, N d b N

90 STEADY-STATE ROBABILITY State b/d b/d. b/d.5b/d b/d

91 Aperodcty, Indecomposablty, and the Approach to the Steady State Defnton Suppose that the sets A,A,,A r partton the state space S. That s, suppose that S r A dsjontly. Assume that a transton from A to A must occur n one jump (where r s taken to be : The partcle "wraps around" to A from A r )- pctorally A A A r That s, f at tme t the partcle s n one of the states of A, then at tme t t must be n one of the states of A. Then the Markov chan s sad to be perodc wth perod r. If the chan s not perodc, t s sad to be aperodc.

92 Example (a) The random walk nfnte n both drectons s perodc wth perod : Let A set of even ntegers and A set of odd ntegers. Then A A A A (b) The random walk wth a boundary at s perodc wth perod f the reflectng probablty at s r. If r<, then the chan s aperodc. Ths s the case snce state can be n one and only one partton set. (c) The random walk wth completely reflectng boundares at and N s perodc wth perod just as n part (a). In lookng for condtons mplyng exstence of a steady state, we ought to exclude perodc chans. Fortunately, there's an easy crteron to use to determne f a Markov chan s aperodc: Suppose that k, k > for any state k. Then the chan s aperodc.

93 Ths may not always apply: that s, there are chans wth k, k for all states k whch are perodc. But t s easy to spot entres k, k along the man dagonal of whch are nonzero; f there are any, the chan must be aperodc. To see why the boxed result s true, suppose that A,,A r s a perodc decomposton of the state space S; thus the A 's are dsjont. But k A for some : f then k A, whch contradcts the fact that A A /. k, k >, Defnton A Markov chan s ndecomposable f for every par,j of states, ether j or j or both.

94 Example Example The random walk on S{,,,,N} wth absorbng barrers and N s not ndecomposable snce nether N nor N. If N s partally reflectng, the chan s ndecomposable snce N, although N stll. Is the chan n Fg., (a) aperodc? (b) ndecomposable? (The arrows ndcate postve transton probabltes.) Soluton (a) Note that for all states. Let us start at state and see where t leads:,,,4,,,4,,, 4 { } { } { } { } { } { } where the next set s obtaned by placng nto t all the states that the states n the prevous set can jump to n one transton. Thus the chan s aperodc Fgure (b) The chan s also ndecomposable snce for all pars,j ether even though for every. j or j

95 Assume that the vector equaton Lmt Theorem has a soluton that s a probablty vector wth < < for all S N Then the followng hold:. s unque: t s the only probablty vector satsfyng.. Regardless of the ntal probablty vector ρ, t t ρ ρ t as.. If the ntal probablty vector s, then for all t, t t ( X t j ntal vector ρ) j as t for all S for all t and all j S. j. And property states ( X t j ntal vector ) j

96 Example The general Two-State rocess of Fg. 4 α α β β Case : α and β. The partcle wll not move; the chan s decomposable. Case : α and β. The partcle jumps back and forth; the chan s perodc wth perod. Case : Ether < α < or < β < or both. α α (, ) (, ) β β mples that α β ( ) ( β ) α Both of these are the same equaton: α β Usng normalzaton, α β

97 Therefore, β α β α α β α β.. α α β Fgure 4

98 Corollary to the Lmt Theorem Under the same assumptons, the tth-order transton t matrx has a lmt matrx lm t t M where M has constant rows each equal to the steady-state probablty vector.

99 Example For the Markov chan of the example earler, successve powers of the transton matrx yeld / / / / / / whch agrees exactly to four sgnfcant dgts wth the steady-state probablty vector found n another example (that s, each row of t approaches ).

100 Example Consder ths cyclc process - a specal case, see Fg. 5 / 9 / / 4 / 4 5 / 7 / 7 Usng the notaton of Example 9 yelds 9 5 p p p 4 7 C p p p 6.5 ( ) Cp Cp Cp

101 Computng powers successvely mples

102 A fnal note about the steady-state probablty vector : / /4 9/.. 5/7 /4. /7 Fgure 5 Exstence Theorem An aperodc and ndecomposable Markov chan wth fnte state space has one and only one steady-state probablty vector.