Continuous-time Markov Chains

Transcription

1 Continuous-time Markov Chains Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester October 31, 2016 Introduction to Random Processes Continuous-time Markov Chains 1

2 Continuous-time Markov chains Continuous-time Markov chains Transition probability function Determination of transition probability function Limit probabilities and ergodicity Introduction to Random Processes Continuous-time Markov Chains 2

3 Definition Continuous-time positive variable t [0, ) Time-dependent random state X (t) takes values on a countable set In general denote states as i = 0, 1, 2,..., i.e., here the state space is N If X (t) = i we say the process is in state i at time t Def: Process X (t) is a continuous-time Markov chain (CTMC) if P ( X (t + s) = j X (s) = i, X (u) = x(u), u < s ) = P ( X (t + s) = j X (s) = i ) Markov property Given the present state X (s) Future X (t + s) is independent of the past X (u) = x(u), u < s In principle need to specify functions P ( X (t + s) = j X (s) = i ) For all times t and s, for all pairs of states (i, j) Introduction to Random Processes Continuous-time Markov Chains 3

4 Notation and homogeneity Notation X [s : t] state values for all times s u t, includes borders X (s : t) values for all times s < u < t, borders excluded X (s : t] values for all times s < u t, exclude left, include right X [s : t) values for all times s u < t, include left, exclude right Homogeneous CTMC if P ( X (t + s) = j ) X (s) = i invariant for all s We restrict consideration to homogeneous CTMCs Still need P ij (t) := P ( X (t + s) = j ) X (s) = i for all t and pairs (i, j) P ij (t) is known as the transition probability function. More later Markov property and homogeneity make description somewhat simpler Introduction to Random Processes Continuous-time Markov Chains 4

5 Transition times T i = time until transition out of state i into any other state j Def: T i is a random variable called transition time with ccdf P (T i > t) = P ( X (0 : t] = i X (0) = i ) Probability of T i > t + s given that T i > s? Use cdf expression P ( T i > t + s Ti > s ) = P ( X (0 : t + s] = i X [0 : s] = i ) = P ( X (s : t + s] = i X [0 : s] = i ) = P ( X (s : t + s] = i X (s) = i ) = P ( X (0 : t] = i X (0) = i ) Used that X [0 : s] = i given, Markov property, and homogeneity From definition of T i P ( T i > t + s Ti > s ) = P (T i > t) Transition times are exponential random variables Introduction to Random Processes Continuous-time Markov Chains 5

6 Alternative definition Exponential transition times is a fundamental property of CTMCs Can be used as algorithmic definition of CTMCs Continuous-time random process X (t) is a CTMC if (a) Transition times T i are exponential random variables with mean 1/ν i (b) When they occur, transition from state i to j with probability P ij P ij = 1, P ii = 0 j=1 (c) Transition times T i and transitioned state j are independent Define matrix P grouping transition probabilities P ij CTMC states evolve as in a discrete-time Markov chain State transitions occur at exponential intervals T i exp(ν i ) As opposed to occurring at fixed intervals Introduction to Random Processes Continuous-time Markov Chains 6

7 Embedded discrete-time Markov chain Consider a CTMC with transition matrix P and rates ν i Def: CTMC s embedded discrete-time MC has transition matrix P Transition probabilities P describe a discrete-time MC No self-transitions (P ii = 0, P s diagonal null) Can use underlying discrete-time MCs to study CTMCs Def: State j accessible from i if accessible in the embedded MC Def: States i and j communicate if they do so in the embedded MC Communication is a class property Recurrence, transience, ergodicity. Class properties... More later Introduction to Random Processes Continuous-time Markov Chains 7

8 Transition rates Expected value of transition time T i is E [T i ] = 1/ν i Can interpret ν i as the rate of transition out of state i Of these transitions, a fraction P ij are into state j Def: Transition rate from i to j is q ij := ν i P ij Transition rates offer yet another specification of CTMCs If q ij are given can recover ν i as ν i = ν i P ij = ν i P ij = j=1 j=1 j=1 ( ) 1 Can also recover P ij as P ij = q ij /ν i = q ij q ij j=1 q ij Introduction to Random Processes Continuous-time Markov Chains 8

9 Birth and death process example State X (t) = 0, 1,... Interpret as number of individuals Birth and deaths occur at state-dependent rates. When X (t) = i Births Individuals added at exponential times with mean 1/λ i Birth or arrival rate = λ i births per unit of time Deaths Individuals removed at exponential times with rate 1/µ i Death or departure rate = µ i deaths per unit of time Birth and death times are independent Birth and death (BD) processes are then CTMCs Introduction to Random Processes Continuous-time Markov Chains 9

10 Transition times and probabilities Q: Transition times T i? Leave state i 0 when birth or death occur If T B and T D are times to next birth and death, T i = min(t B, T D ) Since T B and T D are exponential, so is T i with rate ν i = λ i + µ i When leaving state i can go to i + 1 (birth first) or i 1 (death first) Birth occurs before death with probability = P i,i+1 λ i + µ i µ i Death occurs before birth with probability = P i,i 1 λ i + µ i Leave state 0 only if a birth occurs, then ν 0 = λ 0, P 01 = 1 If CTMC leaves 0, goes to 1 with probability 1 Might not leave 0 if λ 0 = 0 (e.g., to model extinction) Introduction to Random Processes Continuous-time Markov Chains 10 λ i

11 Transition rates Rate of transition from i to i + 1 is (recall definition q ij = ν i P ij ) q i,i+1 = ν i P i,i+1 = (λ i + µ i ) = λ i λ i + µ i Likewise, rate of transition from i to i 1 is q i,i 1 = ν i P i,i 1 = (λ i + µ i ) = µ i λ i + µ i λ i µ i For i = 0 q 01 = ν 0 P 01 = λ 0 0 λ 0 λ i 1 λ i λ i+1... i 1 i i µ 1 µ i µ i+1 Somewhat more natural representation. Similar to discrete-time MCs Introduction to Random Processes Continuous-time Markov Chains 11

12 Poisson process example A Poisson process is a BD process with λ i = λ and µ i = 0 constant State N(t) counts the total number of events (arrivals) by time t Arrivals occur a rate of λ per unit time Transition times are the i.i.d. exponential interarrival times 0 λ λ λ λ... i 1 i i The Poisson process is a CTMC Introduction to Random Processes Continuous-time Markov Chains 12

13 M/M/1 queue example An M/M/1 queue is a BD process with λ i = λ and µ i = µ constant State Q(t) is the number of customers in the system at time t Customers arrive for service at a rate of λ per unit time They are serviced at a rate of µ customers per unit time 0 λ λ λ λ... i 1 i i µ µ µ The M/M is for Markov arrivals/markov departures Implies a Poisson arrival process, exponential services times The 1 is because there is only one server Introduction to Random Processes Continuous-time Markov Chains 13

14 Transition probability function Continuous-time Markov chains Transition probability function Determination of transition probability function Limit probabilities and ergodicity Introduction to Random Processes Continuous-time Markov Chains 14

15 Transition probability function Two equivalent ways of specifying a CTMC 1) Transition time averages 1/ν i + transition probabilities P ij Easier description Typical starting point for CTMC modeling 2) Transition probability function P ij (t) := P ( X (t + s) = j ) X (s) = i More complete description for all t 0 Similar in spirit to Pij n for discrete-time Markov chains Goal: compute P ij (t) from transition times and probabilities Notice two obvious properties P ij (0) = 0, P ii (0) = 1 Introduction to Random Processes Continuous-time Markov Chains 15

16 Roadmap to determine P ij (t) Goal is to obtain a differential equation whose solution is P ij (t) Study change in P ij (t) when time changes slightly Separate in two subproblems (divide and conquer) Transition probabilities for small time h, P ij (h) Transition probabilities in t + h as function of those in t and h We can combine both results in two different ways 1) Jump from 0 to t then to t + h Process runs a little longer Changes where the process is going to Forward equations 2) Jump from 0 to h then to t + h Process starts a little later Changes where the process comes from Backward equations Introduction to Random Processes Continuous-time Markov Chains 16

17 Transition probability in infinitesimal time Theorem The transition probability functions P ii (t) and P ij (t) satisfy the following limits as t approaches 0 P ij (t) 1 P ii (t) lim = q ij, lim = ν i t 0 t t 0 t Since P ij (0) = 0, P ii (0) = 1 above limits are derivatives at t = 0 P ij (t) P ii (t) t = q ij, t=0 t = ν i t=0 Limits also imply that for small h (recall Taylor series) P ij (h) = q ij h + o(h), P ii (h) = 1 ν i h + o(h) Transition rates q ij are instantaneous transition probabilities Transition probability coefficient for small time h Introduction to Random Processes Continuous-time Markov Chains 17

18 Probability of event in infinitesimal time (reminder) Q: Probability of an event happening in infinitesimal time h? Want P (T < h) for small h Equivalent to P (T < h) = P (T < t) t h 0 λe λt dt λh = λ t=0 Sometimes also write P (T < h) = λh + o(h) o(h) o(h) implies lim = 0 h 0 h Read as negligible with respect to h Q: Two independent events in infinitesimal time h? P (T 1 h, T 2 h) (λ 1 h)(λ 2 h) = λ 1 λ 2 h 2 = o(h) Introduction to Random Processes Continuous-time Markov Chains 18

19 Transition probability in infinitesimal time (proof) Proof. Consider a small time h, and recall T i exp(ν i ) Since 1 P ii (h) is the probability of transitioning out of state i 1 P ii (h) = P (T i < h) = ν i h + o(h) Divide by h and take limit to establish the second identity For P ij (t) notice that since two or more transitions have o(h) prob. P ij (h) = P ( X (h) = j X (0) = i ) = P ij P (T i < h) + o(h) Again, since T i is exponential P (T i < h) = ν i h + o(h). Then P ij (h) = ν i P ij h + o(h) = q ij h + o(h) Divide by h and take limit to establish the first identity Introduction to Random Processes Continuous-time Markov Chains 19

20 Chapman-Kolmogorov equations Theorem For all times s and t the transition probability functions P ij (t + s) are obtained from P ik (t) and P kj (s) as P ij (t + s) = P ik (t)p kj (s) k=0 As for discrete-time MCs, to go from i to j in time t + s Go from i to some state k in time t P ik (t) In the remaining time s go from k to j Sum over all possible intermediate states k P kj (s) Introduction to Random Processes Continuous-time Markov Chains 20

21 Chapman-Kolmogorov equations (proof) Proof. P ij (t + s) = P ( X (t + s) = j X (0) = i ) Definition of P ij (t + s) = P ( X (t + s) = j ) ( ) X (t) = k, X (0) = i P X (t) = k X (0) = i k=0 Law of total probability = P ( X (t + s) = j X (t) = k ) P ik (t) Markov property of CTMC k=0 and definition of P ik (t) = P kj (s)p ik (t) Definition of P kj (s) k=0 Introduction to Random Processes Continuous-time Markov Chains 21

22 Combining both results Let us combine the last two results to express P ij (t + h) Use Chapman-Kolmogorov s equations for 0 t h P ij (t + h) = P ik (t)p kj (h) = P ij (t)p jj (h) + k=0 k=0,k j P ik (t)p kj (h) Substitute infinitesimal time expressions for P jj (h) and P kj (h) P ij (t + h) = P ij (t)(1 ν j h) + k=0,k j P ik (t)q kj h + o(h) Subtract P ij (t) from both sides and divide by h P ij (t + h) P ij (t) = ν j P ij (t) + P ik (t)q kj + o(h) h h k=0,k j Right-hand side equals a derivative ratio. Let h 0 to prove... Introduction to Random Processes Continuous-time Markov Chains 22

23 Kolmogorov s forward equations Theorem The transition probability functions P ij (t) of a CTMC satisfy the system of differential equations (for all pairs i, j) P ij (t) t = k=0,k j q kj P ik (t) ν j P ij (t) Interpret each summand in Kolmogorov s forward equations Pij (t)/ t = rate of change of P ij (t) qkj P ik (t) = (transition into k in 0 t) (rate of moving into j in next instant) νj P ij (t) = (transition into j in 0 t) (rate of leaving j in next instant) Change in P ij (t) = k (moving into j from k) (leaving j) Kolmogorov s forward equations valid in most cases, but not always Introduction to Random Processes Continuous-time Markov Chains 23

24 Kolmogorov s backward equations For forward equations used Chapman-Kolmogorov s for 0 t h For backward equations we use 0 h t to express P ij (t + h) as P ij (t + h) = P ik (h)p kj (t) = P ii (h)p ij (t) + k=0 k=0,k i Substitute infinitesimal time expression for P ii (h) and P ik (h) P ij (t + h) = (1 ν i h)p ij (t) + k=0,k i P ik (h)p kj (t) q ik hp kj (t) + o(h) Subtract P ij (t) from both sides and divide by h P ij (t + h) P ij (t) = ν i P ij (t) + q ik P kj (t) + o(h) h h k=0,k i Right-hand side equals a derivative ratio. Let h 0 to prove... Introduction to Random Processes Continuous-time Markov Chains 24

25 Kolmogorov s backward equations Theorem The transition probability functions P ij (t) of a CTMC satisfy the system of differential equations (for all pairs i, j) P ij (t) t = k=0,k i q ik P kj (t) ν i P ij (t) Interpret each summand in Kolmogorov s backward equations Pij (t)/ t = rate of change of P ij (t) qik P kj (t) = (transition into j in h t) (rate of transition into k in initial instant) νi P ij (t) = (transition into j in h t) (rate of leaving i in initial instant) Forward equations change in P ij (t) if finish h later Backward equations change in P ij (t) if start h earlier Where process goes (forward) vs. where process comes from (backward) Introduction to Random Processes Continuous-time Markov Chains 25

26 Determination of transition probability function Continuous-time Markov chains Transition probability function Determination of transition probability function Limit probabilities and ergodicity Introduction to Random Processes Continuous-time Markov Chains 26

27 A CTMC with two states Ex: Simplest possible CTMC has only two states. Say 0 and 1 Transition rates are q 01 and q 10 Given q 01 and q 10 can find rates of transitions out of {0, 1} q ν 0 = j q 0j = q 01, ν 1 = j q 1j = q 10 q 10 Use Kolmogorov s equations to find transition probability functions P 00 (t), P 01 (t), P 10 (t), P 11 (t) Transition probabilities out of each state sum up to one P 00 (t) + P 01 (t) = 1, P 10 (t) + P 11 (t) = 1 Introduction to Random Processes Continuous-time Markov Chains 27

28 Kolmogorov s forward equations Kolmogorov s forward equations (process runs a little longer) P ij(t) = q kj P ik (t) ν j P ij (t) For the two state CTMC k=0,k j P 00(t) = q 10 P 01 (t) ν 0 P 00 (t), P 10(t) = q 10 P 11 (t) ν 0 P 10 (t), P 01(t) = q 01 P 00 (t) ν 1 P 01 (t) P 11(t) = q 01 P 10 (t) ν 1 P 11 (t) Probabilities out of 0 sum up to 1 eqs. in first row are equivalent Probabilities out of 1 sum up to 1 eqs. in second row are equivalent Pick the equations for P 00 (t) and P 11 (t) Introduction to Random Processes Continuous-time Markov Chains 28

29 Solution of forward equations Use Relation between transition rates: ν 0 = q 01 and ν 1 = q 10 Probs. sum 1: P 01 (t) = 1 P 00 (t) and P 10 (t) = 1 P 11 (t) P 00(t) = q 10 [ 1 P00 (t) ] q 01 P 00 (t) = q 10 (q 10 + q 01 )P 00 (t) P 11(t) = q 01 [ 1 P11 (t) ] q 10 P 11 (t) = q 01 (q 10 + q 01 )P 11 (t) Can obtain exact same pair of equations from backward equations First-order linear differential equations Solutions are exponential For P 00 (t) propose candidate solution (just differentiate to check) P 00 (t) = q 10 q 10 + q 01 + ce (q10+q01)t To determine c use initial condition P 00 (0) = 1 Introduction to Random Processes Continuous-time Markov Chains 29

30 Solution of forward equations (continued) Evaluation of candidate solution at initial condition P 00 (0) = 1 yields 1 = q 10 q 10 + q 01 + c c = Finally transition probability function P 00 (t) P 00 (t) = q 01 q 10 + q 01 q 10 q 01 + e (q10+q01)t q 10 + q 01 q 10 + q 01 Repeat for P 11 (t). Same exponent, different constants P 11 (t) = q 01 q 10 + e (q10+q01)t q 10 + q 01 q 10 + q 01 As time goes to infinity, P 00 (t) and P 11 (t) converge exponentially Convergence rate depends on magnitude of q 10 + q 01 Introduction to Random Processes Continuous-time Markov Chains 30

31 Convergence of transition probabilities Recall P 01 (t) = 1 P 00 (t) and P 10 (t) = 1 P 11 (t) Limiting (steady-state) probabilities are lim P 00(t) = t lim P 11(t) = t q 10, q 10 + q 01 q 01, q 10 + q 01 lim P 01(t) = t lim P 10(t) = t q 01 q 10 + q 01 q 10 q 10 + q 01 Limit distribution exists and is independent of initial condition Compare across diagonals Introduction to Random Processes Continuous-time Markov Chains 31

32 Kolmogorov s forward equations in matrix form Restrict attention to finite CTMCs with N states Define matrix R R N N with elements r ij = q ij, r ii = ν i Rewrite Kolmogorov s forward eqs. as (process runs a little longer) N N P ij(t) = q kj P ik (t) ν j P ij (t) = r kj P ik (t) k=1,k j k=1 Right-hand side defines elements of a matrix product P(t) = P 11(t) P 1k (t) P 1N (t) P i1 (t) P ik (t) P in (t) P N1 (t) P Nk (t) P JN (t) r 1j P i1 (t) r kj P ik (t) r Nj P in (t) r 11 r 1j r 1N r k1 r kj r kn r N1 r Nj r NN s 11 s 1j s 1N s i1 s ij s in s N1 s Nk s NN = R = P(t)R = P (t) Introduction to Random Processes Continuous-time Markov Chains 32

33 Kolmogorov s backward equations in matrix form Similarly, Kolmogorov s backward eqs. (process starts a little later) N N P ij(t) = q ik P kj (t) ν i P ij (t) = r ik P kj (t) k=1,k i k=1 Right-hand side also defines a matrix product R = r i1 P 1j (t) r ik P kj (t) r 11 r 1k r 1N r i1 r ik r in r N1 r Nk r JN r in P Nj (t) P 11(t) P 1j (t) P 1N (t) P k1 (t) P kj (t) P kn (t) P N1 (t) P Nj (t) P NN (t) s 11 s 1j s 1N s i1 s ij s in s N1 s Nk s NN = P(t) = RP(t) = P (t) Introduction to Random Processes Continuous-time Markov Chains 33

34 Kolmogorov s equations in matrix form Matrix form of Kolmogorov s forward equation P (t) = P(t)R Matrix form of Kolmogorov s backward equation P (t) = RP(t) More similar than apparent But not equivalent because matrix product not commutative Notwithstanding both equations have to accept the same solution Introduction to Random Processes Continuous-time Markov Chains 34

35 Matrix exponential Kolmogorov s equations are first-order linear differential equations They are coupled, P ij (t) depends on P kj(t) for all k Accepts exponential solution Define matrix exponential Def: The matrix exponential e At of matrix At is the series e At (At) n = n! n=0 = I + At + (At)2 2 + (At) Derivative of matrix exponential with respect to t e At t = 0 + A + A 2 t + A3 t 2 2 ) +... = A (I + At + (At) = Ae At 2 Putting A on right side of product shows that eat t = e At A Introduction to Random Processes Continuous-time Markov Chains 35

36 Solution of Kolmogorov s equations Propose solution of the form P(t) = e Rt P(t) solves backward equations, since derivative is P(t) t = ert t = Re Rt = RP(t) It also solves forward equations P(t) t = ert t = e Rt R = P(t)R Notice that P(0) = I, as it should (P ii (0) = 1, and P ij (0) = 0) Introduction to Random Processes Continuous-time Markov Chains 36

37 Computing the matrix exponential Suppose A R n n is diagonalizable, i.e., A = UDU 1 Diagonal matrix D = diag(λ 1,..., λ n ) collects eigenvalues λ i Matrix U has the corresponding eigenvectors as columns We have the following neat identity ( e At (UDU 1 t) n ) (Dt) n = = U U 1 = Ue Dt U 1 n! n! n=0 n=0 But since D is diagonal, then e Dt (Dt) n = n! n=0 e λ1t... 0 = e λnt Introduction to Random Processes Continuous-time Markov Chains 37

38 Two state CTMC example Ex: Simplest CTMC with two states 0 and 1 Transition rates are q 01 = 3 and q 10 = 1 q Recall transition time rates are ν 0 = q 01 = 3, ν 1 = q 10 = 1, hence ( ) ( ) ν0 q R = = q 10 ν Eigenvalues of R are 0, 4, eigenvectors [1, 1] T and [ 3, 1] T. Thus ( ) ( ) ( ) 1 3 U =, U 1 1/4 3/4 =, e Dt 1 0 = 1 1 1/4 1/1 0 e 4t The solution to the forward equations is ( ) P(t) = e Rt = Ue Dt U 1 1/4 + (3/4)e 4t 3/4 (3/4)e = 4t 1/4 (1/4)e 4t 3/4 + (1/4)e 4t q 10 Introduction to Random Processes Continuous-time Markov Chains 38

39 Unconditional probabilities P(t) is transition prob. from states at time 0 to states at time t Define unconditional probs. at time t, p j (t) := P (X (t) = j) Group in vector p(t) = [p 1 (t), p 2 (t),..., p j (t),...] T Given initial distribution p(0), find p j (t) conditioning on initial state p j (t) = P ( X (t) = j ) X (0) = i P (X (0) = i) = P ij (t)p i (0) i=0 i=0 Using compact matrix-vector notation p(t) = P T (t)p(0) Compare with discrete-time MC p(n) = (P n ) T p(0) Introduction to Random Processes Continuous-time Markov Chains 39

40 Limit probabilities and ergodicity Continuous-time Markov chains Transition probability function Determination of transition probability function Limit probabilities and ergodicity Introduction to Random Processes Continuous-time Markov Chains 40

41 Recurrent and transient states Recall the embedded discrete-time MC associated with any CTMC Transition probs. of MC form the matrix P of the CTMC No self transitions (P ii = 0, P s diagonal null) States i j communicate in the CTMC if i j in the MC Communication partitions MC in classes Induces CTMC partition as well Def: CTMC is irreducible if embedded MC contains a single class State i is recurrent if it is recurrent in the embedded MC Likewise, define transience and positive recurrence for CTMCs Transience and recurrence shared by elements of a MC class Transience and recurrence are class properties of CTMCs Periodicity not possible in CTMCs Introduction to Random Processes Continuous-time Markov Chains 41

42 Limiting probabilities Theorem Consider irreducible, positive recurrent CTMC with transition rates ν i and q ij. Then, lim P ij(t) exists and is independent of the initial state i, i.e., t P j = lim t P ij(t) exists for all (i, j) Furthermore, steady-state probabilities P j 0 are the unique nonnegative solution of the system of linear equations ν j P j = q kj P k, k=0,k j P j = 1 j=0 Limit distribution exists and is independent of initial condition Obtained as solution of system of linear equations Like discrete-time MCs, but equations slightly different Introduction to Random Processes Continuous-time Markov Chains 42

43 Algebraic relation to determine limit probabilities As with MCs difficult part is to prove that P j = lim t P ij(t) exists Algebraic relations obtained from Kolmogorov s forward equations P ij (t) t = k=0,k j q kj P ik (t) ν j P ij (t) If limit distribution exists we have, independent of initial state i P ij (t) lim = 0, lim t t P ij(t) = P j t Considering the limit of Kolomogorov s forward equations yields 0 = k=0,k j q kj P k ν j P j Reordering terms the limit distribution equations follow Introduction to Random Processes Continuous-time Markov Chains 43

44 Two state CTMC example Ex: Simplest CTMC with two states 0 and 1 Transition rates are q 01 and q From transition rates find mean transition times ν 0 = q 01, ν 1 = q 10 Stationary distribution equations q 01 q 10 ν 0 P 0 = q 10 P 1, ν 1 P 1 = q 01 P 0, P 0 + P 1 = 1, q 01 P 0 = q 10 P 1, q 10 P 1 = q 01 P 0 Solution yields P 0 = q 10 q 10 + q 01, P 1 = q 01 q 10 + q 01 Larger rate q 10 of entering 0 Larger prob. P 0 of being at 0 Larger rate q 01 of entering 1 Larger prob. P 1 of being at 1 Introduction to Random Processes Continuous-time Markov Chains 44

45 Ergodicity Def: Fraction of time T i (t) spent in state i by time t T i (t) := 1 t t T i (t) a time/ergodic average, 0 I {X (τ) = i}dτ lim T i(t) is an ergodic limit t If CTMC is irreducible, positive recurrent, the ergodic theorem holds P i = lim T 1 t i(t) = lim I {X (τ) = i}dτ t t t 0 a.s. Ergodic limit coincides with limit probabilities (almost surely) Introduction to Random Processes Continuous-time Markov Chains 45

46 Function s ergodic limit Consider function f (i) associated with state i. Can write f ( X (t) ) as f ( X (t) ) = f (i)i {X (t) = i} i=1 Consider the time average of f ( X (t) ) 1 t lim f ( X (τ) ) 1 t dτ = lim t t 0 t t 0 f (i)i {X (τ) = i}dτ i=1 Interchange summation with integral and limit to say 1 t lim f ( X (τ) ) dτ = t t 0 i=1 1 t f (i) lim I {X (τ) = i}dτ = t t 0 f (i)p i Function s ergodic limit = Function s expectation under limiting dist. i=1 Introduction to Random Processes Continuous-time Markov Chains 46

47 Limit distribution equations as balance equations Recall limit distribution equations ν j P j = q kj P k k=0,k j P j = fraction of time spent in state j ν j = rate of transition out of state j given CTMC is in state j ν j P j = rate of transition out of state j (unconditional) q kj = rate of transition from k to j given CTMC is in state k q kj P k = rate of transition from k to j (unconditional) q kj P k = rate of transition into j, from all states k=0,k j Rate of transition out of state j = Rate of transition into state j Balance equations Balance nr. of transitions in and out of state j Introduction to Random Processes Continuous-time Markov Chains 47

48 Limit distribution for birth and death process Birth/deaths occur at state-dependent rates. When X (t) = i Births Individuals added at exponential times with mean 1/λ i Birth rate = upward transition rate = q i,i+1 = λ i Deaths Individuals removed at exponential times with mean 1/µ i Death rate = downward transition rate = q i,i 1 = µ i Transition time rates ν i = λ i + µ i, i > 0 and ν 0 = λ 0 0 λ 0 λ i 1 λ i λ i+1... i 1 i i µ 1 µ i µ i+1 Limit distribution/balance equations: Rate out of j = Rate into j (λ i + µ i )P i = λ i 1 P i 1 + µ i+1 P i+1 λ 0 P 0 = µ 1 P 1 Introduction to Random Processes Continuous-time Markov Chains 48

49 Finding solution of balance equations Start expressing all probabilities in terms of P 0 Equation for P 0 λ0p 0 = µ 1P 1 Sum eqs. for P 1 λ 0P 0 = and P 0 (λ 1 + µ 1)P 1 = Sum result and λ 1P 1 = eq. for P 2 (λ 2 + µ 2)P 2 =. µ 1P 1 λ 0P 0 + µ 2P 2 λ 1P 1 = µ 2P 2 µ 2P 2 λ 1P 1 + µ 3P 3 λ 2P 2 = µ 3P 3 Sum result and eq. for P i λ i 1 P i 1 = (λ i + µ i )P i = µ i P i λ i 1 P i 1 + µ i+1 P i+1 λ i P i = µ i+1p i+1 Introduction to Random Processes Continuous-time Markov Chains 49

50 Finding solution of balance equations (continued) Recursive substitutions on red equations on the right P 1 = λ 0 µ 1 P 0 P 2 = λ 1 µ 2 P 1 = λ 1λ 0 µ 2 µ 1 P 0. P i+1 = λ i µ i+1 P i = λ iλ i 1... λ 0 µ i+1 µ i... µ 1 P 0 To find P 0 use i=0 P i = 1 1 = P 0 + P 0 = [ 1 + i=1 λ i λ i 1... λ 0 µ i+1 µ i... µ 1 ] 1 i=1 λ i λ i 1... λ 0 µ i+1 µ i... µ 1 P 0 Introduction to Random Processes Continuous-time Markov Chains 50

51 Glossary Continuous-time Markov chain Markov property Time-homogeneous CTMC Transition probability function Exponential transition time Transition probabilities Embedded discrete-time MC Transition rates Birth and death process Poisson process M/M/1 queue Chapman-Kolmogorov equations Kolmogorov s forward equations Kolmogorov s backward equations Limiting probabilities Matrix exponential Unconditional probabilities Recurrent and transient states Ergodicity Balance equations Introduction to Random Processes Continuous-time Markov Chains 51