Review of Markov Chain Theory

Transcription

1 Queuig Aalysis: Review of Markov Chai Theory Hogwei Zhag Ackowledgemet: this lecture is partially based o the slides of Dr. Yais A. Korilis.

2 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

3 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

4 Markov Chai? Stochastic process that takes values i a coutable set Example: {0,1,2,,m}, or {0,1,2, } Elemets represet possible states Chai trasits from state to state Memoryless (Markov) Property: Give the preset state, future trasitios of the chai are idepedet of past history Markov Chais: discrete- or cotiuous- time

5 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

6 Discrete-Time Markov Chai Discrete-time stochastic process {X : = 0,1,2, } Takes values i {0,1,2, } Memoryless property: P { X = j X = i, X = i,..., X = i } = P { X = j X = i } P = P{ X = j X = i} ij + 1 Trasitio probabilities P ij P ij 0, P = 1 j= 0 ij Trasitio probability matrix P=[P ij ]

7 Chapma-Kolmogorov Equatios step trasitio probabilities P = P{ X = j X = i},, m 0, i, j 0 ij + m m How to calculate? Chapma-Kolmogorov equatios P ij + m m ij ik kj k= 0 P = P P,, m 0, i, j 0 is elemet (i, j) i matrix P Recursive computatio of state probabilities

8 State Probabilities Statioary Distributio State probabilities (time-depedet) π = P{ X = j}, π = (π,π,...) j 0 1 I matrix form: 1 = = 1 = = 1 = j = i ij i= 0 i= 0 P{ X j} P{ X i} P{ X j X i} π π P π = π P = π P =... = π P If time-depedet distributio coverges to a limit π = limπ π = πp π is called the statioary distributio (or steady state distributio) existece depeds o the structure of Markov chai

9 Classificatio of Markov Chais Irreducible: States i ad j commuicate: Aperiodic: State i is periodic: m, m : P > 0, P > 0 d > 1: P > 0 = αd ij ji Irreducible Markov chai: all Aperiodic Markov chai: oe of states commuicate the states is periodic ii

10 Limit Theorems Theorem 1: Irreducible aperiodic Markov chai For every state j, the followig limit π = lim P{ X = j X = i}, i = 0,1,2,... j exists ad is idepedet of iitial state i N j (k): umber of visits to state j up to time k N j( k) P π j = lim X 0 = i = 1 k k =>π j : frequecy the process visits state j 0

11 Existece of Statioary Distributio Theorem 2: Irreducible aperiodic Markov chai. There are two possibilities for scalars: π = lim P{ X = j X = i} = lim P j 0 ij 1. π j = 0, for all states j No statioary distributio 2. π j > 0, for all states j π is the uique statioary distributio Remark: If the umber of states is fiite, case 2 is the oly possibility

12 Ergodic Markov Chais A state j is positive recurret if the process returs to state j ifiitely ofte Formal defiitio: F ij () ( 1): the probability, give X 0 = i, that state j occurs at some time betwee 1 ad iclusive T ij : the first passage time from i to j A state j is recurret (or persistet) if F jj ( ) = 1, ad trasiet otherwise A state j is positive recurret (or o-ull persistet) if F jj ( ) = 1 ad E(T jj ) < A state j is ull recurret (or ull persistet) if F jj ( ) = 1 but E(T jj ) = Note: positive recurret => irreducible always hold, but irreducible => positive recurret is guarateed to hold oly for fiite MC

13 Ergodic MC (cotd.) Example: a MC with coutably ifiite state space p p p p p p q q = 1-p q q q q q All states are positive recurret if p < ½, ull recurret if p = ½, ad trasiet if p > ½ A state is ergodic if it is aperiodic ad positive recurret A MC is ergodic if every state is ergodic Ergodic chais have a uique statioary distributio π j = 1/E(T jj ), j = 0, 1, 2, Note: Ergodicity Time Averages = Stochastic Averages

14 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

15 Calculatio of Statioary Distributio A. Fiite umber of states B. Ifiite umber of states Solve explicitly the system of Caot apply previous methods to equatios problem of ifiite dimesio m π = π P, j = 0,1,..., m j i ij i = 0 m j ipij πi = 1 i= 0 i= 0 πi = 1 i= 0 Or, umerically from P which coverges to a matrix with rows equal to π Suitable for a small umber of states Guess a solutio to recurrece: π = π, j = 0,1,...,

16 Example: Fiite Markov Chai Abset-mided professor uses two umbrellas whe commutig betwee home ad office. If it rais ad a umbrella is Markov chai formulatio i is the umber of umbrellas available at her curret locatio 1 p available at her locatio, she p takes it. If it does ot rai, she 1 p p always forgets to take a umbrella. Trasitio matrix Let p be the probability of rai P = 0 1 p p each time she commutes. 1 p p 0 Q: What is the probability that she gets wet o ay give day?

17 Example: Fiite Markov Chai 1 p p 1 p p P = 0 1 p p 1 p p 0 π 0 = (1 p)π2 π = πp π = (1 p)π + pπ 1 p 1 1 π,π,π π = 1 p 3 3 p 3 p = 1 = 2 = π2 π0 π i i = + 1 p π0 + π1 + π2 = 1 1 p P{gets wet} = π0 p = p 3 p

18 Example: Fiite Markov Chai Takig p = 0.1: 1 p 1 1 π =,, = 0.310, 0.345, p 3 p 3 p P = ( ) Numerically determie limit of P lim P = ( 150) Effectiveess depeds o structure of P

19 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

20 Global Balace Equatios Global Balace Equatios (GBE) π P = π P π P = π P, j 0 j ji i ij j ji i ij i= 0 i= 0 i j i j P is the frequecy of trasitios from j to i π j ji Frequecy of Frequecy of trasitios out of j = trasitios ito j Ituitio: 1) j visited ifiitely ofte; 2) for each trasitio out of j there must be a subsequet trasitio ito j with probability 1

21 Global Balace Equatios (cotd.) Alterative Form of GBE { } π P = π P, S 0,1,2,... j ji i ij j S i S i S j S If a probability distributio satisfies the GBE, the it is the uique statioary distributio of the Markov chai Fidig the statioary distributio: Guess distributio from properties of the system Verify that it satisfies the GBE Special structure of the Markov chai simplifies task

22 Global Balace Equatios Proof First form: π = π P ad P = 1 j i ij ji i= 0 i= 0 π P = π P π P = π P j ji i ij j ji i ij i= 0 i= 0 i j i j Secod form: π P = π P π P = π P j ji i ij j ji i ij i= 0 i= 0 j S i= 0 j S i= 0 π j Pji + Pji = πipij + πipij j S i S i S j S i S i S π j Pji = πi Pij j S i S i S j S

23 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

24 Birth-Death Process P 01 S P 1, S c P, P 00 P 10 P, 1 P, P + 1, Oe-dimesioal Markov chai with trasitios oly betwee eighborig states: P ij =0, if i-j >1 Detailed Balace Equatios (DBE) π P = π P = 0,1,..., , Proof: GBE with S ={0,1,,} give: π P = π P π P = π P j ji i ij, , j= 0 i= + 1 j= 0 i= + 1

25 Example: Discrete-Time Queue I a time-slot, oe packet arrival with probability p or zero arrivals with probability 1-p I a time-slot, the packet i service departs with probability q or stays with probability 1-q Idepedet arrivals ad service times State: umber of packets i system p p(1 q) p p(1 q) (1 p) q(1 p) q(1 p) (1 p)(1 q) + pq q(1 p) (1 p)(1 q) + pq

26 Example: Discrete-Time Queue (cotd.) p p(1 q) p(1 q) p(1 q) (1 p) q(1 p) q(1 p) (1 p)(1 q) + pq q(1 p) p / q π 0 p = π 1 q (1 p ) π1 = π0 1 p p(1 q) π p(1 q) = π + 1q(1 p) π+ 1 = π, 1 q(1 p) p(1 q) Defie: ρ p / q, α q(1 p) ρ π 1 = π 0 1 ρ 1 p π = α π 0, 1 1 p π+ 1 = απ, 1 (1 p)(1 q) + pq

27 Example: Discrete-Time Queue (cotd.) Havig determied the distributio as a fuctio of π 0 1 ρ π = α π 0, 1 1 p How to calculate the ormalizatio costat π 0? Probability coservatio law: Notig that 1 1 ρ 1 ρ 1 π 1 π 0 = 0 = 1+ α 1 = = + = 1 1 p (1 p) (1 α) q(1 p) p(1 q) q(1 p) q p q ( 1 p)( 1 α ) = ( 1 p) = = 1 ρ π = 1 ρ 0 1 π = ρ(1 α) α, 1

28 Detailed Balace Equatios Geeral case: π P = π P i, j = 0,1,... j ji i ij Need NOT hold for every Markov chai If hold, it implies the GBE; greatly simplify the calculatio of statioary distributio Methodology: Assume DBE hold have to guess their form Solve the system defied by DBE ad Σ i π i = 1 If system is icosistet, the DBE does ot hold If system has a solutio {π i : i=0,1, }, the it is the uique statioary distributio

29 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

30 Geeralized Markov Chais Markov chai o a set of states {0,1, }, that wheever eters state i The ext state that will be etered is j with probability P ij Give that the ext state etered will be j, the time it speds at state i util the trasitio occurs is a RV with distributio F ij {Z(t): t 0} describig the state of the chai at time t: Geeralized Markov chai, or Semi-Markov process Does GMC have the Markov property? Future depeds o 1) the preset state, ad 2) the legth of time the process has spet i this state

31 Geeralized Markov Chais (cotd.) T i : time process speds at state i, before makig a trasitio holdig time Probability distributio fuctio of T i H ( t) = P{ T t} = P{ T t ext state j} P = F ( t) P i i i ij ij ij j= 0 j= 0 E[ T ] = t dh ( t) i = 0 i T ii : time betwee successive trasitios to i X is the th state visited. {X : =0,1, } Is a Markov chai: embedded Markov chai Has trasitio probabilities P ij Semi-Markov process irreducible: if its embedded Markov chai is irreducible

32 Limit Theorems Theorem 3: give a irreducible semi-markov process w/ E[T ii ] < For ay state j, the followig limit p = lim P{ Z( t) = j Z(0) = i}, i = 0,1,2,... j t exists ad is idepedet of the iitial state. E [ Tj ] p j = E[ T ] T j (t): time spet at state j up to time t jj Tj( t) P p j = lim Z(0) = i = 1 t t p j is equal to the proportio of time spet at state j

33 Occupacy Distributio Theorem 4: give a irreducible semi-markov process where E[T ii ] <, ad the embedded Markov chai is ergodic w/ statioary distributio π π = π P, j 0; π = 1 j i ij i i= 0 i= 0 the, with probability 1, the occupacy distributio of the semi-markov process p j π je[ Tj ] =, j = 0,1,... π E[ T ] π j : proportio of trasitios ito state j i E[T j ]: mea time spet at j Probability of beig at j is proportioal to π j E[T j ] i i

34 Outlie Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

35 Cotiuous-Time Markov Chais (def.?) Cotiuous-time process {X(t): t 0} takig values i {0,1,2, }. Wheever it eters state i Time it speds at state i is expoetially distributed with parameter ν i Whe it leaves state i, it eters state j with probability P ij, where Σ j i P ij = 1 Cotiuous-time Markov chai is a semi-markov process with ν i F ( t) = 1 e t, i, j = 0,1,... ij Expoetial holdig time => a cotiuous-time Markov chai has the Markov property

36 Cotiuous-Time Markov Chais Whe at state i, the process makes trasitios to state j i with rate: q ν P ij i ij Total rate of trasitios out of state i j i q = ν P = ν ij i ij i j i Average time spet at state i before makig a trasitio: E[ T ] = 1/ ν i i

37 Occupacy Probability A cotiuous-time Markov chai is irreducible ad regular, if Embedded Markov chai is irreducible Number of trasitios i a fiite time iterval is fiite with probability 1 From Theorem 3: for ay state j, the limit p = lim P{ X ( t) = j X (0) = i}, i = 0,1,2,... j t exists ad is idepedet of the iitial state p j is the steady-state occupacy probability of state j p j is equal to the proportio of time spet at state j

38 Global Balace Equatios Two possibilities for the occupacy probabilities: p j = 0, for all j p j > 0, for all j, ad Σ j p j = 1 Global Balace Equatios p q = p q, j = 0,1,... j ji i ij i j i j Rate of trasitios out of j = rate of trasitios ito j If a distributio {p j : j = 0,1, } satisfies GBE, the it is the uique occupacy distributio of the Markov chai Alterative form of GBE: p q = p q, S {0,1,...} j ji i ij j S i S i S j S

39 Detailed Balace Equatios Detailed Balace Equatios p q = p q, i, j = 0,1,... j ji i ij Simplify the calculatio of the statioary distributio Need ot hold for every Markov chai Examples: birth-death processes, ad reversible Markov chais

40 Birth-Death Process S S c λ 0 λ 1 λ 1 λ µ 1 µ 2 µ µ + 1 Trasitios oly betwee eighborig states q = λ, q = µ, q = 0, i j > 1 i, i+ 1 i i, i 1 i ij Detailed Balace Equatios λ, 0,1,... p = µ + 1p + 1 = Proof: GBE with S ={0,1,,} give: p q = p q λ p = µ p j ji i ij j= 0 i= + 1 j= 0 i= + 1

41 Birth-Death Process µ p = λ p λ 1 λ 1 λ 2 λ 1λ 2Lλ0 λi = 1 = 2 =... = 0 = 0 µ µ µ 1 µ µ 1Lµ 1 i= 0 µ i+ 1 p p p p p λ i λ i λi p 1 p0 1 1 p0 1, if = 0 = 1 i= 0 µ i + 1 = 1 i= 0 µ i + 1 = 1 i= 0 µ i + 1 = + = = + < Use DBE to determie state probabilities as a fuctio of p 0 Use the probability coservatio law to fid p 0 Usig DBE i solvig problems: Prove that DBE hold, or Justify validity (e.g. reversible process), or Assume they hold have to guess their form ad solve system

42 M/M/1 Queue Arrival process: Poisso with rate λ Service times: iid, expoetial with parameter µ Service times ad iterarrival times: idepedet Sigle server Ifiite waitig room N(t): Number of customers i system at time t (state) λ λ λ λ µ µ µ µ

43 M/M/1 Queue λ λ λ λ µ Birth-death process DBE Normalizatio costat Statioary distributio µ µ p = λ p 1 λ p = p = ρ p =... = ρ p µ = 1 1+ ρ = 1 = 1 ρ, if ρ < 1 p p0 p0 = 0 = 1 p = ρ (1 ρ), = 0,1,... µ µ

44 The M/M/1 Queue Average umber of customers (1 ) (1 ) = 0 = 0 = 0 N = p = ρ ρ = ρ ρ ρ 1 ρ λ N = ρ(1 ρ) = = (1 ) 2 ρ 1 ρ µ λ Applyig Little s Theorem, we have N T = = λ 1 λ 1 = λ µ λ µ λ Similarly, the average waitig time ad umber of customers i the queue is give by 2 1 ρ ρ W = T = ad NQ = λw = µ µ λ 1 ρ 1

45 Summary Markov Chai Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Birth-Death Process Detailed Balace Equatios Geeralized Markov Chais Cotiuous-Time Markov Chais

46 Homework #8 Problem 3.14 of R1 Hits: For a service system, the expected umber of customers is fiite if the service rate is greater tha the customer arrival rate. To solve the problem, thik of how to model the system as a Markov process. You may also fid Little's Theorem be of some use i solvig the problem. Gradig: Overall poits poits for 3.14(a) 70 poits for 3.14(b)