Markov Chains and Queueing Networks

Transcription

1 CS 797 Independent Study Report on Markov Chains and Queueing Networks By: Vibhu Saujanya Sharma (Roll No. Y211165, CSE, IIT Kanpur) Under the supervision of: Prof. S. K. Iyer (Dept. Of Mathematics, IIT Kanpur)

2 Abstract Quantitative analysis of computer systems has become very important as they are being used to handle various mission critical and performance intensive applications. In this independent study I studied methods to model computer systems for analyzing them for reliability, availability, and performance metrics. The techniques that were covered include modeling using Discrete Time Markov Chains (DTMC), Continuous Time Markov Chains (CTMC) and Queueing Networks. Each of the topics, which were studied, are very extensive and have many books devoted solely to each one of them. Thus, in this report I present a summary of the important definitions, assumptions, and results pertaining to each one of these as well as provide some handle on using them practically. 1. Introduction Modeling of computer systems has become very necessary with quantitative analysis gaining importance these days. Almost in every related field the emphasis is now on concrete numbers rather than intuitive arguments in support of the system being built or sold. Ascertaining a number of desirable properties of computer systems such as reliability, availability, and performance metrics such as throughput, response times, etc. mandate the need for accurately modeling these systems. Queueing networks and Markov chains offer fairly simple yet effective and practical modeling solutions to such problems. The aim of this independent study was to study these topics in detail and also learn about their practical applications. The text Probability and Statistics with Reliability, Queuing, and Computer Science Applications by Dr. Kishor S. Trivedi [1] was used as the guiding reference for this study along with some reading from [2] and [3]. Discrete Time Markov Chains, Continuous Time Markov Chains, Stochastic Petri Nets, and Network of Queues, were studied as a part of this course along with some other paradigms and basics required for stochastic modeling. I also solved many problems given in the text as a part of this course. This report contains a gist of some of the important and basic topics, which were studied. I have tried to present the important results and techniques along with some intuition behind them. I have also included some examples showing the use of the techniques. The reader is suggested to refer to [1], for detailed derivations as well as other examples. 2 Discrete Time Markov Chains 2.1 Introduction A Markov Chain is a stochastic process whose state space I is discrete (finite or countably infinite) and the probability distributions for its future development depend only on the present state and not on the path (consisting of past states), that was followed to reach this state. If we further assume that the parameter space T is also discrete, then we have a discrete time Markov chain or a DTMC. We can state the Markov property as where X i s are random variables at time step i = {0,1,2 } and if X n =j, then the state of the system at time step n is j, X 0 being the initial state of the system. The transition probability of the Markov Chain is given by and if its value depends only on the difference n-m, i.e. the number of steps, then such chains are called homogenous Markov Chains. In such cases denotes the n-step transition probabilities if the Markov chain. The zero step transition probabilities are given by

3 The one step transition probabilities of a DTMC could be specified in the form of a transition probability matrix: This is a stochastic matrix as it is a square matrix with non-negative entries and the row sums equal to unity. An example of a discrete time Markov chain is: The n-step transition probabilities can be computed one form of the Chapman-Kolmogrov equation: Let P(n) be the n-step transition probability matrix, then we can write the above equation as: Hence, the one step transition probability matrix could be multiplied (n-1) times to itself to get the n-step transition probability matrix. Also using the theorem of total probability we have: Hence, the step-dependent probability vector of, denoting pmf of X n at time n could be expressed as 2.2 State Classification & Limiting Probabilities A state I is said to be transient (or nonrecurrent) iff there is a positive probability that the process will not return to this state. A state I is said to be recurrent iff, starting from state i, the process returns to state i eventually with probability 1. If a state I is recurrent, then it is said to be positive recurrent (recurrent nonnull) if starting in the expected time until the process returns to state i (mean recurrence time) is finite, otherwise it is said to be recurrent null. For a recurrent state i, the period d i is defined as the GCD of the set of positive integers such that p ii (n)>0. A state is said to be aperiodic if its period is 1, else is called periodic if it is greater than 1. A state is said to be absorbing iff p ii =1 If the Markov chain is irreducible, periodic, and recurrent nonnull, the following limit exists and is independent of initial probability: As Hence if In matrix form it is represented as Where v is the limiting probability vector

4 2.3 Discrete Time Birth Death Process If we consider a special case of the DTMCs with one-step transitions only to neighboring states, then we can visualize it as below: Where b i and d i represent the probabilities of birth and death respectively in state i, and are defined as: If the steady state probability vector exists, we use v=vp and get also as and we get the steady state probabilities as: where 2.4 Finite DTMCs with absorbing states Consider a Finite Discrete Time Markov Chain (DTMC), with the transition probability matrix as P. Let s 1 to s n-1 be the transient states. The transition probability from the n th state of this DTMC is zero to all other states and 1 to itself, and it is an absorbing state in the DTMC. Hence in the transition probability matrix the last row would contain all zeros except the last entry to be one. So we could represent this matrix as Q C P = 0 1 Now Q here is a (n-1) by (n-1) sub-stochastic matrix (with at least one row sum <1) describing the probabilities of transition only between transient states. Now the k step transition probability matrix will be given by P k as k k Q C' P = 0 1 The (i,j) th entry in Q k denotes the probability of arriving in state s j after exactly k steps, starting from t state s i. It could be shown that k k = Q converges as t approaches infinity. Hence the inverse matrix (I- 0 Q) -1 exists. This is called as the fundamental matrix M.

5 M = ( I Q) 1 = I + Q + Q = It can be proved that the expected number of visits to a state s j with the system starting from state s i before entering the absorbing state s n is given by the (i,j) th entry in the fundamental matrix M. If X ij is a random variable denoting the number of times a state s j is visited starting from s i, before entering the absorbing state, then k = 0 Q k The visit count to a particular state is very helpful for calculating other quantities such as the average time spent in the system, the overall reliability of the system, etc. by attaching suitable rewards with the states. Consider a software architecture, wherein a number of components participate in functioning of the whole system. Let us consider the number of components to be 5. Also define the architecture of the system using a DTMC as below with the associated transition probability matrix. We assume that each state s i represents that the flow of control has reached component i. The absorbing state s 5 represents that the computation has now ended, pertaining to the output being produced as the control flow finally reaches component 5. Now applying the technique as above, we can find out the average visits made to a component j starting from component 1, by using the (1,j) th element of the fundamental matrix M. If we know the average execution times of each of the components given by t 1,t 2,..,t 5, then we can find the total execution time of the system as t t t t 4 + t 5 time units. Note that the last component is visited only once. Similarly other metrics like the expected reliability of the whole system could also be calculated.

6 3. Continuous Time Markov Chains 3.1 Introduction Unlike DTMCs, Continuous Time Markov Chains (CTMCs) do not have discrete time or index. The state changes could occur at any instant of time. So if X(t) represents the random variable depicting the state at time t then: We define the transition probabilities similar to as in case of DTMCs as : where We could also denote the pmf of X(t) or the state probabilities as The transition probabilities of the CTMC satisfy the Chapman-Kolmogorov equation which states that for all i,j in the state space I, We can also define the infinitesimal generator matrix Q(t) =[q ij (t)] where Where, It can be shown that, If then, Hence for homogenous CTMCs (HCTMCs) wherein the transition probabilities depend on the time difference (t-v) rather than v and t, we have the following Consider that we define

7 Then it can be shown that L j (t) is the cumulative time spent in state j till time t. If we define the vector L(t)=[L j (t)] then by using integration and the above equations we have, The CTMCs do not have a concept of periodic or aperiodic states. A state i is said to be absorbing if q ij =0 for all j different from i. A state i is said to be reachable from a state j if for t>0, p ji (t)>0. If all states of a CTMC are reachable from all other states, the CTMC is said to be irreducible. For an irreducible continuous-time Markov chain, the limits always exist and are independent of the initial state i. If the limiting probabilities π j exist, then we have To determine a non zero unique solution to this equation we use Irreducible Markov chains that yield positive limiting probabilities {π j} in this way are called recurrent nonnull or positive recurrent, and these probabilities are called steady state probabilities. 3.2 The Birth Death Process The birth- death process is a CTMC with states {0,1, } for which transitions from state i may go only to state i-1 or i+1. We can further consider λ i and µ i, called the birth and death rates of state i, respectively defined as below: This could be shown as below : In steady state the rate of flow into and out of each state would be equal. Hence we get the set of balance equations for this CTMC as: Also for k=0, we have, so we get, And so, Hence,

8 The last expression comes from the fact that all limiting probabilities sum up to 1. In case of finite state space, k varies from 1 to n, and the corresponding sums in above expressions change accordingly. The birth death process can be used to model queues. Queues can form wherever, there is a serving station and customers arrive in to receive service at them. There could be many kinds of queues, based upon the arrival discipline, the service discipline, the number of servers, the number of jobs allowed in the system, etc. The M/M/1 Queue: Consider a system consisting of a single server and an associated queue, with jobs being sent by clients to this system. Assume that the arrival process is Poisson, with rate λ. The service times of jobs are independent identically distributed random variables, with exponential distribution with mean µ. Assume that FCFS scheduling is done at the server. If N(t) denotes the number of customers in the system, the {N(t), t>0} is a birth death process with λ k=λ (k 0) and µ k=µ (k>0). Using the expressions derived for a birth death process we have the following: and Here ρ = λ /µ is called the traffic intensity of the system and has units as Erlangs. A large number of measures can be calculated from the limiting probabilities. The expected number of jobs in the system is given by: We can use Little s Formula to calculate the response time E[R] of the system. It states that the mean number of jobs in a queuing system in the steady state is equal to the product of the arrival rate and the mean response time. In this scenario, Little s Formula gives : Using the expression for π k we get Other measures could also be found easily. If W denotes the random variable denotes the waiting time in the queue i.e W=R-S, then, Hence, Also by applying Little s Formula to the queue, we can get the expected number of jobsin the queue as E[Q] given by :

9 In a similar fashion many other queues such as the M/M/m, the M/M/1/n, etc can be analyzed. The reader can refer to [1] for the same. 3.3 Non Birth Death Processes There are many instances of CTMCs, which are not birth-death processes. These models are generally used to calculate the availability of the system (Availability Models), the performance metrics such as mean response time, mean number in the system, etc. of the system (Performance models), etc. Some times models, which take performance, and availability both into consideration, are also used. The method employed for steady state models is essentially to find the limiting probabilities from the balance equations and then proceed to calculate the metrics by using Little s Formula etc. 3.4 CTMCs with absorbing states by: If the CTMC is not irreducible and has an absorbing state,we have to solve the set of equations given for finding the different π j(t)s. Once these are found, one can then find metrics as the mean time to failure (MTTF) of the system. There is however another method that does not require solution of the above system of equations. Denote the set of all system failure states by set D (all the absorbing states belong to set D), and the set of all states in which the system is operational by set U. Let Q denote the sub matrix of Q pertaining to states in U only. Also, let π (0) be the sub vector of π (0) pertaining to states in U. Now, if then the set of equations become, Because τ i is the expected time the CTMC spends in (nonfailure) state i until system failure (entering D), the mean time to (system) failure (MTTF) is given by Hence this analysis is helpful in predicting the MTTFs of many real life systems which need to be modeled by CTMCs having absorbing state(s).

10 4. Network of Queues 4.1 Introduction We saw that in case of birth death processes, a product form solution for the probabilities exists. In practical systems one might consider a network of queues. These could model system with contention for a set of resources. However the pertinent question could be whether the product form applies to these or not. We can divide queueing networks into two broad categories: 1. Open Networks: One or more sources of incoming traffic and one or more sinks that absorb departing traffic. 2. Closed Networks: Does not have a source or a sink, and the total number of jobs in the system remains constant. Burke has shown that the output of an M/M/1 queue is also Poisson with rate λ [1]. So if we have a tandem network as shown below with ρ 0 = λ/µ 0, the second queue is also an M/M/1 queue with server utilization ρ 1 = λ/µ 1 (assumed to be < 1). Hence the probability of finding k 0 jobs at node 0 and k 1 at node 1 is given by This equation could also be derived by constructing the CTMC for this network, and solving it. However for complex networks such a CTMC will be very difficult to analyze. 4.2 Open Networks Jackson showed that the product-form solution also applies to open networks of Markovian queues with feedback [1]. Besides requiring that the distributions of job interarrival times and service times at all nodes be exponential, assume that the scheduling discipline at each node is FCFS. This result is very helpful in analyzing various practical systems. Consider an open queueing network with two nodes as below: In this example, let λ 0 and λ 1 are the arrival rates at the CPU and I/O node, respectively. The arrivals to CPU are either from outside at a rate λ or from the I/O node at rate λ 1, therefore, λ 0 =λ +λ 1 Also a job after completion of the CPU burst would go to the I/O node with probability p 1, therefore, λ 1 =λ 0 p 1 =λ 0 (1-p 0 ). Solving these two traffic equations, we get, Using Jackson s result, the product form solutio n for the 2-node network with feedback:

11 We next consider a Central Server system, with 1 CPU and m I/O nodes for the same. These are interconnected as shown below : This system difficult to analyze in general, however let us consider the behavior of a single tagged program as it traverses the system. We observe the system only at times when the job arrives at a node for service. The underlying stochastic process forms a DTMC, which is characterized by its transition probability matrix as below: Now we can find the visits Vj, the average number of visits made by a job to node j before leaving the system as: If λ jobs arrive per unit time in the system from outside then the arrival rate λj of jobs at node j is, According to Jackson s result th e steady state joint probability for finding k j customers at node j is given by, Where Once we have these steady state joint probabilities we can find measures like expected number in the system, expected response time etc. 4.3 Closed Networks In practical systems the main memory and other resources are limited therefore the number of jobs that can be allowed access to these has an upper limit. Usually a job scheduler does this regulatory job,

12 and we can assume that a job scheduler queue exists as shown below: This kind of network does not have a product form solution. However, the open network case we considered could be thought of as a light load approximation to this, wherein the value of λ is very small and this queue is always empty. For the conditions of very heavy load, we might notice that there are always jobs (in the scheduler queue) waiting to be scheduled when an existing job leaves the queueing network. Modeling using closed networks could then be used here. This is shown as: Gordon and Newell showed that a product form solution would exist for this network also [1]. If we again consider a tagged job through this network, we could model this as a DTMC again, and get the relative visit count to each node by using the system of equations We could chose v 0 as per our convenience. Gordon Newell showed that the steady-state probability p(k 0, k 1,..., k m ) of finding k i jobs at nodes i,( i = 0, 1,..., m), is given by The computation of C(n) could be cumbersome if the number of states if very high, but there are methods, which will allow the computations to be done easily. For more details on these methods one can refer to [1]. 4.4 General Service Distribution And Multiple Job Classes Many other queueing networks apart from the types mentioned earlier also have product form solutions. Moreover the scheduling discipline also does not need to be necessarily FCFS. Consider a closed

13 network with r classes of customers. A class t customer has a routing matrix X t, and its service rate at node i is denoted by µ it. Therefore we can get the visit counts at various nodded by solving the traffic equations for each job class. Assume that Node i is said to be a type 1 node if it has a single server with exponentially distributed service times, FCFS scheduling, and identical service rates for all job types (i.e., µ it = µ i for all t). A node is said to be of type 2 if it has a single server, PS scheduling, and a service time distribution that is differentiable. Each job type may have a distinct service time distribution. Node i is said to be a type 3 node if it has an enough servers so that no queue ever forms at the node. A node is said to be of type 4 if it has a single server with LCFS-PR scheduling. In case of both type 3 and 4 nodes, any differentiable service time distribution is allowed, and each job type may have a distinct service time distribution. Let k it be the number of jobs of type t at node i. Assume that there are n t jobs of type t in the network so that we have We define a vector Y i as (k i1, k i2,, k ir ). Then (Y 0, Y 1,, Y m ) is the state of the system. If k i is the total number of jobs at a node then the joint probability of such a state is given by: Again we can derive C(n 1, n 2,, n r ) using recursive techniques [1]. Thereby other measure like utilization of various nodes, etc. could also be computed. 4.5 Non Product Form Networks Notice that when we studied the central server model, we found approximate ways of modeling the system either in case of light load or very heavy load. But the system does not have a product form solution in general. We could either resolve the system using Stochastic Petri Nets [1], or we need to again approximate the system by using some equivalent representation. Consider that we represent the whole CPU-I/O system as an equivalent server with the queue being the scheduler queue, then we could try analyzing the system using birth-death process. The service rate of the equivalent server depends could be determined from the average throughput of the CPU-I/O system and it is given by

14 where n is the degree of multi-programming of the system. Hence we could the average throughput vector (E[T(1)], E[T(2)],..., E[T(n)]) of the inner model is and then the steady-state probability that i jobs reside in the queuing system could be found for this birth-death process as: We can thus determine the value of π 0 using equating the sum of all probabilities to 1. We can also determine the expected number of jobs in the system as : From this we can get the response time of the system using Little s law. This method is also called the equivalent flow server method. 5. Conclusion I introduced the basics of Discrete and Continuous Time Markov Chains (DTMCs and CTMCs) in this report along with Queueing Networks. The stress was on introducing the essential concepts regarding these topics. Moreover modeling of some practical systems such as the software composed of many individual components or the central system multi-programmed systems were also introduced in this report. 6. References [1] Kishor S. Trivedi : Probability and Statistics with Reliability, Queuing, and Computer Science Applications, John Wiley and Sons, [2] Sheldon M. Ross : Introduction to Probability Models, Academic Press, [3] Robin A. Sahner, Kishor S. Trivedi and Antonio Puliafito : Performance and Reliability Analysis of Computer Systems: An Example-Based Approach Using the SHARPE Software Package, Kluwer Academic Publishers, 1996.