Fluid Models for Production-Inventory Systems

Fluid Models for Production-Inventory Systems by Keqi Yan A dissertation submitted to the faculty of the University of North Carolina at Chapel Hill in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Statistics and Operations Research. Chapel Hill 2006 Approved by Advisor: Vidyadhar G. Kulkarni Reader: Amarjit Budhiraja Reader: Tugrul Sanli Reader: Jayashankar M. Swaminathan Reader: Paul H. Zipkin

c 2006 Keqi Yan ALL RIGHTS RESERVED ii

ABSTRACT Keqi Yan: Fluid Models for Production-Inventory Systems (Under the direction of Professor Vidyadhar G. Kulkarni) We consider a single stage production-inventory system whose production and demand rates are modulated by a finite state Markov chain called the environment. Supplementary orders can be placed from external suppliers when needed. We model this system by a fluid-flow system and derive the limiting distribution of the bivariate process (fluid level, environment state). We present a stochastic decomposition property for this fluid model and hence prove that the classical deterministic Economic- Order-Quantity (EOQ) policy is still optimal in this stochastic environment under certain assumptions. We extend the results to more general models: 1. When backlogging is allowed, we investigate the optimal reorder-point / orderquantity (r, q) policy. We prove that for a given order quantity q, the optimal reorder point r (q) can be explicitly given by the well-known newsboy solution. We also show that in a special case the optimality of the deterministic EOQ policy with backlogging holds. 2. When the order quantity can be environment-dependent, we derive the limiting distribution and then calculate the optimal order quantity for each ordering state. 3. When there are positive leadtimes, we consider three cases: orders are processed iii

sequentially, in parallel, or the leadtimes have different distributions depending on the number of outstanding orders. Assuming there exists an upper limit N for the number of outstanding orders, this model generalizes the emergency-supply model, selective lost-sales model, and can also be an approximation of infinite-supplier model when N is large enough or the probability that there are N outstanding orders is small. We derive the limiting distribution and the optimal (r, q) policy. We prove that for a given q, the optimality of the newsboy solution for r still holds. We also illustrate numerically how to calculate the optimal N which balances the backlogging cost and emergency-supply / lost-sale cost. iv

ACKNOWLEDGEMENTS I have waited for a long time for this chance to express my deepest gratitude to my advisor, Professor Vidyadhar G. Kulkarni. He always earns the respect and admiration of all his students, and for me, he is the hero in my four years journey in the pursuit of the Ph.D. at Chapel Hill. His impact on my attitude in research and philosophy about the world will benefit the reminder of my life. I would also like to thank Professor Paul Zipkin for introducing me to the field of inventory management. During the process of finishing this dissertation, he gave me priceless suggestions and always kindly encouraged me like a father figure. Professor Jayashankar M. Swaminathan s teaching opened a wide door to the supply chain area. With Dr. Tugrul Sanli, I worked on a demonstration of the software Inventory Replenishment Planning, which was my first exposure to the concept of inventory control and spurred my interest in this area. I am also very thankful to Professor Amarjit Budhiraja for his inspiring questions and suggestions. Special thanks to my supervisors and friends at SAS Institute from where I have received tuition support for three years, and have gained valuable experience in implementing operations research methodologies in software and solving real-world problems. I really enjoyed and have learned so much from these past years internship there. It was an inseparable part of my student life in North Carolina. Of course I would also like to thank my parents Yan Yuchi, He Daying and my sister Yan Shuli. However, I know that words are redundant here, in comparison to their unconditional love and support, which are far beyond any language in the world. v

CONTENTS LIST OF FIGURES............................... xi LIST OF SYMBOLS............................... xii 1 Introduction 1 1.1 Stochastic Fluid Models......................... 2 1.2 Stochastic Inventory Control Problems................. 3 1.2.1 Basic EOQ Model......................... 3 1.2.2 Backlogging EOQ Model..................... 4 1.2.3 Environment-Dependent Order Quantities........... 5 1.2.4 Stochastic Leadtimes....................... 7 2 Fluid Model 9 2.1 Introduction................................ 9 2.2 The Standard Fluid Model........................ 11 2.3 The Fluid Model with Jumps...................... 14 vi

2.4 Differential Equations for the Limiting Distribution.......... 16 2.5 Solution to the Differential Equations.................. 19 2.5.1 Case of Distinct Eigenvalues................... 20 2.5.2 Case of Repeated Eigenvalues.................. 22 2.6 A Special Case: A = I.......................... 23 2.6.1 Stochastic Decomposition Property............... 24 2.6.1.1 Laplace Stieltjes Transform Method......... 24 2.6.1.2 Sample Path Method.................. 26 2.6.2 Uniform Limiting Distribution.................. 34 2.7 Examples................................. 35 2.7.1 A Two-State Example...................... 35 2.7.2 A Machine Shop Example.................... 38 3 A Basic Production-Inventory Model 40 3.1 Introduction................................ 40 3.2 The Model................................. 42 3.3 Optimal Order Quantity......................... 43 3.3.1 Stochastic EOQ Theorem.................... 44 3.3.2 Minimum Cost Rate....................... 46 vii

3.4 Inventory Model with Backlogging.................... 49 3.4.1 Cost Rate Calculation of the (r, q) Policy............ 50 3.4.2 Optimal (r, q) Policy....................... 52 3.4.2.1 Newsboy Solution for the Optimal r for a Given q. 52 3.4.2.2 Stochastic EOQ Policy with Backlogging....... 53 3.4.2.3 A Numerical Example................. 54 4 Environment-Dependent Order Quantities 58 4.1 Introduction................................ 58 4.2 Piecewise Function Method....................... 60 4.3 Sample Path Decomposition Method.................. 64 4.4 Laplace-Stieltjes Transform Method................... 71 4.5 The Cost Model.............................. 76 4.6 A Numerical Example.......................... 77 5 Stochastic Leadtimes 80 5.1 Introduction................................ 80 5.2 Serial Processing System......................... 84 5.3 Parallel Processing System........................ 87 5.4 General Order Processing........................ 89 viii

5.5 Selective Lost Sale Model........................ 91 5.6 The Cost Model.............................. 93 5.6.1 Cost Rate Calculation...................... 93 5.6.2 Newsboy Solution for the Optimal Reorder Point....... 97 5.7 A Numerical Example.......................... 99 5.7.1 Limiting Distribution....................... 99 5.7.2 Optimal Ordering Policy..................... 102 5.7.3 Optimal Production Rate.................... 104 5.7.4 Sensitivity Analysis........................ 104 6 Conclusions and Future Research 111 6.1 Conclusions................................ 111 6.2 Future Research.............................. 114 6.2.1 Model with Semi-Markov Process as Background process... 114 6.2.2 Environment-Dependent Order Quantities and Reorder Points 115 6.2.3 Numerically Stable Methods for the Stochastic Leadtime Model in Serial Processing System................... 115 6.2.4 Environment-Dependent Ordering Policies with Stochastic leadtimes................................ 115 Bibliography 117 ix

LIST OF FIGURES 2.1 A sample path of the (buffer level, environment state) process..... 10 2.2 Decomposition of the X(t) process.................... 27 2.3 Correspondence of the processes X(t), Z(t), X 1 (t), Y 0 (t) and Z 0 (t).. 30 2.4 Correspondence of the processes X(t), Z(t), X 1 (t), Y 1 (t) and Z 1 (t).. 32 2.5 Limiting distribution when r > d..................... 37 2.6 Limiting distribution when r < d..................... 38 2.7 The steady-state ccdf........................... 39 2.8 The steady-state pdf............................ 39 3.1 The optimal order quantity vs. production rate............. 46 3.2 The minimum total cost and the optimal production rate....... 48 3.3 The inventory level process when allowing backlogging......... 49 3.4 The optimal order quantity vs. production rate............. 54 3.5 The optimal reorder point vs. production rate.............. 55 3.6 The optimal order-up-to level q + r vs. production rate........ 56 3.7 The minimum cost vs. production rate.................. 57 4.1 A sample path of X(t) and Z(t) with environment-dependent order quantities.................................. 59 x

4.2 Piecewise function method......................... 61 4.3 Sample path decomposition method................... 65 4.4 The optimal order quantities vs. production rate............ 78 5.1 Sample paths of P (t) and X(t) with stochastic leadtimes........ 81 5.2 The steady-state ccdf of the P (t) process................ 99 5.3 The steady-state pdf of the P (t) process................. 100 5.4 The steady-state ccdf of X(t) and P (t)................. 101 5.5 The steady-state pdf of X(t) and P (t)................. 101 5.6 The optimal order quantity vs. production rate (varying n)...... 102 5.7 The optimal reorder point vs. production rate (varying n)....... 103 5.8 The minimal cost vs. production rate (varying n)............ 104 5.9 The optimal order quantity vs. production rate (varying ν)...... 105 5.10 The optimal reorder point vs. production rate (varying ν)....... 105 5.11 The minimum cost vs. production rate (varying ν)........... 106 5.12 The optimal order quantity vs. production rate (varying N)...... 107 5.13 The optimal reorder point vs. production rate (varying N)....... 108 5.14 The probability that there are N outstanding orders (varying N)... 108 5.15 The minimum cost vs. production rate (varying N)........... 109 xi

LIST OF SYMBOLS a j Equation (2.17) and (4.27), Theorem 4.2, Theorem 4.6. a A row vector of a j s in Equation (4.27). a kj Defined in Equation (5.15) and (5.22). A Transition probability matrix [α ij ], Section 2.3. A (i) Defined in Theorem 4.3. Ā Defined in Equation (5.26). b Backorder penalty-cost rate, Section 3.4.1. c b c h c o c p Steady-state backlogging cost rate. Steady-state holding cost rate. Steady-state ordering cost rate. Steady-state production cost rate. c j Coefficients in Equation (2.16). c (i) k Defined in Theorem 4.2. d Demand rate in Section 2.7.1. d i Demand rate when the environment process is in state i, Section 2.7.2. D(s) Defined in Theorem 4.6. e e = [1,..., 1] t. E(x) Expectation of the random variable x. Ẽ ii (s) Defined in Equation (4.31). f(j, x) Defined in Equation (2.33). F j (x) Limiting cdf of the inventory level process at state j, Equation (2.21). F j (s) LST of F j (x), Equation (2.22). F (s) A row vector of LST of F i (x) s, Equation (2.23). xii

g(i, x) Defined in Equation (2.36). G(x) G (x) G (x) = G(x) = [G 1 (x),..., G n (x)]. [ dg1 (x) dx,..., dgn(x) dx ]. G j (t, x) Defined in Equation (2.11). G j (x) Defined in Equation (2.12). G (i) j (x) Defined in Section 4.3. G (i) (x) Defined in Section 4.3. Ḡ (i) j (x) Defined in Section 5.1. Ḡ (i) (x) Defined in Section 5.1. Ḡ(x) Defined in Section 5.1. h Inventory holding-cost rate, Section 3.1. H j (x) Limiting Cumulative distribution function of the fluid level in state j, in standard fluid model without jumps, Section 2.2. H(x) [H 1 (x),..., H n (x)], Section 2.2. H (x) [ ] dh1 (x),..., dhn(x), Section 2.2. dx dx I (i) Defined in Equation (4.2). k Fixed set-up cost to place an order, Section 3.1. k 1 k 2 Fixed set-up cost to place an order from a regular supplier, Section 5.6. Fixed set-up cost to place an order from an emergency supplier, Section 5.6. L Defined in Equation (4.30). m + m 0 m m Number of background states with positive input rate. Number of background states with zero input rate. Number of background states with negative input rate. Number of background states with nonzero input rate. xiii

M (j) ii Defined in Equation (4.39). M (j) n A diagonal matrix diag(m (j) ii ). Number of the states of the environment processes. N Upper limit of the number of outstanding orders, Section 5.1. O(t) Number of outstanding orders at time t, Section 5.1. p (i) Defined in Section 4.3. p ik Defined in Section 4.3. p (k) (x) Defined in Section 4.3. p ik (x) Defined in Section 4.3. P, P = [p ik ], Section 4.3. P (t) Inventory position at time t. P (t) Defined in Equation (5.1). p 1 Purchasing cost rate, Section 3.3. Purchasing cost rate from a regular supplier, Section 5.6.1. p 2 Production cost rate, Section 3.3. Purchasing cost rate from the emergency supplier, Section 5.6.1. p 3 Production cost rate, Section 5.6.1. q Order quantity. q i Order quantity when the order is place in state i. q ij Transition rate of the environment process from state i to state j. ˆq ij Defined in Equation (2.30). Q Q = [q ij ]. Q Defined in Equation (5.4), (5.12) and (5.19). r Reorder point. R Defined in Equation (5.25). xiv

R i Net input rate when the environment process is in state i. R i = r i d i. S i i-th order epoch, Section 2.6.1.2. t Time variable. T First passage time T = inf{t 0 : X(t) = 0}, Equation (4.10). T 1 Defined in Section 2.6.1.2. T 2 Defined in Section 2.6.1.2. T 2n+1 Defined in Section 2.6.1.2. T 2n+2 Defined in Section 2.6.1.2. T j (x) Steady-state complementary cdf of the inventory level in the backlogging model, Equation (3.8). T (x) Vector of T j (x) s, Equation (3.8). u Production rate of one machine, Section 2.7.2. x Inventory level variable. X(t) Inventory level at time t. X 1 (t) Defined in Section 2.6.1.2. X 2 (t) Defined in Section 2.6.1.2. Y Defined in Section 5.6.2. Y 0 (t) Defined in Section 2.6.1.2. Y 1,n Defined in Section 2.6.1.2. Z(t) State of the environment process at time t. Z 0 (t) Defined in Section 2.6.1.2. xv

α ij Defined in Section 2.3. β Defined in Theorem 2.3. β (i) β (i) = G (0)RI (i), Theorem 4.1. δ A small positive number, Section 2.4. δ ij δ ij = 1 if i = j, and 0 otherwise. Net demand rate in steady state, Section 3.2. η kj Defined in Equation (2.31). θ Defined in Section 2.7.1. λ Repair rate of failed machine, Section 2.7. λ i i-th generalized eigenvalue, Equation (2.5). λ (k) i Defined in Section 5.3, and Equation (5.20). µ Failure rate of one machine, Section 2.7. ν Leadtime distribution parameter, Section 5.1. π Defined in Equation (2.1). ˆπ i Defined in Equation (2.32). π(j, x) Defined in Equation (2.35). τ(j, x) Expected sojourn time of the SMP in state (j, x), Section 2.6.1.2. τ j τ j = τ(j, x), x, Section 2.6.1.2. φ i The row vector (eigenvector) corresponding to λ i such that φ i (λ i R Q) = 0. φ 0 A constant row vector in the expression of G(x), Theorem 2.4. φ (i) 0 Defined in Theorem 4.2. φ (k) i Defined in Section 5.3, and Equation (5.20). Φ Defined in Equation (4.4). Φ (k) i Defined in Equation (5.14) and (5.21). xvi

ψ j (λ j R Q)ψ j = 0, Section 4.4. Ψ Ψ = R 1 Φ 1, Section 4.4. Ω State space of the environment process. Ω + Ω + = {i Ω : R i 0}. Ω Ω = {i Ω : R i < 0}. xvii

Chapter 1 Introduction In this thesis we study a type of production-inventory models that can be seen as a stochastic fluid-flow system. We consider a single product, single location problem. The system has production, demand, and external supply. As the environment evolves over time, the production and demand rates are piecewise constant functions determined by the exogenous environment process. When the production rate exceeds the demand rate, the inventory increases, and when the demand rate exceeds the production rate, it decreases. When needed, replenishment orders can be placed from external suppliers. The inventory under continuous review thus can be viewed as a fluid process that fluctuates according to the evolution of the underlying background process. We assume the external environment undergoes recurring changes in a stochastic fashion, and may be modeled as Markovian. For example, production rates and demand rates change due to weather, economy, competition, seasonal promotion, customer status, and forecasting, etc. Some other example are as described in Mitra (1988) where the author studied a producer and consumer problem in a machine shop where the production rate changes according to the number of working machines. There are costs to hold products in inventory, to backlog unsatisfied orders,

to purchase and to produce. There is also a fixed set-up cost every time an order is placed with an external supplier. Our objective is to find an optimal ordering and production policies that minimizes the long-run average cost. 1.1 Stochastic Fluid Models First in Chapter 2 we study a fluid model to establish the fundamental theory for the production-inventory system. We view the inventory level under continuous review as a fluid level process. When the buffer is empty the fluid level jumps to a predetermined level q instantaneously, and at the same time the environment state jumps to another state with a given probability (it may stay unchanged). Between two consecutive jumps the background process is a continuous time Markov chain. At the jump epoch the environment process jumps according to a transition matrix A. We first derive the stability condition for this system and then derive a set of first order non-homogeneous linear differential equations to describe the limiting behavior of the bivariate (buffer level, environment state) process. We also determine the boundary conditions and give explicit solutions to the differential equations. Particularly for a special case A = I, we use two parallel methods to obtain an interesting stochastic decomposition property: in steady state, the buffer content in the fluid model with jumps is the sum of two independent random variables, one of which has a uniform distribution over [0, q], and the other is the steady-state buffer level in the standard fluid model without jumps. We also consider a more specific case where the fluid input rate is always negative over all the environment states. In this case the fluid level has uniform distribution in steady state, and is independent of the environment state. Most of the classical research on stochastic fluid systems allows only continuous 2

change in the buffer content. Under that assumption, the limiting joint distribution of the (buffer level, environment state) process is computed as a solution of a set of ordinary differential equations in terms of the eigenvalues and eigenvectors of the underlying system. There are also studies about fluid models where instantaneous jumps occur when the environment state changes and the size of the jump depends on the state of the environment (see Kulkarni, Tzenova and Adan (2005), Miyazawa and Takada (2002) and Sengupta (1989)). Another related model is the so called clearing system (see El-Taha (2002), Serfozo and Stidham (1978) and Whitt (1981)). A clearing system can be regarded as the reverse of our model. In a clearing system the fluid process jumps to zero when it reaches a certain positive level. However, there is no explicit environment process in these models. The paper that comes closest to our analysis is Berman, Stadje and Perry (2006) where the authors consider a two state CTMC as the environment process. The methodology of their analysis is different with ours. 1.2 Stochastic Inventory Control Problems Beginning from Chapter 3, we study the production-inventory problem using the theories developed in Chapter 2. 1.2.1 Basic EOQ Model In Chapter 3 we start from the basic model where there is no backlogging and zero leadtime, i.e., when the inventory on hand is zero, a supplementary order is placed and arrives instantaneously, and the order size q is independent of the environment state when the order is placed. 3

In a deterministic setting with constant demand rate, the classical Economic- Order-Quantity (EOQ) model describes the trade-off between the constant set-up cost and the variable holding cost. The earliest work on this is Harris (1913). For a modern review of the determinist models, see Zipkin (2000). In this thesis, we establish the stochastic EOQ theorem that shows in a CTMC environment the standard deterministic EOQ formula remains optimal if we replace deterministic demand rate by the expected net demand rate in steady state. In addition to the ordering policy, we also consider the optimal production policy: choose the optimal production capacity that achieves the best combination of outsourcing and inhouse-production. We show this mainly with numerical results. Interestingly, the optimal policy does not suggest always depending on inhouse-production, even if the production cost is less than the outsourcing price. 1.2.2 Backlogging EOQ Model Later in Chapter 3, from Section 3.4 we extend the basic model to allow backlogging: an external order is not placed until the inventory level reaches the preset reorder point r. We derive the optimal ordering-production policy, which achieves the tradeoff point of production cost, fixed ordering cost, holding cost and backlogging cost. Particularly, when production is always less than the demand rate (for example, the system does not make its own product), we prove that the optimality of the deterministic EOQ formula with backlogging still holds in this stochastic environment if one replaces the deterministic demand rate by the expected net demand rate in steady state. In the literature, dynamic control of inventory systems have been classified as periodic review models and continuous review models. The continuous review model 4

can be further classified according to the demand and production are discrete or continuous. (1) Discrete demands and production: Poisson demands in the continuous review models are studied in depth, see Scarf (1958), Karlin and Scarf (1958), Galliher et al. (1959), and Morse (1958). These early papers are reviewed in Scarf (1963). Poisson demand is generalized by Finch (1961), Rubalskiy (1972a,b) and Sivazlian (1974) to unit demands arriving at epochs following a renewal process. Song and Zipkin (1993) considered the case of Markov modulated Poisson demands. (2) Continuous fluid models: Berman and Perry (2004) studied a fluid model where the production and demand rates depend on the inventory level. Browne and Zipkin (1991) studied a model with continuous demand driven by a Markov process, which can be regarded as a special case of the model in this thesis. To our knowledge, although there are papers studying similar problems, none includes the result about the explicit Stochastic EOQ theorem presented in this thesis. 1.2.3 Environment-Dependent Order Quantities In Chapter 4 we relax the assumption that the order quantity is predetermined, i.e., the order quantity is allowed to depend upon the environmental state when the order is placed. For example, if we can observe the environmental state and can base our inventory replenishment decisions on that information, the policy that allows the order quantities to depend on the state of the environment is certainly no worse than the simple reorder-point/order-quantity policy. We use three methods to derive the limiting distributions of the inventory level: 5

(1) Consider appropriate non-overlapping intervals of the inventory level and within each interval derive the differential equations following the methodology of Chapter 2. Then the final limiting distribution is a piecewise function consisting of the functions derived in all these intervals. (2) Decompose the sample path of the inventory level into different cycles and reduce this problem to the basic fluid model of Chapter 2 in individual cycles. Then the overall limiting distribution is a weighted average of the limiting distribution functions in all cycles. (3) Consider individual intervals of the inventory level as in (1), but use Laplace- Stieltjes transforms instead of solving differential equations to obtain the limiting distribution function. Then based on the limiting distribution of the inventory level, we derive the longrun average cost and hence determine the optimal environment-dependent order quantities. In the literature Berman, Stadje and Perry (2006) studied a similar model with a two-state random environment. They consider order quantities that depend on the state of the environmental state and derive the optimal order quantities to maximize the system revenue. However, their calculation of optimal order quantities is based on the explicit results of the steady-state distribution of this two-state system. When the background has more than two states, their method becomes impractical. The general fluid EOQ models with multiple order quantities studied in this thesis seem to be new. 6

1.2.4 Stochastic Leadtimes In Chapter 5 we extend the model further to allow stochastic leadtimes. Three order processing fashions are considered: (1) Orders are processed sequentially, and hence orders never cross in time. Interarrival times between orders are i.i.d. exponential. (2) Orders are processed in parallel fashion, and leadtimes are i.i.d. exponential random variables. So orders can cross in time. (3) Inter-arrival times of the outstanding orders have exponential distributions whose parameters depends on the number of outstanding orders. This generalizes the previous two cases. We assume there exists an upper limit N of the number of outstanding orders. When there are N outstanding orders and the inventory position decreases to the reorder point again, we either obtain an emergency order instantaneously with higher ordering costs, or lose sales with the penalty costs. If the limiting probability that the number of outstanding orders is N is very small, this provides a good approximation to the models with no upper limit on the number of outstanding orders We derive the optimal ordering-production policy which minimizes the sum of the production cost, fixed ordering cost, holding cost, backlogging cost, and emergency ordering cost (or lost-sale penalty). Minimum cost and limiting distribution under the optimal policy are also calculated. In the literature, there is a sizeable body of work on inventory systems with stochastic leadtimes and Markov modulated demands. However, most of this literature is concerned with Markov modulated Poisson process models of demands. An 7

extensive review of this literature is given in Zipkin (2000). As far as we know, there is very little work on Markov modulated fluid models in the context of the productioninventory systems. One relevant work is that of Browne and Zipkin (1991), where the authors assume continuous stochastic demand, but no production. 8

Chapter 2 Fluid Model 2.1 Introduction In this chapter we study a stochastic fluid-flow system consisting of a single infinite capacity buffer. The buffer content increases or decreases according to a fluid-flow rate modulated by an environment which is a stochastic process with finite state space. Whenever the buffer is empty, it is refilled to a predetermined level instantaneously, and at the same time the environment state jumps to another state with a given probability (it may stay unchanged). Figure 2.1 illustrates a sample path of the (buffer level, environment state) process. Our primary motivation for considering this model is to provide fundamental theories to study a production-inventory system modulated by a Markovian environment. For example, the fluid process can be viewed as the inventory level under continuous review. The environment process represents the background state, for example, production or sales seasons. A jump in the fluid level represents an external order placement or order arrival, and the transition of the background state at the jump point can be a result of repairs of production facility, etc.

Figure 2.1: A sample path of the (buffer level, environment state) process. The outline of this chapter is as follows. In section 2.2, we present some preliminary results about the standard fluid model without jumps. In section 2.3 we describe the model with jumps in detail and derive the stability condition. In section 2.4 we derive a system of first order non-homogeneous linear differential equations for the limiting distribution of the bivariate (buffer level, environment state) process. We also determine the boundary conditions needed to solve those differential equations. In section 2.5, we derive explicit solutions to the differential equations. An interesting stochastic decomposition property is given in Section 2.6 about a special case where the background state does not change at jump epochs: in steady state, the buffer content in the fluid model with jumps is the sum of two independent random variables, one of which has a uniform distribution over [0, q], and the other is the steady-state buffer level in the standard fluid model without jumps. We also consider a more specific case where the fluid input rate is always negative over all the environment states. In this case the fluid level has uniform distribution in steady state, and is independent of the environment state. In section 2.7, we illustrate our methodology 10

with an analytic example as well as a numerical one. 2.2 The Standard Fluid Model In this section, we present some preliminary results about the standard fluid model with infinite capacity buffer. See the survey paper Kulkarni (1997) for an extensive overview of the research in this area. Let X(t) be the fluid level in the buffer at time t. The rate of change of the fluid level is modulated by a continuous time Markov chain {Z(t), t 0} on a finite state space Ω = {1, 2,..., n} with generator matrix Q = [q ij ]. As long as Z(t) is in state i, the fluid level process {X(t), t 0} changes at rate R i. Note that R i may be either negative or positive. Let π = [π 1, π 2,..., π n ] be the limiting distribution of the {Z(t), t 0} process, i.e., π is the unique solution to πq = 0, n π i = 1. i=1 The system is stable if and only if the expected input rate is negative, i.e., (2.1) n π i R i < 0. i=1 Let R = diag(r 1,..., R n ) be the diagonal n n matrix with the input rate R i as the ith entry on the diagonal. Let e = [1,..., 1] t be an n 1 column vector of ones. Then the stability condition can be written in matrix form as follows πre < 0. (2.2) 11

When the stability condition (2.2) holds, the following limits exist: H j (x) = lim t P {X(t) x, Z(t) = j}, x 0, j Ω. Let H(x) = [H 1 (x),..., H n (x)], and [ H dh1 (x) (x) = dx,..., dh ] n(x). dx The next theorem gives the differential equations satisfied by H(x). Theorem 2.1. Assume the stability condition (2.2) holds. The vector H(x) satisfies H (x)r = H(x)Q, x 0. (2.3) The boundary conditions are given by H j (0) = 0, j : R j > 0, (2.4a) H( )e = 1. (2.4b) Let (λ, φ) be a generalized (eigenvalue, eigenvector) pair that solves φq = λφr. (2.5) 12

Let Ω + = {i Ω : R i > 0}, (2.6) Ω 0 = {i Ω : R i = 0}, (2.7) Ω = {i Ω : R i < 0}, (2.8) and m + = Ω +, m 0 = Ω 0 and m = Ω. It is known that the number of eigenvalues that satisfy Equation (2.5) is m = m + + m (counting multiplicities). When the stability condition holds, one eigenvalue is 0, m + have negative real part, and m 1 have positive real part. We index the eigenvalues so that λ 1,..., λ m+ have negative real parts, λ m+ +1 = 0, and λ m+ +2,..., λ m have positive real parts. It is easy to see that φ m+ +1 = π is a valid eigenvector corresponding to the eigenvalue 0. When the eigenvalues are all distinct, the solution to the differential equations in Theorem 2.1 is given by m + H(x) = a i e λix φ i + π, if x > 0, i=1 where the coefficients a 1,..., a m+ are given by the unique solution to the following system of m + linear equations: m + a i φ ij + π j = 0, j Ω +, i=1 where φ ij is the j-th element in φ i. Let H j (s) = 0 e sx dh j (x) 13

be the Laplace-Stieltjes transform (LST) of H j (x) and H(s) = [ H1 (s), H 2 (s),..., H n (s)]. Taking transforms of (2.3), and noticing that H(x) has a jump at 0 of size H(0), and a density H (x) for x > 0, we get H(s) = sh(0)r(sr Q) 1. (2.9) It follows that there is a unique vector H(0) satisfying conditions (2.4a) and (2.4b) that makes H(s) a valid LST of a vector of random variables. We shall use this fact in deriving results in section 2.6. 2.3 The Fluid Model with Jumps Now we describe a fluid-flow model with infinite capacity buffer that we analyze in this chapter. As before, let X(t) be the fluid level in the buffer at time t. The rate of change of the fluid level is modulated by a stochastic process {Z(t), t 0} on a finite state space Ω = {1, 2,..., n}. As long as Z(t) is in state i, the fluid level process {X(t), t 0} changes at rate R i. When X(t) reaches zero it jumps to a predetermined level q instantaneously. Let S 0 = 0 and S k be the kth jump time. We assume that over (S k, S k+1 ) the process {Z(t), t (S k, S k+1 )} behaves as an irreducible CTMC on Ω with generator matrix Q = [q ij ]. Furthermore, when the {X(t), t 0} process jumps at time S k, the {Z(t), t 0} process changes instantaneously with probability α ij defined as follows: α ij = P {Z(S k +) = j Z(S k ) = i}, i, j Ω. 14

Let A = [α ij ]. It is clear that {(X(t), Z(t)), t 0} is a bivariate Markov process. Next we derive the condition when this process is stable, i.e., it has a limiting distribution. Let π be as in Equation (2.1). Note that π is not the limiting distribution of Z unless A = I. Theorem 2.2. The process {(X(t), Z(t)), t 0} is stable if and only if n π i R i < 0. (2.10) i=1 Proof. Let S k be the k-th jump epoch in the {X(t), t 0} process, with S 0 = 0. Let Z k = Z(S k +). It is easy to see that {Z k, k 0} is a DTMC on state space Ω. Since Q is assumed to be irreducible it can be seen that {Z k, k 0} has a single closed communication class Ω Ω that is positive recurrent. Without loss of generality, suppose (X(0), Z(0)) = (q, i) for some i Ω. Let N = min{k 0 : Z k = i}. It is clear that X(S N ) = q, Z(S N ) = i and that {(X(t), Z(t)), t 0} is a regenerative process that regenerates at time S N. Thus from the theory of the regenerative process (see Heyman (1982)), the limiting distribution of the {(X(t), Z(t)), t 0} process exists if E(S N ) <. Since Z(0) = i Ω, it follows that N is the number of steps needed by the {Z k, k 0} process to go from state i to state i. Since Ω is finite, E(N) <. Now from Kulkarni (2002), it follows that E(S 1 X(0) = x, Z(0) = j) < n if and only if π i R i < 0. Now, i=1 E(S N ) E(N) max j Ω {E(S 1 X(0) = q, Z(0) = j)} <. 15

This proves the theorem. 2.4 Differential Equations for the Limiting Distribution Let G j (t, x) = P {X(t) > x, Z(t) = j}, x 0, t 0, j Ω. (2.11) Assume the stability condition (2.10) holds so that the following limits exist: G j (x) = lim t P {X(t) > x, Z(t) = j}, x 0, j Ω. (2.12) In this section we show how to compute G(x) = [G 1 (x),..., G n (x)]. (2.13) We use the notation [ G dg1 (x) (x) = dx,..., dg ] n(x). dx The next theorem gives the differential equations satisfied by G(x). Theorem 2.3. Assume the stability condition (2.10) holds. The limiting distribution G(x) is continuous on [0, ) and is a piecewise differentiable function on (0, q) and (q, ). It satisfies G (x)r = G(x)Q + β, 0 < x < q, (2.14a) G (x)r = G(x)Q, x > q, (2.14b) 16

where the row vector β is given by β = G (0)RA. The boundary conditions are given by G( ) = 0, (2.15a) G j (q + ) = G j (q ), j / Ω 0, (2.15b) G j(0) = 0, j Ω +, (2.15c) G(0)e = 1. (2.15d) Proof. The differential equations follow from the standard derivation of Chapman Kolmogorov equations for Markov processes. We assume at time 0, (X(0), Z(0)) is in steady-state, i.e., for all state j Ω, P {X(0) > x, Z(0) = j} = G j (x). First consider the x < q case. We consider a time interval [0, δ] where δ > 0. During [0, δ], the {Z(t), t 0} process behaves like a usual CTMC with generator matrix Q if the {X(t), t 0} process does not hit zero. Otherwise the {Z(t), t 0} process changes state according to the matrix A and the {X(t), t 0} process jumps to q. Thus we have G j (x) = P {X(δ) > x, Z(δ) = j} = n P {X(δ) > x, Z(δ) = j X(0) > x R i δ, Z(0) = i} P {X(0) > x R i δ, Z(0) = i} i=1 + P {X(δ) > x, Z(δ) = j X(0) R i δ, Z(0) = i} P {X(0) R i δ, Z(0) = i} i Ω 17

When X(0) > x R i δ, we get P {X(δ) > x, Z(δ) = j X(0) > x R i δ, Z(0) = i} = P (Z(δ) = j Z(0) = i) = δ ij + q ij δ + o(δ), where δ ij = 1 if i = j, and 0 otherwise. Also note that when X(0) R i δ, P {X(δ) > x, Z(δ) = j X(0) R i δ, Z(0) = i} = α ij + O(δ), where O(δ) is a function of δ that goes to 0 as δ goes to 0. Using the fact that O(δ)(G i (0) G i ( R i δ)) = o(δ), we get the following: G j (x) = n (δ ij + q ij δ) G i (x R i δ) + α ij (G i (0) G i ( R i δ)) + o(δ) i=1 i Ω = n G j (x R j δ) + q ij δg i (x R i δ) + α ij (G i (0) G i ( R i δ)) + o(δ). i=1 i Ω Rearrange and divide both sides by δ to get G j (x) G j (x R j δ) δ = n q ij G i (x R i δ) + G i (0) G i ( R i δ) α ij + o(δ). δ i Ω i=1 Letting δ 0, we get G j(x)r j = n G i (x) q ij + i=1 i Ω α ij R i G i(0). This shows that G(x) is differentiable over (0, q). Later we shall show that G j(0) = 0 if j Ω +, (boundary condition (2.15c)). Hence we get Equation (2.14a), with β j = n G i(0)r i α ij. i=1 Now consider the x > q case. When x > q we do not need to consider the jumps 18

in the {X(t), t 0} process from 0 to q. Thus G j (x) = P {X(δ) > x, Z(δ) = j} = n P {X(δ) > x, Z(δ) = j X(0) > x R i δ, Z(0) = i} P {X(0) > x R i δ, Z(0) = i}. i=1 Following the same steps as in x < q case, we get Equation (2.14b). This also proves that G(x) is differentiable for x (q, ). As for boundary conditions, Equation (2.15a) follows because G(x) is the complementary distribution function of the fluid level in steady state. The boundary condition (2.15b) for all states j / Ω 0 is obvious from the fact that there is no probability mass at (q, j), i.e., G j (x) is continuous at x = q, if j / Ω 0. Equation (2.15c) holds because 1/(G j(0)r j ) can be seen to be the expected time between two consecutive visits by the {(X(t), Z(t)), t 0} process to the state (0, j). If j Ω +, this mean time is infinity. Hence G j(0) = 0 when j Ω +. From the definition of G j (x) we have Therefore, G j (0) = lim t P {X(t) [0, ), Z(t) = j}. n G j (0) = 1, which is Equation (2.15d). j=1 2.5 Solution to the Differential Equations In this section, we give the solution to the differential equations (2.14a) and (2.14b). We shall treat the cases with distinct eigenvalues and repeated eigenvalues separately. 19

2.5.1 Case of Distinct Eigenvalues Assume that all eigenvalues are distinct and hence the eigenvectors φ i s are linearly independent. The next theorem gives the main result. Theorem 2.4. The solution to the differential equations in Theorem 2.3 is given by m G(x) = c i e λix φ i + c 0 xπ + φ 0, 0 x q, (2.16) i=1 m + G(x) = a i e λix φ i, x > q, (2.17) i=1 where the coefficients a 1, a 2,..., a m+, c 1, c 2,..., c m, c 0 and the vector φ 0 are given by the unique solution to the following system of linear equations: m c i φ i QA + c 0 πr(a I) + φ 0 Q = 0, (2.18a) i=1 m + a i e λiq φ ij i=1 i=1 m c i e λiq φ ij c 0 qπ j φ 0j = 0, j / Ω 0, (2.18b) m c i λ i φ ij + c 0 π j = 0, j Ω +, (2.18c) i=1 ( m ) c i φ i + φ 0 e = 1, (2.18d) i=1 φ 0 e = 1, (2.18e) where φ ij is the j-th element in φ i, i = 0, 1,..., m. Proof. In section 2.2 we see that the homogenous equations (2.14b) have solutions of form G(x) = m c i e λix φ i. i=1 20

It can be shown that the nonhomogeneous equations (2.14a) have solutions of form G(x) = m c i e λix φ i + c 0 xπ + φ 0 i=1 if and only if c 0 xπ + φ 0 is a particular solution to (2.14a). Using G(x) = c 0 xπ + φ 0 in (2.14a), we get c 0 πr = c 0 xπq + φ 0 Q + β = φ 0 Q + β. The last equation holds because πq = 0. Substituting ( m ) β = G (0)RA = c i λ i φ i + c 0 π RA i=1 and noting that λ i φ i R = φ i Q, we obtain c 0 πr = φ 0 Q + m c i φ i QA + c 0 πra, i=1 which can be rearranged to get Equation (2.18a). When x > q, G(x) has a solution m + of the form G(x) = a i e λix φ i (Note that boundary condition 2.15a implies that the i=1 coefficient a i has to be zero when Re(λ i ) 0). Because there is no probability mass in (q, j) for j / Ω 0, the boundary condition in Equation (2.15b) reduces to m + a i e λiq φ ij = i=1 m c i e λiq φ ij + c 0 qπ j + φ 0j, j / Ω 0, (2.19) i=1 Rearranging (2.19) we get (2.18b). Equation (2.18c) and (2.18d) follow directly from boundary conditions (2.15c) and (2.15d). The total number of unknown coefficients is m + + m + n + 1. Notice that the number of independent equations in (2.18a) is n 1, since the rank of the matrix Q is n 1; the number of independent equations is m in (2.18b), and m + in (2.18c). 21

Including Equation (2.18d) we have m + + m + n independent equations satisfied by m + + m + n + 1 coefficients. Since any particular solution will work, we use Equation (2.18e) to determine a unique particular solution. Thus we have as many equations as unknowns. 2.5.2 Case of Repeated Eigenvalues When there are repeated eigenvalues we solve this problem using generalized eigenvectors. Let (λ 1, φ (1) 1 ), (λ 2, φ (2) 1 ),, (λ K, φ (K) 1 ) be K solutions to Equation (2.5), and λ 1, λ 2,...λ K are K distinct eigenvalues. Assume λ 1,...λ K+ have negative real part, λ K+ +1 = 0, and λ K+ +2,...λ K have positive real part. Let n i be the multiplicity of the K + K eigenvalue λ i. Clearly n i 1 and n i = m +, n i = m 1. i=1 i=k + +2 The general solution to the homogeneous equations G (x)r = G(x)Q is given by G(x) = K i=1 n i e λ ix c (i) j j=1 j k=1 x j k (j k)! φ(i) k, (2.20) where c (i) j s are constant coefficients, and φ(i) s are generalized eigenvectors satisfying φ (i) k Q = λ iφ (i) k R + φ(i) k 1 R, k = 2,..., n i. Theorem 2.5. The solution to the differential equations in Theorem 2.3 is given by G(x) = K i=1 K + G(x) = i=1 n i e λ ix c (i) j j=1 n i e λ ix j=1 a (i) j j k=1 k=1 k x j k (j k)! φ(i) k + c 0xπ + φ 0, if 0 x q, j x j k (j k)! φ(i) k, if x > q, 22

where the coefficients a (i) j s, c(i) j to the following system of linear equations: K + e λ iq i=1 n i j=1 a (i) j ( K i=1 j k=1 n i λ i j=1 s, c 0 and the vector φ 0 are given by the unique solution c (i) j φ(i) j + q j k (j k)! φ(i) kl K i=1 n i j=2 c (i) j φ(i) j 1 n i e λ iq c (i) j j=1 ( K i=1 n i λ i j=1 ) j k=1 RA + c 0 πr(a I) + φ 0 Q = 0, q j k (j k)! φ(i) kl c 0 qπ l φ 0l = 0, l / Ω 0, ) c (i) j φ(i) jl + n i i=1 j=2 c (i) j φ(i) j 1,l j=1 + c 0 π l = 0, l Ω +, ( K ) n i c (i) j φ(i) j + φ 0 e = 1, φ 0 e = 1, where φ (i) kl is the l-th element in φ (i) k. Proof. Follow the same lines as in the proof of Theorem 2.4 with only changes in the general solution to the homogeneous equations. Remark. When dealing with large matrices, the generalized eigenvectors are often numerically difficult to compute. There are alternative methods that are numerically better behaved to evaluate the general solution to the homogeneous equations, e.g., Putzer (1966). 2.6 A Special Case: A = I In this section we consider a special case where the background process state does not change when the fluid level jumps to q, i.e., the case A = I. 23

2.6.1 Stochastic Decomposition Property When A = I, as mentioned before, there exists an interesting stochastic decomposition property of the limiting distribution of the {(X(t), Z(t)), t 0} process, which says in steady state the buffer content in the fluid model with jumps is the sum of two independent random variables: a U(0, q) random variable and the buffer content in a fluid model with no jumps. Next we shall prove this decomposition property from two different aspects. 2.6.1.1 Laplace Stieltjes Transform Method Let F j (x) = lim t P {X(t) x, Z(t) = j}, x 0, j Ω, (2.21) and F j (s) = 0 e sx df j (x) (2.22) be the Laplace Stieltjes transform (LST) of F j (x), and F (s) = [ F1 (s), F 2 (s),..., F n (s)]. (2.23) The next theorem gives the stochastic decomposition property of the limiting distribution of the {(X(t), Z(t)), t 0} process. Theorem 2.6. Suppose A = I and the stability condition (2.10) holds. Then F (s) = 1 e sq sq H(s), (2.24) where H(s) is the LST of the limiting distribution function of the standard fluid model without jumps, given by Equation (2.9). 24

Proof. Since A = I and the stability condition (2.10) holds, we have G(0) = π and hence G(x) = π F (x). Clearly, F (0) = 0. Thus F (x)r = F (x)q + F (0)R, 0 x q i, F (x)r = F (x)q, x > q i. Thus we have q 0 e sx F (x)dx R = q 0 e sx (F (x)q + F (0)R)dx and e sx F (x)dx R = q q e sx F (x)qdx. Thus F (s)r = 0 e sx F (x)qdx + = 1 s F (s)q + 1 s q 0 e sx F (0)Rdx ( 1 e sq ) F (0)R. (2.25) Rearranging Equation (2.25), we get F (s) = 1 e sq sqf (0)R(sR Q) 1. (2.26) sq Recalling from section 2.2, for the standard fluid model without jumps, we have H(s) = sh(0)r(sr Q) 1. (2.27) We have seen that there is a unique vector H(0) satisfying Equations (2.4a) and (2.4b) that makes H(s) in Equation (2.27) a valid LST of a random vector. From 25

equation (2.26) it is clear that sqf (0)R(sR q) 1 must be a valid LST of a random vector since (1 e sq )/(sq) is the LST of U(0, 1) random variable. Since the boundary conditions of Equation (2.15c) implies that F j(0) = 0 if j Ω +, which are the same conditions satisfied by H(0) (see Equation (2.4a)), we must have qf (0)R = ch(0)r, for some constant c. The condition F ( )e = 1 implies that c = 1. This proves our result. Remark. Theorem 2.6 indicates that in steady state the buffer content in the fluid model with jumps is the sum of two independent random variables: a U(0, q) random variable and the buffer content in a fluid model with no jumps. Interestingly, similar property has been observed in queuing models with server vacations. See Fuhrmann (1984), Fuhrmann (1985) and Shanthikumar (1986). 2.6.1.2 Sample Path Method We begin by decomposing the {X(t), t 0} process into two components. Let S 0 = 0, X(0) = q and S i be the i-th order point (i 1). Define X 1 (t) = min S n u t {X(u)}, S n t < S n+1 and X 2 (t) = X(t) X 1 (t). Figure 2.2 illustrates the sample paths of the original {X(t), t 0} process and the two resulting processes {X 1 (t), t 0} and {X 2 (t), t 0}. The following two theorems together state the stochastic decomposition properties represented by these 26

Figure 2.2: Decomposition of the X(t) process. component processes {X 1 (t), t 0} and {X 2 (t), t 0}. Theorem 2.7. The process {X 2 (t), t 0} is independent of q. Proof. Assume that Z(0) Ω and define T 1 = min{t 0 : Z(t) Ω + Ω 0 }. Regardless of the value of q, X(t) always decreases over (0, T 1 ), except for possible jumps of size q when it hits zero. Thus X 2 (t) is zero over (0, T 1 ). T 1 is independent of q and hence {X 2 (t), t [0, T 1 )} is independent of q. 27

Now define T 2 = min{t > T 1 : X(t) = X(T 1 )}. Note that T 2 is also independent of q, X 2 (T 1 ) = X 2 (T 2 ) = 0 and X 2 (t) > 0 for t (T 1, T 2 ). The sample path of {X(t), t (T 1, T 2 )} is independent of q, since X(t) never reaches 0 for any t (T 1, T 2 ). Thus the sample path of {X 2 (t), t (T 1, T 2 )} is independent of q. Define T 2n+1 = min{t T 2n : Z(t) Ω + Ω 0 }, and T 2n+2 = min{t T 2n+1 : X(t) = X(T 2n+1 )}. Since {X 2 (t), t 0} goes through these two cycles alternately over (T 2n, T 2n+1 ) and (T 2n+1, T 2n+2 ) independently, it is clear that {X 2 (t), t 0} is independent of q. Theorem 2.8. The limiting distribution of the process {X 1 (t), t 0} is uniform over (0, q). Proof. First note that the sample paths of {X 1 (t), t 0} have right derivative everywhere. Define I(t) = 0 if the right derivative of X 1 (t) is strictly negative at t, and I(t) = 1 if the right derivative of X 1 (t) is zero at t. Now lim P (X 1(t) x) t = lim t P (X 1 (t) x I(t) = 0)P (I(t) = 0) + lim t P (X 1 (t) x I(t) = 1)P (I(t) = 1). (2.28) Next we will show that lim P (X 1(t) x I(t) = ζ) = x/q, ζ {0, 1}. (2.29) t 28

First we construct two new processes {Y 0 (t), t 0} and {Z 0 (t), t 0} by eliminating the segments of the sample paths of {X 1 (t), t 0} and {Z(t), t 0} over the time intervals (T 2n+1, T 2n+2 ] for all n 0. The sample paths of the {Y 0 (t), t 0} and {Z 0 (t), t 0} processes corresponding to the sample paths of {X 1 (t), t 0} and {Z(t), t 0} are shown in Figure 2.3. From Figure 2.3 we can see that {Y 0 (t), t 0} can be thought of as a fluid model modulated by the stochastic process {Z 0 (t), t 0} with state space Ω. It can be seen that {Z 0 (t), t 0} is a CTMC with generator matrix ˆQ = [ˆq ij ], (i, j Ω ) given by ˆq ij = q ij + q ik η kj, i, j Ω, (2.30) k Ω + Ω 0 where η kj = P (Z(T 2n+2 ) = j Z(T 2n+1 ) = k), k Ω + Ω 0, j Ω. (2.31) Thus the {(Y 0 (t), Z 0 (t)), t 0} process satisfies the hypothesis of Theorem 2.9. Hence it follows that lim P (Y 0(t) x, Z 0 (t) = i) = x t q ˆπ i, (2.32) where ˆπ i is the steady-state probability of the CTMC with generator matrix ˆQ in state i. However, our construction of the Y 0 process implies that lim P (Y 0(t) x, Z 0 (t) = i) = lim P (X 1(t) x I(t) = 0). t t This proves Equation (2.29) for ζ = 0. Now for ζ = 1, we define Y 1,n = X 1 (T + 2n+1) and Z 1,n = Z(T + 2n+1) for n 0. Now construct a semi-markov process (SMP) {(Z 1 (t), Y 1 (t)), t 0} with embedded DTMC {(Z 1,n, Y 1,n ), n 0}, so that the n-th sojourn time of this SMP is given by 29

Figure 2.3: Correspondence of the processes X(t), Z(t), X 1 (t), Y 0 (t) and Z 0 (t). 30

T 2n+2 T 2n+1. Clearly the sample path of {Y 1 (t), t 0} is identical to the one obtained by eliminating the segments of the sample path of {X 1 (t), t 0} over the intervals (T 2n, T 2n+1 ] for all n 0. Figure 2.4 illustrates the sample paths of the {Y 1 (t), t 0} and {Z 1 (t), t 0} processes corresponding to the sample paths of {X 1 (t), t 0} and {Z(t), t 0} processes. Define f(j, x)dx = lim t P {Z(t) = j, x Y 1 (t) x + dx}. (2.33) According to the theory of SMP (see Kulkarni (1995)), where π(j, x)τ(j, x)dx f(j, x)dx = q, (2.34) π(k, y)τ(k, y)dy y=0 k Ω + Ω 0 π(j, x)dx = lim n P {Z 1,n = j, x Y 1,n x + dx}, (2.35) and τ(j, x) is the expected sojourn time of the SMP in state (j, x). Clearly τ(j, x) is independent of x, hence we denote τ(j, x) as τ j for all x. Let g(i, x)dx = lim t P {Z(t) = i, x Y 0 (t) x + dx}, (i Ω ). (2.36) From Equation (2.32), we see that g(i, x) = ˆπ i q, (i Ω ). (2.37) Hence using Equation (2.37), π(j, x) = g(i, x)q ij = 1 ˆπ i q ij. (2.38) q i Ω i Ω 31