Stochastic processes

Transcription

1 Stochastic processes M. Veeraraghavan; Feb. 10, 2004 A stochastic process (SP is a family of random variables { Xt (t T} defined on a given probability space, indexed by the time variable t, where t varies over an index set T. [1] Just as a random variable assigns a number to each outcome s in a sample space S, a stochastic process assigns a sample function xts (, to each outcome s. 1 A sample function xts (, is the time function associated with outcome s of an experiment. The ensemble of a stochastic process (sp is the set of all possible time functions that can result from an experiment. xts (, 1 Sample function s 1 s 2 s 3 xts (, 2 xts (, 3 t = t 1 Xt ( 1 is a random variable Figure 1:Representation of a stochastic process - relation to random variables Example of an sp: Number of active calls Mt ( at a switch at time t. One trial of the experiment yields the sample function mts (, where the number of active calls is measured for every second over one 15 minute interval. Say this measurement is taken every day starting at 10AM. An ensemble average can be obtained from all measurements for t = 2min average can be obtained over a 15-minute interval based on one-day s measurements. after 10AM. Or a time 1. Most of the statements in this writeup have been taken verbatim from [2]; exceptions are primary from [1] as noted. 1

2 Types of stochastic processes: Discrete value and continuous value; Discrete time and continuous time. If each random variable Xt ( for different values of t are discrete rv, then the sp is a discrete value. If the process is defined only for discrete time instants, then it is a discrete time sp. Random sequence: for a discrete time process, a random sequence X n is an ordered sequence of random variables X 0, X 1,...- Essentially a random sequence is a discrete-time stochastic process. Relation between sp and rv: A discrete value sp is defined by the joint PMF, ( ( x 1 x k, while a continuous value sp is defined by the joint PDF. P Xt1 ( Xt k Independent, identically distributed random sequence: Let X n be an iid random sequence. All X i has the same distribution. Therefore P Xi ( x = P X ( x. For a discrete value process, the sample vector X n1 X nk has the joint PMF (by property of independent random variables P Xt1 ( Xt ( k, ( x 1 x k = P X ( x 1 P X ( x 2 P X ( x n = P X ( x i For continuous value sp, same operation but with pdf. n i = 1 (1 A counting process: A stochastic process Nt ( is a counting process if for every sample function nts (, = 0 for t < 0 and nts (, is integer valued and nondecreasing with time. N(t S 1 S 2 S 3 S 4 X 1 X 2 X 3 X 4 Figure 2:Sample path of a counting process 2

3 A counting process is always a discrete-value SP (because either continuous-time or discrete-time. nts (, is integer valued but can be A renewal process is a counting process in which the interarrival times X 1, X 2, are an iid random sequence. Poisson process: is a counting process: in which number of arrivals in ( t 0, t 1 ], which is Nt ( 1 Nt ( 0, is a Poisson random variable with expected value λ( t 1 t 0 For any pair of non-overlapping intervals ( t 0, t 1 ] and ( t 0 ', t 1 ' ], number of arrivals in each interval Nt ( 1 Nt ( 0 and Nt ( 1 ' Nt ( 0 ' are independent random variables. M, the number of arrivals in the interval ( t 0, t 1 ], which is Nt ( 1 Nt ( 0, has a PMF P M ( m = [ λ( t 1 t 0 ] m e λ ( t 1 t m = 01,, m! 0 otherwise (see RV.pdf for Poisson rv pmf (2 A Poisson process is a renewal process, i.e., the X s form an iid random sequence. Consider a set of sample functions of a Poisson process as in Fig. 1. If we take the random variable Nt ( 1 at time instant t 1 (just like Xt ( 1 in Fig. 1 then the distribution of this random variable is Poisson with parameter λ t 1 because it represents the cumulative number of arrivals until time t 1 from time 0. It is not sufficient to describe the distribution of the random variable at any one instant in time to define an SP. To completely define a stochastic process, we need to give the joint PMF, which is given by the following Theorem. Theorem 6.2 of [2]: For a Poisson process of rate λ, the joint PMF of Nt ( 1 Nt ( k, t 1 < < t k is 3

4 P Nt1 ( Nt ( k,, ( n 1 n k = n α 1 1 e α α 2 n 2 n 1 e α 2 n k n k 1 e α k n 1! ( n 2 n 1! α k ( n k n k 1! 0 n n 1 k 0 otherwise (3 where α i = λ( t i t i 1. Why is the following incorrect?,, ( n 1 n k = P Nt1 ( ( n 1P Nt2 ( ( n 2 P Ntk ( ( n k P Nt1 ( Nt ( k (4 This is because Nt ( 1, Nt ( 2 Nt ( k are not independent random variables. ( 0, t 2 overlaps with ( 0, t 1, hence Nt ( 2 is not independent of Nt ( 1. We use the independence rule PA ( B = PA ( PB (, where A = N( t 1 and k α α 1e 1 B = N( t 2 Nt ( 1, both of which are Poisson r.v. and hence we use the PMF k! To fully specify a stochastic process, is it sufficient to define the distributions of the random variables at different instants in time? Answer is no: we need the joint PMF of these random variables. If the SP is a renewal process, the intervals are independent and so using the independence rule, the joint PMF is easy to define. Properties of Poisson processes: I. Relation between a Poisson process and exponential distribution: For a Poisson process of rate λ, the interarrival times X 1, X 2, are an iid with the exponential PDF with parameter λ. [2, page 214]: Proof [4, page 35]: The first time interval, i.e., time to the first arrival follows: X 1, can be characterized as PX ( 1 > t = PNt ( ( = 0 = e λt (5 This means X 1 is exponentially distributed with parameter λ. Now to find the distribution of X 2 conditional on : X 1 4

5 PX ( 2 > t X 1 = s = P{ 0 arrivals in ( s, s+ t] X 1 = s} = P{ 0 arrivals in ( s, s+ t] } = e λt (6 Since the number of arrivals in the interval ( s, s + t] is independent of X 1, we drop the condition in the second step of the above equation. 1 The above means that X 2 is also an exponential random variable with mean --. Furthermore we λ can conclude from the above that X 2 is independent of X 1. Note that Nt ( 2 t 1 and Nt ( 3 t 2 are not iid unles ( t 2 t 1 is equal to ( t 3 t 2 if they are non-overlapping. They are independent, but do not have the same parameter in order to be identically distributed. Extending the argument, let t n 1 = x 1 + x x n 1, where X 1 = x 1 X n 1 = x n 1. PX ( n > x X 1 = x 1, X n 1 = x n 1 = PNt ( ( n 1 + x Nt ( (7 n = 0 X 1 1 = x 1, X n 1 = x n 1 because if Nt ( n 1 + x Nt ( n 1 is not 0 then it means X n ended before x time passed. Since the event Nt ( n 1 + x Nt ( is independent of the lengths of, we can n = 0 X 1 1 X n 1 equate (7 to be PNt ( ( n 1 + x Nt (. Setting, n = 0 Nt ( 1 n 1 + x Nt ( n = M P 1 M ( 0 from (2 is equal to P M ( 0 ( λx 0 e λx = = e λx. Since the number of arrivals in the interval 0! [ t n 1, t n 1 + x] is independent of the past history described in X 1, X n 1. Conclusion: X n are an exponentially distributed random variable, and is independent of the interval n. In other words, X 1, X 2, are iid random variables. II. Approximating exponential functions: For every t 0 and δ 0, P{ N( t + δ Nt ( = 0} = 1 λδ + o( δ P{ N( t + δ Nt ( = 1} = λδ + o( δ P{ N( t+ δ Nt ( = 2} = o( δ (8 (9 (10 where lim o( δ = 0 (11 δ 0 δ These can be verified by using Taylor s series expansion of e λδ ( λδ 2 = 1 λδ

6 III. Merging of Poisson processes: If two more independent Poisson processes A 1, A k are merged into a single process A = A A k, the latter is a Poisson process with its parameter equal to the sum of the rates of its components. IV. Splitting of a Poisson process: If a Poisson process is split into two processes by independently assigning each arrival to the first and second of these processes with probability p and ( 1 p, respectively, the two arrival processes are Poisson (note: it is essential that the assignment of each arrival is independent of the previous assignment; if for example, all even arrivals are sent to the first queue and all odd to the second, the two processes will not be Poisson. Examples of stochastic processes: 1. Counting process 2. Renewal process 3. Poisson process 4. Markov process 5. Brownian motion Note that autocovariance and autocorrelation are time-varying functions (unlike covariance and correlation of two random variables, which are numbers. From [2, pg. 217, 226]: Autocovariance and autocorrelation functions indicate the rate of change of the sample functions of a stochastic process. Autocovariance function of a stochastic process Xt ( is C X τ = Cov[ X(Xt t, ( + τ ] (12 Autocorrelation function of a stochastic process Xt ( is R X τ = EXt [ (Xt ( + τ ] (13 Relation between these functions: C X τ µ X (µ t X ( t + τ (14 Stationary process [2, pg. 220, 226]: An SP is stationary if the randomness does not vary with time. [Strict stationary] 6

7 A stochastic process Xt ( is stationary if and only if for all sets of time instants t 1, t m, and any time difference τ (i.e., joint pdf does not change with time f Xt1 ( Xt ( m, ( x 1 x m = f Xt1 ( + τ, Xt ( m + τ ( x 1 x m (15 Same thing for a stationary random sequence - where the time difference is a discrete value k. Properties of a stationary process (mean stays constant and autocovariance and autocorrelation only depend on time interval: µ X ( t = µ X R X ( 0, ( τ 2 C X ( τ µ X = C X ( τ (16 Wide-sense stationary process [2, pg. 223, 226]: An SP is w.s.stationary if the expected value is constant with time and the autocorrelation depends only on the time difference between two random variables. Xt ( is a w.s. stationary process if and only if for all t : EXt [ (] = µ X R X ( 0, ( τ (17 Similar definition for a wide-sense stationary random sequence. Markov process [1, pg. 337] is an sp whose dynamic behavior is such that probability distribution for its future development depends only on its present state and not how the process arrived in that state. If state space is discrete, then it is a Markov chain. If X 's are discrete random variables we get this Definition of a DTMC: A DTMC { X n n = 012,,, } is a discrete time discrete value random sequence such that given X 0, X 1 X n 1, the next random variable X n depends only on X n 1 through the transition probability PX [ n = j X n 1 = i, X n 2 = i n 2 X 0 = i o ] = PX [ n = j X n 1 = i] = P ij (18 7

8 Definition of a CTMC [3]: A CTMC is { Xt (t 0} is a continuous time, discrete value random process if for t o < t 1 < < t n < t, with t and t r 0 for r = 12,,, n, PXt [ ( = j Xt ( n = ixt, ( n 1 = i n 1 Xt ( 0 = i o ] = PXt [ ( = j Xt ( n = i] (19 Relationship between Markov chain and Poisson process: Is the Poisson process a special case of a Markov chain? It is indeed a CTMC. Here s why. By definition of Poisson process, the number of arrivals in two non-overlapping intervals are independent; therefore the number of arrivals in ( t n, t is independent of the arrivals in ( t 0, t n if t o < t 1 < < t n < t. The random variable Nt ( does depend upon Nt ( n. Its dependence on Nt ( n is as follows: PNt [ ( = j Nt ( n = i] = PNt [ ( Nt ( n = j i] (20 But it does not depend on Nt ( 0, Nt ( 1, etc. Therefore, P[ Nt ( = j Nt ( n = int, ( n 1 = i n 1 Nt ( 0 = i o ] = PNt [ ( = j Nt ( n = i], (21 which by the definition of a CTMC as per (19 implies the Poisson process is a Markov chain. Sure enough, [see 3, page 389], says a pure birth Markov process with the constraint of constant birth rates is the same as a Poisson process. A pure birth process is a counting process. Definition of a CTMC: [2, page 381] - actually true for time-homogeneous CTMCs A CTMC { Xt (t 0} is a continuous time, discrete value random process such that for an infinitesimal time step of size, PXt [ ( + = j Xt ( = i] = q ij PXt [ ( + = i Xt ( = i] = 1 q ij j i (22 The above definition is somewhat simplistic since it treats transition rates time, which is true for time-homogeneous CTMCs. Compare the definition of a CTMC in (22 with the definition of a Poisson process. q ij to be independent of 8

9 Xt ( in (22 is the same as Nt ( of the Poisson process definition, and q ij = 0 if ( j ( i+ 1 q ii ( + 1 = λ (23 Time-homogeneous Markov chains (3, page 361, 363: A Markov chain { Xt (t 0} is said to be time-homogeneous (or is said to have stationary transition probabilities if p ij ( ν, t depends only on the time difference t ν, where p ij ( ν, t = PXt [ ( = jxν ( = i]. In other words, p ij ( t = PXt [ ( + u = jxu ( = i] for any u 0. (24 In time-homogeneous Markov chains, the transition rates q ij ( t and q j ( t are independent of time. For example, in Poisson process definition, we see a similar dependence on just the interval, and λ being independent of time. q ij ( t lim p ij ( tt, + h = pij ( ν, t = t ν = t h 0 h (25 lim 1 p ij t+ h q j ( t = ( p t jj ( ν, t = ν = t h 0 h Relation between counting process and a Markov chain: A Markov chain could be a counting process (such as a pure-birth Markov process, but in general it could be a non-counting process (note that a Markov chain is a discrete state process as is a counting process. The state of the variable represented by Markov chain can increase or decrease in a sample function. (26 Counting processes are necessarily non-stationary (they are even not wide sense stationary because Nt ( keeps increasing with time (unless they are constant, because the definition of a counting process says non-decreasing, not increasing. A Markov process is hence quite general - the q ij could be functions of time; it could be nondecreasing (counting or a process that increases and decreases. And hence M in M/M/1 should 9

10 stand for Memoryless not Markov. The arrival process is Poisson, which is a special case of Markov. References [1] K. S. Trivedi, Probability, Statistics with Reliability, Queueing and Computer Science Applications, Second Edition, Wiley, 2002, ISBN [2] R. Yates and D. Goodman, Probability and Stochastic Processes, Wiley, ISBN [3] K. S. Trivedi, Probability, Statistics with Reliability, Queueing and Computer Science Applications, First Edition, Prentice Hall, ISBN [4] S. Ross, Stochastic Processes, Wiley