Markov Chains: An Introduction/Review
|
|
|
- Job Owen
- 7 years ago
- Views:
Transcription
1 Markov Chains: An Introduction/Review David Sirl dsirl/ AUSTRALIAN RESEARCH COUNCIL Centre of Excellence for Mathematics and Statistics of Complex Systems Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 1
2 Andrei A. Markov ( ) Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 2
3 Random Processes A random process is a collection of random variables indexed by some set I, taking values in some set S. I is the index set, usually time, e.g. Z +, R, R +. S is the state space, e.g. Z +, R n, {1, 2,...,n}, {a,b,c}. We classify random processes according to both the index set (discrete or continuous) and the state space (finite, countable or uncountable/continuous). Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 3
4 Markov Processes A random process is called a Markov Process if, conditional on the current state of the process, its future is independent of its past. More formally, X(t) is Markovian if has the following property: P(X(t n ) = j n X(t n 1 ) = j n 1,..., X(t 1 ) = j 1 ) = P(X(t n ) = j n X(t n 1 ) = j n 1 ) for all finite sequences of times t 1 <... < t n I and of states j 1,...,j n S. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 4
5 Time Homogeneity A Markov chain (X(t)) is said to be time-homogeneous if P(X(s + t) = j X(s) = i) is independent of s. When this holds, putting s = 0 gives P(X(s + t) = j X(s) = i) = P(X(t) = j X(0) = i). Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 5
6 Time Homogeneity A Markov chain (X(t)) is said to be time-homogeneous if P(X(s + t) = j X(s) = i) is independent of s. When this holds, putting s = 0 gives P(X(s + t) = j X(s) = i) = P(X(t) = j X(0) = i). Probabilities depend on elapsed time, not absolute time. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 5
7 Discrete-time Markov chains At time epochs n = 1, 2, 3,... the process changes from one state i to another state j with probability p ij. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 6
8 Discrete-time Markov chains At time epochs n = 1, 2, 3,... the process changes from one state i to another state j with probability p ij. We write the one-step transition matrix P = (p ij, i,j S). Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 6
9 Discrete-time Markov chains At time epochs n = 1, 2, 3,... the process changes from one state i to another state j with probability p ij. We write the one-step transition matrix P = (p ij, i,j S). Example: a frog hopping on 3 rocks. Put S = {1, 2, 3}. P = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 6
10 DTMC example Example: A frog hopping on 3 rocks. Put S = {1, 2, 3}. P = We can gain some insight by drawing a picture: 3 1/2 1/3 2/3 1/4 1/ /8 1/8 Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 7
11 DTMCs: n-step probabilities We have P, which tells us what happens over one time step; lets work out what happens over two time steps: p (2) ij = P(X 2 = j X 0 = i) = k S P(X 1 = k X 0 = i) P(X 2 = j X 1 = k, X 0 = i) = k S p ik p kj. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 8
12 DTMCs: n-step probabilities We have P, which tells us what happens over one time step; lets work out what happens over two time steps: p (2) ij = P(X 2 = j X 0 = i) = k S P(X 1 = k X 0 = i) P(X 2 = j X 1 = k, X 0 = i) = k S p ik p kj. So P (2) = PP = P 2. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 8
13 DTMCs: n-step probabilities We have P, which tells us what happens over one time step; lets work out what happens over two time steps: p (2) ij = P(X 2 = j X 0 = i) = k S P(X 1 = k X 0 = i) P(X 2 = j X 1 = k, X 0 = i) = k S p ik p kj. So P (2) = PP = P 2. Similarly, P (3) = P 2 P = P 3 and P (n) = P n. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 8
14 DTMC: Arbitrary initial distributions We may wish to start the chain according to some initial distribution π (0). Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 9
15 DTMC: Arbitrary initial distributions We may wish to start the chain according to some initial distribution π (0). We can then calculate the state probabilities π (n) = (π (n) j, j S) of being in state j at time n as follows: π (n) j = k S P(X 0 = k) P(X n = j X 0 = k) = k S π (0) j p (n) ij. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 9
16 DTMC: Arbitrary initial distributions We may wish to start the chain according to some initial distribution π (0). We can then calculate the state probabilities π (n) = (π (n) j, j S) of being in state j at time n as follows: π (n) j = k S P(X 0 = k) P(X n = j X 0 = k) = k S π (0) j p (n) ij. Or, in matrix notation, π (n) = π (0) P n ; similarly we can show that π (n+1) = π (n) P. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 9
17 Class structure We say that a state i leads to j (written i j) if it is possible to get from i to j in some finite number of jumps: p (n) ij > 0 for some n 0. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 10
18 Class structure We say that a state i leads to j (written i j) if it is possible to get from i to j in some finite number of jumps: p (n) ij > 0 for some n 0. We say that i communicates with j (written i j) if i j and j i. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 10
19 Class structure We say that a state i leads to j (written i j) if it is possible to get from i to j in some finite number of jumps: p (n) ij > 0 for some n 0. We say that i communicates with j (written i j) if i j and j i. The relation partitions the state space into communicating classes. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 10
20 Class structure We say that a state i leads to j (written i j) if it is possible to get from i to j in some finite number of jumps: p (n) ij > 0 for some n 0. We say that i communicates with j (written i j) if i j and j i. The relation partitions the state space into communicating classes. We call the state space irreducible if it consists of a single communicating class. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 10
21 Class structure We say that a state i leads to j (written i j) if it is possible to get from i to j in some finite number of jumps: p (n) ij > 0 for some n 0. We say that i communicates with j (written i j) if i j and j i. The relation partitions the state space into communicating classes. We call the state space irreducible if it consists of a single communicating class. These properties are easy to determine from a transition probability graph. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 10
22 Classification of states We call a state i recurrent or transient according as P(X n = i for infinitely many n) is equal to one or zero. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 11
23 Classification of states We call a state i recurrent or transient according as P(X n = i for infinitely many n) is equal to one or zero. A recurrent state is a state to which the process always returns. A transient state is a state which the process eventually leaves for ever. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 11
24 Classification of states We call a state i recurrent or transient according as P(X n = i for infinitely many n) is equal to one or zero. A recurrent state is a state to which the process always returns. A transient state is a state which the process eventually leaves for ever. Recurrence and transience are class properties; i.e. if two states are in the same communicating class then they are recurrent/transient together. We therefore speak of recurrent or transient classes Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 11
25 Classification of states We call a state i recurrent or transient according as P(X n = i for infinitely many n) is equal to one or zero. A recurrent state is a state to which the process always returns. A transient state is a state which the process eventually leaves for ever. Recurrence and transience are class properties; i.e. if two states are in the same communicating class then they are recurrent/transient together. We therefore speak of recurrent or transient classes We also assume throughout that no states are periodic. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 11
26 DTMCs: Two examples S irreducible: 3 1/2 1/3 2/3 1/4 1/ /8 1/8 P = S = {0} C, where C is a transient class: 1/ /2 1/4 3 2/3 1/4 5/8 1/3 1/8 P = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 12
27 DTMCs: Quantities of interest Quantities of interest include: Hitting probabilities. Expected hitting times. Limiting (stationary) distributions. Limiting conditional (quasistationary) distributions. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 13
28 DTMCs: Hitting probabilities Let α i be the probability of hitting state 1 starting in state i. Clearly α 1 = 1; and for i 1, α i = P(hit 1 start in i) = P(X 1 = k X 0 = i) P(hit 1 start in k) k S = k S p ik α k Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 14
29 DTMCs: Hitting probabilities Let α i be the probability of hitting state 1 starting in state i. Clearly α 1 = 1; and for i 1, α i = P(hit 1 start in i) = P(X 1 = k X 0 = i) P(hit 1 start in k) k S = k S p ik α k Sometimes there may be more than one solution α = (α i, i S) to this system of equations. If this is the case, then the hitting probabilites are given by the minimal such solution. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 14
30 Example: Hitting Probabilities 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P = Let α i be the probability of hitting state 3 starting in state i. So α 3 = 1 and α i = k p ikα k : α 0 = α 0 α 1 = 2 1α α α 3 α 2 = 5 8 α α α 3 Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 15
31 Example: Hitting Probabilities 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P = Let α i be the probability of hitting state 3 starting in state i. α = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 15
32 DTMCs: Hitting probabilities II Let β i be the probability of hitting state 0 before state N, starting in state i. Clearly β 0 = 1 and β N = 0. For 0 < i < N, β i = P(hit 1 before n start in i) = k S P(X 1 = k X 0 = i) P(hit 1 before n start in k) = k S p ik β k Again, we take the minimal solution of this system of equations. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 16
33 Example: Hitting Probabilities II 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P = Let β i be the probability of hitting 0 before 3 starting in i. So β 0 = 1, β 3 = 0 and β i = k p ikβ k : β 1 = 2 1β β β 3 β 2 = 5 8 β β β 3 Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 17
34 Example: Hitting Probabilities II 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P = Let β i be the probability of hitting 0 before 3 starting in i. β = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 17
35 DTMCs: Expected hitting times Let τ i be the expected time to hit state 1 starting in state i. Clearly τ 1 = 0; and for i 0, τ i = E(time to hit 1 start in i) = 1 + P(X 1 = k X 0 = i) E(time to hit 1 start in k) k S = 1 + k S p ik τ k If there are multiple solutions, take the minimal one. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 18
36 Example: Expected Hitting Times 1/2 3 2/3 1/4 1/ /8 1/3 1/8 P = Let τ i be the expected time to hit 2 starting in i. So τ 2 = 0 and τ i = 1 + k p ikτ k : τ 1 = τ τ 3 τ 3 = τ τ 2 Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 19
37 Example: Expected Hitting Times 1/2 3 2/3 1/4 1/ /8 1/3 1/8 P = Let τ i be the expected time to hit 2 starting in i. τ = = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 19
38 DTMCs: Hitting Probabilities and Times Just systems of linear equations to be solved. In principle can be solved analytically when S is finite. When S is an infinite set, if P has some regular structure (p ij same/similar for each i) the resulting systems of difference equations can sometimes be solved analytically. Otherwise we need numerical methods. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 20
39 DTMCs: The Limiting Distribution Assume that the state space is irreducible, aperiodic and recurrent. What happens to the state probabilities π (n) j as n? Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 21
40 DTMCs: The Limiting Distribution Assume that the state space is irreducible, aperiodic and recurrent. What happens to the state probabilities π (n) j as n? We know that π (n+1) = π (n) P. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 21
41 DTMCs: The Limiting Distribution Assume that the state space is irreducible, aperiodic and recurrent. What happens to the state probabilities π (n) j as n? We know that π (n+1) = π (n) P. So if there is a limiting distribution π, it must satisfy π = πp (and i π i = 1). (Such a distribution is called stationary.) Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 21
42 DTMCs: The Limiting Distribution Assume that the state space is irreducible, aperiodic and recurrent. What happens to the state probabilities π (n) j as n? We know that π (n+1) = π (n) P. So if there is a limiting distribution π, it must satisfy π = πp (and i π i = 1). (Such a distribution is called stationary.) This limiting distribution does not depend on the initial distribution. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 21
43 DTMCs: The Limiting Distribution Assume that the state space is irreducible, aperiodic and recurrent. What happens to the state probabilities π (n) j as n? We know that π (n+1) = π (n) P. So if there is a limiting distribution π, it must satisfy π = πp (and i π i = 1). (Such a distribution is called stationary.) This limiting distribution does not depend on the initial distribution. When the state space is infinite, it may happen that π (n) j 0 for all j. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 21
44 Example: The Limiting Distribution 3 1/2 1/3 2/3 1/4 1/ /8 1/8 P = Substituting P into π = πp gives π 1 = 5 8 π π 3, π 2 = 1 2 π π π 3, π 3 = 1 2 π π 2, which together with i π i = 1 yields π = ( ) ( ). Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 22
45 TMCs: The Limiting Conditional Dist n Assume that the state space is consists of an absorbing state and a transient class (S = {0} C). The limiting distribution is (1, 0, 0,...). Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 23
46 TMCs: The Limiting Conditional Dist n Assume that the state space is consists of an absorbing state and a transient class (S = {0} C). The limiting distribution is (1, 0, 0,...). Instead of looking at the limiting behaviour of we need to look at P(X n = j X 0 = i) = p (n) ij, P(X n = j X n 0, X 0 = i) = p(n) ij 1 p (n) i0 for i,j C. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 23
47 TMCs: The Limiting Conditional Dist n It turns out we need a solution m = (m i, i C) of for some r (0, 1). mp C = rm, Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 24
48 TMCs: The Limiting Conditional Dist n It turns out we need a solution m = (m i, i C) of mp C = rm, for some r (0, 1). If C is a finite set, there is a unique such r. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 24
49 TMCs: The Limiting Conditional Dist n It turns out we need a solution m = (m i, i C) of for some r (0, 1). mp C = rm, If C is a finite set, there is a unique such r. If C is infinite, there is r (0, 1) such that all r in the interval [r, 1) are admissible; and the solution corresponding to r = r is the LCD. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 24
50 Example: Limiting Conditional Dist n 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 25
51 Example: Limiting Conditional Dist n 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P C = Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 25
52 Example: Limiting Conditional Dist n 3 1/4 1/3 2/3 1/4 1/ /2 5/8 1/8 P C = Solving mp C = rm, we get r and m (0.45, 0.30, 0.24) Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 25
53 DTMCs: Summary From the one-step transition probabilities we can calculate: n-step transition probabilities, hitting probabilities, expected hitting times, limiting distributions, and limiting conditional distributions. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 26
54 Continuous Time In the real world, time is continuous things do not happen only at prescribed, equally spaced time points. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 27
55 Continuous Time In the real world, time is continuous things do not happen only at prescribed, equally spaced time points. Continuous time is slightly more difficult to deal with as there is no real equivalent to the one-step transition matrix from which one can calculate all quantities of interest. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 27
56 Continuous Time In the real world, time is continuous things do not happen only at prescribed, equally spaced time points. Continuous time is slightly more difficult to deal with as there is no real equivalent to the one-step transition matrix from which one can calculate all quantities of interest. The study of continuous-time Markov chains is based on the transition function. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 27
57 CTMCs: Transition Functions If we denote by p ij (t) the probability of a process starting in state i being in state j after elapsed time t, then we call P(t) = (p ij (t), i,j S, t > 0) the transition function of that process. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 28
58 CTMCs: Transition Functions If we denote by p ij (t) the probability of a process starting in state i being in state j after elapsed time t, then we call P(t) = (p ij (t), i,j S, t > 0) the transition function of that process. P(t) is difficult/impossible to write down in all but the simplest of situations. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 28
59 CTMCs: Transition Functions If we denote by p ij (t) the probability of a process starting in state i being in state j after elapsed time t, then we call P(t) = (p ij (t), i,j S, t > 0) the transition function of that process. P(t) is difficult/impossible to write down in all but the simplest of situations. However all is not lost: we can show that there exist quantities q ij, i,j S satisfying p ij (t) lim, i j, q ij = p ij(0 + ) = t 0 t 1 p ii (t) lim, i = j. t 0 t Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 28
60 CTMCs: The q-matrix We call the matrix Q = (q ij, i,j S) the q-matrix of the process and can interpret it as follows: For i j, q ij [0, ) is the instantaneous rate the process moves from state i to state j, and q i = q ii [0, ] is the rate at which the process leaves state i. We also have j i q ij q i. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 29
61 CTMCs: The q-matrix We call the matrix Q = (q ij, i,j S) the q-matrix of the process and can interpret it as follows: For i j, q ij [0, ) is the instantaneous rate the process moves from state i to state j, and q i = q ii [0, ] is the rate at which the process leaves state i. We also have j i q ij q i. When we formulate a model, it is Q that we can write down; so the question arises, can we recover P( ) from Q = P (0)? Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 29
62 CTMCs: The Kolmogorov DEs If we are given a conservative q-matrix Q, then a Q-function P(t) must satisfy the backward equations P (t) = QP(t), t > 0, and may or may not satisfy the forward (or master) equations P (t) = P(t)Q, t > 0, with the initial condition P(0) = I. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 30
63 CTMCs: The Kolmogorov DEs If we are given a conservative q-matrix Q, then a Q-function P(t) must satisfy the backward equations P (t) = QP(t), t > 0, and may or may not satisfy the forward (or master) equations P (t) = P(t)Q, t > 0, with the initial condition P(0) = I. There is always one such Q-function, but there may also be infinitely many such functions so Q does not necessarily describe the whole process. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 30
64 CTMCs: Interpreting the q-matrix Suppose X(0) = i: The holding time H i in state i is exponentially distributed with parameter q i, i.e. P(H i t) = 1 e q it, t 0. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 31
65 CTMCs: Interpreting the q-matrix Suppose X(0) = i: The holding time H i in state i is exponentially distributed with parameter q i, i.e. P(H i t) = 1 e q it, t 0. After this time has elapsed, the process jumps to state j with probability q ij /q i. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 31
66 CTMCs: Interpreting the q-matrix Suppose X(0) = i: The holding time H i in state i is exponentially distributed with parameter q i, i.e. P(H i t) = 1 e q it, t 0. After this time has elapsed, the process jumps to state j with probability q ij /q i. Repeat... Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 31
67 CTMCs: Interpreting the q-matrix Suppose X(0) = i: The holding time H i in state i is exponentially distributed with parameter q i, i.e. P(H i t) = 1 e q it, t 0. After this time has elapsed, the process jumps to state j with probability q ij /q i. Repeat... Somewhat surprisingly, this recipe does not always describe the whole process. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 31
68 CTMCs: An Explosive Process Consider a process described by the q-matrix λ i if j = i + 1, q ij = λ i if j = i, 0 otherwise. Assume λ i > 0, i S. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 32
69 CTMCs: An Explosive Process Consider a process described by the q-matrix λ i if j = i + 1, q ij = λ i if j = i, 0 otherwise. Assume λ i > 0, i S. Suppose we start in state i 0. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 32
70 CTMCs: An Explosive Process Consider a process described by the q-matrix λ i if j = i + 1, q ij = λ i if j = i, 0 otherwise. Assume λ i > 0, i S. Suppose we start in state i 0. Stay for time H i0 exp(λ i0 ) then move to state i 0 + 1, Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 32
71 CTMCs: An Explosive Process Consider a process described by the q-matrix λ i if j = i + 1, q ij = λ i if j = i, 0 otherwise. Assume λ i > 0, i S. Suppose we start in state i 0. Stay for time H i0 exp(λ i0 ) then move to state i 0 + 1, Stay for time H i0 +1 exp(λ i0 +1) then move to i 0 + 2,... Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 32
72 CTMCs: An Explosive Process Consider a process described by the q-matrix λ i if j = i + 1, q ij = λ i if j = i, 0 otherwise. Assume λ i > 0, i S. Suppose we start in state i 0. Stay for time H i0 exp(λ i0 ) then move to state i 0 + 1, Stay for time H i0 +1 exp(λ i0 +1) then move to i 0 + 2,... Define T n = i 0 +n 1 i=i 0 H i to be the time of the nth jump. We would expect T := lim n T n =. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 32
73 CTMCs: An Explosive Process Lemma: Suppose {S n, n 1} is a sequence of independent exponential rv s with respective rates a i, and put S = n=1 S n. Then either S = a.s. or S < a.s., according as i=1 1 a i diverges or converges. We identify S n with the holding times H i0 +n and S with T. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 33
74 CTMCs: An Explosive Process Lemma: Suppose {S n, n 1} is a sequence of independent exponential rv s with respective rates a i, and put S = n=1 S n. Then either S = a.s. or S < a.s., according as i=1 1 a i diverges or converges. We identify S n with the holding times H i0 +n and S with T. If, for example, λ i = i 2, we have so P(T < ) = 1. i=i 0 1 λ i = i=i 0 1 i 2 <, Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 33
75 CTMCs: Reuter s Uniqueness Condition For there to be no explosion possible, we need the backward equations to have a unique solution. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 34
76 CTMCs: Reuter s Uniqueness Condition For there to be no explosion possible, we need the backward equations to have a unique solution. When Q is conservative, this is equivalent to q ij x j = νx i i S j S having no bounded non-negative solution (x i, i S) except the trivial solution x i 0 for some (and then all) ν > 0. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 34
77 CTMCs: Ruling Out Explosion Analysis of a continuous-time Markov process is greatly simplified if it is regular, that is non-explosive. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 35
78 CTMCs: Ruling Out Explosion Analysis of a continuous-time Markov process is greatly simplified if it is regular, that is non-explosive. A process is regular if Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 35
79 CTMCs: Ruling Out Explosion Analysis of a continuous-time Markov process is greatly simplified if it is regular, that is non-explosive. A process is regular if The state space is finite. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 35
80 CTMCs: Ruling Out Explosion Analysis of a continuous-time Markov process is greatly simplified if it is regular, that is non-explosive. A process is regular if The state space is finite. The q-matrix is bounded, that is sup i q i <. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 35
81 CTMCs: Ruling Out Explosion Analysis of a continuous-time Markov process is greatly simplified if it is regular, that is non-explosive. A process is regular if The state space is finite. The q-matrix is bounded, that is sup i q i <. X 0 = i and i is recurrent. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 35
82 CTMCs: Ruling Out Explosion Analysis of a continuous-time Markov process is greatly simplified if it is regular, that is non-explosive. A process is regular if The state space is finite. The q-matrix is bounded, that is sup i q i <. X 0 = i and i is recurrent. Reuter s condition simplifies considerably for a birth-death process, a process where from state i, the only possible transitions are to i 1 or i + 1. We now assume that the process we are dealing with is non-explosive, so Q is enough to completely specify the process. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 35
83 CTMCs: The Birth-Death Process A Birth-Death Process on {0, 1, 2,...} is a CTMC with q-matrix of the form λ i if j = i + 1 µ i if j = i 1, i 1 q ij = (λ i + µ i ) if j = i 1 λ 0 if j = i = 0 0 otherwise where λ i,µ i > 0, i S. We also set π 1 = 1, and π i = λ 1λ 2 λ i 1 µ 2 µ 3 µ i. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 36
84 CTMCs: Quantities of interest Again we look at Hitting probabilities. Expected hitting times. Limiting (stationary) distributions. Limiting conditional (quasistationary) distributions. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 37
85 CTMCs: Hitting Probabilities Using the same reasoning as for discrete-time processes, we can show that the hitting probabilites α i of a state κ, starting in state i, are given by the minimal non-negative solution to the system α κ = 1 and, for i κ, q ij α j = 0. j S Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 38
86 CTMCs: Hitting Probabilities Using the same reasoning as for discrete-time processes, we can show that the hitting probabilites α i of a state κ, starting in state i, are given by the minimal non-negative solution to the system α κ = 1 and, for i κ, q ij α j = 0. j S For a BDP, we can show that the probability of hitting 0 is one if and only if A := i=1 1 λ n π n =. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 38
87 CTMCs: Hitting times Again, we can use an argument similar to that for discrete-time processes to show that the expected hitting times τ i of state κ, starting in i, are given by the minimal non-negative solution of the system τ κ = 0 and, for i κ, q ij τ j = 1. j S Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 39
88 CTMCs: Hitting times Again, we can use an argument similar to that for discrete-time processes to show that the expected hitting times τ i of state κ, starting in i, are given by the minimal non-negative solution of the system τ κ = 0 and, for i κ, q ij τ j = 1. j S For a BDP, the expected time to hit zero, starting in state i is given by i 1 τ i = π k. µ j π j j=1 k=j Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 39
89 CTMCs: Limiting Behaviour As with discrete-time chains, the class structure is important in determining what tools are useful for analysing the long term behaviour of the process. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 40
90 CTMCs: Limiting Behaviour As with discrete-time chains, the class structure is important in determining what tools are useful for analysing the long term behaviour of the process. If the state space is irreducible and positive recurrent, the limiting distribution is the most useful device. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 40
91 CTMCs: Limiting Behaviour As with discrete-time chains, the class structure is important in determining what tools are useful for analysing the long term behaviour of the process. If the state space is irreducible and positive recurrent, the limiting distribution is the most useful device. If the state space consists of an absorbing state and a transient class, the limiting conditional distribution is of most use. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 40
92 CTMCs: Limiting Distributions Assume that the state space S is irreducible and recurrent. Then there is a unique (up to constant multiples) solution π = (π i, i S) such that πq = 0, where 0 is a vector of zeros. If i π i <, then π is can be normalised to give a probability distribution which is the limiting distribution. (If π is not summable then there is no proper limiting distribution.) Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 41
93 CTMCs: Limiting Distributions Assume that the state space S is irreducible and recurrent. Then there is a unique (up to constant multiples) solution π = (π i, i S) such that πq = 0, where 0 is a vector of zeros. If i π i <, then π is can be normalised to give a probability distribution which is the limiting distribution. (If π is not summable then there is no proper limiting distribution.) For the BDP, the potential coefficients π 1 = 1, π i = λ 1λ 2 λ i 1 µ 2 µ 3 µ i are the essentially unique solution of πq = 0. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 41
94 CTMCs: Limiting Conditional Dist ns If the S = {0} C and the absorbing state zero is reached with probability one, the limiting conditional distribution is given by m = (m i, i C) such that for some ν > 0. mq C = νm, Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 42
95 CTMCs: Limiting Conditional Dist ns If the S = {0} C and the absorbing state zero is reached with probability one, the limiting conditional distribution is given by m = (m i, i C) such that for some ν > 0. mq C = νm, When C is a finite set then there is a unique such ν. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 42
96 CTMCs: Summary Countable state Markov chains are stochastic modelling tools which have been analysed extensively. Where closed form expressions are not available there are accurate numerical methods for approximating quantities of interest. They have found application in fields as diverse as ecology, physical chemistry and telecommunications systems modelling. Markov Chains: An Introduction/Review MASCOS Workshop on Markov Chains, April 2005 p. 43
LECTURE 4. Last time: Lecture outline
LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random
IEOR 6711: Stochastic Models, I Fall 2012, Professor Whitt, Final Exam SOLUTIONS
IEOR 6711: Stochastic Models, I Fall 2012, Professor Whitt, Final Exam SOLUTIONS There are four questions, each with several parts. 1. Customers Coming to an Automatic Teller Machine (ATM) (30 points)
M/M/1 and M/M/m Queueing Systems
M/M/ and M/M/m Queueing Systems M. Veeraraghavan; March 20, 2004. Preliminaries. Kendall s notation: G/G/n/k queue G: General - can be any distribution. First letter: Arrival process; M: memoryless - exponential
Fluid models in performance analysis
Fluid models in performance analysis Marco Gribaudo 1, Miklós Telek 2 1 Dip. di Informatica, Università di Torino, marcog@di.unito.it 2 Dept. of Telecom., Technical University of Budapest telek@hitbme.hu
Markov Chains. Chapter 4. 4.1 Stochastic Processes
Chapter 4 Markov Chains 4 Stochastic Processes Often, we need to estimate probabilities using a collection of random variables For example, an actuary may be interested in estimating the probability that
Tutorial: Stochastic Modeling in Biology Applications of Discrete- Time Markov Chains. NIMBioS Knoxville, Tennessee March 16-18, 2011
Tutorial: Stochastic Modeling in Biology Applications of Discrete- Time Markov Chains Linda J. S. Allen Texas Tech University Lubbock, Texas U.S.A. NIMBioS Knoxville, Tennessee March 16-18, 2011 OUTLINE
Memorandum No. 1486. Families of birth-death processes with similar time-dependent behaviour
Faculty of Mathematical Sciences University of Twente University for Technical and Social Sciences P.O. Box 217 7500 AE Enschede The Netherlands Phone: +31-53-4893400 Fax: +31-53-4893114 Email: memo@math.utwente.nl
Random access protocols for channel access. Markov chains and their stability. Laurent Massoulié.
Random access protocols for channel access Markov chains and their stability laurent.massoulie@inria.fr Aloha: the first random access protocol for channel access [Abramson, Hawaii 70] Goal: allow machines
Master s Theory Exam Spring 2006
Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem
1. (First passage/hitting times/gambler s ruin problem:) Suppose that X has a discrete state space and let i be a fixed state. Let
Copyright c 2009 by Karl Sigman 1 Stopping Times 1.1 Stopping Times: Definition Given a stochastic process X = {X n : n 0}, a random time τ is a discrete random variable on the same probability space as
1 Gambler s Ruin Problem
Coyright c 2009 by Karl Sigman 1 Gambler s Ruin Problem Let N 2 be an integer and let 1 i N 1. Consider a gambler who starts with an initial fortune of $i and then on each successive gamble either wins
10.2 Series and Convergence
10.2 Series and Convergence Write sums using sigma notation Find the partial sums of series and determine convergence or divergence of infinite series Find the N th partial sums of geometric series and
Chapter 2 Discrete-Time Markov Models
Chapter Discrete-Time Markov Models.1 Discrete-Time Markov Chains Consider a system that is observed at times 0;1;;:::: Let X n be the state of the system at time n for n D 0;1;;:::: Suppose we are currently
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete
Introduction to Markov Chain Monte Carlo
Introduction to Markov Chain Monte Carlo Monte Carlo: sample from a distribution to estimate the distribution to compute max, mean Markov Chain Monte Carlo: sampling using local information Generic problem
a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.
Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given
Homework set 4 - Solutions
Homework set 4 - Solutions Math 495 Renato Feres Problems R for continuous time Markov chains The sequence of random variables of a Markov chain may represent the states of a random system recorded at
MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.
MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column
Statistics in Retail Finance. Chapter 6: Behavioural models
Statistics in Retail Finance 1 Overview > So far we have focussed mainly on application scorecards. In this chapter we shall look at behavioural models. We shall cover the following topics:- Behavioural
Probability Generating Functions
page 39 Chapter 3 Probability Generating Functions 3 Preamble: Generating Functions Generating functions are widely used in mathematics, and play an important role in probability theory Consider a sequence
On the mathematical theory of splitting and Russian roulette
On the mathematical theory of splitting and Russian roulette techniques St.Petersburg State University, Russia 1. Introduction Splitting is an universal and potentially very powerful technique for increasing
Systems of Linear Equations
Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and
Optimal Design of Sequential Real-Time Communication Systems Aditya Mahajan, Member, IEEE, and Demosthenis Teneketzis, Fellow, IEEE
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 11, NOVEMBER 2009 5317 Optimal Design of Sequential Real-Time Communication Systems Aditya Mahajan, Member, IEEE, Demosthenis Teneketzis, Fellow, IEEE
Robust Staff Level Optimisation in Call Centres
Robust Staff Level Optimisation in Call Centres Sam Clarke Jesus College University of Oxford A thesis submitted for the degree of M.Sc. Mathematical Modelling and Scientific Computing Trinity 2007 Abstract
Markov Chains And Existtime Stabilization
50 15. MARKOV CHAINS: DEFINITIONS AND EXAMPLES Along with the discussion of martingales, we have introduced the concept of a discrete-time stochastic process. In this chapter we will study a particular
Random variables, probability distributions, binomial random variable
Week 4 lecture notes. WEEK 4 page 1 Random variables, probability distributions, binomial random variable Eample 1 : Consider the eperiment of flipping a fair coin three times. The number of tails that
The Exponential Distribution
21 The Exponential Distribution From Discrete-Time to Continuous-Time: In Chapter 6 of the text we will be considering Markov processes in continuous time. In a sense, we already have a very good understanding
Periodic Capacity Management under a Lead Time Performance Constraint
Periodic Capacity Management under a Lead Time Performance Constraint N.C. Buyukkaramikli 1,2 J.W.M. Bertrand 1 H.P.G. van Ooijen 1 1- TU/e IE&IS 2- EURANDOM INTRODUCTION Using Lead time to attract customers
Markov random fields and Gibbs measures
Chapter Markov random fields and Gibbs measures 1. Conditional independence Suppose X i is a random element of (X i, B i ), for i = 1, 2, 3, with all X i defined on the same probability space (.F, P).
Analysis of a Production/Inventory System with Multiple Retailers
Analysis of a Production/Inventory System with Multiple Retailers Ann M. Noblesse 1, Robert N. Boute 1,2, Marc R. Lambrecht 1, Benny Van Houdt 3 1 Research Center for Operations Management, University
4.1 Introduction and underlying setup
Chapter 4 Semi-Markov processes in labor market theory 4.1 Introduction and underlying setup Semi-Markov processes are, like all stochastic processes, models of systems or behavior. As extensions of Markov
Development of dynamically evolving and self-adaptive software. 1. Background
Development of dynamically evolving and self-adaptive software 1. Background LASER 2013 Isola d Elba, September 2013 Carlo Ghezzi Politecnico di Milano Deep-SE Group @ DEIB 1 Requirements Functional requirements
Modeling and Performance Evaluation of Computer Systems Security Operation 1
Modeling and Performance Evaluation of Computer Systems Security Operation 1 D. Guster 2 St.Cloud State University 3 N.K. Krivulin 4 St.Petersburg State University 5 Abstract A model of computer system
Applying Mean-field Approximation to Continuous Time Markov Chains
#08 2013 IMT LUCCA CSA TECHNICAL REPORT SERIES 08 June 2013 RA Computer Science and Applications Applying Mean-field Approximation to Continuous Time Markov Chains Anna Kolesnichenko Alireza Pourranjabar
Dynamic Assignment of Dedicated and Flexible Servers in Tandem Lines
Dynamic Assignment of Dedicated and Flexible Servers in Tandem Lines Sigrún Andradóttir and Hayriye Ayhan School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332-0205,
Essentials of Stochastic Processes
i Essentials of Stochastic Processes Rick Durrett 70 60 50 10 Sep 10 Jun 10 May at expiry 40 30 20 10 0 500 520 540 560 580 600 620 640 660 680 700 Almost Final Version of the 2nd Edition, December, 2011
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 51, NO. 7, JULY 2005 2475. G. George Yin, Fellow, IEEE, and Vikram Krishnamurthy, Fellow, IEEE
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 51, NO. 7, JULY 2005 2475 LMS Algorithms for Tracking Slow Markov Chains With Applications to Hidden Markov Estimation and Adaptive Multiuser Detection G.
Row Echelon Form and Reduced Row Echelon Form
These notes closely follow the presentation of the material given in David C Lay s textbook Linear Algebra and its Applications (3rd edition) These notes are intended primarily for in-class presentation
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a
Markov Chains. Table of Contents. Schedules
Markov Chains These notes contain material prepared by colleagues who have also presented this course at Cambridge, especially James Norris. The material mainly comes from books of Norris, Grimmett & Stirzaker,
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +
Fluid Models for Production-Inventory Systems
Fluid Models for Production-Inventory Systems by Keqi Yan A dissertation submitted to the faculty of the University of North Carolina at Chapel Hill in partial fulfillment of the requirements for the degree
What is Linear Programming?
Chapter 1 What is Linear Programming? An optimization problem usually has three essential ingredients: a variable vector x consisting of a set of unknowns to be determined, an objective function of x to
1 Norms and Vector Spaces
008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)
1/3 1/3 1/3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0 1 2 3 4 5 6 7 8 0.6 0.6 0.6 0.6 0.6 0.6 0.6
HOMEWORK 4: SOLUTIONS. 2. A Markov chain with state space {, 2, 3} has transition probability matrix /3 /3 /3 P = 0 /2 /2 0 0 Show that state 3 is absorbing and, starting from state, find the expected
Gambling and Data Compression
Gambling and Data Compression Gambling. Horse Race Definition The wealth relative S(X) = b(x)o(x) is the factor by which the gambler s wealth grows if horse X wins the race, where b(x) is the fraction
3.2 Roulette and Markov Chains
238 CHAPTER 3. DISCRETE DYNAMICAL SYSTEMS WITH MANY VARIABLES 3.2 Roulette and Markov Chains In this section we will be discussing an application of systems of recursion equations called Markov Chains.
Analysis of an Artificial Hormone System (Extended abstract)
c 2013. This is the author s version of the work. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purpose or for creating
Optimal proportional reinsurance and dividend pay-out for insurance companies with switching reserves
Optimal proportional reinsurance and dividend pay-out for insurance companies with switching reserves Abstract: This paper presents a model for an insurance company that controls its risk and dividend
Average System Performance Evaluation using Markov Chain
Designing high performant Systems: statistical timing analysis and optimization Average System Performance Evaluation using Markov Chain Weiyun Lu Supervisor: Martin Radetzki Sommer Semester 2006 Stuttgart
The Analysis of Dynamical Queueing Systems (Background)
The Analysis of Dynamical Queueing Systems (Background) Technological innovations are creating new types of communication systems. During the 20 th century, we saw the evolution of electronic communication
Numerical Solution of Differential
Chapter 13 Numerical Solution of Differential Equations We have considered numerical solution procedures for two kinds of equations: In chapter 10 the unknown was a real number; in chapter 6 the unknown
TD(0) Leads to Better Policies than Approximate Value Iteration
TD(0) Leads to Better Policies than Approximate Value Iteration Benjamin Van Roy Management Science and Engineering and Electrical Engineering Stanford University Stanford, CA 94305 bvr@stanford.edu Abstract
9.2 Summation Notation
9. Summation Notation 66 9. Summation Notation In the previous section, we introduced sequences and now we shall present notation and theorems concerning the sum of terms of a sequence. We begin with a
Application of Markov chain analysis to trend prediction of stock indices Milan Svoboda 1, Ladislav Lukáš 2
Proceedings of 3th International Conference Mathematical Methods in Economics 1 Introduction Application of Markov chain analysis to trend prediction of stock indices Milan Svoboda 1, Ladislav Lukáš 2
A Single-Unit Decomposition Approach to Multi-Echelon Inventory Systems
A Single-Unit Decomposition Approach to Multi-Echelon Inventory Systems Alp Muharremoğlu John N. sitsiklis July 200 Revised March 2003 Former version titled: Echelon Base Stock Policies in Uncapacitated
Lectures on Stochastic Processes. William G. Faris
Lectures on Stochastic Processes William G. Faris November 8, 2001 2 Contents 1 Random walk 7 1.1 Symmetric simple random walk................... 7 1.2 Simple random walk......................... 9 1.3
Optimal Charging Strategies for Electrical Vehicles under Real Time Pricing
1 Optimal Charging Strategies for Electrical Vehicles under Real Time Pricing Mohammad M. Karbasioun, Ioannis Lambadaris, Gennady Shaikhet, Evangelos Kranakis Carleton University, Ottawa, ON, Canada E-mails:
1 Error in Euler s Method
1 Error in Euler s Method Experience with Euler s 1 method raises some interesting questions about numerical approximations for the solutions of differential equations. 1. What determines the amount of
Performance Analysis of a Telephone System with both Patient and Impatient Customers
Performance Analysis of a Telephone System with both Patient and Impatient Customers Yiqiang Quennel Zhao Department of Mathematics and Statistics University of Winnipeg Winnipeg, Manitoba Canada R3B 2E9
Exact FCFS matching rates for two infinite multi-type sequences
Exact FCFS matching rates for two infinite multi-type sequences Ivo Adan Gideon Weiss September 2, 20 Abstract Motivated by queues with multi-type servers and multi-type customers, we consider an infinite
Math 4310 Handout - Quotient Vector Spaces
Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable
Safety margins for unsystematic biometric risk in life and health insurance
Safety margins for unsystematic biometric risk in life and health insurance Marcus C. Christiansen June 1, 2012 7th Conference in Actuarial Science & Finance on Samos Seite 2 Safety margins for unsystematic
Solved Problems. Chapter 14. 14.1 Probability review
Chapter 4 Solved Problems 4. Probability review Problem 4.. Let X and Y be two N 0 -valued random variables such that X = Y + Z, where Z is a Bernoulli random variable with parameter p (0, ), independent
Probability and Random Variables. Generation of random variables (r.v.)
Probability and Random Variables Method for generating random variables with a specified probability distribution function. Gaussian And Markov Processes Characterization of Stationary Random Process Linearly
NOTES ON LINEAR TRANSFORMATIONS
NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all
Some Problems of Second-Order Rational Difference Equations with Quadratic Terms
International Journal of Difference Equations ISSN 0973-6069, Volume 9, Number 1, pp. 11 21 (2014) http://campus.mst.edu/ijde Some Problems of Second-Order Rational Difference Equations with Quadratic
Notes on Probability Theory
Notes on Probability Theory Christopher King Department of Mathematics Northeastern University July 31, 2009 Abstract These notes are intended to give a solid introduction to Probability Theory with a
The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].
![The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].](/thumbs/40/21161256.jpg "The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].")
Probability Theory Probability Spaces and Events Consider a random experiment with several possible outcomes. For example, we might roll a pair of dice, flip a coin three times, or choose a random real
Big Data Technology Motivating NoSQL Databases: Computing Page Importance Metrics at Crawl Time
Big Data Technology Motivating NoSQL Databases: Computing Page Importance Metrics at Crawl Time Edward Bortnikov & Ronny Lempel Yahoo! Labs, Haifa Class Outline Link-based page importance measures Why
LS.6 Solution Matrices
LS.6 Solution Matrices In the literature, solutions to linear systems often are expressed using square matrices rather than vectors. You need to get used to the terminology. As before, we state the definitions
A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form
Section 1.3 Matrix Products A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form (scalar #1)(quantity #1) + (scalar #2)(quantity #2) +...
To discuss this topic fully, let us define some terms used in this and the following sets of supplemental notes.
INFINITE SERIES SERIES AND PARTIAL SUMS What if we wanted to sum up the terms of this sequence, how many terms would I have to use? 1, 2, 3,... 10,...? Well, we could start creating sums of a finite number
n k=1 k=0 1/k! = e. Example 6.4. The series 1/k 2 converges in R. Indeed, if s n = n then k=1 1/k, then s 2n s n = 1 n + 1 +...
6 Series We call a normed space (X, ) a Banach space provided that every Cauchy sequence (x n ) in X converges. For example, R with the norm = is an example of Banach space. Now let (x n ) be a sequence
A Profit-Maximizing Production Lot sizing Decision Model with Stochastic Demand
A Profit-Maximizing Production Lot sizing Decision Model with Stochastic Demand Kizito Paul Mubiru Department of Mechanical and Production Engineering Kyambogo University, Uganda Abstract - Demand uncertainty
Hydrodynamic Limits of Randomized Load Balancing Networks
Hydrodynamic Limits of Randomized Load Balancing Networks Kavita Ramanan and Mohammadreza Aghajani Brown University Stochastic Networks and Stochastic Geometry a conference in honour of François Baccelli
4.5 Linear Dependence and Linear Independence
4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then
STABILITY OF LU-KUMAR NETWORKS UNDER LONGEST-QUEUE AND LONGEST-DOMINATING-QUEUE SCHEDULING
Applied Probability Trust (28 December 2012) STABILITY OF LU-KUMAR NETWORKS UNDER LONGEST-QUEUE AND LONGEST-DOMINATING-QUEUE SCHEDULING RAMTIN PEDARSANI and JEAN WALRAND, University of California, Berkeley
The Heat Equation. Lectures INF2320 p. 1/88
The Heat Equation Lectures INF232 p. 1/88 Lectures INF232 p. 2/88 The Heat Equation We study the heat equation: u t = u xx for x (,1), t >, (1) u(,t) = u(1,t) = for t >, (2) u(x,) = f(x) for x (,1), (3)
Random variables P(X = 3) = P(X = 3) = 1 8, P(X = 1) = P(X = 1) = 3 8.
Random variables Remark on Notations 1. When X is a number chosen uniformly from a data set, What I call P(X = k) is called Freq[k, X] in the courseware. 2. When X is a random variable, what I call F ()
PYKC Jan-7-10. Lecture 1 Slide 1
Aims and Objectives E 2.5 Signals & Linear Systems Peter Cheung Department of Electrical & Electronic Engineering Imperial College London! By the end of the course, you would have understood: Basic signal
1. Prove that the empty set is a subset of every set.
1. Prove that the empty set is a subset of every set. Basic Topology Written by Men-Gen Tsai email: b89902089@ntu.edu.tw Proof: For any element x of the empty set, x is also an element of every set since
Metric Spaces. Chapter 7. 7.1. Metrics
Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some
Applications to Data Smoothing and Image Processing I
Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is
ALGORITHMIC TRADING WITH MARKOV CHAINS
June 16, 2010 ALGORITHMIC TRADING WITH MARKOV CHAINS HENRIK HULT AND JONAS KIESSLING Abstract. An order book consists of a list of all buy and sell offers, represented by price and quantity, available
THE DYING FIBONACCI TREE. 1. Introduction. Consider a tree with two types of nodes, say A and B, and the following properties:
THE DYING FIBONACCI TREE BERNHARD GITTENBERGER 1. Introduction Consider a tree with two types of nodes, say A and B, and the following properties: 1. Let the root be of type A.. Each node of type A produces
Supplement to Call Centers with Delay Information: Models and Insights
Supplement to Call Centers with Delay Information: Models and Insights Oualid Jouini 1 Zeynep Akşin 2 Yves Dallery 1 1 Laboratoire Genie Industriel, Ecole Centrale Paris, Grande Voie des Vignes, 92290
1 if 1 x 0 1 if 0 x 1
Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or
Wald s Identity. by Jeffery Hein. Dartmouth College, Math 100
Wald s Identity by Jeffery Hein Dartmouth College, Math 100 1. Introduction Given random variables X 1, X 2, X 3,... with common finite mean and a stopping rule τ which may depend upon the given sequence,
6.1. The Exponential Function. Introduction. Prerequisites. Learning Outcomes. Learning Style
The Exponential Function 6.1 Introduction In this block we revisit the use of exponents. We consider how the expression a x is defined when a is a positive number and x is irrational. Previously we have
CHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e.
CHAPTER II THE LIMIT OF A SEQUENCE OF NUMBERS DEFINITION OF THE NUMBER e. This chapter contains the beginnings of the most important, and probably the most subtle, notion in mathematical analysis, i.e.,
Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur
Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special Distributions-VI Today, I am going to introduce
2x + y = 3. Since the second equation is precisely the same as the first equation, it is enough to find x and y satisfying the system
1. Systems of linear equations We are interested in the solutions to systems of linear equations. A linear equation is of the form 3x 5y + 2z + w = 3. The key thing is that we don t multiply the variables
HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!
Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following
A Markovian Model for Investment Analysis in Advertising
Proceedings of the Seventh Young Statisticians Meeting Andrej Mrvar (Editor) Metodološki zvezki, 21, Ljubljana: FDV, 2003 A Markovian Model for Investment Analysis in Advertising Eugenio Novelli 1 Abstract
Second Order Linear Nonhomogeneous Differential Equations; Method of Undetermined Coefficients. y + p(t) y + q(t) y = g(t), g(t) 0.
Second Order Linear Nonhomogeneous Differential Equations; Method of Undetermined Coefficients We will now turn our attention to nonhomogeneous second order linear equations, equations with the standard
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding
Matrix Differentiation
1 Introduction Matrix Differentiation ( and some other stuff ) Randal J. Barnes Department of Civil Engineering, University of Minnesota Minneapolis, Minnesota, USA Throughout this presentation I have
Nonparametric adaptive age replacement with a one-cycle criterion
Nonparametric adaptive age replacement with a one-cycle criterion P. Coolen-Schrijner, F.P.A. Coolen Department of Mathematical Sciences University of Durham, Durham, DH1 3LE, UK e-mail: Pauline.Schrijner@durham.ac.uk
Special Situations in the Simplex Algorithm
Special Situations in the Simplex Algorithm Degeneracy Consider the linear program: Maximize 2x 1 +x 2 Subject to: 4x 1 +3x 2 12 (1) 4x 1 +x 2 8 (2) 4x 1 +2x 2 8 (3) x 1, x 2 0. We will first apply the