# 1 Limiting distribution for a Markov chain

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Copyright c 2009 by Karl Sigman Limiting distribution for a Markov chain In these Lecture Notes, we shall study the limiting behavior of Markov chains as time n In particular, under suitable easy-to-check conditions, we will see that a Markov chain possesses a limiting probability distribution, π = (π j ) j S, and that the chain, if started off initially with such a distribution will be a stationary stochastic process We will also see that we can find π by merely solving a set of linear equations Communication classes and irreducibility for Markov chains For a Markov chain with state space S, consider a pair of states (i, j) We say that j is reachable from i, denoted by i j, if there exists an integer n 0 such that Pij n > 0 This means that starting in state i, there is a positive probability (but not necessarily equal to ) that the chain will be in state j at time n (that is, n steps later); P (X n = j X 0 = i) > 0 If j is reachable from i, and i is reachable from j, then the states i and j are said to communicate, denoted by i j The relation defined by communication satisfies the following conditions: All states communicate with themselves: P 0 ii = > 0 2 Symmetry: If i j, then j i 3 Transitivity: If i k and k j, then i j The above conditions imply that communication is an example of an equivalence relation, meaning that it shares the properties with the more familiar equality relation = : i = i If i = j, then j = i If i = k and k = j, then i = j Only condition 3 above needs some justification, so we now prove it for completeness: Suppose there exists integers n, m such that Pik n > 0 and P kj m > 0 Letting l = n + m, we conclude that Pij l P ik np kj m > 0 where we have formally used the Chapman-Kolmogorov equations The point is that the chain can (with positive probability) go from i to j by first going from i to k (n steps) and then (independent of the past) going from k to j (an additional m steps) If we consider the rat in the open maze, we easily see that the set of states C = {, 2, 3, 4} all communicate with one another, but state 0 only communicates with itself (since it is an absorbing state) Whereas state 0 is reachable from the other states, i 0, no other state can be reached from state 0 We conclude that the state space S = {0,, 2, 3, 4} can be broken up into two disjoint subsets, C = {, 2, 3, 4} and C 2 = {0} whose union equals S, and such that each of these subsets has the property that all states within it communicate Disjoint means that their intersection contains no elements: C C 2 = A little thought reveals that this kind of disjoint breaking can be done with any Markov chain: Proposition For each Markov chain, there exists a unique decomposition of the state space S into a sequence of disjoint subsets C, C 2,, S = i=c i, in which each subset has the property that all states within it communicate Each such subset is called a communication class of the Markov chain P 0 ii = P (X 0 = i X 0 = i) =, a trivial fact

2 If we now consider the rat in the closed maze, S = {, 2, 3, 4}, then we see that there is only one communication class C = {, 2, 3, 4} = S: all states communicate This is an example of what is called an irreducible Markov chain Definition A Markov chain for which there is only one communication class is called an irreducible Markov chain: all states communicate Examples Simple random walk is irreducible Here, S = {, 0,, } But since 0 < p <, we can always reach any state from any other state, doing so step-by-step, using the fact that P i,i+ = p, P i,i = p For example 4 2 since P 4,2 6 p6 > 0, and 2 4 since P2, 4 6 ( p)6 > 0; thus 4 2 In general Pi,j n > 0 for n = i j 2 Random walk from the gambler s ruin problem is not irreducible Here, the random walk is restricted to the finite state space {0,,, N} and P 00 = P NN = C = {0}, C 2 = {, N }, C 3 = {N} are the communication classes 3 Consider a Markov chain with S = {0,, 2, 3} and transition matrix given by P = /2 /2 0 0 /2 /2 0 0 /3 /6 /6 / Notice how states 0, keep to themselves in that whereas they communicate with each other, no other state is reachable from them (together they form an absorbing set) Thus C = {0, } Whereas every state is reachable from state 2, getting to state 2 is not possible from any other state; thus C 2 = {2} Finally, state 3 is absorbing and so C 3 = {3} This example illustrates the general method of deducing communication classes by analyzing the the transition matrix 2 Recurrence and Stationary distributions 2 Recurrence and transience Let τ ii denote the return time to state i given X 0 = i: τ ii = min{n : X n = i X 0 = i}, τ ii def =, if X n i, n It represents the amount of time (number of steps) until the chain returns to state i given that it started in state i Note how never returning is allowed in the definition by defining τ ii =, so a return occurs if and only if τ ii < def f i = P (τ ii < ) is thus the probability of ever returning to state i given that the chain started in state i A state i is called recurrent if f i = ; transient if f i < By the (strong) Markov property, once the chain revisits state i, the future is independent of the past, and it is as if the chain is starting all over again in state i for the first time: Each time state i is visited, it will be revisited with the same probability f i independent of the past In particular, if f i =, then the chain will return to state i over and over again, an infinite number of times That is why the word recurrent is used If state i is transient (f i < ), then it will only be visited a 2

3 finite (random) number of times (after which only the remaining states j i can be visited by the chain) Counting over all time, the total number of visits to state i, given that X 0 = i, is given by an infinite sequence of indicator rvs and has a geometric distribution, N i = I{X n = i X 0 = i} () n=0 P (N i = n) = f n i ( f i ), n (We count the initial visit X 0 = i as the first visit) The expected number of visits is thus given by E(N i ) = /( f i ) and we conclude that A state i is recurrent (f i = ) if and only if E(N i ) =, or equivalently A state i is transient (f i < ) if and only if E(N i ) < Taking expected value in () yields E(N i ) = n=0 P n i,i, because E(I{X n = i X 0 = i}) = P (X n = i X 0 = i) = P n i,i We thus obtain Proposition 2 A state i is recurrent if and only if transient otherwise n=0 P n i,i =, Proposition 22 For any communication class C, all states in C are either recurrent or all states in C are transient Thus: if i and j communicate and i is recurrent, then so is j Equivalenly if i and j communicate and i is transient, then so is j In particular, for an irreducible Markov chain, either all states are recurrent or all states are transient Proof : Suppose two states i j communicate; choose an appropriate n so that p = P n i,j > 0 Now if i is recurrent, then so must be j because every time i is visited there is this same positive probability p ( success probability) that j will be visited n steps later But i being recurrent means it will be visited over and over again, an infinite number of times, so viewing this as sequence of Bernoulli trials, we conclude that eventually there will be a success (Formally, we are using the Borel-Cantelli theorem) Definition 2 For an irreducible Markov chain, if all states are recurrent, then we say that the Markov chain is recurrent; transient otherwise 3

4 The rat in the closed maze yields a recurrent Markov chain The rat in the open maze yields a Markov chain that is not irreducible; there are two communication classes C = {, 2, 3, 4}, C 2 = {0} C is transient, whereas C 2 is recurrent Clearly if the state space is finite for a given Markov chain, then not all the states can be transient (for otherwise after a finite number a steps (time) the chain would leave every state never to return; where would it go?) Thus we conclude that Proposition 23 An irreducible Markov chain with a finite state space is always recurrent: all states are recurrent Finally observe (from the argument that if two states communicate and one is recurrent then so is the other) that for an irreducible recurrent chain, even if we start in some other state X 0 i, the chain will still visit state i an infinite number of times: For an irreducible recurrent Markov chain, each state j will be visited over and over again (an infinite number of times) regardless of the initial state X 0 = i For example, if the rat in the closed maze starts off in cell 3, it will still return over and over again to cell 22 Expected return time to a given state: positive recurrence and null recurrence A recurrent state j is called positive recurrent if the expected amount of time to return to state j given that the chain started in state j has finite first moment: E(τ jj ) < A recurrent state j for which E(τ jj ) = is called null recurrent def In general even for i j, we define τ ij = min{n : X n = j X 0 = i}, the time (after time 0) until reaching state j given X 0 = i Proposition 24 Suppose i j are both recurrent If i and j communicate and if j is positive recurrent (E(τ jj ) < ), then i is positive recurrent (E(τ ii ) < ) and also E(τ ij ) < In particular, all states in a recurrent communication class are either all together positive recurrent or all together null recurrent Proof : Assume that E(τ jj ) < and that i and j communicate Choose the smallest n such that P n j,i > 0 With X 0 = j, let A = {X k j, k n, X n = i}; P (A) > 0 Then E(τ j,j ) E(τ j,j A)P (A) = (n + E(τ i,j ))P (A); hence E(τ i,j ) < (for otherwise E(τ j,j ) =, a contradiction) With X 0 = j, let {Y m : m } be iid distributed as τ j,j denote the interarrival times between visits to state j Thus the n th revisit of the chain to state j is at time t n = Y + + Y n, and E(Y ) = E(τ j,j ) < Let p = P ({X n } visits state i before returning to state j X 0 = j) p P (A) > 0, where A is defined above Every time the chain revisits state j, there is, independent of the past, this probability p that the chain will visit state i before revisiting state j again Letting N denote the number of revisits the chain makes to state j until first visiting 4

5 state i, we thus see that N has a geometric distribution with success probability p, and so E(N) < N is a stopping time with respect to the {Y m }, and τ j,i N Y m, and so by Wald s equation E(τ j,i ) E(N)E(Y ) < Finally, E(τ i,i ) E(τ i,j ) + E(τ j,i ) < Proposition 22 together with Proposition 24 immediately yield Proposition 25 All states in a communication class C are all together either positive recurrent, null recurrent or transient In particular, for an irreducible Markov chain, all states together must be positive recurrent, null recurrent or transient Definition 22 If all states in an irreducible Markov chain are positive recurrent, then we say that the Markov chain is positive recurrent If all states in an irreducible Markov chain are null recurrent, then we say that the Markov chain is null recurrent If all states in an irreducible Markov chain are transient, then we say that the Markov chain is transient 23 Limiting stationary distribution When the limits exist, let π j denote the long run proportion of time that the chain spends in state j: π j = lim I{X m = j} wp (2) Taking into account the initial condition X 0 = i, this is more precisely stated as: π j = lim I{X m = j X 0 = i} wp, for all initial states i (3) Taking expected values (E(I{X m = j X 0 = i}) = P (X m = j X 0 = i) = Pij m ) we see that if π j exists then it can be computed alternatively by (via the bounded convergence theorem) π j = lim = lim P (X m = j X 0 = i), P m ij, for all initial states i (4) For simplicity of notation, we assume for now that the state space S = N = {0,, 2, } or some finite subset of N Definition 23 If for each j S, π j exists as defined in (3) and is independent of the initial state i, and j S π j =, then the probability distribution π = (π 0, π, ) on the state space S is called the limiting or stationary or steady-state distribution of the Markov chain 5

6 Recalling that Pij m is precisely the (ij)th component of the matrix P m, we conclude that (4) can be expressed in matrix form by lim π π 0, π, P m = π = π 0, π, (5) That is, when we average the m-step transition matrices, each row converges to the vector of stationary probabilities π = (π 0, π, ) The i th row refers to the intial condition X 0 = i in (4), and for each such fixed row i, the j th element of the averages converges to π j A nice way of interpreting π: If you observe the state of the Markov chain at some random time way out in the future, then π j is the probability that the state is j To see this: Let N (our random observation time) have a uniform distribution over the integers {, 2, n}, and be independent of the chain; P (N = m) = /n, m {, 2, n} Now assume that X 0 = i and that n is very large Then by conditioning on N = m we obtain P (X N = j) = = n where we used (4) for the last line P (X m = j X 0 = i)p (N = m) π j, P m i,j 24 Connection between E(τ jj ) and π j The following is intuitive and very useful: Proposition 26 If {X n } is a positive recurrent Markov chain, then a unique stationary distribution π exists and is given by π j = > 0, for all states j S E(τ jj ) If the chain is null recurrent or transient then the limits in (2) are all 0 wp; no stationary distribution exists The intuition: On average, the chain visits state j once every E(τ jj ) amount of time; thus π j = E(τ jj ) Proof : First, we immediately obtain the transient case result since by definition, each fixed state i is then only visited a finite number of times; hence the limit in (2) must be 0 wp Thus we need only consider now the two recurrent cases (We will use the fact that for any fixed state j, returns to state j constitute recurrent regeneration points for the chain; thus this result is a consequence of standard theory about regenerative processes; but even if the reader has not yet learned such theory, the proof will be understandable) First assume that X 0 = j Let t 0 = 0, t = τ jj, t 2 = min{k > t : X k = j} and in general t n+ = min{k > t n : X k = j}, n These t n are the consecutive times at which the chain 6

7 visits state j If we let Y n = t n t n (the interevent times) then we revisit state j for the n th time at time t n = Y + + Y n The idea here is to break up the evolution of the Markov chain into iid cycles where a cycle begins every time the chain visits state j Y n is the n th cycle-length By the Markov property, the chain starts over again and is independent of the past everytime it enters state j (formally this follows by the Strong Markov Property) This means that the cycle lengths {Y n : n } form an iid sequence with common distribution the same as the first cycle length τ jj In particular, E(Y n ) = E(τ jj ) for all n Now observe that the number of revisits to state j is precisely n visits at time t n = Y + + Y n, and thus the long-run proportion of visits to state j per unit time can be computed as π j = lim m m = lim n = m I{X k = j} k= n n i= Y i E(τ jj ), wp, where the last equality follows from the Strong Law of Large Numbers (SLLN) Thus in the positive recurrent case, π j > 0 for all j S, where as in the null recurrent case, π j = 0 for all j S Finally, if X 0 = i j, then we can first wait until the chain enters state j (which it will eventually, by recurrence), and then proceed with the above proof (Uniqueness follows by the unique representation π j = E(τ jj ) ) The above result is useful for computing E(τ jj ) if π has already been found: For example, consider the rat in the closed off maze problem from HMWK 2 Given that the rat starts off in cell, what is the expected number of steps until the rat returns to cell? The answer is simply /π But how do we compute π? We consider that problem next 25 Computing π algebraically Theorem 2 Suppose {X n } is an irreducible Markov chain with transition matrix P Then {X n } is positive recurrent if and only if there exists a (non-negative, summing to ) solution, π = (π 0, π, ), to the set of linear equations π = πp, in which case π is precisely the unique stationary distribution for the Markov chain For example consider state space S = {0, } and the matrix ( ) P =, which is clearly irreducible For π = (π 0, π ), π = πp yields the two equations π 0 = 05π π, π = 05π π We can also utilize the probability condition that π 0 + π = Solving yields π 0 = 4/9, π = 5/9 We conclude that this chain is positive recurrent with stationary distribution (4/9, 5/9) The long run proportion of time the chain visits state 0 is equal to 4/9 and the long run proportion of time the chain visits state is equal to 5/9 Furthermore, since π j = /E(τ jj ), we conclude that the expected number of steps (time) until the chain revisits state given that it is in state now is equal to 9/5 7

8 How to use Theorem 2 Theorem 2 can be used in practice as follows: if you have an irreducible MC, then you can try to solve the set of equations: π = πp, j S π j = If you do solve them for some π, then this solution π is unique and is the stationary distribution, and the chain is positive recurrent It might not even be necessary to solve the equations to obtain π: suppose you have a candidate π for the stationary distribution (perhaps by guessing), then you need only plug in the guess to verify that it satisfies π = πp If it does, then your guess π is the stationary distribution, and the chain is positive recurrent 2 Proof of Theorem 2 Proof : Assume the chain is irreducible and positive recurrent Then we know from Proposition 26 that π exists (as defined in Equations (3), (4)), has representation π j = /E(τ jj ) > 0, j S, and is unique On the one hand, if we multiply (on the right) each side of Equation (5) by P, then we obtain But on the other hand, lim lim π P m+ = π P P m+ = lim π = π π = π, + lim n P m + lim (P n+ P ) n (P n+ P ) (from (5)) because for any k, lim P k = 0 (since p k ij for all i, j) Thus, we obtain π π π = π P, yielding (from each row) π = πp Summarizing: If a Markov chain is positive recurrent, then the stationary distribution π exists as defined in Equations (3), (4), is given by π j = /E(τ jj ), j S, and must satisfy π = πp Now we prove the converse : For an irreducible Markov chain, suppose that π = πp has a non-negative, summing to solution π; that is, a probability solution We must show that the chain is thus positive recurrent, and that this solution π is the stationary distribution as defined by (4) We first will prove that the chain cannot be either transient or null recurrent, 2 0 = (0, 0,, 0) is always a solution to π = πp, but is not a probability Moreover, for any solution π, and any constant c, cπ = cπp by linearity, so cπ is also a solution 8

9 hence must be positive recurrent from Proposition 25 To this end assume the chain is either transient or null recurrent From Proposition 26, we know that then the limits in (4) are identically 0, that is, as n, P m 0 (6) n But if π = πp, then (by multiplying both right sides by P ) π = πp 2 and more generally π = πp m, m, and so [ π n P m] = n πp m = π, n, yielding from (6) that π0 = π, or π = (0,, 0), contradicting that π is a probability distribution Having ruled out the transient and null recurrent cases, we conclude from Proposition 25 that the chain must be positive recurrent For notation, suppose π = π P denotes the nonnegative, summing to solution, and that π denotes the stationary distribution in (4), given by π j = /E(τ jj ), j S We now show that π = π To this end, multiplying both sides of (5) (on the left) by π, we conclude that π π 0, π, π = π π = π π 0, π, Since i S π i =, we see that the above yields π j = π j, j S; π = π as was to be shown 26 Finite state space case When the state space of a Markov chain is finite, then the theory is even simpler: Theorem 22 Every irreducible Markov chain with a finite state space is positive recurrent and thus has a stationary distribution (unique probability solution to π = πp ) Proof : From Prop 23, we know that the chain is recurrent We will now show that it can t be null recurrent, hence must be positive recurrent by Proposition 25 To this end, note that for any fixed m, the rows of P m = P (m) (m-step transition matrix) must sum to, that is, j S P m i,j =, i S (7) Moreover, if null recurrent, we know from Proposition 26, that for all i S, Summing up (8) over j then yields lim j S lim n n P m i,j = 0, j S (8) P m i,j = 0, i S 9

10 But since the state space is finite, we can interchange the outer finite sum with the limit, lim j S P m i,j = 0, i S But then we can interchange the order of summation and use (7), to obtain a contradiction: 0 = lim j S Thus the chain must be positive recurrent Pi,j m = lim Pi,j m =, i S j S This is a very useful result For example, it tells us that the rat in the maze Markov chain, when closed off from the outside, is positive recurrent, and we need only solve the equations π = πp to compute the stationary distribution 27 Stationarity of positive recurrent chains The use of the word stationary here means does not change with time, and we now proceed to show why that describes π For a given probability distribution ν = (ν j ) on the state space S, we use the notation X ν to mean that X is a random variable with distribution ν: P (X = j) = ν j, j S Given any such distribution ν = (ν j ), and Markov chain with transition matrix P, note that if X 0 ν then the vector νp is the distribution of X, that is, the j th coordinate of the vector νp is i S P i,jν i, which is precisely P (X = j): For each j S, P (X = j) = i S P (X = j X 0 = i)p (X 0 = i) = i S P i,j ν i In other words if the initial distribution of the chain is ν, then the distribution one time unit later is given by νp and so on: Lemma 2 If X 0 ν, then X νp, X 2 νp 2,, X n νp n, n Proposition 27 For a positive recurrent Markov chain with stationary distribution π, if X 0 π, then X n π for all n 0 In words: By starting off the chain initially with its stationary distribution, the chain remains having that same distribution for ever after X n has the same distribution for each n This is what is meant by stationary, and why π is called the stationary distribution for the chain Proof : From Theorem 2, we know that π satisfies π = πp, and hence by multiplying both sides (on the right) by P yields π = πp 2 and so on yielding π = πp n, n Thus from Lemma 2, we conclude that if X 0 π, then X n π, n 0

11 Definition 24 A stochastic process {X n } is called a stationary process if for every k 0, the process {X k+n : n 0} has the same distribution, namely the same distribution as {X n } same distribution means in particular that all the finite dimensional distributions are the same: for any integer l and any integers 0 n < n 2 < < n l, the joint distribution of the l vector (X k+n, X k+n2,, X k+nl ) has the same distribution for each k, namely the same distribution as (X n, X n2,, X nl ) It is important to realize (via setting l = in the above definition) that for a stationary process, it of course holds that X n has the same distribution for each fixed n, namely that of X 0 ; but this is only the marginal distribution of the process Stationarity means much more and in general is a much stronger condition It means that if we change the time origin 0 to be any time k in the future, then the entire process still has the same distribution (eg, same finite dimensional distributions) as it did with 0 as the origin For Markov chains, however, stationarity is the same as for the marginals only: Proposition 28 For a positive recurrent Markov chain with stationary distribution π, if X 0 π, then the chain is a stationary process Proof : This result follows immediately from Proposition 27 because of the Markov property: Given X k, the future is independent of the past; its entire distribution depends only on the distribution of X k and the transition matrix P Thus if X k has the same distribution for each k, then it s future {X k+n : n 0} has the same distribution for each k The above result is quite intuitive: We obtain π by choosing to observe the chain way out in the infinite future After doing that, by moving k units of time further into the future does not change anything (we are already out in the infinite future, going any further is still infinite) 28 Convergence to π in the stronger sense π j = lim n P (X n = j); aperiodicity For a positive recurrent chain, the sense in which it was shown to converge to its stationary distribution π was in a time average or Cesàro sense (recall Equations (3), (4)) If one wishes to have the convergence without averaging, π j = lim n P (X n = j), j S (regardless of initial conditions X 0 = i), then a further condition known as aperiodicity is needed on the chain; we will explore this in this section Since P (X n = j X 0 = i) = Pi,j n, we need to explore when it holds for each j S that Pi,j n π j for all i S For simplicity of notation in what follows we assume that S = {0,, 2, } or some subset It is easily seen that for any probability distribution ν, the matrix π π 0, π, M = π = π 0, π,, satisfies νm = π Thus from Lemma 2, we see that in order to obtain π j = lim n P (X n = j), j S, regardless of initial conditions, we equivalently need to have P n M as n We already know that the averages converge, n n P n M, and it is easily seen that in general the stronger convergence P n M will not hold For example take S = {0, } with P = ( 0 0 ),

12 yielding an alternating sequence: If X 0 = 0, then {X n : n 0} = {0,, 0,, }; if X 0 =, then {X n : n 0} = {, 0,, 0, } Clearly, if X 0 = 0, then P (X n = 0) =, if n is even, but P (X n = 0) = 0, if n is odd; P (X n = 0) does not converge as n In terms of P, this is seen by noting that for any n, whereas P 2n = P 2n+ = ( 0 0 ( 0 0 So, although π = (/2, /2) and indeed n n P n M = ), hold that P n M The extra condition needed is explained next For a state j S, consider the set Q = {n : Pj,j n ) ( /2 /2 /2 /2 ), it does not > 0} If there exists an integer d 2 such that Q = {kd : k }, then state j is said to be periodic of period d This implies that Pj,j n = 0 whenever n is not a multiple of d If no such d exists, then the state j is said to be aperiodic It can be shown that periodicity is a class property: all states in a communication class are either aperiodic, or all are periodic with the same period d Thus an irreducible Markov chain is either periodic or aperiodic In the above two-state counterexample, d = 2; the chain is periodic with period d = 2 Clearly, if a chain is periodic, then by choosing any subsequence of integers n m, as m, such that n m / Q, we obtain for any state j that P (X nm = j X 0 = j) = P nm j,j = 0 and so convergence to π j > 0 is not possible along this subsequence; hence convergence of P (X n = j) to π j does not hold The converse is also true, the following is stated without proof: Proposition 29 A positive recurrent Markov chain converges to π via π j = lim n P (X n = j), j S, if and only if the chain is aperiodic Remark For historical reasons, in the literature a positive recurrent and aperiodic Markov chain is sometimes called an ergodic chain The word ergodic, however, has a precise meaning in mathematics (ergodic theory) and this meaning has nothing to do with aperiodicity! In fact any positive recurrent Markov chain is ergodic in this precise mathematical sense (So the historical use of ergodic in the context of aperiodic Markov chains is misleading and unfortunate) Remark 2 The type of convergence π j = lim n P (X n = j), j S, is formally called weak convergence, and we then say that X n converges weakly to π 2

### 1. (First passage/hitting times/gambler s ruin problem:) Suppose that X has a discrete state space and let i be a fixed state. Let

Copyright c 2009 by Karl Sigman 1 Stopping Times 1.1 Stopping Times: Definition Given a stochastic process X = {X n : n 0}, a random time τ is a discrete random variable on the same probability space as

### MATH 56A SPRING 2008 STOCHASTIC PROCESSES 31

MATH 56A SPRING 2008 STOCHASTIC PROCESSES 3.3. Invariant probability distribution. Definition.4. A probability distribution is a function π : S [0, ] from the set of states S to the closed unit interval

### Markov Chains. Chapter Introduction and Definitions 110SOR201(2002)

page 5 SOR() Chapter Markov Chains. Introduction and Definitions Consider a sequence of consecutive times ( or trials or stages): n =,,,... Suppose that at each time a probabilistic experiment is performed,

### 15 Markov Chains: Limiting Probabilities

MARKOV CHAINS: LIMITING PROBABILITIES 67 Markov Chains: Limiting Probabilities Example Assume that the transition matrix is given by 7 2 P = 6 Recall that the n-step transition probabilities are given

### Introduction to Stationary Distributions

13 Introduction to Stationary Distributions We first briefly review the classification of states in a Markov chain with a quick example and then begin the discussion of the important notion of stationary

### Markov Chains, Stochastic Processes, and Advanced Matrix Decomposition

Markov Chains, Stochastic Processes, and Advanced Matrix Decomposition Jack Gilbert Copyright (c) 2014 Jack Gilbert. Permission is granted to copy, distribute and/or modify this document under the terms

### CHAPTER III - MARKOV CHAINS

CHAPTER III - MARKOV CHAINS JOSEPH G. CONLON 1. General Theory of Markov Chains We have already discussed the standard random walk on the integers Z. A Markov Chain can be viewed as a generalization of

### Markov Chains. Gonzalo Mateos

Markov Chains Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ October 5, 2016 Introduction

### Continuous-time Markov Chains

Continuous-time Markov Chains Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ October 31, 2016

### An introduction to Markov chains

An introduction to Markov chains Jie Xiong Department of Mathematics The University of Tennessee, Knoxville [NIMBioS, March 16, 2011] Mathematical biology (WIKIPEDIA) Markov chains also have many applications

### 0.1 Markov Chains Generalities 0.1. MARKOV CHAINS 1

0.1. MARKOV CHAINS 1 0.1 Markov Chains 0.1.1 Generalities A Markov Chain consists of a countable (possibly finite) set S (called the state space) together with a countable family of random variables X,

### Markov Chains, part I

Markov Chains, part I December 8, 2010 1 Introduction A Markov Chain is a sequence of random variables X 0, X 1,, where each X i S, such that P(X i+1 = s i+1 X i = s i, X i 1 = s i 1,, X 0 = s 0 ) = P(X

### Wald s Identity. by Jeffery Hein. Dartmouth College, Math 100

Wald s Identity by Jeffery Hein Dartmouth College, Math 100 1. Introduction Given random variables X 1, X 2, X 3,... with common finite mean and a stopping rule τ which may depend upon the given sequence,

### Markov Chains and Applications

Markov Chains and Applications Alexander Volfovsky August 7, 2007 Abstract In this paper I provide a quick overview of Stochastic processes and then quickly delve into a discussion of Markov Chains. There

### Continuous-Time Markov Chains - Introduction

25 Continuous-Time Markov Chains - Introduction Prior to introducing continuous-time Markov chains today, let us start off with an example involving the Poisson process. Our particular focus in this example

### Part IB Markov Chains

Part IB Markov Chains Based on lectures by G. R. Grimmett Notes taken by Dexter Chua Michaelmas 2015 Discrete-time chains Definition and basic properties, the transition matrix. Calculation of n-step transition

### Introduction to Stochastic Processes. Lothar Breuer

Introduction to Stochastic Processes Lothar Breuer Contents 1 Some general definitions 1 2 Markov Chains and Queues in Discrete Time 3 2.1 Definition.............................. 3 2.2 Classification

### LECTURE 4. Last time: Lecture outline

LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random

### Markov Chains: An Introduction/Review

Markov Chains: An Introduction/Review David Sirl dsirl@maths.uq.edu.au http://www.maths.uq.edu.au/ dsirl/ AUSTRALIAN RESEARCH COUNCIL Centre of Excellence for Mathematics and Statistics of Complex Systems

### 1 Gambler s Ruin Problem

Coyright c 2009 by Karl Sigman 1 Gambler s Ruin Problem Let N 2 be an integer and let 1 i N 1. Consider a gambler who starts with an initial fortune of \$i and then on each successive gamble either wins

### Lecture notes on Markov chains Olivier Lévêque, olivier.leveque#epfl.ch National University of Ireland, Maynooth, August 2-5, 2011

Lecture notes on Markov chains Olivier Lévêque, olivier.leveque#epfl.ch National University of Ireland, Maynooth, August 2-5, 2011 1 Discrete-time Markov chains 1.1 Basic definitions and Chapman-Kolmogorov

### MATH 289 PROBLEM SET 1: INDUCTION. 1. The induction Principle The following property of the natural numbers is intuitively clear:

MATH 89 PROBLEM SET : INDUCTION The induction Principle The following property of the natural numbers is intuitively clear: Axiom Every nonempty subset of the set of nonnegative integers Z 0 = {0,,, 3,

### 5. Convergence of sequences of random variables

5. Convergence of sequences of random variables Throughout this chapter we assume that {X, X 2,...} is a sequence of r.v. and X is a r.v., and all of them are defined on the same probability space (Ω,

### Markov chain mixing and coupling

Markov chain mixing and coupling Recall from Lecture 2: Definition 0.. Let S be a countable set and (X n ) a sequence of random variables taking values in S. We say that (X n ) is a Markov chain if it

### University of Chicago Autumn 2003 CS Markov Chain Monte Carlo Methods

University of Chicago Autumn 2003 CS37101-1 Markov Chain Monte Carlo Methods Lecture 2: October 7, 2003 Markov Chains, Coupling, Stationary Distribution Eric Vigoda 2.1 Markov Chains In this lecture, we

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 1 9/3/2008 PROBABILISTIC MODELS AND PROBABILITY MEASURES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 1 9/3/2008 PROBABILISTIC MODELS AND PROBABILITY MEASURES Contents 1. Probabilistic experiments 2. Sample space 3. Discrete probability

### An important class of codes are linear codes in the vector space Fq n, where F q is a finite field of order q.

Chapter 3 Linear Codes An important class of codes are linear codes in the vector space Fq n, where F q is a finite field of order q. Definition 3.1 (Linear code). A linear code C is a code in Fq n for

### 1. Markov chains. 1.1 Specifying and simulating a Markov chain. Page 5

1. Markov chains Page 5 Section 1. What is a Markov chain? How to simulate one. Section 2. The Markov property. Section 3. How matrix multiplication gets into the picture. Section 4. Statement of the Basic

### 4.9 Markov matrices. DEFINITION 4.3 A real n n matrix A = [a ij ] is called a Markov matrix, or row stochastic matrix if. (i) a ij 0 for 1 i, j n;

49 Markov matrices DEFINITION 43 A real n n matrix A = [a ij ] is called a Markov matrix, or row stochastic matrix if (i) a ij 0 for 1 i, j n; (ii) a ij = 1 for 1 i n Remark: (ii) is equivalent to AJ n

### NOTES ON ELEMENTARY PROBABILITY

NOTES ON ELEMENTARY PROBABILITY KARL PETERSEN 1. Probability spaces Probability theory is an attempt to work mathematically with the relative uncertainties of random events. In order to get started, we

### 1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

### Notes for Math 450 Matlab listings for Markov chains

Notes for Math 45 Matlab listings for Markov chains Renato Feres Classification of States Consider a Markov chain X, X, X 2..., with transition probability matrix P and set of states S. A state j is said

### 1. R In this and the next section we are going to study the properties of sequences of real numbers.

+a 1. R In this and the next section we are going to study the properties of sequences of real numbers. Definition 1.1. (Sequence) A sequence is a function with domain N. Example 1.2. A sequence of real

### Lecture 6: The Group Inverse

Lecture 6: The Group Inverse The matrix index Let A C n n, k positive integer. Then R(A k+1 ) R(A k ). The index of A, denoted IndA, is the smallest integer k such that R(A k ) = R(A k+1 ), or equivalently,

### 3. Mathematical Induction

3. MATHEMATICAL INDUCTION 83 3. Mathematical Induction 3.1. First Principle of Mathematical Induction. Let P (n) be a predicate with domain of discourse (over) the natural numbers N = {0, 1,,...}. If (1)

### Sequences and Series

Contents 6 Sequences and Series 6. Sequences and Series 6. Infinite Series 3 6.3 The Binomial Series 6 6.4 Power Series 3 6.5 Maclaurin and Taylor Series 40 Learning outcomes In this Workbook you will

### 6. Metric spaces. In this section we review the basic facts about metric spaces. d : X X [0, )

6. Metric spaces In this section we review the basic facts about metric spaces. Definitions. A metric on a non-empty set X is a map with the following properties: d : X X [0, ) (i) If x, y X are points

### ORDERS OF ELEMENTS IN A GROUP

ORDERS OF ELEMENTS IN A GROUP KEITH CONRAD 1. Introduction Let G be a group and g G. We say g has finite order if g n = e for some positive integer n. For example, 1 and i have finite order in C, since

### The Exponential Distribution

21 The Exponential Distribution From Discrete-Time to Continuous-Time: In Chapter 6 of the text we will be considering Markov processes in continuous time. In a sense, we already have a very good understanding

### S = {1, 2,..., n}. P (1, 1) P (1, 2)... P (1, n) P (2, 1) P (2, 2)... P (2, n) P = . P (n, 1) P (n, 2)... P (n, n)

last revised: 26 January 2009 1 Markov Chains A Markov chain process is a simple type of stochastic process with many social science applications. We ll start with an abstract description before moving

### HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!

Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following

### Lecture 3. Mathematical Induction

Lecture 3 Mathematical Induction Induction is a fundamental reasoning process in which general conclusion is based on particular cases It contrasts with deduction, the reasoning process in which conclusion

### B. Maddah ENMG 501 Engineering Management I 03/17/07

B. Maddah ENMG 501 Engineering Management I 03/17/07 Discrete Time Markov Chains (3) Long-run Properties of MC (Stationary Solution) Consider the two-state MC of the weather condition in Example 4. P 0.5749

### a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

### COUNTABLE-STATE MARKOV CHAINS

Chapter 5 COUNTABLE-STATE MARKOV CHAINS 5.1 Introduction and classification of states Markov chains with a countably-infinite state space (more briefly, countable-state Markov chains) exhibit some types

### Prime Numbers. Chapter Primes and Composites

Chapter 2 Prime Numbers The term factoring or factorization refers to the process of expressing an integer as the product of two or more integers in a nontrivial way, e.g., 42 = 6 7. Prime numbers are

### Introduction to Flocking {Stochastic Matrices}

Supelec EECI Graduate School in Control Introduction to Flocking {Stochastic Matrices} A. S. Morse Yale University Gif sur - Yvette May 21, 2012 CRAIG REYNOLDS - 1987 BOIDS The Lion King CRAIG REYNOLDS

### Markov Decision Processes: Lecture Notes for STP 425. Jay Taylor

Markov Decision Processes: Lecture Notes for STP 425 Jay Taylor November 26, 2012 Contents 1 Overview 4 1.1 Sequential Decision Models.............................. 4 1.2 Examples........................................

### Theorem 2. If x Q and y R \ Q, then. (a) x + y R \ Q, and. (b) xy Q.

Math 305 Fall 011 The Density of Q in R The following two theorems tell us what happens when we add and multiply by rational numbers. For the first one, we see that if we add or multiply two rational numbers

### Continuous Time Markov Chains

Continuous Time Markov Chains Dan Crytser March 11, 2011 1 Introduction The theory of Markov chains of the discrete form X 1, X 2,... can be adapted to a continuous form X t, t [0, ). This requires use

### 4.3 Limit of a Sequence: Theorems

4.3. LIMIT OF A SEQUENCE: THEOREMS 5 4.3 Limit of a Sequence: Theorems These theorems fall in two categories. The first category deals with ways to combine sequences. Like numbers, sequences can be added,

### NOTES ON MEASURE THEORY. M. Papadimitrakis Department of Mathematics University of Crete. Autumn of 2004

NOTES ON MEASURE THEORY M. Papadimitrakis Department of Mathematics University of Crete Autumn of 2004 2 Contents 1 σ-algebras 7 1.1 σ-algebras............................... 7 1.2 Generated σ-algebras.........................

### 0 ( x) 2 = ( x)( x) = (( 1)x)(( 1)x) = ((( 1)x))( 1))x = ((( 1)(x( 1)))x = ((( 1)( 1))x)x = (1x)x = xx = x 2.

SOLUTION SET FOR THE HOMEWORK PROBLEMS Page 5. Problem 8. Prove that if x and y are real numbers, then xy x + y. Proof. First we prove that if x is a real number, then x 0. The product of two positive

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### MATHEMATICAL INDUCTION AND DIFFERENCE EQUATIONS

1 CHAPTER 6. MATHEMATICAL INDUCTION AND DIFFERENCE EQUATIONS 1 INSTITIÚID TEICNEOLAÍOCHTA CHEATHARLACH INSTITUTE OF TECHNOLOGY CARLOW MATHEMATICAL INDUCTION AND DIFFERENCE EQUATIONS 1 Introduction Recurrence

### Elementary Number Theory We begin with a bit of elementary number theory, which is concerned

CONSTRUCTION OF THE FINITE FIELDS Z p S. R. DOTY Elementary Number Theory We begin with a bit of elementary number theory, which is concerned solely with questions about the set of integers Z = {0, ±1,

### Finite Markov Chains and Algorithmic Applications. Matematisk statistik, Chalmers tekniska högskola och Göteborgs universitet

Finite Markov Chains and Algorithmic Applications Olle Häggström Matematisk statistik, Chalmers tekniska högskola och Göteborgs universitet PUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGE

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J. Olver 4. Gaussian Elimination In this part, our focus will be on the most basic method for solving linear algebraic systems, known as Gaussian Elimination in honor

### NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### Math212a1010 Lebesgue measure.

Math212a1010 Lebesgue measure. October 19, 2010 Today s lecture will be devoted to Lebesgue measure, a creation of Henri Lebesgue, in his thesis, one of the most famous theses in the history of mathematics.

### 16.1. Sequences and Series. Introduction. Prerequisites. Learning Outcomes. Learning Style

Sequences and Series 16.1 Introduction In this block we develop the ground work for later blocks on infinite series and on power series. We begin with simple sequences of numbers and with finite series

### LEARNING OBJECTIVES FOR THIS CHAPTER

CHAPTER 2 American mathematician Paul Halmos (1916 2006), who in 1942 published the first modern linear algebra book. The title of Halmos s book was the same as the title of this chapter. Finite-Dimensional

### Section 1. Statements and Truth Tables. Definition 1.1: A mathematical statement is a declarative sentence that is true or false, but not both.

M3210 Supplemental Notes: Basic Logic Concepts In this course we will examine statements about mathematical concepts and relationships between these concepts (definitions, theorems). We will also consider

### princeton university F 02 cos 597D: a theorist s toolkit Lecture 7: Markov Chains and Random Walks

princeton university F 02 cos 597D: a theorist s toolkit Lecture 7: Markov Chains and Random Walks Lecturer: Sanjeev Arora Scribe:Elena Nabieva 1 Basics A Markov chain is a discrete-time stochastic process

### IEOR 6711: Stochastic Models, I Fall 2012, Professor Whitt, Final Exam SOLUTIONS

IEOR 6711: Stochastic Models, I Fall 2012, Professor Whitt, Final Exam SOLUTIONS There are four questions, each with several parts. 1. Customers Coming to an Automatic Teller Machine (ATM) (30 points)

### Reading 7 : Program Correctness

CS/Math 240: Introduction to Discrete Mathematics Fall 2015 Instructors: Beck Hasti, Gautam Prakriya Reading 7 : Program Correctness 7.1 Program Correctness Showing that a program is correct means that

### Students in their first advanced mathematics classes are often surprised

CHAPTER 8 Proofs Involving Sets Students in their first advanced mathematics classes are often surprised by the extensive role that sets play and by the fact that most of the proofs they encounter are

### REAL ANALYSIS I HOMEWORK 2

REAL ANALYSIS I HOMEWORK 2 CİHAN BAHRAN The questions are from Stein and Shakarchi s text, Chapter 1. 1. Prove that the Cantor set C constructed in the text is totally disconnected and perfect. In other

### MAT2400 Analysis I. A brief introduction to proofs, sets, and functions

MAT2400 Analysis I A brief introduction to proofs, sets, and functions In Analysis I there is a lot of manipulations with sets and functions. It is probably also the first course where you have to take

### The purpose of this section is to derive the asymptotic distribution of the Pearson chi-square statistic. k (n j np j ) 2. np j.

Chapter 7 Pearson s chi-square test 7. Null hypothesis asymptotics Let X, X 2, be independent from a multinomial(, p) distribution, where p is a k-vector with nonnegative entries that sum to one. That

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

### Logic, Sets, and Proofs

Logic, Sets, and Proofs David A. Cox and Catherine C. McGeoch Amherst College 1 Logic Logical Statements. A logical statement is a mathematical statement that is either true or false. Here we denote logical

### Mathematical Induction

Chapter 2 Mathematical Induction 2.1 First Examples Suppose we want to find a simple formula for the sum of the first n odd numbers: 1 + 3 + 5 +... + (2n 1) = n (2k 1). How might we proceed? The most natural

### Chapter 6 Finite sets and infinite sets. Copyright 2013, 2005, 2001 Pearson Education, Inc. Section 3.1, Slide 1

Chapter 6 Finite sets and infinite sets Copyright 013, 005, 001 Pearson Education, Inc. Section 3.1, Slide 1 Section 6. PROPERTIES OF THE NATURE NUMBERS 013 Pearson Education, Inc.1 Slide Recall that denotes

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### Math 421, Homework #5 Solutions

Math 421, Homework #5 Solutions (1) (8.3.6) Suppose that E R n and C is a subset of E. (a) Prove that if E is closed, then C is relatively closed in E if and only if C is a closed set (as defined in Definition

### Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

### I. GROUPS: BASIC DEFINITIONS AND EXAMPLES

I GROUPS: BASIC DEFINITIONS AND EXAMPLES Definition 1: An operation on a set G is a function : G G G Definition 2: A group is a set G which is equipped with an operation and a special element e G, called

### IEOR 6711: Stochastic Models I Fall 2012, Professor Whitt, Tuesday, September 11 Normal Approximations and the Central Limit Theorem

IEOR 6711: Stochastic Models I Fall 2012, Professor Whitt, Tuesday, September 11 Normal Approximations and the Central Limit Theorem Time on my hands: Coin tosses. Problem Formulation: Suppose that I have

### PROVING STATEMENTS IN LINEAR ALGEBRA

Mathematics V2010y Linear Algebra Spring 2007 PROVING STATEMENTS IN LINEAR ALGEBRA Linear algebra is different from calculus: you cannot understand it properly without some simple proofs. Knowing statements

### BIN504 - Lecture XI. Bayesian Inference & Markov Chains

BIN504 - Lecture XI Bayesian Inference & References: Lee, Chp. 11 & Sec. 10.11 Ewens-Grant, Chps. 4, 7, 11 c 2012 Aybar C. Acar Including slides by Tolga Can Rev. 1.2 (Build 20130528191100) BIN 504 - Bayes

### Math Homework 7 Solutions

Math 450 - Homework 7 Solutions 1 Find the invariant distribution of the transition matrix: 0 1 0 P = 2 1 0 3 3 p 1 p 0 The equation πp = π, and the normalization condition π 1 + π 2 + π 3 = 1 give the

### Long Time Properties of Reducible Markov Chains

Long Time Properties of Reducible Markov Chains Tuesday, March 04, 2014 2:03 PM Homework 2 due Friday, March 7. Last time we showed how to prove existence of stationary distributions for irreducible finite

### CARDINALITY, COUNTABLE AND UNCOUNTABLE SETS PART ONE

CARDINALITY, COUNTABLE AND UNCOUNTABLE SETS PART ONE With the notion of bijection at hand, it is easy to formalize the idea that two finite sets have the same number of elements: we just need to verify

### 2. Methods of Proof Types of Proofs. Suppose we wish to prove an implication p q. Here are some strategies we have available to try.

2. METHODS OF PROOF 69 2. Methods of Proof 2.1. Types of Proofs. Suppose we wish to prove an implication p q. Here are some strategies we have available to try. Trivial Proof: If we know q is true then

### 11 Ideals. 11.1 Revisiting Z

11 Ideals The presentation here is somewhat different than the text. In particular, the sections do not match up. We have seen issues with the failure of unique factorization already, e.g., Z[ 5] = O Q(

### Lecture 13: Martingales

Lecture 13: Martingales 1. Definition of a Martingale 1.1 Filtrations 1.2 Definition of a martingale and its basic properties 1.3 Sums of independent random variables and related models 1.4 Products of

### 4.5 Linear Dependence and Linear Independence

4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then

### Markov Chains. Chapter 4. 4.1 Stochastic Processes

Chapter 4 Markov Chains 4 Stochastic Processes Often, we need to estimate probabilities using a collection of random variables For example, an actuary may be interested in estimating the probability that

### HOLOMORPHIC FRAMES FOR WEAKLY CONVERGING HOLOMORPHIC VECTOR BUNDLES

HOLOMORPHIC FRAMES FOR WEAKLY CONVERGING HOLOMORPHIC VECTOR BUNDLES GEORGIOS D. DASKALOPOULOS AND RICHARD A. WENTWORTH Perhaps the most useful analytic tool in gauge theory is the Uhlenbeck compactness

### MATH 2030: SYSTEMS OF LINEAR EQUATIONS. ax + by + cz = d. )z = e. while these equations are not linear: xy z = 2, x x = 0,

MATH 23: SYSTEMS OF LINEAR EQUATIONS Systems of Linear Equations In the plane R 2 the general form of the equation of a line is ax + by = c and that the general equation of a plane in R 3 will be we call

### CHAPTER 3. Sequences. 1. Basic Properties

CHAPTER 3 Sequences We begin our study of analysis with sequences. There are several reasons for starting here. First, sequences are the simplest way to introduce limits, the central idea of calculus.

### 1.3 Induction and Other Proof Techniques

4CHAPTER 1. INTRODUCTORY MATERIAL: SETS, FUNCTIONS AND MATHEMATICAL INDU 1.3 Induction and Other Proof Techniques The purpose of this section is to study the proof technique known as mathematical induction.

### 1. Sigma algebras. Let A be a collection of subsets of some fixed set Ω. It is called a σ -algebra with the unit element Ω if. lim sup. j E j = E.

Real Analysis, course outline Denis Labutin 1 Measure theory I 1. Sigma algebras. Let A be a collection of subsets of some fixed set Ω. It is called a σ -algebra with the unit element Ω if (a), Ω A ; (b)

### Time Series Analysis

Time Series Analysis Time series and stochastic processes Andrés M. Alonso Carolina García-Martos Universidad Carlos III de Madrid Universidad Politécnica de Madrid June July, 2012 Alonso and García-Martos

### P (A) = lim P (A) = N(A)/N,

1.1 Probability, Relative Frequency and Classical Definition. Probability is the study of random or non-deterministic experiments. Suppose an experiment can be repeated any number of times, so that we

### Approximating the entropy of a 2-dimensional shift of finite type

Approximating the entropy of a -dimensional shift of finite type Tirasan Khandhawit c 4 July 006 Abstract. In this paper, we extend the method used to compute entropy of -dimensional subshift and the technique