Math 117 Chapter 10 Markov Chains

Transcription

1 Math 117 Chapter 10 Flathead Valley Community College Regular Page 1 of 10

2 1. When modeling in areas such as business, psychology, biology, sociology, and economics it is often useful to classify the items of study into specific classes or states. For example socio-economic status is often classified as upper class, middle class and upper class. The real area of interest is what happens to the distribution when there is movement from one state to another. Through a great deal of research it is possible to determine the probability of moving from one state to another or even stay in that state over a fixed period of time. Each time step is called and experiment. Once all of the probabilities have been determined it is now possible to make a transition diagram or a transition matrix. Regular 1.1. Transition Diagrams and Matrices A transition matrix P must satisfy the following characteristics: 1. It is square, since all possible states must be used both as row and as columns. 2. All entries are between 0 and 1, inclusive; this is because all entries represent probabilities. 3. The sum of the entries in any row must be 1, since the numbers in the row give the probability of changing from the stat at the left to one of the states indicated across the top. Page 2 of 10

3 It will be very important that you can make a create a transition matrix from a transition diagram as well as draw a transition diagram given a transition matrix Markov Chain A Markov Chain can be thought of as a sequence of repeatable experiments that satisfy some very specific conditions. An experiment is called a Markov chain if 1. the outcome of each experiment is one of a set of discrete states; Regular 2. the outcome of an experiment depends only on the present state, and not on any past states; 3. the transition probabilities remain constant from one transition to the next. Notice that the transition matrix conditions satisfies the three main features of a Markov chain. Page 3 of Repeated Experiments A probability vector is a matrix of only one row, having nonnegative entries. The probability vector is a row matrix that gives the probability distribution of the population over the given states. If the transition matrix P and the initial probability vector X 0 is given, then after running the experiment once the resulting probability vector is X 1 = X 0 P. Notice that on the right-hand side we have a

4 row matrix times a square matrix. If you have forgotten how to multiply matrices, go back to Section 2.4 for review. You will be responsible for multiplication by a hand as well as using your calculator. The idea of is to take a given probability vector, apply the experiment over and over to find new probability distributions at the end of each experiment. The goal of the Markov Chain is to determine the long-term outcome of the system. Applying Suppose a Markov chain has initial probability vector X 0 = [i 1 i 2 i 3 i n ] and transition matrix P. The probability vector after n repetitions of the experiment is X 0 P n. P k gives the probabilities of a transition from one state to another in k repetitions of an experiment, provided the transition probabilities remain constant from one repetition to the next. For the rest of this chapter we will investigate two long-term outcomes for Markov chains, equilibrium distributions and absorbing states. In the equilibrium case repeated experiments will result in a fixed probability distribution regardless of the initial state. In the absorbing case, all of the population will be absorbed into one or more states. Regular Page 4 of 10

5 2. Regular A transition matrix is regular if some power of the matrix contains all positive entries. A regular transition matrix will not have any zero entries in the longterm. A Markov chain is a regular Markov chain if its transition matrix is regular. If a transition matrix P k and P k+1 have zeros in identical places for any k, they will appear in those places for all higher powers of P. In this case P is not regular. Regular 2.1. Equilibrium Vector of a Markov Chain If a Markov chain with transition matrix P is regular, then there is a unique vector V such that for any initial probability vector v and for large values of n, v P n V. Vector V is called the equilibrium vector or the fixed vector of the Markov chain. If the initial vector is the equilibrium vector then V P = V. Theorem If a Markov chain with transition matrix P is regular, then there exists a probability vector V such that V P = V. Page 5 of 10

6 Instead of running the experiment many times it would be nice to find the equilibrium vector right away. To achieve this goal we must solve the matrix problem V P = V. The technique used here will be similar to solving the d Input-Output models from Section 2.6. (You may want to go back and do some review there as well.) The main difference now is that the row vector V is multiplied on the left of P. (In the d Input-Output models the column vector was multiplied on the right.) Begin by subtracting V to the left, V P V = 0. Notice the 0 on the right-hand side is the zero row vector. Now factor V, but don t forget that when factoring matrices one often is left with the identity I, V (P I) = 0 At this point it will be best to subtract the identity from the transition matrix and then multiply by V. Now you should have a system of linear equations all equal to zero. Trying to solve this system using your calculator will result in a free variable, not what we want for an equilibrium vector. To finish the problem recall that V is a probability vector so the sum of all of the entries must equal 1. Add this equation to the equations from V (P I) = 0. Your text places the row of 1s in the first row. I will put the row of 1s at the bottom of the system. It really doesn t matter since we are going to let the calculator solve the system. Either way, write the system as a matrix problem and solve with your calculator using rref. You will be asked to do this on the exam, so make sure you practice a few. Regular Page 6 of 10

7 3. Absorbing The last application involves absorbing Markov chains. In this case there is one or more states that will eventually absorb the entire population. While there is no escaping the absorbing states there are still a few questions that can be answered: 1. Starting in a nonabsorbing state, on average, how many times is another non-absorbing state visited before absorption? 2. Starting in a nonabsorbing state, on average, how many times can the experiment be repeated before that state is totally absorbed? 3. Starting in a nonabsorbing state what is the probability of ending in a specific absorbing state? As it turns out the initial probability distribution v 0 will have a significant effect on the answers to the two questions above. Before we get too far ahead of ourselves, lets look at some definitions. Absorbing State State i of a Markov chain is an absorbing state if p ii = 1. All other probabilities in the row will be 0. Regular Page 7 of 10 Absorbing Markov Chain A Markov chain is an absorbing chain if and only if the following two conditions are satisfied:

8 1. the chain has at least one absorbing state; and 2. it is possible to go from any nonabsorbing state to an absorbing state (perhaps in more than one step). The first thing to do is to identify the absorbing states. Here again you must be comfortable working back and forth between the transition diagram and the transition matrix. Regardless of what you start with, diagram or matrix, you must rewrite the transition matrix with the absorbing states in the first rows followed by the remaining states. This may amount to some relabeling of your rows and columns. When this is completed your transition matrix should be of the form [ ] Im 0 P = R Q The matrix above calls for a bit of explanation. The matrix P is not a 2 2 matrix, but rather a square matrix that can be divided into four distinct blocks. I m is the m m identity matrix. The size of the matrix is determined by the number of absorbing states; 2 absorbing states, I 2, 4 absorbing states, I 4. The matrix 0 is actually a matrix of zeros. Lastly R and Q will be used to help us answer the questions posed at the beginning of this section. Note that Q will always be a square matrix. As with the regular Markov chain, repetitions of the the experiment result in a probability vector v k = v 0 P k where P k will have the same entries for k very large. The proof of the following result will not be given here, please consult your Regular Page 8 of 10

9 text pages for the proof. Now as k gets large (denoted k ), [ ] P k Im 0 F R Again this matrix needs some explanation. The top matrices do not change. The square matrix in the lower right corner approaches a zero matrix. The part of the matrix that is most interesting and carries the most information is in the lower left. With repeated experiments R F R, where F = (I n Q) 1. F is called the fundamental matrix. The fundamental matrix is the inverse of the identity minus matrix Q. You will always use your calculator to compute the fundamental matrix. The entries in the fundamental matrix F give the expected number of visits to each state before absorption. This is the same expected value we encountered in Chapter 7, but without all of the work. The sum of the entries in any row will given the expected number of repetitions of the experiment before that state is totally absorbed. The fundamental matrix answers two of our original questions. Once the fundamental matrix has been determined, use your calculator to compute F R. The entries in F R give the probability that if an object started in one state, then it ended in the given absorbing state. This is the answer to our last question posed earlier. This an awful lot to absorb (pardon the pun) so lets summarize: 0 n Regular Page 9 of 10

10 Properties of an Absorbing Markov Chain 1. Regardless of the initial state, in a finite number of steps the chain will enter and absorbing state and then stay in that state. 2. The powers of the transition matrix get closer and closer to some particular matrix. 3. The long-term trend depends on the initial state. 4. Let P be the transition matrix for an absorbing Markov chain. Rearrange the rows and columns of P so that the absorbing states come first. Matrix P will have the form [ ] Im 0 P =, R Q where I m is an identity matrix, with m equal to the number of absorbing states, and 0 is a matrix of all zeros. The fundamental matrix is defined as F = (I n Q) 1, where I n has the same size as Q. The elements in row i, column j of the fundamental matrix gives the number of visits to state j that are expected to occur before absorption, given that the current state is state i. 5. The product F R gives the matrix of probabilities that a particular initial nonabsorbing state will lead to a particular absorbing state. Regular Page 10 of 10