EE365: Markov Chains. Markov chains. Transition Matrices. Distribution Propagation. Other Models

Transcription

1 EE365: Markov Chains Markov chains Transition Matrices Distribution Propagation Other Models 1

2 Markov chains 2

3 Markov chains a model for dynamical systems with possibly uncertain transitions very widely used, in many application areas one of a handful of core effective mathematical and computational tools often used to model systems that are not random; e.g., language 3

4 Stochastic dynamical system representation x t+1 = f(x t, w t) t = 0, 1,... x 0, w 0, w 1,... are independent random variables state transitions are nondeterministic, uncertain combine system x t+1 = f(x t, u t, w t) with policy u t = µ t(x t) 4

5 Example: Inventory model with ordering policy x t+1 = [x t d t + u t] [0,C] u t = µ(x t) x t X = {0,..., C} is inventory level at time t C > 0 is capacity d t {0, 1,...} is demand that arrives just after time t u t {0, 1,...} is new stock added to inventory µ : X {0, 1,..., C} is ordering policy [z] [0,C] = min(max(z, 0), C) (z clipped to interval [0, C]) assumption: d 0, d 1,... are independent 5

6 Example: Inventory model with ordering policy x t+1 = [x t d t + u t] [0,C] u t = µ(x t) capacity C = 6 demand distribution p i = Prob(d t = i) policy: refill if x t 1 µ(x) = start with full inventory x 0 = C p = [ ] { C x if x 1 0 otherwise 6

7 Simulation x t t given initial state x 0 for t = 0,..., T 1 simulate process noise w t := sample(p) update the state x t+1 = f(x t, w t) 7

8 Simulation called a particle simulation gives a sample of the joint distribution of (x 0,..., x T ) useful to approximately evaluate probabilities and expectations via Monte Carlo simulation to get a feel for what trajectories look like (e.g., for model validation) 8

9 Graph representation p 0 p 1 p 2 p 0 p 2 p p 0 3 p 0 4 p 0 5 p 0 6 p 0 p 1 p 1 p 1 p 1 p 1 p 2 p 2 p 2 p 2 p 2 called the transition graph each vertex (or node) corresponds to a state edge i j is labeled with transition probability Prob(x t+1 = j x t = i) 9

10 The tree each vertex of the tree corresponds to a path e.g., Prob(x 3 = 1 x 0 = 2) =

11 The birth-death chain self-loops can be omitted, since they can be figured out from the outgoing probabilities 11

12 Example: Metastability

13 Transition Matrices 13

14 Transition matrix representation we define the transition matrix P R n n P ij = Prob(x t+1 = j x t = i) P 1 = 1 and P 0 elementwise a matrix with these two properties is called a stochastic matrix if P and Q are stochastic, then so is P Q 14

15 Transition matrix representation p 1 p 0 p 2 p 0 p 2 p p 0 3 p 0 4 p 0 5 p 0 6 p 0 p 1 p 1 p 1 p 1 p 1 p 2 p 2 p 2 p 2 p 2 transition matrix P =

16 Particle simulation given the transition matrix Given the distribution of initial states d R 1 n the transition matrix P R n n, with rows p 1,..., p n Algorithm: choose initial state x 0 := sample(d) for t = 0,..., T 1 find the distribution of the next state d next := p xt sample the next state x t+1 := sample(d next) 16

17 Definition of a Markov chain sequence of random variables x t : Ω X is a Markov chain if, for all s 0, s 1,... and all t, Prob(x t+1 = s t+1 x t = s t,..., x 0 = s 0) = Prob(x t+1 = s t+1 x t = s t) called the Markov property means that the system is memoryless x t is called the state at time t; X is called the state space if you know current state, then knowing past states doesn t give additional knowledge about next state (or any future state) 17

18 The joint distribution the joint distribution of x 0, x 1,..., x t factorizes according to Prob ( x 0 = a, x 1 = b, x 2 = c,..., x t = q ) = d ap ab P bc P pq because the chain rule gives Prob(x t = s t, x t 1 = s t 1,..., x 0 = s 0) = Prob(x t = s t x t 1 = s t 1,..., x 0 = s 0) Prob(x t 1 = s t 1,..., x 0 = s 0) abbreviating x t = s t to just x t, we can repeatedly apply this to give Prob(x t, x t 1,..., x 0) = Prob(x t x t 1,..., x 0) Prob(x t 1 x t 2,..., x 0) Prob(x 1 x 0) Prob(x 0) = Prob(x t x t 1) Prob(x t 1 x t 2) Prob(x 1 x 0) Prob(x 0) 18

19 The joint distribution the joint distribution completely specifies the process; for example E f(x 0, x 1, x 2, x 3) = f(a, b, c, d) d ap ab P bc P cd a,b,c,d X in principle we can compute the probability of any event and the expected value of any function, but this requires a sum over n T terms we can compute some expected values far more efficiently (more later) 19

20 Distribution Propagation 20

21 Distribution propagation π t+1 = π tp here π t denotes distribution of x t, so (π t) i = Prob(x t = i) can compute marginals π 1,..., π T in T n 2 operations a useful type of simulation... 21

22 Example: Distribution propagation p0 p1 p2 p0 p1 p p0 3 p0 4 p0 5 p0 6 p0 p1 p1 p1 p1 p1 p2 p2 p2 p2 p t = t = t = t = t = t = t = t = t =

23 Example: Reordering what is the probability we reorder at time t? compute π t by distribution propagation, and use Prob(x t {0, 1}) = π t [ ] T E g(x t) t 23

24 Evaluating expectation of separable function suppose f : X T +1 R is separable, i.e., where f t : X R f(x 0,..., x T ) = f 0(x 0) + + f T (x T ) then we have E f(x) = π 0f π T f T using distribution propagation, we can compute exactly with T n 2 operations 24

25 Powers of the transition matrix as an example of the use of the joint distribution, let s show (P k ) ij = Prob ( x t+k = j x t = i ) this holds because Prob ( x t+k = s t+k x t = s t ) = Prob ( x t+k = s t+k and x t = s t ) Prob ( x t = s t ) = = s 0,s 1,...,s t 1,s t+1,...,s t+k 1 d s0 P s0 s 1 P s1 s 2 P st+k 1 s t+k Prob ( x t = s t ) ( )( d s0 P s0 s 1 P st 1 s t P st s t+1 P st+k 1 s t+k s 0,s 1,...,s t 1 s t+1,...,s t+k 1 Prob ( ) x t = s t ) 25

26 Example: k step transition probabilities two ways to compute Prob(x t = j x 0 = i) direct method: sum products of probabilities over n t state sequences using matrix powers: Prob(x t = j x 0 = i) = (P t ) ij we can compute this in O(n 3 t) arithmetic operations (by multiplying matrices to get P ) O(n 3 ) arithmetic operations (by first diagonalizing P ) 26

27 Other Models 27

28 Time-varying Markov chains we may have a time-varying Markov chain, with one transition matrix for each time (P t) ij = Prob(x t+1 = j x t = i) suppose Prob(x t = a) 0 for all a X and t then the factorization property that there exists stochastic matrices P 0, P 1,... and a distribution d such that Prob ( x 0 = a, x 1 = b, x 2 = c,..., x t = q ) = d a(p 0) ab (P 1) bc (P t 1) pq is equivalent to the Markov property the theory of factorization properties of distributions is called Bayesian networks or graphical models (with Markov chains the simplest case) 28

29 Dynamical system representation revisited x t+1 = f(x t, w t) If x 0, w 0, w 1,... are independent then x 0, x 1,... is Markov 29

30 Dynamical system representation To see that x 0, x 1,... is Markov, notice that Prob(x t+1 = s t+1 x 0 = s 0,..., x t = s t) = Prob ( f(s t, w t) = s t+1 x 0 = s 0,..., x t = s t ) = Prob ( f(s t, w t) = s t+1 φ t(x 0, w 0,..., w t 1) = (s 0,..., s t) ) = Prob ( f(s t, w t) = s t+1 ) (x 0,..., x t) = φ t(x 0, w 0,..., w t 1) follows from x t+1 = f(x t, w t) last line holds since w t (x 0, w 0,..., w t 1) and x y = f(x) g(y) x 0, x 1,... is Markov because the above approach also gives Prob(x t+1 = s t+1 x t = s t) = Prob ( f(s t, w t) = s t+1 ) 30

31 State augmentation we often have models x t+1 = f(x t, w t) where w 0, w 1,... is Markov w 0, w 1,... satisfy w t+1 = g(w t, v t) where v 0, v 1,... are independent [ ] xt construct an equivalent Markov chain, with state z t = dynamics z t+1 = h(z t, v t) where h(z, v) = [ ] f(z1, z 2) g(z 2, v) w t 31

32 Markov k x t+1 = f(x t, x t 1,..., x t k+1, w t) called a Markov-k model assume (x 0, x 1,..., x k 1 ), w 0, w 1, w 2,... are independent initial distribution on (x 0, x 1,..., x k 1 ) Markov-k property: for all s 0, s 1,..., Prob(x t+1 = s t+1 x t = s t,..., x 0 = s 0) = Prob(x t+1 = s t+1 x t = s t, x t 1 = s t 1,..., x t k+1 = s t k+1 ) 32

33 Markov k x t+1 = f(x t, x t 1,..., x t k+1, w t) equivalent Markov system, with state z t Z = X k ; notice that Z = X k z t = x t x t 1. x t k+1 dynamics z t+1 = g(z t, w t) where f ( z 1, z 2,..., z k, w ) z 1 g(z, w) =. z k 1 33

34 Markov language models given text, construct a Markov k model by counting the frequency of word sequences applications: run particle simulation to generate new text suggest correct spellings when spell checking optical character recognition finding similar documents in information retrieval 34

35 Example: Markov language models example: approximately 78,000 words, 8,200 unique He wasnt true and disappointment that wouldnt be very old school song at once it on the grayish white stone well have charlies work much to keep your wand still had a mouse into a very scratchy whiskery kiss. Then yes yes. I hope im pretty but there had never seen me what had to vacuum anymore because its all right were when flint had shot up while uncle vernon around the sun rose up to. Bill used to. Ron who to what he teaches potions one. I suppose they couldnt think about it harry hed have found out that didnt trust me all sorts of a letter which one by older that was angrier about what they would you see here albus dumbledore it was very well get a rounded bottle of everyone follows me an awful lot his face the minute. Well send you get out and see professor snape. But it must have his sleeve. I am i think. Fred and hermione and dangerous said finally. Uncle vernon opened wide open. As usual everyones celebrating all hed just have caught twice. Yehll get out. Well get the snitch yet no way up his feet. 35

36 Markov language models another example Here on this paper. Seven bottles three are poison two are wine one will get us through the corridors. Even better professor flitwick could i harry said the boy miserably. Well i mean its the girls through one of them hardly daring to take rons mind off his rocker said ron loudly. Shut up said ron sounding both shocked and impressed. Id have managed it before and you should have done so because a second later hermione granger telling a downright lie to a baked potato when professor quirrell too the nervous young man made his way along the floor in front of the year the dursleys were his only chance to beat you to get past fluffy i told her you didnt say another word on the wall wiping sweat off his nerves about tomorrow. Why should he yet harry couldnt help trusting him. This is goyle said the giant turning his back legs in anger. For the food faded from the chamber theyd just chained up. Oh no it isnt the same thing. Harry who was quite clear now and gathering speed. The air became colder and colder as they jostled their way through a crowd. 36