Entropy Rates of a Stochastic Process

Transcription

1 Etropy Rates of a Stochastic Process Best Achievable Data Compressio Radu Trîmbiţaş October Etropy Rates of a Stochastic Process Etropy rates The AEP states that H(X) bits suffice o the average for i.i.d. RVs What for depedet RVs? For statioary processes H(X 1, X 2,..., X ) grows (asymptotically) liearly with at a rate H(X ) the etropy rate of the process A stochastic process {X i } i I is a idexed sequece of radom variables, X i : S X is a RV i I If I N, {X 1, X 2,... } is a discrete stochastic process, called also a discrete iformatio source. A discrete stochastic process is characterized by the joit probability mass fuctio P((X 1, X 2,..., X ) = (x 1, x 2,..., x )) = p(x 1, x 2,..., x ) where (x 1, x 2,..., x ) X. 1.1 Markov chais Markov chais Defiitio 1. A stochastic process is said to be statioary if the joit distributio of ay subset of the sequece of radom variables is ivariat with respect to shifts i the time idex P(X 1 = x 1,..., X = x ) = P(X 1+l = x 1,..., X +l = x ) (1), l ad x 1, x 2,..., x X. 1

2 Defiitio 2. A discrete stochastic process {X 1, X 2,... } is said to be a Markov chai or Markov process if for = 1, 2,... P (X +1 = x +1 X = x, X 1 = x 1,..., X 1 = x 1 ) = P(X +1 = x +1 X = x ), x 1, x 2,..., x, x +1 X. (2) The joit pmf ca be writte as p(x 1, x 2,..., x ) = p(x 1 )p(x 2 x 1 )... p(x x 1 ). (3) Defiitio 3. A Markov chai is said to be time ivariat (time homogeeous) if the coditioal probability p(x +1 x ) does ot deped o ; that is for = 1, 2,... P (X +1 = b X = a) = P(X 2 = b X 1 = a), a, b X. (4) This property is assumed uless otherwise stated. {X i } Markov chai, X is called the state at time A time-ivariat Markov chai is characterized by its iitial state ad a probability trasitio matrix P = [P ij ], i, j = 1,..., m, where P ij = P(X +1 = j X = i). The Markov chai {X } is irreducible if it is possible to go from ay state to aother with a probability > 0 The Markov chai {X } is aperiodic if state a, the possible times to go from a to a have highest commo factor = 1. Markov chais are ofte described by a directed graph where the edges are labeled by the probability of goig from oe state to aother. p(x ) - pmf of the radom variable at time p(x +1 ) = x p(x )P x x +1 (5) A distributio o the states such that the distributio at time + 1 is the same as the distributio at time is called a statioary distributio - so called because if the iitial state of a Markov chai is draw accordig to a statioary distributio, the Markov chai form a statioary process. If the fiite-state Markov chai is irreducible ad periodic, the statioary distributio is uique, ad from ay startig distributio, the distributio of X teds to a statioary distributio as. Example 4. Cosider a two-state Markov chai with a probability trasitio matrix [ ] 1 α α P = β 1 β (Figure 1) 2

3 Figure 1: Two-state Markov chai The statioary probability is the solutio of µp = µ or (I P T )µ T = 0. We add the coditio µ 1 + µ 2 = 0. The solutio is µ 1 = β α + β, µ 2 = α α + β. Click here for a Maple solutio Markovex1.html. The etropy of X is ( ) β H(X ) = H α + β, α. α + β 1.2 Etropy rate Etropy rate Defiitio 5. The etropy rate of a stochastic process {X i } is defied by whe the limit exists. Examples 1 H(X ) = lim H (X 1,..., X ) (6) 1. Typewriter - m equally likely output letters; he(she) ca produce m sequeces of legth, all of them equally likely. H (X 1,..., X ) = log m, ad the etropy rate is H(X ) = log m bits per symbol. 2. X 1, X 2,... i.i.d. RVs H (X H(X ) = lim 1,..., X ) H(X = lim 1 ) = H (X 1 ). 3. X 1, X 2,... idepedet, but ot idetically distributed RVs H (X 1,..., X ) = It is possible that 1 H(x i) does ot exists H(X i ) 3

4 Defiitio 6. H (X ) = lim H (X X 1, X 2,... X 1 ). (7) H(X ) is etropy per symbol of the RVs; H (X ) is the coditioal etropy of the last RV give the past. For statioary processes both limits exist ad are equal. Lemma 7. For a statioary stochastic process, H(X X 1,..., X 1 ) is oicreasig i ad has a limit H (X ). Proof. H(X +1 X 1, X 2,..., X ) H(X +1 X,..., X 2 ) coditioig = H(X X 1,..., X 1 ). statioarity (H(X X 1,..., X 1 )) is decreasig ad oegative, so it has a limit H (X ). Lemma 8 (Cesáro). If a a ad b = 1 a i the b a. Theorem 9. For a statioary stochastic process H(X ) (give by (6)) ad H (X ) (give by (7)) exist ad H(X ) = H (X ). (8) Proof. By the chai rule, But, H (X 1,..., X ) = 1 H (X H(X ) = lim 1,..., X ) 1 = lim H(X i X i 1,..., X 1 ) H(X i X i 1,..., X 1 ). = lim H(X X 1,..., X 1 ) (Lemma 8) = H (X ) (Lemma 7) 1.3 Etropy rate for Markov chai Etropy rate for Markov chai For a statioary Markov chai, the etropy rate is give by H (X ) = H (X ) = lim H (X X 1,..., X 1 ) = lim H (X X 1 ) = H(X 2 X 1 ), (9) where the coditioal etropy is calculated usig the give statioary distributio. 4

5 The statioary distributio µ is the solutio of the equatios Expressio of coditioal etropy: µ j = µ i P ij, j. i Theorem 10. {X i } statioary Markov chai with statioary distributio µ ad trasitio matrix P. Let X 1 µ. the the etropy rate is H(X ) = i Proof. H(X ) = H(X 2 X 1 ) = i µ i ( j P ij log P ij ). µ i P ij log P ij. (10) j Example 11 (Two-state Markov chai). The etropy rate of the two-state Markov chai i Figure 1 is H(X ) = H(X 2 X 1 ) = β α + β H(α) + α α + β H(β). Remark. If the Markov chai is irreducible ad aperiodic, it has a uique statioary distributio o the states, ad ay iitial distributio teds to the statioary distributio as. I this case, eve though the iitial distributio is ot the statioary distributio, the etropy rate, which is defied i terms of log-term behavior, is H(X ), as defied i (9) ad (10). 1.4 Fuctios of Markov chais Fuctios of Markov chais X 1, X 2,..., X,... statioary Markov chai, Y i = φ(x i ), H(Y) =? i may cases Y 1, Y 2,..., Y,... is ot a Markov chai, but it is statioary lower boud Lemma 12. Proof. For k = 1, 2,... H(Y Y 1,..., Y 2, X 1 ) H(Y). (11) H(Y Y 1,..., Y 2, X 1 ) (a) = H(Y Y 1,..., Y 2, Y 1, X 1 ) (b) = H(Y Y 1,..., Y 2, Y 1, X 1, X 0, X 1,..., X k ) (c) = H (Y Y 1,..., Y 2, Y 1, X 1, X 0, X 1,..., X k, Y 0,..., Y k ) (d) H (Y Y 1,..., Y 2, Y 1, Y 0,..., Y k ) (e) = H (Y +k+1 Y +k,..., Y 1 ), 5

6 (a) follows from the fact that Y 1 = φ(x 1 ), (b) from the Markovity, (c) from Y i = φ(x i ), (d) coditioig reduces etropy, (e) statioarity. Proof - cotiuatio. Sice iequality is true for all k, i the limit H(Y Y 1,..., Y 2, X 1 ) lim H (Y +k+1 Y +k,..., Y 1 ) k = H(Y). Lemma 13. H(Y Y 1,..., Y 2, X 1 ) H(Y Y 1,..., Y 2, Y 1, X 1 ) 0. (12) Proof. Expressio of iterval legth: H(Y Y 1,..., Y 2, X 1 ) H(Y Y 1,..., Y 2, Y 1, X 1 ) = I(X 1 ; Y Y 1,..., Y 1 ). By properties of mutual iformatio, I(X 1 ; Y 1,..., Y ) H(X 1 ), ad I(X 1 ; Y 1,..., Y ) icreases with. Thus, lim I(X 1 ; Y 1,..., Y ) exists ad lim I(X 1; Y 1,..., Y ) H(X 1 ). Proof - cotiuatio. By the chai rule H(X 1 ) lim I(X 1 ; Y 1,..., Y ) = lim = I(X 1 ; Y i Y i 1,..., Y 1 ) I(X 1 ; Y i Y i 1,..., Y 1 ) The geeral term of the series must ted to 0 lim I (X 1 ; Y Y 1,..., Y 1 ) = 0. The last two lemmas imply Theorem 14. X 1, X 2,..., X,... statioary Markov chai, Y i = φ(x i ) H(Y Y 1,..., Y 1, X 1 ) H(Y) H(Y Y 1,..., Y 1 ) (13) ad lim H(Y Y 1,..., Y 1, X 1 ) = H(Y) = lim H(Y Y 1,..., Y 1 ) (14) 6

7 Hide Markov models We could cosider Y i to be a stochastic fuctio of X i X 1, X 2,..., X,... statioary Markov chai, Y 1, Y 2,..., Y,... a ew process where Y i is draw accordig to p(y i x i ), coditioally idepedet of all the other X j, j = i p(x, y 1 ) = p(x 1 ) p(x i+1 x i ) p(y i x i ). Y 1, Y 2,..., Y,... is called a hidde Markov model (HMM) Applied to speech recogitio, hadwritig recogitio, ad so o. The same argumet as for fuctios of Markov chai works for HMMs. Refereces Refereces [1] Thomas M. Cover, Joy A. Thomas, Elemets of Iformatio Theory, 2d editio, Wiley, [2] David J.C. MacKay, Iformatio Theory, Iferece, ad Learig Algorithms, Cambridge Uiversity Press, [3] Robert M. Gray, Etropy ad Iformatio Theory, Spriger,