On entropy for mixtures of discrete and continuous variables

Transcription

1 On entropy for mxtures of dscrete and contnuous varables Chandra Nar Balaj Prabhakar Devavrat Shah Abstract Let X be a dscrete random varable wth support S and f : S S be a bjecton. hen t s wellknown that the entropy of X s the same as the entropy of f(x). hs entropy preservaton property has been well-utlzed to establsh non-trval propertes of dscrete stochastc processes, e.g. queung process []. Entropy as well as entropy preservaton s well-defned only n the context of purely dscrete or contnuous random varables. However for a mxture of dscrete and contnuous random varables, whch arse n many nterestng stuatons, the notons of entropy and entropy preservaton have not been well understood. In ths paper, we extend the noton of entropy n a natural manner for a mxed-par random varable, a par of random varables wth one dscrete and the other contnuous. Our extensons are consstent wth the exstng defntons of entropy n the sense that there exst natural njectons from dscrete or contnuous random varables nto mxed-par random varables such that ther entropy remans the same. hs extenson of entropy allows us to obtan suffcent condtons for entropy preservaton n mxtures of dscrete and contnuous random varables under bjectons. he extended defnton of entropy leads to an entropy rate for contnuous tme Markov chans. As an applcaton, we recover a known probablstc result related to Posson process. We strongly beleve that the frame-work developed n ths paper can be useful n establshng probablstc propertes of complex processes, such as load balancng systems, queung network, cachng algorthms. Index erms entropy, bjectons. INODUCION he noton of entropy for dscrete random varables as well as contnuous random varables s well defned. Entropy preservaton of dscrete random varable under bjecton map s an extremely useful property. For example, Prabhakar and Gallager [] used ths entropy preservaton property to obtan an alternate proof of the known result that Geometrc processes are fxed ponts under certan queung dscplnes. Author affatons: C. Nar s wth Mcrosoft esearch, edmond, WA. B. Prabhakar s wth the EE and CS departments at Stanford Unversty. D. Shah s wth the EECS department at MI. Emals: cnar@mcrosoft.com, balaj@stanford.edu, devavrat@mt.edu. In many nterestng stuatons, ncludng Example. gven below, the underlyng random varables are mxtures of dscrete and contnuous random varables. Such systems exhbt natural bjectve propertes whch allow one to obtan non-trval propertes of the system va entropy preservaton arguments. However, the man dffculty n establshng such arguments s the lack of noton of entropy for mxed random varables and approprate suffcent condtons for entropy preservaton. In ths paper, we wll extend the defnton of entropy for random varables whch are mxed par of dscrete and contnuous varables as well as obtan suffcent condtons for preservaton of entropy. Subsequently, we wll provde a rgorous justfcaton of mathematcal denttes that follow n the example below. Example. (Posson Splttng): Consder a Posson Process, P, of rate λ. Splt the Posson process nto two baby-processes P and P 2 as follows: for each pont of P, toss an ndependent con of bas p; f con turns up heads then the pont s assgned to P, else to P 2. It s well-known that P and P 2 are ndependent Posson processes wth rates λp and λ( p) respectvely. Entropy rate of a Posson process wth rate µ s known to be µ( log µ) bts per second. hat s, entropy rates of P, P, and P 2 are gven by λ( log λ), λp( log λp) and λ( p)( log λ( p)) respectvely. Further observe that the con of bas p s tossed at a rate λ and each con-toss has an entropy equal to p log p ( p) log( p) bts. It s clear that there s a bjecton between the tuple (P, con-toss process) and the tuple (P,P 2 ). Observe that the jont entropy rate of the two ndependent babyprocesses are gven by ther sum. hs leads to the followng obvous set of equaltes. H E (P, P 2 ) = H E (P ) + H E (P 2 ) = λp( log λp) + λ( p)( log λ( p)) = λ( log λ) + λ( p log p ( p) log( p)) = H E (P) + λ( p log p ( p) log( p)). (.) he last sum can be dentfed as sum of the entropy

2 rate of the orgnal Posson process and the entropy rate of the con tosses. However the presence of dfferental entropy as well as dscrete entropy prevents ths nterpretaton from beng rgorous. In ths paper, we shall provde rgorous justfcaton to the above equaltes. A. Organzaton he organzaton of the paper s as follows. In secton 2, we ntroduce the mxed-par of random varables and extend the defnton of entropy for the mxed-par of random varables. We wll also derve some propertes for the extended defnton of entropy that agree wth the propertes of entropy for the dscrete and contnuous random varables. In secton 3, we wll establsh suffcent condtons under whch bjectons preserve entropy for mxed-par random varables and random vectors. In secton 4, we wll defne the entropy rate of a contnuous tme Markov chan usng the defnton of entropy for mxed-par random varables. In the subsequent secton we use these defntons to recover an old result (Posson splttng). We conclude n secton DEFINIIONS AND SEUP hs secton provdes techncal defntons and sets up the frame-work for ths paper. Frst, we present some prelmnares. A. Prelmnares Consder a measure space (Ω, F, P), wth P beng a probablty measure. Let (, B ) denote the measurable space on wth the Borel σ-algebra. A random varable X s a measurable mappng from Ω to. Let µ X denote the nduced probablty measure on (, B ) by X. We call X as dscrete random varable f there s a countable subset {x, x 2,...} of that forms a support for the measure µ X. Let p = P(X = x ) and note that p =. he entropy of a dscrete random varable s defned by the sum H(X) = p log p. Note that ths entropy s non-negatve and has several well known propertes. One natural nterpretaton of ths number s n terms of the maxmum compressblty (n bts per symbol) of an..d. sequence of the random varables, X (cf. Shannon s data compresson theorem [2]). A random varable Y, defned on (Ω, F, P), s sad to be a contnuous random varable f the probablty measure, µ Y, nduced on (, B ) s absolutely contnuous wth respect to the Lebesgue measure. hese probablty measures can be characterzed by a non-negatve densty functon f(x) that satsfes f(x)dx =. he entropy (dfferental entropy) of a contnuous random varable s defned by the ntegral h(y ) = f(y) log f(y)dy. he entropy of a contnuous random varable s not non-negatve, though t satsfes several of the other propertes of the dscrete entropy functon. Due to negatvty, dfferental entropy clearly does not have nterpretaton of maxmal compressblty. However, t does have the nterpretaton of beng the lmtng dfference between the maxmally compressed quantzaton of the random varable and an dentcal quantzaton of an ndependent U[0, ] random varable [3] as the quantzaton resoluton goes to zero. Hence the term dfferental entropy s usually preferred to entropy when descrbng ths number. B. Our Setup In ths paper, we are nterested n a set of random varables that ncorporate the aspects of both dscrete and contnuous random varables. Let Z = (X, Y ) be a measurable mappng from the space (Ω, F, P) to the space (, B B ). Observe that ths mappng nduces a probablty measure µ Z on the space (, B B ) as well as two probablty measures µ X and µ Y on (, B ) obtaned va the projecton of the measure µ Z. Defnton 2. (Mxed-Par): Consder a random varables Z = (X, Y ). We call Z a mxed-par f X s a dscrete random varable whle Y s a contnuous random varable. hat s, the support of µ Z s on the product space S, wth S = {x, x 2,...} s a countable subset of. hat s S forms a support for µ X whle µ Y s absolutely contnuous wth respect to the Lebesgue measure. Observe that Z = (X, Y ) nduces measures {µ, µ 2,...} that are absolutely contnuous wth respect to the Lebesgue measure, where µ (A) = P(X = x, Y A), for every A B. Assocated wth these measures µ, there are non-negatve densty functons U[0,] represents a random varable that s unformly dstrbuted on the nterval [0,] For the rest of the paper we shall adopt the notaton that random varables X represent dscrete random varables, Y represent contnuous random varables and Z represent mxed-par of random varables. 2

3 g (y) that satsfy g (y)dy =. Let us defne p = g (y)dy. Observe that p s are non-negatve numbers that satsfy p = and corresponds to the probablty measure µ X. Further g(y) = g (y) corresponds to the probablty measure µ Y. Let g (y) = p g (y) be the probablty densty functon of Y condtoned on X = x. he followng non-negatve sequence s well defned for every y for whch g(y) > 0, p (y) = g (y) g(y),. Now g(y) s fnte except possbly on a set, A, of measure zero. For y A c, we have that p (y) = ; p (y) corresponds to the probablty that X = x condtoned on Y = y. It follows from defntons of p and p (y) that p = p (y)g(y)dy. Defnton 2.2 (Good Mxed-Par ): A mxed-par random varable Z = (X, Y ) s called good f the followng condton s satsfed: g (y) log g (y) dy <. (2.) Essentally, the good mxed-par random varables possess the property that when restrcted to any of the X values, the condtonal dfferental entropy of Y s well-defned. he followng lemma provdes a smple suffcent condtons for ensurng that a mxedpar varable s good. Lemma 2.3: he followng condtons are suffcent for a mxed-par random varable to be a good par: (a) andom varable Y possess a fnte ɛ th moment for some ɛ > 0,.e. M ɛ = y ɛ g(y)dy <. (b) here exsts δ > 0 such that g(y) satsfes g(y) +δ dy <. (c) he dscrete random varable X has fnte entropy,.e. p log p <. Proof: he proof s presented n the appendx. Defnton 2.4 (Entropy of a mxed-par): he entropy of a good mxed-par random varable s defned by H(Z) = g (y) log g (y)dy. (2.2) Defnton 2.5 (Vector of Mxed-Pars): Consder a random vector (Z,..., Z d ) = {(X, Y ),..., (X d, Y d )}. We call (Z,..., Z d ) a vector of mxed-pars f the support of µ (Z,...,Z d ) s on the product space S d d, where S d d s a countable set. hat s, S d forms the support for the probablty measure µ (X,..,X d ) whle the measure µ (Y,..,Y d ) s absolutely contnuous wth respect to the Lebesgue measure on d. Defnton 2.6 (Good Mxed-Par Vector ): A vector of mxed-par random varables (Z,..., Z d ) s called good f the followng condton s satsfed: g x (y) log g x (y) dy <, (2.3) y d x S d where g x (y) s the densty of the contnuous random vector Y d condtoned on the event that X d = x. Analogous to Lemma 2.3, the followng condtons guarantee that a vector of mxed-par random varables s good. Lemma 2.7: he followng condtons are suffcent for a mxed-par random varable to be a good par: (a) andom varable Y d possess a fnte ɛ th moment for some ɛ > 0,.e. M ɛ = y ɛ g(y)dy <. d (b) here exsts δ > 0 such that g(y) satsfes d g(y) +δ dy <. (c) he dscrete random varable X d has fnte entropy,.e. x S d p x log p x <. Proof: he proof s smlar to that of Lemma 2.3 and s omtted. In rest of the paper, all mxed-par varables and vectors are assumed to be good,.e. assumed to satsfy the condton (2.). Defnton 2.8 (Entropy of a mxed-par vector): he entropy of a good mxed-par vector of random varables s defned by H(Z) = x S d d g x (y) log g x (y)dy. (2.4) 3

4 Defnton 2.9 (Condtonal entropy): Gven a par of random varables (Z, Z 2 ), the condtonal entropy s defned as follows H(Z Z 2 ) = H(Z, Z 2 ) H(Z 2 ). It s not hard to see that H(Z Z 2 ) evaluates to g x,x 2 (y, y 2 ) log g x,x 2 (y, y 2 ) dy dy 2. g 2 x2 (y 2 ) x,x 2 Defnton 2.0 (Mutual Informaton): Gven a par of random varables (Z, Z 2 ), the mutual nformaton s defned as follows I(Z ; Z 2 ) = H(Z ) + H(Z 2 ) H(Z, Z 2 ). he mutual nformaton evaluates to g x,x 2 (y, y 2 ) log g x,x 2 (y, y 2 ) g 2 x (y )g x2 (y 2 ) dy dy 2. x,x 2 Usng the fact that + log x < x for x > 0 t can be shown that I(Z ; Z 2 ) s non-negatve. C. Old Defntons Stll Work We wll now present njectons from the space of dscrete (or contnuous) random varables nto the space of mxed-par random varable so that the entropy of the mxed-par random varable s the same as the dscrete (or contnuous) entropy. Injecton: Dscrete nto Mxed-Par: Let X be a dscrete random varable wth fnte entropy. Let {p, p 2,...} denote the probablty measure assocated wth X. Consder the mappng σ d : X Z (X, U) where U s an ndependent contnuous random varable dstrbuted unformly on the nterval [0,]. For Z, we have g (y) = p for y [0, ]. herefore H(Z) = g (y) log g (y) dy = 0 p log p dy = p log p = H(X) <. herefore we see that H(Z) = H(X). Injecton: Contnuous nto Mxed-Par: Let Y be a contnuous random varable wth a densty functon g(y) that satsfes g(y) log g(y) dy <. Consder the mappng σ c : Y Z (X 0, Y ) where X 0 s the constant random varable, say P(X 0 = ) =. Observe that g(y) = g (y) and that the par Z (X 0, Y ) s a good mxed-par that satsfes H(Z) = h(y ). hus σ d and σ c are njectons from the space of contnuous and dscrete random varables nto the space of good mxed-pars that preserve the entropy functon. D. Dscrete-Contnuous Varable as Mxed-Par Consder a random varable V whose support s combnaton of both dscrete and contnuous. hat s, t satsfes the followng propertes: () here s a countable set (possbly fnte) S = {x, x 2,...} such that µ V (x ) = p > 0; () measure µ V wth an assocated non-negatve functon g(y) (absolutely contnuous w.r.t. the Lebesgue measure), and () the followng holds: g(y) dy + p =. hus, the random varable V ether takes dscrete values x, x 2,... wth probabltes p, p 2,... or else t s dstrbuted accordng to the densty functon p g(y); where p = p. Observe that V has nether a countable support nor s ts measure absolutely contnuous wth respect to Lebesgue measure. herefore, though such random varables are encountered nether the dscrete entropy nor the contnuous entropy s approprate. o overcome ths dffculty, we wll treat such varables as mxed-par varables by approprate njecton of such varables nto mxed-par varables. Subsequently, we wll be able to use the defnton of entropy for mxed-par varables. Injecton: Dscrete-Contnuous nto Mxed-Par: Let V be a dscrete-contnuous varable as consdered above. Let the followng two condtons be satsfed: p log p < and g(y) log g(y) dy <. Consder the mappng σ m : V Z (X, Y ) descrbed as follows: When V takes a dscrete value x, t s mapped on to the par (x, u ) where u s chosen ndependently and unformly at random n [0, ]. When V does not take a dscrete value and say takes value y, t gets mapped to the par (x 0, y) where x 0 x,. One can thnk of x 0 as an ndcator value that V takes when t s not dscrete. he mxed-par varable Z has ts assocated functons {g 0 (y), g (y),...} where g (y) = Normally such random varables are referred to as mxed random varables. 4

5 p, y [0, ], and g 0 (y) = g(y). he entropy of Z as defned earler s H(Z) = g (y) log g (y) dy = p log p g(y) log g(y) dy. emark 2.: In the rest of the paper we wll treat every random varable that s encountered as a mxedpar random varable. hat s, a dscrete varable or a contnuous varable would be assumed to be njected nto the space of mxed-pars usng the map σ d or σ c, respectvely. 3. BIJECIONS AND ENOPY PESEVAION In ths secton we wll consder bjectons between mxed-par random varables and establsh suffcent condtons under whch the entropy s preserved. We frst consder the case of mxed-par random varables and then extend ths to vectors of mxed-par random varables. A. Bjectons between Mxed-Pars Consder mxed-par random varables Z (X, Y ) and Z 2 (X 2, Y 2 ). Specfcally, let S = {x } and S 2 = {x 2j } be the countable (possbly fnte) supports of the dscrete measures µ X and µ X2 such that µ X (x ) > 0 and µ X2 (x 2j ) > 0 for all S and j S 2. herefore a bjecton between mxed-par varables Z and Z 2 can be vewed as bjectons between S and S 2. Let F : S S 2 be a bjecton. Gven Z, ths bjecton nduces a mxed-par random varable Z 2. We restrct our attenton to the case when F s contnuous and dfferentable. Let the nduced projectons be F d : S S 2 and F c : S. Let the assocated projectons of the nverse map F : S 2 S be : S 2 respectvely. As before, let {g (y )}, {h j (y 2 )} denote the nonnegatve densty functons assocated wth the mxed-par random varables Z and Z 2 respectvely. Let (x 2j, y 2 ) = F (x, y ),.e. x 2j = F d (x, y ) and y 2 = F c (x, y ). Now, consder a small neghborhood x [y, y + dy ) of (x, y ). From the contnuty of F, for small enough dy, the neghborhood x [y, y + dy ) s mapped to some small neghborhood of (x 2j, y 2 ), say F d : S 2 S and F c he contnuty of mappng between two copes of product space S essentally means that the mappng s contnuous wth respect to rght (or Y ) co-ordnate for fxed x S. Smlarly, dfferentablty essentally means dfferentablty wth respect to Y co-ordnate. x 2j [y 2, y 2 + dy 2 ). he measure of x [y, y + dy ) s g (y ) dy, whle measure of x 2j [y 2, y 2 + dy 2 ) s h j (y 2 ) dy 2. Snce dstrbuton of Z 2 s nduced by the bjecton from Z, we obtan g (y ) dy dy 2 = h j(y 2 ). (3.) Further from y 2 = F c (x, y ) we also have, dy 2 = df c(x, y ). (3.2) dy dy hese mmedately mply a suffcent condton under whch bjectons between mxed-par random varables mply that ther entropes are preserved. Lemma 3.: If dfc(x,y) dy = for all ponts (x, y ) S, then H(Z ) = H(Z 2 ). Proof: hs essentally follows from the change of varables and repeated use of Fubn s theorem (to nterchange the sums and the ntegral). o apply Fubn s theorem, we use the assumpton that mxed-par random varables are good. Observe that, H(Z ) = g (y ) log g (y )dy (a) = j (b) = j = H(Z 2 ). ( h j (y 2 ) log h j (y 2 ) h j (y 2 ) log h j (y 2 )dy 2 df c (x, y ) dy ) dy 2 (3.3) Here (a) s obtaned by repeated use of Fubn s theorem along wth (3.) and (b) follows from the assumpton of the Lemma that df c(x,y ] dy =. B. Some Examples In ths secton, we present some examples to llustrate our defntons, setup and the entropy preservaton Lemma. Example 3.2: Let Y be a contnuous random varable that s unformly dstrbuted n the nterval [0, 2]. Let X 2 be the dscrete random varable that takes value 0 when Y [0, ] and otherwse. Let Y 2 = Y X 2. Clearly Y 2 [0, ], s unformly dstrbuted and ndependent of X 2. Let Z (X, Y ) be the natural njecton, σ c of Y (.e. X s just the constant random varable.). Observe that the bjecton between Z to the par Z 2 (X 2, Y 2 ) that satsfes condtons of Lemma 3. and mples log 2 = H(Z ) = H(Z 2 ). 5

6 However, also observe that by pluggng n the varous defntons of entropy n the approprate spaces, H(Z 2 ) = H(X 2, Y 2 ) = H(X 2 ) + h(y 2 ) = log 2 + 0, where the frst term s the dscrete entropy and the second term s the contnuous entropy. In general t s not dffcult to see that the two defntons of entropy (for dscrete and contnuous random varables) are compatble wth each other f the random varables themselves are thought of as a mxed-par. Example 3.3: hs example demonstrates that some care must be taken when consderng dscrete and contnuous varables as mxed-par random varables. Consder the followng contnuous random varable Y that s unformly dstrbuted n the nterval [0, 2]. Now, consder the mxed random varable V 2 that takes the value 2 wth probablty 2 and takes a value unformly dstrbuted n the nterval [0, ] wth probablty 2. Clearly, there s a mappng that allows us to create V 2 from Y by just mappng Y (, 2] to the value V 2 = 2 and by settng Y = V 2 when Y [0, ]. However, gven V 2 = 2 we are not able to reconstruct Y exactly. herefore, ntutvely one expects that H(Y ) > H(V 2 ). However, f you use the respectve njectons, say Y Z and V 2 Z 2, to the space of mxed-pars of random varables, we can see that H(Y ) = H(Z ) = log 2 = H(Z 2 ). hs shows that f we thnk of H(Z 2 ) as the entropy of the mxed random varable V 2 we get an ntutvely paradoxcal result where H(Y ) = H(V 2 ) where n realty one would expect H(Y ) > H(V 2 ). he careful reader wll be quck to pont out that the njecton from V 2 to Z 2 ntroduces a new contnuous varable, Y 22, assocated wth the dscrete value of 2, as well as a dscrete value x 0 assocated wth the contnuous part of V 2. Indeed the new random varable Y 22 allows us to precsely reconstruct Y from Z 2 and thus complete the nverse mappng of the bjecton. emark 3.4: he examples show that when one has mappngs nvolvng varous types of random varables and one wshes to use bjectons to compare ther entropes; one can perform ths comparson as long as the random varables are thought of as mxed-pars. C. Vector of Mxed-Par andom Varables Now, we derve suffcent condtons for entropy preservaton under bjecton between vectors of mxedpar varables. o ths end, let Z = (Z,..., Zd ) and Z 2 = (Z, 2..., Zd 2 ) be two vectors of mxed-par random varables wth ther support on S d and S 2 d respectvely. (Here S, S 2 are countable subsets of d.) Let F : S d S 2 d be a contnuous and dfferentable bjecton that nduces Z 2 by ts applcaton on Z. As before, let the projectons of F be F d : S d S 2 and F c : S d d. We consder stuaton where F c s dfferentable. Let g (y), y d for x S and h j (y), y d for w j S 2 be densty functons as defned before. Let (x, y ) S d be mapped to (w j, y 2 ) S 2 d. hen, consder d d Jacoban ] [ y J(x, y 2 ) k y l, k,l d where we have used notaton y = (y,..., yd ) and y 2 = (y, 2..., yd 2 ). Now, smlar to Lemma 3. we obtan the followng entropy preservaton for bjecton between vector of mxed-par random varables. Lemma 3.5: If for all (x, y ) S d, det(j(x, y )) =, then H(Z ) = H(Z 2 ). Here det(j) denotes the determnant of matrx J. Proof: he man ngredents for the proof of Lemma 3. for the scalar case were the equaltes (3.) and (3.2). For a vector of mxed-par varable we wll obtan the followng equvalent equaltes: For change of dy at (x, y ), let dy 2 be nduced change at (x j, y 2 ). Let vol(dy) denote the volume of d dmensonal rectangular regon wth sdes gven by components of dy n d. hen, Further, at (x, y ), g (y )vol(dy ) = h j (y 2 )vol(dy 2 ). (3.4) vol(dy 2 ) = det(j(x, y )) vol(dy ). (3.5) Usng exactly the same argument that s used n (3.3) (replacng dy k by vol(dy k ), k =, 2), we obtan the desred result. hs completes the proof of Lemma ENOPY AE OF CONINUOUS IME MAKOV CHAINS A contnuous tme Markov chan s composed of the pont process that characterzes the tme of transtons of the states as well as the dscrete states between whch the transton happens. Specfcally, let x denote the tme of th transton or jump wth Z. Let V S denote the state of the Markov chan after the jump at tme x, where S be some countable state space. For smplcty, we assume S = N. Let transton probabltes be p kl = P(V = l V = k), k, l N for all. 6

7 We recall that the entropy rate of a pont process P was defned n secton 3.5 of [4] accordng to the followng: Observaton of process conveys nformaton of two knds: the actual number of ponts observed and the locaton of these ponts gven ther number. hs led them to defne the entropy of a realzaton {x,..., x N } as H(N) + H(x,..., x N N) he entropy rate of the pont process P s defned as follows: let N( ) be the number of ponts arrved n tme nterval (0, ] and the nstances be x( ) = (x,..., x N( ) ). hen, the entropy rate of the process s H E (P) = lm [H(N( )) + H(x( ) N( ))], f the above lmt exsts. We extend the above defnton to the case of Markov chan n a natural fashon. Observaton of a contnuous tme Markov chan over a tme nterval (0, ] conveys nformaton of three types: the number of ponts/jumps of the chan n the nterval, the locaton of the ponts gven the number as well as the value of the chan after each jump. reatng each random varable as a mxedpar allows us to consder all the random varables n a sngle vector. As before, let N( ) denote the number of ponts n an nterval (0, ]. Let x( ) = (x,..., x N( ) ), V( ) = (V 0, V,..., V N( ) ) denote the locatons of the jumps as well as the values of the chan after the jumps. hs leads us to defne the entropy of the process durng the nterval (0, ] as H (0, ] = H(N( ), V( ), x( )). (4.) Observe that the (N( ), V( ), x( )) s a random vector of mxed-par varables. For a sngle state Markov chan the above entropy s the same as that of the pont process determne the jump/transton tmes. Smlar to the development for pont processes, we defne the entropy rate of the Markov chan as H E = lm H (0, ], f t exsts. Proposton 4.: Consder a Markov chan wth underlyng Pont process beng Posson of rate λ, ts statonary dstrbuton beng π = (π()) wth transton probablty matrx P = [p j ]. hen, ts entropy rate s well-defned and H E = λ( log λ) + λh MC, where H MC = π() j p j log p j. Proof: For Markov Chan as descrbed n the statement of proposton, we wsh to establsh that H (0, ] lm = H E, as defned above. Now H (0, ] = H(x( ), N( ), V( )) = H(x( ), N( )) + H(V( ) N( ), x( )). Consder the term on the rght hand sde of the above equalty. hs corresponds to the ponts of a Posson process of rate λ. It s well-known (cf. equaton (3.5.0), pg. 565 [4]) that lm H(x( ), N( )) = λ( log λ). (4.2) Now consder the term H(V( ) x( ), N( )). Snce V( ) s ndependent of x( ), we get from the defnton of condtonal entropy that H(V( ) x( ), N( )) = H(V( ) N( )). (4.3) One can evaluate H(V( ) N( )) as follows, H(V( ) N( )) = k p k H(V 0,..., V k ), where p k s the probablty that N( ) = k. he sequence of states V 0,..., V k can be thought of as sequence of states of a dscrete tme Markov chan wth transton matrx P. For a Markov chan, wth statonary dstrbuton π (.e. P π = π), t s well-known that lm k k H(V 0,..., V k ) = = H MC. π() j p j log p j hus, for any ɛ > 0, there exsts k(ɛ) large enough such that for k > k(ɛ) k H(V 0,..., V k ) H MC < ɛ. For large enough, usng tal-probablty estmates of Posson varable t can be shown that ( P (N( ) k(ɛ)) exp λ ). 8 Puttng these together, we obtan that for gven ɛ there exsts (ɛ) large enough such that for (ɛ) ( ) H(V( ) N( )) = H(V 0,..., V k ) kp k k k k k(ɛ) = kp k(h MC ± ɛ) + O(k(ɛ)) λ + O(k(ɛ)) = (H MC ± ɛ) = λh MC ± 2ɛ. 7

8 hat s H(V( ) N( )) lm = λh MC. Combnng (4.2), (4.3) and the above equaton we complete the proof of the Proposton 4.. Fact 4.2 (cf. Ch. 3.5 [4]): Consder the set of statonary ergodc pont processes wth mean rate λ. hen the entropy of ths collecton s maxmzed by a Posson Process wth rate λ. hat s, f P s a statonary ergodc pont process wth rate λ then H E (P) λ( log λ). Example 4.3: Consder the queue sze process of an M/M/ queue wth arrval rate λ a and servce rate λ s > λ a. he queue sze s a contnuous tme Markov chan. Snce the locaton of the jumps s a Posson process of rate λ a + λ s, one can see that the number of ponts, N[ ] n the nterval (0, ] satsfes the condton that N[ ] λ eff = lm = λ a + λ s a.s. he statonary dstrbuton of the brth-death Markov chan of the queues s gven by P(Q = 0) = λ a, λ s ( ) [ λa P(Q = ) = λ ] a,. λ s λ s (4.4) From here one can compute the entropy rate of the dscrete tme Markov chan of the queue-sze values to be λ a H MC = log λ a + λ s λ a λ a + λ s Puttng these together we obtan λ s λ a + λ s log H E = (λ a + λ s ) ( log(λ a + λ s )) [ + (λ a + λ s ) λ a λ a log λ a + λ s λ a + λ s λ ] s λ s log λ a + λ s λ a + λ s = λ a ( log λ a ) + λ s ( log λ s ). 5. APPLICAION A. Posson Splttng va Entropy Preservaton λ s λ a + λ s. (4.5) In ths secton, we use the suffcent condtons developed n Lemma 3.5 to obtan proof of the followng property. Lemma 5.: Consder a Posson process, P, of rate λ. Splt the process P nto two baby-processes P and P 2 as follows: for each pont of P, toss an ndependent con of bas p. Assgn the pont to P f con turns up head, else assgn t to P 2. hen, the baby-processes P and P 2 have the same entropy rate as Posson processes of rates λp and λ( p) respectvely. Proof: Consder a Posson Process, P, of rate λ n the nterval [0, ]. Let N( ) be the number of ponts n ths nterval and let a( ) = {a,..., a N( ) } be ther locatons. Further, let C( ) = {C,..., C N( ) } be the outcomes of the con-tosses and M( ) denote the number of heads among them. Denote r( ) = {,..., M( ) }, b( ) = {B,..., B N( ) M( ) } as the locatons of the baby-processes P, P 2 respectvely. It s easy to see that the followng bjecton holds: {a( ), C( ), N( ), M( )} {r( ), b( ), N( ) M( ), M( ).} (5.) Gven the outcomes of the con-tosses C( ), {r( ), b( )} s a permutaton of a( ). Hence, the Jacoban correspondng to any realzaton of {C( ), N( ), M( )} that maps a( ) to {r( ), b( )} s a permutaton matrx. It s well-known that the determnant of a permutaton matrx s ±. herefore, Lemma 3.5 mples that H(a( ), C( ), N( ), M( )) = H(r( ), b( ), N( ) M( ), M( )) H(b( ), N( ) M( )) + H(r( ), M( )). (5.2) M( ) s completely determned by C( ) and t s easy to deduce from the defntons that Hence H(M( ) a( ), C( ), N( )) = 0. H(a( ), C( ), N( ), M( )) = H(a( ), C( ), N( )) + H(M( ) a( ), C( ), N( )) = H(a( ), C( ), N( )). (5.3) Snce the outcome of the con-tosses along wth ther locatons form a contnuous tme Markov chan, usng Proposton 4. we can see that lm H(a( ), C( ), N( ), M( )) = lm H(a( ), C( ), N( )) = λ( log λ) λ(p log p + ( p) log( p)) = λp( log λp) + λ( p)( log λ( p)). (5.4) It s well known that P, P 2 are statonary ergodc processes of rates λp, λ( p) respectvely. Hence from 8

9 Fact 4.2 we have lm H(r( ), M( )) λp( log λp), lm H(b( ), N( ) M( )) λ( p)( log λ( p)). (5.5) Combnng equatons (5.2), (5.4), (5.5) we can obtan lm lm H(r( ), M( )) = λp( log λp), H(b( ), N( ) M( )) = λ( p)( log λ( p)). (5.6) hus, the entropy rates of processes P and P 2 are the same as that of Posson processes of rates λp and λ( p) respectvely. hs completes the proof of Lemma CONCLUSIONS hs paper deals wth notons of entropy for random varables that are mxed-par,.e. par of dscrete and contnuous random varables. Our defnton of entropy s a natural extenson of the known dscrete and dfferental entropy. Stuatons where both contnuous and dscrete varables arse are common n the analyss of randomzed algorthms that are often employed n networks of queues, load balancng systems, etc. We hope that the technques developed here wll be very useful for the analyss of such systems and for computng entropy rates for the processes encountered n these systems. A. Proof of Lemma 2.3 APPENDIX We wsh to establsh that the condtons of Lemma 2.3 guarantee that g (y) log g (y) dy <. (6.) Let (a) + = max(a, 0) and (a) = mn(a, 0) for a. hen, a = a + + a, and a = a + a. By defnton g (y) 0. Observe that log g (y) = 2(log g (y)) + log g (y). herefore to guarantee (6.) t suffces to show the followng two condtons: g (y)(log g (y)) + dy <, (6.2) g (y) log g (y)dy <. (6.3) he next two lemmas show that equatons (6.2) and (6.3) are satsfed and hence completes the proof of Lemma 2.3. Lemma 6.: Let Y be a contnuous random varable wth a densty functon g(y) such that for some δ > 0 g(y) +δ dy <. Further f g(y) can be wrtten as sum of non-negatve functons g (y), the g (y)(log g (y)) + dy <. Proof: For gven δ, there exsts fnte B δ > such that for x B δ, log x x δ. Usng ths, we obtan g (y)(log g (y)) + dy = g (y) log g (y)dy g (y) = g (y) log g (y)dy g (y)<b δ + g (y) log g (y)dy (6.4) B δ g (y) log B δ g (y)dy + g (y) +δ dy = p log B δ + g (y) +δ dy. herefore, g (y)(log g (y)) + dy ( p log B δ + (a) = log B δ + (b) log B δ + <. ) g (y) +δ dy g (y) +δ dy g(y) +δ dy In (a) we use the fact that g (y) s postve to nterchange the sum and the ntegral. In (b), we agan use the fact that g (y) 0 to bound g (y) +δ wth ( g (y)) +δ. Lemma 6.2: In addton to the hypothess of Y n Lemma 6. assume that Y has a fnte ɛ moment for some ɛ > 0. hen the followng holds: g (y) log g (y)dy <. 9

10 Proof: Let for some ɛ > 0, M ɛ = y ɛ g(y) dy <. Note that for any ɛ > 0, there s a constant C ɛ > 0, such that C ɛe y ɛ dy =. Further, observe that the densty g (y) = g (y)/p s absolutely contnuous w.r.t. the densty f(y)( = C ɛ e y ɛ ). hus from the fact that the Kullback-Lebler dstance D( g f) s non-negatve we have 0 g (y) log g (y) p f(y) dy g (y) = g (y) log dy p C ɛ e y ɛ = g (y) log g (y) dy p log p p log C ɛ + y ɛ g (y)dy. herefore g (y) log g (y) dy p log p + p log C ɛ + y ɛ g (y)dy. From (6.4) we have g (y) log g (y) dy g (y)(log g (y)) + dy p log B δ + g (y) +δ dy. (6.5) (6.6) Combnng equatons (6.5) and (6.6), we obtan g (y) log g (y) dy p log p + p log C ɛ + y ɛ g (y) (6.7) + p log B δ + g (y) +δ dy. Now usng the facts we obtan from (6.7) that g (y) log g (y)dy <. EFEENCES [] B. Prabhakar and. Gallager, Entropy and the tmng capacty of dscrete queues, IEEE rans. Info. heory, vol. I-49, pp , February, [2] C. E. Shannon, A mathermatcal theory of communcaton, Bell System echncal Journal, vol. 27, pp and , July and October, 948. [3]. Cover and J. homas, Elements of Informaton heory. Wley Interscence, 99. [4] D. J. Daley and D. Vere-Jones, An ntroducton to the theory of Pont Processes. Sprnger Verlag, 988. p log p <, y ɛ g (y)dy = y ɛ g(y)dy <, g (y) +δ dy < g(y) +δ dy <, 0