The Common Information for N Dependent Random Variables

Transcription

1 Syracuse Uiversity SURFACE Electrical Egieerig ad Computer Sciece Techical Reports College of Egieerig ad Computer Sciece The Commo Iformatio for N Depedet Radom Variables Wei Liu Syracuse Uiversity Ge Xu Syracuse Uiversity Follow this ad additioal works at: Part of the Computer Scieces Commos Recommeded Citatio Liu, Wei ad Xu, Ge, "The Commo Iformatio for N Depedet Radom Variables" (20). Electrical Egieerig ad Computer Sciece Techical Reports. Paper This Report is brought to you for free ad ope access by the College of Egieerig ad Computer Sciece at SURFACE. It has bee accepted for iclusio i Electrical Egieerig ad Computer Sciece Techical Reports by a authorized admiistrator of SURFACE. For more iformatio, please cotact surface@syr.edu.

2 SYR-EECS April, 20 The Commo Iformatio for N Depedet Radom Variables Wei Liu Ge Xu Biao Che wliu28@syr.edu gexu@syr.edu biche@syr.edu ABSTRACT: This paper geeralizes Wyer s defiitio of commo iformatio of a pair or radom variables to that of N radom variables. We prove codig theorems that show the same operatioal meaigs for the commo iformatio of two radom variables geeralize to that of N radom variables. As a byproduct of our proof, we show that the Gray-Wyer source codig etwork ca be geeralized to N source sequeces with N decoders. We also establish a mootoe property of Wyer s commo iformatio which is i cotrast to other otios of the commo iformatio, specifically Shao s mutual iformatio ad Gács ad Körer s commo radomess. Examples about the computatio of Wyer s commo iformatio of N radom variables are also give. KEYWORDS: Wyer's commo iformatio, Gray-Wyer source codig etwork, distributio approximatio, circularly symmetric biary source Syracuse Uiversity - Departmet of EECS, CST, Syracuse, NY 3244 (P) (F)

3 The Commo Iformatio of N Depedet Radom Variables Wei Liu Departmet of EECS Syracuse Uiversity wliu28@syr.edu Ge Xu Departmet of EECS Syracuse Uiversity gexu@syr.edu Biao Che Departmet of EECS Syracuse Uiversity biche@syr.edu Abstract This paper geeralizes Wyer s defiitio of commo iformatio of a pair of radom variables to that of N radom variables. We prove codig theorems that show the same operatioal meaigs for the commo iformatio of two radom variables geeralize to that of N radom variables. As a byproduct of our proof, we show that the Gray-Wyer source codig etwork ca be geeralized to N source squeces with N decoders. We also establish a mootoe property of Wyer s commo iformatio which is i cotrast to other otios of the commo iformatio, specifically Shao s mutual iformatio ad Gács ad Körer s commo radomess. Examples about the computatio of Wyer s commo iformatio of N radom variables are also give. I. INTRODUCTION Cosider a pair of depedet radom variables X ad Y with joit distributio P(x, y). Characterizig the commo iformatio betwee X ad Y has bee a topic of research iterest i the past decades [] [5]. There have bee three classical otios reported i the literature. Shao s [6] mutual iformatio I(X; Y ) Shao s mutual iformatio measures how much ucertaity ca be reduced with respect to oe radom variable by observatio the other radom variable. I the case that X ad Y are idepedet, mutual iformatio I(X; Y ) = 0, idicatig that observig oe variable X does ot give ay iformatio about Y ad vice versa. Shao s mutual iformatio carries operatioal meaigs that are istrumetal i layig the foudatio for iformatio theory. Gács ad Körer s [] commo radomess K(X, Y ) Cosider a pair of idepedet ad idetically distributed radom sequeces X, Y with each pair (X i, Y i ) P(x, y). These two sequeces are observed respectively by two odes, which attempt to map the sequeces oto a commo message set W. Specifically, let f ad g be such mappigs, i.e., f : X W, g : Y W. Defie ǫ = Pr(W W 2 ) where W = f (X ) ad W 2 = g (Y ). Gács ad Körer s commo radomess is defied as K(X, Y ) = lim sup,ǫ 0 H(W ). Gács ad Körer s commo radomess has foud extesive applicatios i cryptography, i.e., for key geeratio [7] [9]. O the other had, the commo radomess otio is rather restrictive as it equals 0 i most cases except for the followig special case (or radom variable pairs that ca be coverted to such distributios through relabelig of realizatios, i.e., permutatio of joit distributio matrix). Let X ad Y be X = (X, V ) ad Y = (Y, V ), respectively, where X, Y, V are idepedet. Clearly, the commo part betwee X ad Y is V ad it follows that K(X; Y ) = H(V ). Note that for this example I(X; Y ) = K(X; Y ) = H(V ). Wyer s [4] commo iformatio C(X, Y ) Wyer s commo iformatio is defied as C(X, Y ) = mi I(XY ; W). () X W Y Thus the hidde (or auxiliary) variable W iduces a Markov chai X W Y, or, equivaletly, a coditioal idepedece structure of X, Y beig idepedet give W. Wyer gave two operatioal meaigs for the above defiitio. The first approach is show i Fig.. The ecoder observes a pair of sequeces (X, Y ), ad map them to three messages W 0, W, W 2, takig values i alphabets of respective sizes 2 R0, 2 R, 2 R2. Decoder, upo receivig (W 0, W ), eeds to reproduce X reliably while decoder 2, upo receivig (W 0, W 2 ), eeds to reproduce Y reliably. Let C be the ifimum of all admissible R 0 for the system i Fig. such that the total rate R 0 + R + R 2 H(X, Y ). The secod approach is show i Fig. 2. A commo iput W, uiformly distributed o W = {,, 2 R0 } is give to two separate processors which are otherwise idepedet of each other. These processors (radom variable geerators) geeratig idepedet ad idetically distributed sequeces accordig to q (X W) ad q 2 (Y W) respectively. The output sequeces of the two processors are deoted by X ad Ỹ respectively. Thus the joit distributio of the output sequeces is, Q( X, Ỹ ) = w W W q (X W)q 2 (Y W). (2) Defie C 2 of (X, Y ) to be ifimum of rate R 0 for the commo iput such that q( X, Ỹ ) close to p(x, Y ), where the closeess is defied usig the average divergece of the two distributios D (P, Q) = P(x, y )log P(x, y ) Q(x, y ). (3) x X,y Y Wyer proved that It was observed i [4] that C = C 2 = C(X, Y ). (4) K(X, Y ) I(X; Y ) C(X, Y ). (5)

4 X, Y Ecoder W W 0 W 2 Decoder Decoder 2 ˆX Ŷ commo iformatio is to defie a similar measure for N radom variables by preservig the coditioal idepedece structure through the itroductio of a auxiliary radom variable. Specifically, we defie C(X, X 2,, X N ) if I(X, X 2,, X N ; W), (6) W Fig.. Fig. 2. Source codig over a simple etwork. Processor Processor 2 Radom variable geerators. X Ỹ Wyer [4] ad Witsehause [5] also provide several examples o how to calculate the commo iformatio C(X, Y ). For the example of X = (X, V ) ad Y = (Y, V ) with (X, Y, V ) mutually idepedet, C(X, Y ) = I(X; Y ) = K(X, Y ) = H(V ). Geeralizig of mutual iformatio to N radom variables was first reported i [0]. The geeralizatio comes from the observatio that for a pair of radom variables, Shao s iformatio measures is cosistet with the Ve diagram for set operatio ad a comprehesive treatmet was available i [], [2]. Gács ad Körer s commo radomess was recetly geeralized to multiple radom variables by Tyagi, Naraya ad Gupta i [3], which exteds the ecodig process i the defiitio of commo radomess to that of N termials. I this paper, we geeralize Wyer s commo iformatio of a pair of radom variables to that of N depedet variables. We show that the operatioal meaig defied i both approaches are still valid. Moreover, we establish some mootoe property of such geeralizatio which cotrast to the otio of commo iformatio. Specifically, we show that the commo iformatio does ot decrease as the umber of variables icreases while keepig the same margial distributio. This is differet from the other two otios of commo iformatio. Examples o evaluatig C(X, X 2,, X N ) are give for circularly symmetric biary sources ad the asymptotic results are also studied. The rest of this paper is orgaized as follows. Sectio II gives the problem formulatio ad mai results. Sectio III gives some examples ad discussios. Sectio IV cocludes the paper. II. PROBLEM STATEMENT AND MAIN RESULTS Let X, X 2,, X N be radom variables that take values o the fiite alphabet sets X, X 2,, X N with joit distributio P(x, x 2,, x N ). Our geeralizatio of Wyer s where the ifimum is take over all the joit distributios of (X, X 2,, X N, W) such that P(x, x 2,, x, w) = P(x, x 2,, x N ), (7) w P(x,..., x w) = P(x i w). (8) Thus the margial distributio of (X, X 2,, X N ) is P(x, x 2,, x N ) ad (X,, X N ) are coditioally idepedet give W. We ow give formal defiitios of C ad C 2 for N radom variables. Cosider N legth- idepedet ad idetically distributed source sequeces (x,x 2,, x N ) with (X i, X 2i,, X Ni ) p(x, x 2,, x N ), i.e., P () (x,x 2,, x N) = P(x i, x 2i,, x Ni ). (9) For the Gray-Wyer source codig etwork, we start with the defiitio of ecoder-decoders. Defiitio : A (, M 0, M,, M N ) code cosists of the followig: A ecoder mappig f : X X 2 X N M 0 M M N, where M i = {, 2,, 2 Ri }. N decoders g i, for i =, 2,, N, g i : M i M 0 X i. (0) The probability of error is defied as P () e = Pr{( ˆX ˆX 2 ˆX N) (X, X 2,, X N)}, () where ˆX i = g i (M i, M 0 ) for i =,, N. Defiitio 2: A umber R 0 is said to be achievable if for ay ǫ > 0, we ca fid a sufficietly large such that there exists a (, M 0, M,, M N ) code with M 0 2 R0 (2) P e () ǫ, (3) log M i H(X, X 2,, X N ) + ǫ. (4) i=0 As with the case for two radom variables, C is defied as the ifimum of all achievable R 0. For the secod approach of approximatig joit distributio, we agai start with the followig defiitio. Defiitio 3: A (, M, ) geerator cosists of the followig: a message set W {, 2,, 2 R };

5 for all w W ad N coditioal probability distributios q () i (x i w), for i =, 2,, N, defie the probability distributio o X X 2 X N M Q () (X, X2,, XN) = w W (5) Thus the N processors serve as radom umber geerators each geeratig idepedet ad idetically distributed (i.i.d.) sequece ˆX i accordig to q(x i w) ad the output of the processors follow joit distributio defied i (9). Let N q () i (x i w). = D (P () ; Q () ) = P () log P () Q, () x i X i,,2,,n (6) where P () ad Q () are defied as i (9) ad (5) respectively. Defiitio 4: A umber R is said to be achievable if for all ǫ > 0, we ca fid a sufficietly large such that there exists a (, M, ) geerator with M 2 R ad ǫ. We defie C 2 as the ifimum of all achievable R. The mai result of this paper is the followig theorm. Theorem : C = C 2 = C(X, X 2,, X N ). (7) The proof of Theorem is give i the Appedix. Thus both C ad C 2 admit sigle letter characterizatio which coicides with C(X,, X N ). III. EXAMPLES AND DISCUSSIONS We start with the followig example. Let X = (X, U, V ), Y = (Y, V, W) ad Z = (Z, W, U) where the radom variables X, Y, Z, U, V, W are mutually idepedet. It is easy to show that for this example whereas O the other had, I(X; Y ; Z) = K(X, Y, Z) = 0, C(X, Y, Z) = H(UV W). C(X, Y ) = H(V ), C(X, Z) = C(Y, Z) = H(U), H(W). What is iterestig is that the iclusio of a additioal variable icreases the commo iformatio. This is somewhat surprisig: if the iformatio is commo it ought to be oicreasig whe more radom variables are icluded. Ideed, we ca prove the followig geeral result: Lemma : Let (X,, X N ) p(x,, x N ). For ay two sets A, B that satisfy A B N = {, 2,, N}, C(X A ) C(X B ), (8) where X A = {X i, i A} ad X B = {X i, i B}. Proof: Let W be the W that achieves C(X B ), i.e., I(W ;X B ) = if I(W;X B ). But A B, thus X B coditioally idepedet give W implies that X A is coditioally idepedet give W. Thus I(X B ; W ) I(X A ; W ) if I(X A ; W) where the ifimum is take over all W such that X A is idepedet give W. This mootoe property perhaps suggests that the ame commo iformatio, while meaigful for pair of variables, o loger suits the geeralizatio to N variables. We commet here that Gács ad Körer s commo radomess follows a differet mootoe property K(X A ) K(X B ) while there is o defiitive iequality relatioship for mutual iformatio. As a cosequece, we have for ay N radom variables C(X, X 2,, X N ) K(X, X 2,, X N ). We ow examie aother example i which Wyer s commo iformatio icreases as the umber of the observatios icreases. Moreover the commo iformatio evetually coverges ad the asymptote suggests that the otio of commo iformatio may have potetial applicatio i certai iferece problem. Cosider first the example of three biary radom variables X, X 2, X 3 with joit distributio { P(x, x 2, x 3 ) = a 0 if x = x 2 = x 3 4 a 0 otherwise where the parameter a 0 satisfies 0 a 0 2. It ca be easily verified that (9) Pr{X i = 0} = 2, (20) for i =, 2, 3 ad that for i, j 3, i j, Pr(X i = x i, X j = x j ) = 2 ( a 0)δ xi,x j + 2 a 0( δ xi,x j ), (2) where δ a,b = if a = b ad 0 otherwise. Thus, each pair of (X i, X j ), i j, ca be viewed as a doubly symmetric biary source as defied i [4]. We refer to this set of exchageable biary sources circularly symmetric biary source. For such circularly symmetric biary source (X, X 2, X 3 ) with joit distributio give i (9) ad radom variables (X, X 2, X 3, W) that satisfy (7) ad (8), we have the followig lemma. Lemma 2: H(X W) + H(X 2 W) + H(X 3 W) 3h(a ), (22) where a = 2 2 ( 2a 0) 2.

6 W Compariso of commo iformatio C(X,Y) ad C(X,Y,Z) C(X,Y) C(X,Y,Z) X X X N 2 Fig. 3. A simple Bayesia graph model. Commo iformatio This lemma is a direct cosequece of Wyer s result o doubly symmetric biary source [4]. Therefore, we have, I(X X 2 X 3 ; W) = H(X X 2 X 3 ) H(X X 2 X 3 W), 3 = H(X X 2 X 3 ) H(X i W), H(X X 2 X 3 ) 3h(a ), ( = + h(a 0 ) + a 0 + ( a 0 )h a 0 2( a 0 ) ) 2h(a ), (23) This lower boud ca ideed be achieved by choosig the followig radom variables. Let W be a radom variable with p W (0) = p W () = /2, i.e., a Beroulli(/2) radom variable. Let each X i be the output of a biary symmetric chael (BSC) with crossover probability a with W as iput. The chaels share the commo iput W but are otherwise idepedet of each other. This is illustrated i the simple Bayesia graph model i Fig. 3 with N = 3 where each lik represets a BSC with crossover probability a. Thus, the commo iformatio of this circularly symmetric biary source is, C(X, X 2, X 3 ) = + a 0 + h(a 0 ) + ( ( a 0 )h a 0 2( a 0 ) ) 3h(a ), (24) Notice that ay pair of (X i, X j ) is a doubly symmetric biary source [4], therefore, C(X, Y ) = + h(a 0 ) 2h(a ). It is straightforward to check that C(X, Y, Z) > C(X, Y ) whe 0 < a 0 < 2. This is also show umerically i Fig. 4. We ow study the geeralizatio of above example to arbitrary N ad i particular the asymptotic value of the commo iformatio for the circularly symmetric biary sources. Cosider N biary radom variables X, X 2,, X N with joit distributio p(x, X 2,, X N ) geerated by a uderlyig Bayesia graph model as i Fig. 3, where W is a a 0 Fig. 4. Compariso of commo iformatio. Beroulli(/2) radom variable ad each X i, i =, 2,, N, is the output of a BSC with crossover probability a (0 a 2 ) with a commo iput W. Hece, for x, x 2,, x N {0, }, P(x, x 2,, x ) = w {0,} 2 N P i (x i w), (25) where for each i =, 2,, N, p i (x i w) = ( a ) if x i = w ad a otherwise. Similarly, we have, H(X i W) Nh(a ), (26) for ay radom variable W that satisfies (7) ad (8). Therefore, C(X, X 2,, X N ) ca be lower bouded by C(X, X 2,, X N ) H(X, X 2,, X N ) Nh(a ). (27) O the other had, the above lower boud is achievable by exactly the same W i the above Bayesia model. Hece, we have, C(X, X 2,, X N ) = H(X, X 2,, X N ) Nh(a ), (28) where H(X, X 2,, X N ) ca be calculated from (25). Now cosider the above model but with icreasig N. For ay ǫ ad a < /2, it is clear that H(W X, X 2,, X N ) < ǫ for N sufficietly large. This ca be established by the Fao s iequality as oe ca estimate W with arbitrary reliability give X,, X N for sufficietly large N. Therefore, C(X, X 2,, X N ) = H(X, X 2,, X N ) Nh(a ), = H(X, X 2,, X N, W) Nh(a ) H(W X, X 2,, X N ), H(W) ǫ, (29)

7 where the last step is from the fact that H(X, X 2,, X N W) = Nh(a ). O the other had, C(X,, X N ) H(W) for ay N. Thus, for a < /2, lim C(X, X 2,, X N ) = H(W) = N If a = /2, the X,, X N are mutually idepedet hece C(X,, X N ) = 0. IV. CONCLUSIONS This paper geeralized Wyer s commo iformatio, defied for a pair of radom variables, to that of N depedet radom variables. We showed that it is the miimum commo iformatio rate R 0 eeded for N separate decoders to recover their iteded sources losslessly while keepig the total rate close to the etropy boud. It is also equivaletly to the smallest rate of the commo iput to N idepedet processors (radom umber geerators), such that the output distributio is approximately the same as the give joit distributio. It was show that such geeralizatio leads to the pheomeo of commo iformatio o-decreasig as the umber of sources icreases. For the example of circularly symmetric biary sources, we show that commo iformatio ot oly icreases as N grows, but evetually coverges to the etropy of W that achieves C(X,, X N ). APPENDIX I this appedix, we give the proof of Theorem. First, as with [4], we defie a quatity Γ(δ, δ 2 ) which plays a importat role i the proof. Let (X, X 2,, X N ) P(x, x 2,, x N ) where X,, X N take values i fiite alphabet X,, X N. Let ( ˆX, ˆX 2,, ˆX N, W) be a (N + )tuple of radom variables where ˆX X, ˆX 2 X 2,, ˆX N X N ad W W, a fiite set. Deote the margial distributio of ( ˆX, ˆX 2,, ˆX N ) by Q(x, x 2,, x N ) = Pr( ˆX = x, ˆX 2 = x 2,, ˆX N = x ), (30) for x i X i, i =, 2,, N. For ay δ, δ 2 0, defie Γ(δ, δ 2 ) = suph( ˆX, ˆX 2,, ˆX N W), (3) where the sumpremum is take over all (N + )-tuples ( ˆX, ˆX 2,, ˆX N, W) that satisfy D(P; Q) = P(x, x 2,, x N )log P(x, x 2,, x N ) Q(x x,y, x 2,, x N ) δ, Substitute (43) ito (4), we get, (32) log M 0 H(X X 2 X N ) ad H( ˆX i W) H( ˆX, ˆX 2,, ˆX N W) δ 2. (33) H(X X 2 X N ) Γ ( It follows that C(X, X 2,, X N ) as defied i Theorem, is equivalet to C(X, X 2,, X N ) = H(X, X 2,, X N ) Γ(0, 0). (34) The followig lemma gives some properties of Γ(δ, δ 2 ). Lemma 3: ) For all δ, δ 2 0, there exists a (N +)-tuple ( ˆX, ˆX 2,, ˆX N, W) such that (32) ad (33) are satisfied ad Γ(δ, δ 2 ) = H( ˆX, ˆX 2,, ˆX N W). (35) Moreover, for δ, δ 2 = 0, W N X i. (36) 2) Γ(δ, δ 2 ) is a cocave fuctio of (δ, δ 2 ) ad it is cotiuous for all δ, δ ) For δ 0, defie Γ (δ) = Γ(0, δ) ad Γ 2 (δ) = Γ(δ, 0), the Γ (δ) ad Γ 2 (δ) are cocave ad cotiuous for δ 0. The proof of Lemma follows similarly as the proof of Theorem 4.4 i [4]. A. Proof of C = C(X, X 2,, X N ). I this sectio, we prove the first part of Theorem, that is C = C(X, X 2,, X N ). We first prove the coverse part, that is for ay R 0 that is achievable for the Gray-Wyer source codig etwork, we have, Theorem 2 (Coverse): C C(X, X 2,, X N ). (37) To prove the coverse, first let (f, g i ), i =, 2,, N be ay (, M 0, M,, M N ) code that satisfies (2), (3) ad (4). The, we have, log M 0 H(M 0 ), (38) I(X X 2 X N ; M 0), (39) = H(X X2 XN) H(X X2 XN M 0 ), (40) = H(X X 2 X N ) H(X j X 2j X Nj W j (4) ), where W j (M 0, X j, X j 2,, X j N ) ad Xj i = (X i, X i2,, X i,j ) for i =, 2,, N. Notice that, the (N + )-tuple (X j, X 2j,, X Nj, W j ) satisfies coditio (32) ad (33) with δ = 0 ad δ (j) 2 = H(X i,j W j ) H(X j, X 2j,, X Nj W j ). (42) Hece, by the defiitio of Γ(δ, δ 2 ), we have H(X j X 2j X Nj W j ) Γ (δ (j) 2 ). (43) Γ (δ (j) 2 ), (44) δ (j) 2 ).(45)

8 where the last step is from the cocavity of Γ ( ) fuctio. Now defie η = δ (j) 2. (46) The followig lemma gives a upper boud o η. Lemma 4: For ay (, M 0, M,, M N ) code that satisfies (2), (3) ad (4), we have η (N + )ǫ. (47) Proof : By Fao s iequality, we have, for i =, 2,, N, Hece, we have, for i =, 2,, N, H(X i M 0 M i ) ǫ. (48) log M i H(M i ), (49) The, we get, H(M i M 0 ), (50) = H(X i M i M 0 ) H(X i M i M 0 ), (5) H(X i M i M 0 ) ǫ, (52) = H(X M 0 ) ǫ. (53) log M i H(Xi M 0) ǫ. (54) where ǫ = Nǫ. Together with (4), we get, log M i H(X X 2 X N ) i=0 + Together with (4), we get, H(Xi M 0) H(X j X 2j X Nj W j ) H(Xi M 0 ) ǫ. (55) H(X j X 2j X Nj W j ) ǫ. (56) where ǫ = (N + )ǫ. O the other had, we have, = = H(Xi M 0) H(X ij X j i M 0 ), (57) H(X ij X j, X j 2,, X j N, M 0), (58) H(X ij W j ). (59) Combie (56) ad (59), we have, [ N ] H(X ij W j ) H(X j X 2j X Nj W j ) ǫ. Hece, we have, This completes the proof of Lemma 4. Now, from Lemma 4 ad (45), we get, (60) δ (j) 2 ǫ. (6) R 0 log M 0 H(X, X 2,, X N ) Γ (η). (62) Together with the cotiuity of Γ ( ), we have, as, R 0 H(X, X 2,, X N ) Γ (0), (63) = C(X, X 2,, X N ). (64) This completes the proof of coverse part. We ow prove the achievability part, that is, let the joit distributio P(x, x 2,, x N ) be give, we have, Theorem 3 (Achievability): C C(X, X 2,, X N ). (65) Our proof maily ivolves geeralizig Gray-Wyer source codig etwork [4] to that of N sources. The system model we cosidered here is the same as Fig. described i sectio II except that defiitio 2 is replaced by, Defiitio 5: A rate tuple (R 0, R,, R N ) is said to be achievable if for all ǫ > 0, we ca fid a sufficietly large such that there exists a (, 2 R0, 2 R,, 2 RN ) code with P () e ǫ. (66) Our purpose is to fid all achievable rate tuples (R 0, R,, R N ). The rate regio of this source codig problem is summarized i the followig theorem. Theorem 4: For the source codig model described above, a rate tuple (R 0, R,, R N ) is achievable if ad oly if the followig coditios are satisfied, R 0 I(X, X 2,, X N ; W), (67) R i H(X i W), (68) for i =, 2,, N, ad for some W P(w x, x 2,, x N ), where W W ad W N X i + 2. Proof of Theorem 4 (Sketch): For the achievability part, we wat to show that for ay rate tuple (R 0, R,, R N ) that satisfies above coditios, we ca costruct a (, 2 R0, 2 R,, 2 RN ) code such that the decodig error P e () 0 as codeword legth. Codeword Geeratio: for ay give distributios P(x, x 2,, x N ) ad P(w x, x 2,, x N ), we calculate the margial distributio P(w). ) Codebook C 0 : we first radomly geerate 2 R0 sequeces w i.i.d. P(w), ad idex them by m 0 {, 2,, 2 R0 }.

9 2) Codebook C(X i ): for each i =, 2, N, for each x i X i, radomly put them ito 2Ri bis ad idex them bis by m i {, 2,, 2 Ri }. Ecodig: ) for each source sequeces (x,x 2,, x N ), ecoder f 0 fids a w (m 0 ) C 0 such that (x, x 2,, x N, w (m 0 )) Tǫ, where T ǫ is the joitly typical set as defied i [5], ad sed the idex m 0 to the decoder. If there is o more tha oe w, choose the sequece w with the smallest idex; if there exist o such sequece, choose sequece w (), 2) for i =, 2,, N, ecoder f i seds the bi idex m i of sequece x i. Decodig: for i =, 2,, N, decoder i looks at bi m i for codebook C(X i ) ad fids the sequece ˆx i such that (ˆx i, w (m 0 )) Tǫ. If there is more tha oe or oe such sequece, declare a error. Error aalysis: Assumig m i, i = 0,,, N are the chose idices for ecodig (x, x 2,, x N ). There are three error evets. ) E : (x, x 2,, x N, w (m 0 )) / T ǫ for all m 0 {, 2,, 2 R0 }. 2) E 2 : (x i, w (m 0 )) / T ǫ for each i. 3) E 3 : for some i, there exists x i x i i bi m i of codebook C(X i ) such that ( x i, w (m 0 )) T ǫ. Hece, P e () P(E ) + P(E 2 E c ) + P(E 3 E c, Ec 2 ). (69) By some stadard argumet, we ca get, as, ) P(E ) 0 if R 0 I(X, X 2,, X N ; W) + ǫ, (70) 2) P(E 2 E) c 0, 3) P(E 3 E c, Ec 2 ) 0 if for each i =, 2,, N, This completes the achievability proof. R i H(X i W) + ǫ. (7) For the coverse part, we wat to show that for ay achievable rate tuple (R 0, R,, R N ), it should satisfy (67) ad (68). By Fao s iequality, we have H(X i M i M 0 ) ǫ. (72) Hece, we have, for i =, 2,, N ad R i H(M i ), (73) H(M i M 0 ), (74) H(M i M 0 ) + H(X i M im 0 ) ǫ, (75) = H(X i M i M 0 ) ǫ, (76) = H(Xi M 0) ǫ, (77) = H(X ij M 0 X j i ) ǫ, (78) H(X ij M 0, X j, X j 2,, X j ) ǫ.(79) R 0 H(M 0 ), (80) I(M 0 ; X, X 2,, X N ), (8) = I(M 0 ; X j X 2j X Nj X j X j 2 X j N ), = I(M 0 X j X j 2 X j N ; X jx 2j X Nj ).(83) Defie W j = (M 0, X j, X j 2,, X j N ), ad usig a stadard time sharig argumet, we ca get, for i =, 2,, N, N R i H(X i W) ǫ, (84) R 0 I(X X 2 X N ; W). (85) Let, the ǫ 0, ad this completes the proof of coverse. The cardiality boud ca be obtaied usig the techique itroduced i [6, Appdedix C]. We skip the details. This completes the proof of Theorem 4. Now we proceed to prove Theorem 3. We will show that if R 0 > C(X, X 2,, X N ), it is achievable for Model I. Let R 0 > C(X, X 2,, X N ) ad ay ǫ > 0 be give ad let radom variables (X, X 2,, X N, W) satisfy (7) ad (8), such that C(X, X 2,, X N ) = I(X X 2 X N ; W). (86) Notice that, the existece of such radom variables is guarateed by Lemma 3. Now defie ǫ ǫ = mi{ N +, R 0 C(X, X 2,, X N )}, (87) ad hece ǫ > 0. By Theorem 4, there exists a (, M 0, M,, M N ) code with P e () ǫ ad ǫ ǫ. Hece, log M 0 C(X, X 2,, X N ) + ǫ R 0, (88) log M i H(X i W) + ǫ. (89)

10 Hece, we have, log M i i=0 C(X, X 2,, X N ) + H(X i W) + ǫ, (90) (a) = H(X, X 2,, X N ) + ǫ. (9) where (a) is from coditio (8). Thus, coditio (4) is also satisfied. This implies that R 0 is achievable i Model I, which completes the proof of achievability part. This completes the proof of Theorem 3. B. Proof of C 2 = C X,X 2,,X N. I this sectio, we prove the secod part of theorem, that is C 2 = C(X, X 2,, X N ). We have the followig theorem. Theorem 5: C 2 C(X, X 2,, X N ), (92) C 2 C(X, X 2,, X N ). (93) For the coverse part, that is (92), the proof follows almost the same lie as i [4, Sectio 5.2]. For the achievability part, that is (93), the proof follows similarly as i [4, Secito 6.2] by applyig U = X X 2, X N i [4, Theorem 6.3]. We omit the details here. REFERENCES [] P. Gács ad J. Körer, Commo iformatio is much less tha mutual iformatio, Problems Cotr. Iform. Theory, vol. 2, pp , 973. [2] R. Ahlswede ad J. Körer, O commo iformatio ad related characteristics of correlated iformatio sources, Proc. of the 7th Prague Coferece of Iformatio Theory, 974. [3] H. S. Witsehause, O sequeces of pairs of depedet radom variables, SIAM J. Appl. Math., vol. 28, pp. 00-3, Ja [4] A. D. Wyer, The commo iformatio of two depedet radom variables, IEEE Tras. If. Theory, vol. 2, o. 2, pp , March [5] H. S. Witsehause, Values ad bouds for the commo iformatio of two discrete radom variables, SIAM J. Appl. Math., vol. 3, o. 2, pp , 976. [6] C. E. Shao, A mathematical theory of commuicatio, Bell System Techical Joural, vol. 27, pp , 948. [7] R. Ahlswede ad I. Csiszár, Commo radomess i iformatio thoery ad cryptography, Part I: Secret sharig, IEEE Tras. If. Theory, vol. 39, pp. 2-32, July 993. [8] R. Ahlswede ad I. Csiszár, Commo radomess i iformatio thoery ad cryptography, Part I: CR capacity, IEEE Tras. If. Theory, vol. 44, pp , Ja [9] U. M. Maurer, Secret key agreemet by public discussio from commo iformatio, IEEE Tras. If. Theory, vol. 39, pp , May 993. [0] G. Hu O the amout of iformatio, Teor. Veroyatost. i Primee., 4: , 962 (i Russia). [] I. Csiszár ad J. Körer, Iformatio Theory: Codig Theorems for Discrete Memoryless Systems, Academic, New York, 98. [2] R. W. Yeug, Iformatio Theory ad Network Codig, Spriger, [3] H. Tyagi ad P. Naraya ad P. Gupta, Whe is a Fuctio Securely Computable?, [4] R. M. Gray ad A. D. Wyer, Source codig for a simple etwork, Bell Syst. Tech. J., vol. 58, pp , Nov [5] T. M. Cover ad J. A. Thomas, Elemets of Iformatio Theory, Wiley, New York, 99. [6] A. El Gamal ad Y. H. Kim, Lecture otes o etwork iformatio theory,