( ) = = , log

Transcription

1 Sample Midterm Solutions The examination is for 0 minutes. The maximum score is 70 points. Your answers should be unambiguous. Please show all work to allow for the possibility of partial credit.. (8 points) I have a coin in my pocket. It is either fair (it comes up either heads or tails equiprobably when tossed) or it is biased with the probability of it coming up heads being. A priori, it is equally likely to be fair or biased. Let X denote the 4 identity of the coin. Thus P (X = u) = P (X = b) =. The coin is tossed ten times. The outcomes of the tosses are independent and identically distributed. The coin comes up heads times. What is the conditional entropy of X given this event? You need not reduce your answer to a fraction. Solution : Let Y denote the number of heads among the ten tosses of the coin. Hence ( ) 0 P (Y = k X = u) = ( k )0, 0 k 0 ( ) 0 P (Y = k X = b) = ( k 4 )k ( 4 )0 k, 0 k 0. We are asked to find H(X Y = ). To this end we find ( ) 0 ( P (X = b Y = ) = and P (X = u Y = ) = Hence. (7 points) k 4 ) ( 4 )7 ) ( )0 ( ) 0 k ( 4 ) ( 4 )7 + ( 0 k = + 0 = 7 05, H(X Y = ) = 7 05 log log A Stanford student claims that if I(X; Y Z) = I(X; Z Y ) = I(Y ; Z X) then X, Y, and Z are independent. Either prove or disprove this statement. Suppose X = Y = Z. Then I(X; Y Z) = H(X Z) H(X Y, Z) = 0, and similarly for I(X; Z Y ) and I(Y ; Z X). However in this case X, Y, and Z are not mutually independent unless X is a constant. Thus the claim is false.

2 . (7 points) Let X be a random variable taking values in the finite set {,, }. Let l(x) denote the mean length of an optimal Huffman code for X. An M.I.T. student claims that if P (X = i) > 0 for all i =,,, then l(x) H(X) 5 log. Either prove or disprove this statement. Let (p, p, p ) denote the probability distribution of X. We may assume without loss of generality that p p p. Then l(x) = + (p + p ) = p. This depends only on p. For fixed p, H(X) is largest when p = p = p, and equals h(p ) + ( p ), where h(p) = [p log p + ( p) log( p)]. Thus l(x) H(X) h(p ). We must also have p. However, setting p = makes h(p ) = 0, so the M.I.T. student is wrong: if X has the probability distribution (,, ) then l(x) = H(X) =. Now we might remember that for 4 4 any probability distribution where all entries are a power of the mean length of a binary Huffman code equals the entropy of the probability distribution, so it was not necessary to go through this calculation. 4. ( points) Let (X n, n Z) be a stationary Markov chain with state space {0,, } and transition probability matrix In writing this matrix, the states are written in the order 0 followed by followed by.. Let (Y n, n Z) be the stationary {0, } valued process defined as follows : Y n = 0 if X n = 0 or X n = Y n = if X n = Show that the entropy rate of (Y n, n Z) is at most 4 5. Since each Y n is a deterministic function of X n we have H(Y,..., Y n ) H(X,..., X n ), for all n. H(Y Recall that the entropy rate of (Y n, n Z) is defined as lim,...,y n) n and n H(X likewise the entropy rate of (X n, n Z) is defined as lim,...,x n) n. Hence the n

3 entropy rate of (Y n, n Z) is upper bounded by that of (X n, n Z). The latter can be computed as = 4 5. This demonstrates the desired upper bound. 5. (6 + 6 points) The triple mutual information of three random variables X, X, and X, which take values in finite sets, is defined as I(X X X ) = H(X ) + H(X ) + H(X ) H(X, X ) H(X, X ) H(X, X ) + H(X, X, X ). Note that this quantity is symmetric in the variables X, X, and X. (a) Give an example of random variables X, X, and X for which I(X X X ) is strictly positive. (b) Give an example of random variables X, X, and X for which I(X X X ) is strictly negative. The triple mutual information can be manipulated into the form I(X X X ) = I(X ; X ) I(X ; X X ). (a) If I(X ; X X ) = 0 while I(X ; X ) 0, then the triple mutual information will be strictly positive. Thus we may seek triples (X, X, X ) such that X and X are not independent but they are conditionally independent when given X. An easy example is to take X to be uniformly distributed on {0, } and X and X to each equal X. Then I(X X X ) =. (b) If I(X ; X ) = 0 while I(X ; X X ) 0, the triple mutual information will be strictly negative. Thus we may seek triples (X, X, X ) such that X and X are independent but are not conditionally independent when given X. An easy example is to take X and X to be independent and uniform on {0, } and to take X to be the binary sum of X and X, which makes X also uniformly distributed on {0, }. In this case I(X ; X X ) = H(X X ) H(X X, X ) =, while I(X ; X ) = 0, so I(X X X ) =.

4 6. ( points) Consider the discrete memoryless channel with input and output alphabets X = Y = {,,, 4} and with the channel probability matrix Find the information capacity of this channel. Solution : Let X be an arbitrary X valued random variable and let Y a Y valued random variable with the conditional distribution of Y given X being given by the channel probability matrix. We note that H(Y X = x) = for all x X. Hence I(X; Y ) = H(Y ) H(Y X) = H(Y ). We also note that, since Y = 4, we have H(Y ). On the other hand, if we choose for the distribution of X the uniform distribution on the two inputs and 4 (with zero probability for the other two inputs) then Y is uniform on {,,, 4} and hence has entropy. We conclude that the information capacity of the channel is. 7. ( points) Let P and Q be probability distributions on a finite set X. For n, X, X,..., X n are random variables that are i.i.d. with distribution P and X n+, X n+,..., X n are random variables that are i.i.d with distribution Q. Further (X,..., X n ) and (X n+,..., X n ) are mutually independent. Recall that P n denotes the set of types with denominator n, and for any R P n T (R) denotes {x = (x,..., x n ) : P x = R}, where P x denotes the type of x. For R P n, find a formula for P ((X,..., X n ) T (R)). We may get a reasonably nice formula for the probabilities of interest by focusing on the types of each of (X,..., X n ) and (X n+,..., X n ) separately, call them S and S respectively; they are both in P n. We have P ((X,..., X n ) T (R)) = (S +S )=R 4 P ((X,..., X n ) T (S ))P ((X n+,..., X n ) T (S )),

5 where we have also used the independence of (X,..., X n ) and (X n+,..., X n ). We thus get P ((X,..., X n ) T (R)) = This can also be written as P ((X,..., X n ) T (R)) = (S +S )=R (S +S )=R T (S ) T (S ) n[h(s )+D(S P )+H(S )+D(S Q)]. n!n! x(ns (x))!(ns (x))! n[h(s )+D(S P )+H(S )+D(S Q)]. 5