ELEC3028 Digital Transmission Overview & Information Theory. Example 1

Transcription

1 Example. A source emits symbols i, i 6, in the BCD format with probabilities P( i ) as given in Table, at a rate R s = 9.6 kbaud (baud=symbol/second). State (i) the information rate and (ii) the data rate of the source. 2. Apply Shannon-Fano coding to the source signal characterised in Table. Are there any disadvantages in the resulting code words? 3. What is the original symbol sequence of the Shannon- Fano coded signal ? 4. What is the data rate of the signal after Shannon-Fano coding? What compression factor has been achieved? i Table. P( i ) BCD word A B C D E F Derive the coding efficiency of both the uncoded BCD signal as well as the Shannon-Fano coded signal. 6. Repeat parts 2 to 5 but this time with Huffman coding. 72

2 Example - Solution. (i) Entropy of source: 6 H = P( i ) log 2 P( i ) = 0.30 log log log i= 0.5 log log log = = bits/symbol Information rate: R = H R s = [bits/symbol] 9600 [symbols/s] = 9750 [bits/s] (ii) Data rate = 3 [bits/symbol] 9600 [symbols/s] = [bits/s] 2. Shannon-Fano coding: P() I (bits) steps code E A D B F C Disadvantage: the rare code words have maximum possible length of q = 6 = 5, and a buffer of 5 bit is required. 73

3 3. Shannon-Fano encoded sequence: = DEFEEEDADE 4. Average code word length: Data rate: Compression factor: d = = 2. [bits/symbol] 5. Coding efficiency before Shannon-Fano: d R s = = 2060 [bits/s] 3 [bits] d [bits] = 3 2. =.4286 CE = information rate data rate = = 68.58% Coding efficiency after Shannon-Fano: CE = information rate data rate == = 97.97% Hence Shannon-Fano coding brought the coding efficiency close to 00%. 6. Huffman coding: 74

4 P() steps code E 0.4 A D B F C step 3 step 4 step 5 E 0.40 E 0.40 ADBFC A 0.30 A E 0.40 D DBFC 0.30 BFC 0.5 step step 2 E 0.40 E 0.40 A 0.30 A 0.30 D 0.5 D 0.5 B 0.0 B F FC 0.05 C 0.02 Same disadvantage as Shannon-Fano: the rare code words have maximum possible length of q = 6 = 5, and a buffer of 5 bit is required = EEAEEEEAAEEDEEA The same data rate and the same compression factor achieved as Shannon-Fano coding. The coding efficiency of the Huffman coding is identical to that of Shannon-Fano coding. 75

5 Example 2. Considering the binary symmetric channel (BSC) shown in the figure: P( 0 ) = p P( ) = p 0 p e p e p e P(Y ) p e Y 0 Y P(Y 0 ) From the definition of mutual information, I(, Y ) = i P( i, Y ) log 2 P( i Y ) P( i ) [bits/symbol] derive both (i) a formula relating I(, Y ), the source entropy H(), and the average information lost per symbol H( Y ), and (ii) a formula relating I(, Y ), the destination entropy H(Y ), and the error entropy H(Y ). 2. State and ustify the relation (>,<,=,, or ) between H( Y ) and H(Y ). 3. Considering the BSC in Figure, we now have p = 4 and a channel error probability p e = 0. Calculate all probabilities P( i, Y ) and P( i Y ), and derive the numerical value for the mutual information I(, Y ). 76

6 Example 2 - Solution. (i) Relating to source entropy and average information lost: I(, Y ) = i = i = i = i P( i, Y ) log 2 P( i Y ) P( i ) P( i, Y ) log 2 P( i ) i P( i, Y ) A log 2 P( i ) P(Y ) P( i ) log 2 P( i ) i! P( i Y ) log 2 P( i Y ) P( i, Y ) log 2 P( i Y ) P(Y ) I( Y ) = H() H( Y ) (ii) Bayes rule : P( i Y ) P( i ) = P( i, Y ) P( i ) P(Y ) = P(Y i ) P(Y ) 77

7 Hence, relating to destination entropy and error entropy: P(Y i ) I(, Y ) = P( i, Y ) log 2 P(Y ) i i P(Y, i ) log 2 P(Y i ) = i P(Y, i ) log 2 P(Y ) = H(Y ) H(Y ) 2. Unless p e = 0.5 or for equiprobable source symbols, the symbols Y at the destination are more balanced, hence H(Y ) H(). Therefore, H(Y ) H( Y ). 3. Joint probabilities: Destination total probabilities: P( 0, Y 0 ) = P( 0 ) P(Y 0 0 ) = = P( 0, Y ) = P( 0 ) P(Y 0 ) = 4 0 = P(, Y 0 ) = P( ) P(Y 0 ) = = P(, Y ) = P( ) P(Y ) = = P(Y 0 ) = P( 0 ) P(Y 0 0 ) + P( ) P(Y 0 ) = =

8 P(Y ) = P( 0 ) P(Y 0 ) + P( ) P(Y ) = = 0.7 Conditional probabilities: P( 0 Y 0 ) = P( 0, Y 0 ) = P(Y 0 ) 0.3 = 0.75 Mutual information: P( 0 Y ) = P( 0, Y ) P(Y ) P( Y 0 ) = P(, Y 0 ) P(Y 0 ) P( Y ) = P(, Y ) P(Y ) I(, Y ) = P( 0, Y 0 ) log 2 P(Y 0 0 ) P(Y 0 ) +P(, Y 0 ) log 2 P(Y 0 ) P(Y 0 ) = = = = 0.25 = = P( 0, Y ) log 2 P(Y 0 ) P(Y ) + P(, Y ) log 2 P(Y ) P(Y ) = = [bits/symbol] 79

9 Example 3 A digital communication system uses a 4-ary signalling scheme. Assume that 4 symbols -3,-,,3 are chosen with probabilities 8, 4, 2, 8, respectively. The channel is an ideal channel with AWGN, the transmission rate is 2 Mbaud (2 0 6 symbols/s), and the channel signal to noise ratio is known to be 5.. Determine the source information rate. 2. If you are able to employ some capacity-approaching error-correction coding technique and would like to achieve error-free transmission, what is the minimum channel bandwidth required? 80

10 Example 3 - Solution. Source entropy: H = 2 8 log log log 2 2 = 7 4 [bits/symbol] Source information rate: R = H R s = = 3.5 [Mbits/s] 2. To be able to achieve error-free transmission ( R C = B log 2 + S ) P N P B log 2 ( + 5) Thus B [MHz] 8

11 Example 4 A predictive source encoder generates a bit stream, and it is known that the probability of a bit taking the value 0 is P(0) = p = The bit stream is then encoded by a run length encoder (RLC) with a codeword length of n = 5 bits.. Determine the compression ratio of the RLC. 2. Find the encoder input patterns that produce the following encoder output cordwords What is the encoder input sequence of the RLC coded signal ? 82

12 Example 4 - Solution. Codeword length after RLC is n = 5 bits, and average codeword length d before RLC with N = 2 n N d = (l + ) p l ( p) + N p N = pn p Compression ratio l=0 d n = pn n( p) = = RLC table {z } {z } {z } {z } {z } the encoder input sequence 00 0 {z } {z 0000} 30 83