Chapter 1 Introduction

Transcription

1 Chapter 1 Introduction 1. Shannon s Information Theory 2. Source Coding theorem 3. Channel Coding Theory 4. Information Capacity Theorem 5. Introduction to Error Control Coding Appendix A : Historical Notes

2 1. Shannon's Information Theory The theory provides answers to two fundamental questions (among others: (awhat is the irreducible complexity below which a signal cannot be compressed? (bwhat is the ultimate transmission rate for reliable communication over a noisy channel? 2. Source Coding Theorem (Shannon's first theorem The theorem can be stated as follows: Given a discrete memoryless source of entropy H (S, the average code-word length L for any distortionless source coding is bounded as L H(S

3 This theorem provides the mathematical tool for assessing data compaction, i.e. lossless data compression, of data generated by a discrete memoryless source. The entropy of a source is a function of the probabilities of the source symbols that constitute the alphabet of the source. Entropy of Discrete Memoryless Source Assume that the source output is modeled as a discrete random variable, S, which taes on symbols from a fixed finite alphabet With probabilities P(S = s S = { s, s, L, } sk K -1 = p, = 0,1,2, L, K 1 with p = 1 = 0 Define the amount of information gain after observing the event S = s as the logarithmic function

4 I(s = log 2 ( 1 p bits the entropy of the source is defined as the mean of I ( s over source alphabet S given by H(S = = = E[ I ( s K 1 = 0 K 1 = 0 p p ] I( s log 2 ( 1 p bits The entropy is a measure of the average information content per source symbol. The source coding theorem is also nown as the "noiseless coding theorem" in the sense that it establishes the condition for error-free encoding to be possible.

5 3. Channel Coding Theorem (Shannon's 2 nd theorem The channel coding theorem for a discrete memoryless channel is stated in two parts as follows: (alet a discrete memoryless source with an alphabet S have entropy H (S and produce symbols once every T S seconds. Let a discrete memoryless channel have capacity C and be used once every T C seconds. Then if H(S T C S T C There exists a coding scheme for which the source output can be transmitted over the channel and be reconstructed with an small probability of error. arbitrarily

6 (b Conversely, if H(S > T C S T C It is not possible to transmit information over the channel and reconstruct it with an arbitrarily small probability of error. The theorem specifies the channel capacity C as a fundamental limit on the rate at which the transmission of reliable error-free message can tae place over a discrete memoryless channel. Mutual Information alphabet X alphabet Y Channel input x (selected from X with entropy H. X = { x0, x1,..., x J 1} Channel output y (selected from Y, Y = K { y0, y1,..., y 1} How can we measure the uncertainty about x after observing y?

7 Define the conditional entropy of X selected from X, given that Y = y as H ( Y = y = J 1 j = 0 P ( x j y log 2 1 [ P ( x The mean of entropy H( y over the output alphabet Y is given by H ( Y = = K 1 = 0 K -1 J 1 = 0 H ( j= 0 P ( x Y j, y = y log P ( y 2 1 [ P ( x The conditional entropy represents the amount of uncertainty remaining the channel input after the channel output has been observed. The difference H( H( is called the mutual information, which represents the uncertainty about the channel input that is resolved by observing the channel output. Denoting the mutual information by I ( ;, we may thus write I ( ; = H( H( I ( ; = I ( ;. Note that j j y y ] ]

8 Channel Capacity The mutual information can be expressed by K -1 J 1 P( x j y I ( ; = P( x j, y log2[ ] P( y = 0 j= 0 The input probability distribution { P ( x j } is obviously independent of the channel. We can then maximize I ( ; with respect to { P ( x } j Hence we define the channel capacity C as C = max I ( ; { p( x j } The channel capacity C is measured in bits per channel use. The Channel Coding Theorem is also nown as noisy coding theorem.

9 4. Information Capacity Theorem (also nown as Shannon-Hartley law or Shannon's 3 rd theorem It can be stated as follows: The information capacity of a continuous channel of bandwidth B Hz, perturbed by additive white Gaussian noise of power spectral density 2 and limited in bandwidth to B, is given by C = B log2 ( 1 + P N B 0 N 0 bits/secon d where P is the average transmitted power. This theorem implies that, for given average transmitted power P and channel bandwidth B, we can transmit information at the rate C bits per second, with arbitrarily small probability of error by employing sufficiently complex encoding systems.

10 Shannon limit: For an ideal system that transmits data at a rate R b = C Thus P = Eb Rb = Eb C C B C B Then = log 2 (1 + b E N C B b 1 o 2 = c B For infinite bandwidth, the approaches the limiting value E N o E / N Eb Eb ( = lim( = ln 2 = = 1.6 db N B N o o This value is called Shannon limit. b o

11 Appendix A. Historical Notes The seeds of Shannon's information theory appeared first in a Bell Labs classified memorandum dated 1 September 1945, A Mathematical Theory of Cryptography (revised form was published in BSTJ, Shannon's wor on cryptography problem at Bell Labs brought him into contact with two scholars, R.V.L. Hartley and H. Nyquist, who were major consultants to the cryptography research. Nyquist (1924 has shown that a certain bandwidth was necessary in order to send telegraph signals at a definite rate. Hartley (1928 had attempted to quantify of information as the logarithm of the number of possible message built from a pool of symbols.

12 Shannon was aware of Norbert Wiener's wor on cybernetics (he cited Wiener's boo in his two 1948 articles on information theory. Wiener had recognized that the communication of information was a problem in statistics. Shannon had taen a mathematics course from Norbert Wiener when he was studying at MIT, and Shannon had access to the "Yellow Peril" report (Wiener's boo before publication while he wored on cryptographic research. Major Publications of Shannon A Mathematical Theory of Communication, I, II, published in BSTJ, 1948, , Communication Theory of Secrecy Systems, published in BSTJ, 1949,

13 Paperbac edition of "The Mathematics Theory of Communication" was published by University of Illinois Press, ,000 copies have been sold up to In a review of the accomplishments of Bell Labs. in communication science, Millman in his edited boo (1984, A history of engineering and science in the Bell system: Communication Science ( says: Probably the most specular development in communication mathematics to tae place at Bell Laboratories was the formulation in the 1940's of information theory by C. E. Shannon. In his 1948 articles on information theory, Shannon credits J.W. Tuey (a professor at Princeton Univ. with suggesting the word "bit" as a measure of information.

14 In 1984 J.L. Massey wrote a paper Information Theory: the Copernican System of Communications. Published in IEEE Communications Magazine, vol. 22, No. 12, pp The Shannon s theory of information is the scientific basis of communications in the same sense that the Copernicus heliocentric theory is the scientific basis of astronomy. IEEE Transaction on Information Theory Oct Vol.44 No.6. Commemorate Issue (

15 5. Introduction to Error Control Coding 5.1 Purpose of Error Control Coding In data communications, coding is used for controlling transmission errors induced by channel noise or other impairments, such as fading and interference, so that error-free communication can be achieved. In data storage system, coding is used for controlling storage errors (during retrieval caused by storage medium defects, dust particles and radiation so that error-free storage can be achieved.

16 5.2 Coding Principle Coding is achieved by adding properly designed redundant digits (bits to each message. These redundant digits (bits are used for detecting and/or correcting transmission (or storage errors. 5.3 Types of Codings Bloc and Convolutional coding. Bloc coding: A message of digits is mapped into a structured sequence of n digits, called a codeword. : message length n : code length n : code rate The mapping operation is called encoding. Each encoding operation is independent of the past encoding, i.e. memoryless. The collection of all codeword is called a "bloc code".

17 Convolutional Coding: An information sequence is divided into (short blocs of digits each. Each -digit message is encoded into an n-digit coded bloc. The n-digit coded bloc depends not only the corresponding -digit message bloc but also on m ( 1 previous message blocs. That is, the encoder has memory of order m. The encoder has inputs and n outputs. An information is encoded into a coded sequence. The collection of all possible code sequences is called an (n,, m convolutional code. Normally, 1 2 n 8 9 n : code rate

18 5.4 Type of Errors and Channels Random errors and burst errors. Type of Channels Random error channels: Deep space channel, satellite channels, line of sight transmission channel, etc. Burst error channels: Radio lins, terrestrial microwave lins, wire and cable transmission channels, etc.

19 5.5 Decoding Suppose a codeword corresponding to a message is transmitted over a noisy channel. Let r be the received sequence. Based on r, the encoding rules and the noise characteristics of the channel, the receiver (or decoder maes a decision which message was actually transmitted. This decision maing operation is called decoding. The device which performing the decoding operation is called a decoder. Two types of decoding: Hard-decision decoding and soft-decision decoding. Hard-decision decoding: When binary coding is used, the modulator has only binary inputs. If binary demodulator output quantization

20 is used, the decoder has only binary inputs. In this case, the demodulator is said to mae hard decisions. Decoding based on hard decision made by the demodulator is called hard decision decoding. Soft-decision decoding: If the output of demodulator consists of more than two quantization levels or is left unquantized, the demodulator is said to mae soft decisions. Decoding based on soft decision made by demodulator is called soft-decision decoding. Hard-decision decoding is much easier to implement than soft-decision decoding. However, soft-decision decoding offers much better performance.

21 5.6 Some Channel Models Binary Symmetric Channel (BSC p: transition probability When BPSK modulation is used over the AWGN channel with optimum coherent detection and binary output quantization, the bit-error probability for equally liely signal is given by p = Q( 2E N 0 where Q(x E : bit energy 1 2p x e 2 y 2 dy N 0 2 : power spectral density of AWGN

22 Binary-input, 8-ary Output Discrete Channel 5.7 Optimum Decoding Suppose the codeword c corresponding to a certain message m is transmitted. Let r be the corresponding output of the demodulator. The decoder produces an estimate mˆ of the message based on r. An optimum decoding rule is that minimize the probability of a decoding error. That is, P( cˆ c r is minimized. Or, equivalently, maximizing P ( cˆ = c r.

23 The decoding error is minimized for a given r by choosing ĉ to be a codeword c that maximizes P ( c r = P( r c P( c P( r That is, ĉ is chosen to be the most liely codeword, given that r is received. 5.8 Maximum a posteriori decoding (MAP decoding In general, we have P ( c r = P( r c P( c P( r If nowledge or an estimation of (c used for decoding, the technique is called MAP decoding. P is

24 5.9 Maximum Lielihood Decoding (MLD Suppose all the messages are equally liely. An optimum decoding can be done as follows: For every codeword c j, compute the conditional probability P ( r c j c j with the largest The codeword conditional probability is chosen as the estimate ĉ for the transmitted codeword c. This decoding rule is called the Maximum Lielihood decoding (MLD.

25 MLD for a BSC Let a = ( a1,a2, L, an and b = ( b1,b2, L, bn be two binary sequences of n components. The Hamming distance between a and b, denoted as d H ( a, b, is defined as the number of places where a and b differ. In coding for a BSC, every codeword and every received sequence are binary sequences. Suppose some codeword is transmitted and the received sequence is r = ( r,r, L, n 1 2 r For a codeword c i probability P ( r ci is, the conditional d H ( r,ci n-dh ( ci ( r c ( r, i p 1 - p P = 1 For p < 2, P ( r ci is a monotonically decreasing function of d H ( r, ci. Then P ( r ci > P( r c j if and only if d H ( r,ci < dh ( r, c j

26 The MLD is completed by the following steps: (i Compute d H ( r, ci for all c i (ii c i is taen as the transmitted codeword if d H ( r,ci < dh ( r, c j for j i (iii Decode c i into message m i That is, the received vector r is decoded into the closest codeword. This is also called the minimum distance (nearest neighbor decoding.

27 5.10 Bounded Distance Decoding Given a received word r, a t-error correcting, bounded distance decoder selects that codeword c which minimizes d H ( r, c if and only if there exists c such that d H ( r,c t. If no such c exists, then a decoder failure is declared. All practical bounded distance decoder use some form of syndrome decoding. The bounded distance decoding is usually an incomplete decoding since it decodes only those received words lying in a radius-t sphere about a codeword.