FUNDAMENTALS of INFORMATION THEORY and CODING DESIGN

Transcription

1 DISCRETE "ICS AND ITS APPLICATIONS Series Editor KENNETH H. ROSEN FUNDAMENTALS of INFORMATION THEORY and CODING DESIGN Roberto Togneri Christopher J.S. desilva CHAPMAN & HALL/CRC A CRC Press Company Boca Raton London New York Washington, D.C.

2 Contents 1 Entropy and Information Structure Structure in Randomness First Concepts of Probability Theory Surprise and Entropy Units of Entropy The Minimum and Maximum Values of Entropy A Useful Inequality Joint Probability Distribution Functions Conditional Probability and Bayes'Theorem Conditional Probability Distributions and Conditional Entropy ' Information Sources Memoryless Information Sources Markov Sources and n-gram Models Stationary Distributions The Entropy of Markov Sources Sequences of Symbols The Adjoint Source of a Markov Source Extensions of Sources Infinite Sample Spaces Exercises References 49 2 Information Channels What Are Information Channels? BSC and BEC Channels Mutual Information Importance of Mutual Information Properties of the Mutual Information Noiseless and Deterministic Channels Noiseless Channels Deterministic Channels Cascaded Channels Additivity of Mutual Information Channel Capacity: Maximum Mutual Information Channel Capacity of a BSC Channel Capacity of a BEC 80 IX

3 2.7.3 Channel Capacity of Weakly Symmetric Channels Continuous Channels and Gaussian Channels Information Capacity Theorem Rate Distortion Theory Properties of R(D) Exercises References 103 Source Coding Introduction Instantaneous Codes Construction of Instantaneous Codes Decoding Instantaneous Codes Properties of Instantaneous Codes Sensitivity to Bit Errors The Kraft Inequality and McMillan's Theorem The Kraft Inequality McMillan's Theorem Average Length and Compact Codes Average Length Lower Bound on Average Length Shannon's Noiseless Coding Theorem Shannon's Theorem for Zero-Memory Sources Shannon's Theorem for Markov Sources Code Efficiency and Channel Capacity Fano Coding Huffman Coding Huffman Codes Binary Huffman Coding Algorithm Software Implementation of Binary Huffman Coding r -ary Huffman Codes Arithmetic Coding Encoding and Decoding Algorithms Encoding and Decoding with Scaling Is Arithmetic Coding Better Than Huffman Coding? Higher-order Modelling Higher-order Huffman Coding Higher-order Arithmetic Coding Exercises ; References 168 Data Compression Introduction Basic Concepts of Data Compression Run-length Coding 173

4 XI 4.4 The CCITT Standard for Facsimile Transmission Block-sorting Compression Dictionary Coding Statistical Compression Prediction by Partial Matching Image Coding Exercises References 196 Fundamentals of Channel Coding Introduction Code Rate Decoding Rules Hamming Distance Hamming Distance Decoding Rule for BSCs Error Detection/Correction Using the Hamming Distance Bounds on M, Maximal Codes and Perfect Codes Upper Bounds on M and the Hamming Bound Maximal Codes and the Gilbert Bound Redundancy Requirements for t-bit Error Correction Perfect Codes for i-bit Error Correction Error Probabilities Bit and Block Error Probabilities and Code Rate Probability of Undetected Block Error Shannon's Fundamental Coding Theorem Exercises References 234 Error-Correcting Codes Introduction Groups Rings and Fields Linear Spaces Linear Spaces over the Binary Field Linear Codes Encoding and Decoding Codes Derived from Hadamard Matrices Exercises References 278 Cyclic Codes Introduction Rings of Polynomials Cyclic Codes Encoding and Decoding of Cyclic Codes 296

5 Xll 7.5 Encoding and Decoding Circuits for Cyclic Codes The Golay Code Hamming Codes Cyclic Redundancy Check Codes Reed-Muller Codes Exercises References Burst-Correcting Codes Introduction Finite Fields Irreducible Polynomials Construction of Finite Fields Bursts of Errors Fire Codes Minimum Polynomials Bose-Chaudhuri-Hocquenghem Codes Other Fields Reed-Solomon Codes Exercises References Convolutional Codes Introduction A Simple Example Binary Convolutional Codes Decoding Convolutional Codes The Viterbi Algorithm Sequential Decoding Trellis Modulation Turbo Codes Exercises References 379 Index 381