Problem Set I Lossless Coding

Transcription

1 Problem Set I Lossless Coding Christopher Tsai Problem #2 Markov- Source Model The two-dimensional entropy rate function, is, Problem 2[ii] - Entropy Rate Function of a Markov- Source.9 Entropy H(α, β) β α. For both special cases, the entropy rate reduces to log log. For the case, the transition probabilities are identical, leading to an equal probability and proportion of ones and zeros. In other words, such a source is state-independent; every transition matrix looks identical. For the case, the two transition probabilities are complements; any given bit is zero with probability β, and one with probability α. In this case, α directly denotes the proportion of ones in the sequence, whereas β = α approximates the number of zeros. For, our test sequence has a tenth-order entropy of approximately bit, which closely approximates the analytical entropy rate of bit. Page

2 Problem #3 Run-Length Coding If we compress a Markov- source sequence with run-length encoding, we can achieve superior compression ratios for low and for high than we can for intermediate values Problem 3[iii] - Compression Ratio of a Markov- Source Bit Stream 4 Run-Length Compression Ratio Transition Probability α This plot reveals that, when the probability α is low (less than.), the low probability of transition permits extremely lengthy runs, which we can compress quite easily into a few numbers. For example, a run of zeros may occur, leading to a -fold reduction in the number of bits. However, the gain from the massive summarization of data quickly yields to the large number of possible run-lengths, spanning a number of nearly equally likely possibilities; the compression ratio quickly decreases to unity as the amount of randomness in the sheer number of possible run lengths ( to +) that we must encode overtakes the savings gained from shrinking long runs. On the other end of the scale, for high transition probabilities α >.9, the runs are short, and the bit alternations frequent, but the number of run-lengths we must encode also shrinks to a small typical set {mainly and 2}, allowing us to describe the potential outcomes in fewer bits. Page 2

3 The probability mass functions for three representative values of α confirm our hypothesis: Probability Problem 3[iii] - Run Length Probability Mass Function (α =.5) Run Length Problem 3[iii] - Run Length Probability Mass Function (α = ) Probability.3. Probability Run Length Problem 3[iii] - Run Length Probability Mass Function (α =.95) Run Length The low probability (α =.5) yields a large alphabet of possible run-lengths that we must encode, but the longer runs facilitate data reduction from many bits to a single representative count. The intermediate probability (α = ) balances alphabet size ( 6) with common lengths, but the mere possibility of longer runs forces us to accommodate them, and, alas, the savings gained from compression fails to exceed the amount of uncertainty in the large alphabet of run-lengths. The high probability (α =.95) yields an easily encodable alphabet, thereby achieving a maximal compression ratio despite the rarity of actual runs. When we apply run-length encoding to the images Leisler and Snoopy, we observe that the resultant compression ratio depends on how we vectorize our two-dimensional image into a single Page 3

4 one-dimensional bit stream; unsurprisingly, the way we scan our image changes the distribution of runs, as the following table reveals: IMAGE COLUMNWISE (Up Down) ROWWISE (Left Right) Leisler Snoopy Thus, we can maximize our compression ratio by juxtaposing the image and its transposition to determine how one might exploit the distribution of runs for the best compression. For the Snoopy image, for instance, the inclusion of text in the transcript bubble creates several runs (lines) of white pixels, allowing us to exploit their redundancy in run-length coding; armed with this prescience, we scan left to right to encounter these white lines and maximize the length of our runs. Page 4

5 Problem #4 Adaptive Golomb Coder If we compress a Markov- source sequence with Golomb-coded run-length encoding, then we can achieve different compression ratios since the Golomb code should optimal for geometric sources. However, it remains to be seen whether or not our Golomb code can compete with the ideal entropy coding. We juxtapose Golomb compression ratios with ideal entropy coding: Compression Ratio Problem 4[ii] - Compression Ratio of a Markov- Source Bit Stream Transition Probability α The Golomb code compresses the run-length bit stream admirably well for Markov- source probabilities α <, its compression ratio essentially coinciding with the ideal optimum computed as binary entropy. However, for higher transition probabilities more state alternation, shorter and fewer runs the Golomb code breaks down, failing to compress the data at all; most likely, this shortcoming unfolds because the data grow less and less geometric as the state transitions (bit flips) occur more frequently. With fewer runs to exploit in unary coding, the Golomb code mirrors the complexity and length of the run-length sequence itself. Ideal Golomb Page 5

6 Solid initial estimates for A and materialize from repeated trial and error. In Christopher Tsai particular, A represents the average run-length (or most frequently Golomb-encoded value); hence, based on our observation from the histograms for α =.5,, and.95, we might select A of 4 or 5, since these run lengths occur quite frequently for intermediate alpha. Of course, Golomb code targeting lower values of α benefit from higher values of A since the possible run lengths stretch past ; conversely, higher values of α demand A of unity, as runs hardly ever occur for frequently transitioning source codes. Meanwhile, because modern technology makes memory an afterthought especially for the simple images we compress here we need not worry about limiting the amount of hindsight our coder taps; instead, we choose well above the number of elements in our system, ensuring that our Golomb coding algorithm considers the entire input stream. Compressing the Leisler and Snoopy images, we obtain the following compression ratios: IMAGE COLUMNWISE (GOLOMB) COLUMNWISE (IDEAL) ROWWISE (GOLOMB) ROWWISE (IDEAL) Leisler Snoopy Juxtaposition with the ideal entropies reveals that the Golomb coder performs quite far from the optimum for Leisler and Snoopy images. Whereas the run-length codes could be compressed as much as.7 with ideal coding, the Golomb coding can achieve only.4 for the Leisler image and approximately.5 for the Snoopy image. The ideal entropy coder prevails over the Golomb coder. Page 6