Huffman Codes. Spring 2010

Transcription

1 Huffmn Coes C Lecture Notes Spring 2010 Shnnon s noiseless coing theorem tells us how compctly we cn compress messges in which ll letters re rwn inepenently from n lphet A n we re given the proility p of ech letter A ppering in the messge. Shnnon s theorem sys tht, for rnom messges with n letters, the expecte numer of its we nee to trnsmit is t lest nh(p) = n A p log 2 p its, n there exist coes which trnsmit n expecte numer of its eyon this lower oun of nh(p) which grows more slowly thn n s n tens to infinity (formlly, the numer f(n) of itionl its neee stisfies f(n) tens to 0 s n n tens to infinity). We will see now very efficient wy to construct coe tht lmost chieves this Shnnon lower oun. The ie is very simple. To every letter in A, we ssign string of its. For exmple, we my ssign to, 100 to n so on. woul e encoe s But to mke sure tht it is esy to ecoe messge, we mke sure this gives prefix coe. In prefix coe, for ny two letters x in y of our lphet the string corresponing to x cnnot e prefix of the string corresponing to n vice vers. For exmple, we woul not e llowe to ssign 1001 to c n to s. There is very convenient wy to escrie prefix coe s inry tree. The leves of the tree contin the letters of our lphet, n we cn re off the string corresponing to ech letter y looking t the pth from the root to the lef corresponing to this letter. Every time we go to the left chil, we hve 0 s the next it, n every time we go to the right chil, we get 1. For exmple, the following tree for the lphet A = {,, c,, e, f}: e c f 1

2 correspons to the prefix coe: c e 001 f 111 The min vntge of prefix coe is tht it is very to ecoe string of its y just repetely mrching own this tree from the root until one reches lef. For exmple, to ecoe , we mrch own the tree n see tht the first 3 its correspon to f, get for the next 2 its, n e, n thus our string ecoes to fe. Our gol now is to esign prefix coe which minimizes the expecte numer of its we trnsmit. If we encoe letter with it string of length l, the expecte length for encoing one letter is L = A p l, n our gol is to minimize this quntity L over ll possile prefix coes. By linerity of expecttions, encoing messge n letters results in it strength of expecte length n A p l. We ll show now n optiml prefix coe n this is known s the Huffmn coe, se on the nme of the MIT grute stuent who invente it in As we erive it in its full generlity, we will lso consier n exmple to illustrte it. Suppose our lphet is {,, c,, e, f} n the proilities for ech letter re: 0.05 c e 0.20 f 0.10 Let us ssume we know n optimum prefix coe; letter is encoe with string s of length l for ny letter of our lphet. First, we cn ssume tht, in this optimum prefix coe, the letter with smllest proility, sy s in our exmple (with proility 0.05) correspons to one of the longest strings. Inee, if tht is not the cse, we cn keep the sme strings ut just interchnge the string s with the longest string, n ecrese the expecte length of n encoe letter. If we now look t the prent of in the tree representtion of this optimum prefix coe, this prent must hve nother chil. If not, the lst it of s ws unnecessry n coul e remove (therey ecresing the expecte length). But, since correspons to one of the longest strings, tht siling of in the tree representtion must lso e lef n thus correspon to nother letter of our lphet. By the sme interchnge rgument s efore, we cn ssume tht this other letter hs the secon smllest proility, in our cse it is with proility Being silings men tht the strings s n s iffer only in the lst it. Thus, our optimum prefix coe cn e ssume to hve the following sutree: 2

3 Now suppose we replce n (the two letters with smllest proilities) y one new letter, sy α with proility p α = p + p (in our cse P α = 12). Our new lphet is thus A = A \ {, } {α}. In our exmple, we hve: c 0.18 e 0.20 f 0.10 α 0.12 = p + p n given our prefix coe for A, we cn esily get one for A y simply encoing this new letter α with the l 1 = l 1 common its of s n s. This correspons to replcing in the tree, the sutree y α In this prefix coe for A, we hve l α = l 1 = l 1. Thus, if we enote y L n L, the expecte length for our prefix coes for A n A, oserve tht L = L p α. On the other hn, if one consiers ny prefix coe for A, it is esy to get prefix coe for A such tht L = L + p α y simply oing the reverse trnsformtion. This shows tht the prefix coe for A must e optiml s well. But, now, we cn repet the process. For A, the two letters with smllest proilities re in our exmple f (p f = 0.1) n α (p α = 0.12), n thus they must correspon to the sutree: f α Expning the tree for α, we get tht one cn ssume tht the following sutree is prt of n optimum prefix coe for A: 3

4 f We cn now ggregte f n α (i.e. f, n ), into single letter β with p β = p f +p α = Our new lphet n corresponing proilities re therefore: c 0.18 e 0.20 β 0.22 = p f + p α = p f + p + p This process cn e repete n les to the Huffmn coe. At every stge, we tke the two letters, sy x n y, with smllest proilities of our new lphet n replce them y new letter with proility p x + p y. At the en, we cn unrvel our tree representtion to get n optiml prefix coe. Continuing the process with our exmple, we get the sequence of lphets: n β 0.22 = p f + p α = p f + p + p γ 0.38 = p c + p e δ 0.60 = p β + p γ = p f + p + p + p c + p e ɛ 1 = p + p δ = p + p + p c + p + p + p e + p f. We cn o the reverse process to get n optimum prefix coe for A. First, we hve: ɛ δ 4

5 β γ β γ β c e f α c e n finlly: 5

6 f c e This correspons to the prefix coe: c e 111 f 100 The expecte numer of its per letter is thus p l = = A This is to e compre to Shnnon s lower oun of A p log 2 (p ) = 2.25, n the trivil upper oun of log 2 ( A ) = We will now show tht Huffmn coing is lwys close to the Shnnon oun, no mtter how lrge our lphet A is n our choice of proilities. Our first lemm (known s Krft s inequlity) gives n inequlity for the epths of the leves of inry tree. Lemm 1 (Krft s inequlity) Let l 1, l 2,, l k N. Then there exists inry tree with k leves t istnce (epth) l i for i = 1, k from the root if n only if k 2 l i 1. i=1 Prove this lemm (for exmple y inuction) s n exercise. With this lemm, we hve now the require tools to show tht Huffmn coing is lwys within 1 it per letter of the Shnnon oun. 6

7 Theorem 1 Let A e n lphet, n let p for A enote the proility of occurrence of letter A. Then, Huffmn coing gives n expecte length L per letter stisfying: H(p) L H(p) + 1. Proof: The first inequlity follows from Shnnon s noiseless coing theorem. To prove the secon inequlity, efine l = log 2 (p ), for A. Notice tht l is n integer for every. Furthermore, we hve tht p = 1. A 2 l = A 2 log2(p) A 2 log2 p = A Thus, y Krft s inequlity, there exists inry tree with A leves n the corresponing prefix tree hs string of length l for A. As we hve rgue tht the Huffmn coe gives the smllest possile expecte length, we get tht L p l A = p log 2 (p ) A p (1 log 2 p ) A = H(p) + 1. This proves the theorem. 7