More Bits and Bytes Huffman Coding. Natural Language and Dialogue Systems Lab

Transcription

1 More Bits and Bytes Huffman Coding Natural Language and Dialogue Systems Lab

2 Announcements Privacy essay deadline extended to 5PM Saturday To be assigned a week from today, a one week creativity pair programming assignment. FIND A PARTNER for your PAIR Winter2//pages/syllabus 3 homeworks this week: 2 processing homeworks, one easy (Thurs), one harder (Tues), One principles homework, due next Thursday. Ready today. Practice kinds of problems that will be on the midterm/quiz Next Tuesday: Social Media guest lecture There will be midterm questions on that lecture

3 Announcements (Repeat) If you have a question on your homework please ask Gabby or Chao if you want a v. timely response Ask the person whose section you signed up for: Gabby T/Th ghalberg@soe.ucsc.edu Chao M/W zhu@soe.ucsc.edu Questions about grading: ask Gabby but CC me.

4 Next Processing Homework: DUE THURS! Write your own code from scratch, not just change existing code. But lots of hints and instructions in the homework PDF

5 Brief Review: How do Computers Compute? Natural Language and Dialogue Systems Lab

6 MP Neuron for LOGICAL AND X + Y + =2 IS *X + *Y 2?? X AND Y

7 Fundamental units of computers: Logic Gates

8 We can read it off the truth Table for XOR P Q P xor Q

9 Logical XOR Neuron exists? How long do we keep looking for a solution? We need to be able to calculate the weights, not just keep looking for the answer by trial and error. Each possible pair of inputs corresponds to an equation (a linear inequality) for the output in terms of the inputs, the weights and the threshhold. E.g. for AND it was IS *X + *Y 2?? These can be used to compute the weights and thresholds. Equations for XOR are incompatible => MP Neuron for XOR can t be built.

10 What would you ever want XOR for anyway? Binary Addition (will do some of this after the MIDTERM)

11 Binary combinations, True/False possibilities One bit Three bits Four bits?????? Two Bits??????????

12 ASCII C A T /25/2 2 Lawrence Snyder, CSE

13 With 8 bits how many different letters? 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 => 2 8 = 256

14 How do Computers represent data? Natural Language and Dialogue Systems Lab

15 Encoding Text: How is it done? ASCII, UTF, Huffman algorithm Natural Language and Dialogue Systems Lab

16 ASCII C A T /26/2 2 Lawrence Snyder, CSE

17 With 8 places how many different letters?

18 UTF-8: All the alphabets in the world Uniform Transformation Format: a variable- width encoding that can represent every character in the Unicode Character set,2,64 of them!!! 8 UTF- 8 is the dominant character encoding for the World- Wide Web, accounting for more than half of all Web pages. The Internet Engineering Task Force (IETF) requires all Internet protocols to identify the encoding used for character data The supported character encodings must include UTF- 8.

19 UTF is a VARIABLE LENGTH ALPHABET CODING Remember ASCII can only represent 256 characters (8 bits) UTF encodes over M Why would you want a variable length coding scheme?

20 Cyrillic vs. English

21 Huffman Coding: Natural Language and Dialogue Systems Lab

22 Coding can be used to do Compression What is CODING? The conversion of one representation into another What is COMPRESSION? Change the representation (digitization) in order to reduce size of data (number of bits needed to represent data) Benefits Reduce storage needed Consider growth of digitized data. Reduce transmission cost / latency / bandwidth When you have a 56K dialup modem, every savings in BITS counts, SPEED Also consider telephone lines, texting

23 What makes it possible to do Compression? IN OTHER WORDS: When is Coding USEFUL? When there is Redundancy Recognize repeating patterns Exploit using Dictionary Variable length encoding When Human perception is less sensitive to some information Can discard less important data

24 How easy is it to do it? Depends on data Random data hard Example:? Organized data easy Example: WHAT DOES THAT MEAN? There is NO universally best compression algorithm It depends on how tuned the coding is to the data you have

25 Can you lose information with Compression? Lossless Compression is not guaranteed Pigeonhole principle Reduce size bit can only store ½ of data Example,,,,,,,,,, CONSIDER THE ALTERNATIVE IF LOSSLESS COMPRESSION WERE GUARANTEED THEN Compress file (reduce size by bit) Recompress output Repeat (until we can store data with bits) OBVIOUS CONTRADICTION => IT IS NOT GUARANTEED.

26 Huffman Code: A Lossless Compression Use Variable Length codes based on frequency (like UTF does) Approach Exploit statistical frequency of symbols What do I MEAN by that? WE COUNT!!! HELPS when the frequency for different symbols varies widely Principle Use fewer bits to represent frequent symbols Use more bits to represent infrequent symbols A A B A A A B A

27 Huffman Code Example dog cat cat bird bird bird bird fish Symbol Dog Cat Bird Fish Frequency /8 /4 /2 /8 Original Encoding Huffman Encoding 2 bits 2 bits 2 bits 2 bits 3 bits 2 bits bit 3 bits Expected size Original /8 2 + /4 2 + /2 2 + /8 2 = 2 bits / symbol Huffman /8 3 + /4 2 + /2 + /8 3 =.75 bits / symbol DOES EVERYONE SEE THIS?? ASK ME A QUESTION.

28 Huffman Code Algorithm: Data Structures Binary (Huffman) tree Represents Huffman code Edge code ( or ) Leaf symbol Path to leaf encoding Example A =, H =, C = Good when??? A, H less frequent than C in messages Want to efficiently build a binary tree Also showed you binary tree for ASCII A H C

29 Huffman Code Algorithm Overview Order the symbols with least frequent first (will explain) Build a tree piece by piece Encoding Calculate frequency of symbols in the message, language JUST COUNT AND DIVIDE BY TOTAL NUMBER OF SYMBOLS Create binary tree representing best encoding Use binary tree to encode compressed file For each symbol, output path from root to leaf Size of encoding = length of path Save binary tree

30 Huffman Code Creating Tree Algorithm (Recipe) Place each symbol in leaf Weight of leaf = symbol frequency Select two trees L and R (initially leafs) Such that L, R have lowest frequencies in tree Which L, R have the lowest number of occurrences in the message? Create new (internal) node Left child L Right child R New frequency frequency( L ) + frequency( R ) Repeat until all nodes merged into one tree

31 Huffman Tree Construction A C E H I

32 Huffman Tree Step 2: can first re-order by frequency A H C E I

33 Huffman Tree Construction 3 A H E I C 5 5

34 Huffman Tree Construction 4 A H E I C 5 5

35 Huffman Tree Construction 5 A 3 5 H 2 C E I 7 E = I = C = A = H = 25

36 Huffman Coding Example Huffman code Input ACE Output ()()() = E = I = C = A = H =

37 Huffman Code Algorithm Overview Decoding Read compressed file & binary tree Use binary tree to decode file Follow path from root to leaf

38 Huffman Decoding A 3 H 2 C E I

39 Huffman Decoding 2 A 3 H 2 C E I

40 Huffman Decoding 3 A 3 H 2 C E I A 5 25

41 Huffman Decoding 4 A 3 H 2 C E I A 5 25

42 Huffman Decoding 5 A 3 H 2 C E I AC 5 25

43 Huffman Decoding 6 A 3 H 2 C E I AC 5 25

44 Huffman Decoding 7 A 3 H 2 C E I ACE 5 25

45 Huffman Code Properties Prefix code No code is a prefix of another code Example Huffman( dog ) Huffman( cat ) // not legal prefix code Can stop as soon as complete code found No need for end-of-code marker Nondeterministic Multiple Huffman coding possible for same input If more than two trees with same minimal weight

46 Huffman Code Properties Greedy algorithm Chooses best local solution at each step Combines 2 trees with lowest frequency Still yields overall best solution Optimal prefix code Based on statistical frequency Better compression possible (depends on data) Using other approaches (e.g., pattern dictionary)

47 Huffman Coding. Another example. Natural Language and Dialogue Systems Lab

48 Huffman Tree Example 2. Step T O B E R TO BE OR NOT TO BE T = 3 O = 4 B = 2 E = 2 R =

49 Huffman Tree: TO BE OR NOT TO BE E R B T O

50 Huffman Tree 3: TO BE OR NOT TO BE E R T O B 3 2 5

51 Huffman Tree 4: TO BE OR NOT TO BE E R T O B 2 7 5

52 Huffman Tree Construction 5 E 2 3 R B T O 4 E = R = B = T = O = 2 2 NUMBER OF LETTERS IN MESSAGE

53 Huffman Tree 5: TO BE E 2 3 R B T O 4 E = R = B = T = O = 2...N..

54 How Much For A Fixed Length Code? E 2 3 R B T O 4 E = R = B = T = O = 2...N.. 28 BITS HERE

55 Huffman Code Algorithm Overview Decoding Read compressed file & binary tree Use binary tree to decode file Follow path from root to leaf FIRST EXAMPLE. ACE

56 Huffman Decoding A 3 H 2 C E I

57 Huffman Decoding 2 A 3 H 2 C E I

58 Huffman Decoding 3 A 3 H 2 C E I A 5 25

59 Huffman Decoding 4 A 3 H 2 C E I A 5 25

60 Huffman Decoding 5 A 3 H 2 C E I AC 5 25

61 Huffman Decoding 6 A 3 H 2 C E I AC 5 25

62 Huffman Decoding 7 A 3 H 2 C E I ACE 5 25

63 DECODING: 2 nd example BEBE E 2 3 R B T O 4 E = R = B = T = O = 2 BEBE = ORIGINAL 6 BITS FOR BEBE

64 Huffman Code Properties Prefix code No code is a prefix of another code Example Huffman( dog ) Huffman( cat ) // not legal prefix code Can stop as soon as complete code found No need for end-of-code marker Nondeterministic Multiple Huffman coding possible for same input If more than two trees with same minimal weight

65 Huffman Code Properties Greedy algorithm Chooses best local solution at each step Combines 2 trees with lowest frequency Still yields overall best solution Optimal prefix code Based on statistical frequency Better compression possible (depends on data) But needs look ahead, not prefix.

66 Encoding Information: There s more! Bits and bytes encode the information, but that s not all Tags encode format and some structure in word processors Tags encode format and some structure in HTML Tags are one form of meta- data Meta- data is information about information We will return to this when we talk about HTML and the WEB in Week 7.