Lecture 1: Encoding Language. LING 1330/2330: Introduction to Computational Linguistics Na-Rae Han

Transcription

1 Lecture 1: Encoding Language LING 1330/2330: Introduction to Computational Linguistics Na-Rae Han

2 Objectives Understand the fundamentals of how language is encoded on a computer Text encoding systems ASCII ISO-8859 Unicode 1/12/2017 2

3 How is language represented on a computer? Natural ("Human") languages: Spoken form Written form *Also: sign languages The language of computers: 1/12/2017 3

4 The language of computers At the lowest level, computer language is binary: Information on a computer is stored in bits A bit is either: ON (=1, =yes) or OFF (=0, =no) This language essentially contains two alphabetic characters Next level up: byte A byte is made up of a sequence of 8 bits ex Historically, a byte was the number of bits used to encode a single character of text in a computer Byte is a basic addressable unit in most computer architecture 1/12/2017 4

5 Encoding a written language How to represent a text with 0s and 1s? Hello world! Each character is mapped to a code point (=character code), e.g., a unique integer. H 72 dec e 101 dec Each code point is represented as a binary number, using a fixed number of bits. 8 bits == 1 byte in the example above H 72 dec ( = = 72) e 101 dec ( = = 101) One byte can represent 256 (=2 8 ) different characters dec dec 1/12/2017 5

6 ASCII encoding for English How many bits are needed to encode English? 26 lowercase letters: a, b, c, d, e, 26 uppercase letters: A, B, C, D, E, 10 Arabic digits: 0, 1, 2, 3, 4, Punctuation:., : ;?! ' " Symbols: ( ) < > & % * $ + - We are already up to 80 6 bits (2 6 = 64) is not enough; we will need at least 7 (2 7 = 128) ASCII (the American Standard Code for Information Interchange) did just that Uses 7-bit code (= 128 characters) for storing English text Range 0 to 127 1/12/2017 6

7 The ASCII chart CII%20Conversion%20Chart.pdf Decimal Binary (7-bit) Character Decimal Binary (7-bit) Character (NULL) A B # C & a b c (DEL) 1/12/2017 7

8 ASCII (the American Standard Code for Information Interchange) The ASCII encoding scheme First published in 1963 Uses 7-bit code (= 128 characters) for storing English text, ranging from 0 to 127 In an 8-bit (1 byte) representation, the highest bit is always 0 Printable characters Upper and lower case roman alphabet Digits Punctuation marks, symbols, and space Includes 32 non-printing characters Control characters: BELL, ACKNWOLEDGE, BACKSPACE, DELETE, etc. originally for typewriters, many obsolete now WHITESPACE characters: TAB, LINE FEED, CARRIAGE RETURN 1/12/2017 8

9 Practice What is this English text? Note: byte (=8-bit) ASCII representation instead of 7-bit Space provided for your convenience only! Answer: Hi! 1/12/2017 9

10 Extending ASCII: ISO-8859, etc. ASCII (=7 bit, 128 characters) was sufficient for encoding English. But what about characters used in other languages? Solution: Extend ASCII into 8-bit (=256 characters) and use the additional 128 slots for non-english characters ISO-8859: has 16 different implementations! ISO ISO ISO aka Latin-1: French, German, Spanish, etc. Greek alphabet Hebrew alphabet JIS X 0208: Japanese characters Problem: overlapping character code space. 224 dec means à in Latin-1 but א in ISO ! 1/12/

11 The problem with multiple encoding systems Problem: Multiple coding systems map different characters to the same character code Solution 1: Provide meta-information on coding system Ex. MIME (Multipurpose Internet Mail Extensions) Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit But what if your message contains characters from multiple coding systems? Solution 2: Have a single universal code system for all writing systems UNICODE 1/12/

12 Unicode A character encoding standard developed by the Unicode Consortium Provides a single representation for all world's writing systems "Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language. ( 1/12/

13 How big is Unicode? Version 9.0 (2016) has codes for 128,237 characters Full Unicode standard uses 32 bits (4 bytes) : it can represent 2 32 = 4,294,967,296 characters! In reality, only 21 bits are needed Unicode has three encoding versions UTF-32 (32 bits/4 bytes): direct representation UTF-16 (16 bits/2 bytes): 2 16 =65,536 possibilities UTF-8 (8 bits/1 byte): 2 8 =256 possibilities 1/12/

14 8-bit, 16-bit, 32-bit UTF-32 UTF-16 UTF-8 (32 bits/4 bytes): direct representation (16 bits/2 bytes): 2 16 =65,536 possibilities (8 bits/1 byte): 2 8 =256 possibilities Wait! But how do you represent all of 2 32 (=4 billion) code points with only one byte (UTF-8: 2 8 =256 slots)? You don't. In reality, only 2 21 bits are ever utilized for 128K characters. UTF-8 and UTF-16 use a variable-width encoding. Why UTF-16 and UTF-8? They are more compact (more so for certain languages, i.e., English) 1/12/

15 Variable-width encoding 'H' as 1 byte (8 bits): cf. 'H' as 2 bytes (16 bits): UTF-8 as a variable-width encoding ASCII characters get encoded with just 1 byte ASCII is originally 7-bits, so the highest bit is always 0 in an 8-bit encoding All other characters are encoded with multiple bytes How to tell? The highest bit is used as a flag. Highest bit 0: single character Highest bit 1: part of a multi-byte character Advantage for English: 8-bit ASCII is already a valid UTF-8! É 1/12/

16 A look at Unicode chart How to find your Unicode character: Basic Latin (ASCII) 1/12/

17 Code point for M. But "004D"? 1/12/

18 Another representation: hexadecimal Hexadecimal (hex) = base-16 Utilizes 16 characters: A B C D E F Designed for human readability & easy byte conversion 2 4 =16: 1 hexadecimal digit is equivalent to 4 bits 1 byte (=8 bits) is encoded with just 2 hex chars! Letter Base-10 (decimal) Base-2 (binary) Base-16 (hex) M D Unicode characters are usually referenced by their hexadecimal code Lower-number characters go by their 4-char hex codes (2 bytes), e.g. U+004D ("M", U+ designates Unicode) Higher-number characters go by 5 or 6 hex codes, e.g. U+1D122 ( 1/12/