Introduction to Unicode. By: Atif Gulzar Center for Research in Urdu Language Processing

Transcription

1 Introduction to Unicode By: Atif Gulzar Center for Research in Urdu Language Processing

2 Introduction to Unicode Unicode Why Unicode? What is Unicode? Unicode Architecture

3 Why Unicode?

4 Pre-Unicode Standards and their Limitations ASCII by ANSI in 1964 (7 bit code) ISO adopt ASCII in 1967 as ISO 646 ISO 2022 (8 bit code) ISO 8859 ISO 8859 is a family of 16 Standards Code Page And plenty of standards for East Asian languages

5 ISO 8859 cont. ISO , Latin-1, Western European ISO , Latin-2, Eastern European ISO , Latin-3, Southern European ISO , Latin-4, Northern European ISO , 5, Cyrillic, Russian, Bulgarian.. ISO , Arabic, Arabic ISO , Greek, Greek ISO , 8, Hebrew, Hebrew

6 ISO 8859 ISO , 9, Latin-5, Turkish ISO , Latin-6, Northern European ISO , Thai, Thai ISO , Latin-7, Baltic ISO , Latin-8, Celtic ISO , Latin-9, Western European ISO , Latin-10, Eastern European

7 ASCII Upper case (A-Z) 26 Digits (0-9) 10 Space 1 Punctuation marks (.,+{)%) 32 Lower case (a-z) 26 Control characters (tab, cr, lf) 33 ====================== Total 128

8 ASCII code page

9 ANSI 1252 code page

10 ANSI 1256 code page Arabic

11 ANSI 1252 code page Central Europe Arabic Č ب È 00C8 Hebrew Greek Θ Cyrillic Thai И ศ

12 Code page for Windows Characters not common to other codepages

13 The Code Page Problem cont. Each character set: ASCII (common) + ANSI (Western Europe) Eastern Europe Baltic Greek Cyrillic Thai, Turkish, Arabic, Hebrew, etc. ASCII + extended Extensions for many countries Characters above 128 change meaning

14 The Code Page Problem cont. Characters in most languages are traditionally represented by single-byte values Allows for 256 characters max Real limit for most encodings is 192 characters This includes letters, digits, punctuation, symbols When a system is used for a new language, the encoding has to be adapted to use that language s characters

15 The Code Page Problem Each language or group of languages gets its own encoding Different vendors or standards committees devise different encodings, so generally each language has several, often incompatible, encodings

16 Interoperability Problems Can t t easily mix languages in a document or system Data not tagged with encoding, so loss can occur when transferring between systems Most encodings are ASCII-based, so problems often not seen with English-only data Two possible solutions: Systematic tagging of textual data with encoding ID Universal encoding standard with all languages characters

17 What is Unicode?

18 Unicode or Universal Code One Universal Code for every character no matter what the platform, no matter what the program, no matter what the language. Unicode is not just a bunch of code points Initially it was a 2 byte code, that can support over 65,000 characters Unicode Standard, Version 4.0 provides codes for 96,447 characters Adopted by ISO as ISO 10464

19 Principles of the Unicode Standard Universality Efficiency Characters, not glyphs Semantics Plain text Logical order Unification Convertibility Accurate

20 Universality (Unicode Coverage) European scripts Latin, Greek, Cyrillic, Armenian, Georgian, IPA Bidirectional (Middle Eastern) scripts Hebrew, Arabic, Syriac,, Thaana Indic (Indian and Southeast Asian) scripts Devanagari,, Bengali, Gurmukhi,, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Sinhala,, Thai, Lao, Khmer, Myanmar, Tibetan, Philippine East Asian scripts Chinese (Han) characters, Japanese (Hiragana and Katakana), Korean (Hangul), Yi Other modern scripts Mongolian, Ethiopic, Cherokee, Canadian Aboriginal Historical scripts Runic, Ogham,, Old Italic, Gothic, Deseret Punctuation and symbols Numerals, math symbols, scientific symbols, arrows, blocks, geometric shapes, Braille, musical notation, etc.

21 Characters and Glyphs fi fi لا ل ا -->

22 Plain Text Its is a plain text Its is a formatted text

23 Logical Data Ordering and Bidi- Algorithm

24 Unification Unicode standard avoid duplicate encoding of characters within Scripts across languages. In Chinese, Japanese and Korean many ideographs are common. The character code U+0057 Y is same in English, German and French

25 Character Semantics cont. The Unicode standard includes an extensive database that specifies a large number of character properties, including: Name Type (e.g., letter, digit, punctuation mark) Decomposition Case and case mappings (for cased letters) Numeric value (for digits and numerals) Combining class (for combining characters) Directionality Line-breaking behavior Cursive joining behavior For Chinese characters, mappings to various other standards and many other properties

26 Character semantics 1781; KHMER LETTER KHA;Lo;0;L;;;;;N;;;;; 17BE KHMER VOWEL SIGN OE;Mc;0;L;;;;;N;;;;; 17E5 KHMER DIGIT FIVE;Nd;0;L;;5;5;5;N;;;;;

27 Unicode Architecture

28 Unicode Architecture Initially Unicode was designed for 16-bit encoding space, consisting of 256 rows of 256 characters each ISO was designed for 32 bit encoding space, thus ISO has room for 2,147,483,648 characters After Unicode came out that 16-bit encoding is too small In Unicode 3.0 the length is increased to 21-bit, allows for 1,114,112 characters

29 Encoding Space Early versions of Unicode used 16 bits Unicode now uses 21 bits Plane number Row number Character number

30 The Unicode Encoding Space 10 F E D C B A Basic Multilingual Plane

31 The Unicode Encoding Space 10 F E D C B A Supplementary Planes

32 The Unicode Encoding Space 10 F E D C B A Private Use Planes Supplementary Special-Purpose Plane Supplementary Ideographic Plane Supplementary Multilingual Plane

33 The Basic Multilingual Plane A B C D E F General Scripts Area Symbols Area CJK Punct. Yi Private Use Area Han Hangul Surrogates Area CJK Punct. Compatibility Area

34 The General Scripts Area 00/01 02/03 04/05 06/07 08/09 0A/0B 0C/0D 0E/0F 10/11 12/13 14/15 16/17 18/19 1A/1B 1C/1D 1E/1F Latin IPA Diacriticals Greek Cyrillic Armenian Hebrew Arabic Syriac Thaana Devanagari Bengali Gurmukhi Gujarati Oriya Tamil Telugu Kannada Malayalam Sinhala Thai Lao Tibetan Myanmar Georgian Hangul Ethiopic Cherokee Canadian Aboriginal Syllabics Ogh Khmer am Runic Philippine Mongolian Latin Greek

35 Unicode Storage Formats or UTF-32 UTF-16 UTF-8 UTF-7 CESU-8 UTF-EBCDIC BOCU Unicode Encodings

36 Storage formats cont. UTF-32: The 21-bit abstract Unicode value is simply zero-padded to 32 bits:

37 UTF-16: Storage formats For characters in the BMP, the 21-bit value is simply truncated to 16 bits: For other characters, the 21-bit value is turned into a sequence of two 16-bit values called a surrogate pair: A particular numeric value is either a BMP character, a high surrogate, or a low surrogate.

38 UTF-8: Storage formats For ASCII characters, the 21-bit value is truncated to 8 bits: For other characters, the 21-bit value is turned into a sequence of two, three, or four 8-bit values: Different numeric ranges are used for ASCII characters and leading and trailing bytes. Different ranges are used for leading bytes of different-length sequences.

39 Detecting Unicode Storage Format If the files starts with 0xFE 0xFF 0xFF 0xFE 0x00 0x00 0xFE 0xFF 0xFF 0xFE 0x00 0x00 0xEF 0xBB 0x BF 0xDD 0x73 0x73 0x73 0x0E 0xFE 0xFE Any thing else The file contains UTF-16 Byte-swapped UTF-16 UTF-32 Byte-swapped UTF-32 UTF-8 UTF-EBCDIC CESU Non-Unicode or Untagged Unicode

40 The Unicode standard The Unicode standard consists of: The standard text, published in book form (this includes a complete set of printed code charts) The Unicode Character Database, a set of data files providing complete property information on every character Various Web-published supplemental materials: Unicode Standard Annexes (UAX): Amendments to the standard since the last book was published Unicode Technical Standards (UTS): Allied standards maintained separately from Unicode itself Unicode Technical Reports (UTR): Non-normative normative documents providing background info, implementation hints, or other useful information Unicode Technical Notes (UTN): Other articles of

41 References The Unicode Standard Version 4.0 by Unicode Consortium Unicode Demystified by Richard Gillam Unicode Character Database ( Unicode Charts (

42 Questions?