Introduction to Unicode and Writing Systems

Transcription

1 Introduction to Unicode and Writing Systems Denis Kiryaev If you are a programmer working in 2003 and you don't know the basics of characters, character sets, encodings, and Unicode, and I catch you, I'm going to punish you by making you peel onions for 6 months in a submarine. I swear I will. Joel Spolsky, Joel on Software 1

2 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 2

3 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 3

4 ASCII 4

5 Windows

6 6

7 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 7

8 Unicode as a standard Characters table Encodings Rendering rules & Algorithms 8

9 Private Use Area characters Supplementary Ideographic Plane (SIP) Supplementary Multilingual Plane (SMP) 17 planes characters each Basic Multilingual Plane (BMP) 9

10 10

11 Encodings UTF-8 UTF-16 UTF-32 Character set, codepage and encoding 11

12 UTF-8 0 Invalid bytes in UTF-8 sequence: C0, C1, F5 FF 12

13 UTF-8 example Character Binary code point Binary UTF-8 Hexadecimal UTF-8 $ U U+00A C2 A2 U+20AC E2 82 AC 𤭢 U+24B F0 A4 AD A2 13

14 UTF-16 Codepoint range U U+D7FF U+E000 - U+FFFF U U+10FFFF U+D800 - U+DFFF Encoding scheme 16-bit integer equal to codepoint number Surrogate pair: Lead surrogate = 0xD800 + ((code point - 0x10000) >> 10 Trail surrogate = 0xDC00 + (code point & 0x3FF) Reserved for UTF-16 surrogates 14

15 UTF-16 example Code point Glyph character UTF-16 code units (hex) U+007A Latin small letter z 007A z U+6C34 U U+1D11E CJK unified ideograph-6c34 (water) Linear B syllable B008 A (first non-bmp code point) Musical symbol G clef 6C34 D800 DC00 D834 DD1E U+10FFFD Private use character- 10FFFD (last Unicode code point) DBFF DFFD 15

16 Byte order mark (BOM) H e l l o Intel x C 00 6C 00 6F System/ C 00 6C 00 6F 00 Byte order mark (BOM) = U+FEFF (zero-width non-breaking space) U+FFFE reserved not a character BOM H e l l o Intel x86 FF FE C 00 6C 00 6F System/360 FE FF C 00 6C 00 6F 00 UTF-16BE and UTF-16LE IANA approved encoding names 16

17 UTF-32 The most simple fixed length The most redundant BOM is also applicable Not recommended in HTML 5 standard 17

18 Character set, codepage and encoding Pre-Unicode era Character set = codepage = charmap = encoding Unicode era Codepage = charmap = legacy encoding Defines table encoding for locale-specific Unicode subset Character set Means whole Unicode repertoire For legacy encodings means Unicode subset encoded by that encoding Encoding One of the Unicode encodings 18

19 Rendering rules & Algorithms Unlike Russian and English, most languages in the world use complex scripting Rendering complexities: Diacritics Ligatures Right-to-left and other writing directions Sorting complexities: Diacritics Ligatures Ideographs 19

20 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 20

21 Writing systems - French Père Noël 21

22 Precomposed and decomposed forms Precomposed form: P + è + r + e N + o + ë + l Decomposed form: P + e + + r + e N + o + e + + l Both forms are equivalent 22

23 Normalization forms NFD Normalization Form Canonical Decomposition NFC Normalization Form Canonical Composition NFKD Normalization Form Compatibility Decomposition NFKC Normalization Form Compatibility Composition 23

24 Compatibility ff (U+FB00) f+f (U+0066), but compatible ß (U+00DF) s+s (U+0073), but compatible Ⅻ (U+216B) compatible to Ⅹ (U+2169) + Ⅰ (U+2160) + Ⅰ 24

25 Canonical ordering e = e + (U+0301) + (U+031C) or e + (U+031C) + (U+0301) e + (U+031C, CCC=220) + (U+0301, CCC=230) e = e + (U+0303, CCC=230) + (U+0300, CCC=230) e = e + (U+0300, CCC=230) + (U+0303, CCC=230) 25

26 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 26

27

28

29 Arabic script Right-to-left Cursive script each letter has 4 glyphs: initial, medial, final, and isolated No upper or lower letters Diacritics used for short vowels 29

30 Arabic short vowels 30

31 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 31

32 32

33 Chinese character types Pictographs ( 木 tree) Ideographs ( 上 up, 日 sun) Logical aggregates ( 東 east, sun rising in the trees) Phonetic complexes ( 晴 clear weather, sun + blue) 33

34 Writing rules Each character is fit into the square; complex characters are scaled as necessary Writing direction: traditional top to bottom, right to left; modern left to right Traditional writing has little or even no punctuation while modern has enough Latin characters and numbers have 2 forms: halfwidth (1) and fullwidth (1) with different codepoints 34

35 Horisontal vs. Vertical 35

36 Traditional and Simplified Chinese Simplified Chinese is a standard in PRC, Singapore, Malaysia Traditional Chinese is used in Hong Kong, Taiwan, Macau, and overseas Chinese communities Different codepoints for the same character in traditional and simplified Chinese 36

37 GB18030 PRC national standard Treated as Unicode encoding, although not a part of the standard Is a superset of double-byte code page GB2312 which in turn is a superset of ASCII GB18030 support is required for software sold in China 37

38 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 38

39 Writing systems - Japanese Mix of 3 scripts: Kanji (Chinese characters) Hiragana (syllabary used for grammatical elements) Katakana (syllabary used for foreign words and names, etc.) Traditional writing direction same as Chinese, modern left to right Sorting is based on kana 39

40 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 40

41 Writing systems - Korean 2 scripts: Hanja Chinese characters Hangul alphabet created in 1446 ㅂㄷㅈㄱㅃㄸㅉㄲㅍㅌㅊㅋㅅㅎㅆㅁㄴㅇㄹㅣㅔㅚㅐㅏㅗㅜㅓㅡㅢㅖㅒㅑㅛㅠㅕㅟㅞㅙㅘㅝ 41

42 Korean hangul syllables ㅂㄷㅈㄱㅃㄸㅉㄲㅍㅌㅊㅋㅅㅎㅆㅁㄴㅇㄹ ㅣㅔㅚㅐㅏㅗㅜㅓㅡㅢㅖㅒㅑㅛㅠㅕㅟㅞㅙㅘㅝ 다국어 42

43 Han unification One codepoint for the same ideograph in Chinese, Japanese, Korean Simplified Chinese is not unified Font is different for each of the languages resulting in localized rendering characters assigned Many disadvantages of unification; proposals exist for de-unification (e.g. ISO/IEC 2022) 43

44 Ideographs sorting in Han Radical ideograph component 214 radicals defined in Unicode now 44

45 Input Method Editor 4 main methods: Typing latin translit of the character Typing local translit (kana Japan, pinyin China) Drawing shape with mouse/pen, IME recognizes character 45

46 Agenda Introduction Unicode as a standard Writing Systems, or the fun begins French Arabic Chinese Japanese Korean Hindi 46

47 Hindi combining characters 47

48 Ligatures in Hindi 48

49 More info Internationalization & Unicode conference (Santa Clara, CA) Wikipedia (English version only!) has good simple articles about Unicode 49

50 THANK YOU Denis Kiryaev, EMC 50