Unicode Demystified. A Practical Programmer s Guide to the Encoding Standard. by Richard Gillam

Transcription

1 Unicode Demystified A Practical Programmer s Guide to the Encoding Standard by Richard Gillam

2 Copyright by Richard T. Gillam. All rights reserved. Pre-publication draft number Tuesday, January 15, 2002

3 To Mark, Kathleen, Laura, Helena, Doug, John F, John R, Markus, Bertrand, Alan, Eric, and the rest of the old JTCSV Unicode crew, without whom this book would not have been possible and To Ted and Joyce Gillam, without whom the author would not have been possible

4

5 Table of Contents Table of Contents v Preface xv About this book xvi How this book was produced xviii The author s journey xviii Acknowledgements xix A personal appeal xx Unicode in Essence An Architectural Overview of the Unicode Standard 1 CHAPTER 1 Language, Computers, and Unicode 3 What Unicode Is 6 What Unicode Isn t 8 The challenge of representing text in computers 10 What This Book Does 14 How this book is organized 15 Section I: Unicode in Essence 15 Section II: Unicode in Depth 16 Section III: Unicode in Action 17 CHAPTER 2 A Brief History of Character Encoding 19 Prehistory 19 The telegraph and Morse code 20 v

6 Table of Contents The teletypewriter and Baudot code 21 Other teletype and telegraphy codes 22 FIELDATA and ASCII 23 Hollerith and EBCDIC 24 Single-byte encoding systems 26 Eight-bit encoding schemes and the ISO 2022 model 27 ISO Other 8-bit encoding schemes 29 Character encoding terminology 30 Multiple-byte encoding systems 32 East Asian coded character sets 32 Character encoding schemes for East Asian coded character sets 33 Other East Asian encoding systems 36 ISO and Unicode 36 How the Unicode standard is maintained 41 CHAPTER 3 Architecture: Not Just a Pile of Code Charts 43 The Unicode Character-Glyph Model 44 Character positioning 47 The Principle of Unification 50 Alternate-glyph selection 53 Multiple Representations 54 Flavors of Unicode 56 Character Semantics 58 Unicode Versions and Unicode Technical Reports 60 Unicode Standard Annexes 60 Unicode Technical Standards 61 Unicode Technical Reports 61 Draft and Proposed Draft Technical Reports 61 Superseded Technical Reports 62 Unicode Versions 62 Unicode stability policies 63 Arrangement of the encoding space 64 Organization of the planes 64 The Basic Multilingual Plane 66 The Supplementary Planes 69 Non-Character code point values 72 Conforming to the standard 73 General 74 Producing text as output 75 Interpreting text from the outside world 75 Passing text through 76 Drawing text on the screen or other output devices 76 Comparing character strings 77 Summary 77 CHAPTER 4 Combining character sequences and Unicode normalization 79 How Unicode non-spacing marks work 81 vi Unicode Demystified

7 Dealing properly with combining character sequences 83 Canonical decompositions 84 Canonical accent ordering 85 Double diacritics 87 Compatibility decompositions 88 Singleton decompositions 90 Hangul 91 Unicode normalization forms 93 Grapheme clusters 94 CHAPTER 5 Character Properties and the Unicode Character Database 99 Where to get the Unicode Character Database 99 The UNIDATA directory 100 UnicodeData.txt 103 PropList.txt 105 General character properties 107 Standard character names 107 Algorithmically-derived names 108 Control-character names 109 ISO comment 109 Block and Script 110 General Category 110 Letters 110 Marks 112 Numbers 112 Punctuation 113 Symbols 114 Separators 114 Miscellaneous 114 Other categories 115 Properties of letters 117 SpecialCasing.txt 117 CaseFolding.txt 119 Properties of digits, numerals, and mathematical symbols 119 Layout-related properties 120 Bidirectional layout 120 Mirroring 121 Atabic contextual shaping 122 East Asian width 122 Line breaking property 123 Normalization-related properties 124 Decomposition 124 Decomposition type 124 Combining class 126 Composition exclusion list 127 Normalization test file 127 Derived normalization properties 128 Grapheme-cluster-related properties 128 Unihan.txt 129 A Practical Programmer s Guide to the Encoding Standard vii

8 Table of Contents CHAPTER 6 Unicode Storage and Serialization Formats 131 A historical note 132 UTF UTF-16 and the surrogate mechanism 134 Endian-ness and the Byte Order Mark 136 UTF CESU UTF-EBCDIC 141 UTF Standard Compression Scheme for Unicode 143 BOCU 146 Detecting Unicode storage formats 147 Unicode in Depth A Guided Tour of the Character Repertoire 149 CHAPTER 7 Scripts of Europe 151 The Western alphabetic scripts 151 The Latin alphabet 153 The Latin-1 characters 155 The Latin Extended A block 155 The Latin Extended B block 157 The Latin Extended Additional block 158 The International Phonetic Alphabet 159 Diacritical marks 160 Isolated combining marks 164 Spacing modifier letters 165 The Greek alphabet 166 The Greek block 168 The Greek Extended block 169 The Coptic alphabet 169 The Cyrillic alphabet 170 The Cyrillic block 173 The Cyrillic Supplementary block 173 The Armenian alphabet 174 The Georgian alphabet 175 CHAPTER 8 Scripts of The Middle East 177 Bidirectional Text Layout 178 The Unicode Bidirectional Layout Algorithm 181 Inherent directionality 181 Neutrals 184 Numbers 185 The Left-to-Right and Right-to-Left Marks 186 The Explicit Embedding Characters 187 viii Unicode Demystified

9 Mirroring characters 188 Line and Paragraph Boundaries 188 Bidirectional Text in a Text-Editing Environment 189 The Hebrew Alphabet 192 The Hebrew block 194 The Arabic Alphabet 194 The Arabic block 199 Joiners and non-joiners 199 The Arabic Presentation Forms B block 201 The Arabic Presentation Forms A block 202 The Syriac Alphabet 202 The Syriac block 204 The Thaana Script 205 The Thaana block 207 CHAPTER 9 Scripts of India and Southeast Asia 209 Devanagari 212 The Devanagari block 217 Bengali 221 The Bengali block 223 Gurmukhi 223 The Gurmukhi block 225 Gujarati 225 The Gujarati block 226 Oriya 226 The Oriya block 227 Tamil 227 The Tamil block 230 Telugu 230 The Telugu block 232 Kannada 232 The Kannada block 233 Malayalam 234 The Malayalam block 235 Sinhala 235 The Sinhala block 236 Thai 237 The Thai block 238 Lao 239 The Lao block 240 Khmer 241 The Khmer block 243 Myanmar 243 The Myanmar block 244 Tibetan 245 The Tibetan block 247 The Philippine Scripts 247 CHAPTER 10 Scripts of East Asia 251 The Han characters 252 A Practical Programmer s Guide to the Encoding Standard ix

10 Table of Contents Variant forms of Han characters 261 Han characters in Unicode 263 The CJK Unified Ideographs area 267 The CJK Unified Ideographs Extension A area 267 The CJK Unified Ideographs Extension B area 267 The CJK Compatibility Ideographs block 268 The CJK Compatibility Ideographs Supplement block 268 The Kangxi Radicals block 268 The CJK Radicals Supplement block 269 Indeographic description sequences 269 Bopomofo 274 The Bopomofo block 275 The Bopomofo Extended block 275 Japanese 275 The Hiragana block 281 The Katakana block 281 The Katakana Phonetic Extensions block 281 The Kanbun block 281 Korean 282 The Hangul Jamo block 284 The Hangul Compatibility Jamo block 285 The Hangul Syllables area 285 Halfwidth and fullwidth characters 286 The Halfwidth and Fullwidth Forms block 288 Vertical text layout 288 Ruby 292 The Interlinear Annotation characters 293 Yi 294 The Yi Syllables block 295 The Yi Radicals block 295 CHAPTER 11 Scripts from Other Parts of the World 297 Mongolian 298 The Mongolian block 300 Ethiopic 301 The Ethiopic block 303 Cherokee 303 The Cherokee block 304 Canadian Aboriginal Syllables 304 The Unified Canadian Aboriginal Syllabics block 305 Historical scripts 305 Runic 306 Ogham 307 Old Italic 307 Gothic 308 Deseret 309 CHAPTER 12 Numbers, Punctuation, Symbols, and Specials 311 Numbers 311 x Unicode Demystified