MySQL and Unicode Daniël van Eeden Percona Live Amsterdam 23 September 2015

Transcription

1 MySQL and Unicode Daniël van Eeden Percona Live Amsterdam 23 September 2015

2 Booking.com is available in more than 40 languages So Unicode is important to us.

3

4 Also my name is Daniël, not Daniel

5 Also my name is Daniël, not Daniël

6 First some history

7 ASCII

8 Encodes characters as 7-bit

9 The 8 th bit can be used as parity, but that was never common.

10 The 3568 ASCII astroid is named after it.

11 ISO-8859

12 Uses the extra bit to be able to store a second set of 127 characters

13 The base characters (<127) are shared between ASCII and ISO-8859-?

14 The other characters differ per country/region

15 Windows-1252 (CP1252) is mostly identical with ISO

16 ISO is also known as Latin1

17 Latin1 in MySQL is not ISO , but CP1252.

18 Unicode

19 Allows you to store text in any language

20 Allows you to store text combining multiple languages in the same file

21 Each character gets a number (a.k.a. code point) and a description.

22 That doesn't guarantee your font will display it.

23 UTF-8

24 This is an character encoding for unicode.

25 This translates from code points to a binary string.

26 UTF-8 and ASCII share the same characters for 0<127.

27 Non-ASCII characters are stored as 2, 3 or 4-bytes.

28 UTF-32 UTF-16 UTF-8 ISO ASCII Baudot

29 If a byte starts with '0xxxxxxx' then it is a 1-byte character

30 If a byte starts with '110' it is a start of a 2- byte character.

31 If a byte starts with '10' then it is a continuation of a multibyte character.

32 If a byte starts with '1110' it is the start of a 3-byte character.

33 If a byte starts with '11110' it is the start of a 4-byte character.

34 Examples: a = ë =

35 UTF-8 And MySQL

36 Some reasons to use UTF-8 in MySQL

37 Non-english scripts like Chinese, Cyrillic or Greek.

38 Emoji (including in the help text of your mobile app)

39 utf8 in MySQL is an alias for utf8mb3

40 utf8mb3 can store 3-byte UTF-8

41 utf8mb4 can store 4-byte UTF-8

42 Best practice: Always use utf8mb4, don't use utf8

43 Where to set the encoding?

44 It is set on a per-column basis

45 There is a per-table default

46 There is a per-database default

47 There is a per-server default: character_set_server

48 Connections also have a character set

49 Drawbacks of UTF-8

50 So just set everything to utf8mb4?

51 It depends

52 Does your application support it?

53 CHAR(10) suddenly needs 40 bytes!

54 TINYTEXT has a size limit in bytes

55 The MEMORY storage engine expands VARCHAR(10) to 40 bytes

56 With InnoDB your index grows over 767 bytes.

57 Best practice: Use latin1 for server, database and table default. Enable Unicode on columns which need it.

58 Converting your data

59 How to convert from latin1 to utf8mb4?

60 ALTER TABLE t1 MODIFY COLUMN c1 VARCHAR(100) CHARACTER SET utf8mb4;

61 But I have many columns!

62 Use CONCAT() and information_schema to generate the statements

63 Or convert all columns: ALTER TABLE t1 CONVERT TO CHARACTER SET utf8mb4;

64 Change defaults

65 Set character_set_server

66 ALTER SCHEMA s1 DEFAULT CHARACTER SET utf8mb4;

67 ALTER TABLE t1 DEFAULT CHARACTER SET utf8mb4;

68 Common failures

69 Application looks okay, but in MySQL the data looks wrong

70 The latin1 column was holding utf8 data already

71 Wrong conversion == garbage

72 Change column to varbinary and then to utf8mb4 to not convert the data.

73 The conversion fails and eats your data

74 Use sql_mode='strict_all_tables'

75 Now the operation will fail instead of truncate your data

76 Connection set to utf8, but data is 4-byte UTF-8.

77 Collation support There is no utf8mb4_general_cs (case sensitive) There is utf8mb4_unicode_ci And utf8mb4_unicode_520_ci And utf8mb4_bin

78

79 Special collations get lost during conversion

80 ALTER TABLE CONVERT TO only supports one collation

81 Safe collation before the ALTER and then restore it for columns which have a nondefault collation.

82 Collation mismatch

83

84 Use COLLATE to set the desired collation for the operation.

85

86 é and + e are not identical

87 Unicode normalization forms NFC Composed NFD Decomposed NFKC Composed NFKD Decomposed NFK removes compatibility distinction and will lose information. But this is useful for search etc.

88 Best practice: Normalize strings in your application

89 4-byte characters get silently lost on dump/restore

90 Set utf8mb4 as default charset for the connection mysqldump uses

91

92

93

94 Did you know a BOM can be in the middle of a string?

95 Fulltext search & CJK

96 Not every character has the same width

97 Not even if we use a monospace font

98 A character can have a width of 0, 1, 2 or -1 positions

99 Punycode

100 Ligatures, Glyphs, Characters

101 Using Unicode

102 Fonts

103 Typing

104 Virtual keyboards

105 Control characters

106 Charset is not a constraint

107 Replacement characters

108 MySQL and Unicode Daniël van Eeden Percona Live Amsterdam 23 September 2015 Welcome. My name is Daniël and I work for Booking.com This presentation is about MySQL and Unicode.

109 Booking.com is available in more than 40 languages So Unicode is important to us. Booking.com is available in more that 40 languages so Unicode is of critical importance to us

110 This is the booking.com website in Arabic Note that the text is also flowing from right to left

111 Also my name is Daniël, not Daniel My name is Daniël. There are dots on the 'e'. Those are important to me.

112 Also my name is Daniël, not Daniël The image here shows examples from letters I got in the mail. This happens quite often. Also on websites. Marking every special character as illegal is not a solution.

113 First some history Let's first start with some history about character sets

114 Before ASCII: baudot (5-bit, 1870) ASCII Let's start with ASCII. ASCII was invented in 1963 to allow comunication between systems of different vendors. A little known fact is that ASCII was not made for computers, but for teleprinters. One of the interesting decisions of ASCII was that it does not require state. It does not encode the 'shift'

115 Encodes characters as 7-bit In ASCII 1 character equals 1 byte.

116 The 8 th bit can be used as parity, but that was never common.

117 The 3568 ASCII astroid is named after it. Fun fact

118 ISO-8859 ISO-8859 was created in 1985 and is a set of 16 character sets. The most known one is ISO This included more than just english When the euro was introduced they replaced with and named it ISO

119 Uses the extra bit to be able to store a second set of 127 characters ISO-8859 Replaces ISO 646 (1972) which was a 7-bit mess. ECMA, the European Computer Manufacturers Association

120 The base characters (<127) are shared between ASCII and ISO-8859-? The base characters are shared between ASCII and ISO-8859

121 The other characters differ per country/region

122 Windows-1252 (CP1252) is mostly identical with ISO

123 ISO is also known as Latin1

124 Latin1 in MySQL is not ISO , but CP1252. mysql> SHOW CHARSET LIKE 'latin1'; Charset Description Default collation Maxlen latin1 cp1252 West European latin1_swedish_ci row in set (0.00 sec)

125 Unicode The work on Unicode started in 1987

126 Allows you to store text in any language Both alive and dead

127 Allows you to store text combining multiple languages in the same file

128 Each character gets a number (a.k.a. code point) and a description.

129 That doesn't guarantee your font will display it. You might see a replacement character instead. This can be a question mark or some square.

130 UTF-8 Unicode Transformation Format

131 This is an character encoding for unicode. It is not the only unicode enconding. UTF-16 (fixed: ucs2) UTF-32 (ucs4)

132 This translates from code points to a binary string.

133 UTF-8 and ASCII share the same characters for 0<127.

134 Non-ASCII characters are stored as 2, 3 or 4-bytes.

135 UTF-32 UTF-16 UTF-8 ISO ASCII Baudot Here you can see the minimum and maximum number of bytes required to store one character. The blue show minimum and the red shows the variable part. This shows that UTF-8 is efficient in terms of storage for latin scripts

136 If a byte starts with '0xxxxxxx' then it is a 1-byte character

137 If a byte starts with '110' it is a start of a 2- byte character.

138 If a byte starts with '10' then it is a continuation of a multibyte character.

139 If a byte starts with '1110' it is the start of a 3-byte character.

140 If a byte starts with '11110' it is the start of a 4-byte character.

141 Examples: a = ë = Here you can see the letter a and the letter e with the dots (diaeresis, trema)

142 UTF-8 And MySQL Now we get into MySQL specifics

143 Some reasons to use UTF-8 in MySQL

144 Non-english scripts like Chinese, Cyrillic or Greek. Names Comments URL's addresses

145 Emoji (including in the help text of your mobile app) Hamburger icon

146 utf8 in MySQL is an alias for utf8mb3

147 utf8mb3 can store 3-byte UTF-8

148 utf8mb4 can store 4-byte UTF-8 utf8mb4 exists since 5.5.3

149 Best practice: Always use utf8mb4, don't use utf8

150 Where to set the encoding?

151 It is set on a per-column basis

152 There is a per-table default

153 There is a per-database default Stored in db.opt Use ALTER DATABASE to change it

154 There is a per-server default: character_set_server

155 Connections also have a character set Set the character set in your connection properties If that isn't possible: Use SET NAMES utf8mb4

156 Drawbacks of UTF-8

157 So just set everything to utf8mb4? The question we want to answer is...

158 It depends The answer is...

159 Does your application support it? Input validation Character length Security

160 CHAR(10) suddenly needs 40 bytes!

161 TINYTEXT has a size limit in bytes With utf8m4 you can store between 63 and 255 characters. This also happens to other TEXT types and BLOB types

162 The MEMORY storage engine expands VARCHAR(10) to 40 bytes This affects: - User created tables - Internal temporary tables

163 With InnoDB your index grows over 767 bytes. Use innodb_large_prefex with COMPRESSED or DYNAMIC

164 Best practice: Use latin1 for server, database and table default. Enable Unicode on columns which need it. Or use utf8mb4 all the way if you don't need the efficiency and performance of latin1 Changing everything to VARBINARY and BLOB will not solve your issue.

165 Converting your data

166 How to convert from latin1 to utf8mb4?

167 ALTER TABLE t1 MODIFY COLUMN c1 VARCHAR(100) CHARACTER SET utf8mb4;

168 But I have many columns!

169 Use CONCAT() and information_schema to generate the statements

170 Or convert all columns: ALTER TABLE t1 CONVERT TO CHARACTER SET utf8mb4; Also for INSERTs!

171 Change defaults

172 Set character_set_server

173 ALTER SCHEMA s1 DEFAULT CHARACTER SET utf8mb4;

174 ALTER TABLE t1 DEFAULT CHARACTER SET utf8mb4;

175 Common failures

176 Application looks okay, but in MySQL the data looks wrong 'Search' in the application might not function correctly

177 The latin1 column was holding utf8 data already

178 Wrong conversion == garbage Don't just convert this data. Run a latin1 to UTF-8 conversion on data which already was UTF-8 will result in garbage.

179 Change column to varbinary and then to utf8mb4 to not convert the data.

180 The conversion fails and eats your data MySQL tries really hard to convert your data but this might not be possible.

181 Use sql_mode='strict_all_tables'

182 Now the operation will fail instead of truncate your data Also for inserts

183 Connection set to utf8, but data is 4-byte UTF-8. You can't insert 4-byte or request 4- byte characters

184 Collation support There is no utf8mb4_general_cs (case sensitive) There is utf8mb4_unicode_ci And utf8mb4_unicode_520_ci And utf8mb4_bin unicode_ci = UCA Unicode_520 UCA Latest 8.0.0

185 Here we compare the sun and moon emoji.

186 Special collations get lost during conversion Collation = Sorting & Equality

187 ALTER TABLE CONVERT TO only supports one collation

188 Safe collation before the ALTER and then restore it for columns which have a nondefault collation.

189 Collation mismatch

190 Here MySQL does not now which collation to use.

191 Use COLLATE to set the desired collation for the operation.

192

193 é and + e are not identical Combining characters

194 Unicode normalization forms NFC Composed NFD Decomposed NFKC Composed NFKD Decomposed NFK removes compatibility distinction and will lose information. But this is useful for search etc.

195 Best practice: Normalize strings in your application

196 4-byte characters get silently lost on dump/restore

197 Set utf8mb4 as default charset for the connection mysqldump uses

198 This shows what we can do with a patched mysql client. This uses unicode drawing characters

199 This shows the unicode character database imported into MySQL

200

201 Did you know a BOM can be in the middle of a string? Also MySQL doesn't handle BOM's well

202 Fulltext search & CJK

203 Not every character has the same width

204 Not even if we use a monospace font

205 A character can have a width of 0, 1, 2 or -1 positions

206 Punycode

207 Ligatures, Glyphs, Characters

208 Using Unicode

209 Fonts

210 Typing

211 Virtual keyboards

212 Control characters

213 Charset is not a constraint

214 Replacement characters