MySQL BREAK/FIX LAB. Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 2014

Transcription

1 MySQL BREAK/FIX LAB Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214

2 Who we are: black bird /blackbird/ noun 1. (of a people) constantly in a state of shock caused by series of incidents.

3 Tutorial Agenda Fix standalone MySQL instance Replication issues Performance issues

4 Getting started Each attendee has its own instance (not shared) we break it, you fix it :-) One standalone MySQL instance Several MySQL instances using MySQL sandbox

5 Getting started Access: Username/password: user-lab / blackbirdlab123 ssh user-lab@hostname Pre-requirement: ssh client

6 Agenda - Part 1 Recover a mysql instance unable to start misconfiguration files permission corrupted files Connectivity issues misconfiguration recover password server gone away Learn how to troubleshoot crash read error log fix misconfiguration tune variables

7 Start mysqld ~]# ps aux grep mysql root pts/2 S+ 13:8 : grep mysql [root@hostdb ~]# service mysqld start chown: invalid user: `mysql:mysql' chown: invalid user: `mysql:mysql' Initializing MySQL database: chown: invalid user: `mysql' Cannot change ownership of the database directories to the 'mysql' user. Check that you have the necessary permissions and try again. [FAILED] chown: invalid user: `mysql:mysql'

8 mysql user ~]# id mysql id: mysql: No such user ~]# useradd mysql ~]# id mysql uid=5(mysql) gid=51(mysql) groups=51(mysql)

9 Unknown variable ~]# service mysqld start Initializing MySQL database: Installing MySQL system tables :19:9 [ERROR] /usr/libexec/mysqld: unknown variable 'tmpd1r=/var/tmp' :19:9 [ERROR] Aborting :19:9 [Note] /usr/libexec/mysqld: Shutdown complete [FAILED] ~]#!ps ps aux grep mysql root pts/2 S+ 13:19 : grep mysql

10 Which config file? ~]# grep tmpd /etc/my.cnf ~]# grep tmpd /etc/mysql/my.cnf grep: /etc/mysql/my.cnf: No such file or directory Multiple configuration file(s)?? The easy way: The hard way: strace

11 strace ~]# strace mysqld strace: mysqld: command not found ~]# strace /usr/libexec/mysqld ~]# strace -e trace=open,stat /usr/libexec/mysqld... stat("/etc/my.cnf", {st_mode=s_ifreg644, st_size=243,...}) = open("/etc/my.cnf", O_RDONLY) = 3 stat("/etc/mysql/my.cnf", x7fffea4cd8) = -1 ENOENT (No such file or directory) stat("/usr/etc/my.cnf", {st_mode=s_ifreg644, st_size=25,...}) = open("/usr/etc/my.cnf", O_RDONLY) = 3 stat("/root/.my.cnf", {st_mode=s_ifreg644, st_size=33,...}) = open("/root/.my.cnf", O_RDONLY) = 3...

12 strace : mysqld --print-defaults # strace -e stat64 /usr/libexec/mysqld --print-defaults stat64("/etc/my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) stat64("/etc/mysql/my.cnf", {st_mode=s_ifreg644, st_size=3564,...}) = stat64("/usr/etc/my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) stat64("/root/.my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) mysqld would have been started with the following arguments: --user=mysql --socket=/var/run/mysqld/mysqld.sock --port=336 --basedir=/usr -datadir=/var/lib/mysql --tmpdir=/tmp --skip-external-locking --bind-address= key_buffer=16m --max_allowed_packet=16m --thread_stack=192k -thread_cache_size=8 --myisam-recover=backup --query_cache_limit=1m -query_cache_size=16m --log_error=/var/log/mysql/error.log --expire_logs_days=1 -max_binlog_size=1m

13 Fix tmpdir ~]# cat /usr/etc/my.cnf [mysqld] tmpd1r=/var/tmp ~]# sed -i -e 's/tmpd1r/tmpdir/' /usr/etc/my.cnf ~]# cat /usr/etc/my.cnf [mysqld] tmpdir=/var/tmp

14 Start again ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld:?? Not very descriptive output [FAILED]

15 Check error log ~]# cat /var/log/mysqld.log :5: mysqld_safe Starting mysqld daemon with databases from /var/lib/msql :5: [Note] Plugin 'FEDERATED' is disabled. /usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist :5: [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it :5: InnoDB: The InnoDB memory heap is disabled :5: InnoDB: Mutexes and rw_locks use GCC atomic builtins :5: InnoDB: Compressed tables use zlib :5: InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/iblmkyv' (Errcode: 13) :5: InnoDB: Error: unable to create temporary file; errno: :5: [ERROR] Plugin 'InnoDB' init function returned error :5: [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :5: [ERROR] Unknown/unsupported storage engine: InnoDB :5: [ERROR] Aborting :5: [Note] /usr/libexec/mysqld: Shutdown complete :5: mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

16 Fix permissions -rw-rw rw-rw rw-rw ~]# ls -l root root root root root root /var/lib/mysql/mysql/plugin.* 8586 Mar 13 12:3 /var/lib/mysql/mysql/plugin.frm Mar 13 12:3 /var/lib/mysql/mysql/plugin.myd 124 Mar 13 12:3 /var/lib/mysql/mysql/plugin.myi [root@hostdb ~]# chown -R mysql:mysql /var/lib/mysql/mysql/ [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

17 Check error log :23:4 mysqld_safe Starting mysqld daemon with databases from /var/lib/msql :23:4 [Note] Plugin 'FEDERATED' is disabled. /usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist :23:4 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it :23:4 InnoDB: The InnoDB memory heap is disabled :23:4 InnoDB: Mutexes and rw_locks use GCC atomic builtins :23:4 InnoDB: Compressed tables use zlib :23:4 InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/ib9c5cf6' (Errcode: 13) :23:5 InnoDB: Error: unable to create temporary file; errno: :23:5 [ERROR] Plugin 'InnoDB' init function returned error :23:5 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :23:5 [ERROR] Unknown/unsupported storage engine: InnoDB :23:5 [ERROR] Aborting :23:5 [Note] /usr/libexec/mysqld: Shutdown complete :23:5 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

18 Fix datadir path ~]# grep datadir /etc/my.cnf datadir=/var/lib/msql ~]# sed -i -e 's/datadir=\/var\/lib\/msql/datadir=\/var\/lib\/mysql/' /etc/my.cnf ~]# grep datadir /etc/my.cnf datadir=/var/lib/mysql

19 Check error log :51:6 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :51:6 [Note] Plugin 'FEDERATED' is disabled :51:6 InnoDB: The InnoDB memory heap is disabled :51:6 InnoDB: Mutexes and rw_locks use GCC atomic builtins :51:6 InnoDB: Compressed tables use zlib :51:6 InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/ibzhtuux' (Errcode: 13) :51:6 InnoDB: Error: unable to create temporary file; errno: :51:6 [ERROR] Plugin 'InnoDB' init function returned error :51:6 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :51:6 [ERROR] Unknown/unsupported storage engine: InnoDB :51:6 [ERROR] Aborting :51:6 [Note] /usr/libexec/mysqld: Shutdown complete :51:6 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

20 Fix tmpdir ~]# ls -ld /var/tmp drwxrwx--t 2 root root 496 Mar 13 13:23 /var/tmp [root@hostdb ~]# chmod a+rwx /var/tmp [root@hostdb ~]# ls -ld /var/tmp drwxrwxrwt 2 root root 496 Mar 13 13:23 /var/tmp [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

21 Cannot allocate memory :58:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :58:48 [Note] Plugin 'FEDERATED' is disabled :58:48 InnoDB: The InnoDB memory heap is disabled :58:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins :58:48 InnoDB: Compressed tables use zlib :58:48 InnoDB: Using Linux native AIO :58:48 InnoDB: Initializing buffer pool, size = 1.G InnoDB: mmap( bytes) failed; errno :58:48 InnoDB: Completed initialization of buffer pool :58:48 InnoDB: Fatal error: cannot allocate memory for the buffer pool :58:48 [ERROR] Plugin 'InnoDB' init function returned error :58:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :58:48 [ERROR] Unknown/unsupported storage engine: InnoDB :58:48 [ERROR] Aborting :58:48 [Note] /usr/libexec/mysqld: Shutdown complete :58:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

22 Fix innodb_buffer_pool_size ~]# perror 12 OS error code 12: Cannot allocate memory ~]# grep 1 /etc/my.cnf innodb_buffer_pool_size=1g [root@hostdb ~]# sed -i -e 's/1g/256m/' /etc/my.cnf [root@hostdb ~]# grep innodb_buffer_pool_size /etc/my.cnf innodb_buffer_pool_size=256m [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

23 Again error :11:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :11:57 [Note] Plugin 'FEDERATED' is disabled :11:57 InnoDB: The InnoDB memory heap is disabled :11:57 InnoDB: Mutexes and rw_locks use GCC atomic builtins :11:57 InnoDB: Compressed tables use zlib :11:57 InnoDB: Using Linux native AIO :11:57 InnoDB: Initializing buffer pool, size = 256.M :11:57 InnoDB: Completed initialization of buffer pool :11:57 InnoDB: Operating system error number 13 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. InnoDB: File name./ibdata1 InnoDB: File operation call: 'open'. InnoDB: Cannot continue operation :11:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

24 Fix permissions -rw-rw [root@hostdb total rw-rw rw-rw rw-rw drwx drwx drwx ~]# ls -l /var/lib/mysql/ibdata Mar 13 12:34 /var/lib/mysql/ibdata1 ~]# ls -l /var/lib/mysql mysql root root Mar 13 12:34 ibdata Mar 13 12:34 ib_logfile Mar 13 12:34 ib_logfile1 mysql 496 Mar 13 12:3 mysql root 496 Mar 13 12:3 performance_schema root 496 Mar 13 12:3 test [root@hostdb ~]# chown -R mysql:mysql /var/lib/mysql [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

25 innodb_log_file_size :15: mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :15: [Note] Plugin 'FEDERATED' is disabled :15: InnoDB: The InnoDB memory heap is disabled :15: InnoDB: Mutexes and rw_locks use GCC atomic builtins :15: InnoDB: Compressed tables use zlib :15: InnoDB: Using Linux native AIO :15: InnoDB: Initializing buffer pool, size = 256.M :15:1 InnoDB: Completed initialization of buffer pool InnoDB: Error: log file./ib_logfile is of different size bytes InnoDB: than specified in the.cnf file bytes! :15:1 [ERROR] Plugin 'InnoDB' init function returned error :15:1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :15:1 [ERROR] Unknown/unsupported storage engine: InnoDB :15:1 [ERROR] Aborting :15:1 [Note] /usr/libexec/mysqld: Shutdown complete :15:1 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

26 Fix innodb_log_file_size ~]# grep innodb_log_file_size /etc/my.cnf innodb_log_file_size=64m ~]# ls -l /var/lib/mysql/ib_logfile* -rw-rw mysql mysql Mar 13 12:34 /var/lib/mysql/ib_logfile -rw-rw mysql mysql Mar 13 12:34 /var/lib/mysql/ib_logfile1 [root@hostdb ~]# sed -i -e 's/innodb_log_file_size=64m/innodb_log_file_size=32m/' /etc/my.cnf [root@hostdb ~]# grep innodb_log_file_size /etc/my.cnf innodb_log_file_size=32m [root@hostdb ~]# service mysqld start Starting mysqld: [ OK ]

27 Access to mysqld ~]# mysql ERROR 22 (HY): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) ~]# perror 2 OS error code 2: No such file or directory [root@hostdb ~]# ls -l /tmp/mysql.sock ls: cannot access /tmp/mysql.sock: No such file or directory

28 Access to mysqld ~]# grep socket /var/log/mysqld.log tail -n 1 Version: '5.5.34' socket: '/var/lib/mysql/mysql.sock' port: 336 Community Server (GPL) [root@hostdb ~]# lsof -n grep mysqld grep unix mysqld mysql 12u unix xffff882edd4 t /var/lib/mysql/mysql.sock In /etc/my.cnf : [client] socket=/tmp/mysql.sock MySQL 22829

29 Access to mysqld ~]# mysql ERROR 145 (28): Access denied for user (using password: YES) ~]# strace -e trace=open mysql open("/etc/my.cnf", O_RDONLY) = 3 open("/usr/etc/my.cnf", O_RDONLY) = 3 open("/root/.my.cnf", O_RDONLY) = 3 [root@hostdb ~]# cat ~/.my.cnf [client] password=adummypassword

30 Access to mysqld ~]# mysql --no-defaults ~]# mysql -p

31 Change root password ~]# echo "SET PASSWORD=PASSWORD('$RANDOM$RANDOM')" mysql ~]# mysql ERROR 145 (28): Access denied for user (using password: NO) In /etc/my.cnf : skip-grant-tables [root@hostdb ~]# service mysqld restart Stopping mysqld: Starting mysqld: [ [ OK OK ] ]

32 Change root password ~]# mysql mysql> UPDATE mysql.user SET password=password('newpass') WHERE user='root'; mysql> FLUSH PRIVILEGES; Remove skip-grant-tables from /etc/my.cnf

33 removed ib_logfile (DON T!) Free space was low: I deleted some log files... [root@hostdb ~]# cd /var/lib/mysql [root@hostdb mysql]# rm -rf ib_logfile* [root@hostdb mysql]# lsof -n grep ib_logfile mysqld 1276 mysql 8uW REG 22, mysqld 1276 mysql 9uW REG 22, /var/lib/mysql/ib_logfile (deleted) 4355 /var/lib/mysql/ib_logfile1 (deleted)

34 removed ib_logfile (DON T!) mysqld and InnoDB continue working normally mysql> use test; mysql> create table tbl1 (id int auto_increment primary key, v varchar(1)) engine=innodb; mysql> insert into tbl1 values(null,'aa'); mysql> insert into tbl1 select null, v from tbl1; mysql> SELECT * FROM tbl1; [root@hostdb ~]# service mysqld restart

35 removed ib_logfile (DON T!) InnoDB Redo Log are automatically recreated :42:2 InnoDB: Log file./ib_logfile did not exist: new to be created InnoDB: Setting log file./ib_logfile size to 32 MB InnoDB: Database physically writes the file full: wait :42:21 InnoDB: Log file./ib_logfile1 did not exist: new to be created InnoDB: Setting log file./ib_logfile1 size to 32 MB InnoDB: Database physically writes the file full: wait :42:22 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! :42:22 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the.ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer...

36 removed ibdata1 (really, DON T!) Free space was low: I deleted a large file... [root@hostdb mysql]# rm -f ibdata1 [root@ip mysql]# lsof -n grep deleted mysqld 7478 mysql 3uW REG 22, (deleted) mysqld 7478 mysql 4u REG 22, (deleted) mysqld 7478 mysql 5u REG 22, (deleted) mysqld 7478 mysql 6u REG 22, (deleted) mysqld 7478 mysql 7u REG 22, (deleted) mysqld 7478 mysql 11u REG 22, (deleted) [root@hostdb mysql]# lsof -n grep ibdata1 mysqld 7478 mysql 3uW REG 22, (deleted) /var/lib/mysql/ibdata1 /var/tmp/ibtkakep /var/tmp/ibf1ray /var/tmp/ib3piqic /var/tmp/ibl8k16o /var/tmp/ib7sjqkq /var/lib/mysql/ibdata1

37 removed ibdata1 (really, DON T!) mysqld and InnoDB continue working normally : mysql> insert into tbl1 values(null,'aa'); mysql> insert into tbl1 select null, v from tbl1; mysql> SELECT * FROM tbl1; until restart : [root@hostdb ~]# service mysqld restart Stopping mysqld: [ OK ] MySQL Daemon failed to start. Starting mysqld: [FAILED]

38 removed ibdata1 (really, DON T!) You can now recover from backup!

39 removed ibdata1 (really, DON T!) InnoDB: The first specified data file./ibdata1 did not exist: InnoDB: a new database to be created! :48:28 InnoDB: Setting file./ibdata1 size to 18 MB InnoDB: Database physically writes the file full: wait :48:28 InnoDB: Error: all log files must be created at the same time :48:28 InnoDB: All log files must be created also in database creation :48:28 InnoDB: If you want bigger or smaller log files, shut down the :48:28 InnoDB: database and make sure there were no errors in shutdown :48:28 InnoDB: Then delete the existing log files. Edit the.cnf file :48:28 InnoDB: and start the database again :48:28 [ERROR] Plugin 'InnoDB' init function returned error :48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :48:28 [ERROR] Unknown/unsupported storage engine: InnoDB :48:28 [ERROR] Aborting Misleading errors

40 removed ibdata1 (really, DON T!) mysql]# rm -f ibdata1 ib_logfile* test/* ~]# service mysqld restart

41 MySQL server has gone away mysql> SELECT SLEEP(1); ERROR 213 (HY): Lost connection to MySQL server during query # kill -9 `pidof mysqld` # kill -6 `pidof mysqld` # kill -11 `pidof mysqld`

42 MySQL server has gone away Version: '5.5.34' socket: '/var/lib/mysql/mysql.sock' port: 336 MySQL Community Server (GPL) :6:1 mysqld_safe Number of processes running now: :6:1 mysqld_safe mysqld restarted

43 MySQL server has gone away OOM Killer crashing bug signals others Always check logs: - mysql error log - syslog

44 MySQL server has gone away ~]# ( echo -n "SELECT '" ; for i in `seq 1 1` ; do echo -n " " ; done ; echo -n "' a") mysql a [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 2` ; do echo -n " " ; done ; echo -n "' a") mysql a [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 1` ; do echo -n " " ; done ; echo -n "' a") mysql wc [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 11` ; do echo -n " " ; done ; echo -n "' a") mysql wc ERROR 26 (HY) at line 1: MySQL server has gone away

45 MySQL server has gone away ~]# mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_allowed_packet'" Variable_name Value max_allowed_packet ~]# mysql -e "SET GLOBAL max_allowed_packet=148576" ~]# ( echo -n "SELECT '" ; for i in `seq 1 11` ; do echo -n " " ; done ; echo -n "' a") mysql wc

46 MySQL server has gone away server crash server killed session terminated/killed session timing out (wait_timeout) big packet (max_allowed_packet)

47 MySQL BREAK/FIX LAB Replication Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214

48 Agenda - Part 2 Replication overview and tools Bad server-id slave id identical to master slave id identical to other slave Incorrect slave data Duplicate key error Data drift Relay log corruption

49 Replication Overview asynchronous: based on copying and executing of binary logs allows for delayed slaves, incremental backups binary logs have two formats statement: SQL row events: starting with 5.1 replication is run by two threads on slave IO thread: copies master binary logs to slave relay logs SQL thread: executes binlog events on data

50 Replication Processlist Threads Slave: slave1 [localhost] {msandbox} ((none)) > show processlist; Id User Host db Command Time State Info system user NULL Connect 31 Slave has read all relay log; waiting for the slave I/O thread to update it NULL 2 system user NULL Connect 31 Waiting for master to send event NULL 3 msandbox localhost NULL Query NULL show processlist rows in set (. sec)

51 Replication Processlist Threads Master: master [localhost] {msandbox} ((none)) > show processlist; Id User Host db Command Time State Info rsandbox localhost:4494 NULL Binlog Dump 147 Master has sent all binlog to slave; waiting for binlog to be updated NULL 2 rsandbox localhost:4495 NULL Binlog Dump 144 Master has sent all binlog to slave; waiting for binlog to be updated NULL 4 msandbox localhost NULL Query NULL show processlist rows in set (. sec)

52 Replication Tools MySQL Sandbox Percona Toolkit pt-table-checksum pt-table-sync pt-slave-restart mysqlbinlog error logs

53 Replication Tools (cont.) slave commands SHOW SLAVE STATUS\G STOP SLAVE; START SLAVE; STOP/START SLAVE IO_THREAD; STOP/START SLAVE SQL_THREAD; master commands SHOW MASTER STATUS; SHOW SLAVE HOSTS;

54 MySQL Sandbox created by Giuseppe Maxia (The Data Charmer) Linux/FreeBSD/MacOSX only not for production instances allows you to install and run multiple instances of MySQL on same server can easily set up replication groups default is master and two slaves can be run as regular (non-root) user

55 Replication Sandbox repl_test]$ pwd /home/user-lab/sandboxes/repl_test repl_test]$ ls check_slaves m clear_all master connection.json node1 default_connection.json node2 initialize_slaves README restart_all s1 s2 send_kill_all start_all status_all stop_all use_all [user-lab@hostdb repl_test]$./start_all # executing "start" on /home/user-lab/sandboxes/repl_test executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started executing "start" on slave 2... sandbox server started

56 Sandbox Master repl_test]$ cd master/ master]$ ls change_paths grants.mysql change_ports json_in_db clear load_grants connection.json msb data my default_connection.json my.sandbox.cnf proxy_start README rescue_mysql_dump.sql restart send_kill start master]$ ls data/ ibdata1 msandbox.err mysql-bin.index ib_logfile mysql mysql_sandbox23992.pid ib_logfile1 mysql-bin.1 performance_schema test status stop tmp use USING

57 Sandbox Slave master]$ cd../node1 change_paths change_ports clear connection.json data node1]$ ls default_connection.json grants.mysql json_in_db load_grants msb my my.sandbox.cnf proxy_start README restart node1]$ ls data ibdata1 mysql_sandbox23993.pid ib_logfile mysql_sandbox23993-relay-bin.1 ib_logfile1 mysql_sandbox23993-relay-bin.2 master.info mysql_sandbox23993-relay-bin.index msandbox.err performance_schema mysql relay-log.info mysql-bin.1 test mysql-bin.index send_kill start status stop tmp use USING

58 Connecting to MySQL Instances repl_test]$ ls check_slaves m clear_all master connection.json node1 default_connection.json node2 initialize_slaves README restart_all s1 s2 send_kill_all start_all status_all stop_all use_all [user-lab@hostdb repl_test]$./m Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 master [localhost] {msandbox} ((none)) > [user-lab@hostdb repl_test]$./s1 -uroot Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 slave1 [localhost] {root} ((none)) >

59 Problem #1: Duplicate server-id value the server-id uniquely identifies a server in a replication topology recorded in binary log dynamic variable (doesn t require restart) set in the my.cnf file (my.sandbox.cnf) if a server is cloned (e.g., ec2 snapshot), the server-id needs to be edited in new instance

60 Problem #1: Duplicate server-id value Example of slave with identical server_id to master: $./s1 -e "show slave status\g" *************************** 1. row *************************** Slave_IO_State: Slave_IO_Running: No Slave_SQL_Running: Yes Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 1593 Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have equal MySQL server ids; these ids must be different for replication to work (or the --replicate-same-server-id option must be used on slave but this does not always make sense; please check the manual before using it). Master_Server_Id: 1

61 Problem #1: Duplicate server-id values with two slaves with identical server-ids, the symptoms and error messages are misleading master will show only one slave in processlist, continuously reconnecting

62 Problem #1: Duplicate server-id values slave status looks okay on both, but error logs will fill up: ~]$ cd sandboxes/ sandboxes]$ cd dupl_server_id2 dupl_server_id2]$./start_all dupl_server_id2]$ tail node1/data/msandbox.err :25:47 [Note] Slave: received end packet from server, apparent master shutdown: :25:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.3' at position :25:47 [Note] Slave: received end packet from server, apparent master shutdown: :25:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.3' at position 17

63 Solution #1: Modify slave server-id on one slave 1. set server_id variable dupl_server_id2]$./s1 slave1 [localhost] {msandbox} ((none)) > show global variables like 'server_id'; Variable_name Value server_id row in set (. sec) slave1 [localhost] {msandbox} ((none)) > set global server_id = 12; Query OK, rows affected (. sec) slave1 [localhost] {msandbox} ((none)) > stop slave; start slave; Query OK, rows affected (.2 sec) Query OK, rows affected (. sec) slave1 [localhost] {msandbox} ((none)) > show slave status\g

64 Solution #1: Modify slave server-id on one slave 2. edit node1/my.sandbox.cnf to use different server-id dupl_server_id2]$ cd node1 node1]$ sed -i 's/server-id=11/server-id=12/g' my.sandbox.cnf node1]$ grep server-id my.sandbox.cnf server-id=12

65 Problem #2: Duplicate key errors very common replication error number of possible causes slave created from bad backup slave replication started in wrong file/pos direct write to the slave data drift

66 Problem #2: Duplicate key errors Example sandboxes]$ cd dupl_key/ dupl_key]$./start_all # executing "start" on /home/user-lab/sandboxes/dupl_key executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started [user-lab@hostdb dupl_key]$./s1 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3

67 Problem #2: Duplicate key errors Example slave1 [localhost] {msandbox} (test) > show slave status\g *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: Master_User: rsandbox... Slave_IO_Running: Yes Slave_SQL_Running: No... Last_Errno: 162 Last_Error: Error 'Duplicate entry '3' for key 'PRIMARY'' on query. Default database: 'test'. Query: 'insert into dupe_test (val) values ('master3'), ('master4')'... Seconds_Behind_Master: NULL

68 Problem #2: Duplicate key errors Example (cont.) slave1 [localhost] {msandbox} ((none)) > use test; slave1 [localhost] {msandbox} (test) > show create table dupe_test\g *************************** 1. row *************************** Table: dupe_test Create Table: CREATE TABLE `dupe_test` ( `id` int(11) NOT NULL AUTO_INCREMENT, `val` varchar(2) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 1 row in set (. sec) slave1 [localhost] {msandbox} (test) > select * from dupe_test where id = 3; id val slave row in set (. sec)

69 Problem #2: Duplicate key errors Example (cont.) dupl_key]$ cd master master]$./my sqlbinlog data/mysql-bin.3... # at 521 #1441 7:5:14 server id 1 end_log_pos 589 Querythread_id=3 error_code= SET TIMESTAMP= /*!*/; BEGIN /*!*/; # at 589 #1441 7:5:14 server id 1 end_log_pos 617 Intvar SET INSERT_ID=3/*!*/; # at 617 #1441 7:5:14 server id 1 end_log_pos 739 Querythread_id=3 error_code= SET TIMESTAMP= /*!*/; insert into dupe_test (val) values ("master3"), ('master4') /*!*/;... exec_time= exec_time=

70 Solution #2: Duplicate key errors General solutions: 1. recreate slave from fresh backup of master (preferred) 2. get replication running again, fix data differences later We will be doing solution #2.

71 Solution #2: Duplicate key errors To check if the error is due to a direct write to slave: [user-lab@hostdb dupl_key]$ cd node1 [user-lab@hostdb node1]$ pwd /home/user-lab/sandboxes/dupl_key/node1 [user-lab@hostdb node1]$ ls data/mysql-bin.* data/mysql-bin.1 data/mysql-bin.2 data/mysql-bin.index [user-lab@hostdb node1]$./my sqlbinlog --server-id=11 data/mysql-bin. 2 grep dupe_test insert into dupe_test (val) values ('slave1')