MySQL BREAK/FIX LAB. Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 2014

Transcription

1 MySQL BREAK/FIX LAB Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214

2 Who we are: black bird /blackbird/ noun 1. (of a people) constantly in a state of shock caused by series of incidents.

3 Tutorial Agenda Fix standalone MySQL instance Replication issues Performance issues

4 Getting started Each attendee has its own instance (not shared) we break it, you fix it :-) One standalone MySQL instance Several MySQL instances using MySQL sandbox

5 Getting started Access: Username/password: user-lab / blackbirdlab123 ssh user-lab@hostname Pre-requirement: ssh client

6 Agenda - Part 1 Recover a mysql instance unable to start misconfiguration files permission corrupted files Connectivity issues misconfiguration recover password server gone away Learn how to troubleshoot crash read error log fix misconfiguration tune variables

7 Start mysqld ~]# ps aux grep mysql root pts/2 S+ 13:8 : grep mysql [root@hostdb ~]# service mysqld start chown: invalid user: `mysql:mysql' chown: invalid user: `mysql:mysql' Initializing MySQL database: chown: invalid user: `mysql' Cannot change ownership of the database directories to the 'mysql' user. Check that you have the necessary permissions and try again. [FAILED] chown: invalid user: `mysql:mysql'

8 mysql user ~]# id mysql id: mysql: No such user ~]# useradd mysql ~]# id mysql uid=5(mysql) gid=51(mysql) groups=51(mysql)

9 Unknown variable ~]# service mysqld start Initializing MySQL database: Installing MySQL system tables :19:9 [ERROR] /usr/libexec/mysqld: unknown variable 'tmpd1r=/var/tmp' :19:9 [ERROR] Aborting :19:9 [Note] /usr/libexec/mysqld: Shutdown complete [FAILED] ~]#!ps ps aux grep mysql root pts/2 S+ 13:19 : grep mysql

10 Which config file? ~]# grep tmpd /etc/my.cnf ~]# grep tmpd /etc/mysql/my.cnf grep: /etc/mysql/my.cnf: No such file or directory Multiple configuration file(s)?? The easy way: The hard way: strace

11 strace ~]# strace mysqld strace: mysqld: command not found ~]# strace /usr/libexec/mysqld ~]# strace -e trace=open,stat /usr/libexec/mysqld... stat("/etc/my.cnf", {st_mode=s_ifreg644, st_size=243,...}) = open("/etc/my.cnf", O_RDONLY) = 3 stat("/etc/mysql/my.cnf", x7fffea4cd8) = -1 ENOENT (No such file or directory) stat("/usr/etc/my.cnf", {st_mode=s_ifreg644, st_size=25,...}) = open("/usr/etc/my.cnf", O_RDONLY) = 3 stat("/root/.my.cnf", {st_mode=s_ifreg644, st_size=33,...}) = open("/root/.my.cnf", O_RDONLY) = 3...

12 strace : mysqld --print-defaults # strace -e stat64 /usr/libexec/mysqld --print-defaults stat64("/etc/my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) stat64("/etc/mysql/my.cnf", {st_mode=s_ifreg644, st_size=3564,...}) = stat64("/usr/etc/my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) stat64("/root/.my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) mysqld would have been started with the following arguments: --user=mysql --socket=/var/run/mysqld/mysqld.sock --port=336 --basedir=/usr -datadir=/var/lib/mysql --tmpdir=/tmp --skip-external-locking --bind-address= key_buffer=16m --max_allowed_packet=16m --thread_stack=192k -thread_cache_size=8 --myisam-recover=backup --query_cache_limit=1m -query_cache_size=16m --log_error=/var/log/mysql/error.log --expire_logs_days=1 -max_binlog_size=1m

13 Fix tmpdir ~]# cat /usr/etc/my.cnf [mysqld] tmpd1r=/var/tmp ~]# sed -i -e 's/tmpd1r/tmpdir/' /usr/etc/my.cnf ~]# cat /usr/etc/my.cnf [mysqld] tmpdir=/var/tmp

14 Start again ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld:?? Not very descriptive output [FAILED]

15 Check error log ~]# cat /var/log/mysqld.log :5: mysqld_safe Starting mysqld daemon with databases from /var/lib/msql :5: [Note] Plugin 'FEDERATED' is disabled. /usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist :5: [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it :5: InnoDB: The InnoDB memory heap is disabled :5: InnoDB: Mutexes and rw_locks use GCC atomic builtins :5: InnoDB: Compressed tables use zlib :5: InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/iblmkyv' (Errcode: 13) :5: InnoDB: Error: unable to create temporary file; errno: :5: [ERROR] Plugin 'InnoDB' init function returned error :5: [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :5: [ERROR] Unknown/unsupported storage engine: InnoDB :5: [ERROR] Aborting :5: [Note] /usr/libexec/mysqld: Shutdown complete :5: mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

16 Fix permissions -rw-rw rw-rw rw-rw ~]# ls -l root root root root root root /var/lib/mysql/mysql/plugin.* 8586 Mar 13 12:3 /var/lib/mysql/mysql/plugin.frm Mar 13 12:3 /var/lib/mysql/mysql/plugin.myd 124 Mar 13 12:3 /var/lib/mysql/mysql/plugin.myi [root@hostdb ~]# chown -R mysql:mysql /var/lib/mysql/mysql/ [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

17 Check error log :23:4 mysqld_safe Starting mysqld daemon with databases from /var/lib/msql :23:4 [Note] Plugin 'FEDERATED' is disabled. /usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist :23:4 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it :23:4 InnoDB: The InnoDB memory heap is disabled :23:4 InnoDB: Mutexes and rw_locks use GCC atomic builtins :23:4 InnoDB: Compressed tables use zlib :23:4 InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/ib9c5cf6' (Errcode: 13) :23:5 InnoDB: Error: unable to create temporary file; errno: :23:5 [ERROR] Plugin 'InnoDB' init function returned error :23:5 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :23:5 [ERROR] Unknown/unsupported storage engine: InnoDB :23:5 [ERROR] Aborting :23:5 [Note] /usr/libexec/mysqld: Shutdown complete :23:5 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

18 Fix datadir path ~]# grep datadir /etc/my.cnf datadir=/var/lib/msql ~]# sed -i -e 's/datadir=\/var\/lib\/msql/datadir=\/var\/lib\/mysql/' /etc/my.cnf ~]# grep datadir /etc/my.cnf datadir=/var/lib/mysql

19 Check error log :51:6 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :51:6 [Note] Plugin 'FEDERATED' is disabled :51:6 InnoDB: The InnoDB memory heap is disabled :51:6 InnoDB: Mutexes and rw_locks use GCC atomic builtins :51:6 InnoDB: Compressed tables use zlib :51:6 InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/ibzhtuux' (Errcode: 13) :51:6 InnoDB: Error: unable to create temporary file; errno: :51:6 [ERROR] Plugin 'InnoDB' init function returned error :51:6 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :51:6 [ERROR] Unknown/unsupported storage engine: InnoDB :51:6 [ERROR] Aborting :51:6 [Note] /usr/libexec/mysqld: Shutdown complete :51:6 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

20 Fix tmpdir ~]# ls -ld /var/tmp drwxrwx--t 2 root root 496 Mar 13 13:23 /var/tmp [root@hostdb ~]# chmod a+rwx /var/tmp [root@hostdb ~]# ls -ld /var/tmp drwxrwxrwt 2 root root 496 Mar 13 13:23 /var/tmp [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

21 Cannot allocate memory :58:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :58:48 [Note] Plugin 'FEDERATED' is disabled :58:48 InnoDB: The InnoDB memory heap is disabled :58:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins :58:48 InnoDB: Compressed tables use zlib :58:48 InnoDB: Using Linux native AIO :58:48 InnoDB: Initializing buffer pool, size = 1.G InnoDB: mmap( bytes) failed; errno :58:48 InnoDB: Completed initialization of buffer pool :58:48 InnoDB: Fatal error: cannot allocate memory for the buffer pool :58:48 [ERROR] Plugin 'InnoDB' init function returned error :58:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :58:48 [ERROR] Unknown/unsupported storage engine: InnoDB :58:48 [ERROR] Aborting :58:48 [Note] /usr/libexec/mysqld: Shutdown complete :58:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

22 Fix innodb_buffer_pool_size ~]# perror 12 OS error code 12: Cannot allocate memory ~]# grep 1 /etc/my.cnf innodb_buffer_pool_size=1g [root@hostdb ~]# sed -i -e 's/1g/256m/' /etc/my.cnf [root@hostdb ~]# grep innodb_buffer_pool_size /etc/my.cnf innodb_buffer_pool_size=256m [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

23 Again error :11:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :11:57 [Note] Plugin 'FEDERATED' is disabled :11:57 InnoDB: The InnoDB memory heap is disabled :11:57 InnoDB: Mutexes and rw_locks use GCC atomic builtins :11:57 InnoDB: Compressed tables use zlib :11:57 InnoDB: Using Linux native AIO :11:57 InnoDB: Initializing buffer pool, size = 256.M :11:57 InnoDB: Completed initialization of buffer pool :11:57 InnoDB: Operating system error number 13 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. InnoDB: File name./ibdata1 InnoDB: File operation call: 'open'. InnoDB: Cannot continue operation :11:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

24 Fix permissions -rw-rw [root@hostdb total rw-rw rw-rw rw-rw drwx drwx drwx ~]# ls -l /var/lib/mysql/ibdata Mar 13 12:34 /var/lib/mysql/ibdata1 ~]# ls -l /var/lib/mysql mysql root root Mar 13 12:34 ibdata Mar 13 12:34 ib_logfile Mar 13 12:34 ib_logfile1 mysql 496 Mar 13 12:3 mysql root 496 Mar 13 12:3 performance_schema root 496 Mar 13 12:3 test [root@hostdb ~]# chown -R mysql:mysql /var/lib/mysql [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]

25 innodb_log_file_size :15: mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql :15: [Note] Plugin 'FEDERATED' is disabled :15: InnoDB: The InnoDB memory heap is disabled :15: InnoDB: Mutexes and rw_locks use GCC atomic builtins :15: InnoDB: Compressed tables use zlib :15: InnoDB: Using Linux native AIO :15: InnoDB: Initializing buffer pool, size = 256.M :15:1 InnoDB: Completed initialization of buffer pool InnoDB: Error: log file./ib_logfile is of different size bytes InnoDB: than specified in the.cnf file bytes! :15:1 [ERROR] Plugin 'InnoDB' init function returned error :15:1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :15:1 [ERROR] Unknown/unsupported storage engine: InnoDB :15:1 [ERROR] Aborting :15:1 [Note] /usr/libexec/mysqld: Shutdown complete :15:1 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

26 Fix innodb_log_file_size ~]# grep innodb_log_file_size /etc/my.cnf innodb_log_file_size=64m ~]# ls -l /var/lib/mysql/ib_logfile* -rw-rw mysql mysql Mar 13 12:34 /var/lib/mysql/ib_logfile -rw-rw mysql mysql Mar 13 12:34 /var/lib/mysql/ib_logfile1 [root@hostdb ~]# sed -i -e 's/innodb_log_file_size=64m/innodb_log_file_size=32m/' /etc/my.cnf [root@hostdb ~]# grep innodb_log_file_size /etc/my.cnf innodb_log_file_size=32m [root@hostdb ~]# service mysqld start Starting mysqld: [ OK ]

27 Access to mysqld ~]# mysql ERROR 22 (HY): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) ~]# perror 2 OS error code 2: No such file or directory [root@hostdb ~]# ls -l /tmp/mysql.sock ls: cannot access /tmp/mysql.sock: No such file or directory

28 Access to mysqld ~]# grep socket /var/log/mysqld.log tail -n 1 Version: '5.5.34' socket: '/var/lib/mysql/mysql.sock' port: 336 Community Server (GPL) [root@hostdb ~]# lsof -n grep mysqld grep unix mysqld mysql 12u unix xffff882edd4 t /var/lib/mysql/mysql.sock In /etc/my.cnf : [client] socket=/tmp/mysql.sock MySQL 22829

29 Access to mysqld ~]# mysql ERROR 145 (28): Access denied for user (using password: YES) ~]# strace -e trace=open mysql open("/etc/my.cnf", O_RDONLY) = 3 open("/usr/etc/my.cnf", O_RDONLY) = 3 open("/root/.my.cnf", O_RDONLY) = 3 [root@hostdb ~]# cat ~/.my.cnf [client] password=adummypassword

30 Access to mysqld ~]# mysql --no-defaults ~]# mysql -p

31 Change root password ~]# echo "SET PASSWORD=PASSWORD('$RANDOM$RANDOM')" mysql ~]# mysql ERROR 145 (28): Access denied for user (using password: NO) In /etc/my.cnf : skip-grant-tables [root@hostdb ~]# service mysqld restart Stopping mysqld: Starting mysqld: [ [ OK OK ] ]

32 Change root password ~]# mysql mysql> UPDATE mysql.user SET password=password('newpass') WHERE user='root'; mysql> FLUSH PRIVILEGES; Remove skip-grant-tables from /etc/my.cnf

33 removed ib_logfile (DON T!) Free space was low: I deleted some log files... [root@hostdb ~]# cd /var/lib/mysql [root@hostdb mysql]# rm -rf ib_logfile* [root@hostdb mysql]# lsof -n grep ib_logfile mysqld 1276 mysql 8uW REG 22, mysqld 1276 mysql 9uW REG 22, /var/lib/mysql/ib_logfile (deleted) 4355 /var/lib/mysql/ib_logfile1 (deleted)

34 removed ib_logfile (DON T!) mysqld and InnoDB continue working normally mysql> use test; mysql> create table tbl1 (id int auto_increment primary key, v varchar(1)) engine=innodb; mysql> insert into tbl1 values(null,'aa'); mysql> insert into tbl1 select null, v from tbl1; mysql> SELECT * FROM tbl1; [root@hostdb ~]# service mysqld restart

35 removed ib_logfile (DON T!) InnoDB Redo Log are automatically recreated :42:2 InnoDB: Log file./ib_logfile did not exist: new to be created InnoDB: Setting log file./ib_logfile size to 32 MB InnoDB: Database physically writes the file full: wait :42:21 InnoDB: Log file./ib_logfile1 did not exist: new to be created InnoDB: Setting log file./ib_logfile1 size to 32 MB InnoDB: Database physically writes the file full: wait :42:22 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! :42:22 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the.ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer...

36 removed ibdata1 (really, DON T!) Free space was low: I deleted a large file... [root@hostdb mysql]# rm -f ibdata1 [root@ip mysql]# lsof -n grep deleted mysqld 7478 mysql 3uW REG 22, (deleted) mysqld 7478 mysql 4u REG 22, (deleted) mysqld 7478 mysql 5u REG 22, (deleted) mysqld 7478 mysql 6u REG 22, (deleted) mysqld 7478 mysql 7u REG 22, (deleted) mysqld 7478 mysql 11u REG 22, (deleted) [root@hostdb mysql]# lsof -n grep ibdata1 mysqld 7478 mysql 3uW REG 22, (deleted) /var/lib/mysql/ibdata1 /var/tmp/ibtkakep /var/tmp/ibf1ray /var/tmp/ib3piqic /var/tmp/ibl8k16o /var/tmp/ib7sjqkq /var/lib/mysql/ibdata1

37 removed ibdata1 (really, DON T!) mysqld and InnoDB continue working normally : mysql> insert into tbl1 values(null,'aa'); mysql> insert into tbl1 select null, v from tbl1; mysql> SELECT * FROM tbl1; until restart : [root@hostdb ~]# service mysqld restart Stopping mysqld: [ OK ] MySQL Daemon failed to start. Starting mysqld: [FAILED]

38 removed ibdata1 (really, DON T!) You can now recover from backup!

39 removed ibdata1 (really, DON T!) InnoDB: The first specified data file./ibdata1 did not exist: InnoDB: a new database to be created! :48:28 InnoDB: Setting file./ibdata1 size to 18 MB InnoDB: Database physically writes the file full: wait :48:28 InnoDB: Error: all log files must be created at the same time :48:28 InnoDB: All log files must be created also in database creation :48:28 InnoDB: If you want bigger or smaller log files, shut down the :48:28 InnoDB: database and make sure there were no errors in shutdown :48:28 InnoDB: Then delete the existing log files. Edit the.cnf file :48:28 InnoDB: and start the database again :48:28 [ERROR] Plugin 'InnoDB' init function returned error :48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed :48:28 [ERROR] Unknown/unsupported storage engine: InnoDB :48:28 [ERROR] Aborting Misleading errors

40 removed ibdata1 (really, DON T!) mysql]# rm -f ibdata1 ib_logfile* test/* ~]# service mysqld restart

41 MySQL server has gone away mysql> SELECT SLEEP(1); ERROR 213 (HY): Lost connection to MySQL server during query # kill -9 `pidof mysqld` # kill -6 `pidof mysqld` # kill -11 `pidof mysqld`

42 MySQL server has gone away Version: '5.5.34' socket: '/var/lib/mysql/mysql.sock' port: 336 MySQL Community Server (GPL) :6:1 mysqld_safe Number of processes running now: :6:1 mysqld_safe mysqld restarted

43 MySQL server has gone away OOM Killer crashing bug signals others Always check logs: - mysql error log - syslog

44 MySQL server has gone away ~]# ( echo -n "SELECT '" ; for i in `seq 1 1` ; do echo -n " " ; done ; echo -n "' a") mysql a [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 2` ; do echo -n " " ; done ; echo -n "' a") mysql a [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 1` ; do echo -n " " ; done ; echo -n "' a") mysql wc [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 11` ; do echo -n " " ; done ; echo -n "' a") mysql wc ERROR 26 (HY) at line 1: MySQL server has gone away

45 MySQL server has gone away ~]# mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_allowed_packet'" Variable_name Value max_allowed_packet ~]# mysql -e "SET GLOBAL max_allowed_packet=148576" ~]# ( echo -n "SELECT '" ; for i in `seq 1 11` ; do echo -n " " ; done ; echo -n "' a") mysql wc

46 MySQL server has gone away server crash server killed session terminated/killed session timing out (wait_timeout) big packet (max_allowed_packet)

47 MySQL BREAK/FIX LAB Replication Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214

48 Agenda - Part 2 Replication overview and tools Bad server-id slave id identical to master slave id identical to other slave Incorrect slave data Duplicate key error Data drift Relay log corruption

49 Replication Overview asynchronous: based on copying and executing of binary logs allows for delayed slaves, incremental backups binary logs have two formats statement: SQL row events: starting with 5.1 replication is run by two threads on slave IO thread: copies master binary logs to slave relay logs SQL thread: executes binlog events on data

50 Replication Processlist Threads Slave: slave1 [localhost] {msandbox} ((none)) > show processlist; Id User Host db Command Time State Info system user NULL Connect 31 Slave has read all relay log; waiting for the slave I/O thread to update it NULL 2 system user NULL Connect 31 Waiting for master to send event NULL 3 msandbox localhost NULL Query NULL show processlist rows in set (. sec)

51 Replication Processlist Threads Master: master [localhost] {msandbox} ((none)) > show processlist; Id User Host db Command Time State Info rsandbox localhost:4494 NULL Binlog Dump 147 Master has sent all binlog to slave; waiting for binlog to be updated NULL 2 rsandbox localhost:4495 NULL Binlog Dump 144 Master has sent all binlog to slave; waiting for binlog to be updated NULL 4 msandbox localhost NULL Query NULL show processlist rows in set (. sec)

52 Replication Tools MySQL Sandbox Percona Toolkit pt-table-checksum pt-table-sync pt-slave-restart mysqlbinlog error logs

53 Replication Tools (cont.) slave commands SHOW SLAVE STATUS\G STOP SLAVE; START SLAVE; STOP/START SLAVE IO_THREAD; STOP/START SLAVE SQL_THREAD; master commands SHOW MASTER STATUS; SHOW SLAVE HOSTS;

54 MySQL Sandbox created by Giuseppe Maxia (The Data Charmer) Linux/FreeBSD/MacOSX only not for production instances allows you to install and run multiple instances of MySQL on same server can easily set up replication groups default is master and two slaves can be run as regular (non-root) user

55 Replication Sandbox repl_test]$ pwd /home/user-lab/sandboxes/repl_test repl_test]$ ls check_slaves m clear_all master connection.json node1 default_connection.json node2 initialize_slaves README restart_all s1 s2 send_kill_all start_all status_all stop_all use_all [user-lab@hostdb repl_test]$./start_all # executing "start" on /home/user-lab/sandboxes/repl_test executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started executing "start" on slave 2... sandbox server started

56 Sandbox Master repl_test]$ cd master/ master]$ ls change_paths grants.mysql change_ports json_in_db clear load_grants connection.json msb data my default_connection.json my.sandbox.cnf proxy_start README rescue_mysql_dump.sql restart send_kill start master]$ ls data/ ibdata1 msandbox.err mysql-bin.index ib_logfile mysql mysql_sandbox23992.pid ib_logfile1 mysql-bin.1 performance_schema test status stop tmp use USING

57 Sandbox Slave master]$ cd../node1 change_paths change_ports clear connection.json data node1]$ ls default_connection.json grants.mysql json_in_db load_grants msb my my.sandbox.cnf proxy_start README restart node1]$ ls data ibdata1 mysql_sandbox23993.pid ib_logfile mysql_sandbox23993-relay-bin.1 ib_logfile1 mysql_sandbox23993-relay-bin.2 master.info mysql_sandbox23993-relay-bin.index msandbox.err performance_schema mysql relay-log.info mysql-bin.1 test mysql-bin.index send_kill start status stop tmp use USING

58 Connecting to MySQL Instances repl_test]$ ls check_slaves m clear_all master connection.json node1 default_connection.json node2 initialize_slaves README restart_all s1 s2 send_kill_all start_all status_all stop_all use_all [user-lab@hostdb repl_test]$./m Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 master [localhost] {msandbox} ((none)) > [user-lab@hostdb repl_test]$./s1 -uroot Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 slave1 [localhost] {root} ((none)) >

59 Problem #1: Duplicate server-id value the server-id uniquely identifies a server in a replication topology recorded in binary log dynamic variable (doesn t require restart) set in the my.cnf file (my.sandbox.cnf) if a server is cloned (e.g., ec2 snapshot), the server-id needs to be edited in new instance

60 Problem #1: Duplicate server-id value Example of slave with identical server_id to master: $./s1 -e "show slave status\g" *************************** 1. row *************************** Slave_IO_State: Slave_IO_Running: No Slave_SQL_Running: Yes Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 1593 Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have equal MySQL server ids; these ids must be different for replication to work (or the --replicate-same-server-id option must be used on slave but this does not always make sense; please check the manual before using it). Master_Server_Id: 1

61 Problem #1: Duplicate server-id values with two slaves with identical server-ids, the symptoms and error messages are misleading master will show only one slave in processlist, continuously reconnecting

62 Problem #1: Duplicate server-id values slave status looks okay on both, but error logs will fill up: ~]$ cd sandboxes/ sandboxes]$ cd dupl_server_id2 dupl_server_id2]$./start_all dupl_server_id2]$ tail node1/data/msandbox.err :25:47 [Note] Slave: received end packet from server, apparent master shutdown: :25:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.3' at position :25:47 [Note] Slave: received end packet from server, apparent master shutdown: :25:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.3' at position 17

63 Solution #1: Modify slave server-id on one slave 1. set server_id variable dupl_server_id2]$./s1 slave1 [localhost] {msandbox} ((none)) > show global variables like 'server_id'; Variable_name Value server_id row in set (. sec) slave1 [localhost] {msandbox} ((none)) > set global server_id = 12; Query OK, rows affected (. sec) slave1 [localhost] {msandbox} ((none)) > stop slave; start slave; Query OK, rows affected (.2 sec) Query OK, rows affected (. sec) slave1 [localhost] {msandbox} ((none)) > show slave status\g

64 Solution #1: Modify slave server-id on one slave 2. edit node1/my.sandbox.cnf to use different server-id dupl_server_id2]$ cd node1 node1]$ sed -i 's/server-id=11/server-id=12/g' my.sandbox.cnf node1]$ grep server-id my.sandbox.cnf server-id=12

65 Problem #2: Duplicate key errors very common replication error number of possible causes slave created from bad backup slave replication started in wrong file/pos direct write to the slave data drift

66 Problem #2: Duplicate key errors Example sandboxes]$ cd dupl_key/ dupl_key]$./start_all # executing "start" on /home/user-lab/sandboxes/dupl_key executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started [user-lab@hostdb dupl_key]$./s1 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3

67 Problem #2: Duplicate key errors Example slave1 [localhost] {msandbox} (test) > show slave status\g *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: Master_User: rsandbox... Slave_IO_Running: Yes Slave_SQL_Running: No... Last_Errno: 162 Last_Error: Error 'Duplicate entry '3' for key 'PRIMARY'' on query. Default database: 'test'. Query: 'insert into dupe_test (val) values ('master3'), ('master4')'... Seconds_Behind_Master: NULL

68 Problem #2: Duplicate key errors Example (cont.) slave1 [localhost] {msandbox} ((none)) > use test; slave1 [localhost] {msandbox} (test) > show create table dupe_test\g *************************** 1. row *************************** Table: dupe_test Create Table: CREATE TABLE `dupe_test` ( `id` int(11) NOT NULL AUTO_INCREMENT, `val` varchar(2) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 1 row in set (. sec) slave1 [localhost] {msandbox} (test) > select * from dupe_test where id = 3; id val slave row in set (. sec)

69 Problem #2: Duplicate key errors Example (cont.) dupl_key]$ cd master master]$./my sqlbinlog data/mysql-bin.3... # at 521 #1441 7:5:14 server id 1 end_log_pos 589 Querythread_id=3 error_code= SET TIMESTAMP= /*!*/; BEGIN /*!*/; # at 589 #1441 7:5:14 server id 1 end_log_pos 617 Intvar SET INSERT_ID=3/*!*/; # at 617 #1441 7:5:14 server id 1 end_log_pos 739 Querythread_id=3 error_code= SET TIMESTAMP= /*!*/; insert into dupe_test (val) values ("master3"), ('master4') /*!*/;... exec_time= exec_time=

70 Solution #2: Duplicate key errors General solutions: 1. recreate slave from fresh backup of master (preferred) 2. get replication running again, fix data differences later We will be doing solution #2.

71 Solution #2: Duplicate key errors To check if the error is due to a direct write to slave: [user-lab@hostdb dupl_key]$ cd node1 [user-lab@hostdb node1]$ pwd /home/user-lab/sandboxes/dupl_key/node1 [user-lab@hostdb node1]$ ls data/mysql-bin.* data/mysql-bin.1 data/mysql-bin.2 data/mysql-bin.index [user-lab@hostdb node1]$./my sqlbinlog --server-id=11 data/mysql-bin. 2 grep dupe_test insert into dupe_test (val) values ('slave1')

72 Solution #2: Duplicate key errors Do this (if it s a read slave): [mysqld] read_only Do not do this: [mysqld] slave-skip-errors = all

73 Solution #2: Duplicate key errors Hack #1: Get slave caught up Manually: dupl_key]$./s1 slave1 > stop slave; set global sql_slave_skip_counter = 1; start slave; slave1 > show slave status\g If you have a lot of events to skip, you can use pt-slaverestart.

74 Solution #2: Duplicate key errors Hack #1: Get slave caught up note this skips binary log events for InnoDB tables: entire transaction can t easily guarantee that master and slave data are equivalent later: run checksum to compare master and slave

75 Problem #3: Data Drift Statement-based binary logging can result in incorrect data on the slave nondeterministic updates with LIMIT but no ORDER BY certain system and math functions etc. ROW and MIXED binary logging avoids this

76 Problem #3: Data Drift dupl_key]$./m master [localhost] {msandbox} ((none)) > use test; master [localhost] {msandbox} (test) > insert into dupe_test (val) values (uuid()); Query OK, 1 row affected, 2 warnings (.4 sec) master [localhost] {msandbox} (test) > show warnings\g *************************** 1. row *************************** Level: Warning Code: 1265 Message: Data truncated for column 'val' at row 1 *************************** 2. row *************************** Level: Note Code: 1592 Message: Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it uses a system function that may return a different value on the slave. 2 rows in set (. sec)

77 Problem #3: Data Drift master [localhost] {msandbox} (test) > select * from dupe_test order by id desc limit 1; id val f6fe11-b932-11e3-a row in set (.2 sec) slave1 [localhost] {msandbox} ((none)) > select * from test.dupe_test order by id desc limit 1; id val f6ff6a8c-b932-11e3-a row in set (. sec)

78 Solution #3: Fixing Data Drift/Incorrect Slave Hack #2: Checksum and sync slave data pt-table-checksum checksums tables in chunks of rows can use replication to compare master and slave can write to checksum table, results on slave pt-table-sync can be run without pt-table-checksum can sync data via REPLACE/DELETE statements run on master or printed out for review

79 Solution #3: Fixing Data Drift/Incorrect Slave Checksum dupl_key]$ pt-table-checksum -uroot --ask-pass --replicate test. checksum --socket=/tmp/mysql_sandbox2.sock Enter MySQL password:... TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE 4-1T:48: mysql.columns_priv 4-1T:48: mysql.db 4-1T:48: mysql.event 4-1T:48: mysql.func...

80 Solution #3: Fixing Data Drift/Incorrect Slave Checksum Results Check dupl_key]$./s1 slave1 [localhost] {msandbox} (test) > SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks -> FROM test.checksum -> WHERE ( -> master_cnt <> this_cnt -> OR master_crc <> this_crc -> OR ISNULL(master_crc) <> ISNULL(this_crc)) -> GROUP BY db, tbl; db tbl total_rows chunks test dupe_test row in set (. sec)

81 Solution #3: Fixing Data Drift/Incorrect Slave Checksum Results - Simplified slave1 [localhost] {msandbox} (test) > select distinct db, tbl from test.checksum where master_crc <> this_crc; db tbl test dupe_test row in set (. sec)

82 Solution #3: Fixing Data Drift/Incorrect Slave Sync Demonstration dupl_key]$ pt-table-sync -uroot --ask-pass --print --replicate test.checksum --sync-to-master --socket=/tmp/mysql_sandbox21.sock D=test, t=dupe_test Enter password for DSN D=test,S=/tmp/mysql_sandbox21.sock,t=dupe_test,u=root: REPLACE INTO `test`.`dupe_test`(`id`, `val`) VALUES ('3', 'master3') /*perconatoolkit src_db:test src_tbl:dupe_test src_dsn:d=test,p=23992, S=/tmp/mysql_sandbox21.sock,h= ,p=...,t=dupe_test,u=root dst_db:test dst_tbl:dupe_test dst_dsn:d=test,s=/tmp/mysql_sandbox21.sock,p=...,t=dupe_test, u=root lock:1 transaction:1 changing_src:test.checksum replicate:test.checksum bidirectional: pid:26192 user:user-lab host:hostdb*/;...

83 Solution #3: Fixing Data Drift/Incorrect Slave Sync to Master Data dupl_key]$ pt-table-sync -uroot --ask-pass --execute --replicate test.checksum --sync-to-master --socket=/tmp/mysql_sandbox21.sock D=test, t=dupe_test Enter password for DSN D=test,S=/tmp/mysql_sandbox21.sock,t=dupe_test,u=root: Note: pt-table-sync is connecting to the slave socket to start, as that s the location of the test.checksum table results.

84 Solution #3: Fixing Data Drift/Incorrect Slave Sync Check dupl_key]$./m -e "checksum table test.dupe_test" Table Checksum test.dupe_test dupl_key]$./s1 -e "checksum table test.dupe_test" Table Checksum test.dupe_test

85 Problem #4: Corrupted relay log Can be caused by slave server crash master log corruption (copied over) network issues filesystem issues

86 Problem #4: Corrupted relay log Simulating relay log corruption (already done): sandboxes]$ cd corrupt_relay/node1/data data]$ ls -l mysql_sandbox241-relay-bin.9 -rw-rw user-lab user-lab 1429 Mar 31 23:24 mysql_sandbox241-relay-bin. 9 [user-lab@hostdb data]$ truncate -s 1 mysql_sandbox241-relay-bin.9 [user-lab@hostdb data]$ ls -l mysql_sandbox241-relay-bin.9 -rw-rw user-lab user-lab 1 Mar 31 23:27 mysql_sandbox241-relay-bin.9

87 Problem #4: Corrupted relay log sandboxes]$ cd corrupt_relay/node1/data corrupt_relay]$./start_all # executing "start" on /home/user-lab/sandboxes/corrupt_relay executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started [user-lab@hostdb corrupt_relay]$ tail node1/data/msandbox.err :29:1 [ERROR] Slave SQL: Error initializing relay log position: I/O error reading event at position 4, Error_code: :29:1 [Note] Slave I/O thread: connected to master 'rsandbox@ : 24',replication started in log 'mysql-bin.4' at position 17...

88 Problem #4: Corrupted relay log corrupt_relay]$./s1 slave1> show slave status\g... Master_Log_File: Read_Master_Log_Pos: Relay_Log_File: Relay_Log_Pos: Relay_Master_Log_File: Slave_IO_Running: Slave_SQL_Running:... Last_Errno: Last_Error: reading event at position 4 Skip_Counter: Exec_Master_Log_Pos: mysql-bin.4 17 mysql_sandbox241-relay-bin mysql-bin.3 Yes No 1593 Error initializing relay log position: I/O error 937

89 Solution #4: Corrupted relay log slave1> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.3', MASTER_LOG_POS=937; ERROR 1198 (HY): This operation cannot be performed with a running slave; run STOP SLAVE first slave1> STOP SLAVE; Query OK, rows affected (.1 sec) slave1> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.3', MASTER_LOG_POS=937; Query OK, rows affected (.3 sec) slave1> START SLAVE; Query OK, rows affected (. sec) slave1> SHOW SLAVE STATUS\G

90 MySQL BREAK/FIX LAB Significant Performance Issues Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214

91 Agenda - Part 3 System Bottlenecks Verify OS indicators Run diagnostics MySQL Bottlenecks MySQL CLI MySQL tools External MySQL tools

92 Significant Performance Issues Diagnosing performance issues via OS tools, performance counters/graphs and MySQL utilities/commands. Mitigating/Triaging issues with such techniques as relaxing durability, dynamic variable changes, command line fixes and managing connections/commands.

93 Possible Bottlenecks Hardware: Disk, Memory, Network Operating System: File System, Memory Management, Drivers, Scheduler RDBMS Specifics: Storage Engine, SQL Layer, Configuration Schema and Application Design: Table structures, Indexes, Data Types

94 Bottlenecks Explained Trends Memory Utilisation CPU Utilisation Disk Utilisation Network Utilisation Current Status High Load Swapping I/O Wait *

95 What to look first? Changes Application changes, patches, new version? Database schema and configuration changes? OS Changes patches, packages, updates? *

96 None of the above? Dig more Operating System Diagnostics vmstat iostat ps top sar strace lsof ifstat *

97 OS stats - vmstat root@sandbox:~# vmstat 1 1 procs memory swap io---- -system cpu---r b swpd free buff cache si so bi bo in cs us sy id wa *

98 #vmstat Cntd. io (1KB blocks) bi : blocks received from block devices bo : blocks sent to block devices system in : interrupts cs : context switches cpu us : in user space sy : in kernel code id : idle wa : waiting for IO (if virtualization is enabled) st : stolen from a virtual machine *

99 OS Stats - iostat root@sandbox:~# iostat -k -d -x 1 3 /dev/sd? Linux generic (sandbox) 3/27/214 Device: sda rrqm/s. wrqm/s.92 r/s.1 w/s.25 Device: sda rrqm/s. wrqm/s r/s 1. Device: sda rrqm/s. wrqm/s r/s. _i686_ (1 CPU) rkb/s.29 wkb/s avgrq-sz avgqu-sz await.53 svctm.3 %util.1 w/s 881. rkb/s wkb/s avgrq-sz avgqu-sz await.38 svctm.28 %util 24.4 w/s rkb/s wkb/s avgrq-sz avgqu-sz await.45 svctm.32 %util 27.7 *

100 #iostat Cntd. rrqm/s : read requests merged per second wrqm/s : write requests merged per second r/s : read requests per second w/s : write requests per second rkb/s : KBs read per second wkb/s : KBs written per second avgrq-sz : average size (in sectors) of requests avgqu-sz : average queue length of requests await : average time (milliseconds) for requests to be time) served ( queue time + service svctm : average service time (milliseconds) for requests to be served %util :percentage of CPU time during the requests : device saturation *

101 OS stats top top top - 2:32:25 up 5 days, 13:22, 2 users, load average: 2.4,.56,.24 Tasks: 8 total, 2 running, 78 sleeping, stopped, zombie Cpu(s): 5.2%us, 45.1%sy,.%ni,.%id, 4.%wa,.%hi,.7%si,.%st Mem: 25628k total, k used, 6444k free, 3512k buffers Swap: k total, 64k used, 11144k free, k cached PID USER mysql root root root root root root PR RT NI VIRT RES SHR S %CPU %MEM 162m 3m 6676 S S R 1.7. S S..6 S.. S.. TIME+ 1:36.1 :2.8 :.71 :.48 :.73 :.2 :. COMMAND mysqld sysbench jbd2/dm--8 kdmflush init kthreadd migration/ *

102 #top Shift +H, u - mysql top - 2:38:21 up 5 days, 13:28, 2 users, load average: 8.39, 6.4, 3.9 Tasks: 129 total, 15 running, 114 sleeping, stopped, zombie Cpu(s): 53.%us, 45.6%sy,.%ni,.%id,.7%wa,.%hi,.7%si,.%st Mem: 25628k total, k used, 376k free, k buffers Swap: k total, 116k used, k free, 61984k cached PID USER mysql mysql mysql mysql mysql mysql mysql mysql mysql mysql PR NI VIRT 174m 174m 174m 174m 174m 174m 174m 174m 174m 174m RES 32m 32m 32m 32m 32m 32m 32m 32m 32m 32m SHR S %CPU %MEM R S R R R R R R S R TIME+ :15.92 :4.87 :16.1 :16.69 :16.2 :15.96 :16.1 :4.79 :4.84 :4.82 COMMAND mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld *

103 OS stats ifstat ifstat -i eth 5 8 eth KB/s in KB/s out *

104 MySQL Tools (built-in) mysql client mysqladmin mysqlbinlog mysql error log mysql slow query log *

105 External Mysql tools Percona tools (pt-query-digest) tcpdump innotop *

106 MySQL CLI - I root@sandbox:~# for i in `seq 1 12` ; do mysql popsdba -e "SHOW ENGINE INNODB STATUS\G" grep "Checkpoint age " ; sleep 1 ; done > checkpoint.txt mysql> show global status like '%threads%'; Variable_name Value Delayed_insert_threads Slow_launch_threads Threads_cached Threads_connected 17 Threads_created 33 Threads_running *

107 MySQL CLI - II mysql> show global status like '%conn%'; Variable_name Value Aborted_connects 14 Connections 222 Max_used_connections 17 Ssl_client_connects Ssl_connect_renegotiates Ssl_finished_connects Threads_connected rows in set (. sec) *

108 MySQL CLI - III mysql> pager cut -d '' -f 4cut -d ':' -f 1sortuniq -c PAGER set to 'cut -d '' -f 4cut -d ':' -f 1sortuniq -c' mysql> show processlist; Host 1 localhost 1 row in set (. sec) *

109 MySQL CLI - IV mysql> \s mysql Ver Distrib , for debian-linux-gnu (i486) using readline 6.1 Connection id: Current database: Current user: SSL: Current pager: Using outfile: Using delimiter: Server version: Protocol version: Connection: Server characterset: Db characterset: Client characterset: Conn. characterset: UNIX socket: Uptime: 221 root@localhost Not in use cut -d '' -f 4cut -d ':' -f 1sortuniq -c '' ; ubuntu (Ubuntu) 1 Localhost via UNIX socket latin1 latin1 latin1 latin1 /var/run/mysqld/mysqld.sock 11 days 3 hours 15 min 26 sec Threads: 17 Questions: Slow queries: Opens: 34 Flush tables: 1 Open tables: 44 Queries per second avg: 7.44 *

110 MySQL CLI - V root@sandbox:~# while true; do echo "show engine innodb status" mysql popsdba -A -N -r grep -i "history"; sleep.5; done History list length 126 History list length 1315 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 *

111 MySQL Tools - #mysqladmin - I root@sandbox:~# mysqladmin popsdba (os_logdata)_writtencom_insert )' Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written extended -i 1 -r -c 12 egrep '(Innodb_ *

112 MySQL Tools - #mysqladmin - II mysqladmin -r -i 5 extend status >> $myadminfilename.txt Variable_name Value Aborted_clients Aborted_connects Binlog_cache_disk_use Binlog_cache_use Bytes_received 4 Bytes_sent 7751 Com_admin_commands Com_assign_to_keycache *

113 MySQL Tools - #mysqladmin - III root@sandbox:~# mysqladmin -p Enter password: Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written -r -i 5 extended-status grep Innodb_os_log_written *

114 MySQL Tools - #mysqlbinlog - I $mysqlbinlog proddb bin-log.42 egrep '^#.*exec_time' egrep -v 'exec_time= ( )' sed -e 's/exec_time=//' sort -r -n -k 1 head -n 2 $ mysqlbinlog /path/to/mysql-bin.999 grep -i -e "^update" -e "^insert" -e "^delete" -e "^replace" -e "^alter" cut -c1-1 tr '[A-Z]' '[a-z]' sed -e "s/\t/ /g;s/\`//g;s/(.*$//;s/ set.*$//;s/ as.*$//" sed -e "s/ where.*$//" sort uniq -c sort -nr update e_acc 1768 insert into r_b 1768 insert into e_rec insert into rcv_c update e_rec 185 update loc 3339 insert into r_att 2781 insert into o_att *

115 MySQL Tools - #mysqlbinlog - II $ mysqlbinlog al-db2.179 /tmp/writes.txt pt-query-digest --type=binlog --group-by=distill > $ head n 1 /tmp/writes.txt > /tmp/writes_1.txt $ egrep -c '^# /tmp/writes_1.txt awk '{print $1}' grep exec_time sort uniq exec_time= exec_time=4 exec_time=5 $ mysqlbinlog pathtobinlog pt-query-digest --type binlog --limit 3 --order-by 'Query_time:cnt' > output.txt *

116 MySQL slow query log Set slow query log seconds (warning) mysql> SET GLOBAL log_slow_verbosity='standard'; Query OK, rows affected (. sec) mysql> SET GLOBAL slow_query_log_use_global_control='long_query_time'; Query OK, rows affected (. sec) mysql> SET GLOBAL long_query_time=; Query OK, rows affected (. sec) mysql> \! mv /var/log/mysql/mysql-slow.log /var/log/mysql/mysql-slow.log mysql> FLUSH LOGS; Query OK, rows affected (.8 sec) Run detailed pt-query-digest *

117 External tools #pt-query-digest All queries ordered by time: #pt-query-digest --limit 1% /var/log/mysql/mysql-slow.log > /root/bb/mysql-slowdb1.time.digest All queries ordered by count: #pt-query-digest --limit 1% /var/log/mysql/mysql-slow.log --order-by 'Query_time:cnt' > /root/bb/mysql-slow-db1.cnt.digest All queries ordered by row examined: #pt-query-digest --limit 1% /var/log/mysql/mysql-slow.log --order-by 'Rows_examined:sum' > /root/palominodb/mysql-slow-db1.rows.digest All queries longer than 1 seconds: #pt-query-digest mysql-slow.log.1 --filter '$event->{query_time} > 1' > /tmp/mysqlslow.log.1_1sec.txt *

118 External tools #pt-query-digest II All queries by time range: #pt-query-digest --limit 1% --since " ::" --until " : :" > spike1sept.diges All queries by time range compressed: #zcat slow-log.2.gz pt-query-digest --limit 1% --since " ::" -until " ::" > spike1sept.digest All queries by time range ordered by rows examined: #pt-query-digest --limit 1% mysql_slow.log --since " ::" --until " ::" --order-by 'Rows_examined:sum' > 1314.spike.txt *

119 External tools #tcpdump Use tcpdump to capture all traffic: #tcpdump -s x -nn -q -tttt -i any -c 1 port 336 > mysql.tcp.txt Generate slowlog from output of tcpdump: #pt-query-digest --output tcpdump.slow.log --no-report --type tcpdump mysql.tcp.txt *

120 External tools #innotop Top queries When Now Total Load.. QPS 8.3k. Slow QCacheHit.%.% KCacheHit 1.%.% BpsIn 26.7k. Cmd ID State User Host Execute 46 statistics alkin localhost DISTINCT c from sbtest where id between and Execute 47 statistics alkin localhost from sbtest where id=5259 Execute 48 Sorting result alkin localhost from sbtest where id between and 524 order by Execute 49 Updating alkin localhost sbtest set k=k+1 where id=? Execute 5 statistics alkin localhost from sbtest where id=43618 BpsOut 7.1M.11 DB sysbench Time : Query SELECT sysbench : SELECT c sysbench : SELECT c sysbench : UPDATE sysbench : SELECT c *