MySQL BREAK/FIX LAB Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214
Who we are: black bird /blackbird/ noun 1. (of a people) constantly in a state of shock caused by series of incidents.
Tutorial Agenda Fix standalone MySQL instance Replication issues Performance issues
Getting started Each attendee has its own instance (not shared) we break it, you fix it :-) One standalone MySQL instance Several MySQL instances using MySQL sandbox
Getting started Access: Username/password: user-lab / blackbirdlab123 ssh user-lab@hostname Pre-requirement: ssh client http://pastebin.com/raw.php?i=rc3saj1g
Agenda - Part 1 Recover a mysql instance unable to start misconfiguration files permission corrupted files Connectivity issues misconfiguration recover password server gone away Learn how to troubleshoot crash read error log fix misconfiguration tune variables
Start mysqld [root@hostdb ~]# ps aux grep mysql root 1856.. 13424 828 pts/2 S+ 13:8 : grep mysql [root@hostdb ~]# service mysqld start chown: invalid user: `mysql:mysql' chown: invalid user: `mysql:mysql' Initializing MySQL database: chown: invalid user: `mysql' Cannot change ownership of the database directories to the 'mysql' user. Check that you have the necessary permissions and try again. [FAILED] chown: invalid user: `mysql:mysql'
mysql user [root@hostdb ~]# id mysql id: mysql: No such user [root@hostdb ~]# useradd mysql [root@hostdb ~]# id mysql uid=5(mysql) gid=51(mysql) groups=51(mysql)
Unknown variable [root@hostdb ~]# service mysqld start Initializing MySQL database: Installing MySQL system tables... 14313 13:19:9 [ERROR] /usr/libexec/mysqld: unknown variable 'tmpd1r=/var/tmp' 14313 13:19:9 [ERROR] Aborting 14313 13:19:9 [Note] /usr/libexec/mysqld: Shutdown complete [FAILED] [root@hostdb ~]#!ps ps aux grep mysql root 2185.. 13424 828 pts/2 S+ 13:19 : grep mysql
Which config file? [root@hostdb ~]# grep tmpd /etc/my.cnf [root@hostdb ~]# grep tmpd /etc/mysql/my.cnf grep: /etc/mysql/my.cnf: No such file or directory Multiple configuration file(s)?? The easy way: https://dev.mysql.com/doc/refman/5.5/en/option-files.html The hard way: strace
strace [root@hostdb ~]# strace mysqld strace: mysqld: command not found [root@hostdb ~]# strace /usr/libexec/mysqld [root@hostdb ~]# strace -e trace=open,stat /usr/libexec/mysqld... stat("/etc/my.cnf", {st_mode=s_ifreg644, st_size=243,...}) = open("/etc/my.cnf", O_RDONLY) = 3 stat("/etc/mysql/my.cnf", x7fffea4cd8) = -1 ENOENT (No such file or directory) stat("/usr/etc/my.cnf", {st_mode=s_ifreg644, st_size=25,...}) = open("/usr/etc/my.cnf", O_RDONLY) = 3 stat("/root/.my.cnf", {st_mode=s_ifreg644, st_size=33,...}) = open("/root/.my.cnf", O_RDONLY) = 3...
strace : mysqld --print-defaults # strace -e stat64 /usr/libexec/mysqld --print-defaults stat64("/etc/my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) stat64("/etc/mysql/my.cnf", {st_mode=s_ifreg644, st_size=3564,...}) = stat64("/usr/etc/my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) stat64("/root/.my.cnf", xbfb9d75) = -1 ENOENT (No such file or directory) mysqld would have been started with the following arguments: --user=mysql --socket=/var/run/mysqld/mysqld.sock --port=336 --basedir=/usr -datadir=/var/lib/mysql --tmpdir=/tmp --skip-external-locking --bind-address=127...1 --key_buffer=16m --max_allowed_packet=16m --thread_stack=192k -thread_cache_size=8 --myisam-recover=backup --query_cache_limit=1m -query_cache_size=16m --log_error=/var/log/mysql/error.log --expire_logs_days=1 -max_binlog_size=1m
Fix tmpdir [root@hostdb ~]# cat /usr/etc/my.cnf [mysqld] tmpd1r=/var/tmp [root@hostdb ~]# sed -i -e 's/tmpd1r/tmpdir/' /usr/etc/my.cnf [root@hostdb ~]# cat /usr/etc/my.cnf [mysqld] tmpdir=/var/tmp
Start again [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld:?? Not very descriptive output [FAILED]
Check error log [root@hostdb ~]# cat /var/log/mysqld.log 14313 14:5: mysqld_safe Starting mysqld daemon with databases from /var/lib/msql 14313 14:5: [Note] Plugin 'FEDERATED' is disabled. /usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist 14313 14:5: [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. 14313 14:5: InnoDB: The InnoDB memory heap is disabled 14313 14:5: InnoDB: Mutexes and rw_locks use GCC atomic builtins 14313 14:5: InnoDB: Compressed tables use zlib 1.2.7 14313 14:5: InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/iblmkyv' (Errcode: 13) 14313 14:5: InnoDB: Error: unable to create temporary file; errno: 13 14313 14:5: [ERROR] Plugin 'InnoDB' init function returned error. 14313 14:5: [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 14313 14:5: [ERROR] Unknown/unsupported storage engine: InnoDB 14313 14:5: [ERROR] Aborting 14313 14:5: [Note] /usr/libexec/mysqld: Shutdown complete 14313 14:5: mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Fix permissions [root@hostdb -rw-rw---- 1 -rw-rw---- 1 -rw-rw---- 1 ~]# ls -l root root root root root root /var/lib/mysql/mysql/plugin.* 8586 Mar 13 12:3 /var/lib/mysql/mysql/plugin.frm Mar 13 12:3 /var/lib/mysql/mysql/plugin.myd 124 Mar 13 12:3 /var/lib/mysql/mysql/plugin.myi [root@hostdb ~]# chown -R mysql:mysql /var/lib/mysql/mysql/ [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]
Check error log 14313 14:23:4 mysqld_safe Starting mysqld daemon with databases from /var/lib/msql 14313 14:23:4 [Note] Plugin 'FEDERATED' is disabled. /usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist 14313 14:23:4 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. 14313 14:23:4 InnoDB: The InnoDB memory heap is disabled 14313 14:23:4 InnoDB: Mutexes and rw_locks use GCC atomic builtins 14313 14:23:4 InnoDB: Compressed tables use zlib 1.2.7 14313 14:23:4 InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/ib9c5cf6' (Errcode: 13) 14313 14:23:5 InnoDB: Error: unable to create temporary file; errno: 13 14313 14:23:5 [ERROR] Plugin 'InnoDB' init function returned error. 14313 14:23:5 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 14313 14:23:5 [ERROR] Unknown/unsupported storage engine: InnoDB 14313 14:23:5 [ERROR] Aborting 14313 14:23:5 [Note] /usr/libexec/mysqld: Shutdown complete 14313 14:23:5 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Fix datadir path [root@hostdb ~]# grep datadir /etc/my.cnf datadir=/var/lib/msql [root@hostdb ~]# sed -i -e 's/datadir=\/var\/lib\/msql/datadir=\/var\/lib\/mysql/' /etc/my.cnf [root@hostdb ~]# grep datadir /etc/my.cnf datadir=/var/lib/mysql
Check error log 14313 14:51:6 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 14313 14:51:6 [Note] Plugin 'FEDERATED' is disabled. 14313 14:51:6 InnoDB: The InnoDB memory heap is disabled 14313 14:51:6 InnoDB: Mutexes and rw_locks use GCC atomic builtins 14313 14:51:6 InnoDB: Compressed tables use zlib 1.2.7 14313 14:51:6 InnoDB: Using Linux native AIO /usr/libexec/mysqld: Can't create/write to file '/var/tmp/ibzhtuux' (Errcode: 13) 14313 14:51:6 InnoDB: Error: unable to create temporary file; errno: 13 14313 14:51:6 [ERROR] Plugin 'InnoDB' init function returned error. 14313 14:51:6 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 14313 14:51:6 [ERROR] Unknown/unsupported storage engine: InnoDB 14313 14:51:6 [ERROR] Aborting 14313 14:51:6 [Note] /usr/libexec/mysqld: Shutdown complete 14313 14:51:6 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Fix tmpdir [root@hostdb ~]# ls -ld /var/tmp drwxrwx--t 2 root root 496 Mar 13 13:23 /var/tmp [root@hostdb ~]# chmod a+rwx /var/tmp [root@hostdb ~]# ls -ld /var/tmp drwxrwxrwt 2 root root 496 Mar 13 13:23 /var/tmp [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]
Cannot allocate memory 14313 14:58:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 14313 14:58:48 [Note] Plugin 'FEDERATED' is disabled. 14313 14:58:48 InnoDB: The InnoDB memory heap is disabled 14313 14:58:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins 14313 14:58:48 InnoDB: Compressed tables use zlib 1.2.7 14313 14:58:48 InnoDB: Using Linux native AIO 14313 14:58:48 InnoDB: Initializing buffer pool, size = 1.G InnoDB: mmap(19897648 bytes) failed; errno 12 14313 14:58:48 InnoDB: Completed initialization of buffer pool 14313 14:58:48 InnoDB: Fatal error: cannot allocate memory for the buffer pool 14313 14:58:48 [ERROR] Plugin 'InnoDB' init function returned error. 14313 14:58:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 14313 14:58:48 [ERROR] Unknown/unsupported storage engine: InnoDB 14313 14:58:48 [ERROR] Aborting 14313 14:58:48 [Note] /usr/libexec/mysqld: Shutdown complete 14313 14:58:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Fix innodb_buffer_pool_size [root@hostdb ~]# perror 12 OS error code 12: Cannot allocate memory [root@hostdb ~]# grep 1 /etc/my.cnf innodb_buffer_pool_size=1g [root@hostdb ~]# sed -i -e 's/1g/256m/' /etc/my.cnf [root@hostdb ~]# grep innodb_buffer_pool_size /etc/my.cnf innodb_buffer_pool_size=256m [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]
Again error 13 14313 15:11:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 14313 15:11:57 [Note] Plugin 'FEDERATED' is disabled. 14313 15:11:57 InnoDB: The InnoDB memory heap is disabled 14313 15:11:57 InnoDB: Mutexes and rw_locks use GCC atomic builtins 14313 15:11:57 InnoDB: Compressed tables use zlib 1.2.7 14313 15:11:57 InnoDB: Using Linux native AIO 14313 15:11:57 InnoDB: Initializing buffer pool, size = 256.M 14313 15:11:57 InnoDB: Completed initialization of buffer pool 14313 15:11:57 InnoDB: Operating system error number 13 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. InnoDB: File name./ibdata1 InnoDB: File operation call: 'open'. InnoDB: Cannot continue operation. 14313 15:11:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Fix permissions [root@hostdb -rw-rw---- 1 [root@hostdb total 8398 -rw-rw---- 1 -rw-rw---- 1 -rw-rw---- 1 drwx------ 2 drwx------ 2 drwx------ 2 ~]# ls -l /var/lib/mysql/ibdata1 27 27 18874368 Mar 13 12:34 /var/lib/mysql/ibdata1 ~]# ls -l /var/lib/mysql 27 27 27 mysql root root 27 18874368 Mar 13 12:34 ibdata1 27 33554432 Mar 13 12:34 ib_logfile 27 33554432 Mar 13 12:34 ib_logfile1 mysql 496 Mar 13 12:3 mysql root 496 Mar 13 12:3 performance_schema root 496 Mar 13 12:3 test [root@hostdb ~]# chown -R mysql:mysql /var/lib/mysql [root@hostdb ~]# service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED]
innodb_log_file_size 14313 17:15: mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 14313 17:15: [Note] Plugin 'FEDERATED' is disabled. 14313 17:15: InnoDB: The InnoDB memory heap is disabled 14313 17:15: InnoDB: Mutexes and rw_locks use GCC atomic builtins 14313 17:15: InnoDB: Compressed tables use zlib 1.2.7 14313 17:15: InnoDB: Using Linux native AIO 14313 17:15: InnoDB: Initializing buffer pool, size = 256.M 14313 17:15:1 InnoDB: Completed initialization of buffer pool InnoDB: Error: log file./ib_logfile is of different size 33554432 bytes InnoDB: than specified in the.cnf file 6718864 bytes! 14313 17:15:1 [ERROR] Plugin 'InnoDB' init function returned error. 14313 17:15:1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 14313 17:15:1 [ERROR] Unknown/unsupported storage engine: InnoDB 14313 17:15:1 [ERROR] Aborting 14313 17:15:1 [Note] /usr/libexec/mysqld: Shutdown complete 14313 17:15:1 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Fix innodb_log_file_size [root@hostdb ~]# grep innodb_log_file_size /etc/my.cnf innodb_log_file_size=64m [root@hostdb ~]# ls -l /var/lib/mysql/ib_logfile* -rw-rw---- 1 mysql mysql 33554432 Mar 13 12:34 /var/lib/mysql/ib_logfile -rw-rw---- 1 mysql mysql 33554432 Mar 13 12:34 /var/lib/mysql/ib_logfile1 [root@hostdb ~]# sed -i -e 's/innodb_log_file_size=64m/innodb_log_file_size=32m/' /etc/my.cnf [root@hostdb ~]# grep innodb_log_file_size /etc/my.cnf innodb_log_file_size=32m [root@hostdb ~]# service mysqld start Starting mysqld: [ OK ]
Access to mysqld [root@hostdb ~]# mysql ERROR 22 (HY): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) [root@hostdb ~]# perror 2 OS error code 2: No such file or directory [root@hostdb ~]# ls -l /tmp/mysql.sock ls: cannot access /tmp/mysql.sock: No such file or directory
Access to mysqld [root@hostdb ~]# grep socket /var/log/mysqld.log tail -n 1 Version: '5.5.34' socket: '/var/lib/mysql/mysql.sock' port: 336 Community Server (GPL) [root@hostdb ~]# lsof -n grep mysqld grep unix mysqld 21737 mysql 12u unix xffff882edd4 t /var/lib/mysql/mysql.sock In /etc/my.cnf : [client] socket=/tmp/mysql.sock MySQL 22829
Access to mysqld [root@hostdb ~]# mysql ERROR 145 (28): Access denied for user 'root'@'localhost' (using password: YES) [root@hostdb ~]# strace -e trace=open mysql open("/etc/my.cnf", O_RDONLY) = 3 open("/usr/etc/my.cnf", O_RDONLY) = 3 open("/root/.my.cnf", O_RDONLY) = 3 [root@hostdb ~]# cat ~/.my.cnf [client] password=adummypassword
Access to mysqld [root@hostdb ~]# mysql --no-defaults [root@hostdb ~]# mysql -p
Change root password [root@hostdb ~]# echo "SET PASSWORD=PASSWORD('$RANDOM$RANDOM')" mysql [root@hostdb ~]# mysql ERROR 145 (28): Access denied for user 'root'@'localhost' (using password: NO) In /etc/my.cnf : skip-grant-tables [root@hostdb ~]# service mysqld restart Stopping mysqld: Starting mysqld: [ [ OK OK ] ]
Change root password [root@hostdb ~]# mysql mysql> UPDATE mysql.user SET password=password('newpass') WHERE user='root'; mysql> FLUSH PRIVILEGES; Remove skip-grant-tables from /etc/my.cnf
removed ib_logfile (DON T!) Free space was low: I deleted some log files... [root@hostdb ~]# cd /var/lib/mysql [root@hostdb mysql]# rm -rf ib_logfile* [root@hostdb mysql]# lsof -n grep ib_logfile mysqld 1276 mysql 8uW REG 22,1 33554432 mysqld 1276 mysql 9uW REG 22,1 33554432 4353 /var/lib/mysql/ib_logfile (deleted) 4355 /var/lib/mysql/ib_logfile1 (deleted)
removed ib_logfile (DON T!) mysqld and InnoDB continue working normally mysql> use test; mysql> create table tbl1 (id int auto_increment primary key, v varchar(1)) engine=innodb; mysql> insert into tbl1 values(null,'aa'); mysql> insert into tbl1 select null, v from tbl1; mysql> SELECT * FROM tbl1; [root@hostdb ~]# service mysqld restart
removed ib_logfile (DON T!) InnoDB Redo Log are automatically recreated 14321 14:42:2 InnoDB: Log file./ib_logfile did not exist: new to be created InnoDB: Setting log file./ib_logfile size to 32 MB InnoDB: Database physically writes the file full: wait... 14321 14:42:21 InnoDB: Log file./ib_logfile1 did not exist: new to be created InnoDB: Setting log file./ib_logfile1 size to 32 MB InnoDB: Database physically writes the file full: wait... 14321 14:42:22 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 14321 14:42:22 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the.ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer...
removed ibdata1 (really, DON T!) Free space was low: I deleted a large file... [root@hostdb mysql]# rm -f ibdata1 [root@ip-1-37-217-26 mysql]# lsof -n grep deleted mysqld 7478 mysql 3uW REG 22,1 27262976 135257 (deleted) mysqld 7478 mysql 4u REG 22,1 14613 (deleted) mysqld 7478 mysql 5u REG 22,1 146391 (deleted) mysqld 7478 mysql 6u REG 22,1 146392 (deleted) mysqld 7478 mysql 7u REG 22,1 146393 (deleted) mysqld 7478 mysql 11u REG 22,1 146394 (deleted) [root@hostdb mysql]# lsof -n grep ibdata1 mysqld 7478 mysql 3uW REG 22,1 27262976 135257 (deleted) /var/lib/mysql/ibdata1 /var/tmp/ibtkakep /var/tmp/ibf1ray /var/tmp/ib3piqic /var/tmp/ibl8k16o /var/tmp/ib7sjqkq /var/lib/mysql/ibdata1
removed ibdata1 (really, DON T!) mysqld and InnoDB continue working normally : mysql> insert into tbl1 values(null,'aa'); mysql> insert into tbl1 select null, v from tbl1; mysql> SELECT * FROM tbl1; until restart : [root@hostdb ~]# service mysqld restart Stopping mysqld: [ OK ] MySQL Daemon failed to start. Starting mysqld: [FAILED]
removed ibdata1 (really, DON T!) You can now recover from backup!
removed ibdata1 (really, DON T!) InnoDB: The first specified data file./ibdata1 did not exist: InnoDB: a new database to be created! 14321 15:48:28 InnoDB: Setting file./ibdata1 size to 18 MB InnoDB: Database physically writes the file full: wait... 14321 15:48:28 InnoDB: Error: all log files must be created at the same time. 14321 15:48:28 InnoDB: All log files must be created also in database creation. 14321 15:48:28 InnoDB: If you want bigger or smaller log files, shut down the 14321 15:48:28 InnoDB: database and make sure there were no errors in shutdown. 14321 15:48:28 InnoDB: Then delete the existing log files. Edit the.cnf file 14321 15:48:28 InnoDB: and start the database again. 14321 15:48:28 [ERROR] Plugin 'InnoDB' init function returned error. 14321 15:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 14321 15:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB 14321 15:48:28 [ERROR] Aborting Misleading errors
removed ibdata1 (really, DON T!) [root@hostdb mysql]# rm -f ibdata1 ib_logfile* test/* [root@hostdb ~]# service mysqld restart
MySQL server has gone away mysql> SELECT SLEEP(1); ERROR 213 (HY): Lost connection to MySQL server during query # kill -9 `pidof mysqld` # kill -6 `pidof mysqld` # kill -11 `pidof mysqld`
MySQL server has gone away Version: '5.5.34' socket: '/var/lib/mysql/mysql.sock' port: 336 MySQL Community Server (GPL) 14325 13:6:1 mysqld_safe Number of processes running now: 14325 13:6:1 mysqld_safe mysqld restarted
MySQL server has gone away OOM Killer crashing bug signals others Always check logs: - mysql error log - syslog
MySQL server has gone away [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 1` ; do echo -n "123456789" ; done ; echo -n "' a") mysql a 123456789 [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 2` ; do echo -n "123456789" ; done ; echo -n "' a") mysql a 123456789123456789 [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 1` ; do echo -n "123456789" ; done ; echo -n "' a") mysql wc 2 2 13 [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 11` ; do echo -n "123456789" ; done ; echo -n "' a") mysql wc ERROR 26 (HY) at line 1: MySQL server has gone away
MySQL server has gone away [root@hostdb ~]# mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_allowed_packet'" +--------------------+---------+ Variable_name Value +--------------------+---------+ max_allowed_packet 148576 +--------------------+---------+ [root@hostdb ~]# mysql -e "SET GLOBAL max_allowed_packet=148576" [root@hostdb ~]# ( echo -n "SELECT '" ; for i in `seq 1 11` ; do echo -n "123456789" ; done ; echo -n "' a") mysql wc 2 2 113
MySQL server has gone away server crash server killed session terminated/killed session timing out (wait_timeout) big packet (max_allowed_packet)
MySQL BREAK/FIX LAB Replication Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214
Agenda - Part 2 Replication overview and tools Bad server-id slave id identical to master slave id identical to other slave Incorrect slave data Duplicate key error Data drift Relay log corruption
Replication Overview asynchronous: based on copying and executing of binary logs allows for delayed slaves, incremental backups binary logs have two formats statement: SQL row events: starting with 5.1 replication is run by two threads on slave IO thread: copies master binary logs to slave relay logs SQL thread: executes binlog events on data
Replication Processlist Threads Slave: slave1 [localhost] {msandbox} ((none)) > show processlist; +----+-------------+-----------+------+---------+-----+----------------------------------------------------------------------------+------------------+ Id User Host db Command Time State Info +----+-------------+-----------+------+---------+-----+----------------------------------------------------------------------------+------------------+ 1 system user NULL Connect 31 Slave has read all relay log; waiting for the slave I/O thread to update it NULL 2 system user NULL Connect 31 Waiting for master to send event NULL 3 msandbox localhost NULL Query NULL show processlist +----+-------------+-----------+------+---------+-----+----------------------------------------------------------------------------+------------------+ 3 rows in set (. sec)
Replication Processlist Threads Master: master [localhost] {msandbox} ((none)) > show processlist; +----+----------+-----------------+------+-------------+-----+-----------------------------------------------------------------------+------------------+ Id User Host db Command Time State Info +----+----------+-----------------+------+-------------+-----+-----------------------------------------------------------------------+------------------+ 1 rsandbox localhost:4494 NULL Binlog Dump 147 Master has sent all binlog to slave; waiting for binlog to be updated NULL 2 rsandbox localhost:4495 NULL Binlog Dump 144 Master has sent all binlog to slave; waiting for binlog to be updated NULL 4 msandbox localhost NULL Query NULL show processlist +----+----------+-----------------+------+-------------+-----+-----------------------------------------------------------------------+------------------+ 3 rows in set (. sec)
Replication Tools MySQL Sandbox Percona Toolkit pt-table-checksum pt-table-sync pt-slave-restart mysqlbinlog error logs
Replication Tools (cont.) slave commands SHOW SLAVE STATUS\G STOP SLAVE; START SLAVE; STOP/START SLAVE IO_THREAD; STOP/START SLAVE SQL_THREAD; master commands SHOW MASTER STATUS; SHOW SLAVE HOSTS;
MySQL Sandbox http://mysqlsandbox.net/ created by Giuseppe Maxia (The Data Charmer) Linux/FreeBSD/MacOSX only not for production instances allows you to install and run multiple instances of MySQL on same server can easily set up replication groups default is master and two slaves can be run as regular (non-root) user
Replication Sandbox [user-lab@hostdb repl_test]$ pwd /home/user-lab/sandboxes/repl_test [user-lab@hostdb repl_test]$ ls check_slaves m clear_all master connection.json node1 default_connection.json node2 initialize_slaves README restart_all s1 s2 send_kill_all start_all status_all stop_all use_all [user-lab@hostdb repl_test]$./start_all # executing "start" on /home/user-lab/sandboxes/repl_test executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started executing "start" on slave 2... sandbox server started
Sandbox Master [user-lab@hostdb repl_test]$ cd master/ [user-lab@hostdb master]$ ls change_paths grants.mysql change_ports json_in_db clear load_grants connection.json msb data my default_connection.json my.sandbox.cnf proxy_start README rescue_mysql_dump.sql restart send_kill start [user-lab@hostdb master]$ ls data/ ibdata1 msandbox.err mysql-bin.index ib_logfile mysql mysql_sandbox23992.pid ib_logfile1 mysql-bin.1 performance_schema test status stop tmp use USING
Sandbox Slave [user-lab@hostdb master]$ cd../node1 [user-lab@hostdb change_paths change_ports clear connection.json data node1]$ ls default_connection.json grants.mysql json_in_db load_grants msb my my.sandbox.cnf proxy_start README restart [user-lab@hostdb node1]$ ls data ibdata1 mysql_sandbox23993.pid ib_logfile mysql_sandbox23993-relay-bin.1 ib_logfile1 mysql_sandbox23993-relay-bin.2 master.info mysql_sandbox23993-relay-bin.index msandbox.err performance_schema mysql relay-log.info mysql-bin.1 test mysql-bin.index send_kill start status stop tmp use USING
Connecting to MySQL Instances [user-lab@hostdb repl_test]$ ls check_slaves m clear_all master connection.json node1 default_connection.json node2 initialize_slaves README restart_all s1 s2 send_kill_all start_all status_all stop_all use_all [user-lab@hostdb repl_test]$./m Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 master [localhost] {msandbox} ((none)) > [user-lab@hostdb repl_test]$./s1 -uroot Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 slave1 [localhost] {root} ((none)) >
Problem #1: Duplicate server-id value the server-id uniquely identifies a server in a replication topology recorded in binary log dynamic variable (doesn t require restart) set in the my.cnf file (my.sandbox.cnf) if a server is cloned (e.g., ec2 snapshot), the server-id needs to be edited in new instance
Problem #1: Duplicate server-id value Example of slave with identical server_id to master: $./s1 -e "show slave status\g" *************************** 1. row *************************** Slave_IO_State: Slave_IO_Running: No Slave_SQL_Running: Yes Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 1593 Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have equal MySQL server ids; these ids must be different for replication to work (or the --replicate-same-server-id option must be used on slave but this does not always make sense; please check the manual before using it). Master_Server_Id: 1
Problem #1: Duplicate server-id values with two slaves with identical server-ids, the symptoms and error messages are misleading master will show only one slave in processlist, continuously reconnecting
Problem #1: Duplicate server-id values slave status looks okay on both, but error logs will fill up: [user-lab@hostdb ~]$ cd sandboxes/ [user-lab@hostdb sandboxes]$ cd dupl_server_id2 [user-lab@hostdb dupl_server_id2]$./start_all [user-lab@hostdb dupl_server_id2]$ tail node1/data/msandbox.err 1433 8:25:47 [Note] Slave: received end packet from server, apparent master shutdown: 1433 8:25:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.3' at position 17 1433 8:25:47 [Note] Slave: received end packet from server, apparent master shutdown: 1433 8:25:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.3' at position 17
Solution #1: Modify slave server-id on one slave 1. set server_id variable [user-lab@hostdb dupl_server_id2]$./s1 slave1 [localhost] {msandbox} ((none)) > show global variables like 'server_id'; +---------------+-------+ Variable_name Value +---------------+-------+ server_id 11 +---------------+-------+ 1 row in set (. sec) slave1 [localhost] {msandbox} ((none)) > set global server_id = 12; Query OK, rows affected (. sec) slave1 [localhost] {msandbox} ((none)) > stop slave; start slave; Query OK, rows affected (.2 sec) Query OK, rows affected (. sec) slave1 [localhost] {msandbox} ((none)) > show slave status\g
Solution #1: Modify slave server-id on one slave 2. edit node1/my.sandbox.cnf to use different server-id [user-lab@hostdb dupl_server_id2]$ cd node1 [user-lab@hostdb node1]$ sed -i 's/server-id=11/server-id=12/g' my.sandbox.cnf [user-lab@hostdb node1]$ grep server-id my.sandbox.cnf server-id=12
Problem #2: Duplicate key errors very common replication error number of possible causes slave created from bad backup slave replication started in wrong file/pos direct write to the slave data drift
Problem #2: Duplicate key errors Example [user-lab@hostdb sandboxes]$ cd dupl_key/ [user-lab@hostdb dupl_key]$./start_all # executing "start" on /home/user-lab/sandboxes/dupl_key executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started [user-lab@hostdb dupl_key]$./s1 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3
Problem #2: Duplicate key errors Example slave1 [localhost] {msandbox} (test) > show slave status\g *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 127...1 Master_User: rsandbox... Slave_IO_Running: Yes Slave_SQL_Running: No... Last_Errno: 162 Last_Error: Error 'Duplicate entry '3' for key 'PRIMARY'' on query. Default database: 'test'. Query: 'insert into dupe_test (val) values ('master3'), ('master4')'... Seconds_Behind_Master: NULL
Problem #2: Duplicate key errors Example (cont.) slave1 [localhost] {msandbox} ((none)) > use test; slave1 [localhost] {msandbox} (test) > show create table dupe_test\g *************************** 1. row *************************** Table: dupe_test Create Table: CREATE TABLE `dupe_test` ( `id` int(11) NOT NULL AUTO_INCREMENT, `val` varchar(2) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 1 row in set (. sec) slave1 [localhost] {msandbox} (test) > select * from dupe_test where id = 3; +----+--------+ id val +----+--------+ 3 slave1 +----+--------+ 1 row in set (. sec)
Problem #2: Duplicate key errors Example (cont.) [user-lab@hostdb dupl_key]$ cd master [user-lab@hostdb master]$./my sqlbinlog data/mysql-bin.3... # at 521 #1441 7:5:14 server id 1 end_log_pos 589 Querythread_id=3 error_code= SET TIMESTAMP=1396335914/*!*/; BEGIN /*!*/; # at 589 #1441 7:5:14 server id 1 end_log_pos 617 Intvar SET INSERT_ID=3/*!*/; # at 617 #1441 7:5:14 server id 1 end_log_pos 739 Querythread_id=3 error_code= SET TIMESTAMP=1396335914/*!*/; insert into dupe_test (val) values ("master3"), ('master4') /*!*/;... exec_time= exec_time=
Solution #2: Duplicate key errors General solutions: 1. recreate slave from fresh backup of master (preferred) 2. get replication running again, fix data differences later We will be doing solution #2.
Solution #2: Duplicate key errors To check if the error is due to a direct write to slave: [user-lab@hostdb dupl_key]$ cd node1 [user-lab@hostdb node1]$ pwd /home/user-lab/sandboxes/dupl_key/node1 [user-lab@hostdb node1]$ ls data/mysql-bin.* data/mysql-bin.1 data/mysql-bin.2 data/mysql-bin.index [user-lab@hostdb node1]$./my sqlbinlog --server-id=11 data/mysql-bin. 2 grep dupe_test insert into dupe_test (val) values ('slave1')
Solution #2: Duplicate key errors Do this (if it s a read slave): [mysqld] read_only Do not do this: [mysqld] slave-skip-errors = all
Solution #2: Duplicate key errors Hack #1: Get slave caught up Manually: [user-lab@hostdb dupl_key]$./s1 slave1 > stop slave; set global sql_slave_skip_counter = 1; start slave; slave1 > show slave status\g If you have a lot of events to skip, you can use pt-slaverestart.
Solution #2: Duplicate key errors Hack #1: Get slave caught up note this skips binary log events for InnoDB tables: entire transaction can t easily guarantee that master and slave data are equivalent later: run checksum to compare master and slave
Problem #3: Data Drift Statement-based binary logging can result in incorrect data on the slave nondeterministic updates with LIMIT but no ORDER BY certain system and math functions etc. ROW and MIXED binary logging avoids this
Problem #3: Data Drift [user-lab@ip-1-74-3-3 dupl_key]$./m master [localhost] {msandbox} ((none)) > use test; master [localhost] {msandbox} (test) > insert into dupe_test (val) values (uuid()); Query OK, 1 row affected, 2 warnings (.4 sec) master [localhost] {msandbox} (test) > show warnings\g *************************** 1. row *************************** Level: Warning Code: 1265 Message: Data truncated for column 'val' at row 1 *************************** 2. row *************************** Level: Note Code: 1592 Message: Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it uses a system function that may return a different value on the slave. 2 rows in set (. sec)
Problem #3: Data Drift master [localhost] {msandbox} (test) > select * from dupe_test order by id desc limit 1; +----+----------------------+ id val +----+----------------------+ 9 f6fe11-b932-11e3-a +----+----------------------+ 1 row in set (.2 sec) slave1 [localhost] {msandbox} ((none)) > select * from test.dupe_test order by id desc limit 1; +----+----------------------+ id val +----+----------------------+ 9 f6ff6a8c-b932-11e3-a +----+----------------------+ 1 row in set (. sec)
Solution #3: Fixing Data Drift/Incorrect Slave Hack #2: Checksum and sync slave data pt-table-checksum checksums tables in chunks of rows can use replication to compare master and slave can write to checksum table, results on slave pt-table-sync can be run without pt-table-checksum can sync data via REPLACE/DELETE statements run on master or printed out for review
Solution #3: Fixing Data Drift/Incorrect Slave Checksum [user-lab@hostdb dupl_key]$ pt-table-checksum -uroot --ask-pass --replicate test. checksum --socket=/tmp/mysql_sandbox2.sock Enter MySQL password:... TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE 4-1T:48:17 1.51 mysql.columns_priv 4-1T:48:17 1.52 mysql.db 4-1T:48:17 1.49 mysql.event 4-1T:48:17 1.5 mysql.func...
Solution #3: Fixing Data Drift/Incorrect Slave Checksum Results Check [user-lab@hostdb dupl_key]$./s1 slave1 [localhost] {msandbox} (test) > SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks -> FROM test.checksum -> WHERE ( -> master_cnt <> this_cnt -> OR master_crc <> this_crc -> OR ISNULL(master_crc) <> ISNULL(this_crc)) -> GROUP BY db, tbl; +------+-----------+------------+--------+ db tbl total_rows chunks +------+-----------+------------+--------+ test dupe_test 4 1 +------+-----------+------------+--------+ 1 row in set (. sec)
Solution #3: Fixing Data Drift/Incorrect Slave Checksum Results - Simplified slave1 [localhost] {msandbox} (test) > select distinct db, tbl from test.checksum where master_crc <> this_crc; +------+-----------+ db tbl +------+-----------+ test dupe_test +------+-----------+ 1 row in set (. sec)
Solution #3: Fixing Data Drift/Incorrect Slave Sync Demonstration [user-lab@hostdb dupl_key]$ pt-table-sync -uroot --ask-pass --print --replicate test.checksum --sync-to-master --socket=/tmp/mysql_sandbox21.sock D=test, t=dupe_test Enter password for DSN D=test,S=/tmp/mysql_sandbox21.sock,t=dupe_test,u=root: REPLACE INTO `test`.`dupe_test`(`id`, `val`) VALUES ('3', 'master3') /*perconatoolkit src_db:test src_tbl:dupe_test src_dsn:d=test,p=23992, S=/tmp/mysql_sandbox21.sock,h=127...1,p=...,t=dupe_test,u=root dst_db:test dst_tbl:dupe_test dst_dsn:d=test,s=/tmp/mysql_sandbox21.sock,p=...,t=dupe_test, u=root lock:1 transaction:1 changing_src:test.checksum replicate:test.checksum bidirectional: pid:26192 user:user-lab host:hostdb*/;...
Solution #3: Fixing Data Drift/Incorrect Slave Sync to Master Data [user-lab@hostdb dupl_key]$ pt-table-sync -uroot --ask-pass --execute --replicate test.checksum --sync-to-master --socket=/tmp/mysql_sandbox21.sock D=test, t=dupe_test Enter password for DSN D=test,S=/tmp/mysql_sandbox21.sock,t=dupe_test,u=root: Note: pt-table-sync is connecting to the slave socket to start, as that s the location of the test.checksum table results.
Solution #3: Fixing Data Drift/Incorrect Slave Sync Check [user-lab@hostdb dupl_key]$./m -e "checksum table test.dupe_test" +----------------+------------+ Table Checksum +----------------+------------+ test.dupe_test 197688199 +----------------+------------+ [user-lab@hostdb dupl_key]$./s1 -e "checksum table test.dupe_test" +----------------+------------+ Table Checksum +----------------+------------+ test.dupe_test 197688199 +----------------+------------+
Problem #4: Corrupted relay log Can be caused by slave server crash master log corruption (copied over) network issues filesystem issues
Problem #4: Corrupted relay log Simulating relay log corruption (already done): [user-lab@hostdb sandboxes]$ cd corrupt_relay/node1/data [user-lab@hostdb data]$ ls -l mysql_sandbox241-relay-bin.9 -rw-rw---- 1 user-lab user-lab 1429 Mar 31 23:24 mysql_sandbox241-relay-bin. 9 [user-lab@hostdb data]$ truncate -s 1 mysql_sandbox241-relay-bin.9 [user-lab@hostdb data]$ ls -l mysql_sandbox241-relay-bin.9 -rw-rw---- 1 user-lab user-lab 1 Mar 31 23:27 mysql_sandbox241-relay-bin.9
Problem #4: Corrupted relay log [user-lab@hostdb sandboxes]$ cd corrupt_relay/node1/data [user-lab@hostdb corrupt_relay]$./start_all # executing "start" on /home/user-lab/sandboxes/corrupt_relay executing "start" on master... sandbox server started executing "start" on slave 1... sandbox server started [user-lab@hostdb corrupt_relay]$ tail node1/data/msandbox.err... 14331 23:29:1 [ERROR] Slave SQL: Error initializing relay log position: I/O error reading event at position 4, Error_code: 1593 14331 23:29:1 [Note] Slave I/O thread: connected to master 'rsandbox@127...1: 24',replication started in log 'mysql-bin.4' at position 17...
Problem #4: Corrupted relay log [user-lab@hostdb corrupt_relay]$./s1 slave1> show slave status\g... Master_Log_File: Read_Master_Log_Pos: Relay_Log_File: Relay_Log_Pos: Relay_Master_Log_File: Slave_IO_Running: Slave_SQL_Running:... Last_Errno: Last_Error: reading event at position 4 Skip_Counter: Exec_Master_Log_Pos: mysql-bin.4 17 mysql_sandbox241-relay-bin.9 9453 mysql-bin.3 Yes No 1593 Error initializing relay log position: I/O error 937
Solution #4: Corrupted relay log slave1> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.3', MASTER_LOG_POS=937; ERROR 1198 (HY): This operation cannot be performed with a running slave; run STOP SLAVE first slave1> STOP SLAVE; Query OK, rows affected (.1 sec) slave1> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.3', MASTER_LOG_POS=937; Query OK, rows affected (.3 sec) slave1> START SLAVE; Query OK, rows affected (. sec) slave1> SHOW SLAVE STATUS\G
MySQL BREAK/FIX LAB Significant Performance Issues Presented by: Alkin Tezuysal, Gillian Gunson, René Cannaò April 1, 214
Agenda - Part 3 System Bottlenecks Verify OS indicators Run diagnostics MySQL Bottlenecks MySQL CLI MySQL tools External MySQL tools
Significant Performance Issues Diagnosing performance issues via OS tools, performance counters/graphs and MySQL utilities/commands. Mitigating/Triaging issues with such techniques as relaxing durability, dynamic variable changes, command line fixes and managing connections/commands.
Possible Bottlenecks Hardware: Disk, Memory, Network Operating System: File System, Memory Management, Drivers, Scheduler RDBMS Specifics: Storage Engine, SQL Layer, Configuration Schema and Application Design: Table structures, Indexes, Data Types
Bottlenecks Explained Trends Memory Utilisation CPU Utilisation Disk Utilisation Network Utilisation Current Status High Load Swapping I/O Wait *
What to look first? Changes Application changes, patches, new version? Database schema and configuration changes? OS Changes patches, packages, updates? *
None of the above? Dig more Operating System Diagnostics vmstat iostat ps top sar strace lsof ifstat *
OS stats - vmstat root@sandbox:~# vmstat 1 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---r b swpd free buff cache si so bi bo in cs us sy id wa 9 1 64 3812 1388 86128 6 14 26 1 16 1 64 3992 14124 8568 12192 19 38386 5 48 2 11 64 3992 14472 85212 1558 1111 36948 46 53 1 5 64 4172 14644 8498 1244 156 37181 49 49 1 1 64 3872 14972 8494 11988 973 32739 54 45 1 8 1 64 688 15244 8172 15572 165 37857 55 44 1 16 1 64 6448 156 8178 15552 142 36428 51 48 1 8 1 64 628 15972 8178 1734 173 35714 54 42 4 11 1 64 5728 1636 817 144 171 3616 48 51 1 14 1 64 5248 16736 81716 15596 157 37619 51 47 2 *
#vmstat Cntd. io (1KB blocks) bi : blocks received from block devices bo : blocks sent to block devices system in : interrupts cs : context switches cpu us : in user space sy : in kernel code id : idle wa : waiting for IO (if virtualization is enabled) st : stolen from a virtual machine *
OS Stats - iostat root@sandbox:~# iostat -k -d -x 1 3 /dev/sd? Linux 2.6.32-38-generic (sandbox) 3/27/214 Device: sda rrqm/s. wrqm/s.92 r/s.1 w/s.25 Device: sda rrqm/s. wrqm/s 2618. r/s 1. Device: sda rrqm/s. wrqm/s 2653.54 r/s. _i686_ (1 CPU) rkb/s.29 wkb/s avgrq-sz avgqu-sz 4.45 36.43. await.53 svctm.3 %util.1 w/s 881. rkb/s wkb/s avgrq-sz avgqu-sz 24. 13176. 29.93.32 await.38 svctm.28 %util 24.4 w/s 848.48 rkb/s wkb/s avgrq-sz avgqu-sz. 13163.64 31.3.38 await.45 svctm.32 %util 27.7 *
#iostat Cntd. rrqm/s : read requests merged per second wrqm/s : write requests merged per second r/s : read requests per second w/s : write requests per second rkb/s : KBs read per second wkb/s : KBs written per second avgrq-sz : average size (in sectors) of requests avgqu-sz : average queue length of requests await : average time (milliseconds) for requests to be time) served ( queue time + service svctm : average service time (milliseconds) for requests to be served %util :percentage of CPU time during the requests : device saturation *
OS stats top root@sandbox:~# top top - 2:32:25 up 5 days, 13:22, 2 users, load average: 2.4,.56,.24 Tasks: 8 total, 2 running, 78 sleeping, stopped, zombie Cpu(s): 5.2%us, 45.1%sy,.%ni,.%id, 4.%wa,.%hi,.7%si,.%st Mem: 25628k total, 244184k used, 6444k free, 3512k buffers Swap: 111414k total, 64k used, 11144k free, 152528k cached PID 1981 4494 192 174 1 2 3 USER mysql root root root root root root PR 2 2 2 2 2 2 RT NI VIRT RES SHR S %CPU %MEM 162m 3m 6676 S 79.5 12.6 15484 3632 142 S 13.6 1.4 R 1.7. S 1.3. 2672 156 1212 S..6 S.. S.. TIME+ 1:36.1 :2.8 :.71 :.48 :.73 :.2 :. COMMAND mysqld sysbench jbd2/dm--8 kdmflush init kthreadd migration/ *
#top Shift +H, u - mysql top - 2:38:21 up 5 days, 13:28, 2 users, load average: 8.39, 6.4, 3.9 Tasks: 129 total, 15 running, 114 sleeping, stopped, zombie Cpu(s): 53.%us, 45.6%sy,.%ni,.%id,.7%wa,.%hi,.7%si,.%st Mem: 25628k total, 246868k used, 376k free, 126316k buffers Swap: 111414k total, 116k used, 1113988k free, 61984k cached PID 456 4541 1998 3166 452 455 457 4537 4539 454 USER mysql mysql mysql mysql mysql mysql mysql mysql mysql mysql PR 2 2 2 2 2 2 2 2 2 2 NI VIRT 174m 174m 174m 174m 174m 174m 174m 174m 174m 174m RES 32m 32m 32m 32m 32m 32m 32m 32m 32m 32m SHR 6588 6588 6588 6588 6588 6588 6588 6588 6588 6588 S %CPU %MEM R 5.3 13.5 S 5.3 13.5 R 5. 13.5 R 5. 13.5 R 5. 13.5 R 5. 13.5 R 5. 13.5 R 5. 13.5 S 5. 13.5 R 5. 13.5 TIME+ :15.92 :4.87 :16.1 :16.69 :16.2 :15.96 :16.1 :4.79 :4.84 :4.82 COMMAND mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld *
OS stats ifstat root@sandbox:~# ifstat -i eth 5 8 eth KB/s in KB/s out.1.3.1.2.1.2.4.5.35.42.32.43 5.52.72.1.2 *
MySQL Tools (built-in) mysql client mysqladmin mysqlbinlog mysql error log mysql slow query log *
External Mysql tools Percona tools (pt-query-digest) tcpdump innotop *
MySQL CLI - I root@sandbox:~# for i in `seq 1 12` ; do mysql popsdba -e "SHOW ENGINE INNODB STATUS\G" grep "Checkpoint age " ; sleep 1 ; done > checkpoint.txt mysql> show global status like '%threads%'; +------------------------+-------+ Variable_name Value +------------------------+-------+ Delayed_insert_threads Slow_launch_threads Threads_cached Threads_connected 17 Threads_created 33 Threads_running 4 +------------------------+-------+ *
MySQL CLI - II mysql> show global status like '%conn%'; +--------------------------+-------+ Variable_name Value +--------------------------+-------+ Aborted_connects 14 Connections 222 Max_used_connections 17 Ssl_client_connects Ssl_connect_renegotiates Ssl_finished_connects Threads_connected 17 +--------------------------+-------+ 7 rows in set (. sec) *
MySQL CLI - III mysql> pager cut -d '' -f 4cut -d ':' -f 1sortuniq -c PAGER set to 'cut -d '' -f 4cut -d ':' -f 1sortuniq -c' mysql> show processlist; 3 +-----+------+-----------+------+---------+------+-------+-----------------+ 1 Host 1 localhost 1 row in set (. sec) *
MySQL CLI - IV mysql> \s -------------mysql Ver 14.14 Distrib 5.1.73, for debian-linux-gnu (i486) using readline 6.1 Connection id: Current database: Current user: SSL: Current pager: Using outfile: Using delimiter: Server version: Protocol version: Connection: Server characterset: Db characterset: Client characterset: Conn. characterset: UNIX socket: Uptime: 221 root@localhost Not in use cut -d '' -f 4cut -d ':' -f 1sortuniq -c '' ; 5.1.73-ubuntu.1.4.1 (Ubuntu) 1 Localhost via UNIX socket latin1 latin1 latin1 latin1 /var/run/mysqld/mysqld.sock 11 days 3 hours 15 min 26 sec Threads: 17 Questions: 7158713 -------------- Slow queries: Opens: 34 Flush tables: 1 Open tables: 44 Queries per second avg: 7.44 *
MySQL CLI - V root@sandbox:~# while true; do echo "show engine innodb status" mysql popsdba -A -N -r grep -i "history"; sleep.5; done History list length 126 History list length 1315 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 History list length 1477 *
MySQL Tools - #mysqladmin - I root@sandbox:~# mysqladmin popsdba (os_logdata)_writtencom_insert )' Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written Com_insert Innodb_data_written Innodb_os_log_written extended -i 1 -r -c 12 egrep '(Innodb_ 487256 3355659264 632927744 456 17918464 551424 422 1959872 5688 422 1799274 592896 415 17952256 551936 *
MySQL Tools - #mysqladmin - II mysqladmin -r -i 5 extend status >> $myadminfilename.txt +-----------------------------------+-------------+ Variable_name Value +-----------------------------------+-------------+ Aborted_clients Aborted_connects Binlog_cache_disk_use Binlog_cache_use Bytes_received 4 Bytes_sent 7751 Com_admin_commands Com_assign_to_keycache......... *
MySQL Tools - #mysqladmin - III root@sandbox:~# mysqladmin -p Enter password: Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written Innodb_os_log_written -r -i 5 extended-status grep Innodb_os_log_written 781725696 282688 27648 284628 2779136 275324 284928 282176 276192 *
MySQL Tools - #mysqlbinlog - I $mysqlbinlog proddb2-433574-bin-log.42 egrep '^#.*exec_time' egrep -v 'exec_time= (4294967295)' sed -e 's/exec_time=//' sort -r -n -k 1 head -n 2 $ mysqlbinlog /path/to/mysql-bin.999 grep -i -e "^update" -e "^insert" -e "^delete" -e "^replace" -e "^alter" cut -c1-1 tr '[A-Z]' '[a-z]' sed -e "s/\t/ /g;s/\`//g;s/(.*$//;s/ set.*$//;s/ as.*$//" sed -e "s/ where.*$//" sort uniq -c sort -nr 33389 update e_acc 1768 insert into r_b 1768 insert into e_rec 14332 insert into rcv_c 13543 update e_rec 185 update loc 3339 insert into r_att 2781 insert into o_att *
MySQL Tools - #mysqlbinlog - II $ mysqlbinlog al-db2.179 /tmp/writes.txt pt-query-digest --type=binlog --group-by=distill > $ head n 1 /tmp/writes.txt > /tmp/writes_1.txt $ egrep -c 2 156 1376 '^# /tmp/writes_1.txt awk '{print $1}' grep exec_time sort uniq exec_time= exec_time=4 exec_time=5 $ mysqlbinlog pathtobinlog pt-query-digest --type binlog --limit 3 --order-by 'Query_time:cnt' > output.txt *
MySQL slow query log Set slow query log seconds (warning) mysql> SET GLOBAL log_slow_verbosity='standard'; Query OK, rows affected (. sec) mysql> SET GLOBAL slow_query_log_use_global_control='long_query_time'; Query OK, rows affected (. sec) mysql> SET GLOBAL long_query_time=; Query OK, rows affected (. sec) mysql> \! mv /var/log/mysql/mysql-slow.log /var/log/mysql/mysql-slow.log mysql> FLUSH LOGS; Query OK, rows affected (.8 sec) Run detailed pt-query-digest *
External tools #pt-query-digest All queries ordered by time: #pt-query-digest --limit 1% /var/log/mysql/mysql-slow.log > /root/bb/mysql-slowdb1.time.digest All queries ordered by count: #pt-query-digest --limit 1% /var/log/mysql/mysql-slow.log --order-by 'Query_time:cnt' > /root/bb/mysql-slow-db1.cnt.digest All queries ordered by row examined: #pt-query-digest --limit 1% /var/log/mysql/mysql-slow.log --order-by 'Rows_examined:sum' > /root/palominodb/mysql-slow-db1.rows.digest All queries longer than 1 seconds: #pt-query-digest mysql-slow.log.1 --filter '$event->{query_time} > 1' > /tmp/mysqlslow.log.1_1sec.txt *
External tools #pt-query-digest II All queries by time range: #pt-query-digest --limit 1% --since "213-9-1 13::" --until "213-9-1 15: :" > spike1sept.diges All queries by time range compressed: #zcat slow-log.2.gz pt-query-digest --limit 1% --since "213-9-1 13::" -until "213-9-1 15::" > spike1sept.digest All queries by time range ordered by rows examined: #pt-query-digest --limit 1% mysql_slow.log --since "214-1-29 19::" --until "214-1-3 3::" --order-by 'Rows_examined:sum' > 1314.spike.txt *
External tools #tcpdump Use tcpdump to capture all traffic: #tcpdump -s 65535 -x -nn -q -tttt -i any -c 1 port 336 > mysql.tcp.txt Generate slowlog from output of tcpdump: #pt-query-digest --output tcpdump.slow.log --no-report --type tcpdump mysql.tcp.txt *
External tools #innotop Top queries When Now Total Load.. QPS 8.3k. Slow QCacheHit.%.% KCacheHit 1.%.% BpsIn 26.7k. Cmd ID State User Host Execute 46 statistics alkin localhost DISTINCT c from sbtest where id between 49875 and 49975 Execute 47 statistics alkin localhost from sbtest where id=5259 Execute 48 Sorting result alkin localhost from sbtest where id between 49925 and 524 order by Execute 49 Updating alkin localhost sbtest set k=k+1 where id=? Execute 5 statistics alkin localhost from sbtest where id=43618 BpsOut 7.1M.11 DB sysbench Time : Query SELECT sysbench : SELECT c sysbench : SELECT c sysbench : UPDATE sysbench : SELECT c *