Monitoring MySQL Geert Vanderkelen MySQL Senior Support Engineer Sun Microsystems
Agenda Short intro into MySQL, the company Monitoring MySQL: some examples Nagios plugins for MySQL MySQL Enterprise Monitor
From MySQL to Sun Whoohoo!
What Makes MySQL Special OpenSource! Commercial and Community Support Storage engines Simplicity Sun Microsystems MySQL 5.1 is coming! MySQL Cluster 6.2 is here!
Monitoring MySQL Basics and Some examples..
Getting Status Information SHOW GLOBAL STATUS > Global keyword is important! Snapshot regularly > mysqladmin -i 20 extended-status >> status.log Also available as information_schema.global_status For example: > Com_xxx counter variables > Questions for Queries/Sec
Index usage Find out if you do lots of table scans mysql> SHOW GLOBAL STATUS LIKE Handler\_read% ; Handler_read_first 83574685 Handler_read_key 11274900036 Handler_read_next 31376780526 Handler_read_prev 34482622 Handler_read_rnd 265239832 Handler_read_rnd_next 307182727076 ((Handler_read_rnd + Handler_read_rnd_next) / ( (Handler_read_%))*100 = 87.8
Temp. Tables Going To Disk? mysql> SHOW GLOBAL STATUS LIKE Created\_tmp% ; Created_tmp_disk_tables 5032520 Created_tmp_files 71 Created_tmp_tables 13117923 (Created_tmp_disk_tables / Created_tmp_tables) *100 = 38.3
Check # Of Processes To much long running process: mostly trouble SHOW PROCESSLIST or: mysql> USE information_schema; mysql> SELECT COUNT(*) FROM processlist WHERE user <> 'replication' AND id <> CONNECTION_ID() AND time > 60 AND command <> 'Sleep';
Check uptime MySQL might crash and restart quickly Other checks might miss it mysql> USE information_schema; mysql> SELECT DATE_SUB(NOW(), INTERVAL variable_value SECOND) AS started, variable_value AS uptime FROM global_status WHERE variable_name = 'UPTIME'; +---------------------+--------+ STARTED UPTIME +---------------------+--------+ 2008-09-08 10:58:24 36604 +---------------------+--------+
InnoDB Table Size Usually auto-extends innodb_data_file_path = ibdata1:200m Before MySQL 5.1 in table comments! mysql> USE information_schema; mysql> SELECT data_free*1024 AS 'InnoDB free space' FROM tables WHERE table_name = 'innotest'; +-------------------+ InnoDB free space +-------------------+ 201326592 +-------------------+
InnoDB Buffer Pool Hit Rate Is the buffer still big enough? mysql> SHOW GLOBAL STATUS LIKE 'InnoDB\_buffer\_pool\_read%'; +-----------------------------------+------------+ Variable_name Value +-----------------------------------+------------+.. Innodb_buffer_pool_read_requests 2226519108 Innodb_buffer_pool_reads 74845 +-----------------------------------+------------+ 100 * ( 1 - (Innodb_buffer_pool_reads / Innodb_buffer_pool_read_requests)) = 99.99%
Checking configuration SHOW GLOBAL VARIABLES > Also information_schema.global_variables What version is running? Default storage engine still InnoDB? Global character set is still UTF-8? Logs enabled? > slow and general query log (online as of 5.1)
Logging Enabled? General log > Log every statement, useful for debugging > SHOW GLOBAL VARIABLES LIKE 'general_log'; Slow query log > Show long running queries, or not using indexes > SHOW GLOBAL VARIABLES LIKE 'slow_query_log'; As of MySQL 5.1 > Can be toggled online, and logged to tables Should not be running all the time
Monitoring Replication On Master > Check number of binary logs > Limit number of slaves On Slave > Use SHOW SLAVE STATUS > Check seconds behind master > Check if still reading from Master > etc.
Master And Her Slaves Master can handle lots of slave > But it s good to set a limit SHOW SLAVE HOSTS > Does list registered hosts, not connected ones mysql> SHOW SLAVE HOSTS; +-----------+--------+------+...+-----------+ Server_id Host Port... Master_id +-----------+--------+------+...+-----------+ 2 cent02 3306... 1 +-----------+--------+------+...+-----------+ 1 row in set (0.00 sec)
Number Of Binary Logs Regularly causing disk space issues > Can use PURGE MASTER LOGS SHOW BINARY LOGS mysql> SHOW BINARY LOGS; +------------------+-----------+ Log_name File_size +------------------+-----------+ cent01bin.000001 125 cent01bin.000002 1021 cent01bin.000004 125 cent01bin.000005 462 +------------------+-----------+
Slave status mysql> SHOW SLAVE STATUS\G.. Master_Host: 192.168.14.100 Master_User: replication Master_Port: 3306.. Slave_IO_Running: Yes Slave_SQL_Running: Yes.. Seconds_Behind_Master: 0.. Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:
Seconds Behind Master Number of seconds elapsed since timestamp of last statement received from Master Only useful in fast networks > Slow I/O gives bad approximation Use a heartbeat table > INSERT INTO replication_ping VALUES (NOW()) > Check on slave when table is updated > Calculate time difference > Or use mk-heartbeat from Maatkit
Monitoring Cluster Still not easy to do Using NDB API Parse logs Using ndb_mgm shell> ndb_mgm -e "SHOW" Connected to Management Server at: localhost:1406 Cluster Configuration --------------------- [ndbd(ndb)] 2 node(s) id=3 @10.100.9.8 (mysql-5.1.27 ndb-6.3.17, Nodegroup: 0, Master) id=4 @10.100.9.9 (mysql-5.1.27 ndb-6.3.17, Nodegroup: 0) [ndb_mgmd(mgm)] 2 node(s) id=1 @10.100.9.6 (mysql-5.1.27 ndb-6.3.17) id=2 (not connected, accepting connect from ndbsup-priv-2) [mysqld(api)] 7 node(s) id=5 @10.100.9.6 (mysql-5.1.27 ndb-6.3.17) id=6 (not connected, accepting connect from any host)
Useful Tools MySQL client tools and SQL! MySQL Enterprise Monitor > http://www.mysql.com/products/enterprise/ innotop > http://innotop.sourceforge.net/ Maatkit > http://www.maatkit.org/
Nagios + MySQL Monitoring MySQL with Nagios
Configuration tips Using hostgroups > Useful for replication setups or Cluster define hostgroup { hostgroup_name mysql-masters } define hostgroup { hostgroup_name mysql-slaves } define host { use generic-virtualbox host_name cent02 alias cent02 (VMWare) address 192.168.14.101 hostgroups virtualboxes,vmware,mysql-slaves } define service { use generic-service hostgroup_name mysql-slaves service_description Slave SQL Thread check_command mysql_repl_sqlthread }
Configuration tips (2) Secure your setup: SSL or SSH Don t use MySQL root user for checks Could use NRPE (Nagios add-on)
Some plugins.. Oli Sennhauser > http://www.shinguz.ch/mysql/mysql_monitoring.html
Some plugins.. (2) check_mysql_perf > http://www.consol.com/opensource/nagios/check-mysql-perf/ Checks various buffers > hit rates for qcache, keycache InnoDB checks > bufferpool, etc.. Checks for slow queries and more..
Write your own! http://nagiosplug.sourceforge.net/developer-guidelines.html Quite easy: here a very simple plugin #!/bin/sh mysql -e "SELECT NOW()" -NB -h $1 -u$2 -p$3 2>/dev/null if [ $?!= 0 ]; then echo "Oops!" ; exit 2 # Criticial fi exit 0 # OK shell>./simple.sh cent02.kemuri.net agent cop 2008-09-08 13:37:09 shell>./simple.sh cent02.kemuri.net agent cap Oops!
Write Your Own with Python Email geert@mysql.com for the full version def get_uptime(): global OPTIONS db = MySQLdb.connection(host=OPTIONS['hostname'], user=options['user'],passwd=options['password']) db.query("show GLOBAL STATUS LIKE 'Uptime'") res = db.store_result() row = res.fetch_row(1) return int(row[0][1]) def check_uptime(uptime): global OPTIONS, ERRORS line = 'Uptime %ds' % uptime err = ERRORS['OK'] if uptime < OPTIONS['critical']: err = ERRORS['CRITICAL'] elif uptime < OPTIONS['warning']: err = ERRORS['WARNING'] print line sys.exit(err)
MySQL Enterprise Monitor What is it, and how combining it with Nagios?
MySQL Enterprise Monitor (MEM) Single, consolidated view Auto discovery of MySQL Servers, Replication Topologies Problem Query Detection, Analysis and Tuning New! Customizable rules-based monitoring and alerts Identifies problems before they occur Reduces risk of downtime Easier to scale-out without requiring more DBAs
Advisors/Rules Comparable with Nagios plugins Possible using Nagios plugins making custom Advisors for MEM Can use SNMP Traps
MEM, Nagios and SNMP Using Passive Service Checks Handle SNMP traps > Using traphandle in snmptrapd configuration Nagios and MEM host names must match MEM View Nagios View
MEM, Nagios and SNMP (cont.) Configure snmptrapd to handle MEM traps # Usually /etc/snmp/snmptrapd.conf traphandle default /path/to/handle_mysql_memtrap.sh Setup Nagios define service { use generic-service service_description MEM SNMP Trap hostgroup_name mysql-masters,mysql-slaves register 0 check_command check_no is_volatile 1 max_check_attempts 1 retry_check_interval 1 normal_check_interval 4 active_checks_enabled 0 passive_checks_enabled 1 }
SNMP Handler Script (part 1) Reading SNMP trap information read host read ip while read oid val do val=`expr "$val" : '"\(.*\)"'` vars="$vars;; $oid: $val" case "$oid" in MYSQLTRAP-MIB::advisor.0) hostname=$val ;; MYSQLTRAP-MIB::advisor.1) memstate=$val ;; MYSQLTRAP-MIB::advisor.4) message=$val ;; esac done
SNMP Handler Script (part 2) Translate and send it to Nagios. state=$state_ok case "$memstate" in critical) state=$state_critical ;; warning info success) state=$state_warning ;; esac datetime=`date +%s` cmd="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$hostname; $service;$state;$message" echo $cmd >> $nagioscmdfile
Some Links.. MySQL Enterprise > http://www.mysql.com/products/enterprise/ Nagios documentation > http://nagios.sourceforge.net/docs/2_0/passivechecks.html > http://www.nagios.org/faqs/viewfaq.php?faq_id=29 Email Geert for the handler script
Monitoring MySQL Geert Vanderkelen geert.vanderkelen@sun.com