Monitoring MySQL. Geert Vanderkelen MySQL Senior Support Engineer Sun Microsystems

Size: px

Start display at page:

Download "Monitoring MySQL. Geert Vanderkelen MySQL Senior Support Engineer Sun Microsystems"

Douglas Knight
10 years ago
Views:

1 Monitoring MySQL Geert Vanderkelen MySQL Senior Support Engineer Sun Microsystems

2 Agenda Short intro into MySQL, the company Monitoring MySQL: some examples Nagios plugins for MySQL MySQL Enterprise Monitor

3 From MySQL to Sun Whoohoo!

4 What Makes MySQL Special OpenSource! Commercial and Community Support Storage engines Simplicity Sun Microsystems MySQL 5.1 is coming! MySQL Cluster 6.2 is here!

5 Monitoring MySQL Basics and Some examples..

6 Getting Status Information SHOW GLOBAL STATUS > Global keyword is important! Snapshot regularly > mysqladmin -i 20 extended-status >> status.log Also available as information_schema.global_status For example: > Com_xxx counter variables > Questions for Queries/Sec

7 Index usage Find out if you do lots of table scans mysql> SHOW GLOBAL STATUS LIKE Handler\_read% ; Handler_read_first Handler_read_key Handler_read_next Handler_read_prev Handler_read_rnd Handler_read_rnd_next ((Handler_read_rnd + Handler_read_rnd_next) / ( (Handler_read_%))*100 = 87.8

Handler_read_next 31376780526 Handler_read_prev 34482622 Handler_read_rnd 265239832

8 Temp. Tables Going To Disk? mysql> SHOW GLOBAL STATUS LIKE Created\_tmp% ; Created_tmp_disk_tables Created_tmp_files 71 Created_tmp_tables (Created_tmp_disk_tables / Created_tmp_tables) *100 = 38.3

Created_tmp_disk_tables 5032520 Created_tmp_files 71

9 Check # Of Processes To much long running process: mostly trouble SHOW PROCESSLIST or: mysql> USE information_schema; mysql> SELECT COUNT(*) FROM processlist WHERE user <> 'replication' AND id <> CONNECTION_ID() AND time > 60 AND command <> 'Sleep';

mysql> SELECT COUNT(*) FROM processlist WHERE user <>

10 Check uptime MySQL might crash and restart quickly Other checks might miss it mysql> USE information_schema; mysql> SELECT DATE_SUB(NOW(), INTERVAL variable_value SECOND) AS started, variable_value AS uptime FROM global_status WHERE variable_name = 'UPTIME'; STARTED UPTIME :58:

variable_value AS uptime FROM global_status WHERE variable_name = 'UPTIME';

11 InnoDB Table Size Usually auto-extends innodb_data_file_path = ibdata1:200m Before MySQL 5.1 in table comments! mysql> USE information_schema; mysql> SELECT data_free*1024 AS 'InnoDB free space' FROM tables WHERE table_name = 'innotest'; InnoDB free space

mysql> USE information_schema; mysql> SELECT data_free*1024 AS 'InnoDB free space'

12 InnoDB Buffer Pool Hit Rate Is the buffer still big enough? mysql> SHOW GLOBAL STATUS LIKE 'InnoDB\_buffer\_pool\_read%'; Variable_name Value Innodb_buffer_pool_read_requests Innodb_buffer_pool_reads * ( 1 - (Innodb_buffer_pool_reads / Innodb_buffer_pool_read_requests)) = 99.99%

13 Checking configuration SHOW GLOBAL VARIABLES > Also information_schema.global_variables What version is running? Default storage engine still InnoDB? Global character set is still UTF-8? Logs enabled? > slow and general query log (online as of 5.1)

14 Logging Enabled? General log > Log every statement, useful for debugging > SHOW GLOBAL VARIABLES LIKE 'general_log'; Slow query log > Show long running queries, or not using indexes > SHOW GLOBAL VARIABLES LIKE 'slow_query_log'; As of MySQL 5.1 > Can be toggled online, and logged to tables Should not be running all the time

LIKE 'general_log'; Slow query log > Show long running queries, or not using

15 Monitoring Replication On Master > Check number of binary logs > Limit number of slaves On Slave > Use SHOW SLAVE STATUS > Check seconds behind master > Check if still reading from Master > etc.

16 Master And Her Slaves Master can handle lots of slave > But it s good to set a limit SHOW SLAVE HOSTS > Does list registered hosts, not connected ones mysql> SHOW SLAVE HOSTS; Server_id Host Port... Master_id cent row in set (0.00 sec)

+-----------+--------+------+...+-----------+ Server_id Host Port.

17 Number Of Binary Logs Regularly causing disk space issues > Can use PURGE MASTER LOGS SHOW BINARY LOGS mysql> SHOW BINARY LOGS; Log_name File_size cent01bin cent01bin cent01bin cent01bin

Log_name File_size +------------------+-----------+ cent01bin.000001 125 cent01bin.

18 Slave status mysql> SHOW SLAVE STATUS\G.. Master_Host: Master_User: replication Master_Port: Slave_IO_Running: Yes Slave_SQL_Running: Yes.. Seconds_Behind_Master: 0.. Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:

19 Seconds Behind Master Number of seconds elapsed since timestamp of last statement received from Master Only useful in fast networks > Slow I/O gives bad approximation Use a heartbeat table > INSERT INTO replication_ping VALUES (NOW()) > Check on slave when table is updated > Calculate time difference > Or use mk-heartbeat from Maatkit

approximation Use a heartbeat table > INSERT INTO replication_ping VALUES (NOW()) >

20 Monitoring Cluster Still not easy to do Using NDB API Parse logs Using ndb_mgm shell> ndb_mgm -e "SHOW" Connected to Management Server at: localhost:1406 Cluster Configuration [ndbd(ndb)] 2 node(s) (mysql ndb , Nodegroup: 0, Master) (mysql ndb , Nodegroup: 0) [ndb_mgmd(mgm)] 2 node(s) (mysql ndb ) id=2 (not connected, accepting connect from ndbsup-priv-2) [mysqld(api)] 7 node(s) (mysql ndb ) id=6 (not connected, accepting connect from any host)

100.9.9 (mysql-5.1.27 ndb-6.3.17, Nodegroup: 0) [ndb_mgmd(mgm)] 2 node(s) id=1 @10.100.9.6 (mysql-5.1.27 ndb-6.3.17) id=2 (not connected, accepting connect from ndbsup-priv-2) [mysqld(api)] 7 node(s) id=5 @10.

21 Useful Tools MySQL client tools and SQL! MySQL Enterprise Monitor > innotop > Maatkit >

22 Nagios + MySQL Monitoring MySQL with Nagios

23 Configuration tips Using hostgroups > Useful for replication setups or Cluster define hostgroup { hostgroup_name mysql-masters } define hostgroup { hostgroup_name mysql-slaves } define host { use generic-virtualbox host_name cent02 alias cent02 (VMWare) address hostgroups virtualboxes,vmware,mysql-slaves } define service { use generic-service hostgroup_name mysql-slaves service_description Slave SQL Thread check_command mysql_repl_sqlthread }

24 Configuration tips (2) Secure your setup: SSL or SSH Don t use MySQL root user for checks Could use NRPE (Nagios add-on)

25 Some plugins.. Oli Sennhauser >

26 Some plugins.. (2) check_mysql_perf > Checks various buffers > hit rates for qcache, keycache InnoDB checks > bufferpool, etc.. Checks for slow queries and more..

27 Write your own! Quite easy: here a very simple plugin #!/bin/sh mysql -e "SELECT NOW()" -NB -h $1 -u$2 -p$3 2>/dev/null if [ $?!= 0 ]; then echo "Oops!" ; exit 2 # Criticial fi exit 0 # OK shell>./simple.sh cent02.kemuri.net agent cop :37:09 shell>./simple.sh cent02.kemuri.net agent cap Oops!

28 Write Your Own with Python for the full version def get_uptime(): global OPTIONS db = MySQLdb.connection(host=OPTIONS['hostname'], user=options['user'],passwd=options['password']) db.query("show GLOBAL STATUS LIKE 'Uptime'") res = db.store_result() row = res.fetch_row(1) return int(row[0][1]) def check_uptime(uptime): global OPTIONS, ERRORS line = 'Uptime %ds' % uptime err = ERRORS['OK'] if uptime < OPTIONS['critical']: err = ERRORS['CRITICAL'] elif uptime < OPTIONS['warning']: err = ERRORS['WARNING'] print line sys.exit(err)

29 MySQL Enterprise Monitor What is it, and how combining it with Nagios?

30 MySQL Enterprise Monitor (MEM) Single, consolidated view Auto discovery of MySQL Servers, Replication Topologies Problem Query Detection, Analysis and Tuning New! Customizable rules-based monitoring and alerts Identifies problems before they occur Reduces risk of downtime Easier to scale-out without requiring more DBAs

32 Advisors/Rules Comparable with Nagios plugins Possible using Nagios plugins making custom Advisors for MEM Can use SNMP Traps

33 MEM, Nagios and SNMP Using Passive Service Checks Handle SNMP traps > Using traphandle in snmptrapd configuration Nagios and MEM host names must match MEM View Nagios View

34 MEM, Nagios and SNMP (cont.) Configure snmptrapd to handle MEM traps # Usually /etc/snmp/snmptrapd.conf traphandle default /path/to/handle_mysql_memtrap.sh Setup Nagios define service { use generic-service service_description MEM SNMP Trap hostgroup_name mysql-masters,mysql-slaves register 0 check_command check_no is_volatile 1 max_check_attempts 1 retry_check_interval 1 normal_check_interval 4 active_checks_enabled 0 passive_checks_enabled 1 }

35 SNMP Handler Script (part 1) Reading SNMP trap information read host read ip while read oid val do val=`expr "$val" : '"$.*$"'` vars="$vars;; $oid: $val" case "$oid" in MYSQLTRAP-MIB::advisor.0) hostname=$val ;; MYSQLTRAP-MIB::advisor.1) memstate=$val ;; MYSQLTRAP-MIB::advisor.4) message=$val ;; esac done

36 SNMP Handler Script (part 2) Translate and send it to Nagios. state=$state_ok case "$memstate" in critical) state=$state_critical ;; warning info success) state=$state_warning ;; esac datetime=`date +%s` cmd="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$hostname; $service;$state;$message" echo $cmd >> $nagioscmdfile

37 Some Links.. MySQL Enterprise > Nagios documentation > > Geert for the handler script

38 Monitoring MySQL Geert Vanderkelen

Maintaining Non-Stop Services with Multi Layer Monitoring

Maintaining Non-Stop Services with Multi Layer Monitoring Lahav Savir System Architect and CEO of Emind Systems [email protected] www.emindsys.com The approach Non-stop applications can t leave on their