Using Cacti To Graph MySQL s Metrics

Using Cacti To Graph MySQL s Metrics Kenny Gryp kenny.gryp@percona.com Principal Consultant @ Percona Collaborate 2011 1

Percona MySQL/LAMP Consulting MySQL Support Percona Server (XtraDB) Percona XtraBackup InnoDB Recovery Toolkit, tcprstat (maatkit, innotop, aspersa, mysql-mmm, mysql-cactitemplates)... http://www.percona.com http://www.mysqlperformanceblog.com 2

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 3

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 4

Why Graph? Troubleshooting -behavior over time -sudden spikes in graphs after application updates Capacity Planning -traffic patterns -predictions Not only for Operations, also for Development and Management 5

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 6

Cacti? open source graphing solution uses RRDTool advanced graphs, scriptable web interface quite complex to add graphs adding servers is a manual process 7

mysql-cacti-templates? collection of templates for cacti graph MySQL metrics Has a lot of non-mysql Metrics too: IO stats, apache, memcache, mongodb, java... 8

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 9

How does Cacti work? Database Server /proc/ diskstats mysqld sshd Cacti df Web Server snmpd fetch Poller (cronjob which fetches data from servers) what stores MySQL (contains cacti configuration) httpd sshd snmpd Apache (serves web interface).rrd Files (contains rrd data) Web Browser 10

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 11

Installation Cacti Mysql-cacti-templates Configuring Servers 12

Installation Cacti Mysql-cacti-templates Configuring Servers 13

Setting up Cacti: Requirements httpd php, php-mysql, php-snmp mysql-server (server and client utilities) net-snmp (snmpget) rrdtool 14

Setting up Cacti: Installation Install rpm/deb/... package or extract yourself Configure webserver: php5, cacti scripts Create cacti mysql user Import cacti database Edit configuration Setup cronjob for poller Finish installation using your webbrowser 15

Setup: Extract Recommended is using apt/yum/... to install cacti if not: cacti# cd /var/www/html/ # may vary cacti# tar xzvf cacti-0.8.7g.tar.gz cacti# ln -s cacti-0.8.7g cacti cacti# chown -R cactiuser rra/ log/ 16

Setup: Webserver Apache (may be something else) Enable php Set DirectoryIndex index.php LoadModule php5_module modules/libphp5.so AddHandler php5-script.php AddType text/html.php DirectoryIndex index.php 17

Setup: MySQL Create Cacti MySQL user: cactimysql> GRANT ALL PRIVILEGES ON cacti.* TO cactiuser@localhost IDENTIFIED BY 'password'; Import Cacti Schema cacti# mysql -e CREATE DATABASE cacti; cacti# mysql cacti < cacti.sql Edit include/config.php $database_type = "mysql"; $database_default = "cacti"; $database_hostname = "localhost"; $database_username = "cactiuser"; $database_password = "password"; 18

Setup: Poller Add cronjob (/etc/cron.d/cacti) */5 * * * * cactiuser php /var/www/html/cacti/ poller.php > /dev/null 2>&1 19

20 Setup: UI

21 Setup: UI

22 Setup: UI

23 Setup: UI

24 Setup: UI

25 Setup: UI

26 Setup: UI

Installation Cacti Mysql-cacti-templates Configuring Servers 27

download & installation cacti# wget http://mysql-cactitemplates.googlecode.com/files/better-cactitemplates-1.1.8.tar.gz cacti# tar -xzf better-cacti-templates-1.1.8.tar.gz cacti# cd better-cacti-templates-1.1.8/ cacti# cp \ scripts/{ss_get_mysql_stats.php,ss_get_by_ssh.php}\ /var/www/html/cacti/scripts/ 28

install: mysql script cacti# vi /var/www/html/cacti/scripts/\ ss_get_mysql_stats.php... $mysql_user = 'cactiuser'; $mysql_pass = 'password';... 29

install: generate ssh key cacti# mkdir /etc/cacti cacti# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/ id_rsa): /etc/cacti/id_rsa Enter passphrase (empty for no passphrase): Your identification has been saved in /etc/cacti/ id_rsa. Your public key has been saved in /etc/cacti/ id_rsa.pub. cacti# chown -R www-data: /etc/cacti/ {id_rsa,id_rsa.pub} cacti# chmod -R 0400 /etc/cacti/id_rsa 30

install: ssh script cacti# vi /var/www/html/cacti/scripts/ ss_get_by_ssh.php... $ssh_user= 'cacti'; # SSH username $ssh_port= 22; # SSH port $ssh_iden='-i /etc/cacti/id_rsa';# SSH identity... 31

32 install: templates

33 install: templates

34 install: templates

Installation Cacti Mysql-cacti-templates Configuring Servers 35

Configure Servers configure net-snmpd add a cacti user in MySQL configure ssh access 36

Configure Servers: snmpd Install snmpd server# apt-get install snmpd Make sure snmpd listens on the network and that a community password is set: server# vi /etc/snmp/snmpd.conf: agentaddress udp:161 rocommunity public server# /etc/init.d/snmpd restart Verify on the cacti server if snmp is working remotely cacti# snmpget -v2c -c public ubuntu-servertest iso.3.6.1.2.1.1.5.0 iso.3.6.1.2.1.1.5.0=string:"ubuntu-server-test" 37

Configure Servers: mysql user Add cacti user servermysql> GRANT SUPER, PROCESS ON *.* TO 'cactiuser'@'cactiserver' IDENTIFIED BY "password"; Verify if login works from cacti servers cacti# mysql -h ubuntu-server-test -e "SELECT VERSION()" -u cactiuser -ppassword +-------------------+ VERSION() +-------------------+ 5.1.49-1ubuntu8.1 +-------------------+ 38

Configure Servers: ssh access Add shell user cacti server# useradd cacti server# getent passwd cacti cacti:x:1001:1002::/home/cacti:/bin/sh Copy the public key server# mkdir /home/cacti/.ssh/ server# cat > /home/cacti/.ssh/authorized_keys << EOF ssh-rsa AAA...r2E7z5 root@ubuntu-server-test EOF server# chmod 0400 /home/cacti/.ssh/authorized_keys Verify cacti# ssh -i /etc/cacti/id_rsa cacti@ubuntu-servertest 39

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 40

41 Adding Graphs

42 Adding Graphs

43 Adding Graphs

Adding Graphs Apply wanted templates on the device - MySQL & GNU/Linux templates Click on Create Graphs: Select graphs to create and Add! Might ask some more questions 44

45 Adding Graphs

Adding Graphs Assign ucd/net SNMP template to have: -cpu usage/load -memory used/available -network statistics -free disk space Different way of creating 46

47 Adding Graphs

Adding Graphs What if I have multiple disk devices to monitor? -Go to Create graphs for this host -Select graph to add in select menu -Create and fill in other device when asked 48

Adding Graphs: Graph Trees Tree to browse through graphs By host/individual graph, mixed 49

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 50

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 51

52 MySQL Handlers

Table scans, backup? MySQL Handlers 53

54 MySQL Handlers

MySQL Handlers Take backup here 55

56 MySQL Connections

MySQL Connections max_connections not reached yet 57 Why more incoming connections?

58 MySQL Replication

MySQL Replication Replication never caught up 59 Replication was stopped

60 MySQL Replication

MySQL Replication In 2 hours, almost 6900 seconds replication lag 61

62 MySQL Temporary Objects

MySQL Temporary Objects 63 keep an eye on: * temp tables * temp disk tables if it grows > query optimization

64 MySQL Select Types

MySQL Select Types Why this sudden peak in rangescans? 65

66 MySQL Select Types

MySQL Select Types Query behavior changed? Other explain plan chosen? 67

68 MySQL Sorts

MySQL Sorts 69 The peaks should be understood, not necessarily be removed as they might be really fast in terms of rows

70 MySQL Command Counters

71 MySQL Binary Logs

expire_logs_days MySQL Binary Logs 72

73 Response Time Distribution +----------------+-------+------------+ time count total +----------------+-------+------------ 0.000001 0 0.000000 0.000010 17 0.000094 0.000100 4301 0.236555 0.001000 1499 0.824450 0.010000 14851 81.680502 0.100000 8066 443.635693 1.000000 0 0.000000 10.000000 0 0.000000 100.000000 1 55.937094 1000.000000 0 0.000000 10000.000000 0 0.000000 100000.000000 0 0.000000

74 MySQL Query Response Time

75 MySQL Query Response Time

76 InnoDB Buffer Pool

InnoDB Buffer Pool 77 DB Restart Time to fill buffer

78 InnoDB Checkpoint Age

sudden increase in unflushed writes InnoDB Checkpoint Age db restart, no write flushing 79

80 InnoDB Buffer Pool Activity

81 InnoDB I/O

More read requests Increase in load InnoDB I/O 82

83 InnoDB Row Operations

InnoDB Row Operations Good view on capacity of the system 84

85

86 more row reads, coming from disk

87 this spike in IO reads did not cause more disk reads

88

89 looking at amount of statements does not always show load

90 CPU Usage

91 Memory

92 Disk Operations

Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 93

non-mysql mysql-cacti-templates Apache JMX Memcached Mongodb Nginx OpenVZ Redis Unix 94

non-mysql mysql-cacti-templates 95

non-mysql mysql-cacti-templates 96

non-mysql mysql-cacti-templates 97

non-mysql mysql-cacti-templates 98

Other Graphing solutions Munin and munin-mysql OpenNMS and MySQL-SNMP Zabbix with appaloosa-zabbix-templates 99

Creating Your Own Graphs Creating cacti graphs not trivial/portable mysql-cacti-templates to the rescue! http://code.google.com/p/mysql-cacti-templates/ wiki/creatinggraphs 100

Using Cacti To Graph MySQL s Metrics Powerful Tool Easy to install Open source! 101

Using Cacti To Graph MySQL s Metrics Cacti: http://www.cacti.net/ RRDtool: http://oss.oetiker.ch/rrdtool/ mysql-cacti-templates: http://code.google.com/p/mysql-cacti-templates/ IRC: Freenode #percona Mailinglist: http://groups.google.com/group/percona-discussion http://www.percona.com http://www.mysqlperformanceblog.com 102

103 Percona Live MySQL Conference NYC May 26, 2011 http://www.percona.com/live/nyc-2011/