Using Cacti To Graph MySQL s Metrics

Transcription

1 Using Cacti To Graph MySQL s Metrics Kenny Gryp [email protected] Principal Percona Collaborate

2 Percona MySQL/LAMP Consulting MySQL Support Percona Server (XtraDB) Percona XtraBackup InnoDB Recovery Toolkit, tcprstat (maatkit, innotop, aspersa, mysql-mmm, mysql-cactitemplates)

3 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 3

4 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 4

5 Why Graph? Troubleshooting -behavior over time -sudden spikes in graphs after application updates Capacity Planning -traffic patterns -predictions Not only for Operations, also for Development and Management 5

6 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 6

7 Cacti? open source graphing solution uses RRDTool advanced graphs, scriptable web interface quite complex to add graphs adding servers is a manual process 7

8 mysql-cacti-templates? collection of templates for cacti graph MySQL metrics Has a lot of non-mysql Metrics too: IO stats, apache, memcache, mongodb, java... 8

9 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 9

10 How does Cacti work? Database Server /proc/ diskstats mysqld sshd Cacti df Web Server snmpd fetch Poller (cronjob which fetches data from servers) what stores MySQL (contains cacti configuration) httpd sshd snmpd Apache (serves web interface).rrd Files (contains rrd data) Web Browser 10

11 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 11

12 Installation Cacti Mysql-cacti-templates Configuring Servers 12

13 Installation Cacti Mysql-cacti-templates Configuring Servers 13

14 Setting up Cacti: Requirements httpd php, php-mysql, php-snmp mysql-server (server and client utilities) net-snmp (snmpget) rrdtool 14

15 Setting up Cacti: Installation Install rpm/deb/... package or extract yourself Configure webserver: php5, cacti scripts Create cacti mysql user Import cacti database Edit configuration Setup cronjob for poller Finish installation using your webbrowser 15

16 Setup: Extract Recommended is using apt/yum/... to install cacti if not: cacti# cd /var/www/html/ # may vary cacti# tar xzvf cacti-0.8.7g.tar.gz cacti# ln -s cacti-0.8.7g cacti cacti# chown -R cactiuser rra/ log/ 16

17 Setup: Webserver Apache (may be something else) Enable php Set DirectoryIndex index.php LoadModule php5_module modules/libphp5.so AddHandler php5-script.php AddType text/html.php DirectoryIndex index.php 17

18 Setup: MySQL Create Cacti MySQL user: cactimysql> GRANT ALL PRIVILEGES ON cacti.* TO IDENTIFIED BY 'password'; Import Cacti Schema cacti# mysql -e CREATE DATABASE cacti; cacti# mysql cacti < cacti.sql Edit include/config.php $database_type = "mysql"; $database_default = "cacti"; $database_hostname = "localhost"; $database_username = "cactiuser"; $database_password = "password"; 18

19 Setup: Poller Add cronjob (/etc/cron.d/cacti) */5 * * * * cactiuser php /var/www/html/cacti/ poller.php > /dev/null 2>&1 19

20 20 Setup: UI

21 21 Setup: UI

22 22 Setup: UI

23 23 Setup: UI

24 24 Setup: UI

25 25 Setup: UI

26 26 Setup: UI

27 Installation Cacti Mysql-cacti-templates Configuring Servers 27

28 download & installation cacti# wget cacti# tar -xzf better-cacti-templates tar.gz cacti# cd better-cacti-templates-1.1.8/ cacti# cp \ scripts/{ss_get_mysql_stats.php,ss_get_by_ssh.php}\ /var/www/html/cacti/scripts/ 28

29 install: mysql script cacti# vi /var/www/html/cacti/scripts/\ ss_get_mysql_stats.php... $mysql_user = 'cactiuser'; $mysql_pass = 'password';... 29

30 install: generate ssh key cacti# mkdir /etc/cacti cacti# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/ id_rsa): /etc/cacti/id_rsa Enter passphrase (empty for no passphrase): Your identification has been saved in /etc/cacti/ id_rsa. Your public key has been saved in /etc/cacti/ id_rsa.pub. cacti# chown -R www-data: /etc/cacti/ {id_rsa,id_rsa.pub} cacti# chmod -R 0400 /etc/cacti/id_rsa 30

31 install: ssh script cacti# vi /var/www/html/cacti/scripts/ ss_get_by_ssh.php... $ssh_user= 'cacti'; # SSH username $ssh_port= 22; # SSH port $ssh_iden='-i /etc/cacti/id_rsa';# SSH identity... 31

32 32 install: templates

33 33 install: templates

34 34 install: templates

35 Installation Cacti Mysql-cacti-templates Configuring Servers 35

36 Configure Servers configure net-snmpd add a cacti user in MySQL configure ssh access 36

37 Configure Servers: snmpd Install snmpd server# apt-get install snmpd Make sure snmpd listens on the network and that a community password is set: server# vi /etc/snmp/snmpd.conf: agentaddress udp:161 rocommunity public server# /etc/init.d/snmpd restart Verify on the cacti server if snmp is working remotely cacti# snmpget -v2c -c public ubuntu-servertest iso iso =string:"ubuntu-server-test" 37

38 Configure Servers: mysql user Add cacti user servermysql> GRANT SUPER, PROCESS ON *.* TO IDENTIFIED BY "password"; Verify if login works from cacti servers cacti# mysql -h ubuntu-server-test -e "SELECT VERSION()" -u cactiuser -ppassword VERSION() ubuntu

39 Configure Servers: ssh access Add shell user cacti server# useradd cacti server# getent passwd cacti cacti:x:1001:1002::/home/cacti:/bin/sh Copy the public key server# mkdir /home/cacti/.ssh/ server# cat > /home/cacti/.ssh/authorized_keys << EOF ssh-rsa AAA...r2E7z5 EOF server# chmod 0400 /home/cacti/.ssh/authorized_keys Verify cacti# ssh -i /etc/cacti/id_rsa 39

40 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 40

41 41 Adding Graphs

42 42 Adding Graphs

43 43 Adding Graphs

44 Adding Graphs Apply wanted templates on the device - MySQL & GNU/Linux templates Click on Create Graphs: Select graphs to create and Add! Might ask some more questions 44

45 45 Adding Graphs

46 Adding Graphs Assign ucd/net SNMP template to have: -cpu usage/load -memory used/available -network statistics -free disk space Different way of creating 46

47 47 Adding Graphs

48 Adding Graphs What if I have multiple disk devices to monitor? -Go to Create graphs for this host -Select graph to add in select menu -Create and fill in other device when asked 48

49 Adding Graphs: Graph Trees Tree to browse through graphs By host/individual graph, mixed 49

50 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 50

51 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 51

52 52 MySQL Handlers

53 Table scans, backup? MySQL Handlers 53

54 54 MySQL Handlers

55 MySQL Handlers Take backup here 55

56 56 MySQL Connections

57 MySQL Connections max_connections not reached yet 57 Why more incoming connections?

58 58 MySQL Replication

59 MySQL Replication Replication never caught up 59 Replication was stopped

60 60 MySQL Replication

61 MySQL Replication In 2 hours, almost 6900 seconds replication lag 61

62 62 MySQL Temporary Objects

63 MySQL Temporary Objects 63 keep an eye on: * temp tables * temp disk tables if it grows > query optimization

64 64 MySQL Select Types

65 MySQL Select Types Why this sudden peak in rangescans? 65

66 66 MySQL Select Types

67 MySQL Select Types Query behavior changed? Other explain plan chosen? 67

68 68 MySQL Sorts

69 MySQL Sorts 69 The peaks should be understood, not necessarily be removed as they might be really fast in terms of rows

70 70 MySQL Command Counters

71 71 MySQL Binary Logs

72 expire_logs_days MySQL Binary Logs 72

73 73 Response Time Distribution time count total

74 74 MySQL Query Response Time

75 75 MySQL Query Response Time

76 76 InnoDB Buffer Pool

77 InnoDB Buffer Pool 77 DB Restart Time to fill buffer

78 78 InnoDB Checkpoint Age

79 sudden increase in unflushed writes InnoDB Checkpoint Age db restart, no write flushing 79

80 80 InnoDB Buffer Pool Activity

81 81 InnoDB I/O

82 More read requests Increase in load InnoDB I/O 82

83 83 InnoDB Row Operations

84 InnoDB Row Operations Good view on capacity of the system 84

85 85

86 86 more row reads, coming from disk

87 87 this spike in IO reads did not cause more disk reads

88 88

89 89 looking at amount of statements does not always show load

90 90 CPU Usage

91 91 Memory

92 92 Disk Operations

93 Topics Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extra s 93

94 non-mysql mysql-cacti-templates Apache JMX Memcached Mongodb Nginx OpenVZ Redis Unix 94

95 non-mysql mysql-cacti-templates 95

96 non-mysql mysql-cacti-templates 96

97 non-mysql mysql-cacti-templates 97

98 non-mysql mysql-cacti-templates 98

99 Other Graphing solutions Munin and munin-mysql OpenNMS and MySQL-SNMP Zabbix with appaloosa-zabbix-templates 99

100 Creating Your Own Graphs Creating cacti graphs not trivial/portable mysql-cacti-templates to the rescue! wiki/creatinggraphs 100

101 Using Cacti To Graph MySQL s Metrics Powerful Tool Easy to install Open source! 101

102 Using Cacti To Graph MySQL s Metrics Cacti: RRDtool: mysql-cacti-templates: IRC: Freenode #percona Mailinglist:

103 103 Percona Live MySQL Conference NYC May 26,