Effective MySQL Monitoring. Baron Schwartz March 2012

Transcription

1 Effective MySQL Monitoring Baron Schwartz March 2012

2 Who Am I?

3 Who Am I?

4 Who Am I? Maatkit Percona Toolkit Innotop Monitoring Plugins Aspersa Online Tools JavaScript Libraries

5 Consulting Percona Server Support Percona XtraBackup Training Percona XtraDB Cluster Conferences Percona Toolkit Engineering Many More

6 Agenda What Kind of Monitoring? Historical Data, Graphing Fault Detection and Alerting What to Monitor Tools Overview of Nagios Plugins Demo of Cacti Templates

7 Two Kinds of Monitoring Capturing metrics Detecting faults

8 Capturing Metrics Useful for troubleshooting, diagnosis, and noticing things Very handy for graphs/charts Typical problem: you're missing the one thing you want What to monitor: everything?

9 Fault Detection Monitor system health Alert when there are problems

10 Fault Detection Monitor system health Alert when there are problems Typical problem: spammy alerts Non-actionable False positives /dev/null filters

11 Fault Detection Monitor system health Alert when there are problems Typical problem: spammy alerts Non-actionable False positives /dev/null filters What to monitor: only as much as needed?

12 Why not do both in one tool? Caching Consistency Staleness

13 Monitoring Tools

14 Why So Many?

15 Why So Many? Thus is born Yet Another Tool

16 Popular Tools Nagios MRTG Cacti Munin Zabbix Monit MySQL Enterprise MonYOG New Relic Zenoss Circonus and many more

17 Percona Monitoring Plugins The philosophy: Don't create Yet Another Tool Make existing tools better and easier to use Nagios and Cacti are good enough for most The plugins are 100% free and open source See

18 Monitoring with Nagios

19 Using Nagios Nagios is primarily for fault detection

20 Using Nagios Nagios is primarily for fault detection Works via plugins Executables that follow specific conventions Exit code Text to STDOUT

21 Using Nagios Nagios is primarily for fault detection Works via plugins Executables that follow specific conventions Exit code Text to STDOUT Nagios plugins are pretty universal Portable to many/most similar systems Zabbix, etc

22 Nagios Plugins for MySQL There are too many plugins, too! My suggestion: The official plugins (there are only a couple) Plus Percona's (there is a decent selection)

23 Why Percona's Plugins? Targeted towards avoiding spammy alerts High quality Good documentation Free

24 Why Percona's Plugins? Integrates with other Percona tools Easy to extend and customize Highly structured Designed for testability Written in Bourne shell

25 Why Percona's Plugins?

26 Why Percona's Plugins? This makes the plugins unit-testable for quality.

27 Overview of Percona Plugins Operating Environment: LVM snapshots Deleted files Privileges PID files System memory

28 Overview of Percona Plugins MySQL: Table checksums Replication Processes InnoDB status Deadlocks Arbitrary status counters

29 What to Monitor Everything appropriate from the previous slides Carefully selected status counters Uptime. If too small, server has restarted Threads_connected approaches max_connections

30 What NOT to Monitor In general, be cautious with... Cache hit ratios $variable-per-second Threshold-based alerts

31 What NOT to Monitor Why? A single right threshold is hard to find These are usually not reliable indicators They are also usually not actionable They generate spam alerts!

32 Writing Custom Checks Use the pmp-check-mysql-status plugin You can easily monitor counters, ratios... Example: -x Threads_running -w 20 -c 40 See the documentation for more examples

33 Monitoring with Cacti

34 Cacti Cacti is for metrics collection and graphing Good: easy to install, easy to get started Bad: tedious to template-ize

35 Cacti Cacti is for metrics collection and graphing Good: easy to install, easy to get started Bad: tedious to template-ize Percona Monitoring Plugins fixes this Ready-made templates Template-generation utilities Easy to extend and customize

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66 Take-away Points Record metrics eagerly Alert cautiously Try not to reinvent the wheel Nothing's perfect; usually 2 systems are best One for graphing/trending/metrics One for fault detection

67 Monitoring Resources Blog posts about our plugins: and White papers about preventing downtime: percona.com/about-us/mysql-white-papers

68

69 Public and on-premises courses Upcoming dates: Dallas May 14 London May 21 Raleigh June 18 Chicago July 9 See

70 MySQL Conference & Expo April 10-12, Santa Clara