Ganglia & Nagios. Maciej Lasyk 11. Sesja Linuksowa Wrocław, /25. Maciej Lasyk, Ganglia & Nagios

Transcription

1 Ganglia & Nagios Maciej Lasyk 11. Sesja Linuksowa Wrocław, /25

2 Ganglia.. what? Ganglia cluster / group of neurons found outside the central nervous system 2/25

3 Just a little about monitoring - the need for monitoring 3/25

4 Just a little about monitoring - the need for monitoring - measuring availability 3/25

5 Just a little about monitoring - the need for monitoring - measuring availability - measuring performance 3/25

6 Just a little about monitoring - the need for monitoring - measuring availability - measuring performance - gathering additional metrics 3/25

7 Monitoring is critical for HA How to measure availability? 4/25

8 Monitoring is critical for HA How to measure availability? A = Uptime / (Uptime + Downtime) 4/25

9 Monitoring is critical for HA How to measure availability? A = Uptime / (Uptime + Downtime) MTTD (Mean Time to Diagnose) The average time it takes to diagnose the problem 4/25

10 Monitoring is critical for HA How to measure availability? A = Uptime / (Uptime + Downtime) MTTD (Mean Time to Diagnose) The average time it takes to diagnose the problem MTTR (Mean Time to Repair) The average time it takes to fix a problem 4/25

11 Monitoring is critical for HA How to measure availability? A = Uptime / (Uptime + Downtime) MTTD (Mean Time to Diagnose) The average time it takes to diagnose the problem MTTR (Mean Time to Repair) The average time it takes to fix a problem MTTF (Mean Time to Failure) The average time there is correct behavior 4/25

12 Monitoring is critical for HA How to measure availability? A = Uptime / (Uptime + Downtime) MTTD (Mean Time to Diagnose) The average time it takes to diagnose the problem MTTR (Mean Time to Repair) The average time it takes to fix a problem MTTF (Mean Time to Failure) The average time there is correct behavior MTBF (Mean Time Between Failures) The average time between different failures of the service 4/25

13 Monitoring is critical for HA 4/25

14 Monitoring is critical for HA A = MTTF / MTBF = MTTF / (MTTF + MTTD + MTTR) 4/25

15 What should we monitor? - hardware housing - devices - storage - network - hosts - software (very deep hole) 5/25

16 What should we monitor? - hardware housing - devices - storage - network - hosts - software (very deep hole) Think dependencies! 5/25

17 When outage hits us don't panic! - Notifications 6/25

18 When outage hits us don't panic! - Notifications - Escalations L1 <-> L2 <-> L3 <-> L4 lol ;) desktop support / devs / ops / networking / / storage / middleware / dc / security 6/25

19 When outage hits us don't panic! - Notifications - Escalations L1 <-> L2 <-> L3 <-> L4 lol ;) desktop support / devs / ops / networking / / storage / middleware / dc / security - Clock is ticking it should be simple 6/25

20 When outage hits us don't panic! - Notifications - Escalations L1 <-> L2 <-> L3 <-> L4 lol ;) desktop support / devs / ops / networking / / storage / middleware / dc / security - Clock is ticking it should be simple - What if cell is offline or someone is out? 6/25

21 Monitoring: notifications issues - false positives 7/25

22 Monitoring: notifications issues - false positives - major events 7/25

23 Monitoring: notifications issues - false positives - major events - failover notifications? 7/25

24 Monitoring: notifications issues - false positives - major events - failover notifications? - tolerance & critical thresholds 7/25

25 Monitoring: reporting - baseline 8/25

26 Monitoring: reporting - baseline - correlation between incidents and change management 8/25

27 Monitoring: reporting - baseline - correlation between incidents and change management - trending info 8/25

28 Monitoring: reporting - baseline - correlation between incidents and change management - trending info - reporting 8/25

29 Monitoring: good practices - don't NIH! 9/25

30 Monitoring: good practices - don't NIH! - DVCS 9/25

31 Monitoring: good practices - don't NIH! - DVCS - testing envs 9/25

32 Monitoring: good practices - don't NIH! - DVCS - testing envs - think usability! 9/25

33 Monitoring: good practices - don't NIH! - DVCS - testing envs - think usability! - passive checks 9/25

34 Monitoring: good practices - don't NIH! - DVCS - testing envs - think usability! - passive checks - automate don't hardcode 9/25

35 Monitoring: good practices - don't NIH! - DVCS - testing envs - think usability! - passive checks - automate don't hardcode - security 9/25

36 Monitoring: good practices Last but not least... Quis custodiet ipsos custodes? (Who will guard the guards?) 9/25

37 Nagios recap Host / Services / Contacts - hosts, hostgroups 10/25

38 Nagios recap Host / Services / Contacts - hosts, hostgroups - services, service groups 10/25

39 Nagios recap Host / Services / Contacts - hosts, hostgroups - services, service groups - templates 10/25

40 Nagios recap Host / Services / Contacts - hosts, hostgroups - services, service groups - templates - time periods 10/25

41 Nagios recap Host / Services / Contacts - hosts, hostgroups - services, service groups - templates - time periods - host and services dependencies 10/25

42 Nagios recap Host / Services / Contacts - hosts, hostgroups - services, service groups - templates - time periods - host and services dependencies - regular expressions 10/25

43 Nagios recap 10/25

44 Nagios recap 10/25

45 Nagios recap Checks and states - frequencies & thresholds 10/25

46 Nagios recap Checks and states - frequencies & thresholds - scheduling downtimes 10/25

47 Nagios recap Checks and states - frequencies & thresholds - scheduling downtimes - outages and flapping 10/25

48 Nagios recap Notifications - periods 10/25

49 Nagios recap Notifications - periods - groups 10/25

50 Nagios recap Notifications - periods - groups - which states to be notified about? 10/25

51 Nagios recap Notifications - periods - groups - which states to be notified about? - escalations / rotations 10/25

52 Nagios recap Notifications - periods - groups - which states to be notified about? - escalations / rotations - custom notifications method 10/25

53 Nagios recap Monitoring remotes - NRPE daemons - checks via SSH 10/25

54 Nagios recap Web interface tactical overview 10/25

55 Nagios recap Web interface availability reports 10/25

56 Nagios recap Web interface trends 10/25

57 Nagios recap Web interface network maps 10/25

58 Networking recap Unicast 11/25

59 Networking recap Multicast 11/25

60 Networking recap Broadcast 11/25

61 Ganglia what is it? Problems of big scale: 20k hosts with zylion metrics probed every 10 seconds It is fully redundant (until you spoil it) It is very scalable Regexp searches and creating of views adhoc :) 12/25

62 Ganglia architecture 13/25

63 Ganglia architecture 13/25

64 Ganglia topologies Default multicast topology 14/25

65 Ganglia topologies Deaf / mute multicast topology 14/25

66 Ganglia topologies Unicast topology 14/25

67 Ganglia topologies Gmetad topology 14/25

68 Ganglia topologies Gmetad HA topology (active - active) 14/25

69 Ganglia topologies Gmetad hierarchical topology 14/25

70 Ganglia RRDcached 15/25

71 Ganglia sflow 16/25

72 Ganglia web (grid view) 17/25

73 Ganglia web (cluster view) 17/25

74 Ganglia web (physical view) 17/25

75 Ganglia web (host view) 17/25

76 Ganglia web (compare hosts) 17/25

77 Ganglia web (events) Events have API json based Think integration with whatever app :) 17/25

78 Ganglia web (dashboards) - Create view -> apply as dashboard - Create dashboard from XML - Generate graphs and add to views 17/25

79 Ganglia web (graphs) 17/25

80 Ganglia metrics - base / extended metrics - own modules - c / c++ - mod_python - spoofing - gmetric - gmetric4j / java - Which to choose? gmetric / python / c/c++? 18/25

81 Ganglia metrics - base / extended metrics 18/25

82 Ganglia metrics - base / extended metrics - own modules 18/25

83 Ganglia metrics - base / extended metrics - own modules - c / c++ 18/25

84 Ganglia metrics - base / extended metrics - own modules - c / c++ - mod_python 18/25

85 Ganglia metrics - base / extended metrics - own modules - c / c++ - mod_python - spoofing 18/25

86 Ganglia metrics - base / extended metrics - own modules - c / c++ - mod_python - spoofing - gmetric - gmetric4j / java 18/25

87 Ganglia metrics - base / extended metrics - own modules - c / c++ - mod_python - spoofing - gmetric - gmetric4j / java - Which to choose? gmetric / python / c/c++? 18/25

88 Ganglia and logfiles? ganglia-logtailer parser logfiles (realtime) - pushes data to ganglia (via gmetric) - yup based on specific log formats - yet still open source so poke around ;) 19/25

89 So... Nagios + Ganglia! 3 ways of integration: - ganglia-web/nagios (PHP & bash based) - ganglia-nagios-bridge (Python & cron based) - check-ganglia-metric (Python) 20/25

90 Nagios + Ganglia: ganglia-web/nagios Sending Nagios Data to Ganglia service_perfdata_command Or replace Nagios checks with Ganglia! - Check heartbeat. - Check a single metric on a specific host. - Check multiple metrics on a specific host. - Check multiple metrics across a regex-defined range of hosts 21/25

91 Nagios + Ganglia: ganglia-web/nagios Nagios pulls info from Ganglia via HTTP 21/25

92 Nagios + Ganglia: ganglia-nagios-bridge Python script run in e.g. in crontab - pulls data from Ganglia XML via sockets - parses XML - send data to Nagios - Nagios commits only passive checks 22/25

93 Nagios + Ganglia: check_ganglia_metric basically Nagios plugin - pulls data from Ganglia XML via sockets - check_ganglia_metric.py \ --gmetad_host=gmetad-server.example.com \ --metric_host=host.example.com --metric_name=cpu_idle 23/25

94 Nagios + Ganglia Which one integration should I use? 24/25

95 Nagios + Ganglia Which one integration should I use? Seriously try yourself and test 24/25

96 Freenode #ganglia /25

97 Thank you :) Ganglia & Nagios Maciej Lasyk 11. Sesja Linuksowa , Wrocław 25/25