TF-NOC Dublin. Alexandros Kosiaris GRNET NOC Use puppet and network inventory to populate nagios/icinga configuration

Transcription

1 r GRNET NOC Use puppet and network inventory to populate nagios/icinga configuration TF-NOC Dublin Alexandros Kosiaris (alex@noc.grnet.gr)

2 Network & Equipment Optical Network: ~70 cities (+30 within next year) 15years-leased dark fiber DWDM/CWDM network Optical Equipment: Alcatel 1626LM, 1696MS, 1678MCC Adva FSP2000 Routing Equipment: Juniper T1600, Juniper MX960 ~10x Cisco 12000s, a few Cisco 7200s/7300s Switching Equipment: Cisco 6500 Several Cisco 3750, Cisco 2970, Juniper ex4200, Extreme X450a/X350 Storage Equipment: Netapp/IBM N5300 EMC Celerra NS-480 Computing Equipment: Virtualization (KVM) 12 Blade servers, HP BL-460c 12 IBM 1U Servers 128 1U Fujitsu Servers 275 2U HP Proliant Servers ~200 Vms

3 Nagios + Network Equipment or (more accurately) Switching and Routing In-house developed Network Inventory (a.k.a. GRNETDB) A MySQL database of almost 150 tables Populated multiple times a day by a PHP discovery script SNMP, telnet + expect Basic Concepts: Node Interface Layer Domain Location These concepts get extended to represent functionality Routing, Switching nodes Layer2, Layer3 interfaces Switching, administrative domains

4 Nagios + Network Equipment or (more accurately) Switching and Routing In-house developed python Django project, with multiple sub-apps Network (the interface to the database) RG (router graphs, take a peek at Maps (take a look at Hostmaster Optical network (built mostly on Location info) Nadjicingo Builts on network app and generates a nagios/icinga configuration Nagvis Same thing but generates/updates nagvis config

5 Nadjicingo A Django management command outputing nagios/icinga configuration Run by crontab every hour (manage.py nadjicingo) Will generate nagios configuration objects for Routers Switches Interfaces L3 Topology aware (nagios hates cyclic dependencies aka redundant links), populates parents field for most devices. Hardware checks in devices Business logic embedded in interface descriptions: Part of it is a unique identifier for a customers link [.NTUA-4] => National Technical University's L3 link [AUTH@ERMOU-1] => Aristotle University of Thessaloniki L2 link at Ermou PoP

6 Nagvis A Django management command (again...) Run by crontab every hour (manage.py nagvis) Will update a specific nagvis map configuration by: Removing obsolete nodes Adding new nodes to a special area for manual positioning on map Also features an automated positioning mode based on devices Latitude Longitude. Nice for showoff but not for overview in monitoring applications Will only populate host objects in map. Service objects cluttered it too much and information is rightly available anyway

7 Nagvis Network Map

8 Servers, Services? A little bit of history For years, GRNET only had very basic services (DNS, , Web) And some router supporting services (Looking glass, mrtg, rancid) And very few servers (<=10) 3 years ago, major paradigm shift from networking to services 20 Servers bought, and then 132 and recently 275 more End user services were born: Public cloud storage service (Pithos) Virtual Private Servers (ViMa) Students books statements (Eudoxus) Student Id cards (Paso) Public IaaS (Okeanos) Academic Professor Elections (Apella) Plus many other services and projects (TCS, Whois, NTP, VoD, ) The result? => 200 Vms were created for managing all this infrastructure

9 Puppet to the rescue What is Puppet? It's a stack of applications It's a language (a declarative one as well) It's a policy and state enforcing tool It's a attribute and state discovery tool (kind of...) It's a new paradigm in managing systems! What is Puppet not? Not just an automation tool Not a For loop Not a command execution framework (it can be reduced to that though) AGAIN: A new paradigm, you need to change the way you work

10 Puppet Concepts Facts Attributes of a system: OS Version and family Available memory CPUs Block devices IP addresses/netmasks MAC addresses And anything else you can write code for it to be discovered LLDP neighbours IPMI functionality Hardware info Apache vhosts Discovered by facter and then made available to Puppet

11 Puppet Concepts(2) Resources Files, Directories Users, Groups Packages Vlans Interfaces Nagios objects!!!! And a lot more ( Classes A way to group resources Support inheritance and mixins (aka including) The standard class has 3 resources defined Package {'software': File { '/etc/software.conf': Service { 'softwared':

12 Puppet Concepts(3) Nodes A.k.a. machines (VM or hardware) A node CAN (and probably will) have multiple puppet classes Node population can be done in multiple ways: Puppet language config LDAP External script Puppetd agents running in each machine (daemon or crontab) Central Puppetmaster (with an RDBMS) holds all the configuration and data

13 Hello World example class helloworld { file { '/tmp/helloworld': ensure => present, owner => root, group => root, mode => 640, content => 'Hello world' node mynode { include helloworld Will create the /tmp/helloworld with all the attributes as defined above More importantly, if run again it will make sure to wipe any possible changes and restore the state as is defined above

14 Back to nagios Let s use a puppet native type nagios_host { $hostname : address => , alias => myhost, contact_groups => hostadmins, hostgroups => 'Puppeted Servers', /etc/nagios/nagios_host.cfg gets populated Problem is... This is executed in the machine running puppetd not the nagios server. No problem. Puppet supports exported resources.

15 Exported resources Let s prepend the definition with { 'myservice' contact_groups => hostadmins, host_name => $hostname, tag => 'collect_me_nagios_server', Exports the resource but does not realize it on the machine running puppetd No /etc/nagios/nagios_service.cfg file created << Nagios_service tag == 'collect_me_nagiosserver' >> In nagios server s manifest. /etc/nagios/nagios_service.cfg populated. nagios,icinga.cfg can now just include the file/directory and monitoring begins

16 Simple example A manifest for all authoritative DNS servers Install bind9, install configuration and ensure it is running Open up firewall Setup a simple DNS check class authoritativedns { include bind9 include { "authdns": command => "check_dig! servicegroups => "DNS,DNS:Authoritative"

17 Interesting use cases Class hierarchy means: A base class nagios::host that is included in all other So all servers nagios-monitored without any intervention But: A Server is physical and has IPMI capabilities: So export another nagios host for it if $ipmi_capable { "$ipmi_dns": address => $ipmi_ipaddress, tag => "hardwarehost",

18 Interesting use cases (2) Server is an HP Proliant Server class hp-health { package { [ 'hp-health', 'hpacucli' ]: ensure => present, nagios::host::service { 'hpacucli': ensure => present, servicegroups => 'HARDWARE', command => 'check_nrpe!dsa-check-hpacucli!0', nagios::host::service { 'hpasm': ensure => present, servicegroups => 'HARDWARE', command => 'check_nrpe!dsa-check-hpasm!0',

19 Interesting use cases (3) Multicast beacons (double exported resources!!!) define ssmping_check($ipv4, $ipv6) { $local = $::fqdn $remote = $name if ($::ipaddress and $ipv4 and $local!= $remote) { "ping-ssm-$remote-$local-v4": ensure => present, check_command => "check_nrpe!check_ssmping!$ipv4", host_name => $local, service_description => "Multicast from $remote SSM IPv4", # export the { $fqdn: ipv4 => $ipaddress, ipv6 => $ipv6address

20 Interesting use cases (4) Standard checks for all servers nagios::host::service { "disk": command => "check_nrpe!check_disk!13% 7%", nagios::host::service { "load": command => "check_nrpe!check_load!4,3,2 5,4,3", nagios::host::service { "users": command => "check_nrpe!check_load!20 30", nagios::host::service { "swap": command => "check_nrpe!check_swap!60 40", nagios::host::service { "check_tainted": command => "check_nrpe!check_tainted!0", nagios::host::service { "check_firewall": command => "check_nrpe!check_firewall!0",

21 Problems arise /etc/nagios/*.cfg files can become quickly large However each resource collection reads the entire file Problem solved by disabling collections and creating the entire config file every time, however a more elegant solution would be nice Exported resources cost Each is an entry in the database and they are not used for nagios alone. Execution speed suffers and sometimes times out Problem solved in database by adding some indexes... but is bound to show up again Puppet devs know it, some effort goes there

22 Problems arise (2) Puppet's declarative language can cause problems at times { 'myhost': Hostgroups => $myhostgroups And host also has classes A,B,C apart from nagios class. Which class is going to declare $myhostgroups? Multiple solutions exist, all of them not elegant. Externally (via LDAP) Fact based Populated hostgroups, not hosts

23 Problems arise (3) Active checks cost. Not a Puppet issue but a nagios one check_mk Distributed monitoring Well obsess_over_services sucks mod_gearman For now splitting the infrastructure in Networking Services But if Services grow more? Variable tagging on { 'myservice' contact_groups => hostadmins, host_name => $hostname, tag => 'collect_me_nagios_server_n',

24 r Questions? Alexandros Kosiaris GRNET NOC Systerms Admin alex@noc.grnet.gr