The road to lazy monitoring with Icinga2 & Puppet Tom De Vylder @tomdevylder
whois Ops guy Rusty, old, Bavarian vehicles enthousiast The one who always gets the patdown at checkpoints Open Source Consultant & Enthousiast at Inuits Team Packages & Tools at Icinga
Monitoring!
Monitoring! Image credits: http://mta.hu/data/cikk/13/00/73/cikk_130073/20120611-144546.jpg
Monitoring :((
Monitoring is... Boring Tedious Repetitive Production = deployment + monitoring + backup Often neglected
Monitoring is... Boring Tedious Repetitive Production = deployment + monitoring + backup Often neglected
Why can't someone else do it?
Someone else! Tedious Repetitive People make mistakes Nobody likes Nagios configs
Someone else! Tedious Repetitive People make mistakes Nobody likes Nagios configs Monitor servers not people Image credits: https://openclipart.org/image/2400px/svg_to_png/219371/you-are-being-monitored.png
Single source of truth
Config management!
Puppet Rudder Juju Chef Config management! Ansible Saltstack TrueConfig CFEngine
Puppet
Puppet Config management tool Exported resources PuppetDB Nagios_* {} resource types
node web.example.org { @@nagios_host { $::fqdn: ensure => present, alias => $::hostname, address => $::ipaddress, use => "generic-host", } @@nagios_service { "check_ping_${::fqdn}": check_command => "check_ping", use => "generic-service", host_name => $::fqdn", notification_period => "24x7", service_description => "${::fqdn}_check_ping", } }
Nagios_* {} resource types node icinga.example.org { Nagios_host << >> Nagios_service << >> }
Nagios_* {} resource types @@nagios_host { $::fqdn: ensure => absent, } @@nagios_service { "check_ping_${::fqdn}": ensure => absent, }
Nagios_* {} resource types Nagios_host << >> Nagios_service << >>
Nagios_* {} resource types /etc/nagios/
Puppet Config management tool Exported resources PuppetDB Nagios_* resource types
Puppet Config management tool Exported resources PuppetDB Nagios_* resource types https://github.com/icinga/puppet-icinga2 https://github.com/icinga/puppet-icingaweb2
Puppet-Icinga2 puppet-icinga2 (develop) git log tail -4 Author: Nick Chappell Date: Fri May 2 19:36:32 2014-0700 Initial commit.
Other tools... https://github.com/icinga/puppet-icinga2 https://github.com/icinga/puppet-icingaweb2 https://github.com/icinga/icinga2-ansible https://github.com/icinga/docker-icinga2 https://github.com/icinga/chef-icinga2
Puppet as central source of truth
Exported resources
Exported resources
Collected resources
Icinga2
Icinga2 Apply rules Assign rules
Icinga2 template Service "generic-service" { max_check_attempts = 3 check_interval = 5min retry_interval = 5min enable_perfdata = true } apply Service "ping4" { import "generic-service" check_command = "ping4" assign where host.address } apply Service "ping6" { import "generic-service" check_command = "ping6 assign where host.address6 }
Icinga2 apply Service "ssh" { import "generic-service" check_command = "ssh" assign where host.address && host.vars.os == "Linux" }
Icinga2 object HostGroup "database" { display_name = "Database Servers" assign where match("mysql*", host.name) assign where match("psql*", host.name) assign where match("db*", host.name) }
Icinga2 Apply rules Assign rules Exported resources
node 'web.example.org' { @@::icinga2::object::host groups => target_dir => target_file_name => template_to_import => vars => distro => distro_version => virtual_machine => puppet_version => puppet_environment => }, } } { $::fqdn: ['puppet', $::operatingsystem], '/etc/icinga2/objects/hosts', "${::fqdn}.conf", 'linux-host', { $::operatingsystem, $::operatingsystemrelease, $::is_virtual, $::puppetversion, $::environment,
node 'web.example.org' { @@::icinga2::object::host groups => target_dir => target_file_name => template_to_import => vars => distro => distro_version => virtual_machine => puppet_version => puppet_environment => puppet_classes => puppet_modules => }, } } { $::fqdn: ['puppet', $::operatingsystem], '/etc/icinga2/objects/hosts', "${::fqdn}.conf", 'linux-host', { $::operatingsystem, $::operatingsystemrelease, $::is_virtual, $::puppetversion, $::environment, $::puppet_classes, $::puppet_modules,
classes_file = '/var/lib/puppet/classes.txt' classes_hash = {} modules_array = [] File.foreach(classes_file) do l modules_array << l.chomp.gsub(/::.*/, '') end modules_array = modules_array.sort.uniq modules_array.each do i classes_array = [] classes_array << i File.foreach(classes_file) do l classes_array << l.chomp if l =~ /^#{i}/ classes_array = classes_array.sort.uniq end classes_hash[i] = classes_array end
Facter.add(:puppet_modules, :timeout => 10) do confine :kernel => 'Linux' setcode do modules_array.sort.uniq.join(', ').to_s end end Facter.add(:puppet_classes, :timeout => 10) do confine :kernel => 'Linux' setcode do classes_hash.map { _k, v v }.sort.uniq.join(', ').to_s end end
# facter -p puppet_modules --yaml --puppet_modules: - apache - collectd - concat - dnsclient - firewall - grafana - icinga2 - logrotate - mysql - postfix - ssh - stdlib - sudo - sysctl - ulimit
# facter -p puppet_classes --yaml --puppet_classes: mysql: - mysql - "mysql::client" - "mysql::client::install" - "mysql::server" - "mysql::server::config" - "mysql::server::install" - "mysql::server::installdb" - "mysql::server::service" icinga2: - icinga2 - "icinga2::server" - "icinga2::server::config" - "icinga2::server::features" - "icinga2::server::install" - "icinga2::server::service"
@@::icinga2::object::host { $::fqdn: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${::fqdn}.conf", template_to_import => 'linux-host', vars => { distro => $::operatingsystem, distro_version => $::operatingsystemrelease, virtual_machine => $::is_virtual, puppet_version => $::puppetversion, puppet_environment => $::environment, puppet_classes => inline_template( '<%= puppet_classes.collect { k, v v }.sort.join(" ") %>' ), puppet_modules => inline_template( '<%= puppet_modules.sort.join(" ") %>' ), }, }
icinga2::object::apply_service_to_host { 'mysql_server': check_command => 'by_ssh', display_name => 'MySQL', target_dir => '/etc/icinga2/objects/applys', assign_where => '"match(*mysql::server*", host.vars.puppet_classes)"', vars => { by_ssh_command => "check_tcp -H localhost -p 3306", by_ssh_logname => 'root', by_ssh_quiet => true, by_ssh_timeout => 60, }, }
Collected resources
There's a better way
Query PuppetDB
Query PuppetDB curl \ -s \ -X GET \ http://puppetdb:8080/pdb/query/v4/resources \ --data-urlencode \ 'query=["extract", "certname", ["and", ["=", "type", "Class"], ["~", "title", "^Settings"] ] ]'
PuppetDBquery
PuppetDBquery $nodes = query_nodes('class[settings]', 'fqdn')
PuppetDBquery $nodes = query_nodes('class[settings]', 'fqdn') notify { "${nodes}": } # puppet agent --test noop Info: Applying configuration version '1447591034' Notice: /Stage[main]/Main/Notify[web.example.org, db.example.org, icinga2.example.org, puppet.example.org]/message: current_value absent, should be [web.example.org, db.example.org, icinga2.example.org, puppet.example.org] (noop) Notice: Applied catalog in 0.21 seconds
PuppetDBquery $nodes = query_nodes('class[mysql::server]', 'fqdn') notify { "${nodes}": } # puppet agent --test noop Info: Applying configuration version '1447591391' Notice: /Stage[main]/Main/Notify[db.example.org]/message: current_value absent, should be db.example.org (noop) Notice: Applied catalog in 0.21 seconds
PuppetDBquery $nodes = query_nodes('class[settings]', 'fqdn') define monitor::host ($host_name) { $foo.each Integer $index, String $host { ::icinga2::object::host { $host: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${host}.conf", template_to_import => 'linux-host', } } } ::monitor::host { 'all': host_name => $nodes, }
Summary No more exported resources Automagically works Production = deployment + monitoring + backup Purge decommissioned nodes
Contact Tom De Vylder @tomdevylder tomdv@inuits.eu INUITS bvba Heidestraat 257 2070 Zwijndrecht Belgium +32 475 961221 info@inuits.eu