The road to lazy monitoring with Icinga2 & Puppet. Tom De

Transcription

1 The road to lazy monitoring with Icinga2 & Puppet Tom De

2 whois Ops guy Rusty, old, Bavarian vehicles enthousiast The one who always gets the patdown at checkpoints Open Source Consultant & Enthousiast at Inuits Team Packages & Tools at Icinga

3 Monitoring!

4 Monitoring! Image credits:

5 Monitoring :((

6 Monitoring is... Boring Tedious Repetitive Production = deployment + monitoring + backup Often neglected

7 Monitoring is... Boring Tedious Repetitive Production = deployment + monitoring + backup Often neglected

8 Why can't someone else do it?

9 Someone else! Tedious Repetitive People make mistakes Nobody likes Nagios configs

10 Someone else! Tedious Repetitive People make mistakes Nobody likes Nagios configs Monitor servers not people Image credits:

11 Single source of truth

12 Config management!

13 Puppet Rudder Juju Chef Config management! Ansible Saltstack TrueConfig CFEngine

14 Puppet

15 Puppet Config management tool Exported resources PuppetDB Nagios_* {} resource types

16 node web.example.org { $::fqdn: ensure => present, alias => $::hostname, address => $::ipaddress, use => "generic-host", { "check_ping_${::fqdn}": check_command => "check_ping", use => "generic-service", host_name => $::fqdn", notification_period => "24x7", service_description => "${::fqdn}_check_ping", } }

17 Nagios_* {} resource types node icinga.example.org { Nagios_host << >> Nagios_service << >> }

18 Nagios_* {} resource types { $::fqdn: ensure => absent, { "check_ping_${::fqdn}": ensure => absent, }

19 Nagios_* {} resource types Nagios_host << >> Nagios_service << >>

20 Nagios_* {} resource types /etc/nagios/

21 Puppet Config management tool Exported resources PuppetDB Nagios_* resource types

22 Puppet Config management tool Exported resources PuppetDB Nagios_* resource types

23 Puppet-Icinga2 puppet-icinga2 (develop) git log tail -4 Author: Nick Chappell Date: Fri May 2 19:36: Initial commit.

24 Other tools

25 Puppet as central source of truth

26 Exported resources

27 Exported resources

28 Collected resources

29 Icinga2

30 Icinga2 Apply rules Assign rules

31 Icinga2 template Service "generic-service" { max_check_attempts = 3 check_interval = 5min retry_interval = 5min enable_perfdata = true } apply Service "ping4" { import "generic-service" check_command = "ping4" assign where host.address } apply Service "ping6" { import "generic-service" check_command = "ping6 assign where host.address6 }

32 Icinga2 apply Service "ssh" { import "generic-service" check_command = "ssh" assign where host.address && host.vars.os == "Linux" }

33 Icinga2 object HostGroup "database" { display_name = "Database Servers" assign where match("mysql*", host.name) assign where match("psql*", host.name) assign where match("db*", host.name) }

34 Icinga2 Apply rules Assign rules Exported resources

35 node 'web.example.org' groups => target_dir => target_file_name => template_to_import => vars => distro => distro_version => virtual_machine => puppet_version => puppet_environment => }, } } { $::fqdn: ['puppet', $::operatingsystem], '/etc/icinga2/objects/hosts', "${::fqdn}.conf", 'linux-host', { $::operatingsystem, $::operatingsystemrelease, $::is_virtual, $::puppetversion, $::environment,

36 node 'web.example.org' groups => target_dir => target_file_name => template_to_import => vars => distro => distro_version => virtual_machine => puppet_version => puppet_environment => puppet_classes => puppet_modules => }, } } { $::fqdn: ['puppet', $::operatingsystem], '/etc/icinga2/objects/hosts', "${::fqdn}.conf", 'linux-host', { $::operatingsystem, $::operatingsystemrelease, $::is_virtual, $::puppetversion, $::environment, $::puppet_classes, $::puppet_modules,

37 classes_file = '/var/lib/puppet/classes.txt' classes_hash = {} modules_array = [] File.foreach(classes_file) do l modules_array << l.chomp.gsub(/::.*/, '') end modules_array = modules_array.sort.uniq modules_array.each do i classes_array = [] classes_array << i File.foreach(classes_file) do l classes_array << l.chomp if l =~ /^#{i}/ classes_array = classes_array.sort.uniq end classes_hash[i] = classes_array end

38 Facter.add(:puppet_modules, :timeout => 10) do confine :kernel => 'Linux' setcode do modules_array.sort.uniq.join(', ').to_s end end Facter.add(:puppet_classes, :timeout => 10) do confine :kernel => 'Linux' setcode do classes_hash.map { _k, v v }.sort.uniq.join(', ').to_s end end

39 # facter -p puppet_modules --yaml --puppet_modules: - apache - collectd - concat - dnsclient - firewall - grafana - icinga2 - logrotate - mysql - postfix - ssh - stdlib - sudo - sysctl - ulimit

40 # facter -p puppet_classes --yaml --puppet_classes: mysql: - mysql - "mysql::client" - "mysql::client::install" - "mysql::server" - "mysql::server::config" - "mysql::server::install" - "mysql::server::installdb" - "mysql::server::service" icinga2: - icinga2 - "icinga2::server" - "icinga2::server::config" - "icinga2::server::features" - "icinga2::server::install" - "icinga2::server::service"

41 { $::fqdn: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${::fqdn}.conf", template_to_import => 'linux-host', vars => { distro => $::operatingsystem, distro_version => $::operatingsystemrelease, virtual_machine => $::is_virtual, puppet_version => $::puppetversion, puppet_environment => $::environment, puppet_classes => inline_template( '<%= puppet_classes.collect { k, v v }.sort.join(" ") %>' ), puppet_modules => inline_template( '<%= puppet_modules.sort.join(" ") %>' ), }, }

42 icinga2::object::apply_service_to_host { 'mysql_server': check_command => 'by_ssh', display_name => 'MySQL', target_dir => '/etc/icinga2/objects/applys', assign_where => '"match(*mysql::server*", host.vars.puppet_classes)"', vars => { by_ssh_command => "check_tcp -H localhost -p 3306", by_ssh_logname => 'root', by_ssh_quiet => true, by_ssh_timeout => 60, }, }

43 Collected resources

44 There's a better way

45 Query PuppetDB

46 Query PuppetDB curl \ -s \ -X GET \ \ --data-urlencode \ 'query=["extract", "certname", ["and", ["=", "type", "Class"], ["~", "title", "^Settings"] ] ]'

47 PuppetDBquery

48

49 PuppetDBquery $nodes = query_nodes('class[settings]', 'fqdn')

50 PuppetDBquery $nodes = query_nodes('class[settings]', 'fqdn') notify { "${nodes}": } # puppet agent --test noop Info: Applying configuration version ' ' Notice: /Stage[main]/Main/Notify[web.example.org, db.example.org, icinga2.example.org, puppet.example.org]/message: current_value absent, should be [web.example.org, db.example.org, icinga2.example.org, puppet.example.org] (noop) Notice: Applied catalog in 0.21 seconds

51 PuppetDBquery $nodes = query_nodes('class[mysql::server]', 'fqdn') notify { "${nodes}": } # puppet agent --test noop Info: Applying configuration version ' ' Notice: /Stage[main]/Main/Notify[db.example.org]/message: current_value absent, should be db.example.org (noop) Notice: Applied catalog in 0.21 seconds

52 PuppetDBquery $nodes = query_nodes('class[settings]', 'fqdn') define monitor::host ($host_name) { $foo.each Integer $index, String $host { ::icinga2::object::host { $host: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${host}.conf", template_to_import => 'linux-host', } } } ::monitor::host { 'all': host_name => $nodes, }

53 Summary No more exported resources Automagically works Production = deployment + monitoring + backup Purge decommissioned nodes

54

55 Contact Tom De INUITS bvba Heidestraat Zwijndrecht Belgium