Bernd Ahlers Michael Friedrich. Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2

Transcription

1 Bernd Ahlers Michael Friedrich Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2

2 BEFORE WE START

3 Agenda

4 AGENDA Introduction Tools Log History Logs & Monitoring Demo The Future Resources Q&A

5 Introduction

6 WHO S WHO Bernd Michael German, 34, Graylog2 Developer Austrian, 31, Icinga Developer Graylog2 Team since 2014 Icinga Team since May 2009 TORCH GmbH Application NETWAYS

7 Tools: Graylog2

8 TOOLS: GRAYLOG2 Started as open source project by Lennart Koopmann in 2010 Developed entirely in his free time Free & open source log management tool TORCH GmbH founded as company behind Graylog2 in late 2012 after seeing massive growth and worldwide distribution in large scale setups Team of 8 engineers working full-time on it

9 TOOLS: GRAYLOG2 Big rewrite of Graylog2 started in 2012 Finished with releasing a final v in February 2014 Addresses what we learnt from our first customers and all users Unified REST API communication easy extending and integrating with other products, tools and scripts New web interface focusing on powerful analytics Current stable version:

10 TOOLS: GRAYLOG2

11 Tools: Icinga 2

12 TOOLS: ICINGA 2 Monitoring core engine Checks, alerts, notifications Backend interfaces for frontend visualization Scalable for high performance & real-time monitoring check_interval = 1s Dynamic configuration format Cluster & remote clients, SSL x509 & IPv4/6

13 TOOLS: ICINGA 2 Modular feature set & connectors DB IDO, Livestatus, Perfdata, Graphite, Gelf Supports Monitoring Plugins API Rewritten from scratch Stable version: ( )

14 Log History

15 LOG HISTORY Logs everywhere How to collect them? Splunk (4500$+ for 1GB/day) Syslog-ng + Custom scripts Purpose of your collection? Regex for log parsing Filters Alerts? Notifications? Correlation? Reporting #devops Stack Graylog2, Logstash (ELK) + $monitoring + $metrics + $cfgmgmt

16 LOG HISTORY Problems with remote syslog checks Failure: where s the context? Pattern matching Seek files (state history, rate calculation) Configuration inside Icinga/Plugin Collect them Central log cluster (failover) Correlate events from other servers Defined streams and alert triggers Defined input types (e.g. GELF) Query alert API from Icinga

17 Logs & Monitoring

18 LOG & MONITORING Monitor your logs Call check plugin or receive passive events Generate alerts based on thresholds (configuration) Notifications based on alerts Visualize the current state & history for SLA reporting Trigger event handlers (e.g. iptables on flood) Popular plugins check_logfiles check_splunk Collector APIs & Hooks Graylog2 alert API & alert callback plugin Logstash Nagios output

19 Logs & Monitoring: Strategy

20 STRATEGY Out-of-the-box support or external addons? Add hook to streams for passive event sending? Query a defined API for alerts? Visualize alerts, and where? (we want dashboards!) Re-usable & customizable URL for notifications Combine Log Events & Monitoring notifications and handlers

21 Logs & Monitoring: Push

22 PUSH: GRAYLOG2 ALARM CALLBACK Requirements Icinga API (Command Pipe) Graylog2 Plugin Alarm Callback Ideas Exec Callback+NSCA (Ab)Use the notification plugin Custom Rake Plugin Solution There is no simple & secure unified Core API (yet) Use local Icinga2 client & poll check plugin instead

23 Logs & Monitoring: Poll

24 POLL: ICINGA CHECK Requirements Graylog2 REST API Icinga Check Plugin Ideas Wrapper for Python API calls? Compile check_graylog2_stream? Solution New Icinga Plugin by Graylog2

25 POLL: ICINGA CHECK

26 POLL: ICINGA CHECK #./check-graylog2-stream usage: -condition="<id>": Condition ID, set only to check a single alert (optional) -password="<password>": API password (mandatory) -stream="<id>": Stream ID (mandatory) -url=" URL to Graylog2 api (optional) -user="<username>": API username (mandatory)

27 Combining Graylog2 & Icinga 2

28 COMBINING GRAYLOG2 & ICINGA 2 Events triggered by Icinga 2 Check results State changes Notifications Sent to Graylog2 using `GelfWriter` feature # icinga2 feature enable gelf && service icinga2 restart Visualize in Graylog2 Filter based on type (e.g. state!= OK) Alert streams based on counts, etc

29 NOTIFICATIONS Default Monitoring Alerts are awful You want to see what s wrong. No additional click on your mobile. Icinga 2 triggers a notification Fetch additional information from Graylog2 API Include notes_url with stream id in notification Requirements Custom notification script Stream ids as custom attributes Icinga2 v2.2 Apply For Rules

30 MONITOR THE MONITORING CORE Check Plugin Query Graylog2 Alert Stream API for Icinga 2 alerts Use Stream ID for notifications & notes_url See what s happening in Icinga 2 Restrict views based on user roles Debug plugin & check problems Combine cluster mal-function log Filter events Additional dashboard

31 GRAYLOG2: GELFWRITER VISUALIZED

32 Demo

33 DEMO Graylog x Icinga check-graylog2-stream Plugin Configuration Graylog2 icinga2 stream & alert Icinga2 check plugin & host/service/notification apply rules

34 The FUTURE

35 THE FUTURE Build your own stack Combine existing interfaces into one Graylog2 streams in Icinga Web 2 (ask Tom!) Icinga 2 Events in Graylog2 (more? We want more!) Correlate your monitoring events with events & logs of any kind Think about Simple and secure event receiver Auto-Discover checkable objects from log alerts Alert stream rules for monitoring

36 RESOURCES

37 Code Vagrant Box icinga2x-graylog2 Documentation

38 Q&A? Questions & Answers Web Releases github.com/{graylog2,icinga} IRC #graylog2 #icinga on FreeNode Support support.{graylog2,icinga}.org Twitter twitter.com/{graylog2,icinga}.. Everywhere!