LOG- UND EVENTMANAGEMENT MIT LOGSTASH UND GRAPHITE

Transcription

1 LOG- UND EVENTMANAGEMENT MIT LOGSTASH UND GRAPHITE LINUXTAG BERND ERK NETWAYS GMBH

2 AGENDA Kurzvorstellung Einführung Architektur Installation Routing und Filterung von Events Interfaces & API Integration in Nagios und Icinga Fragen & Antworten

3 KURZVORSTELLUNG

4 KURZVORSTELLUNG NETWAYS Firmengründung 1995 Open Source seit Mitarbeiter Spezialisierung in den Bereichen Open Source Systems Management und Open Source Datacenter Infrastructure

5 KURZVORSTELLUNG BERND ERK Gründung 1977 Open Source seit 2007 Spezialisierung in Open Source Unterhaltung und allen anderen Dingen die links und rechts runterfallen

6 NETWAYS KOMPETENZEN OPEN SOURCE SYSTEMS MANAGEMENT OPEN SOURCE DATA CENTER Monitoring & Reporting Configuration Management Service Management Knowledge Management Backup & Recovery High Availability & Clustering Cloud Computing Load Balancing Virtualization Database Management MANAGED SERVICES MONITORING HARDWARE KONFERENZEN

7 NETWAYS KONFERENZEN Open Source Backup Conference September Köln PuppetCamp Düsseldorf 16. Oktober 2014 Düsseldorf Open Source Monitoring Conference November Nürnberg OpenNebula Conf Dezember Berlin

8 EINFÜHRUNG

9 LOGS Logs -> Fluss an unstrukturierten Daten Oct 4 16:57:24 web sshd[25828]: Received disconnect from : 11: disconnected by user bestehend aus Timestamp und Message

10 EVENTS Event -> Fluss an strukturierten Daten Event { Time: Oct 4 16:57:24 Process: sshd State: Received disconnect from Client: bestehend aus konkreten Attributen

11 LOG & EVENTMANAGEMENT Logs > Event > Analyse (Korrelation) > Aktion

12 TOOLS Nagios & Icinga Addons check_logfiles NagTrap EventDB EDBC Logmanagement-Tools Graylog2 Fluentd Logstash

13 LOGSTASH Logstash

14 ARCHITEKTUR & INSTALLATION

15 LOGSTASH Logmanagement auf Basis von JRuby Konfigurierbare Pipe Flexible Plugin-Architektur für Input Filter Output Standardplugins für alle gängige Protokolle Webinterface

16 LOGSTASH - IO Inputs Outputs amqp relp amqp http s3 drupal_dblog s3 boundary irc sns elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc log4j lumberjack pipe rabbitmq snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq circonus cloudwatch datadog datadog_metrics elasticsearch elasticsearch_http elasticsearch_river exec file ganglia gelf gemfire google_cloud_storage graphite graphtastic jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis riak sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq redis hipchat riemann

17 INSTALLATION - LOGSTASH Download - tar xvf logstash-x.x.x.tar bin/logtash agent -f <config-file>

18 ARCHITEKTUR Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface

19 REDIS NoSQL in memory auf Basis von C Unterstützung verschiedener Datentypen strings hashes lists sets and sorted sets Support für verschiedene Replikationsszenarien SCHNELL $./redis-benchmark -r n t get,set,lpush,lpop -q SET: requests per second GET: requests per second LPUSH: requests per second LPOP: requests per second

20 INSTALLATION - REDIS Download - make make test make install /usr/local/bin/redis-server

21 ELASTICSEARCH Schemafreier RESTful Suchserver auf Basis von Java Basierend auf Lucene Core Vergleichbar mit Apache Solr Verteilte Architektur durch Shards Replicas Gateways Realtime-Suche als Basis für Kibana

22 INSTALLATION - ELASTICSEARCH Download Entpacken des Archives Ausführung von bin/elasticsearch

23 ROUTING UND FILTERUNG VON EVENTS

24 ÜBERSICHT Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface

25 KONFIGURATION - LOGSTASH - SHIPPER Übermittlung von Logs an Logstash Logstash Lumberjack Syslog Log4J Gelf File-Read u.v.a.m.

26 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH - SHIPPER Konfiguration input { file { path => "/root/osmc/demodata/access.log.1 type => "apache-access" output { redis { host => " " data_type => "list" key => "logstash.apache" Shipper Shipper Shipper bin/logstash agent -f logstash_shipper.conf

27 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH - INDEXER Konfiguration input { redis { host => " " type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache output { elasticsearch { host => " " Shipper Shipper Shipper

28 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH INDEXER - APACHE Konfiguration für Apache-Logs input { redis { host => " " type => "apache-access data_type => "list" key => "logstash.apache format => "json_event" filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG" ] output { elasticsearch {host => " Shipper Shipper Shipper

29 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH INDEXER - GEOIP Konfiguration für Geo-Daten input { redis { host => " " type => "apache-access data_type => "list" key => "logstash.apache filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG" geoip { source => "clientip" add_tag => ["geotag"] output { elasticsearch {host => " Shipper Shipper Shipper

30 INTERFACES & API

31 KIBANA Kibana

32 KIBANA

33 ELASTICHQ

34 KIBANA - DEMO DEMO

35 INTEGRATION NAGIOS UND ICINGA

36 REALTIME LOGANALYSE Analyse verschiedener Quellen in Realtime Prüfung auf Patterns und States Facilitites Regex Programs Übermittlung als Passiver Event

37 ÜBERSICHT LOGSTASH UND ICINGA Indexer Search & Storage Webinterface Icinga - Commandpipe Icinga Web

38 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH INDEXER - ICINGA Konfiguration für Icinga-Alert input { Shipper Shipper Shipper filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE" ] grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2 ] output { elasticsearch {host => " nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd"

39 LOGSTASH ICINGA - DEMO DEMO

40 ZUGABE

41 REALTIME GRAPHING

42 STATSD & GRAPHITE StatsD Netzwerkdaemon auf Basis von UDP Bucket -> Value -> Flush Entkoppelte Zwischenaggretion für Statisik Graphite Graphing-Framework bestehend aus Whisper (Datenbank) Carbon (Engine) Graphite-Web (Interface)

43 INSTALLATION STATSD - NODEJS apt-get install make python g++ checkinstall mkdir nodejs && cd nodejs wget -N tar xzvf node-latest.tar.gz && cd `ls -rd node-v*` checkinstall

44 INSTALLATION STATSD wget unzip master.zip node stats.js config.js

45 MONITORING - STATSD Status Informationen echo stats nc echo health nc Timer- und Counterinfo echo counters nc echo timers nc

46 INSTALLATION GRAPHITE Download der Sources git clone git clone git clone

47 INSTALLATION GRAPHITE Installation Whisper pushd whisper sudo python setup.py install popd Installation Carbon pushd carbon sudo python setup.py install popd Konfiguration Carbon pushd /opt/graphite/conf cp carbon.conf.example carbon.conf cp storage-schemas.conf.example storage-schemas.conf

48 INSTALLATION GRAPHITE - WEBAPP Check Dependencies Graphite webapp pushd graphite-web python check-dependencies.py popd Installation Graphite webapp pushd graphite-web python setup.py install popd Konfiguration Apache example-graphite-vhost.conf

49 ÜBERSICHT STATSD UND GRAPHITE Indexer Search & Storage Webinterface Statsd Graphite

50 KONFIGURATION - LOGSTASH INDEXER - STATSD Konfiguration für Statsd Shipper Shipper input { redis { host => " " type => "apache-access data_type => "list" key => "logstash.apache format => "json_event add_field=> ["sitename"," filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG" ] output { stdout { debug => true if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename.response.%{response count => ["apache.%{sitename.bytes", "%{bytes"] elasticsearch {host => " Shipper Broker Search & Storage Webinterface Indexer StatsD

51 GRAPHITE - DEMO DEMO

52 FRAGEN & ANTWORTEN

53 FRAGEN & ANTWORTEN NETWAYS GmbH Deutschherrnstrasse DANKE Nürnberg Tel: ICINGA- VORTRAG UM UHR Fax: Website: Twitter: twitter.com/netways Facebook: facebook.com/netways Blog: blog.netways.de