LOG- UND EVENTMANAGEMENT

Transcription

1 LOG- UND EVENTMANAGEMENT OSMC BERND ERK NETWAYS GMBH

2 AGENDA Kurzvorstellung Einführung Architektur Installation Routing und Filterung von Events Interfaces & API Integration in Nagios und Icinga Eventkorrelation mit EDBC Fragen & Antworten

3 KURZVORSTELLUNG

4 KURZVORSTELLUNG NETWAYS Firmengründung 1995 Open Source seit Mitarbeiter Spezialisierung in den Bereichen Open Source Systems Management und Open Source Datacenter Infrastructure

5 NETWAYS KOMPETENZEN OPEN SOURCE SYSTEMS MANAGEMENT OPEN SOURCE DATA CENTER Monitoring & Reporting Configuration Management Service Management Knowledge Management Backup & Recovery High Availability & Clustering Cloud Computing Load Balancing Virtualization Database Management MANAGED SERVICES MONITORING HARDWARE KONFERENZEN

6 NETWAYS KONFERENZEN PuppetCamp 2013/ November München 11. April Berlin CfP für Berlin läuft noch Open Source Datacenter Conference April Teilnehmer (2012) Datacenter Automation DevOps CfP läuft bis zum 31. Dezember 2013

7 EINFÜHRUNG

8 LOGS Logs -> Fluss an unstrukturierten Daten Oct 4 16:57:24 web sshd[25828]: Received disconnect from : 11: disconnected by user bestehend aus Timestamp und Message

9 EVENTS Event -> Fluss an strukturierten Daten Event { Time: Oct 4 16:57:24 Process: sshd State: Received disconnect from Client: bestehend aus konkreten Attributen

10 LOG & EVENTMANAGEMENT Logs > Event > Analyse (Korrelation) > Aktion

11 TOOLS Nagios & Icinga Addons check_logfiles NagTrap EventDB EDBC Logmanagement-Tools Graylog Fluentd Logstash

12 LOGSTASH Logstash

13 ARCHITEKTUR & INSTALLATION

14 LOGSTASH Logmanagement auf Basis von JRuby Konfigurierbare Pipe Flexible Plugin-Architektur für Input Filter Output Standardplugins für alle gängige Protokolle Webinterface Single File Deployment

15 LOGSTASH - IO Inputs Outputs amqp relp amqp http s3 drupal_dblog s3 boundary irc sns elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc log4j lumberjack pipe rabbitmq snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq circonus cloudwatch datadog datadog_metrics elasticsearch elasticsearch_http elasticsearch_river exec file ganglia gelf gemfire google_cloud_storage graphite graphtastic jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis riak sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq redis hipchat riemann

16 INSTALLATION - LOGSTASH Download - java -jar logstash-x.x.x-flatjar.jar agent -f <config-file>

17 ARCHITEKTUR Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface

18 REDIS NoSQL in memory auf Basis von C Unterstützung verschiedener Datentypen strings hashes lists sets and sorted sets Support für verschiedene Replikationsszenarien SAUSCHNELL $./redis-benchmark -r n t get,set,lpush,lpop -q SET: requests per second GET: requests per second LPUSH: requests per second LPOP: requests per second

19 INSTALLATION - REDIS Download - make make test make install /usr/local/bin/redis-server

20 ELASTICSEARCH Schemafreier RESTful Suchserver auf Basis von Java Basierend auf Lucene Core Vergleichbar mit Apache Solr Verteilte Architektur durch Shards Replicas Gateways Realtime-Suche als Basis für Kibana

21 INSTALLATION - ELASTICSEARCH Download Entpacken des Archives Ausführung von bin/elasticsearch (-f)

22 ROUTING UND FILTERUNG VON EVENTS

23 ÜBERSICHT Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface

24 KONFIGURATION - LOGSTASH - SHIPPER Übermittlung von Logs an Logstash Logstash Lumberjack Syslog Log4J Gelf File-Read u.v.a.m.

25 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH - SHIPPER Konfiguration input { file { path => "/root/osmc/demodata/access.log.1 type => "apache-access" output { stdout { debug => true redis { host => " " data_type => "list" key => "logstash.apache" java -jar logstash-current.jar agent -f logstash_shipper.conf Shipper Shipper Shipper

26 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH - INDEXER Konfiguration input { redis { host => " " type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache output { stdout { debug => true elasticsearch { host => " " Shipper Shipper Shipper

27 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH INDEXER - APACHE Konfiguration für Apache-Logs input { redis { host => " " type => "apache-access data_type => "list" key => "logstash.apache format => "json_event" filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG" ] output { elasticsearch { host => " Shipper Shipper Shipper

28 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH INDEXER - GEOIP Konfiguration für Geo-Daten input { redis { host => " " type => "apache-access data_type => "list" key => "logstash.apache filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG" geoip { source => "clientip" add_tag => ["geotag"] output { elasticsearch {host => " Shipper Shipper Shipper

29 INTERFACES & API

30 KIBANA 3

31 KIBANA

32 ELASTICHQ

33 KIBANA - DEMO DEMO

34 INTEGRATION NAGIOS UND ICINGA

35 REALTIME LOGANALYSE Analyse verschiedener Quellen in Realtime Prüfung auf Patterns und States Facilitites Regex Programs Übermittlung als Passiver Event

36 ÜBERSICHT LOGSTASH UND ICINGA Indexer Search & Storage Webinterface Icinga - Commandpipe Icinga Web

37 Broker Indexer Search & Storage Webinterface // blog.netways.de KONFIGURATION - LOGSTASH INDEXER - ICINGA Konfiguration für Icinga-Alert input { Shipper Shipper Shipper filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE" ] grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2 ] output { elasticsearch {host => " nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd"

38 LOGSTASH ICINGA - DEMO DEMO

39 EVENTKORRELATION MIT EDBC

40 EDBC - EINFÜHRUNG EDBC EventDB Correlator Receptors Eingangskanäle für verschiedene Eventquellen Processors Verarbeitet konfigurierte Filterregeln Chains Verbindet verschiedene Receptoren und Processoren zu komplexeren Prozesseketten

41 EDBC - ARCHITEKTUR Filter-Chain Event A Cleared Acknowledge Group Event B Event C Receptor Aggregator Persister Clearing Aggregate / No match Event A

42 EDBC - BEISPIEL [example-aggregator] class: processor type: aggregation matcher: message REGEXP 'The server (?P<HOSTNAME>\w+) just went down. Errorcode (?P<CODE>\d+)' aggregatemessage: Server $HOSTNAME is down (Code : $CODE) ($_COUNT events)

43 ZUGABE

44 REALTIME GRAPHING

45 STATSD & GRAPHITE StatsD Netzwerkdaemon auf Basis von UDP Bucket -> Value -> Flush Entkoppelte Zwischenaggretion für Statisik Graphite Graphing-Framework bestehend aus Whisper (Datenbank) Carbon (Engine) Graphite-Web (Interface)

46 INSTALLATION STATSD - NODEJS apt-get install make python g++ checkinstall mkdir nodejs && cd nodejs wget -N tar xzvf node-latest.tar.gz && cd `ls -rd node-v*` checkinstall

47 INSTALLATION STATSD wget unzip master.zip node stats.js config.js

48 MONITORING - STATSD Status Informationen echo stats nc echo health nc Timer- und Counterinfo echo counters nc echo timers nc

49 INSTALLATION GRAPHITE Download der Sources git clone git clone git clone

50 INSTALLATION GRAPHITE Installation Whisper pushd whisper sudo python setup.py install popd Installation Carbon pushd carbon sudo python setup.py install popd Konfiguration Carbon pushd /opt/graphite/conf cp carbon.conf.example carbon.conf cp storage-schemas.conf.example storage-schemas.conf

51 INSTALLATION GRAPHITE - WEBAPP Check Dependencies Graphite webapp pushd graphite-web python check-dependencies.py popd Installation Graphite webapp pushd graphite-web python setup.py install popd Konfiguration Apache example-graphite-vhost.conf

52 ÜBERSICHT STATSD UND GRAPHITE Indexer Search & Storage Webinterface Statsd Graphite

53 KONFIGURATION - LOGSTASH INDEXER - STATSD Konfiguration für Statsd Shipper Shipper input { redis { host => " " type => "apache-access data_type => "list" key => "logstash.apache format => "json_event add_field=> ["sitename"," filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG" ] output { stdout { debug => true if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename.response.%{response count => ["apache.%{sitename.bytes", "%{bytes"] elasticsearch {host => " Shipper Broker Search & Storage Webinterface Indexer StatsD

54 GRAPHITE - DEMO DEMO

55 FRAGEN & ANTWORTEN

56 FRAGEN & ANTWORTEN NETWAYS GmbH Deutschherrnstrasse Nürmberg Tel: DANKE Fax: Website: Twitter: twitter.com/netways Facebook: facebook.com/netways Blog: blog.netways.de