LOG- UND EVENTMANAGEMENT OSMC 2013 23.10.2013 BERND ERK NETWAYS GMBH
AGENDA Kurzvorstellung Einführung Architektur Installation Routing und Filterung von Events Interfaces & API Integration in Nagios und Icinga Eventkorrelation mit EDBC Fragen & Antworten
KURZVORSTELLUNG
KURZVORSTELLUNG NETWAYS Firmengründung 1995 Open Source seit 1997 40 Mitarbeiter Spezialisierung in den Bereichen Open Source Systems Management und Open Source Datacenter Infrastructure http://jobs.netways.de
NETWAYS KOMPETENZEN OPEN SOURCE SYSTEMS MANAGEMENT OPEN SOURCE DATA CENTER Monitoring & Reporting Configuration Management Service Management Knowledge Management Backup & Recovery High Availability & Clustering Cloud Computing Load Balancing Virtualization Database Management MANAGED SERVICES MONITORING HARDWARE KONFERENZEN
NETWAYS KONFERENZEN PuppetCamp 2013/2014 28. November München 11. April Berlin CfP für Berlin läuft noch Open Source Datacenter Conference 08. 10. April 2014 125 Teilnehmer (2012) Datacenter Automation DevOps CfP läuft bis zum 31. Dezember 2013
EINFÜHRUNG
LOGS Logs -> Fluss an unstrukturierten Daten Oct 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11: disconnected by user bestehend aus Timestamp und Message
EVENTS Event -> Fluss an strukturierten Daten Event { Time: Oct 4 16:57:24 Process: sshd State: Received disconnect from 10.10.0.31 Client: 10.10.0.31 bestehend aus konkreten Attributen
LOG & EVENTMANAGEMENT Logs > Event > Analyse (Korrelation) > Aktion
TOOLS Nagios & Icinga Addons check_logfiles NagTrap EventDB EDBC Logmanagement-Tools Graylog Fluentd Logstash
LOGSTASH Logstash
ARCHITEKTUR & INSTALLATION
LOGSTASH Logmanagement auf Basis von JRuby Konfigurierbare Pipe Flexible Plugin-Architektur für Input Filter Output Standardplugins für alle gängige Protokolle Webinterface Single File Deployment
LOGSTASH - IO Inputs Outputs amqp relp amqp http s3 drupal_dblog s3 boundary irc sns elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc log4j lumberjack pipe rabbitmq snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq circonus cloudwatch datadog datadog_metrics elasticsearch elasticsearch_http elasticsearch_river email exec file ganglia gelf gemfire google_cloud_storage graphite graphtastic jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis riak sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq redis hipchat riemann
INSTALLATION - LOGSTASH Download - http://logstash.net java -jar logstash-x.x.x-flatjar.jar agent -f <config-file>
ARCHITEKTUR Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface
REDIS NoSQL in memory auf Basis von C Unterstützung verschiedener Datentypen strings hashes lists sets and sorted sets Support für verschiedene Replikationsszenarien SAUSCHNELL $./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q SET: 122556.53 requests per second GET: 123601.76 requests per second LPUSH: 136752.14 requests per second LPOP: 132424.03 requests per second
INSTALLATION - REDIS Download - http://redis.io/download make make test make install /usr/local/bin/redis-server
ELASTICSEARCH Schemafreier RESTful Suchserver auf Basis von Java Basierend auf Lucene Core Vergleichbar mit Apache Solr Verteilte Architektur durch Shards Replicas Gateways Realtime-Suche als Basis für Kibana
INSTALLATION - ELASTICSEARCH Download http://elasticsearch.org/download/ Entpacken des Archives Ausführung von bin/elasticsearch (-f)
ROUTING UND FILTERUNG VON EVENTS
ÜBERSICHT Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface
KONFIGURATION - LOGSTASH - SHIPPER Übermittlung von Logs an Logstash Logstash Lumberjack Syslog Log4J Gelf File-Read u.v.a.m.
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH - SHIPPER Konfiguration input { file { path => "/root/osmc/demodata/access.log.1 type => "apache-access" output { stdout { debug => true redis { host => "127.0.0.1" data_type => "list" key => "logstash.apache" java -jar logstash-current.jar agent -f logstash_shipper.conf Shipper Shipper Shipper
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH - INDEXER Konfiguration input { redis { host => "127.0.0.1" type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache output { stdout { debug => true elasticsearch { host => "127.0.0.1" Shipper Shipper Shipper
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH INDEXER - APACHE Konfiguration für Apache-Logs input { redis { host => "127.0.0.1" type => "apache-access data_type => "list" key => "logstash.apache format => "json_event" filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG" ] output { elasticsearch { host => "127.0.0.1 Shipper Shipper Shipper
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH INDEXER - GEOIP Konfiguration für Geo-Daten input { redis { host => "127.0.0.1" type => "apache-access data_type => "list" key => "logstash.apache filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG" geoip { source => "clientip" add_tag => ["geotag"] output { elasticsearch {host => "127.0.0.1 Shipper Shipper Shipper
INTERFACES & API
KIBANA 3
KIBANA
ELASTICHQ
KIBANA - DEMO DEMO
INTEGRATION NAGIOS UND ICINGA
REALTIME LOGANALYSE Analyse verschiedener Quellen in Realtime Prüfung auf Patterns und States Facilitites Regex Programs Übermittlung als Passiver Event
ÜBERSICHT LOGSTASH UND ICINGA Indexer Search & Storage Webinterface Icinga - Commandpipe Icinga Web
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH INDEXER - ICINGA Konfiguration für Icinga-Alert input { Shipper Shipper Shipper filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE" ] grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2 ] output { elasticsearch {host => "127.0.0.1 nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd"
LOGSTASH ICINGA - DEMO DEMO
EVENTKORRELATION MIT EDBC
EDBC - EINFÜHRUNG EDBC EventDB Correlator Receptors Eingangskanäle für verschiedene Eventquellen Processors Verarbeitet konfigurierte Filterregeln Chains Verbindet verschiedene Receptoren und Processoren zu komplexeren Prozesseketten
EDBC - ARCHITEKTUR Filter-Chain Event A Cleared Acknowledge Group Event B Event C Receptor Aggregator Persister Clearing Aggregate / No match Event A
EDBC - BEISPIEL [example-aggregator] class: processor type: aggregation matcher: message REGEXP 'The server (?P<HOSTNAME>\w+) just went down. Errorcode (?P<CODE>\d+)' aggregatemessage: Server $HOSTNAME is down (Code : $CODE) ($_COUNT events) datasource: @mysql http://docs.netways.org/edbc/
ZUGABE
REALTIME GRAPHING
STATSD & GRAPHITE StatsD Netzwerkdaemon auf Basis von UDP Bucket -> Value -> Flush Entkoppelte Zwischenaggretion für Statisik Graphite Graphing-Framework bestehend aus Whisper (Datenbank) Carbon (Engine) Graphite-Web (Interface)
INSTALLATION STATSD - NODEJS apt-get install make python g++ checkinstall mkdir nodejs && cd nodejs wget -N http://nodejs.org/dist/node-latest.tar.gz tar xzvf node-latest.tar.gz && cd `ls -rd node-v*` checkinstall
INSTALLATION STATSD wget https://github.com/etsy/statsd/archive/master.zip unzip master.zip node stats.js config.js
MONITORING - STATSD Status Informationen echo stats nc 127.0.0.1 8126 echo health nc 127.0.0.1 8126 Timer- und Counterinfo echo counters nc 127.0.0.1 8126 echo timers nc 127.0.0.1 8126
INSTALLATION GRAPHITE Download der Sources git clone https://github.com/graphiteproject/graphite-web.git git clone https://github.com/graphiteproject/carbon.git git clone https://github.com/graphiteproject/whisper.git
INSTALLATION GRAPHITE Installation Whisper pushd whisper sudo python setup.py install popd Installation Carbon pushd carbon sudo python setup.py install popd Konfiguration Carbon pushd /opt/graphite/conf cp carbon.conf.example carbon.conf cp storage-schemas.conf.example storage-schemas.conf
INSTALLATION GRAPHITE - WEBAPP Check Dependencies Graphite webapp pushd graphite-web python check-dependencies.py popd Installation Graphite webapp pushd graphite-web python setup.py install popd Konfiguration Apache example-graphite-vhost.conf
ÜBERSICHT STATSD UND GRAPHITE Indexer Search & Storage Webinterface Statsd Graphite
KONFIGURATION - LOGSTASH INDEXER - STATSD Konfiguration für Statsd Shipper Shipper input { redis { host => "127.0.0.1" type => "apache-access data_type => "list" key => "logstash.apache format => "json_event add_field=> ["sitename","www.icinga.org"] filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG" ] output { stdout { debug => true if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename.response.%{response count => ["apache.%{sitename.bytes", "%{bytes"] elasticsearch {host => "127.0.0.1 Shipper Broker Search & Storage Webinterface Indexer StatsD
GRAPHITE - DEMO DEMO
FRAGEN & ANTWORTEN
FRAGEN & ANTWORTEN NETWAYS GmbH Deutschherrnstrasse 15-19 90429 Nürmberg Tel: +49 911 92885-0 DANKE Fax: +49 911 92885-77 Email: info@netways.de Website: www.netways.de Twitter: twitter.com/netways Facebook: facebook.com/netways Blog: blog.netways.de