LOG- UND EVENTMANAGEMENT MIT LOGSTASH UND GRAPHITE LINUXTAG 2014 08.05.2014 BERND ERK NETWAYS GMBH
AGENDA Kurzvorstellung Einführung Architektur Installation Routing und Filterung von Events Interfaces & API Integration in Nagios und Icinga Fragen & Antworten
KURZVORSTELLUNG
KURZVORSTELLUNG NETWAYS Firmengründung 1995 Open Source seit 1997 40 Mitarbeiter Spezialisierung in den Bereichen Open Source Systems Management und Open Source Datacenter Infrastructure http://jobs.netways.de
KURZVORSTELLUNG BERND ERK Gründung 1977 Open Source seit 2007 Spezialisierung in Open Source Unterhaltung und allen anderen Dingen die links und rechts runterfallen
NETWAYS KOMPETENZEN OPEN SOURCE SYSTEMS MANAGEMENT OPEN SOURCE DATA CENTER Monitoring & Reporting Configuration Management Service Management Knowledge Management Backup & Recovery High Availability & Clustering Cloud Computing Load Balancing Virtualization Database Management MANAGED SERVICES MONITORING HARDWARE KONFERENZEN
NETWAYS KONFERENZEN Open Source Backup Conference 22. 23. September 2014 - Köln PuppetCamp Düsseldorf 16. Oktober 2014 Düsseldorf Open Source Monitoring Conference 18. 20. November 2014 - Nürnberg OpenNebula Conf 02. 04. Dezember 2014 - Berlin
EINFÜHRUNG
LOGS Logs -> Fluss an unstrukturierten Daten Oct 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11: disconnected by user bestehend aus Timestamp und Message
EVENTS Event -> Fluss an strukturierten Daten Event { Time: Oct 4 16:57:24 Process: sshd State: Received disconnect from 10.10.0.31 Client: 10.10.0.31 bestehend aus konkreten Attributen
LOG & EVENTMANAGEMENT Logs > Event > Analyse (Korrelation) > Aktion
TOOLS Nagios & Icinga Addons check_logfiles NagTrap EventDB EDBC Logmanagement-Tools Graylog2 Fluentd Logstash
LOGSTASH Logstash
ARCHITEKTUR & INSTALLATION
LOGSTASH Logmanagement auf Basis von JRuby Konfigurierbare Pipe Flexible Plugin-Architektur für Input Filter Output Standardplugins für alle gängige Protokolle Webinterface
LOGSTASH - IO Inputs Outputs amqp relp amqp http s3 drupal_dblog s3 boundary irc sns elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc log4j lumberjack pipe rabbitmq snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq circonus cloudwatch datadog datadog_metrics elasticsearch elasticsearch_http elasticsearch_river email exec file ganglia gelf gemfire google_cloud_storage graphite graphtastic jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis riak sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq redis hipchat riemann
INSTALLATION - LOGSTASH Download - http://logstash.net tar xvf logstash-x.x.x.tar bin/logtash agent -f <config-file>
ARCHITEKTUR Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface
REDIS NoSQL in memory auf Basis von C Unterstützung verschiedener Datentypen strings hashes lists sets and sorted sets Support für verschiedene Replikationsszenarien SCHNELL $./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q SET: 122556.53 requests per second GET: 123601.76 requests per second LPUSH: 136752.14 requests per second LPOP: 132424.03 requests per second
INSTALLATION - REDIS Download - http://redis.io/download make make test make install /usr/local/bin/redis-server
ELASTICSEARCH Schemafreier RESTful Suchserver auf Basis von Java Basierend auf Lucene Core Vergleichbar mit Apache Solr Verteilte Architektur durch Shards Replicas Gateways Realtime-Suche als Basis für Kibana
INSTALLATION - ELASTICSEARCH Download http://elasticsearch.org/download/ Entpacken des Archives Ausführung von bin/elasticsearch
ROUTING UND FILTERUNG VON EVENTS
ÜBERSICHT Shipper Shipper Shipper Broker Indexer Search & Storage Webinterface
KONFIGURATION - LOGSTASH - SHIPPER Übermittlung von Logs an Logstash Logstash Lumberjack Syslog Log4J Gelf File-Read u.v.a.m.
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH - SHIPPER Konfiguration input { file { path => "/root/osmc/demodata/access.log.1 type => "apache-access" output { redis { host => "127.0.0.1" data_type => "list" key => "logstash.apache" Shipper Shipper Shipper bin/logstash agent -f logstash_shipper.conf
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH - INDEXER Konfiguration input { redis { host => "127.0.0.1" type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache output { elasticsearch { host => "127.0.0.1" Shipper Shipper Shipper
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH INDEXER - APACHE Konfiguration für Apache-Logs input { redis { host => "127.0.0.1" type => "apache-access data_type => "list" key => "logstash.apache format => "json_event" filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG" ] output { elasticsearch {host => "127.0.0.1 Shipper Shipper Shipper
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH INDEXER - GEOIP Konfiguration für Geo-Daten input { redis { host => "127.0.0.1" type => "apache-access data_type => "list" key => "logstash.apache filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG" geoip { source => "clientip" add_tag => ["geotag"] output { elasticsearch {host => "127.0.0.1 Shipper Shipper Shipper
INTERFACES & API
KIBANA Kibana
KIBANA
ELASTICHQ
KIBANA - DEMO DEMO
INTEGRATION NAGIOS UND ICINGA
REALTIME LOGANALYSE Analyse verschiedener Quellen in Realtime Prüfung auf Patterns und States Facilitites Regex Programs Übermittlung als Passiver Event
ÜBERSICHT LOGSTASH UND ICINGA Indexer Search & Storage Webinterface Icinga - Commandpipe Icinga Web
Broker Indexer Search & Storage Webinterface www.netways.de // blog.netways.de // @netways KONFIGURATION - LOGSTASH INDEXER - ICINGA Konfiguration für Icinga-Alert input { Shipper Shipper Shipper filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE" ] grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2 ] output { elasticsearch {host => "127.0.0.1 nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd"
LOGSTASH ICINGA - DEMO DEMO
ZUGABE
REALTIME GRAPHING
STATSD & GRAPHITE StatsD Netzwerkdaemon auf Basis von UDP Bucket -> Value -> Flush Entkoppelte Zwischenaggretion für Statisik Graphite Graphing-Framework bestehend aus Whisper (Datenbank) Carbon (Engine) Graphite-Web (Interface)
INSTALLATION STATSD - NODEJS apt-get install make python g++ checkinstall mkdir nodejs && cd nodejs wget -N http://nodejs.org/dist/node-latest.tar.gz tar xzvf node-latest.tar.gz && cd `ls -rd node-v*` checkinstall
INSTALLATION STATSD wget https://github.com/etsy/statsd/archive/master.zip unzip master.zip node stats.js config.js
MONITORING - STATSD Status Informationen echo stats nc 127.0.0.1 8126 echo health nc 127.0.0.1 8126 Timer- und Counterinfo echo counters nc 127.0.0.1 8126 echo timers nc 127.0.0.1 8126
INSTALLATION GRAPHITE Download der Sources git clone https://github.com/graphiteproject/graphite-web.git git clone https://github.com/graphiteproject/carbon.git git clone https://github.com/graphiteproject/whisper.git
INSTALLATION GRAPHITE Installation Whisper pushd whisper sudo python setup.py install popd Installation Carbon pushd carbon sudo python setup.py install popd Konfiguration Carbon pushd /opt/graphite/conf cp carbon.conf.example carbon.conf cp storage-schemas.conf.example storage-schemas.conf
INSTALLATION GRAPHITE - WEBAPP Check Dependencies Graphite webapp pushd graphite-web python check-dependencies.py popd Installation Graphite webapp pushd graphite-web python setup.py install popd Konfiguration Apache example-graphite-vhost.conf
ÜBERSICHT STATSD UND GRAPHITE Indexer Search & Storage Webinterface Statsd Graphite
KONFIGURATION - LOGSTASH INDEXER - STATSD Konfiguration für Statsd Shipper Shipper input { redis { host => "127.0.0.1" type => "apache-access data_type => "list" key => "logstash.apache format => "json_event add_field=> ["sitename","www.icinga.org"] filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG" ] output { stdout { debug => true if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename.response.%{response count => ["apache.%{sitename.bytes", "%{bytes"] elasticsearch {host => "127.0.0.1 Shipper Broker Search & Storage Webinterface Indexer StatsD
GRAPHITE - DEMO DEMO
FRAGEN & ANTWORTEN
FRAGEN & ANTWORTEN NETWAYS GmbH Deutschherrnstrasse 15-19 DANKE 90429 Nürnberg Tel: +49 911 92885-0 ICINGA- VORTRAG UM 16.00 UHR Fax: +49 911 92885-77 Email: info@netways.de Website: www.netways.de Twitter: twitter.com/netways Facebook: facebook.com/netways Blog: blog.netways.de