Monitoring Linux and Windows Logs with Graylog Collector Bernd Ahlers Graylog, Inc.
Structured Logging & Introduction to Graylog Collector Bernd Ahlers Graylog, Inc.
Introduction: Graylog Open source log management platform Collect, index and analyze structured and unstructured log data Alerts based on log data Extensible via custom plugins
More about Graylog www.graylog.org marketplace.graylog.org docs.graylog.org github.com/graylog2
Why are we writing logs? Getting insight & collecting business metrics Debugging problems Building an audit trail Monitoring
How do we access our logs? Applications write to local files SSH into machines tail, grep, awk If lucky: central log management
What do they look like? Syslog RFC 3164 (BSD) Syslog RFC 5424
Syslog RFC 3164 (BSD) Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Syslog RFC 5424 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [examplesdid@32473 iut="3" eventsource="application" eventid="1011"] BOMAn application event log entry...
Apache 127.0.0.1 - bernd [28/Dec/2014:06:43:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910 "-" "Mozilla/5.0 (Linux) mirall/1.7.1"
Postfix Aug 5 17:05:26 hostname postfix/qmgr[308]: A44F828C71: from=<bamm@example.com>, size=153136, nrcpt=1 (queue active)
Squid sq18.wikimedia.org 1715898 2010-1201T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200 13208 GET http://en.wikipedia.org/wiki/main_page NONE/text/html - - Mozilla/4.0%20(compatible;%20MSIE %206.0;%20Windows%20NT%205.1;%20.NET%20CLR %201.1.4322) en-us -
log4j 0 36 51 [main] INFO MyApp - Entering application. [main] DEBUG com.foo.bar - Did it again! [main] INFO MyApp - Exiting application.
Ruby Logger I, [2015-11-18T00:16:27.723972 #3609] Hello world! INFO -- :
#1 Problem: Timestamps Everyone likes to invent one Missing most of the time: timezone, year
How to get value out of unstructured logs? Regex More regex Even more regex
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4} :)) (([0-9A-Fa-f]{1,4}:){6}(: [0-9A-Fa-f]{1,4} ((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3}) :)) (([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}) {1,2}) :((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [19]?\d)){3}) :)) (([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3}) ((: [0-9A-Fa-f]{1,4})?:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[04]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f] {1,4}){1,4}) ((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5] 2[0-4]\d 1\d\d [19]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f] {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5}) ((:[0-9A-Fa-f]{1,4}){0,3}:((25[05] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)) {3})) :)) (([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6}) ((:[0-9AFa-f]{1,4}){0,4}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (:(((:[0-9A-Fa-f]{1,4}){1,7}) ((:[0-9A-Fa-f] {1,4}){0,5}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)))(%.+)?
Grok IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4} :)) (([0-9... USERNAME [a-za-z0-9._-]+ USER %{USERNAME} HOSTNAME \b(?:[0-9a-za-z][0-9a-za-z-]{0,62})(?:\.(?:[0-9a-za-z][0-9aza-z-]{0,62}))*(\.? \b) EMAILLOCALPART [a-za-z][a-za-z0-9_.+-=:]+ EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}... COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
Graylog: Extractors Regular expressions based Extracts data into message fields
How to fix this? Central log collection (Graylog, ELK, others) Use structured log formats Structured Syslog RFC 5424 CEF Format GELF JSON
Structured Syslog RFC 5424 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [examplesdid@32473 iut="3" eventsource="application" eventid="1011"] BOMAn application event log entry...
CEF by ArcSight/HP Sep 19 08:26:10 host CEF:0 HP siem 1.0 100 service successfully stopped 10 src=10.0.0.1 dst=2.1.2.2 spt=1232
GELF { "version": "1.1", "timestamp": 1385053862.3072, "host": "example.org", "short_message": "A short message", "full_message": "Backtrace here\n\nmore stuff", "level": 1, "_user_id": 9001, "_some_info": "foo", "_some_env_var": "bar"}
JSON { "source": "example.org", "message": "A log message", "timestamp": "2015-11-15T10:43:21Z", "user_id": 9001, "http_method": "GET"}
How we try to improve the ecosystem Icinga2 GELF output for events Docker GELF logging driver (since Docker 1.8) apache-mod_log_gelf (beta) log4j2-gelf gelfclient Java library svloggelfd (log forwarding for runit)
We at Graylog <3 structured data and you should too!
Introduction: Graylog Collector Reads local log files and ships them to Graylog Windows EventLog support (limited for now) Transport encryption via TLS Runs on Linux, Windows, Mac OS X and AIX
Why another Collector? There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng We want integration and centralized management of collectors in Graylog
Collector Installation OS packages for Linux distributions Manual installation on Windows via ZIP file (MSI upcoming) Runs as Windows service
Collector Configuration server-url = "http://your-graylog-server:12900" inputs { windows-application-log { type = "windows-eventlog" source-name = "Application" } } outputs { gelf-tcp { type = "gelf" host = "your-graylog-server" port = 12201 } }
Collector: Current State Windows EventLog support needs update to support new Windows APIs File reading needs improvement Centralized management needs to be implemented :-(
Tomorrow: Hackathon
Thank you! Thank you for your time!
QA Ask me anything! Bernd Ahlers / Graylog, Inc. @berndahlers www.graylog.org github.com/graylog2