Monitoring Linux and Windows Logs with Graylog Collector. Bernd Ahlers Graylog, Inc.

Transcription

1 Monitoring Linux and Windows Logs with Graylog Collector Bernd Ahlers Graylog, Inc.

2 Structured Logging & Introduction to Graylog Collector Bernd Ahlers Graylog, Inc.

3 Introduction: Graylog Open source log management platform Collect, index and analyze structured and unstructured log data Alerts based on log data Extensible via custom plugins

4

5

6

7

8

9

10

11

12 More about Graylog marketplace.graylog.org docs.graylog.org github.com/graylog2

13 Why are we writing logs? Getting insight & collecting business metrics Debugging problems Building an audit trail Monitoring

14 How do we access our logs? Applications write to local files SSH into machines tail, grep, awk If lucky: central log management

15 What do they look like? Syslog RFC 3164 (BSD) Syslog RFC 5424

16 Syslog RFC 3164 (BSD) Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

17 Syslog RFC T22:14:15.003Z mymachine.example.com evntslog - ID47 [examplesdid@32473 iut="3" eventsource="application" eventid="1011"] BOMAn application event log entry...

18 Apache bernd [28/Dec/2014:06:43: ] "PROPFIND /remote.php/webdav/ HTTP/1.1" "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

19 Postfix Aug 5 17:05:26 hostname postfix/qmgr[308]: A44F828C71: from=<[email protected]>, size=153136, nrcpt=1 (queue active)

20 Squid sq18.wikimedia.org T21:57: TCP_MEM_HIT/ GET NONE/text/html - - Mozilla/4.0%20(compatible;%20MSIE %206.0;%20Windows%20NT%205.1;%20.NET%20CLR % ) en-us -

21 log4j [main] INFO MyApp - Entering application. [main] DEBUG com.foo.bar - Did it again! [main] INFO MyApp - Exiting application.

22 Ruby Logger I, [ T00:16: #3609] Hello world! INFO -- :

23 #1 Problem: Timestamps Everyone likes to invent one Missing most of the time: timezone, year

24 How to get value out of unstructured logs? Regex More regex Even more regex

25 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4} :)) (([0-9A-Fa-f]{1,4}:){6}(: [0-9A-Fa-f]{1,4} ((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3}) :)) (([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}) {1,2}) :((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [19]?\d)){3}) :)) (([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3}) ((: [0-9A-Fa-f]{1,4})?:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[04]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f] {1,4}){1,4}) ((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5] 2[0-4]\d 1\d\d [19]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f] {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5}) ((:[0-9A-Fa-f]{1,4}){0,3}:((25[05] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)) {3})) :)) (([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6}) ((:[0-9AFa-f]{1,4}){0,4}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (:(((:[0-9A-Fa-f]{1,4}){1,7}) ((:[0-9A-Fa-f] {1,4}){0,5}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(\.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)))(%.+)?

26 Grok IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4} :)) (([ USERNAME [a-za-z0-9._-]+ USER %{USERNAME} HOSTNAME \b(?:[0-9a-za-z][0-9a-za-z-]{0,62})(?:\.(?:[0-9a-za-z][0-9aza-z-]{0,62}))*(\.? \b) LOCALPART [a-za-z][a-za-z0-9_.+-=:]+ ADDRESS COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

27 Graylog: Extractors Regular expressions based Extracts data into message fields

28

29 How to fix this? Central log collection (Graylog, ELK, others) Use structured log formats Structured Syslog RFC 5424 CEF Format GELF JSON

30 Structured Syslog RFC T22:14:15.003Z mymachine.example.com evntslog - ID47 [examplesdid@32473 iut="3" eventsource="application" eventid="1011"] BOMAn application event log entry...

31 CEF by ArcSight/HP Sep 19 08:26:10 host CEF:0 HP siem service successfully stopped 10 src= dst= spt=1232

32 GELF { "version": "1.1", "timestamp": , "host": "example.org", "short_message": "A short message", "full_message": "Backtrace here\n\nmore stuff", "level": 1, "_user_id": 9001, "_some_info": "foo", "_some_env_var": "bar"}

33 JSON { "source": "example.org", "message": "A log message", "timestamp": " T10:43:21Z", "user_id": 9001, "http_method": "GET"}

34 How we try to improve the ecosystem Icinga2 GELF output for events Docker GELF logging driver (since Docker 1.8) apache-mod_log_gelf (beta) log4j2-gelf gelfclient Java library svloggelfd (log forwarding for runit)

35 We at Graylog <3 structured data and you should too!

36 Introduction: Graylog Collector Reads local log files and ships them to Graylog Windows EventLog support (limited for now) Transport encryption via TLS Runs on Linux, Windows, Mac OS X and AIX

37 Why another Collector? There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng We want integration and centralized management of collectors in Graylog

38

39 Collector Installation OS packages for Linux distributions Manual installation on Windows via ZIP file (MSI upcoming) Runs as Windows service

40 Collector Configuration server-url = " inputs { windows-application-log { type = "windows-eventlog" source-name = "Application" } } outputs { gelf-tcp { type = "gelf" host = "your-graylog-server" port = } }

41 Collector: Current State Windows EventLog support needs update to support new Windows APIs File reading needs improvement Centralized management needs to be implemented :-(

42 Tomorrow: Hackathon

43 Thank you! Thank you for your time!

44 QA Ask me anything! Bernd Ahlers / Graylog, github.com/graylog2