Log Management with Open-Source Tools. Risto Vaarandi rvaarandi 4T Y4H00 D0T C0M

Transcription

1 Log Management with Open-Source Tools Risto Vaarandi rvaarandi 4T Y4H00 D0T C0M

2 Outline Why do we need log collection and management? Why use open source tools? Widely used logging protocols and recently introduced new standards Open-source syslog servers Open-source log management tools

3 Why collect logs from your IT system and network? Observation logs contain information which is often not available from other sources Real-time monitoring analyze logs in real-time (or near-real-time) fashion, in order to discover important changes in the state of the IT system Post-factum incident analysis leverage collected data for discovering unknown past incidents and getting detailed insights into them

4 Why use open source tools for log management? Commercial SIEM and log management frameworks: many frameworks are consultant-oriented have complex design and insufficient documentation prohibitive deployment and licensing costs many frameworks repeat a number of design mistakes of network management solutions (made almost two decades ago!) Past experience with network management solutions: Phase 1: initial marketing hype, followed by a number of success stories in the context of large and wealthy institutions Phase2: disappointment among many potential customers (failed deployments, prohibitive pricing, etc.) and search for alternatives Phase3: appearance of well-designed open-source solutions which become widely used and acknowledged, especially by small- and midsize enterprises

5 Traditional log collection protocols The scene of log collection protocols was relatively stable for two decades BSD syslog the only cross-vendor protocol designed specifically for logging UDP based plaintext, thus resource-efficient, but unreliable and not secure Simple message layout in the UDP frame priority, simple timestamp, host name, program name, unstructured message text

6 New log collection protocols IETF syslog (2009) support for including structured data in messages, UDP and TCP based transport, encryption and authentication, detailed timestamps CEE (Common Event Expression) logging standard (2012) use JSON format inside originally unstructured BSD/IETF syslog message fields Other protocols non-rfc flavors of BSD and IETF syslog (e.g., BSD syslog over TCP), GELF, SNMP trap messages, etc.

7 Examples # Traditional BSD syslog priority value 28 encapsulates facility value 3 # (daemon) and severity value 4 (warning): 3*8 + 4 = 28 <28>Nov 17 12:33:59 myhost2 ids[1299]: port scan from # IETF syslog note high granularity timestamps with timezone information # and two blocks of structured data <28> T12:33: :00 myhost2 ids [timequality tzknown="1" issynced="1"][origin ip=" "] port scan from # CEE message format use standard BSD syslog message for transporting # structured data in JSON format <28>Nov 17 12:33:59 myhost2 scan from ", "originip":" ","action":"portscan","src":" "}

8 Why pass structured data in log messages? Unstructured message fields often contain additional information about event which needs to be highlighted It is much easier to parse structured data (keywordvalue pairs) than unstructured free-format strings Some structured data can be used without extra parsing JSON format is supported by several log management frameworks and databases (e.g., Elasticsearch)

9 Log collection on Linux platform db GUI local programs openlog(3) syslog(3)... /dev/log network port incoming messages kernel /proc/kmsg syslog server configuration to local logfiles /var/log/... /etc/syslog-server.conf messages to remote syslog servers messages from other nodes 9

10 Syslog servers rsyslog + fast message processing, efficient multithreading, designed to handle at least K messages per second (see the paper Rsyslog: going up from 40K messages per second to 250K by Rainer Gerhards from Linux Kongress 2010) + backwards compatible with UNIX syslogd configuration directives + has a number of unique features and advantages over competitors (disk based buffers, support for Elasticsearch database, etc.) - documentation could be better - configuration language has a non-intuitive syntax - filtering conditions can not be named which prevents their reuse

11 Syslog servers syslog-ng + a flexible and readable configuration language which allows for specifying complex configurations + single-threaded until the 3.2 version, but multi-threading has been introduced into recent versions which considerably improves scalability and performance + well documented - open-source edition does not support disk based buffers - no support for Elasticsearch (although could be configured through a self-developed output plugin)

12 Syslog servers nxlog + native support for Windows platform and Windows Event Log + supports the use of embedded Perl constructs for message processing + supports a number of input and output types not supported by competitors (e.g., accepting input events from SQL databases, producing output events in GELF format, etc.) - poor message filtering performance

13 Elasticsearch DB for log management Apache Lucene based nosql database technology that is frequently used for storing log data native support for distributed operations and building clusters allows for splitting indexes into parts (shards) and distributing shards over several nodes (e.g., split an index into 2 shards and distribute them over 2 nodes, turning disks at individual nodes into a single logical storage space) indexes can be configured to have one or more replicas which increases fault tolerance (e.g., split an index into 2 shards and configure the index to have 1 replica, and distribute resulting 4 shards across 4 nodes) builtin support for data compression (important when storing large volumes of log data) supported by several log management tools (Kibana, Graylog2, logstash, rsyslog)

14 Log management tools Kibana Kibana is a GUI for searching log data stored into Elasticsearch DB Kibana is designed to work with logstash log preprocessing tool, but can accept data from any other tool which is able to store it to Elasticsearch in a recognizable way (e.g., rsyslog) Kibana is lightweight, written in Ruby, accessible over HTTP, and contains only searching and reporting functionality (e.g., user authentication and SSL connectivity has to be accomplished with external tools like Apache reverse proxy) When building a Kibana based log management solution, you are creating the system from well-documented and well-established building blocks, and thus having the opportunity for many customizations during initial installation and later maintenance

15 Kibana web interface

16 Log management tools Graylog2 A full log management solution consisting of a server for log message reception (syslog, GELF) and a GUI The GUI is user-friendly with builtin help, and is intuitive to use Many configuration tasks (such as setting log data retention intervals, etc.) can be accomplished through a web interface Graylog2 supports users with different roles and password authentication Earlier versions of Graylog2 employed single-server approach which limited the system scalability, while most recent versions allow to run several servers in parallel

17 Graylog2 web interface

18 Other log management tools Logstash ( - has a web interface for searching logs stored to Elasticsearch database, but since it supports large number of input and output types, it is mostly used as a log parsing and conversion tool ELSA ( - a log management system which is built on top of syslog-ng, MySQL and Sphinx

19 Netflow protocol Proposed by Cisco in 1990s, nowadays supported by many major vendors A Netflow-enabled network device (e.g., router, switch, dedicated probe) collects network traffic statistics and exports it to collector over UDP Traffic statistics consists of flow records, where each record describes some network flow Network flow unidirectional sequence of packets which share transport protocol, source and destination IP, source and destination port, and few other parameters (e.g., type of service)

20 Example of collected Netflow data The following two records represent a successfully negotiated and completed TCP connection from client port to the HTTP service (port 80) running at the server : Start = :04: Duration = TCP : > :80 TCPflags =.AP.SF Packets = 5 Bytes = 513 Start = :04: Duration = TCP :80 -> :48896 TCPflags =.AP.SF Packets = 4 Bytes = 375

21 How to collect/use netflow data Enable Netflow collection at your network device or use dedicated probes (e.g., fprobe) Open-source software packages for collecting Netflow NfSen ( SiLK ( Flow-tools ( - unmaintained What you might be interested in finding in Netflow data Flows with unusual combinations of TCP flags (e.g., FIN without ACK) Flows which represent connections to/from known bad IP addresses Unexpected spikes in traffic volumes (measured in number of bytes, packets, flows) associated with certain sources (e.g., foreign IP addresses or bad IP addresses)