Log Management with Open-Source Tools Risto Vaarandi SEB Estonia
Outline Why use open source tools for log management? Widely used logging protocols and recently introduced new standards Open-source syslog servers Open-source log management tools 2
Why use open source tools for log management? Commercial SIEM and log management frameworks: many frameworks are consultant-oriented have complex design and insufficient documentation prohibitive deployment and licensing costs some frameworks repeat a number of design mistakes of network management solutions (made almost two decades ago!) Past experience with network management solutions: Phase 1: initial marketing hype, followed by a number of success stories in the context of large and wealthy institutions Phase2: disappointment among many potential customers (failed deployments, prohibitive pricing, etc.) and search for alternatives Phase3: appearance of well-designed open-source solutions which become widely used and acknowledged, especially by small- and mid-size enterprises 3
Traditional log collection protocols The scene of log collection protocols was relatively stable for two decades (from 1990s until 2009) BSD syslog the only cross-vendor protocol designed specifically for logging UDP based plaintext, thus resource-efficient, but unreliable and not secure Simple message layout in the UDP frame priority, simple timestamp, host name, program name, unstructured message text 4
New log collection protocols IETF syslog (2009) support for including structured data in messages, UDP and TCP based transport, encryption and authentication, detailed timestamps CEE (Common Event Expression) logging standard proposal by MITRE (2013) use JSON format inside originally unstructured BSD/IETF syslog message fields. Note that CEE is no longer developed by MITRE. Other protocols various flavors of BSD and IETF syslog (e.g., BSD syslog over TCP), GELF, SNMP trap messages, etc. 5
Examples # Traditional BSD syslog priority value 28 encapsulates facility value 3 # (daemon) and severity value 4 (warning): 3*8 + 4 = 28 <28>Nov 17 12:33:59 myhost2 ids[1299]: port scan from 192.168.1.102 # IETF syslog note high granularity timestamps and two blocks of structured data <28>1 2012-11-17T12:33:59.223+02:00 myhost2 ids 1299 - [timequality tzknown="1" issynced="1"][origin ip="10.1.1.2"] port scan from 192.168.1.102 # CEE message format use standard BSD syslog message for transporting # structured data in JSON format <28>Nov 17 12:33:59 myhost2 ids[1299]: @cee:{"pname":"ids","pid":1299,"msg":"port scan from 192.168.1.102", "originip":"10.1.1.2","action":"portscan","src":"192.168.1.102"} 6
Why pass structured data in log messages? Unstructured message fields often contain additional information about event which needs to be highlighted It is much easier to parse structured data (keyword-value pairs) than unstructured free-format strings Structured data can be used without extra parsing JSON format is supported by several log management frameworks and databases (e.g., Elasticsearch) 7
Log collection on Linux platform db GUI openlog(3) syslog(3)... local programs /dev/log network port incoming messages kernel /proc/kmsg syslog server configuration to local logfiles /var/log/... /etc/syslog-server.conf messages to remote syslog servers messages from other nodes 8
Syslog servers rsyslog http://www.rsyslog.com + fast message processing, efficient multithreading, designed to handle at least 150-200K messages per second (see the paper Rsyslog: going up from 40K messages per second to 250K by Rainer Gerhards from Linux Kongress 2010) + backwards compatible with UNIX syslogd configuration directives + has a number of unique features and advantages over competitors (disk based buffers, support for Elasticsearch database, Guardtime keyless signatures, etc.) + default logging system for most Linux distributions - configuration language supports several syntax flavors which might be sometimes confusing for inexperienced users 9
Syslog servers syslog-ng http://www.balabit.com/network-security/syslog-ng/ + a flexible and readable configuration language which allows for specifying complex configurations + single-threaded until the 3.2 version, but multi-threading has been introduced into recent versions which considerably improves scalability and performance + well documented - open-source edition does not support disk based buffers - no support for Elasticsearch (although could be configured through a self-developed output plugin) 10
Syslog servers nxlog http://nxlog-ce.sourceforge.net/ + native support for Windows platform and Windows Event Log + supports the use of embedded Perl constructs for message processing + supports a number of input and output types not supported by competitors (e.g., accepting input events from SQL databases, producing output events in GELF format, etc.) - poor message filtering performance 11
Elasticsearch DB for log management http://www.elasticsearch.org/ Apache Lucene based nosql database technology that is frequently used for storing log data native support for distributed operations and building clusters allows for splitting indexes into parts (shards) and distributing shards over several nodes (e.g., split an index into 2 shards and distribute them over 2 nodes, turning disks at individual nodes into a single logical storage space) indexes can be configured to have one or more replicas which increases fault tolerance (e.g., split an index into 2 shards and configure the index to have 1 replica, and distribute resulting 4 shards across 4 nodes) builtin support for data compression (important when storing large volumes of log data) supported by several log management tools (Kibana, Graylog2, logstash, rsyslog) 13
Log management tools Kibana http://www.elasticsearch.org/overview/kibana/ Kibana is a GUI for searching log data stored into Elasticsearch DB Kibana is designed to work with logstash log preprocessing tool, but can accept data from any other tool which is able to store events to Elasticsearch in a recognizable way (e.g., rsyslog) Kibana is lightweight, implemented as Javascript which runs in the client web browser and talks directly to Elasticsearch, and contains only searching and reporting functionality (e.g., user authentication and SSL connectivity has to be accomplished with external tools like Apache reverse proxy) When building a Kibana based log management solution, you are creating the system from well-documented and well-established building blocks, and thus having the opportunity for many customizations during initial installation and later maintenance 14
Kibana web interface 15
Log management tools Graylog2 http://graylog2.org/ A full log management solution consisting of a server for log message reception (syslog, GELF, etc.) and a GUI Many configuration and admin tasks (setting up event inputs, checking the status of Elasticsearch DB, interactive creation and testing of message parsers, etc.) can be accomplished through a web interface in a user-friendly way Graylog2 supports users with different roles and password authentication Earlier versions of Graylog2 employed single-server approach which limited the system scalability, while most recent versions allow to run several servers in parallel 16
Graylog2 web interface 17
Other log management tools Logstash (http://logstash.net/) - flexible log parsing and conversion tool, supports flexible Grok parsing language. Supports large number of input and output protocols, often used for receiving raw log data and storing it to Elasticsearch ELSA (http://code.google.com/p/enterprise-log-search-and-archive/) - a log management system which is built on top of syslog-ng, MySQL and Sphinx 18
Writing to Elasticsearch rsyslog or logstash? Logstash parsing is more powerful and flexible, can be done with Grok and regular expressions. Also, logstash supports adding GeoIP information to parsed events. Rsyslog parsing is much faster, but less flexible than with logstash Use both in parallel 19
Netflow protocol Proposed by Cisco in 1990s, nowadays supported by many major vendors A Netflow-enabled network device (e.g., router, switch, dedicated probe) collects network traffic statistics and exports it to collector over UDP Traffic statistics consists of flow records, where each record describes some network flow Network flow unidirectional sequence of packets which share transport protocol, source and destination IP, source and destination port, and few other parameters (e.g., type of service) 20
Example of collected Netflow data The following two records represent a successfully negotiated and completed TCP connection from client 10.3.1.1 port 48896 to the HTTP service (port 80) running at the server 10.2.1.1: Start = 2013-02-18 00:04:05.733 Duration = 0.014 TCP 10.3.1.1:48896 -> 10.2.1.1:80 TCPflags =.AP.SF Packets = 5 Bytes = 513 Start = 2013-02-18 00:04:05.734 Duration = 0.010 TCP 10.2.1.1:80 -> 10.3.1.1:48896 TCPflags =.AP.SF Packets = 4 Bytes = 375 21
How to collect and use netflow data Open-source software packages for collecting Netflow NfSen (http://nfsen.sourceforge.net/) SiLK (http://tools.netsa.cert.org/silk/) Flow-tools (http://www.splintered.net/sw/flow-tools/) unmaintained Logstash can receive Netflow and store it to Elasticsearch! What you might be interested in finding in Netflow data Flows with unusual combinations of TCP flags (e.g., FIN without ACK) Flows which represent connections to/from known bad IP addresses Unexpected spikes in traffic volumes (measured in number of bytes, packets, flows) associated with certain sources (e.g., foreign or bad IP addresses) 22
Thank you! 23