Performance Guideline for syslog-ng Premium Edition 5 LTS

Table of Contents 1. Preface... 3 2. Executive summary... 4 2.1. Optimizing multithreaded performance... 5 2.2. Use cases for performance improvement... 5 2.3. Configuration guidelines... 6 2.4. Configuration guidelines for Windows Agent... 9 3. The test environment... 10 4. Summary... 11 4.1. About BalaBit... 11 2

Preface 1. Preface This document summarizes the findings of several performance tests and measurements carried out on different versions of syslog-ng Premium Edition. The main focus was syslog-ng PE version 5 LTS, but similar performance tests will be performed and published regularly with every major syslog-ng PE release. 3

Executive summary 2. Executive summary The main findings of the performance tests are as follows: The fastest way the syslog-ng PE application can receive log messages from the network is using plain TCP transport with the syslog() source driver. Using the tcp() driver is not significantly slower. Starting with version 4 F1, syslog-ng PE can be run in multithreaded mode to scale to multiple CPUs or cores for increased performance. In case of multiple connections, the performance rates have been tested with 10 and 100 connections, and in both cases the results are the same. Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing: over 185000 messages per second when receiving messages from a single connection and storing them in text files; over 185000 messages per second when receiving messages from a single connection and storing them in logstore files; over 435000 messages per second when receiving messages from multiple connections and storing them in text files; over 350000 messages per second when receiving messages from multiple connections and storing them in logstore files; over 410000 messages per second when receiving messages from multiple secure (TLS-encrypted) connections and storing them in text files. Note By default, syslog-ng PE runs in single-thread mode. Multithreading must be explicitly enabled in the syslog-ng PE configuration file using the threaded(yes) option. The tests were performed using 200-byte log messages and ran for 120 seconds. Note In certain cases, the rate of writing into plain-text files has increased from 180000 messages per second to 500000 messages per second since syslog-ng PE version 4.0. The rate of writing into logstore files has increased in certain cases from 170000 messages per second to 500000 messages per second since syslog-ng PE version 4.0. Using TLS-encrypted transport seriously degrades the rate of receiving messages, down to a rate of 30% of normal performance. PatternDB only works on one thread. Therefore, using PatternDB can significantly decrease performance depending on how many messages match a pattern in the specified ruleset. If a large amount of messages are being received but the specified pattern matches a small number of messages, there is no significant performance degradation. 4

Optimizing multithreaded performance Using disk buffer on the client side can significantly degrade performance. This is especially the case if relays that are heavily loaded are used together with diskbuffer enabled on relays. Performance can also be degraded if lots of messages are received on the syslog-ng server, and are forwarded through the network (for example to a log analyzer or SIEM), and therefore disk buffer is enabled on the server as well. Regarding flow-control, using soft flow-control is faster by 10% than using hard flow-control. Higher stats_level decreases the performance. For example, stats_level(2) means -10% in performance. Simple filtering (for example, filtering on facility or tag) decreases the performance only by a few percent. However, regular expressions significantly decrease the message-processing rate, by about 50-75%, depending on the complexity of the regular expression. From version 4.1.1, syslog-ng PE provides true multi-core support. For the best performance, use multi-core hardware for both the source and the destination side. For details on using multi-threading, see Chapter Multithreading and scaling in syslog-ng PE in The syslog-ng Premium Edition Administrator Guide. 2.1. Optimizing multithreaded performance Destinations that have a queue process that queue in a single thread. Multiple sources can send messages to the same queue, so the queue can scale to multiple CPUs. However, when the writer thread writes the queue contents to the destination, it will be single-threaded. Message parsing, rewrite rules, filters, and other types of message processing is performed by the reader thread in a sequential manner. This means that such operations can scale only if reading messages from the source can be multithreaded. For example, if a tcp source can process messages from different connections (clients) in separate threads. If the source cannot use multiple threads to process the messages, the operations will not scale. To improve the processing power of syslog-ng PE and scale to more processors, use the following methods: To improve scaling on the source side, use more sources, for example, more source files, or receive the messages from more parallel connections. For network sources, you can also configure a part of your clients to send the messages to a different port of your syslog-ng server, and use separate source definitions for each port. On the destination side, when writing the log messages to files, use macros in the filename to split the messages to separate files (for example, using the $HOST macro). Files with macros in their filenames are processed in separate writer threads. On the destination side, when sending messages to a syslog-ng server, you can use multiple connections to the server if you configure the syslog-ng server to receive messages on multiple ports, and configure the clients to use both ports. 2.2. Use cases for performance improvement Improving performance with lots of connections: If there are several thousand active connections simultaneously, it is advised to place relay syslog-ng-s on another computer in front of the syslog-ng. Switching between the active connections is time-consuming, while 5

Configuration guidelines the amount of incoming messages is usually not significant. This problem is solved by using relays, since they are collecting the logs. The syslog-ng solution can handle lots of log messages sent from a few connections easily. Storing lots of log messages: It requires large free disk space if syslog-ng receives lots of messages per second (note that syslog-ng can process messages at even 100 MB/sec speed). In this case it is advised to use compressed logstore instead of file for storing data. The size of a compressed logstore (compress(3)) is only a few percent of the file destination. Obviously, the effectiveness of the compression depends on the pattern of the incoming messages, but since most of the time the log messages consist of simple text messages, they can be compressed quite effectively (around 90% compression rate). Writing data into SQL database: The syslog-ng will only commit after the amount of inserts set in flush-lines(), if flags(explicit-commits) is enabled and the value of flush-lines() is raised. The speed of writing into an SQL database can be 2-3 times faster if flush-lines(100) and flags(explicit-commits) is enabled. Filtering messages: It is advised to use the simplest filters when filtering incoming messages. If a message can be filtered with several types of filters, check the measured data. For example, if a message is filtered with a complicated regexp, the performance of syslog-ng can drop down to 5%, whereas if tag or facility, there is no performance decrease. Identifying bottlenecks: It is advised to first identify the bottlenecks if the performance of syslog-ng seems to be too low. For example, if syslog-ng is writing on slow disks at the destination side, upgrading the source side device will not help increasing the performance. 2.3. Configuration guidelines Log messages can be collected and processed at a faster rate in the latest version of syslog-ng Premium Edition compared to version 4 LTS and earlier versions but several configuration aspects will affect the rate at which log messages are collected and stored. The following tables show the results of tests performed with syslog-ng PE 5 LTS. Taking the following factors into consideration will optimize syslog-ng Premium Edition performance: Number of network connections: In a multi-threaded environment, an increase in connections will a significantly increase the rate at which syslog-ng PE processes log messages. Threaded Number of Connections Messages Per Second 1 115,000 10 485,000 Not Threaded Number of Connections Messages Per Second 1 145,000 10 135,000 6

Configuration guidelines Number of Connections 100 Scalability: Threaded Messages Per Second 420,000 Number of Connections 100 Not Threaded Messages Per Second 115,000 Configuration: path TCP, destination multiple files(using macros), message size: 200 bytes Table 1. Number of network connections To improve processing power, syslog-ng PE uses multi-thread processing to scale to multiple CPU cores. The syslog-ng PE application has a main thread that is always running, and a number of worker threads that process the messages. The maximum number of worker-threads is set in the license file which is necessary to operate syslog-ng Premium Edition in server mode. Messages per second Two Threads 195,000 Four Threads 340,000 Eight Threads 515,000 Configuration: path TCP, destination multiple files(using macros), 10 connections, message size: 200 bytes Encrypted log transfer: Table 2. Scalability The syslog-ng PE application uses the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates. Not Encrypted Messages per second 510,000 Legacy syslog TLS Encryption 405,000 Not Encrypted 485,000 IETF syslog TLS Encryption 410,000 Configuration: path TCP, multi-threaded, 10 connections, destination multiple files (using macros), message size: 200 bytes Type of storage: Table 3. Encrypted log transfer The syslog-ng PE application can store log messages securely in encrypted, compressed and timestamped binary files. While encrypted files are more secure than plain text files, there is an approximately 13% decrease in processing speed. Messages per second Plain Text File 475,000 Encrypted Logstore 350,000 Configuration: path TCP, multi-threaded, 10 connections, destination multiple files (using macros)/logstores, message size: 200 bytes 7

Configuration guidelines Plain Text File Encrypted Logstore Note: Writing into only one logstore cannot be scaled because of technical limitations even when syslog-ng receives messages from more connections. Disk buffer: Table 4. Type of storage The syslog-ng Premium Edition stores messages on the local hard disk if the central log server or the network connection to the server becomes unavailable. Without Buffer Messages per second 340,000 Reliable Disk With Disk Buffer 25,000 Without Buffer 320,000 Normal Disk With Disk Buffer 33,000 Configuration: path TCP, multi-threaded, disk buffer 100Mb, 10 connections, message size: 200 bytes Log pre-processing: Table 5. Disk buffer Depending of the type of pre-processing, the rate at which syslog-ng PE collects messages can vary. Regular Expressions, both simple and complex, as well as pattern recognition processing through PatternDB have a significant impact on the message processing rate. Facility filtering and rewriting result in a modest reduction in the message processing rate. Tag filtering results in no reduction of the message collection rate. No Pre-Processing Complex regexp (for example matching a valid IP address in hostname field) Simple regexp (for example matching a single string) Facility filter PatternDB ( 10 % of messages matched) Simple Rewrite (for example rewrite hostname) Tag Filter Messages Per Second 490,000 50,000 375,000 520,000 95,000 500,000 525,000 Configuration: path TCP, multi-threaded, 10 connections, message size: 200 bytes Application-level acknowledgement via Reliable Log Transfer Protocol (RLTP ): Table 6. Log pre-processing The latest version of syslog-ng Premium Edition supports the RLTP that allows the logserver to notify clients when a message is received. The application level acknowledgment has an effect on the number of log messages processed per second. 8

Configuration guidelines for Windows Agent Messages per second TCP RLTP 485,000 385,000 Configuration: multi-threaded, 10 connections, message size: 200 bytes Note: This result is measured on server side. When RLTP is applied on a relay server both source and destination side apply RLTP which would decrease the performance slightly. Table 7. Application-level acknowledgement via Reliable Log Transfer Protocol (RLTP ) 2.4. Configuration guidelines for Windows Agent The following tables show the results of tests performed with syslog-ng Agent 5 LTS. Taking the following factors into consideration will optimize syslog-ng Agent for Windows performance: Source Eventlog File Messages per second 7,800 10,000 Configuration: path TCP Table 8. Configuration guidelines for Windows Agent based on log sources 9

The test environment 3. The test environment The test environment consisted of a single client and a server hardware, connected via a Gigabit switch. Note that in certain test runs, the client opened several separate connections to the servers to simulate real-life logging environments. The syslog-ng Premium Edition application was installed from the.run package. The client and the server hardware were identical with the following main parameters: 2x Intel Xeon Processor E5620 (12M Cache, 2.40 GHz, 5.86 GT/s Intel QPI, quad-core) 18 GB RAM Western Digital RE3 WD2502ABYS 250GB 7200 RPM 16MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drives The main operating system of the hosts was ubuntu-lucid-x64, but the syslog-ng PE client and server applications were running in chrooted debian-etch-x64 environments. The following settings were used for performance improvement: Improving performance with lots of connections: log_iw_size = number of max_connections() * 1000 log_fifo_size = number of max_connections() * 1000 log_fetch_limit = 1000 flush_lines = 1000 Improving performance with a few connections but high amount of traffic: log_iw_size = number of max_connections() * 10000 log_fifo_size = number of max_connections() * 10000 log_fetch_limit = 1000 flush_lines = 1000 10

Summary 4. Summary The test measurements show that the processing capabilities of syslog-ng Premium Edition have increased significantly from version 4.2, and that syslog-ng PE is capable of receiving and processing high-volume log traffic. 4.1. About BalaBit BalaBit S.a.r.l. is an innovative information security company, a global leader in the development of privileged activity monitoring, trusted logging and proxy-based gateway technologies to help protect customers against internal and external threats and meet security and compliance regulations. As an active member of the open source community, we provide solutions to a uniquely wide range of both open source and proprietary platforms, even for the most complex and heterogeneous IT systems across physical, virtual and cloud environments. BalaBit is also known for its flagship product, the open source log server application syslog-ng. BalaBit, the fastest-growing IT Security company in the Central European region according to Deloitte Technology Fast 50 (2012) list, has local offices in France, Germany, Russia, and in the USA, and cooperates with partners worldwide. Our R&D and global support centers are located in Hungary, Europe. To learn more about commercial and open source BalaBit products, request an evaluation version, or find a reseller, visit the following links: The syslog-ng homepage Contact us and request an evaluation version Find a reseller All questions, comments or inquiries should be directed to <info@balabit.com> or by post to the following address: BalaBit S.a.r.l. 1117 Budapest, Alíz Str. 2 Phone: +36 1 398 6700 Fax: +36 1 208 0875 Web: https:/// Copyright 2015 BalaBit S.a.r.l. All rights reserved. This document is protected by copyright and is distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this document may be reproduced in any form by any means without prior written authorization of BalaBit. The latest version is always available at the BalaBit Documentation Page. 11