Performance measurements of syslog-ng Premium Edition 4 F1

Transcription

1 Performance measurements of syslog-ng Premium Edition 4 F1 October 13, 2011 Abstract Performance analysis of syslog-ng Premium Edition Copyright BalaBit IT Security Ltd.

2 Table of Contents 1. Preface Executive summary Optimizing multithreaded performance Use cases for performance improvement The test environment Interpreting the measurement results Summary About BalaBit Appendix 1. Detailed results

3 Preface 1. Preface This document summarizes the findings of several performance tests and measurements carried out on different versions of syslog-ng Premium Edition. The main focus was syslog-ng PE version 4 F1, but similar performance tests will be performed and published regularly with every major syslog-ng PE release. 3

4 Executive summary 2. Executive summary The main findings of the performance tests are as follows: The fastest way the syslog-ng PE application can receive log messages from the network is using plain TCP transport with the syslog() source driver. Using the tcp() driver is not significantly slower. Starting with version, syslog-ng PE can be run in multithreaded mode to scale to multiple CPUs or cores for increased performance. In case of multiple connections, the performance rates have been tested with 10 and 100 connections, and in both cases the results are the same. Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing: over messages per second when receiving messages from a single connection and storing them in text files; over messages per second when receiving messages from a single connection and storing them in logstore files; over messages per second when receiving messages from multiple connections and storing them in text files; over messages per second when receiving messages from multiple connections and storing them in logstore files; over messages per second when receiving messages from secure (TLS-encrypted) connections and storing them in text files. Note By default, syslog-ng PE runs in single-thread mode. Multithreading must be explicitly enabled in the syslog-ng PE configuration file using the threaded(yes) option. The tests were performed using 200-byte log messages and ran for 120 seconds. Note In certain cases, the rate of writing into plain-text files has increased from messages per second to messages per second since syslog-ng PE version 4.0. The rate of writing into logstore files has increased in certain cases from messages per second to messages per second since syslog-ng PE version 4.0. Using TLS-encrypted transport seriously degrades the rate of receiving messages, down to a rate of 30% of normal performance. PatternDB only works on one thread. Therefore, using PattnerDB can significantly decrease performance. If you are receiving lots of messages, but not all of them are processed with PatternDB, there is no significant performance degradation. If you are receiving messages through one connection only, performance will be better without multithreading (threaded(no)). 4

5 Optimizing multithreaded performance Using disk buffer on the client side can significantly degrade performance. This is especially the case if relays that are heavily loaded are used together with diskbuffer enabled on relays. Performance can also be degraded if lots of messages are received on the syslog-ng server, and are forwarded through the network (for example to a log analyzer or SIEM), and therefore disk buffer is enabled on the server as well. Regarding flow-control, using soft flow-control is faster by 10% than using hard flow-control. Higher stats_level decreases the performance. For example, stats_level(2) means -10% in performance. Simple filtering (for example, filtering on facility or tag) decreases the performance only by a few percent. However, regular expressions significantly decrease the message-processing rate, by about 50-75%, depending on the complexity of the regular expression. Version of syslog-ng PE provides true multi-core support. For the best performance, use multicore hardware for both the source and the destination side. For details on using multi-threading, see Chapter Multithreading and scaling in syslog-ng PE in The syslog-ng Premium Edition Administrator Guide Optimizing multithreaded performance Destinations that have a queue process that queue in a single thread. Multiple sources can send messages to the same queue, so the queue can scale to multiple CPUs. However, when the writer thread writes the queue contents to the destination, it will be single-threaded. Message parsing, rewrite rules, filters, and other types of message processing is performed by the reader thread in a sequential manner. This means that such operations can scale only if reading messages from the source can be multithreaded. For example, if a tcp source can process messages from different connections (clients) in separate threads. If the source cannot use multiple threads to process the messages, the operations will not scale. To improve the processing power of syslog-ng PE and scale to more processors, use the following methods: To improve scaling on the source side, use more sources, for example, more source files, or receive the messages from more parallel connections. For network sources, you can also configure a part of your clients to send the messages to a different port of your syslog-ng server, and use separate source definitions for each port. On the destination side, when writing the log messages to files, use macros in the filename to split the messages to separate files (for example, using the $HOST macro). Files with macros in their filenames are processed in separate writer threads. On the destination side, when sending messages to a syslog-ng server, you can use multiple connections to the server if you configure the syslog-ng server to receive messages on multiple ports, and configure the clients to use both ports Use cases for performance improvement Improving performance with lots of connections. If there are several thousand active connections simultaneously, it is advised to place relay syslog-ng-s on another computer in front of the syslog-ng. Switching between the active connections is time-consuming, while the amount of incoming messages is usually not significant. This problem is solved by using relays, since they are collecting the logs. The syslog-ng solution can handle lots of log messages sent from a few connections easily. 5

6 Use cases for performance improvement Storing lots of log messages. It requires large free disk space if syslog-ng receives lots of messages per second (note that syslog-ng can process messages at even 100 MB/sec speed). In this case it is advised to use compressed logstore instead of file for storing data. The size of a compressed logstore (compress(3)) is only a few percent of the file destination. Obviously, the effectiveness of the compression depends on the pattern of the incoming messages, but since most of the time the log messages consist of simple text messages, they can be compressed quite effectively (around 90% compression rate). Writing data into SQL database. The syslog-ng will only commit after the amount of inserts set in flush-lines(), if flags(explicit-commints) is enabled and the value of flush-lines() is raised. The speed of writing into an SQL database can be 2-3 times faster if flush-lines(100) and flags(explicit-commints) is enabled. Filtering messages. It is advised to use the simplest filters when filtering incoming messages. If a message can be filtered with several types of filters, check the measured data. For example, if a message is filtered with a complicated regexp, the performance of syslog-ng can drop down to 5%, whereas if tag or facility, there is no performance decrease. Identifying bottlenecks. It is advised to first identify the bottlenecks if the performance of syslog-ng seems to be too low. For example, if syslog-ng is writing on slow disks at the destination side, upgrading the source side device will not help increasing the performance. 6

7 The test environment 3. The test environment The test environment consisted of a single client and a server hardware, connected via a Gigabit switch. Note that in certain test runs, the client opened several separate connections to the servers to simulate real-life logging environments. The syslog-ng Premium Edition application was installed from the.run package. The client and the server hardware were identical with the following main parameters: 2x Intel Xeon Processor E5620 (12M Cache, 2.40 GHz, 5.86 GT/s Intel QPI, quad-core) 18 GB RAM Western Digital RE3 WD2502ABYS 250GB 7200 RPM 16MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drives The main operating system of the hosts was ubuntu-lucid-x64, but the syslog-ng PE client and server applications were running in chrooted debian-etch-x64 environments. 7

8 Interpreting the measurement results 4. Interpreting the measurement results Several different measurements were performed in various syslog-ng configurations and settings. These are listed in Appendix 1, Detailed results (p. 11) in the following format. -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 3.2G The results can be interpreted as follows: The configuration parameters of the syslog-ng PE server are listed as a string of name-value pairs. Whitespaces in the name are replaced with underscores. For example -log_destination_file_macro-threaded_no-active-idle_ log_source_legacy_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no: hw_config: The identifier of the hardware configuration used in the tests. For details, see Section 3, The test environment (p. 7). platform: The operating system and architecture used in the test, for example, debian-etch_amd64. log_source: The type of source driver used in the test. The legacy name refers to syslog-ng drivers using the BSD-syslog standard (RFC3164), while syslog to the IETF-syslog standards (RFC5424). log_size: The length of the log messages in bytes. flush_lines: The value of the flush_lines() parameter. modifiers: Modifications (for example, rewriting or filtering) used in the log path (if any). file/logstore: All incoming messages are written into the same file or logstore. file/logstore_macro: All messages from each connection are written into a separate file or logstore. threaded: Whether multi-threading is used or not. active-idle: The number of active and idle connections between the client and the server. Log messages were transferred only in the active connections. fetch_limit-iw_size: The values of the fetch_limit() and iw_size() parameters, respectively. log_destination: The type of destination driver used in the test. runtime: The length of the test run (in seconds). journal_block_size: The value of the journal_block_size parameter. 8

9 Interpreting the measurement results max_connections: The value of the max_connections parameter. Test results measured on the syslog-ng PE client, for example average rate = msg/sec, count= , time= , (last) msg size=200, bandwidth= kb/sec: Average rate: The rate of messages the client sent the log messages to the server (messages per second). Count: The total number of log messages sent by the clients during the test run. Time: The length of the test run (in seconds). Msg size: The size of the log messages sent by the client (in bytes). Bandwidth: The total bandwith used by the client connections (in kilobytes per seconds). 9

10 Summary 5. Summary The test measurements show that the processing capabilities of syslog-ng Premium Edition have increased significantly in version 4.1, and that syslog-ng PE is capable of receiving and processing high-volume log traffic About BalaBit BalaBit IT Security Ltd. is a developer of network security solutions satisfying the highest standards. BalaBit was founded and is currently owned by Hungarian individuals. Its main products are the syslog-ng system logging software, which is the most widely used alternative syslog solution of the world; the syslog-ng Store Box logserver appliance; Zorp, a modular proxy gateway capable of inspecting over twenty protocols, including encrypted ones like SSL and SSH, and the Shell Control Box, an appliance that can transparently control, audit, and replay SSH, RDP, VNC, Citrix, and Telnet traffic. To learn more about commercial and open source BalaBit products, request an evaluation version, or find a reseller, visit the following links: The syslog-ng homepage Shell Control Box homepage syslog-ng Store Box (SSB) homepage Product manuals, guides, and other documentation Register and request an evaluation version Find a reseller All questions, comments or inquiries should be directed to <[email protected]> or by post to the following address: BalaBit IT Security 1115 Budapest, Bártfai str. 54 Phone: Fax: Web: Copyright 2011 BalaBit IT Security Ltd. Some rights reserved. This document is published under the Creative Commons Attribution Noncommercial No Derivative Works (byncnd) 3.0 license. All other product names mentioned herein are the trademarks of their respective owners. The latest version is always available at the BalaBit Documentation Page. 10

11 Appendix 1. Detailed results The following measurement results were obtained using syslog-ng PE in the environment described in Section 3, The test environment (p. 7). -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 3.2G -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 926M -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec

12 Size of stored outputfiles: 1.7G -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.1G -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 888M -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec 12

13 Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 958M -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 2.3G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.3G 13

14 -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 892M -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 3.2G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 990M 14

15 -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 2.9G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 853M -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no 15

16 average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 11G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 2.7G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 1.7G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 16

17 0 0 0 Size of stored outputfiles: 11G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.7G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 13G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec 17

18 Size of stored outputfiles: 4.1G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 2.6G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 12G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 18

19 Size of stored outputfiles: 3.8G -log_destination_file_macro-threaded_yes-active-idle_1-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 2.5G -log_destination_file_macro-threaded_yes-active-idle_1-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 890M -log_destination_file_macro-threaded_yes-active-idle_1-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec 19

20 Size of stored outputfiles: 2.0G -log_destination_file_macro-threaded_yes-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 2.5G -log_destination_file_macro-threaded_yes-active-idle_1-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 828M -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec

21 0 Size of stored outputfiles: 3.0G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.3G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec

22 Size of stored outputfiles: 9.8G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 12G -log_destination_file_macro-threaded_yes-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 2.6G -log_destination_file-threaded_no-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec

23 Size of stored outputfiles: 4.0G -log_destination_file-threaded_no-active-idle_10-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 4.1G -log_destination_file-threaded_no-active-idle_1-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 4.3G -log_destination_file-threaded_yes-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec

24 Size of stored outputfiles: 8.6G -log_destination_file-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 9.6G -log_destination_file-threaded_yes-active-idle_1-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.1G -log_destination_logstore_macro-threaded_no-active-idle_100-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 0 0 Used compress : 3 Used journal_block_size :

25 Size of stored outputfiles: 54M -log_destination_logstore_macro-threaded_no-active-idle_10-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size : Size of stored outputfiles: 55M -log_destination_logstore_macro-threaded_no-active-idle_1-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size : Size of stored outputfiles: 57M -log_destination_logstore_macro-threaded_yes-active-idle_100-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec

26 Used compress : 3 Used journal_block_size : Size of stored outputfiles: 157M -log_destination_logstore_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size : Size of stored outputfiles: 189M -log_destination_logstore_macro-threaded_yes-active-idle_1-0-log_source_syslog_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size : Size of stored outputfiles: 46M -log_destination_logstore-threaded_no-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no 26

27 average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 0 0 Used compress : 3 Used journal_block_size : Size of stored outputfiles: 52M -log_destination_logstore-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size : Size of stored outputfiles: 65M -log_destination_logstore-threaded_no-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size :

28 Size of stored outputfiles: 64M -log_destination_logstore-threaded_yes-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 0 0 Used compress : 3 Used journal_block_size : Size of stored outputfiles: 70M -log_destination_logstore-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size : Size of stored outputfiles: 63M -log_destination_logstore-threaded_yes-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used compress : 3 Used journal_block_size :

29 Size of stored outputfiles: 58M -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_all_modifiers-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_easy_regexp) Used modifier : filter(f_filter_tag_regexp) Used modifier : filter(f_filter_facility_filter) Used modifier : parser(pdb) Used modifier : rewrite(rewrite_set) Size of stored outputfiles: 1.3G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_complex_regexp-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_complex_regexp) Size of stored outputfiles: 1.2G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_easy_regexp-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 29

30 Used modifier : filter(f_filter_easy_regexp) Size of stored outputfiles: 2.2G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_facility_filter-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_facility_filter) Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_patterndb-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 30

31 Used modifier : parser(pdb) Size of stored outputfiles: 1.9G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_rewrite_set-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : rewrite(rewrite_set) Size of stored outputfiles: 2.9G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_tag_filter-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_tag_filter) Size of stored outputfiles: 3.1G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_all_modifiers-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 31

32 Used modifier : filter(f_filter_easy_regexp) Used modifier : filter(f_filter_tag_regexp) Used modifier : filter(f_filter_facility_filter) Used modifier : parser(pdb) Used modifier : rewrite(rewrite_set) Size of stored outputfiles: 863M -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_complex_regexp-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_complex_regexp) Size of stored outputfiles: 635M -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_easy_regexp-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_easy_regexp) Size of stored outputfiles: 3.4G 32

33 -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_facility_filter-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_facility_filter) Size of stored outputfiles: 12G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 12G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_patterndb-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : parser(pdb) Size of stored outputfiles: 884M 33

34 -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_rewrite_set-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : rewrite(rewrite_set) Size of stored outputfiles: 11G -log_destination_file_macro-threaded_yes-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_tag_filter-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Used modifier : filter(f_filter_tag_filter) Size of stored outputfiles: 12G -log_destination_file_macro-threaded_yes-active-idle_30-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = ,1 msg/sec, count= , time= , (average) msg size=150, bandwidth=119723,012 kb/sec Size of stored outputfiles: NA 34

35 Since in certain measurements the maximum capacity of the Gigabit network interface and the hard disk proved to be a bottleneck, a slightly modified environment was used for the last measurement: The server received log messages on 2 gigabit network cards instead of 1 The length of the messages was reduced to 150 bytes To overcome the I/O limitation of the hard disk, 30% of the messages was written to /dev/null, because syslog-ng reached the maximum performance of the specified hard disk. Note that writing messages to /dev/null is generally slower than writing them to a hard drive. 35