1 Performance measurements of syslog-ng Premium Edition 4 F1 October 13, 2011 Abstract Performance analysis of syslog-ng Premium Edition Copyright BalaBit IT Security Ltd.
2 Table of Contents 1. Preface Executive summary Optimizing multithreaded performance Use cases for performance improvement The test environment Interpreting the measurement results Summary About BalaBit Appendix 1. Detailed results
3 Preface 1. Preface This document summarizes the findings of several performance tests and measurements carried out on different versions of syslog-ng Premium Edition. The main focus was syslog-ng PE version 4 F1, but similar performance tests will be performed and published regularly with every major syslog-ng PE release. 3
4 Executive summary 2. Executive summary The main findings of the performance tests are as follows: The fastest way the syslog-ng PE application can receive log messages from the network is using plain TCP transport with the syslog() source driver. Using the tcp() driver is not significantly slower. Starting with version, syslog-ng PE can be run in multithreaded mode to scale to multiple CPUs or cores for increased performance. In case of multiple connections, the performance rates have been tested with 10 and 100 connections, and in both cases the results are the same. Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing: over messages per second when receiving messages from a single connection and storing them in text files; over messages per second when receiving messages from a single connection and storing them in logstore files; over messages per second when receiving messages from multiple connections and storing them in text files; over messages per second when receiving messages from multiple connections and storing them in logstore files; over messages per second when receiving messages from secure (TLS-encrypted) connections and storing them in text files. Note By default, syslog-ng PE runs in single-thread mode. Multithreading must be explicitly enabled in the syslog-ng PE configuration file using the threaded(yes) option. The tests were performed using 200-byte log messages and ran for 120 seconds. Note In certain cases, the rate of writing into plain-text files has increased from messages per second to messages per second since syslog-ng PE version 4.0. The rate of writing into logstore files has increased in certain cases from messages per second to messages per second since syslog-ng PE version 4.0. Using TLS-encrypted transport seriously degrades the rate of receiving messages, down to a rate of 30% of normal performance. PatternDB only works on one thread. Therefore, using PattnerDB can significantly decrease performance. If you are receiving lots of messages, but not all of them are processed with PatternDB, there is no significant performance degradation. If you are receiving messages through one connection only, performance will be better without multithreading (threaded(no)). 4
5 Optimizing multithreaded performance Using disk buffer on the client side can significantly degrade performance. This is especially the case if relays that are heavily loaded are used together with diskbuffer enabled on relays. Performance can also be degraded if lots of messages are received on the syslog-ng server, and are forwarded through the network (for example to a log analyzer or SIEM), and therefore disk buffer is enabled on the server as well. Regarding flow-control, using soft flow-control is faster by 10% than using hard flow-control. Higher stats_level decreases the performance. For example, stats_level(2) means -10% in performance. Simple filtering (for example, filtering on facility or tag) decreases the performance only by a few percent. However, regular expressions significantly decrease the message-processing rate, by about 50-75%, depending on the complexity of the regular expression. Version of syslog-ng PE provides true multi-core support. For the best performance, use multicore hardware for both the source and the destination side. For details on using multi-threading, see Chapter Multithreading and scaling in syslog-ng PE in The syslog-ng Premium Edition Administrator Guide Optimizing multithreaded performance Destinations that have a queue process that queue in a single thread. Multiple sources can send messages to the same queue, so the queue can scale to multiple CPUs. However, when the writer thread writes the queue contents to the destination, it will be single-threaded. Message parsing, rewrite rules, filters, and other types of message processing is performed by the reader thread in a sequential manner. This means that such operations can scale only if reading messages from the source can be multithreaded. For example, if a tcp source can process messages from different connections (clients) in separate threads. If the source cannot use multiple threads to process the messages, the operations will not scale. To improve the processing power of syslog-ng PE and scale to more processors, use the following methods: To improve scaling on the source side, use more sources, for example, more source files, or receive the messages from more parallel connections. For network sources, you can also configure a part of your clients to send the messages to a different port of your syslog-ng server, and use separate source definitions for each port. On the destination side, when writing the log messages to files, use macros in the filename to split the messages to separate files (for example, using the $HOST macro). Files with macros in their filenames are processed in separate writer threads. On the destination side, when sending messages to a syslog-ng server, you can use multiple connections to the server if you configure the syslog-ng server to receive messages on multiple ports, and configure the clients to use both ports Use cases for performance improvement Improving performance with lots of connections. If there are several thousand active connections simultaneously, it is advised to place relay syslog-ng-s on another computer in front of the syslog-ng. Switching between the active connections is time-consuming, while the amount of incoming messages is usually not significant. This problem is solved by using relays, since they are collecting the logs. The syslog-ng solution can handle lots of log messages sent from a few connections easily. 5
6 Use cases for performance improvement Storing lots of log messages. It requires large free disk space if syslog-ng receives lots of messages per second (note that syslog-ng can process messages at even 100 MB/sec speed). In this case it is advised to use compressed logstore instead of file for storing data. The size of a compressed logstore (compress(3)) is only a few percent of the file destination. Obviously, the effectiveness of the compression depends on the pattern of the incoming messages, but since most of the time the log messages consist of simple text messages, they can be compressed quite effectively (around 90% compression rate). Writing data into SQL database. The syslog-ng will only commit after the amount of inserts set in flush-lines(), if flags(explicit-commints) is enabled and the value of flush-lines() is raised. The speed of writing into an SQL database can be 2-3 times faster if flush-lines(100) and flags(explicit-commints) is enabled. Filtering messages. It is advised to use the simplest filters when filtering incoming messages. If a message can be filtered with several types of filters, check the measured data. For example, if a message is filtered with a complicated regexp, the performance of syslog-ng can drop down to 5%, whereas if tag or facility, there is no performance decrease. Identifying bottlenecks. It is advised to first identify the bottlenecks if the performance of syslog-ng seems to be too low. For example, if syslog-ng is writing on slow disks at the destination side, upgrading the source side device will not help increasing the performance. 6
7 The test environment 3. The test environment The test environment consisted of a single client and a server hardware, connected via a Gigabit switch. Note that in certain test runs, the client opened several separate connections to the servers to simulate real-life logging environments. The syslog-ng Premium Edition application was installed from the.run package. The client and the server hardware were identical with the following main parameters: 2x Intel Xeon Processor E5620 (12M Cache, 2.40 GHz, 5.86 GT/s Intel QPI, quad-core) 18 GB RAM Western Digital RE3 WD2502ABYS 250GB 7200 RPM 16MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drives The main operating system of the hosts was ubuntu-lucid-x64, but the syslog-ng PE client and server applications were running in chrooted debian-etch-x64 environments. 7
8 Interpreting the measurement results 4. Interpreting the measurement results Several different measurements were performed in various syslog-ng configurations and settings. These are listed in Appendix 1, Detailed results (p. 11) in the following format. -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 3.2G The results can be interpreted as follows: The configuration parameters of the syslog-ng PE server are listed as a string of name-value pairs. Whitespaces in the name are replaced with underscores. For example -log_destination_file_macro-threaded_no-active-idle_ log_source_legacy_tcp-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no: hw_config: The identifier of the hardware configuration used in the tests. For details, see Section 3, The test environment (p. 7). platform: The operating system and architecture used in the test, for example, debian-etch_amd64. log_source: The type of source driver used in the test. The legacy name refers to syslog-ng drivers using the BSD-syslog standard (RFC3164), while syslog to the IETF-syslog standards (RFC5424). log_size: The length of the log messages in bytes. flush_lines: The value of the flush_lines() parameter. modifiers: Modifications (for example, rewriting or filtering) used in the log path (if any). file/logstore: All incoming messages are written into the same file or logstore. file/logstore_macro: All messages from each connection are written into a separate file or logstore. threaded: Whether multi-threading is used or not. active-idle: The number of active and idle connections between the client and the server. Log messages were transferred only in the active connections. fetch_limit-iw_size: The values of the fetch_limit() and iw_size() parameters, respectively. log_destination: The type of destination driver used in the test. runtime: The length of the test run (in seconds). journal_block_size: The value of the journal_block_size parameter. 8
9 Interpreting the measurement results max_connections: The value of the max_connections parameter. Test results measured on the syslog-ng PE client, for example average rate = msg/sec, count= , time= , (last) msg size=200, bandwidth= kb/sec: Average rate: The rate of messages the client sent the log messages to the server (messages per second). Count: The total number of log messages sent by the clients during the test run. Time: The length of the test run (in seconds). Msg size: The size of the log messages sent by the client (in bytes). Bandwidth: The total bandwith used by the client connections (in kilobytes per seconds). 9
10 Summary 5. Summary The test measurements show that the processing capabilities of syslog-ng Premium Edition have increased significantly in version 4.1, and that syslog-ng PE is capable of receiving and processing high-volume log traffic About BalaBit BalaBit IT Security Ltd. is a developer of network security solutions satisfying the highest standards. BalaBit was founded and is currently owned by Hungarian individuals. Its main products are the syslog-ng system logging software, which is the most widely used alternative syslog solution of the world; the syslog-ng Store Box logserver appliance; Zorp, a modular proxy gateway capable of inspecting over twenty protocols, including encrypted ones like SSL and SSH, and the Shell Control Box, an appliance that can transparently control, audit, and replay SSH, RDP, VNC, Citrix, and Telnet traffic. To learn more about commercial and open source BalaBit products, request an evaluation version, or find a reseller, visit the following links: The syslog-ng homepage Shell Control Box homepage syslog-ng Store Box (SSB) homepage Product manuals, guides, and other documentation Register and request an evaluation version Find a reseller All questions, comments or inquiries should be directed to or by post to the following address: BalaBit IT Security 1115 Budapest, Bártfai str. 54 Phone: Fax: Web: Copyright 2011 BalaBit IT Security Ltd. Some rights reserved. This document is published under the Creative Commons Attribution Noncommercial No Derivative Works (byncnd) 3.0 license. All other product names mentioned herein are the trademarks of their respective owners. The latest version is always available at the BalaBit Documentation Page. 10
11 Appendix 1. Detailed results The following measurement results were obtained using syslog-ng PE in the environment described in Section 3, The test environment (p. 7). -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 3.2G -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 926M -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec
12 Size of stored outputfiles: 1.7G -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.1G -log_destination_file_macro-threaded_no-active-idle_100-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 888M -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec 12
13 Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 958M -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 2.3G -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.3G 13
14 -log_destination_file_macro-threaded_no-active-idle_10-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 892M -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 3.2G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 990M 14
15 -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 2.9G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 3.4G -log_destination_file_macro-threaded_no-active-idle_1-0-log_source_syslog_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec Size of stored outputfiles: 853M -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_legacy_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no 15
16 average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 11G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_legacy_tcp_tls-platform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=200, bandwidth= kb/sec Size of stored outputfiles: 2.7G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_legacy_udpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no server side average rate = msg/sec Size of stored outputfiles: 1.7G -log_destination_file_macro-threaded_yes-active-idle_100-0-log_source_syslog_tcpplatform_debian-etch_amd64-syslogger_syslogd-modifiers_na-logstore_encryption_no average rate = msg/sec, count= , time= , (average) msg size=204, bandwidth= kb/sec 16
StreamServe Persuasion SP5 Microsoft SQL Server Database Guidelines Rev A StreamServe Persuasion SP5 Microsoft SQL Server Database Guidelines Rev A 2001-2011 STREAMSERVE, INC. ALL RIGHTS RESERVED United
theguard! Service Management Center (Valid for Version 6.3 and higher) Introduction Introduction about the minimal system requirements of theguard! Service Management Center. The theguard! Service Management
Best Practices Guide McAfee epolicy Orchestrator for use with epolicy Orchestrator versions 4.5.0 and 4.0.0 COPYRIGHT Copyright 2011 McAfee, Inc. All Rights Reserved. No part of this publication may be
Deployment Planning Guide August 2011 Copyright: 2011, CCH, a Wolters Kluwer business. All rights reserved. Material in this publication may not be reproduced or transmitted in any form or by any means,
Getting Started with Zeus Web Server 4.3 Zeus Technology Limited - COPYRIGHT NOTICE Zeus Technology Limited 2004. Copyright in this documentation belongs to Zeus Technology Limited. All rights are reserved.
The Definitive Guide To tm Building Highly Scalable Enterprise File Serving Solutions Chris Wolf Chapter 3: Data Path Optimization for Enterprise File Serving...44 The Big Picture of File Access...44 Availability
Veeam Backup & Replication for VMware Version 6.x Best Practices for Deployment & Configuration March, 2013 Tom Sightler Solutions Architect, Core Products Veeam Software 2013 Veeam Software. All rights
Redbooks Paper Tuning Windows Server 2003 on IBM System x Servers Phillip Dundas David Watts Windows Server 2003 1 is Microsoft s mainstream server operating system and has been now for almost four years.
Data Center Real User Monitoring Oracle Forms Application Monitoring User Guide Release 12.3 Please direct questions about Data Center Real User Monitoring or comments on this document to: Customer Support
Pervasive PSQL v11 Advanced Operations Guide Procedures and References for Advanced Users Pervasive Software Inc. 12365 Riata Trace Parkway Building B Austin, TX 78727 USA Telephone: 512 231 6000 or 800
WHITE PAPER: DATA PROTECTION Symantec NetBackup, Cisco UCS, and VMware vsphere Joint Backup Performance Benchmark George Winter, Symantec Corporation, Roger Andersson, Cisco Systems, Inc. Paul Vasquez,
Improving the Performance of Passive Network Monitoring Applications with Memory Locality Enhancements Antonis Papadogiannakis a,, Giorgos Vasiliadis a, Demetres Antoniades a, Michalis Polychronakis b,
2X ApplicationServer & LoadBalancer Manual 2X ApplicationServer & LoadBalancer Contents 1 URL: www.2x.com E-mail: email@example.com Information in this document is subject to change without notice. Companies,
Department of Computer Department Science and Engineering, of Computer CUHK Science 2010-2011 Final and Year Engineering Project Report The Chinese University of Hong Kong (1st Term) Cloud computing technologies
Network Monitoring Based on IP Data Flows Best Practice Document Produced by CESNET led working group on Network monitoring (CBPD131) Authors: Martin Žádník March 2010 TERENA 2010. All rights reserved.
Relational Database Management Systems in the Cloud: Microsoft SQL Server 2008 R2 Miles Ward July 2011 Page 1 of 22 Table of Contents Introduction... 3 Relational Databases on Amazon EC2... 3 AWS vs. Your
Parallels Parallels Virtuozzo Containers for Windows User's Guide Version 4.0 Copyright 1999-2008 Parallels Software International Inc. ISBN: N/A Parallels Holdings Ltd. 13755 Sunrise Valley Drive Suite
Contents Copyright 1995-2015 SolarWinds Worldwide, LLC. All rights reserved worldwide. No part of this document may be reproduced by any means nor modified, decompiled, disassembled, published or distributed,
NETAPP TECHNICAL REPORT Open Systems SnapVault Best Practices Guide Jeremy Merrill, Darrin Chapman TR-3466 ABSTRACT This document is a deployment guide for architecting and deploying Open Systems SnapVault
Performance Study VirtualCenter Database Performance for Microsoft SQL Server 2005 VirtualCenter 2.5 VMware VirtualCenter uses a database to store metadata on the state of a VMware Infrastructure environment.
FileMaker Server 12 Getting Started Guide 2007 2012 FileMaker, Inc. All Rights Reserved. FileMaker, Inc. 5201 Patrick Henry Drive Santa Clara, California 95054 FileMaker and Bento are trademarks of FileMaker,