The syslog-ng Premium Edition 5 F3 Administrator Guide

Transcription

1 The syslog-ng Premium Edition 5 F3 Administrator Guide Publication date June 17, 2015 Abstract This manual is the primary documentation of the syslog-ng Premium Edition 5 F3 product.

2 Copyright BalaBit S.a.r.l. Copyright 2015 BalaBit S.a.r.l.. All rights reserved. This document is protected by copyright and is distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this document may be reproduced in any form by any means without prior written authorization of BalaBit. This documentation and the product it describes are considered protected by copyright according to the applicable laws. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit ( This product includes cryptographic software written by Eric Young AIX, AIX 5L, AS/400, BladeCenter, eserver, IBM, the IBM logo, IBM System i, IBM System i5, IBM System x, iseries, i5/os, Netfinity, NetServer, OpenPower, OS/400, PartnerWorld, POWER, ServerGuide, ServerProven, and xseries are trademarks or registered trademarks of International Business Machines. Alliance Log Agent for System i is a registered trademark of Patrick Townsend & Associates, Inc. The BalaBit name and the BalaBit logo are registered trademarks of BalaBit S.a.r.l.. Debian is a registered trademark of Software in the Public Interest Inc. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software Foundation. Linux is a registered trademark of Linus Torvalds. MapR, is a trademark of MapR Technologies, Inc. MySQL is a registered trademark of Oracle and/or its affiliates. Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. Red Hat, Inc., Red Hat Enterprise Linux and Red Hat Linux are trademarks of Red Hat, Inc. SUSE is a trademark of SUSE AG, a Novell business. Solaris is a registered trademark of Oracle and/or its affiliates. The syslog-ng name and the syslog-ng logo are registered trademarks of BalaBit. Windows 95, 98, ME, 2000, XP, Server 2003, Vista, Server 2008, 7, 8, and Server 2012 are registered trademarks of Microsoft Corporation. For details on FIPS-compliance, see Section 11.3, Legal Notice of FIPS Compliance of Syslog-ng Premium Edition (p. 304). All other product names mentioned herein are the trademarks of their respective owners. DISCLAIMER BalaBit is not responsible for any third-party Web sites mentioned in this document. BalaBit does not endorse and is not responsible or liable for any content, advertising, products, or other material on or available from such sites or resources. BalaBit will not be responsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through any such sites or resources. ii

3 Table of Contents Preface... xv 1. Summary of contents... xv 2. Target audience and prerequisites... xvi 3. Products covered in this guide... xvii 4. Typographical conventions... xvii 5. Contact and support information... xvii 5.1. Sales contact... xviii 5.2. Support contact... xviii 5.3. Training... xviii 6. About this document... xviii 6.1. Summary of changes... xviii 6.2. Feedback... xx 6.3. Acknowledgments... xx 1. Introduction to syslog-ng What syslog-ng is What syslog-ng is not Why is syslog-ng needed? What is new in syslog-ng Premium Edition 5 F3? Who uses syslog-ng? Public references of syslog-ng Premium Edition Supported platforms Limitations on Microsoft Windows platforms Certified packages The concepts of syslog-ng The philosophy of syslog-ng Logging with syslog-ng The route of a log message in syslog-ng Modes of operation Client mode Relay mode Server mode Global objects Timezones and daylight saving How syslog-ng PE assigns timezone to the message A note on timezones and timestamps Versions and releases of syslog-ng PE Licensing GPL and LGPL licenses High availability support The structure of a log message BSD-syslog or legacy-syslog messages IETF-syslog messages Message representation in syslog-ng PE Structuring macros, metadata, and other value-pairs Specifying data types in value-pairs iii

4 3. Installing syslog-ng Prerequisites to installing syslog-ng PE Security-enhanced Linux: grsecurity, SELinux Installing syslog-ng using the.run installer Installing syslog-ng PE in client or relay mode Installing syslog-ng PE in server mode Installing syslog-ng PE without user-interaction Installing syslog-ng PE on RPM-based platforms (Red Hat, SUSE, AIX) Using syslog-ng PE on SELinux Installing syslog-ng on Debian-based platforms Installing syslog-ng PE using.pkg installer Installing syslog-ng PE with user-interaction Installing syslog-ng PE without user-interaction Installing syslog-ng PE from a transformed PKG package Installing syslog-ng PE on Windows platforms Installing syslog-ng without user-interaction on Windows Upgrading syslog-ng PE Upgrading from previous syslog-ng PE versions to 5 F Upgrading to syslog-ng PE 5 F Upgrading syslog-ng PE to other package versions Upgrading from syslog-ng PE to syslog-ng OSE Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE Uninstalling syslog-ng PE Configuring Microsoft SQL Server to accept logs from syslog-ng The syslog-ng PE quick-start guide Configuring syslog-ng on client hosts Configuring syslog-ng on server hosts Configuring syslog-ng relays Configuring syslog-ng on relay hosts How relaying log messages works The syslog-ng PE configuration file Location of the syslog-ng configuration file The configuration syntax in detail Notes about the configuration syntax Global and environmental variables Logging configuration changes Modules in syslog-ng PE Loading modules Managing complex syslog-ng configurations Including configuration files Reusing configuration blocks Collecting log messages sources and source drivers How sources work Collecting messages from Windows eventlog sources eventlog() source options Limitations of using the EVT API on Windows Vista or newer Collecting internal messages iv

5 internal() source options Collecting messages from text files Notes on reading kernel messages File sources and the RFC5424 message format file() source options Collecting messages using the RFC3164 protocol (network() driver) network() source options Collecting messages from named pipes pipe() source options Receiving messages from external applications program() source options Collecting messages from tables or relational database Supported SQL sources by platform sql() source options Customizing SQL queries Collecting messages on Sun Solaris sun-streams() source options Collecting messages using the IETF syslog protocol (syslog() driver) syslog() source options Collecting the system-specific log messages of a platform Collecting messages from the systemd-journal system log storage systemd-journal() source options Collecting systemd messages using a socket Collecting messages from remote hosts using the BSD syslog protocol tcp(), tcp6(), udp() and udp6() source options OBSOLETE Collecting messages from UNIX domain sockets unix-stream() and unix-dgram() source options Sending and storing log messages destinations and destination drivers Storing messages in plain-text files file() destination options Storing messages on the Hadoop Distributed File System (HDFS) Prerequisites How syslog-ng PE interacts with HDFS Storing messages with MapR-FS HDSF destination options Storing messages in encrypted files Displaying the contents of logstore files Journal files logstore() destination options Storing messages in a MongoDB database How syslog-ng PE connects the MongoDB server mongodb() destination options Sending messages to a remote log server using the RFC3164 protocol (network() driver) network() destination options Sending messages to named pipes pipe() destination options Sending messages to external applications v

6 program() destination options Generating SMTP messages ( ) from logs smtp() destination options Error handling Sending SNMP traps Converting Cisco syslog messages to "clogmessagegenerated" SNMP traps snmp() destination options Storing messages in an SQL database Using the sql() driver with an Oracle database Using the sql() driver with a Microsoft SQL database The way syslog-ng interacts with the database MySQL-specific interaction methods MsSQL-specific interaction methods Supported SQL destinations by platform sql() destination options Sending messages to a remote log server using the IETF-syslog protocol syslog() destination options Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) tcp(), tcp6(), udp(), and udp6() destination options Sending messages to UNIX domain sockets unix-stream() and unix-dgram() destination options Sending messages to a user terminal usertty() destination Routing messages: log paths, reliability, and filters Log paths Embedded log statements Log path flags Managing incoming and outgoing messages with flow-control Flow-control and multiple destinations Configuring flow-control Using disk-based and memory buffering Enabling reliable disk-based buffering Enabling normal disk-based buffering Enabling memory buffering Client-side failover Filters Using filters Combining filters with boolean operators Comparing macro values in filters Using wildcards, special characters, and regular expressions in filters Tagging messages Filter functions Dropping messages Global options of syslog-ng PE Configuring global syslog-ng options Global options TLS-encrypted message transfer Secure logging using TLS vi

7 10.2. Encrypting log messages with TLS Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server Mutual authentication using TLS Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server TLS options FIPS-compliant syslog-ng Installing FIPS-compliant syslog-ng PE Limitations of the FIPS-compliant syslog-ng PE Legal Notice of FIPS Compliance of Syslog-ng Premium Edition Reliable Log Transfer Protocol Logging using RLTP How RLTP connections work Using RLTP in a client-relay-server scenario RLTP options Examples for using RLTP Manipulating messages Customizing message format Formatting messages, filenames, directories, and tablenames Templates and macros Date-related macros Hard vs. soft macros Macros of syslog-ng PE Using template functions Template functions of syslog-ng PE Modifying messages Replacing message parts Setting message fields to specific values Creating custom SDATA fields Setting multiple message fields to specific values Conditional rewrites Regular expressions Types and options of regular expressions Optimizing regular expressions Parsing and segmenting structured messages Parsing messages with comma-separated and similar values Options of CSV parsers The JSON parser Options of JSON parsers Processing message content with a pattern database Classifying log messages The structure of the pattern database How pattern matching works Artificial ignorance Using pattern databases Using parser results in filters and templates Downloading sample pattern databases vii

8 15.3. Correlating log messages Referencing earlier messages of the context Triggering actions for identified messages Conditional actions External actions Actions and message correlation Creating pattern databases Using pattern parsers What's new in the syslog-ng pattern database format V The syslog-ng pattern database format Statistics of syslog-ng Multithreading and scaling in syslog-ng PE Multithreading concepts of syslog-ng PE Configuring multithreading Optimizing multithreaded performance Troubleshooting syslog-ng Possible causes of losing log messages Creating syslog-ng core files Collecting debugging information with strace, truss, or tusc Running a failure script Stopping syslog-ng Best practices and examples General recommendations Handling large message load Using name resolution in syslog-ng Resolving hostnames locally Collecting logs from chroot Configuring log rotation Appendix A. The syslog-ng manual pages dqtool loggen lgstool pdbtool persist-tool syslog-ng syslog-ng.conf syslog-ng-ctl Appendix B. License contract for BalaBit Product Appendix C. GNU General Public License C.1. Preamble C.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION C.2.1. Section C.2.2. Section C.2.3. Section C.2.4. Section C.2.5. Section C.2.6. Section viii

9 C.2.7. Section C.2.8. Section C.2.9. Section C Section C Section C NO WARRANTY Section C Section C.3. How to Apply These Terms to Your New Programs Appendix D. GNU Lesser General Public License D.1. Preamble D.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION D.2.1. Section D.2.2. Section D.2.3. Section D.2.4. Section D.2.5. Section D.2.6. Section D.2.7. Section D.2.8. Section D.2.9. Section D Section D Section D Section D Section D Section D Section D NO WARRANTY Section D Section D.3. How to Apply These Terms to Your New Libraries Appendix E. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary List of syslog-ng PE parameters Index ix

10 List of Examples 2.1. Licensing syslog-ng PE Licensing syslog-ng PE Licensing syslog-ng PE Counting log source hosts Using type-hinting Using the value-pairs() option Using the rekey() option Extracting syslog-ng PE from a transformed PKG package The default configuration file of syslog-ng PE A simple configuration for clients A simple configuration for servers A simple configuration for relays A simple configuration file Using required and optional parameters Using global variables Reusing configuration blocks Defining blocks with multiple elements Passing arguments to blocks A simple source statement A source statement using two source drivers Setting default priority and facility Source statement on a Linux based operating system Using the eventlog() driver Using the internal() driver Using the file() driver Tailing files Using wildcards in the filename File-related information in message Initial window size of file sources Processing Tomcat logs Monitoring multiple directories Using the network() driver Initial window size of a connection Processing Tomcat logs Using the pipe() driver Initial window size of a connection Using the program() driver Initial window size of a connection Using a MySQL source A sample archive query SQL source option columns A sample connect query SQL source option datetime-column(col_name, [format]) A sample fetch query Initial window size of a connection x

11 6.28. SQL source option prefix() SQL source option template() SQL source fetch-query Using the sun-streams() driver Initial window size of a connection Using the syslog() driver Initial window size of a connection Processing Tomcat logs Sending all fields through syslog protocol using the systemd-journal() driver Filtering for a specific field using the systemd-journal() driver Sending all fields in value-pairs using the systemd-journal() driver Using the systemd-syslog() driver Using the unix-stream() and unix-dgram() drivers Initial window size of a connection Processing Tomcat logs A simple destination statement Using the file() driver Using the file() driver with macros in the file name and a template for the message Storing logfiles on HDFS Storing logfiles with MapR-FS Using the logstore() driver Calculating memory usage of logstore journals Setting journal block number and size Setting journal block number and size Using the mongodb() driver Examples for using disk-buffer() Using the network() driver Examples for using disk-buffer() Specifying failover servers for syslog() destinations Spoofing the source address on Microsoft Windows Using the pipe() driver Using the program() destination driver Examples for using disk-buffer() Using the smtp() driver Simple alerting with the smtp() driver Examples for using disk-buffer() Using the snmp() destination driver Defining a Cisco-specific SNMP destination Defining SNMP objects Using the sql() driver Using the sql() driver with an Oracle database Using the sql() driver with an MSSQL database Examples for using disk-buffer() Setting flags for SQL destinations Using SQL NULL values Value: default Using the syslog() driver Examples for using disk-buffer() xi

12 7.34. Specifying failover servers for syslog() destinations Spoofing the source address on Microsoft Windows Using the unix-stream() driver Examples for using disk-buffer() Using the usertty() driver A simple log statement Using embedded log paths Using log path flags Soft flow-control Hard flow-control Sizing parameters for flow-control Example for using reliable disk-based buffering Example for using normal disk-based buffering Example for using memory buffering A simple filter statement Comparing macro values in filters Filtering with widcards Selecting messages using the in-list filter Adding tags and filtering messages with tags Skipping messages Using global options Calculating memory usage of logstore journals Limiting the memory use of journal files A destination statement using TLS A source statement using TLS Disabling mutual authentication A destination statement using mutual authentication A source statement using TLS Simple RLTP connection RLTP with TLS encryption Using templates and macros Using ${RCPTID} macro Using SDATA macros Using the format-json template function Using the format-welf() template function Using the grep template function Using the $(hash) template function Anonymizing IP addresses Using pattern databases and the if template function Using the indent-multi-line template function Using numerical template functions Using substitution rules Anonymizing IP addresses Setting message fields to a particular value Rewriting custom SDATA fields Using groupset rewrite rules Using conditional rewriting Using Posix regular expressions xii

13 Using PCRE regular expressions Optimizing regular expressions in filters Segmenting hostnames separated with a dash Parsing Apache log files Segmenting a part of a message Adding the end of the message to the last column Using a JSON parser Using the marker option in JSON parser Defining pattern databases Using classification results Using classification results for filtering messages Using pattern parsers as macros How syslog-ng PE calculates context-timeout Using message correlation Sending triggered messages to the internal() source Generating messages for pattern database matches Generating messages with inherited values Actions based on the number of messages Sending triggered messages to external applications Using the inherit-properties option Pattern parser syntax Using the STRING and ESTRING parsers A V4 pattern database containing a single rule Enabling multithreading File destination for log rotation Logstore destination for log rotation Command for cron for log rotation A.1. lgstool cat filter A.2. lgstool tail filter A.3. Using required and optional parameters A.4. Using global options xiii

14 List of Procedures The route of a log message in syslog-ng How syslog-ng PE assigns timezone to the message Installing syslog-ng PE in client or relay mode Installing syslog-ng PE in server mode Installing syslog-ng PE on RPM-based platforms (Red Hat, SUSE, AIX) Using syslog-ng PE on SELinux Installing syslog-ng on Debian-based platforms Installing syslog-ng PE with user-interaction Installing syslog-ng PE without user-interaction Installing syslog-ng PE from a transformed PKG package Installing syslog-ng PE on Windows platforms Upgrading to syslog-ng PE 5 F Configuring Microsoft SQL Server to accept logs from syslog-ng Configuring syslog-ng on client hosts Configuring syslog-ng on server hosts Configuring syslog-ng on relay hosts Change an old source driver to the network() driver Prerequisites How syslog-ng PE interacts with HDFS Storing messages with MapR-FS How syslog-ng PE connects the MongoDB server Change an old destination driver to the network() driver Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server How RLTP connections work How conditional rewriting works Creating syslog-ng core files Resolving hostnames locally Collecting logs from chroot xiv

15 Summary of contents Preface Welcome to the syslog-ng Premium Edition 5 F3 Administrator Guide! This document describes how to configure and manage syslog-ng. Background information for the technology and concepts used by the product is also discussed. 1. Summary of contents Chapter 1, Introduction to syslog-ng (p. 1) describes the main functionality and purpose of syslog-ng PE. Chapter 2, The concepts of syslog-ng (p. 7) discusses the technical concepts and philosophies behind syslog-ng PE. Chapter 3, Installing syslog-ng (p. 28) describes how to install syslog-ng PE on various UNIX-based platforms using the precompiled binaries. Chapter 4, The syslog-ng PE quick-start guide (p. 59) provides a briefly explains how to perform the most common log collecting tasks with syslog-ng PE. Chapter 5, The syslog-ng PE configuration file (p. 66) discusses the configuration file format and syntax in detail, and explains how to manage large-scale configurations using included files and reusable configuration snippets. Chapter 6, Collecting log messages sources and source drivers (p. 75) explains how to collect and receive log messages from various sources. Chapter 7, Sending and storing log messages destinations and destination drivers (p. 151) describes the different methods to store and forward log messages. Chapter 8, Routing messages: log paths, reliability, and filters (p. 255) explains how to route and sort log messages, and how to use filters to select specific messages. Chapter 9, Global options of syslog-ng PE (p. 277) lists the global options of syslog-ng PE and explains how to use them. Chapter 10, TLS-encrypted message transfer (p. 292) shows how to secure and authenticate log transport using TLS encryption. Chapter 12, Reliable Log Transfer Protocol (p. 305) describes the reliable log transport that prevents message loss. Chapter 13, Manipulating messages (p. 310) describes how to customize message format using templates and macros, how to rewrite and modify messages, and how to use regular expressions. Chapter 14, Parsing and segmenting structured messages (p. 335) describes how to segment and process structured messages like comma-separated values. xv

16 Target audience and prerequisites Chapter 15, Processing message content with a pattern database (p. 343) explains how to identify and process log messages using a pattern database. Chapter 16, Statistics of syslog-ng (p. 365) details the available statistics that syslog-ng PE collects about the processed log messages. Chapter 17, Multithreading and scaling in syslog-ng PE (p. 368) describes how to configure syslog-ng PE to use multiple processors, and how to optimize its performance. Chapter 18, Troubleshooting syslog-ng (p. 371) offers tips to solving problems. Chapter 19, Best practices and examples (p. 375) gives recommendations to configure special features of syslog-ng PE. Appendix A, The syslog-ng manual pages (p. 379) contains the manual pages of the syslog-ng PE application. Appendix B, License contract for BalaBit Product (p. 419) includes the text of the End-User License Agreement applicable to syslog-ng Premium Edition. Appendix D, GNU Lesser General Public License (p. 431) includes the text of the LGPLv2.1 license applicable to the core of syslog-ng Premium Edition. Appendix C, GNU General Public License (p. 425) includes the text of the GPLv2 license applicable to syslog-ng Premium Edition. Appendix E, Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License (p. 440) includes the text of the Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License applicable to The syslog-ng Premium Edition 5 F3 Administrator Guide. Glossary (p. 445) defines the important terms used in this guide. List of syslog-ng PE parameters (p. 449) provides cross-references to the definitions of options, parameters, and macros available in syslog-ng PE. The Index provides cross-references to important terms used in this guide. 2. Target audience and prerequisites This guide is intended for system administrators and consultants responsible for designing and maintaining logging solutions and log centers. It is also useful for IT decision makers looking for a tool to implement centralized logging in heterogeneous environments. The following skills and knowledge are necessary for a successful syslog-ng administrator: At least basic system administration knowledge. An understanding of networks, TCP/IP protocols, and general network terminology. Working knowledge of the UNIX or Linux operating system. In-depth knowledge of the logging process of various platforms and applications. An understanding of the legacy syslog (BSD-syslog) protocol) and the new syslog (IETF-syslog) protocol) standard. xvi

17 Products covered in this guide 3. Products covered in this guide This guide describes the use of the following products: syslog-ng Premium Edition (syslog-ng PE) and later 4. Typographical conventions Before you start using this guide, it is important to understand the terms and typographical conventions used in the documentation. For more information on specialized terms and abbreviations used in the documentation, see the Glossary at the end of this document. The following kinds of text formatting and icons identify special information in the document. Tip Tips provide best practices and recommendations. Note Notes provide additional information on a topic, and emphasize important facts and considerations. Warning Warnings mark situations where loss of data or misconfiguration of the device is possible if the instructions are not obeyed. Command Emphasis /path/to/file Parameters Label Menu Button Commands you have to execute. Reference items, additional readings. File names. Parameter and attribute names. GUI output messages or dialog labels. A submenu or menu item in the menu bar. Buttons in dialog windows. 5. Contact and support information This product is developed and maintained by BalaBit-Europe. We are located in Budapest, Hungary. Our address is: BalaBit-Europe 2 Alíz Street xvii

18 Sales contact H-1117 Budapest, Hungary Tel: Fax: <info@balabit.com> Web: Sales contact You can directly contact us with sales related topics at the address <sales@balabit.com>, or leave us your contact information and we call you back Support contact In case you experience a problem that is not covered in this guide, visit the syslog-ng wiki or post it on syslog-ng mailing list. Product support, including 7x24 online support is available in various packages. For support options, visit the following page. To access the BalaBit Online Support System (BOSS), sign up for an account at the MyBalaBit page and request access to the BalaBit Online Support System (BOSS). Online support is available 24 hours a day. BOSS is available only for registered users with a valid support package. Support address: <support@balabit.com>. Support hotline: (available from 9 AM to 5 PM CET on weekdays) 5.3. Training BalaBit-Europe holds courses on using its products for new and experienced users. For dates, details, and application forms, visit the webpage. 6. About this document This guide is a work-in-progress document with new versions appearing periodically. The latest version of this document can be downloaded from the BalaBit website here Summary of changes Version 5 F2-5 F3 Changes in product: Section 7.2, Storing messages on the Hadoop Distributed File System (HDFS) (p. 160) has been added to the document. CSV-parsers can use strings as delimiters. For details, see Section delimiters() (p. 338). xviii

19 Summary of changes Changes in documentation: The tcp(), tcp6(), udp(), udp6() source and destination drivers have been deprecated, as all of their functionality can be achieved with the network() driver. For help on migrating to the network() driver, see Procedure , Change an old source driver to the network() driver (p. 141) and Procedure , Change an old destination driver to the network() driver (p. 245). The beginning of Chapter 18, Troubleshooting syslog-ng (p. 371) has been extended with basic troubleshooting information. Section 1.6, Supported platforms (p. 3) has been updated. Other editorial corrections Version 5 F1-5 F2 Changes in product: Section netmask6() (p. 275) has been added to the document. Section 3.2, Security-enhanced Linux: grsecurity, SELinux (p. 29) has been added to the document. New examples have been added to Section hash (p. 323). The use-rcptid() global option has been deprecated. Section use-uniqid() (p. 291) has been added to the document. The assume-utf8 source flag has been documented. Multiple message fields can be modified using the groupset rewrite rule. For details, see Section , Setting multiple message fields to specific values (p. 329). Platforms CentOS 7, Ubuntu LTS (Trusty Tahr), and Red Hat ES 7 are now supported. Section 7.8, Generating SMTP messages ( ) from logs (p. 206) has been added to the document. Section 6.12, Collecting messages from the systemd-journal system log storage (p. 137) has been added to the document. Log rotation with syslog-ng PE has been described in Section 19.5, Configuring log rotation (p. 378). The inlist() filter has been added to Section 8.5.6, Filter functions (p. 272). The option time-sleep() is now deprecated Version 5 LTS - 5 F1 Changes in product: Section 7.4, Storing messages in a MongoDB database (p. 176) has been added to the document. A new parser is available to parse JSON-formatted messages. For details, see Section 14.2, The JSON parser (p. 340). xix

20 Feedback Section format-json (p. 322) has been added to the document. Changes in documentation: The retry_sql_inserts option has been renamed to retries to increase consistency. Section on-error() (p. 285) can be set locally for MongoDB destinations as well. Section timestamp-freq() (p. 288) has been added to Chapter 9, Global options of syslog-ng PE (p. 277). The FILE_NAME and SOURCE macros have been added to Section , Macros of syslog-ng PE (p. 313). Section 2.12, Structuring macros, metadata, and other value-pairs (p. 22) has been added to the document. Section on-error() (p. 285) has been added to the document. Section 1.6, Supported platforms (p. 3) has been updated. Chapter 12, Reliable Log Transfer Protocol (p. 305) has been expanded, including several clarifications and improvements. Option api() has been renamed to event-api() in Section 6.2.1, eventlog() source options (p. 78) Feedback Any feedback is greatly appreciated, especially on what else this document should cover. General comments, errors found in the text, and any suggestions about how to improve the documentation is welcome at 6.3. Acknowledgments BalaBit would like to express its gratitude to the syslog-ng users and the syslog-ng community for their invaluable help and support, including the community members listed at syslog-ng Community Page. xx

21 What syslog-ng is Chapter 1. Introduction to syslog-ng This chapter introduces the syslog-ng Premium Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure What syslog-ng is The syslog-ng Premium Edition (syslog-ng PE) application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. The main features of syslog-ng PE are summarized below. Reliable log transfer: The syslog-ng application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. The logs of different servers can be collected and stored centrally on dedicated log servers. Transferring log messages using the RLTP protocol ensures that no messages are lost. Secure logging using TLS: Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates. Client-side failover: When transferring messages to a remote server, the syslog-ng PE clients can be configured to send the log messages to secondary servers if the primary server becomes unaccessible. Disk-based message buffering: The Premium Edition of syslog-ng stores messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent no messages are lost even if syslog-ng is restarted. Direct database access: Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite. Encrypted and timestamped log storage: The Premium Edition of syslog-ng can store log messages securely in encrypted, compressed, and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA). Heterogeneous environments: The syslog-ng application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, Microsoft Windows, BSD, Sun Solaris, HP-UX, and AIX. Filter and classify: The syslog-ng application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations. Parse and rewrite: The syslog-ng application can segment log messages to named fields or columns, and also modify the values of these fields. The syslog-ng PE application can receive structured JSON messages 1

22 What syslog-ng is not Structured messages: The syslog-ng PE application can receive and send structured JSON messages. IPv4 and IPv6 support: The syslog-ng application can operate in both IPv4 and IPv6 network environments; it can receive and send messages to both types of networks. Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing: over messages per second when receiving messages from a single connection and storing them in text files; over messages per second when receiving messages from a single connection and storing them in logstore files; over messages per second when receiving messages from multiple connections and storing them in text files; over messages per second when receiving messages from multiple connections and storing them in logstore files; over messages per second when receiving messages from secure (TLS-encrypted) connections and storing them in text files What syslog-ng is not The syslog-ng application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages Why is syslog-ng needed? Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons. The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps you to select only the really interesting messages, and forward them to a central server. Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process. For details on how can you use syslog-ng to comply with various regulations, see the Regulatory compliance and system logging whitepaper available here 1.4. What is new in syslog-ng Premium Edition 5 F3? For details on the news and highlights of syslog-ng Premium Edition 5 F3, see the What is new in syslog-ng Premium Edition 5 F3. 2

23 Who uses syslog-ng? For details on changes in The syslog-ng Premium Edition 5 F3 Administrator Guide, see Section 6.1.1, Version 5 F2-5 F3 (p. xviii) Who uses syslog-ng? The syslog-ng application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using syslog-ng is particularly advantageous for: Internet Service Providers; Financial institutions and companies requiring policy compliance; Server, web, and application hosting companies; Datacenters; Wide area network (WAN) operators; Server farm administrators Public references of syslog-ng Premium Edition Among others, the following companies decided to use syslog-ng PE in their production environment: Air France Coop Denmark DataPath, Inc. (Read Case Study) Facebook Hush Communications Canada Inc. Tecnocom Espana Solutions, S.L. (Read Case Study) Telenor Norge AS (Read Case Study) 1.6. Supported platforms The syslog-ng Premium Edition application is officially supported on the following platforms. Note that the following table is for general reference only, and is not always accurate about the supported platforms and options available for specific platforms. The latest version of this table is available at Unless explicitly noted otherwise, the subsequent releases of the platform (for example, Windows Server 2008 R2 and its service packs in case of Windows Server 2008) are also supported. x86 x86_64 SPARC SPARC64 PowerPC IA64 AIX AIX CentOS CentOS

24 Supported platforms x86 x86_64 SPARC SPARC64 PowerPC IA64 CentOS Debian 6 (squeeze) Debian 7 (wheezy) FreeBSD FreeBSD FreeBSD HP-UX 11i v HP-UX 11i v Oracle Linux Oracle Linux opensuse Red Hat EL Red Hat EL Red Hat EL SLES Solaris Solaris Ubuntu LTS (Lucid Lynx) Ubuntu LTS (Precise Pangolin) Ubuntu LTS (Trusty Tahr) Windows Server Windows Server Windows Server Windows Vista Windows

25 Limitations on Microsoft Windows platforms x86 x86_64 SPARC SPARC64 PowerPC IA64 Windows Table 1.1. Platforms supported by syslog-ng Premium Edition Warning Certain sources and destinations are not supported on every platform (particularly, the sql() source and destination, and the hdfs() destination). For details, see the description of the specific source and destination. The central syslog-ng PE server can be installed on Microsoft Windows platforms as well, in this case you can configure syslog-ng PE using a configuration file, like on any other platform. However, if you want only to forward eventlog and other log messages from Windows to your central logserver, you can use the syslog-ng Agent for Windows application. The syslog-ng Agent for Windows can be managed centrally from a domain controller, and can be configured from a graphical interface as well. The syslog-ng Agent for Windows application is available as part of syslog-ng Premium Edition. For details about the syslog-ng Agent for Windows application, see The syslog-ng Agent for Windows 5 F3 Administrator Guide Limitations on Microsoft Windows platforms The following features and options of syslog-ng PE are not available on Microsoft Windows platforms. IPv6 is not supported, only IPv4 The mongodb() destination The pipe() source The pipe() destination The program() source The program() destination The snmp() destination The sql() source The sql() destination The sun-streams() source The unix-dgram() source The unix-dgram() destination 5

26 Certified packages The unix-stream() source The unix-stream() destination Certified packages Starting from version 4.0, syslog-ng Premium Edition is Novell Ready certified for the following platforms: SUSE Linux Enterprise Server 10 on the x86 and x86_64 AMD64 & Intel EM64T architectures SUSE Linux Enterprise Server 11 on the x86 and x86_64 AMD64 & Intel EM64T architectures Starting from version 4.0, syslog-ng Premium Edition is RedHat Ready certified for the following platforms: Red Hat Enterprise Linux 2.1 on the x86 architecture Red Hat Enterprise Linux 3 on the x86_64 AMD64 & Intel EM64T architecture Red Hat Enterprise Linux 4 on the x86 and x86_64 AMD64 & Intel EM64T architectures Red Hat Enterprise Linux 5 on the x86 and x86_64 AMD64 & Intel EM64T architectures Red Hat Enterprise Linux 6 on the x86 and x86_64 AMD64 & Intel EM64T architectures 6

27 The philosophy of syslog-ng Chapter 2. The concepts of syslog-ng This chapter discusses the technical concepts of syslog-ng The philosophy of syslog-ng Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices called syslog-ng clients all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, which sorts and stores them Logging with syslog-ng The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources. Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations. Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving from a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement. Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path. Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages Procedure The route of a log message in syslog-ng Purpose: The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server. 7