TIBCO LogLogic. Universal Collector (UC) User Guide. Software Release: November 2012

Transcription

1 TIBCO LogLogic Universal Collector (UC) User Guide Software Release: November 2012

2 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE. USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE LICENSE FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME. This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc. TIBCO and LogLogic are either registered trademarks or trademarks of TIBCO Software Inc. and/or subsidiaries of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only. THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. PLEASE SEE THE README.TXT FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM. THIS DOCUMENT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME. THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES. Copyright TIBCO Software Inc. ALL RIGHTS RESERVED. TIBCO Software Inc. Confidential Information

3 Contents Contents 3 List of Figures 7 List of Tables 9 Preface About This Guide 11 Audience Technical Support Information Documentation Support Information Contact Information Conventions Chapter 1 Introduction and Requirements 13 Introduction Aims and Assets of Universal Collector Requirements General Security Considerations Supported Platforms Minimum Hardware Configuration Ports Limitations JRE Documentation Chapter 2 Install UC 19 Recommendations Install Universal Collector in Graphical Mode Start Graphical Mode Install Universal Collector Uninstall Universal Collector Install Universal Collector in Console Mode Start Console Mode Install Universal Collector Uninstall Universal Collector Install Universal Collector in Silent Mode Start Silent Mode Install Universal Collector Uninstall Universal Collector Universal Collector User Guide 3

4 Chapter 3 Collect Logs 29 About the Log Sources Real-Time File Logs Windows Event Logs Syslog Logs Remote Files UC Internal Logs Create and Configure Log Sources Create a Log Source Create Several Log Sources at a Time Create a Complete Configuration Edit Log Sources Edit a Real-Time File Log Source Edit Several Real Time File Log Sources Edit a Windows Event Log Source Edit Several Windows Event Log Sources Edit a Syslog Log Source Edit Several Syslog Log Sources Edit a Remote File Log Source Edit Different Log Sources at a Time Sorting Log Sources Create Tags Sort Log Sources Chapter 4 Forward Logs 69 Create a TCP (Syslog) or UDP (Syslog) Connection Create an LMI Connection Create a Connection in Authentication and/or Encryption Mode Step 1 - Get a Root Certificate Authority from your PKI Step 2 - Create a Certificate Signing Request Step 3 - Create a valid UC certificate using a CA and OpenSSL Step 4 - Import the Certificate into *.ks or *.p Step 5 - Configure the Forwarding Process Step 6 - Enable Secure Connection Manage the list of Forwardings Graphical User Interface Overview Copy a Forwarding Delete a Forwarding Chapter 5 Monitor UC Activities 83 Start UCMon Tool Summary Screen Status Screen Log Source Forwarding Connection Universal Collector User Guide

5 Metrics Screen Log Source Forwarding Connection Trends Screen Log Source Forwarding Connection RealTime Screen Log Sources Forwarding Connection Chapter 6 Command Line Interface 95 Overview cert_mgt: Manage the Security Certificates uc_checkconf: Check the Current Configuration uc_createlogsources: Import and Create Several Log Sources at a Time uc_decodepwd: Decode Passwords for Windows Files uc_encryptpwd: Encrypt Passwords for Windows Files uc_monitor: UCMon Tool uc_reload: Reload Configuration uc_saveactiveconfas: Save an Active Configuration uc_switchto: Make Configuration Active Switching from One Configuration to Another Checking the Impacted Processes Limitations Appendix 103 Sample Configuration Files [UC Configuration] uc.xml [LMI Connection] uldp-samplecommented.uldp.xml [LMI Connection] uldp-samplecommentedauthjks.uldp.xml [LMI Connection] uldp-samplecommentedauthpem.uldp.xml [LMI Connection] uldp-samplecommentedauthpks12.uldp.xml [Log Sources] file-samplecommented.ls.xml [Log Sources] syslog-samplecommented.ls.xml [Log Sources] wmi-samplecommented.ls.xml Regular Expressions Index 1 Universal Collector User Guide 5

6 6 Universal Collector User Guide

7 List of Figures Figure 1: Universal Collector Principle Figure 2: TCP port required for Windows log collection Figure 3: Launch and Activation Permission Figure 4: Windows Event Logs Control Properties Figure 5: Windows Event Logs Control Properties Figure 6: RT File Edition tab Figure 7: Windows Event Log Edition tab Figure 8: Syslog Edition tab Figure 9: All tab Figure 10: Two filters are applied Figure 11: Two values are applied Figure 12: Overview of the creation of a UC certificate Figure 13: Generation of a Root CA Figure 14: Creation of a valid certificate Figure 15: The three supported formats Figure 16: LMI Connection Universal Collector User Guide 7

8 8 Universal Collector User Guide

9 List of Tables Table 1: Supported platforms Table 2: Minimum configuration Table 3: Ports Table 4: MC Agent Ports Table 5: Types of installation recommended Table 6: Silent mode - Values to modify Table 7: Multi-line log sources supported by default Table 8: Audit Policy Table 9: Protocols Table 10: CSV fields Table 11: CSV Example Table 12: General Settings Edition Table 13: Real-Time File edition parameters - General Table 14: Real-Time File edition parameters - Forwarding Connection Table 15: Real-Time File edition parameters - Message Filtering Table 16: Real-Time File edition parameters - Collection Table 17: Windows Event Log edition parameters - General Table 18: Windows Event Log edition parameters - Forwarding Connection Table 19: Windows Event Log edition parameters - Message Filtering Table 20: Windows Event Log edition parameters - Collection Table 21: Syslog edition parameters - Description Table 22: Syslog edition parameters - Forwarding Connection Table 23: Syslog edition parameters - Collection Table 24: Syslog edition parameters - Message Filtering Table 25: Remote File edition parameters - Description Table 26: Remote File edition parameters - Forwarding Connection Table 27: Remote File edition parameters - Collection Table 28: All parameters - General Table 29: All parameters - Forwarding Connection Table 30: Filters Table 31: Forwarding Connection edition - Security Table 32: Forwarding GUI Table 33: UCMon - Summary screen Table 34: UCMon - Log Source Status Table 35: UCMon - Forwarding Status Table 36: UCMon - Log Source Metrics Table 37: UCMon - Forwarding Connection Metrics Table 38: UCMon - Log Source Trends Table 39: UCMon - Forwarding Connection Trends Table 40: UCMon - RealTime Table 41: UCMon - Forwarding Connection RealTime Table 42: Starting the CLI Table 43: cert_mgt Table 44: uc_checkconf Universal Collector User Guide 9

10 Table 45: uc_createlogsources Table 46: Command Line Interface to decode passwords Table 47: Command Line Interface to encrypt passwords Table 48: Command Line Interface Table 49: uc_reload Table 50: uc_saveactiveconfas Table 51: uc_switchto Universal Collector User Guide

11 Preface About This Guide This User Guide provides the information needed to quickly install, configure, and uninstall the Universal Collector (UC), a lightweight but powerful software agent that collects event logs locally or remotely by acting as a Syslog listener or file collector. Audience The guide is intended for Security Network Administrators who are responsible for installing and maintaining network security software. Technical Support Information LogLogic is committed to the success of our customers and to ensuring our products improve customers' ability to maintain secure, reliable networks. Although LogLogic products are easy to use and maintain, occasional assistance might be necessary. LogLogic provides timely and comprehensive customer support and technical assistance from highly knowledgeable, experienced engineers who can help you maximize the performance of your LogLogic Compliance Suites. To reach LogLogic Customer Support: Telephone: Toll Free LOGS Local EMEA support@loglogic.com You can also visit the LogLogic Support website at: When contacting the Support, be prepared to provide the following information: Your name, address, phone number, and fax number Your company name and company address Your machine type and release version A description of the problem and the content of pertinent error messages (if any) Universal Collector User Guide 11

12 Documentation Support Information Documentation Support Information The LogLogic documentation includes Portable Document Format (PDF) files. To read the PDF documentation, you need a PDF file viewer such as Adobe Acrobat Reader. You can download the Adobe Acrobat Reader at Contact Information Your feedback on the LogLogic documentation is important to us. If you have questions or comments, send to DocComments@loglogic.com. In your message, please indicate the software name and version you are using, as well as the title and document release date of your documentation. Your comments will be reviewed and addressed by the LogLogic Technical Publications team. Conventions The LogLogic documentation uses the following conventions to distinguish text and information that might require special attention. Caution: Highlights important situations that could potentially damage data or cause system failure. IMPORTANT! Highlights key considerations to keep in mind. Note: Provides additional information that is useful but not always essential or highlights guidelines and helpful hints. This guide also uses the following typographic conventions to highlight code and command line elements: Monospace is used for programming elements (such as code fragments, objects, methods, parameters, and HTML tags) and system elements (such as file names, directories, paths, and URLs). Monospace bold is used to distinguish system prompts or screen output from user responses, as in this example: username: system home directory: home\app Monospace italic is used for placeholders, which are general names that you replace with names specific to your site, as in this example: LogLogic_home_directory\upgrade\ Straight brackets signal options in command line syntax. ls [-AabCcdFfgiLlmnopqRrstux1] [-X attr] [path...] 12 Universal Collector User Guide

13 CHAPTER 1 Introduction and Requirements Contents General Security Considerations Supported Platforms Minimum Hardware Configuration Ports Limitations Documentation Introduction The Universal Collector is a software agent that collects logs from any device or log source and forwards them to an LMI Appliance or a Syslog server by using UDP, TCP, or the Universal Log Data Protocol proprietary protocol. UC collects the information from four types of log sources: Syslog, Windows event logs, Real-Time File pull, or Remote File pull. Several UC agents can be deployed on a dedicated/shared appliance or physical/virtual hardware to remotely collect hundreds of log sources located at the same site. Aims and Assets of Universal Collector Collecting Logs: UC allows you to gather data from several types of log sources while ensuring confidentiality of the logs. You can easily collect event logs from local or remote instances of MS Windows, including time-stamped or rotated files. The UC agent works as a Syslog listener. Forwarding Logs: UC forwards secure and authenticated data to an LMI server via a ULDP protocol without the need for a dedicated appliance. UC 2.3 also forwards to Syslog server with either UDP or TCP protocols. Monitoring Activity: A UCMon tool is also available to monitor the internal process of the UC and therefore make sure that your collection and forwarding processes are correctly responding. Universal Collector User Guide 13

14 Introduction Easy and Wise Configuration: A UC configuration is composed of Log Sources, Forwarding connections and UC general parameters. UC configuration must be created and updated via the GUI or the Command Line Interface. You can create, save and store a configuration. A stored configuration is useful: - to create a configuration and then activate it whenever you want, even if an active configuration is open, i.e. another configuration is running on the system - to create several configurations and deploy them rapidly on other UCs. Easy Management: Multiple UCs can be remotely managed using LogLogic Management Center (MC) and MC Agent configured and running on each UC Asset. MC is a software solution that allows you to manage Assets, schedule batch upgrade for Assets, monitor system health check, and backup and restore Asset data. Adaptability: UC is a software program with a small footprint and low memory usage on your Domain Controllers, or application services. It is highly adaptable and can be customized easily. Its lightweight although reliable configuration helps the user to manage changes according to his particular needs. Figure 1 Universal Collector Principle 14 Universal Collector User Guide

15 CHAPTER 1 Introduction and Requirements Requirements General Security Considerations The following section provides security recommendations that need to be followed to install the Universal Collector in a secure manner. Full security The administrator must ensure that the machines on which Universal Collectors are installed are fully secured. These machines should be located in a physically secure environment in which only trusted personnel have access. Password The administrator must ensure that the default admin password to connect to the machine is changed. A good password has a combination of alphabetic and numeric characters and comprises at least eight characters in length. It should be known by a very restricted number of people. Updates All operating systems and software installed on the machine must be correctly updated to avoid any security breach which could expose admin rights. Supported Platforms The machine where the Universal Collector is installed must be safe and secured by a reliable administrator. It must also be synchronized according to a NTP server (for reliable time management). Table 1 Supported platforms Type Microsoft Windows Linux/Unix Operating Systems Microsoft Windows 2003 R2 32-bit and 64-bit Microsoft Windows bit and 64-bit Microsoft Windows 2008 R2 64-bit Microsoft Windows 7 32-bit and 64-bit Redhat Enterprise Linux (RHEL) v5 32-bit and 64-bit Redhat Enterprise Linux (RHEL) v6 32-bit and 64-bit SUSE Linux Enterprise Server (SLES) v11 32-bit and 64-bit Solaris 10 x86 64-bit Solaris 10 Sparc 64-bit Solaris 11 x86 64-bit Solaris 11 Sparc 64-bit Note: With Solaris x86_64, UC will run only in 32 bit mode, even under 64 bit jvm. Universal Collector User Guide 15

16 Requirements Minimum Hardware Configuration Table 2 Minimum configuration Type Multi-core CPU Disk Space RAM Minimum Configuration 2GHz min. 5 GB 2 GB for local collection (Agent mode) or 4 GB for remote Windows collection (Collector mode) Note: Once UC Console is installed, you can check the memory used by UC service compared to the maximum memory size allocated to UC in the About screen (Help > About) as well as the version and revision number. Ports The following section describes the ports used by UC and the MC Agent. UC To use the Universal Collector along with the LMI, you will need to open specific TCP ports according to the versions of LMI currently in use. Table 3 Ports Ports Use 5514 For non-authenticated and non-encrypted connection with LMI 5.0 and For authenticated and/or encrypted Secure connection with LMI 5.0 or later (although default can be configured) 5516 For non-authenticated and non-encrypted connection with LMI 5.2 or later Make sure that the UC - LMI network connection is not blocked by a firewall. MC Agent The MC Agent communicates with Management Center about the Asset's features. To use the Universal Collector along with the MC, you will need to open specific TCP ports that are used by the MC Agent. Table 4 Ports MC Agent Ports Use Port number used by MC Agent to notify MC about changes RMI Server Port number. The default value is This must be a number between 1024 to RMI Registry Port number. The default value is This must be a number between 1024 to Make sure that the MC Agent - LMI network connection is not blocked by a firewall. 16 Universal Collector User Guide

17 CHAPTER 1 Introduction and Requirements Limitations UC 2.3 will not provide large scale enabled management software for the standalone UCs as it is assumed that in large scale enterprise environments IT has already implemented scalable solutions for deploying and monitoring software applications. The minimum resolution to display UC Console is 1024x768. The installer requires JRE 7 to be installed on the system, and also the java executable must be in the user's PATH. In Linux it is recommended to install the JRE using an RPM (SuSE, RedHat) so that java is automatically put in the user's path. JRE Before installing UC 2.3.0, you must have JRE 7u5 or later installed on your machine. Older versions of JRE 7 are not supported. On Windows: At startup, UC will attempt to discover JRE from the system PATH. If you install the JRE on Windows using the JRE installer, then you should not experience any problems as long as you install it using the installer defaults and do not change the location where the JRE is to be installed. On a Linux system: In Linux it is recommended to install the JRE using an RPM (SuSE, RedHat). Note: If you make any changes to the location where you install the JRE, you should restart the service(s). On Windows you should run UC_HOME\bin\uninstall-service.bat and then UC_HOME\bin\install-service.bat to ensure that the service has the correct environment. On Linux and Solaris you should run UC_HOME\bin\uninstall-service and then UC_HOME\bin\install-service. Documentation The PDF documentation is available in the doc folder contained in the installation directory. The Online Help is accessible from the UC Console via the Help menu. The Online Help is only supported by the latest versions of Internet Explorer and Mozilla Firefox. Universal Collector User Guide 17

18 Requirements 18 Universal Collector User Guide

19 CHAPTER 2 Install UC Contents Recommendations Install Universal Collector in Graphical Mode Install Universal Collector in Console Mode Install Universal Collector in Silent Mode Recommendations According to your needs, you may decide to install UC on one or several machines. In all cases, make sure you have administrator rights on the machine where you want to install UC. Table 5 Types of installation recommended Type of installation Graphical mode Console mode Silent mode Recommended if... You want to install UC on a sole machine on a Windows platform. You want to install UC on a sole machine on a Red Hat, SuSE Linux Enterprise platform or Solaris. You want to rapidly deploy UCs on several machines with no interaction with your machine. Note: Upgrading UC from a 32-bit to a 64-bit platform and vice-versa is not supported. Universal Collector User Guide 19

20 Install Universal Collector in Graphical Mode Install Universal Collector in Graphical Mode First, you must modify the User Account Control settings. To do so: 1. Go to Control Panel > System and Security > click on Change User Accounts Control Settings. 2. In the User Account Control Settings screen, move the slider to the Never notify position and click OK. 3. If you are asked to enter an administrator password, enter it and confirm. Start Graphical Mode To start the Graphical Mode on MS Windows: 1. Log in as Local Administrator. There is no need to have Domain Administrator rights, a Local Administrator account level is enough to run the setup program. 2. Run UC setup program. An installation screen is displayed. To start the Graphical Mode on Red Hat or SuSE Linux Enterprise or Solaris: 1. Connect to the machine as user root. 2. Set the permission to access to the installer folder with the command: chmod 755 <installer file> 3. Launch the Universal Collector executable file and enter: loglogic-uc setup-unix.bin -i gui Install Universal Collector Note: Microsoft Visual C Redistributable SP1ATL will be automatically installed on your computer. This is the main component to make the UC run so do not uninstall it while the UC is running. 1. Read the Introduction screen and click Next. 2. Read the License Agreement screen then click the option to accept the license and click Next. 3. If necessary, change the installation folder path in the Choose Install Folder and click Next. 4. Enter the absolute path of the data folder in the Get User Input screen. The Data Folder will contain: The spool file with all the collected logs (default size is 100MB) Metadata about collected logs (to restore log collection) LogLogic Universal Collector internal logs 20 Universal Collector User Guide

21 CHAPTER 2 Install UC Note: A file spool of 100 MB is dedicated to each LMI connection. Therefore, a minimum of 200 MB of disk space is required to install the UC with one LMI connection. 5. Select Configure MC Agent if you want to Configure MC Agent now. MC is a product which enables you to centrally monitor health and status of UC Assets and perform upgrade tasks. If you do not intend to install MC, then you do not need to configure the Agent and click Next to go directly to the end of the installation process. 6. In the Get User Input - MC Agent Configuration screen, provide the following values and click Next: In the Management Center Host field, enter the IP Address of MC. This enables MC to manage its Agents and for each MC Agent to send information about its assets and health to MC. In the MC Notification Port field, enter the the port MC Agents use to communicate with Management Center. In the Repositories field, enter the URL that the MC Agent will use to discover upgradeable artifacts. 7. In the Get User Input - MC Agent Advanced Configuration screen, provide the following values and click Next: In the Server Port field, enter the server port used by the MC Agent for monitoring. In the Registry Port field, enter the registry port used by the MC Agent for monitoring. 8. In the Verify MC Agent Settings screen, check if all the data are correct and click Next. Note: The MC Agent is always installed in the mcagent directory located in your installation folder path. For more information on MC and MC Agent, including configuring and verifying MC Agent Settings after installation of MC, as well as instructions on installing the Agent as a service, refer to the MC documentation. 9. Keep the Launch UC Console selected if you want to start the UC Console as soon as the installation process. Otherwise deselect it and click Next. 10. When the installation is complete, click Done. If the installation has not been completed successfully, open the Universal_Collector_Install_[Timestamp].log to check error messages. This file is in <UC_HOME>\logs.A wrapper.log is also created when the UC service starts. Note: For security reasons, only the administrator group, the file owner and the system group are allowed to access this folder. Uninstall Universal Collector To uninstall UC on Windows: 1. On the Windows host, log in as Local Administrator. There is no need to have Domain Administrator rights, a Local Administrator account level is enough to run the uninstall program. Universal Collector User Guide 21

22 Install Universal Collector in Graphical Mode 2. Display the Uninstall window by going to Program Files > LogLogic > Universal Collector > click on Uninstall. 3. In the Uninstall Universal Collector screen, click on Uninstall. 4. Click on the Done button to close the window. To uninstall UC on Red Hat, SuSE Linux Enterprise: 1. Connect to the machine as user root. 2. Go to the installation folder and enter the following command:./uninstaller/uninstall 3. Follow the procedure for Windows. 22 Universal Collector User Guide

23 CHAPTER 2 Install UC Install Universal Collector in Console Mode Start Console Mode To start the Console Mode on Windows: 1. Connect to the machine as local administrator. 2. Open a command prompt. 3. From the installation folder, enter the command: loglogic-uc setup-windows.exe -i console To start the Console Mode on Red Hat or SuSE Linux Enterprise: 1. Connect to the machine as user root. 2. Set the permission to access to the installer folder with the command: chmod 755 <installer file> 3. From the installation folder, launch the Universal Collector executable file: loglogic-uc setup-unix.bin Install Universal Collector 1. Read the introduction and press Enter until you are asked to accept the license. 2. Press y (for yes) to accept the license and press Enter. 3. Enter the path to the UC Installation directory, then press Enter. 4. The default directory for data storage is usually appropriate, but it can be changed. Press Enter. 5. The default Data Folder is usually appropriate, but it can be changed. It will contain: the spool file containing all collected logs (default size is 100MB) metadata about collected logs (in order to restore log collection) Universal Collector internal logs 6. Verify the capacity of the directory in which to store collected data and click Next. 7. Type the number 2 if you want to configure the MC Agent now. MC is a product which enables you to centrally monitor health and status of UC Assets and perform upgrade tasks. If you do not intend to install MC, then you do not need to configure the Agent and click Next to go directly to the end of the installation process. Universal Collector User Guide 23

24 Install Universal Collector in Console Mode 8. In the Get User Input - MC Agent Configuration, provide the following values then press Enter. In the Management Center Host field, enter the IP Address of MC. This enables MC to manage its Agents and for each MC Agent to send information about its assets and health to MC. In the MC Notification Port field, enter the the port MC Agents use to communicate with Management Center. In the Repositories field, enter the URL that the MC Agent will use to discover upgradeable artifacts. 9. In the Get User Input - Servert Port, enter the enter the server port used by the MC Agent for monitoring. 10. In the Get User Input - Server Registry Port Registry Port, enter the registry port used by the MC Agent for monitoring. 11. In the Verify MC Agent Settings screen, check if all the data are correct and click Next. Note: The MC Agent is always installed in the mcagent directory located in your installation folder path. For more information on MC and MC Agent, including configuring and verifying MC Agent Settings after installation of MC, as well as instructions on installing the Agent as a service, refer to the MC documentation. 12. Click Install in the Pre-Installation Summary screen and wait until the installation process has finished. 13. An Installation Complete prompt appears. Press Enter to validate the installation. If the installation has not been completed successfully, open the Universal_Collector_Install_[Timestamp].log to check error messages. This file is located in <UC_HOME>/logs. Note: For security reasons, only the root account and the file owner are allowed to access this folder. Uninstall Universal Collector To uninstall UC on Windows: 1. From the uninstall folder, enter the following command: uninstall.exe -i console To uninstall UC on Red Hat or SuSE Linux Enterprise: 1. Connect to the machine as user root. 2. Go to the installation folder and enter the following command:./uninstaller/uninstall 3. Press Enter and decide whether you want to keep configuration data or not after the uninstallation process by selecting the relevant option. 24 Universal Collector User Guide

25 CHAPTER 2 Install UC 4. Press Enter. The uninstallation is in progress. Universal Collector User Guide 25

26 Install Universal Collector in Silent Mode Install Universal Collector in Silent Mode Start Silent Mode The silent mode is recommended when you decide to rapidly deploy UCs on several machines. You must create an installation file in which you must enter the following values: Table 6 Silent mode - Values to modify Values Description INSTALLER_UI Indicate the installation mode, e.g. silent Note: you can also indicate another type of installation mode such as console or gui. USER_INSTALL_DIR Indicate the UC s installation directory, e.g. /opt/ LogLogic/Universal_Collector USER_INPUT_INSTALL_DATA_DIR Indicate the UC s data directory, e.g. /opt/loglogic/ Universal_Collector configuremcagent Set this value to '1' to tell the installer you want the MC Agent configured and installed as a service. mchost Indicate the MC Host. Default value: but it should be changed to the IP address where MC is running. mcnotificationport Indicate the MC Notification Port. Default value: rmiserverport RMI Server Port. Default value: rmiregistryport RMI Registry Port. Default value: agentaddress Set IP address of the MC Agent. Default value: empty string. This field must be empty unless the MC Agent is hosted on a multi-homed host. enableheartbeat Default value: true. heartbeatinterval Default value: 720. org.ops4j.pax.url.mvn.repositor ies Set the location of repositories (to have the MC Agent point to multiple repositories in case the first one in the list is not available.) Install Universal Collector To install on Windows: 1. Create the installation file, e.g. silent.txt. 2. Enter the installation information: INSTALLER_UI=silent 26 Universal Collector User Guide

27 CHAPTER 2 Install UC USER_INSTALL_DIR=C:\\Program Files\\LogLogic\\Universal Collector USER_INPUT_INSTALL_DATA_DIR=C:\\Program Files\\LogLogic\\Universal Collector 3. Configure MC Agent if you want to manage UC using MC: configuremcagent=1 mchost=mc.host.loglogic.com mcnotificationport= Start the installation: C:\Temp\UC>loglogic-uc setup-windows.exe -f silent.txt 5. The installation is complete. 6. If the installation has not been completed successfully, open the Collector_install.log to check error messages. This file is by default located in C:\Program Files\LogLogic\Universal Collector\<DATA DIR>\data\logs. To install on Red Hat or SuSE Linux Enterprise: 1. Open a Shell program and login as root. 2. Set the permission to access to the installer folder with the command: chmod 755 <installer file> 3. Create and complete the installation file, e.g. silent.txt: 4. Start the installation: bash-3.2# cat silent.txt INSTALLER_UI=silent USER_INSTALL_DIR=/opt/LogLogic/uc USER_INPUT_INSTALL_DATA_DIR=/opt/LogLogic/uc configuremcagent=1 mchost=x.x.x.x mnotificationport=41616 org.ops4j.pax.url.mvn.repositories= bash-3.2# sh./loglogic-uc setup-unix.bin -f silent.txt 5. The installation is complete. Uninstall Universal Collector To uninstall on Windows: 1. In the silent.txt you have created when installing UC, enter the uninstall information: INSTALLER_UI=silent 2. Go to uninstall folder: C:\Program Files\LogLogic\Universal Collector\uninstaller Universal Collector User Guide 27

28 Install Universal Collector in Silent Mode 3. Start the uninstall: uninstall.exe 4. The uninstall is complete. To install on Red Hat or SuSE Linux Enterprise: 1. Open a Shell program and login as root. 2. In the silent.txt you have created when installing UC, enter the uninstall information: INSTALLER_UI=silent 3. Go to the installation folder: /opt/loglogic/universal_collector/uninstaller 4. Start the uninstall:./uninstall 5. The uninstall is complete. 28 Universal Collector User Guide

29 CHAPTER 3 Collect Logs Contents About the Log Sources Create and Configure Log Sources Edit Log Sources Sorting Log Sources About the Log Sources Real-Time File Logs The Universal Collector reads logs from local files - i.e. logs from files generated on the machine where UC is installed - and forwards them to either an LMI or a Syslog server. This chapter describes how Universal Collector handles file collection from files: - Collection of Single-line Messages - Where does collection starts? - What is collected? - What is forwarded? - Are messages lost when UC restarts? - Log File Rotation Collection of Multi-line log messages Universal Collector User Guide 29

30 About the Log Sources Collection of Single-line Messages Where does collection starts? When a file is collected, only the newly added logs at the end of the file are collected. Logs already available in a file before the UC log source creation will not be collected. What is collected? UC operates by monitoring specified text files that are receiving log output from log sources. The log sources append new logs to the end of the text file as events occur. As new records appear at the tail of the monitored file, they are instantly taken into account by UC. What is forwarded? UC forwards single-line log messages to an LMI or Syslog server. By default, UC sends a maximum of characters per line. Are messages lost when UC restarts? UC uses cursors to track the monitored files and to resume continuously after having stopped. 30 Universal Collector User Guide

31 CHAPTER 3 Collect Logs The cursors have information about the file positions at which to restart - called metadata - as well as file identification information. It can determine whether the file to be resumed is the file to which the saved position applies. In other terms, even if the UC is stopped for a while, all the messages contained in the file will be collected thanks to the position cursors, no messages will be lost. Log File Rotation Log file rotation is not managed by UC; it is the user's responsibility to manage the file rotation. In the case of log file rotation, a log file is retired and renamed to a rotated name, and the monitored file is replaced by a new log file. Therefore, periodically during the monitoring of a log file that is rotated, the file is replaced by a fresh log file. Depending on your log rotation tool, UC is able to manage rotation files in two different ways. 1) The log file name contains a date that changes during the rotation process The UC handles the rotation process of logs that contain a date in their name provided you correctly configured the File Log Source configuration file. If you enter the parameter [date] in the file path you must: 1. Activate the file rotation. Enable and enter a date format for the date pattern such as yyyymmdd (see docs.oracle.com/javase/7/docs/api/java/text/simpledateformat.html ) Example with yyyymmdd: Filenames: logfile log, logfile log... Absolute path: c:\logdir\logfile.[date].log 2) The log file name contains an id that changes during the rotation The UC handles the rotation process of logs that contain an Id in their name provided you correctly configured the File Log Source configuration file. Universal Collector User Guide 31

32 About the Log Sources If you enter the parameter [id] in the file path you must: 1. Activate the file rotation. 2. Enable and enter the number of digits expected (1-9) for the nbdigit parameter. Example with only one digit: Filenames: logfile.1.log, logfile.2.log... Absolute path: c:\logdir\logfile.[id].log You can combine the two examples to allow the use of both [id] and [date] parameters in the file path. Recommendations - In the case of resuming after having been stopped, if the log file has been rotated during the period in which the collector was stopped, some log data will be missed. Therefore, you must ensure that the collector is not temporarily stopped during an interval in which a rotation occurs. - To be collected, a file must have been modified after the latest collected file. - The log file name does not change during the rotation. The UC records the identity of a log file in the cursor as a hash of the first several bytes of the file. When the file is rotated and replaced with a fresh one, the hash will be different. File identity checking is performed throughout the log file monitoring process to detect log rotation. - If a log file needs to be replaced and enriched while UC is running, do not copy content in the file but move it on the same partition. Collection of Multi-line log messages The Universal Collector can combine multiple consecutive related lines or multi-lines in a source log file into a single line which will be sent to the LMI. Multi-line message groups may require analysis to determine the correct expression to use if the format is complex. Before sending, groups of lines that represent a logical message are converted to a single-line format. All of the original messages' data is kept intact nothing is altered. UC can collect multi-line messages from default application sources or custom ones: Table 7 Multi-line log sources supported by default Log Sources Tomcat / Servlet Container WebLogic Application Server Description Default log location is CATALINA_BASE/logs. Tomcat and application logs unless configured otherwise. The default format is multi-line, with the first line beginning with a timestamp. It may change due to localization. Logs are rotated daily by default Default log location is under the server root DOMAIN_NAME/servers/ ADMIN_SERVER_NAME/logs/. Each server or cluster maintains a server log and selected events are forwarded to a domain log. Most of the entries are single line, but can contain java exceptions. Each message begins with '####'. There may also be a web access log 32 Universal Collector User Guide

33 CHAPTER 3 Collect Logs Table 7 Multi-line log sources supported by default Log Sources WebSphere Application Server JBoss Application Server Description Default log location is under the WebSphere directory APPSERVER/profiles/PROFILENAME/ logs/servername/. There is no default log rotation. There are server start and stop logs (SystemErr.log, SystemOut.log), JVM log files (native_stderr.log, native_stdout.log), and process log files (startserver.log, stopserver.log). All of these logs contain entries describing the system environment that do not have a timestamp. The error logs do not contain any timestamps. Continuation lines are indented Default log location is JBOSS_HOME/server/NAME/log. The boot log records startup events prior to the initialization of the logging service. The server.log file records activity while the server is running. The boot.log file entries begin with a time with no date. The server.log file entries start with a timestamp in the form 'YYYY-MM-DD HH:MI:SS,FFF'. Log messages can be multi-line and the continuation lines are sometimes indented, but frequently not. Messages start with a timestamp. Note: The regex format for these default applications are indicated in <InstallationFolder>\runtime\conf\static\line_combiner.xml file. Custom multi-line: Custom regex can be defined for custom multiline logs. You need to define - the header regex pattern. - whether you keep orphaned lines, i.e UC sends messages that do not match the Header Regexp - the timeout after which messages are sent even if the regex is not found again. Custom Multi-line log sources: Custom regex can be defined for custom multiline logs. You need to define - the header regex pattern. - whether you keep orphaned lines, i.e UC sends messages that do not match the Header Regexp - the timeout after which messages are sent even if the regex is not found again. Example of a custom application logs: :09:41,344 WARN [main] file.fileimportsqldao (?(think)) - File not found (/home/exaprotect/conf/tbsmp6/report/ etc/export.properties) :09:41,344 WARN [main] config.configurationfactory (ConfigurationFactory.java:127) - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/home/exaprotect/report_tbsmp6/webapps/ ExaReport/WEB-INF/lib/ehcache jar!/ehcache-failsafe.xml java version "1.6.0_18" Java(TM) SE Runtime Environment (build 1.6.0_18-b07) :09:50,723 INFO [main] config.facesconfigurator (FacesConfigurator.java:151) - Reading standard config org/apache/ myfaces/resource/standard-faces-config.xml Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode) Universal Collector User Guide 33

34 About the Log Sources In the UC Console, you can create a regex like: ^\d+-\d+-\d+\s\d+:\d+:\d+,\d+\s[^\s]+\s+\[[^\]]+\]\s with a timeout of 3 seconds and indicating that orphaned lines are kept. It will match the header of the multiline log (date and level), which is: :09:41,344 WARN [main] All the lines will be aggregated and then forwarded as a single log to LMI. The \r and \n will be replaced by escaped ones \\r and\\n), until UC finds another regex header. You can obtain something like: :09:41,344 WARN [main] file.fileimportsqldao (?(think)) - File not found (/home/exaprotect/conf/tbsmp6/report/ etc/export.properties) :09:41,344 WARN [main] config.configurationfactory (ConfigurationFactory.java:127) - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/home/exaprotect/report_tbsmp6/webapps/ ExaReport/WEB-INF/lib/ehcache jar!/ ehcache-failsafe.xml\r\njava version "1.6.0_18"\r\nJava(TM) SE Runtime Environment (build 1.6.0_18-b07) :09:50,723 INFO [main] config.facesconfigurator (FacesConfigurator.java:151) - Reading standard config org/apache/ myfaces/resource/standard-faces-config.xml\r\n Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode) Note: Refer to section Appendix to get the full content of the Real-Time File Log Source commented file. 34 Universal Collector User Guide

35 CHAPTER 3 Collect Logs Windows Event Logs The Universal Collector can collect Windows events. The supported Windows versions for remote collection are Windows 2003 R2 (32-bit/64-bit), Windows 2008 (32/64-bit), Windows 2008 R2 64-bit, Windows 7 (32/64-bit). Note: The Universal Collector forwards Windows logs to the LMI appliance by using an LMI connection. Windows logs collected from UC are forwarded in a format which is based upon the SNARE format although UC and SNARE formats are not 100% similar and a subtle difference may exist for certain messages. Collecting Windows Event Logs in Agent Mode (Windows only) Configuring UC in agent mode for local collection of Windows event logs is straightforward: Create a Windows log source Optionally fine tune the Windows Audit Policy When collecting Windows logs, all log source configuration files (XML files) must refer to the same Windows Journals - although the filters can be different. If Windows Journals are different, the log collection will be duplicated. Collecting Windows Event Logs in Collector mode (Windows, RHEL, SuSE, Solaris) There are 3 main steps to use the Collector mode: Step 1 - Collecting Logs Step 2 - Editing Registry of Remote Windows Log Sources (RHEL, SuSE and Solaris only) Step 3 - Verifying Connection To Remote Windows Log Sources Step 1 - Collecting Logs Firewall filtering when a UC is installed on Windows When a firewall is in the communication path of two Windows hosts, the firewall must be configured to allow communication between the hosts. To do so: 1. Make sure that RPC is allowed by opening the port 135/tcp (EPMAP service) on your firewall. Universal Collector User Guide 35