Data Center Real User Monitoring

Transcription

1 Data Center Real User Monitoring Database Monitoring User Guide Release 12.3

2 Please direct questions about Data Center Real User Monitoring or comments on this document to: Customer Support Copyright 2014 Compuware Corporation. All rights reserved. Unpublished rights reserved under the Copyright Laws of the United States. U.S. GOVERNMENT RIGHTS-Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Compuware Corporation license agreement and as provided in DFARS (a) and (a) (1995), DFARS (c)(1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. Compuware Corporation. This product contains confidential information and trade secrets of Compuware Corporation. Disclosure is prohibited without the prior express written permission of Compuware Corporation. Use of this product is subject to the terms and conditions of the user's License Agreement with Compuware Corporation. Documentation may only be reproduced by Licensee for internal use. The content of this document may not be altered, modified or changed without the express written consent of Compuware Corporation. Compuware Corporation may change the content specified herein at any time, with or without notice. All current Compuware Corporation product documentation can be found at Compuware, FrontLine, Network Monitoring, Enterprise Synthetic, Server Monitoring, Dynatrace Network Analyzer, Dynatrace, VantageView, Dynatrace, Real-User Monitoring First Mile, and Dynatrace Performance Network are trademarks or registered trademarks of Compuware Corporation. Cisco is a trademark or registered trademark of Cisco Systems, Inc. Internet Explorer, Outlook, SQL Server, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation. Firefox is a trademark or registered trademark of Mozilla Foundation. Red Hat and Red Hat Enterprise Linux are trademarks or registered trademarks of Red Hat, Inc. J2EE, Java, and JRE are trademarks or registered trademarks of Oracle Corporation. VMware is a trademark or registered trademark of VMware, Inc. SAP and SAP R/3 are trademarks or registered trademarks of SAP AG. Adobe Reader is a registered trademark of Adobe Systems Incorporated in the United States and/or other countries. All other company and product names are trademarks or registered trademarks of their respective owners. Local Build: December 9, 2014, 4:34

3 Contents Contents Introduction Who Should Read This Guide Organization of the Guide Related Publications Customer Support Information Reporting a Problem Documentation Conventions Chapter 1 Database Monitoring Overview of Database Monitoring Chapter 2 Adding Basic DC RUM Devices Adding an AMD to the Devices List Adding a CAS to Devices List Chapter 3 Verification of Traffic Monitoring Quality Sniffing Point Diagnostics Sniffing Point Diagnostics Reports Network Interface General Statistics Network and Transport Protocol Information Services Detected in the Traffic Session-Related Statistics SSL Diagnostics Application Overview Using RUM Console to Identify Problems Related to Network Hardware Operation... Chapter 4 Basic Monitoring Configuration Configuring General Data Collector Settings Configuring Operation-Related Global Settings Chapter 5 Configuring AMD to Monitor User-Defined Software Services Defining Oracle Forms Software Services Configuring Rules for User-Defined Software Services Excluding IP Ranges from AMD Client Analysis

4 Contents Managing User-Defined Software Services Chapter 6 Configuration Properties Common to All Database Analyzers Chapter 7 Configuration Fine-Tuning Configuring Database Monitoring for Individual Servers Individual Query Monitoring Query Auto-Learning Query Diagnostics Query Aggregation SQL Query Normalization Masking Sensitive Information in SQL Query Literals Configuring Database Availability TDS (Sybase and Microsoft SQL) Errors and Warnings DRDA Errors and Warnings Oracle Errors and Warnings Chapter 8 Monitoring SQL Sequence Transactions Adding SQL Sequence Transactions Filters and Transaction Inspector for SQL Transactions Modifying, Deleting, and Cloning Transactions for a Single AMD Chapter 9 Database Tiers Chapter 10 Database Traffic on CAS Reports Chapter 11 Links to Advanced Diagnostics Server Links to Slow Operation Cause Breakdown Report Appendix A Diagnostics and Troubleshooting Report-Related Issues Appendix B Analyzed Operations Analyzed DRDA Operations Analyzed MySQL Operations Analyzed Oracle Operations Analyzed TDS Operations Appendix C Regular Expression Fundamentals Testing Regular Expressions Best Practices for Regular Expressions Glossary Index

5 INTRODUCTION Who Should Read This Guide This manual is intended for users of Data Center Real User Monitoring who want to monitor the database environment. Organization of the Guide This guide is organized as follows: Overview of Database Monitoring [p. 10] - Describes capabilities of database analyzers and provides an overview of the configuration process. Adding Basic DC RUM Devices [p. 13] - Describes how to add and configure the data sources and report servers using the RUM Console. Verification of Traffic Monitoring Quality [p. 17] - Describes how to verify sniffing points traffic detection quality before the actual monitoring begins. Basic Monitoring Configuration [p. 29] - Explains basic concepts such as software services, global monitoring settings, settings specific to analyzers, and applying settings to the AMD. Configuring AMD to Monitor User-Defined Software Services [p. 37] - Describes the creation and management of database-based software services. Configuration Properties Common to All Database Analyzers [p. 45] - Describes the properties and setting that are common database analyzers. Configuring Database Monitoring for Individual Servers [p. 47] - Explains the configuration of database monitoring including query auto learning and query aggregation. Monitoring SQL Sequence Transactions [p. 63] - Describes how to monitor sequence transactions. Database Tiers [p. 71] - Describes tiers that present data for database analyzers. Links to Advanced Diagnostics Server [p. 75] - Describes database reports available via the ADS link. 5

6 Introduction Diagnostics and Troubleshooting [p. 77] - Lists common support issues in the form of questions and answers. Regular Expression Fundamentals [p. 99] - Describes how to use regular expressions in CAS. Related Publications Documentation for your product is distributed on the product media. For Data Center RUM, it is located in the \Documentation directory. It can also be accessed from the Media Browser. Go online ( for fast access to information about your Dynatrace products. You can download documentation and FAQs as well as browse, ask questions and get answers on user forums (requires subscription). The first time you access FrontLine, you are required to register and obtain a password. Registration is free. PDF files can be viewed with Adobe Reader version 7 or later. If you do not have the Reader application installed, you can download the setup file from the Adobe Web site at Customer Support Information Dynatrace Community For product information, go to and click Support. You can review frequently asked questions, access the training resources in the APM University, and post a question or comment to the product forums. You must register and log in to access the Community. Corporate Website To access the corporate website, go to The Dynatrace site provides a variety of product and support information. Reporting a Problem Use these guidelines when contacting APM Customer Support. When submitting a problem, log on to the Dynatrace Support Portal at click the Open Ticket button and select Data Center Real User Monitoring from the Product list. Refer to the DC RUM FAQ article at to learn know how to provide accurate diagnostics data for your DC RUM components. Most of the required data can be retrieved using RUM Console. 6

7 Introduction Documentation Conventions The following font conventions are used throughout documentation: This font Bold Citation Documentation Conventions [p. 7] Fixed width Fixed width bold Fixed width italic Menu Item Screen Code block Indicates Terms, commands, and references to names of screen controls and user interface elements. Emphasized text, inline citations, titles of external books or articles. Links to Internet resources and linked references to titles in documentation. Cited contents of text files, inline examples of code, command line inputs or system outputs. Also file and path names. User input in console commands. Place holders for values of strings, for example as in the command: cd directory_name Menu items. Text screen shots. Blocks of code or fragments of text files. 7

8 Introduction 8

9 CHAPTER 1 Database Monitoring AMDs can be configured to monitor specific features of database traffic, or to automatically recognize and analyze the SQL queries that appear most frequently. You can also configure many other aspects of database monitoring such as operation recognition or query masking. Database monitoring refers to monitoring and analysis of SQL database traffic. The AMD is capable of monitoring various types of database traffic such as DRDA (DB2 database), TDS (Sybase SQL Server and Microsoft SQL Server), Oracle, MySQL and Informix. All the properties that database analyzers use are set in the RUM Console. Some of the major features include: Query auto-learning Full or partial query reporting Aggregation of similar queries Individual query monitoring Possibility to limit the number of processed queries only to the unique ones Hiding classified data in queries You can turn on simple monitoring of database protocols by enabling monitoring of default software services. Then you should acquire basic data based on global settings common to all analyzers. To broaden or narrow the scope of the monitoring process, you need to create user-defined software services and a number of rules that will override default settings and will match your installations. See Basic Monitoring Configuration [p. 29] for information on how to: Use default and user-defined software services. Configure a particular aspect of database monitoring. Configure settings common to all protocols (such as activating default service monitoring or defining a new user-defined service on a given IP address and port). 9

10 Chapter 1 Database Monitoring Overview of Database Monitoring Some database analysis can be performed using default software services definitions. However, many additional specialized settings and customization are provided for the monitoring of user-defined software services, and for such services information better fitting your needs is collected and reported. Before You Begin Before you start the configuration process: You should be familiar with DC RUM components and basic monitoring concepts. Refer to the Data Center Real User Monitoring Getting Started. You need to identify your monitoring goals. For more information, see Define and Prioritize Goals, Objectives, and Requirements in the Data Center Real User Monitoring Getting Started. You need to install the following DC RUM components: The latest version of AMD Refer to the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. The latest version of RUM Console Refer to the Data Center Real User Monitoring RUM Console Installation Guide. The latest version of CAS Refer to the Data Center Real User Monitoring Central Analysis Server Installation Guide. Optionally: The latest version of ADS Refer to the Data Center Real User Monitoring Advanced Diagnostics Server Installation Guide. Make sure that default ports are available for communications between the individual DC RUM components. For more information, see Network Ports Opened for DC RUM in the Data Center Real User Monitoring Administration Guide. The following steps must be executed in order to begin monitoring the traffic using the DC RUM suite: Configure Devices 1. Add Agentless Monitoring Device (AMD) AMD is the main data source (Data Collector) for DC RUM; it collects and presents the monitored data to DC RUM report servers for analysis and reporting. You need to add at least one AMD to the list of devices in RUM Console. For more information, see Adding an AMD to the Devices List [p. 13]. 2. Add Central Analysis Server (CAS) 10

11 Chapter 1 Database Monitoring CAS is the main report server for DC RUM. It uses data provided by the AMD and its monitoring and alerting mechanisms to identify, track, and report on issues affecting the security, performance, and reliability of your services. Add at least one CAS to the device list and configure its connection with the AMD. Adding a report server to a list of devices is similar to adding the AMD. For more information, see Adding a CAS to Devices List [p. 15]. 3. Verify the traffic monitoring quality and completeness You can verify traffic quality and completeness before the actual monitoring begins. Sniffing point diagnostics allows you to perform pre-monitoring tasks without the need of accessing the AMD console and executing a series of Linux commands which usually serve the purpose of validating AMD physical installation and connection. For more information, see Verification of Traffic Monitoring Quality [p. 17]. Configure Basic Monitoring 4. Configure general settings for your AMD Before you proceed to detailed monitoring rules, you need to define the global settings that are applied to all software services monitored by a given AMD. These global settings include a monitoring interval and thresholds for the basic metrics. These settings can be overridden at a later time with more specific monitoring rules that you can define. For more information, see Configuring General Data Collector Settings [p. 29] and Configuring Operation-Related Global Settings [p. 33]. Customize Monitoring Rules 5. Define your own software services on specified ports and for specified IP addresses If you configure user-defined software services, you will be able to see data on many reports provided by DC RUM. 6. Display the reports to review statistics for monitored traffic Determining the best possible configuration for your needs may be an iterative process, where you adjust the configuration incrementally after viewing your report results. Fine-Tune the Reporting Configuration 7. Configure the sites, areas, and regions A site is a term for a group of users that are located in the same IP network or group of networks sharing similar routing properties. Sites can be grouped together into areas, which, in turn, can be grouped together into regions. The hierarchy of sites, areas, and regions provides an organized view of the monitored network on the reports. For more information, see Configuring Sites, Areas, and Regions in the Data Center Real User Monitoring Administration Guide. 8. Configure the transactions, applications, and reporting groups Transactions are sequences of information exchange that represent particular actions or functions performed by a human user or a client program. They are viewed as higher-level units of self-contained functionality and are tied to applications. For example, they may represent the procedure for an online purchase or ticket booking. AMD monitors traffic 11

12 Chapter 1 Database Monitoring data and prepares it for transaction monitoring by an ADS and CAS. Some of the relevant configuration and processing is performed on the actual RUM Console and some is performed on the AMD. For more information, see Managing Business Units in the Data Center Real User Monitoring Administration Guide. 9. Configure the monitoring of sequence transactions DC RUM enables you to define and monitor transactions that are sequences of steps. For example, adding a product to a cart or selecting payment method could be one of the steps in the purchase transaction. For more information, see Monitoring Sequence Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. 10. Configure the tiers A tier is a specific point where DC RUM collects performance data. You can have monitoring data reported based on the default tier configuration, or you can define tiers that fit your network architecture. Troubleshooting 11. Troubleshoot problems You can review the answers to the most common questions and troubleshoot your setup and report configurations. For more information, see Diagnostics and Troubleshooting [p. 77]. 12

13 CHAPTER 2 Adding Basic DC RUM Devices In a DC RUM configuration, there are two device types: data collectors and report servers. To start using the product, add and configure at least one AMD data collector and one CAS report server. You manage these devices using a configuration tool called the RUM Console. Adding an AMD to the Devices List Before you can monitor traffic with DC RUM, you have to add and configure an Agentless Monitoring Device using the RUM Console. To add an AMD to the list: Adding an AMD 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type list, select AMD. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Information 5. In the Device IP address box, type the device IP address. 6. In the Port number box, type the port number for communication with this device. The standard port number used by AMD is Optional: Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. 13

14 Chapter 2 Adding Basic DC RUM Devices Providing the Authentication Details 8. Type the user name and password of the account that will be used for managing this device. By default, the AMD user is set to compuware and the password is set to vantage. The credentials entered here are used by the RUM Configuration to communicate with the device and are also passed to the report servers so that they can collect monitoring data for processing. Note that the values used here for authentication are not the same as the values you use for logging in to the device via SSH or local console. Configuring Advanced Settings 9. Select the Advanced options tab. 10. Optional: Under Secondary device connection, provide an alternative IP address for this device. 11. Optional: Enable SNMP connection. Optionally, you can define the SNMP connection parameters so that you can obtain more detailed health information about the device. To define SNMP connection parameters: a. Select SNMP Connection check box. b. Type the read community name and port number. 12. Enable Guided Configuration. By default, the Guided Configuration connection is enabled when you add an AMD. However, for performance reasons, the number of AMDs with enabled Guided Configuration is limited to 50. Any additional AMDs do not feed data to the Guided Configuration perspective. This means that the monitoring data from the additional AMDs is not available for generating the web traffic statistics or defining the web software services with a wizard. By default, the port number for communication between the Console Basic Analyzer Agent and the RUM Console Server is set to 9094 and the secure connection is enabled. In most cases, it is not necessary to modify this setting. If the default port number is already in use by other services, however, type the new port number in the Port number box. In this case, you also have to manually change the port number setting on the Console Basic Analyzer Agent side. For more information, see Modifying Connection Settings for Guided Configuration in the Data Center Real User Monitoring Administration Guide. 13. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 14. Click Finish to save the configuration. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. 14

15 Chapter 2 Adding Basic DC RUM Devices Adding a CAS to Devices List To view reports based on the data from the AMD, use the RUM Console to add and configure a CAS report server. Adding a CAS 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type menu, select CAS. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Details 5. In the Device IP address box, type the device IP address. 6. In the Port box, type the port number for communicating with this device. The standard port number used by the CAS when communicating over HTTP is Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. Providing the Authentication Details 8. Choose whether authentication should occur via CSS. 9. Type the user name and password of the account that will be used for managing this device. Configuring the Advanced Settings 10. Select the Advanced options tab. 11. Optional: Under Secondary device connection, provide an alternative IP address for this device. 12. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Click Finish to save the configuration. Configuring the CAS-AMD Connection 14. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 15. Select a report server from the list of devices. Click the server once to display the detailed information for the device. 16. Select the Data Sources tab. 15

16 Chapter 2 Adding Basic DC RUM Devices 17. Click Add Data Source. 18. Select your AMD from the list and then click the button. 19. Click Finish to save the configuration. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. What to Do Next It is important to keep the devices synchronized to avoid improper data interpretation. For more information, see Synchronizing Time Using the NTP Server in the Data Center Real User Monitoring Smart Packet Capture User Guide and Time Synchronization Between AMD and Server in the Data Center Real User Monitoring Administration Guide. 16

17 CHAPTER 3 Verification of Traffic Monitoring Quality Use the RUM Console to verify the traffic monitoring quality using two tightly connected solutions: Sniffing Point Diagnostics and Application Overview. We highly recommend that you perform this step at the beginning of your DC RUM deployment to verify that your hardware is working properly and that the applications you intend to monitor are detected. You can verify the test results and repeat them as needed at any time and for any network conditions. IMPORTANT All verification is based on a traffic recording, either manual or automatic. The outcome may not be representative if the target traffic is low at the time of recording or if you are unable to capture a satisfactory number of complete sessions. Choose automatic or manual traffic recording to capture unfiltered or filtered traffic. Enable automatic recording only during the configuration process and then disable it. It can negatively affect the performance of the AMD during normal operations, especially if you are running a 32-bit AMD in a high-traffic environment or a 64-bit AMD with the native driver. For the most complete and reliable statistics, use the 64-bit customized driver on the AMD. The verification of traffic monitoring quality is possible only for AMD 11.7 or later. Sniffing Point Diagnostics Sniffing Point Diagnostics is a type of hardware state analysis that enables you to perform pre-monitoring tasks without the need to access the AMD terminal. You can use it to validate the operation of the sniffing points, instead of using a series of UNI or rcon commands. This step can be performed at the DC RUM deployment stage or at any other time to determine if the AMD performance is affected by malfunctioning hardware or external networking conditions. The Sniffing Point Diagnostics analysis can detect issues, such as: No traffic detected on sniffing interfaces. Interface or link overload. 17

18 Chapter 3 Verification of Traffic Monitoring Quality Poor quality of traffic due to mirrored ports on switching hardware configuration. Dropped packets (indicates AMD overload). Network conditions when unidirectional traffic prevails. Rejected packets, invalid packets, wrong check sums for packets. Missing packets (either lost or dropped). Missing bytes (how much traffic is lost in general). Conditions affecting AMD calculations, such as: Duplicate traffic that cannot be handled by the AMD. Incorrect choice of packet deduplication method. Incorrect settings for packet deduplication buffer. Incorrect settings for maximum packet size or huge packet size. Conditions affecting AMD performance, such as: Duplicate traffic handled by the AMD. Large percentage of non-ip traffic (noise). Large percentage of non-tcp or non-udp traffic (noise). Reordered sessions. Miscellaneous SSL problems: Unsuccessful decryption (in general). Uninitialized SSL cards unable to decrypt traffic. The ratio of encrypted and successfully decrypted traffic to encrypted and non-decrypted traffic. Incorrect or missing private keys. No match between the key and server certificate. Dropped or corrupted packets preventing decryption. Unsupported cipher methods (for example, Diffie-Hellman based key infrastructure). Unsupported SSL versions or features. Prerequisites and Best Practices To diagnose application detection problems and sniffing point connection problems, ensure that: All cables are connected correctly. The AMD is properly installed and configured. This includes the post-installation steps, such as interface identification and network configuration. Traffic recording lasts long enough to capture a reasonable amount of traffic volume, for example, 20 to 30 minutes of traffic. 18

19 Chapter 3 Verification of Traffic Monitoring Quality Do not use specific capture profiles when recording traffic. Always use the All available option for capture profiles when you do manual recording. When you need to diagnose traffic or capture port problems, enable automatic trace recording. Trace recording provides access to regular and fresh snapshots of the traffic that is traveling on your network. Sniffing Point Diagnostics Reports Sniffing Point Diagnostics reports are organized into several sections, each representing a separate set of metrics related to either hardware or network traffic. This topic provides directions for viewing the reports, but you can follow each step or skip steps to view the only the information important to you. 1. Start either by looking at device health or from the reports section directly. If you enabled automatic trace recording, you can monitor the device state on the Device Status tab of the Devices screen. A separate set of statistics is provided for each AMD added to the console. If there are any alarm messages, go to Devices and Connections Verify quality of monitored traffic. Inspect network interfaces in detail for a selected AMD. Open the Overview report to verify that the proper type of network driver is being used (custom or native) and that traffic has been detected, and check the number of dropped packets and other performance related issues. You can also verify that the NIC drivers are operational. For more information, see Network Interface General Statistics [p. 19]. 2. Switch to the Protocols section to inspect protocols. See whether network protocols are detected (IPv4 or IPv6) and verify detection of transport protocols (TCP or UDP). For more information, see Network and Transport Protocol Information [p. 22]. 3. Switch to the Services section to see the most active services. For more information, see Services Detected in the Traffic [p. 22]. 4. Depending on your goals, switch to the Sessions section either by selecting a particular service on the Services report to see session details or by choosing the Sessions section to see general statistics for all sessions. For more information, see Session-Related Statistics [p. 22]. 5. If you use SSL decryption, you can inspect whether there are problems detected for the currently used SSL engine or keys. For more information, see SSL Diagnostics [p. 24]. Network Interface General Statistics The Overview section of the Sniffing Points Diagnostics reports enables you to verify the general state of capture ports on a selected AMD. The information in the Overview section is gathered directly from the NIC driver operating on the AMD. For the most reliable results, use the 64-bit customized drivers. 19

20 Chapter 3 Verification of Traffic Monitoring Quality Calculation of Analyzed Traffic The calculation of analyzed traffic is performed in several stages, gradually excluding the irrelevant statistics: 1. The overruns are excluded first. When the received packets are counted, the overruns are omitted. 2. The calculation of the received packets depends on the subtraction of errors and filtered-out packets. 3. The dropped packets are counted after the filtered packets are disregarded. 4. The number of analyzed packets is the count of packets remaining after all of the previous categories are subtracted. In default AMD installations, non-tcp/udp packets are not part of this process and are never counted when the number of analyzed packets is given. Non-TCP/UDP traffic increases the amount of analyzed traffic only if you enable the monitoring of the default software services. Figure 1. Graphical Explanation of Analyzed Traffic Calculation for an AMD with 64-bit Customized Network Interface Driver All network packets Overruns Packets not received Received packets Errors and non-conditional filtering Errors: length or bad checksum; filtered out: non-ip Load balancing If active, fraction of the traffic Configuration filtering Based on defined software services Sampling and dropped packets Packets not analyzed due to performance issues Non-TCP, non-udp If default software services enabled Analyzed packets 20

21 Chapter 3 Verification of Traffic Monitoring Quality Interface Operation-Related Metrics The statistics presented on this screen include: Overruns Overruns may indicate a link overload. The overload is typically caused by an exceptionally high traffic volume. This value may also indicate that the network interface or network interface driver cannot manage the amount of traffic received. Other hardware-related issues may also cause overruns. If a high overrun occurs, limit the traffic volume received by the card. Errors (length) Packets of erroneous length are reported when they are too big (such as jumbo frames) or are bigger than the maximum transmission unit (MTU). To avoid such problems, you can increase Maximum packet size in the Entire Configuration perspective. For more information, see Configuring General Data Collector Settings [p. 29]. Errors (bad checksum) Checksum-related errors are typically caused by insufficient signal strength on an optical link. In other cases, checksum errors may indicate Ethernet distortion, such as duplex problems, where the checksum errors may result, for example, when the duplex auto-negotiation process fails. Check the host switch and AMD duplex settings. Filtered out (non-ip) Non-IP packets, such as ARP traffic. Even large numbers of such packets are generally considered harmless. They are not analyzed by the AMD software and are regarded as noise. Preventing such traffic from reaching the AMD may reduce the possibility of performance degradation. Filtered out (load balancing) This setting is only applicable in deployments with multiple AMDs where each device only analyzes a certain part of the same traffic. Filtered out (configuration) Provides additional filtering based on software service definitions. In default installations, where monitoring of the default software services is turned off, the driver limits the number of processed packets to only those that are relevant to the IP addresses included in user-defined software service definitions. Dropped (sampling) Sampling here means dropping packets when the driver performance is degraded. Packets are dropped in a controlled manner, and always with care, to preserve complete and consistent sessions. The packet drops almost always mean that traffic is too heavy for a complete analysis and that, with packet drops, the precision of CAS reports is diminished. Sampling is only active with the customized 64-bit driver and diagnostics always use this sampling mechanism regardless of the settings used in the general AMD configuration. Dropped (driver performance) Drops are always a symptom of problems, especially when SSL analysis is deployed. Drops occur when AMD software is unable to analyze all of the packets it receives from the driver. If you use 32-bit or native drivers, you may experience uncontrolled packet dropping. If you use the 64-bit customized driver, packet dropping may occur, but in a software-controlled manner with care for monitored data contingency. 21

22 Chapter 3 Verification of Traffic Monitoring Quality To avoid packet dropping, decrease the traffic volume that your AMD analyzes or reduce the number of monitored software services. Non TCP/UDP Whether these statistics are classified as analyzed or not depends on the default software service monitoring. The numbers in this section are mostly relevant if you enabled monitoring of default software services. In this case, ICMP traffic is also analyzed. If monitoring of the default software services is disabled and you still see a large percentage of non-tcp and non-udp traffic, it is possible that AMD performance will be affected. Network and Transport Protocol Information Use the Protocols report to check the ratio of supported transport or network protocols. Only supported protocols are shown. In general, this report enables you to check whether traffic that makes sense (from the DC RUM perspective) is present and is heavy enough to give meaningful results for report servers. NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. Problem Detection Low traffic for the IPv4 or IPv6 network protocols may indicate further monitoring problems. The presence solely of multicast or broadcast traffic is an indication that port mirroring is not enabled or inactive. Services Detected in the Traffic This overview report enables you to identify the most active services on your network. You can see what their load is and what protocols they use, and filter the results to display all data, monitored services, or unmonitored services. You can also use filters to display statistics for all, monitored, or unmonitored services with additional protocol filtering. For each service, you can open the Sessions report to verify session-level statistics. NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. Session-Related Statistics The Sessions section enables you to view detailed information about traffic quality. The statistics presented on this screen include: 22

23 Chapter 3 Verification of Traffic Monitoring Quality Duplicates, Unhandled duplicates The value presented on the Sessions screen depends on the currently selected deduplication method in your AMD configuration. Packet duplicates may indicate incorrect configuration of mirroring ports. While this may be a sign of a problem, values of 10 to 20 percent typically are no reason for concern. The AMD is capable of packet deduplication. Higher numbers of duplicate packets will degrade the AMD performance and may negatively influence the monitoring results. The diagnostics mechanism for duplicate detection and counting for this report works with different settings than the network monitoring processes on the AMD. Duplicate detection is performed using both methods of duplicate detection and with different settings (buffer and delay detection size). Based on these settings and calculations, Sniffing Point Diagnostics provides suggestions concerning duplicate handling, such as increasing buffer size or changing the deduplication mechanism. You should check whether there are unhandled duplicates detected, in which case it is suggested that you switch the detection method in the AMD general settings. For more information, see Configuring General Data Collector Settings [p. 29]. Unidirectional TCP sessions and UDP streams This may indicate a problem related to incorrectly configured mirroring ports. If the value of unidirectional traffic exceeds 90 percent, the RUM Console always marks it as an error. The numbers on the Sessions screen are the sums of many measurements; you are able to go deeper and analyze details for each server and check whether this is a problem related to a significant service or protocol. Insignificant traffic may be recorded and included in the general analysis, so always check the detailed reports when you see alarming numbers on the Sessions report. TCP sessions with missing packets Missing packets may result from interface or driver packet drops. If a session with missing packets is shown, the percentage value is counted with regard to all sessions. For example, if two percent of sessions have missing packets reported, this means that two out of a hundred sessions have missing packets. TCP sessions with missing packets and TCP bytes lost in missed packets may provide valuable insight into SSL decryption problems, especially in the case of long SSL sessions. TCP bytes lost in missing packets This is a complementary value to the TCP sessions with missing packets. Verify the number of lost bytes with regard to missing packets to see whether the problem is serious (if there are large sums of missing bytes). This is useful additional information in the case of long TCP sessions; because one lost packet is enough to classify a session as having missing packets, the number here gives insight into the actual loss rate. TCP sessions with reordered packets Reordered packets are typically found when there is a WAN link enabled. Devices transferring WAN packets may affect the packet order. The existence of reordered packets is not a problem in itself, because the AMD software can restore original packet order, but an excessive number of such packets may cause performance degradation. 23

24 Chapter 3 Verification of Traffic Monitoring Quality NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. SSL Diagnostics The traffic for this report is dependent on capturing complete sessions. Incomplete sessions, missing packets, or missed handshakes cause a large number of errors and a large number of errors results in unreliable reports. Always be sure to record enough traffic for an adequate length of time to allow you to capture complete sessions. The Statistics for encrypted traffic, SSL card and keys report is only available after the traffic trace recording is finished. Partial statistics for SSL are not provided for unfinished sessions. General Statistics for Encrypted Traffic For a given time range, defined by the scope of the recorded traffic traces, you can see the recognized SSL engine (for example, OpenSSL or ncipher) and the number of keys exchanged in the traffic. The remaining sections of this diagnostic report show the detailed information about the keys, the overall summary of the captured SSL traffic, and whether there are errors. The servers section shows information for all SSL traffic captured during the traffic trace recording. All of the detected encrypted protocols are listed together with their matching keys, if they are seen in the traffic. You can see whether the key exchange was successful; the matched keys are indicated by the icon. Key and certificate matching enables you to verify that certificates were found and were valid. No matching may indicate that the certificates are out of date. SSL Server Status The Status column shows whether there are errors or whether erroneous sessions prevail. A traffic capture sometimes does not contain session beginnings, or it contains incomplete handshakes, or it has no master session; these sessions are marked as ignored, as indicated by the gray ( ) color bar. The sessions with errors are marked by a red ( ) color bar. The main causes of errors are missing packets or missing keys. Other causes of errors are listed in detail on the Detailed SSL Statistics for servers report. Detailed SSL Statistics for Servers Detailed SSL statistics for servers are accessed from the Server or Status columns. This report shows: The percentage of the sessions without error, with errors, or ignored. The counts of each problem, in detail, for the error or ignored sessions. The number of decrypted sessions if there are no problems. 24

25 Chapter 3 Verification of Traffic Monitoring Quality You can filter the results. Use Sessions finished to display the data for completed sessions. Use Sessions in progress to display the sessions that are still in progress (sessions that did not end before the traffic capture stopped; to see those session statistics). Figure 2. Example of Detailed SSL Statistics for Server, Errors Detected Due to Private Key Mismatch SSL Keys Because invalid or outdated keys are usually not removed from SSL cards, the list of keys for which an error status is indicated may be considerably long. In such cases, sort by the Status column to see keys correctly matched. Note that it may be necessary to format the SSL card storage area to refresh the key list. Application Overview The Application Overview screen enables you to answer several questions about your applications at the onset of your monitoring configuration. Are all my applications or servers detected? What applications or servers are detected? Can the detected applications or servers be successfully monitored? How heavy is the traffic for each application or server? What services are detected on each server? How heavy is the traffic for each detected service? Note that incomplete sessions are not analyzed. If no beginning is recorded for a session, that session is not analyzed. 25

26 Chapter 3 Verification of Traffic Monitoring Quality The Application Overview screen is an optional step towards defining new software services. To access it, select Software Services Add Software Service in the console top menu, then select By traffic lookup. Figure 3. Example of the Application Overview Screen Showing Detected Applications From this screen, you can configure software services either manually or by using the wizard. If it is possible to go through a step-by-step configuration, a wizard icon ( ) is displayed for the given protocol or service. Application Detection Mechanism Application detection is a three-stage process: 1. To provide the most accurate results, packet analysis for SSL, HTTP, HTTPS, SOAP, and related protocols is performed as a first step toward application type detection. Application recognition is based on the first matching pattern found. This means that some services may not be properly classified if multiple protocols are used in one session. For example, if your application uses HTTP and SOAP over HTTP protocols, and plain HTTP communication opens a session, the application is classified as HTTP. 2. Applications are also detected based on discovery of well-known ports. The default protocol definitions are stored on the AMD and can be exported from the RUM Console. For more information, see Exporting the AMD Configuration in the Data Center Real User Monitoring Administration Guide. At times applications may use ports commonly used for other purposes. The AMD is unaware of these circumstances and will report well-known protocol names. For example, if one of your web applications uses port 8080 and uses HTTP for communication, the AMD reports this as an HTTP proxy. 3. If none of the selected conditions matches, the application is labeled as Unknown TCP or Unknown UDP. 26

27 Chapter 3 Verification of Traffic Monitoring Quality Server recognition in application detection is based on heuristic session analysis; results may vary depending on the type of network interface driver used. Using RUM Console to Identify Problems Related to Network Hardware Operation Typical configuration errors related to port mirroring can, at times, severely affect the AMD software traffic analysis capabilities. Faulty hardware configuration may result in no data seen by the AMD, a large number of duplicate packets reaching the AMD, or only a limited portion of traffic visible to the monitoring software. Use the Application Overview and Sniffing Point Diagnostics sections as tools to solve issues related to the switching hardware configuration. The following list describes several common problems and some possible causes and solutions. No data seen by the AMD The cable is connected to the wrong physical port on the destination switch. This can be checked by physically tracing the cable directly to the switch and confirming the port ID. The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror incorrect ports or an incorrect destination. This can be resolved by logging on to the source switch and checking the mirroring ports configuration relevant to the requirements (see the vendor-specific documentation for details). No data seen on Application Overview but non-tcp/udp traffic seen in interface statistics The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror incorrect ports or an incorrect destination. This can be resolved by logging on to the source switch and checking the mirroring ports configuration relevant to the requirements (see the vendor-specific documentation for details). Application Overview does not show all expected data The port mirroring destination may be oversubscribed or dropping packets. Check this by logging on to the switch and checking the SPAN or mirror destination interface. If it is recording many drops, review the configuration of source ports to understand the ratio of source interface bandwidth to destination interface bandwidth. If the ratio is excessive (for example, greater than 4:1), consider reducing the number of source interfaces. If applicable, consider using device-specific filtering to reduce the load on the destination interface (for example, VACL, Rx-only, or Tx-only sources). By design, port mirroring does not forward faulty frames. Check the source device interface statistics to ascertain the nature of the drops (see the vendor-specific documentation for details). Check the interface-related metrics. If there is a high rate of Errors (bad checksum), consider hard-configuring one end of the AMD SPAN connection to prevent auto negotiation. Session-related report shows a high rate of packet duplicates A SPAN or mirror operates by copying frames from source interfaces and directing them to the destination interface. In effect, configurations often result in two copies of a packet. 27

28 Chapter 3 Verification of Traffic Monitoring Quality For example, if the source of a SPAN or mirror is set as a VLAN, any traffic that goes from one switch port to another switch port within the VLAN appears twice on the mirrored port. If the number of duplicates starts to affect AMD performance, consider reducing the number of source interfaces. If applicable, consider using a device-specific filtering control to reduce packet duplication (for example, VACL, receive-only, or transmit-only sources) or consider using tap technology as opposed to port mirroring to collect the data. Only unidirectional streams are seen on session-related overview If the AMD is connected via a SPAN or mirror, the configuration has been set incorrectly to send only one side of a receive or transmit stream to the destination. Log on to the local source switch to check the configuration (see the vendor-specific documentation for details). 28

29 CHAPTER 4 Basic Monitoring Configuration You can define many configuration settings globally for all software services for a given protocol and Data Collector, or locally for specific user-defined software services. If you specify both types of settings, the settings for a user-defined software service take precedence over the corresponding global settings. Use the RUM Console to perform basic monitoring configuration, including the global settings for Data Collectors, operations, and the analyzer, as well as configuring Dynatrace to recognize WAN-optimized traffic. Configuration and recognition of optimized WAN traffic in Dynatrace is optional and depends on whether WAN optimization is used in your network. Refer to the Data Center Real User Monitoring WAN Optimization Getting Started. NOTE If you make any significant changes in the configuration, such as removing defined software services or operations, your are advised to restart the AMD. This is to prevent persistent TCP sessions from blocking your changes. Configuring General Data Collector Settings For any given data collector device such as the AMD, you can set a variety of options, such as time thresholds. The general settings affect the monitoring of default and user-defined software services. Some of these settings can then be overridden by settings for a particular analyzer, software service, or URL. To define the general settings for an AMD: 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 4. Select Configuration Global General to access the list of general configuration settings. 29

30 Chapter 4 Basic Monitoring Configuration While some of the options control only general AMD behavior, some options in the Advanced group affect more specific configurations in application monitoring. For example, if Inherit from global settings is selected in your other configurations while configuring user-defined software services, the global setting takes precedence over the specific monitoring configuration. Configuration options include: Monitoring interval The monitoring interval in minutes. Increasing this value reduces the number of chunks of data that need to be transferred and processed. Default: 5 minutes. Verify that the monitoring interval is synchronized between the data collectors. Operation time threshold The number of seconds after which an operation is considered to be slow. The global threshold value depends on the analyzer. This threshold is used by the following analyzers: Cerner Cerner over MQ Epic Generic with transactions HTTP MS Exchange over HTTP MS Exchange over HTTPS Oracle Applications over HTTP Oracle Applications over HTTPS SAP GUI SAP RFC SAP GUI over HTTP SAP GUI over HTTPS SMTP SSL SSL Decrypted Server time threshold The Server time threshold relates to the server time portion of an overall operation time. Server times above the threshold limit are considered to be slow due to poor datacenter performance. This threshold is used by the following analyzers: HTTP SAP GUI over HTTP SAP GUI over HTTPS IP address of the server authorized to set AMD time The IP address of the report server that has the authority to synchronize the time with this AMD. 30

31 Chapter 4 Basic Monitoring Configuration In an environment with a number of servers sharing the same AMD, it is good practice to designate only one of these servers as a time synchronization server to make changes to AMD settings. Otherwise, the server used for time synchronization will change inadvertently every time you save an AMD configuration. Default analyzer The default setting for the TCP analyzer is Generic (with transactions). To change it, select another analyzer from the list. Client RST packet timeout to mark session as CLOSED If the time between the last ACK for data sent by the server and an RST packet sent by the client is greater than this value, the session is treated as closed instead of aborted. Huge packet size The upper size limit, in bytes, of an HTTP request to be processed successfully by the AMD. Maximum packet size The AMD is capable of processing packets of up to bytes, besides the Ethernet standard MTU (Maximum Transmission Unit) of 1536 bytes. Choose one of the predefined values (2048, 4096, 8192, or bytes) to enable the AMD to process non-standard MTU packets. When you have chosen the Maximum packet size value, make sure that you also set the Huge packet size to an applicable value. Enabling theamd to process nonstandard MTU packets without extending RAM on the machine and leaving Packet buffer size (64-bit AMDs only) and Data memory limit unchanged can cause an excessive packet loss. To avoid this, extend RAM and configure its usage as recommended in the tables below. For more information, see Setting Packet Buffer Size in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Setting Data Memory Limit in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. NOTE Do not enable the processing of large packets for a Small AMD. These devices are not designed to process larger packets. For more information, see Small AMD in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. Table 1. Recommended RAM Configuration for Maximum Packet Size Values for AMDs Maximum packet size 8192 B or less 8192 B B Recommended RAM size for 64-bit platforms 64 GB 96 GB 128 GB 31

32 Chapter 4 Basic Monitoring Configuration Sampling enabled Supported in 64-bit customized AMD drivers and all- native drivers. The sampling mechanism is beneficial when heavy traffic may negatively affect AMD performance and there is a risk of losing IP session consistency. When this option is enabled, the AMD tries to analyze the greatest possible portion of traffic. It drops packets in a controlled manner that preserves complete and consistent sessions. Note that statistics for dropped packets are not shown on the report server. If packets are dropped because of sampling, the CAS shows notification messages. For percentages between 75 and 99, a warning icon is displayed; for values below 75, the report server issues error messages. When this option is disabled and the network interface driver performance is degraded, random packets are dropped. Default: enabled. For more information, see Using Network Interfaces with Native Drivers in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Driver, Network, and Interface Configuration in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. NOTE When capturing packets on an AMD with sampling disabled, if the AMD experiences packet drop due to high traffic volume, the packet capture is not automatically canceled. If this occurs, select Tools Packet Data Mining Tasks on the CAS, find the task that was using the AMD in question, and click to cancel that task. Deduplication method You can choose one of four methods for eliminating duplicate packets: Based on TCP checksum and IP ID Using this method, duplicate packets are detected based on their TCP checksum and IP ID. Based on TCP checksum and IP ID (excluded SEQ and ACK numbers) Using this more complex, two-stage method, duplicate packets are detected based on a modified packet KCP checksum (SEQ and ACK numbers are excluded) and IP ID. This method is useful if the AMD captures packets on various interfaces of the router, rewriting SEQ and ACK numbers. A packet is considered a duplicate when the modified checksum, IP ID, and SEQ and ACK numbers are identical. First, a packet checksum with SEQ and ACK numbers is created and compared to the packets stored in the detection buffer. If the comparison indicates that the packet is not a duplicate, it is checked to determine whether it matches the current session. A packet matches the current session when its SEQ and ACK numbers are different from processed and cached numbers by the value defined in TCP duplicate window. If the difference exceeds the defined value, the AMD assumes the ACK and SEQ numbers were rewritten by the router and the packet is considered a duplicate. 32

33 Chapter 4 Basic Monitoring Configuration TCP checksum, IP ID and MAC address (excluded SEQ and ACK) Using this method, the deduplication process is similar to the one based on TCP checksum and IP ID (excluded SEQ and ACK numbers), but in addition to TCP checksum and IP ID, the source/destination MAC addresses are also taken into account for the calculation. TCP checksum, IP ID and MAC address Using this method, duplicate packets are identified based on their TCP checksum, IP ID and source/destination MAC addresses. TCP duplicate window This setting is useful only if Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers. It is used for determining whether a packet, based on its SEQ and ACK numbers, belongs in the session. If a packet's SEQ and ACK numbers differ from the current session's SEQ and ACK numbers by a value larger than TCP duplicate window, the packet is considered a duplicate. Default: Packet buffer size The number of packets to keep in the buffer for use as a basis for comparison in duplicate packet detection. Newly captured packets are sequentially compared to the packets in the buffer. A newly captured non-duplicate packet (all packets in the buffer are unique) is placed on the top of the stack and the oldest is removed. Range: 6 to 24 packets. Default: 16. Reset duplicate detection time threshold Time of inactivity (in seconds) after which the duplicate packets elimination mechanism is reset. If Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers or TCP checksum, IP ID and MAC address (excluded SEQ and ACK), and the Reset duplicate detection time threshold should be greater than every response generation time (server time). 5. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. 6. Close the AMD Configuration window. Configuring Operation-Related Global Settings The operation-related global settings enable you to define options that apply to all monitored operations. These settings take precedence over the options defined for individual operations. 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 33

34 Chapter 4 Basic Monitoring Configuration 4. Select Configuration Global Operations to display the general configuration settings. The options are: Operation load time threshold The number of seconds after which an operation is considered slow. You can set this value with a precision of one ten-thousandth of a second. Default: seconds. The threshold is used by following analyzers: IBM over MQ Jolt MS Exchange Oracle Forms over HTTP Oracle Forms over HTTPS Oracle Forms over SSL Oracle Forms over TCP SOAP over HTTP SOAP over HTTPS ML ML over HTTP ML over HTTPS ML over MQ ML over SSL Max. operation duration The maximum number of seconds an operation can take. You can set this value with a precision of one ten-thousandth of a second. Default: 3600 seconds (1 hour). User abort threshold The minimum number of seconds between the beginning of a hit and TCP reset to count it as a user abort. Default: seconds. (You can set this value with a precision of one ten-thousandth of a second.) ADS data generation settings The options in the ADS data generation settings section can be used to handle various types of standalone hits, which are hits that cannot be automatically assigned to operations because the reference information, such as correlating response, defined or auto-learned URL, no authorization, or orphaned redirects, is missing. By default, most standalone hits are not taken into account when generating operations data. Report data without monitored URL Select this option to report data for hits without a URL that has been explicitly defined in user-defined services or recorded through auto-learning. Report standalone hits without monitored URL Select this option to report data for standalone hits that at the same time do not refer to a monitored URL, as in Report data without monitored URL. 34

35 Chapter 4 Basic Monitoring Configuration Standalone hits are hits without a response header, unauthorized hits, orphaned redirects, or other hits missing the reference context. Report hits without response header Select this option to report data for discarded hits (hits without a correlating response header). Report hits not added to any operation Select this option to report data for other standalone hits caused by factors not covered by other options of this section. Report unauthorized hits Select this option to report data for hits with rejected authentication. Report orphaned redirects Select this option to report data for redirects to sites that are not being monitored or are not visible and therefore appear as orphaned redirects. Report filtered data This is a diagnostics option. When configuring content type monitoring, you can filter out pages based on the content of the URL. For more information, see Monitoring of Non-HTML Objects Based on Content Type in the Data Center Real User Monitoring Web Application Monitoring User Guide. If you select this option, the filtered out pages are not reported, but are saved in the AMD data files. Ignored clients A list of clients for which TCP setup time are ignored and all operations start from the request packet. Right-click the list to open a menu of command options: Add, Edit, or Delete. 5. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. 6. Close the AMD Configuration window. 35

36 Chapter 4 Basic Monitoring Configuration 36

37 CHAPTER 5 Configuring AMD to Monitor User-Defined Software Services If you require detailed analysis of traffic, you need to specify software services to be monitored on specific IP addresses and ports. Defining Oracle Forms Software Services For software services you intend to monitor that do not work on a well-known port, you can use the specific IP address and port of the service when defining the software service configuration. For such software services, you can measure a wide range of metrics and perform detailed traffic analysis. To add a new software service: 1. Start and log on to RUM Console. 2. In the top menu, select Software Services Add Software Service. The Add Software Service pop-up window appears, listing all ways of adding a new service. 3. Select Manually as a method of adding a new software service definition. The Add Software Service window appears. 4. Specify basic information for your software service. Provide a software service name. Select appropriate analyzer to monitor the traffic. Using check boxes, select the devices that will monitor the new software service. When you later publish the software service definition, the new configuration will be applied only to the selected devices. 5. Click OK to proceed to monitoring rules configuration. 6. Right-click in the Rules table and select Add from the context menu. The Rule Configuration appears. 37

38 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 7. Proceed with the rules configuration.for more information, see Configuring Rules for User-Defined Software Services in the Data Center Real User Monitoring Oracle Forms Application Monitoring User Guide. 8. On the Software Services screen, click Publish Configuration. Configuring Rules for User-Defined Software Services Each software service can have a number of specific rules that define what is to be monitored and what additional options are in effect. You can also assign each software service to existing or newly created tiers and applications. Before You Begin It is assumed that, for this task, you are already familiar with the concept of software services and that you know how to create and edit software services and how to open the Rules window. For more information, see Configuring User-Defined Software Services in the RUM Console Online Help. After a user-defined software service is created, create a group of settings that comprise the rules for the software service. It is necessary to specify, at minimum, the IP addresses and port numbers for the software service. To configure rules for a user-defined software service: 1. Open the Services tab. 2. On the Services tab, select or clear Enabled to activate or de-activate the service definition. 3. In Rule description, type a brief description to identify the rule. The description you enter is shown in the Rules table, in the column Rule Name. If no text is entered here, the IP address specified later is used as the description for this rule. 4. Right-click in the Services table and select Add or Open from the context menu. To quickly navigate to an entry in this table, click in the table and then type some or all of the IP address. Click the magnifying glass icon or press [Ctrl+F] to open a search box to limit the table view to only those rows that contain a match (in any column) to the search string. The Service Details window appears when adding or editing the rules. 5. In the Service Details window, in the IP address(es) fields, enter the server IP address, or enter a range of IP addresses to monitor more than one server. 6. In the Port(s) fields, enter the port number of the monitored service. You can provide a range of port numbers if such a range of ports is used in your environment. Some software services may be active on a number of predefined ports or may change ports dynamically. To allow for this, you can specify a range of ports. However, specifying more than one port for a service prevents the port number from being reported for that service. If you define more than one port for a particular service name and server IP address (by either specifying a range of ports or by creating two or more distinct rules for the same 38

39 Chapter 5 Configuring AMD to Monitor User-Defined Software Services service name and server IP address but with different port numbers), the AMD reports the port number for this service as 0, causing the port number to be ignored in traffic reports. NOTE You can define up to 5000 definitions containing a server and a port. Each association of a server and a port counts as a single definition. Specifying a range of ports counts as providing many individual definitions. On CAS, the number of processed server definitions is limited by the license. For more information, see Per-Measurement Licensing in the Data Center Real User Monitoring Administration Guide. Advanced Configuration 7. Optional: Select Client port(s) for reversed-direction protocols. This option applies only to protocols such as -Window whose client-server meanings are reversed. If you are uncertain, leave this option cleared. 8. Optional: Select or enter a Group name Part of URL auto learning configuration. By default, the URL auto-learning mechanism stores the URLs from all the the servers defined in the software service in one pool. You can create separate pools within a single software service based on a number of servers. This way, you ensure the URLs monitored on a server with a lower traffic do not have to compete with URLs from a much larger server in terms of volume. You achieve this by assigning servers to groups within a single software service which translates to separate pools. To create a seperate pool for a group of server, keep them under a common group name of your choice. For more information, see Details of the URL Auto-Learning Algorithm in the Data Center Real User Monitoring Web Application Monitoring User Guide. NOTE It is important that grouping within the services definition is consistent. Defining services with that same IP address but different ports and assigning them to different groups results in the generation of redundant and irrelevant data. 9. Optional: Enter the main server IP address. If the monitored application runs on several servers that are linked together in a farm, you can monitor the farm as one virtual server. Type the IP address that you want to use as your main server IP address. 10. Optional: Enter the IP address of the server masking the addresses of monitored servers. If the servers you intend to monitor reside behind an appliance that masks and replaces the addresses of the target servers, you need to set NLB NAT masking IP address to the IP address of the masking server. Without doing so, the AMD will see two unidirectional conversations instead of one bi-directional conversation between the servers and appliance: The conversation between the client and server is observed and recorded (IP address A talking to IP address B) 39

40 Chapter 5 Configuring AMD to Monitor User-Defined Software Services When a response travels to the client, a different session (IP address C talking to IP address A) is recorded due to the server's IP address being replaced by the load balancer's IP address. Unless you account for this, CAS reports will return ambiguously granulated data. Using the NLB NAT masking IP address option will ensure that the AMD monitors contiguous conversations. 11. Optional: Map client IP to client group name. The mapping allows you to catalog and report traffic going to the same server IP and port by associating client group names with the originating client IP. On the report, the client group name is be reported as a suffix to the software service name. For example, a software service named SQL configured on a server located at can be configured the following way: Client IP Address Software Service name suffix _ATLANTA _BOSTON The system will differentiate the SQL software service traffic going to the server based on the client IP definition and report data for software service SQL_ATLANTA and SQL_BOSTON individually. The default configuration containing no client IP definitions results in an empty client group name. Similarly, an empty group name is used if a client IP is not included in any of the defined IP ranges. This configuration makes it possible to obtain only the client group name. The same client group name can be used in many client IP ranges. The configuration of each software service is individual per client group name. No cross-relations or cross-checks are performed between the definitions. It is possible to use a different name for the same client IP in each of the software services. 12. Click OK to confirm your changes and close the Service Details window. 13. Configure the settings on the available tabs. The number of available configuration options depends on the analyzer. See the analyzer-specific section for more information. 14. Optional: On the Options tab, define analyzer-specific options. The following list describes all possible options. Depending on the analyzer, some may be unavailable: Operation load time threshold An operation that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. The global threshold value depends on the analyzer. Operation time threshold An operation that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, 40

41 Chapter 5 Configuring AMD to Monitor User-Defined Software Services open the AMD configuration, select Global General and set the Operation time threshold. Server time threshold Server time threshold relates to the server time portion of an overall operation time. Server times above the threshold limit are considered to be slow due to the poor datacenter performance. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, select Global General and set the Server time threshold. SQL query time threshold A database query that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, select Global Database Monitoring General and set the SQL query time threshold. Enable monitoring of persistent TCP sessions When this option is selected, the TCP sessions that do not start with SYN packets are monitored. By default, this option is selected. Persistent TCP sessions are TCP sessions for which the start was not recorded. They are also referred to as non-syn sessions. These sessions can be included in the TCP statistics, based on the configuration properties you enable in RUM Console. The inclusion of these sessions may render the statistics somewhat inaccurate and must be undertaken with care. Generate transactions and ADS data Select this option to provide the report server with, for example, raw HTTP traffic data enabling you to view the full HTTP request-response dialog. SQL Server uses dynamic ports This option only applies to the TDS analyzer. Select this option if the database engine you intend to monitor does not have a static port number assigned (for example, a named instance). In this case SQL Server Browser Service (SSBS) is used to discover the actual port of the service. The AMD uses additional UDP analysis of the SSBS to discover the port number for the service you intend to monitor. If you select this option make sure that the connection details specified on the Services tab identify the SQL Server Browser Service (use the IP address of the server and the port number of the SSBS). Do not enable this option if your SQL Server uses static ports. Convert the ML content URL-encoding This check box defines whether the ML URL-encoding content is enabled. When Inherit from global setting is selected, the global ML setting is used. URL parameter name that contains URL encoded ML document Provide the parameter name that contains a URL encoded ML document. If this field is empty, the AMD will not analyze ML documents sent in URL parameters. 15. Configure availability 41

42 Chapter 5 Configuring AMD to Monitor User-Defined Software Services Select the Availability tab to configure the availability reporting at the software service level, overriding the global settings. The scope of failure reporting depends on an analyzer. For more information, see Configuring Availability in the Data Center Real User Monitoring Administration Guide. 16. Click OK. Excluding IP Ranges from AMD Client Analysis You can exclude particular client IP address ranges from AMD analysis. 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 4. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 5. Select Global Advanced Excluded Client ranges. 6. Provide the start and end IP addresses for each range to exclude from AMD analysis. Be sure not to filter everything out or there will be no data in your reports. 7. On the Devices screen, click Publish Configuration. Managing User-Defined Software Services As a console user you have the ability to add, remove, and edit the properties of any software service defined on any of the AMDs. You can view all user-defined software services on the Software Services screen in the RUM Console. To access this screen, select Software Services Manage Software Services from the top menu. This screen contains information about the user-defined software services that were created on all devices managed from this console. You can add new services, delete existing ones, and copy them to other AMDs. Note that the default software services are specific to a single AMD and cannot be managed centrally. To view user-defined software services monitored by a selected device, access the configuration for this device. Select Devices and Connections Manage devices from the top menu, a list of all devices managed by the console appears. Next, select Open configuration from the context menu for the device. Assigning Software Services to Devices Software services mirrored across different AMDs (that is, having the same name and identical rules) are grouped together. The service name is the name for the group. Whenever you change rules on any of the AMDs in the group, the software services are separated into single entries. After a software service is created on one of the AMDs in your network, you can copy it to another device. To do so, on the Software Services screen, select Copy from the Actions context menu for a given software service. A dialog box appears where you can choose the 42

43 Chapter 5 Configuring AMD to Monitor User-Defined Software Services AMDs by selecting the check box next to their IP addresses. After you click OK, the software service definitions are copied to the selected AMDs. On the Software Services screen, select the Deployment tab and click Change Assignment to modify the list of devices that are monitoring a given software service. 43

44 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 44

45 CHAPTER 6 Configuration Properties Common to All Database Analyzers For any given Agentless Monitoring Device (AMD), you can change global settings for all supported database protocols (TDS, DRDA, Informix, and Oracle) so that the settings are inherited by all user-defined software services for these protocols. In RUM Console, open configuration settings for an AMD and go to Global Database Monitoring General to set global settings for database analyzers. The settings are either common for each analyzer or are dedicated to a particular database (which is indicated in brackets beside the option's label). SQL query time threshold The number of seconds after which a database query is classified as slow. Report full queries Whether full queries are reported in ADS. Full queries are never reported in CAS. When you select full query reporting, you see queries in the form in which they were detected in the network traffic; query normalization rules do not influence the reported strings. If queries contain classified information, you can use masking to prevent certain strings from being reported. See SQL Query Normalization [p. 54] to learn about excluding sensitive information from query reports. Select the check box beside the protocol name that you want to have logged. By default, processing and reporting of full queries is cleared for all protocols, and only normalized queries are reported. Note that only the first 1024 bytes of a full query are logged. Slow query threshold (DRDA only) Whether the slow operation threshold is compared to: Query duration Server delay Report cursor-based queries as single queries Select this option to have cursor-based queries reported as separate operations. Otherwise, all cursor operations are grouped and reported under their respective query operations. 45

46 Chapter 6 Configuration Properties Common to All Database Analyzers Log SQL statements Enables storage and processing of SQL statements on the AMD. Note that the first 1024 bytes of a query are logged. Report database name Enable this option to report the database name, which is used in the CAS reports as a module. For more information, see Multi-Level Hierarchy Reporting in the Data Center Real User Monitoring Central Analysis Server User Guide. Select the check box beside the protocol name for which you want to report the database name. User Identification Select Global Database Monitoring User Identification and enable the AMD to attempt to extract the database user name for the selected database protocols (TDS, DRDA, Informix, MySQL and Oracle). Next, select the protocols for which to extract the database user name. Dynamic ports Select Global Advanced Database to enable or disable the dynamic ports reporting. Use the option with caution, because it switches the AMD into single threaded mode, which may impair the monitoring performance. Dynamic ports monitoring is required for a small fraction of the TDS or Oracle deployments and in new istallations is disabled by default. If you upgraded to current release, the option is selected, but we strongly recommend to disable it, if it is not reuired for your particular deployment. NOTE Some authentication methods do not allow to extract the database user name or database name, for example NTLM authentication for TDS. Also, if the connection traffic is not available, as in case of long-standing connections, the user name and the database name extraction may be not possible. 46

47 CHAPTER 7 Configuration Fine-Tuning While basic database monitoring and analysis can be performed using default settings, you can also fine tune your configuration to obtain a more detailed information about your database traffic. for example, you can configure individual query monitoring, define query aggregation and mask sensitive information in a query. The following chapter will guide you through fine-tuning your database monitoring configuration. Configuring Database Monitoring for Individual Servers Basic database analysis can be performed using default software services definitions. To customize monitoring properties or override global settings, create user-defined software services on a single AMD. Before You Begin See Configuring User-Defined Software Services in the RUM Console Online Help and Configuring Rules for User-Defined Software Services [p. 38] to learn how to create and define user-defined software services. For a broader perspective, see Basic Monitoring Configuration [p. 29]. To configure monitoring of individual SQL servers, you must create user-defined software services. Then you will be able to point to a specific server or servers and, for each of them, create query auto-learning rules and individual masks for SQL query literals. To change monitoring parameters for a defined server: In the Rule Configuration window, set the service address and port and, on the Query Auto-Learning tab, clear the Inherit from global settings check box. Edit the available options when they become active: Limit of queries that can be learned per server The limit of the number of queries that can be learned per server IP address. Default value:

48 Chapter 7 Configuration Fine-Tuning Number of queries tracked in addition to the reported queries The AMD typically tracks more SQL queries than it reports; this is required to accurately pick the most frequently used ones. This parameter defines how many SQL queries are tracked in addition to the reported queries. It is expressed as a percentage of the maximum number of reported queries. Values greater than 100 are allowed. Default value: 100. Number of executions a query has to reach before being learned Defines the number of executions a query has to reach before it will be learned. Default value: 10. Inactivity timeout for query in the learning list Defines, in seconds, the period of inactivity, after which a candidate query is demoted from its candidate status, when the query list is full. Default value: 300 seconds. See, Tuning guidelines [p. 53] for additional help on matching configuration with your needs. On the Query Aggregation tab, you can define additional masking parameters to avoid having sensitive information reported or processed by the AMD and you can also choose one of the available normalization method to be applied at the software service level. For more information, see SQL Query Normalization [p. 54]. On the Options tab, you can change additional monitoring parameters. Enable monitoring of persistent TCP sessions When this is selected, TCP sessions that do not start with SYN packets are monitored. By default, this is selected. Persistent TCP sessions are TCP sessions for which the start was not recorded. They are also referred to as non-syn sessions. These sessions can be included in the TCP statistics, based on the configuration properties you enable in RUM Console. The inclusion of these sessions may render the statistics somewhat inaccurate and must be undertaken with care. SQL query time threshold A database query that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, go to Global Database Monitoring General and set the SQL query time threshold. Generate ADS data Enable this option to provide data to the report server consisting of full SQL queries. SQL Server uses dynamic ports This is a TDS analyzer specific option. Select this option if the database engine you intend to monitor does not have a static port number assigned (for example, a named instance). In this case SQL Server Browser Service (SSBS) will be used to discover the actual port of the service. The AMD will use additional UDP analysis of the SSBS to discover the actual port number for the service you intend to monitor. 48

49 Chapter 7 Configuration Fine-Tuning If you select this option make sure the connection details set in Services tab identify SQL Server Browser Service (use IP address of the server and the port number of the SSBS). Leave this option unselected if your SQL Server uses static ports. Enable user name recognition You can enable the AMD to attempt to extract the database user name for the selected database protocols (TDS, DRDA, Informix, MySQL and Oracle). Next, select the protocols for which to extract the database user name. Report database name Enable this option to report the database name, which is used in the CAS reports as a module. For more information, see Multi-Level Hierarchy Reporting in the Data Center Real User Monitoring Central Analysis Server User Guide. Select the check box beside the protocol name for which you want to report the database name. NOTE Some authentication methods do not allow to extract the database user name or database name, for example NTLM authentication for TDS. Also, if the connection traffic is not available, as in case of long-standing connections, the user name and the database name extraction may be not possible. Individual Query Monitoring The query monitoring feature enables you to define the queries that are always reported regardless of their frequency or performance, in other words, regardless if they comply with query auto-learning requirements. In the Rule Configuration window, select Query Monitoring tab and add or edit the query in the Key Queries table to open Monitored Query dialog. You can either specify a literally given query or use a regular expression that the query will match. Exact Select Exact to add a literally given query. In such case, you have to provide a literal string for the query and its custom name. Regular expression Select Regular expression to use a pattern to match a query. In such case you have to provide a regular expression to match a query and a custom name assigned to the matched occurrences. Note that when applying a regular expression, you can extract part of the query, getting rid of redundant elements. Optionally, you can specify the normalization method applied to queries found by means of regular expression. For more information, see SQL Query Normalization [p. 54]. Optionally, for both methods, you can override a default SQL query time threshold defined at the software service level. A query that takes more than this many seconds is considered slow. When Inherit setting from the rule is selected, the software service setting is used. 49

50 Chapter 7 Configuration Fine-Tuning To report the key queries bypassing the auto-learning scheme, select always report query. If you already defined the key queries and temporarily wish to leave the decision on their monitoring to the query auto-learning scheme, clear always report query, which is a default setting. Note that the queries that bypass the auto-learning mechanism will not be displayed on reports showing queries which most frequently occur or the ones having longest execution time. Configuration effects on reported data Individual query monitoring settings affect data presentation on the CAS with regard to reporting hierarchy levels. Values that you enter in the Query definition section will be reported as operation names. They can be either combined or only the value from either field will be used. If set, Query name will be always visible on reports. If you use regular expressions to define individual queries they will also be reported as operation names. If you do not set the name then the value from Query field will be reported as operation name. Query Auto-Learning The query auto-learning feature enables you to define the set of queries appearing in per-query reporting statistics. The queries found (learned) to fall into at least one of these three categories are reported: the most frequently executed queries; queries with the longest average execution time; queries with the longest overall execution time. The learning process occurs on the AMD. You can turn this feature off, thus causing all queries to be reported, but normally it is desirable to report queries complying one of the above requirements optionally extended by a set of always reported queries defined manually. For more information, see Individual Query Monitoring [p. 49]. The query auto-learning configuration in RUM Console consists of the following properties: High-level Reporting SImplified reporting based on the autodiscovered software services. In this scenario, operation types are used for both tasks and operation names, which saves the system resources. Detailed operation names are reserved for monitoring based on user-defined software services. The autodiscovery feature detects the following operation types: Operation Types ORACLE TDS DRDA MySQL Database statement execution cancel Database login Database logout SQL select 50

51 Chapter 7 Configuration Fine-Tuning Operation Types ORACLE TDS DRDA MySQL SQL select distinct SQL select count SQL create statement SQL execute statement SQL insert SQL update SQL delete SQL alter SQL other statements (such as describe, lock, explain, noaudit, grant, purge, declare, savepoint, rollback, comment, begin, etc) APPS SYS Database other RPC statements SYSPROC DB SYS APPS Prepared statement definition Prepared statement execution 51

52 Chapter 7 Configuration Fine-Tuning Operation Types ORACLE TDS DRDA MySQL Prepared statement destruction Database other operations Transaction rollback Transaction commit Transaction other operations Cursor definition Cursor execution Cursor destruction Detailed Reporting - No Auto-Learning Auto-learning disabled, reporting based on user-defined software services. Auto-Learn Reporting Select this option if you want to enable query learning. If this check box is not selected, all queries are reported. Default: enabled. Limit of queries that can be learned per server The limit of the number of queries that can be learned per server IP address. Default value: 50. Number of queries tracked in addition to the reported queries The AMD typically tracks more SQL queries than it reports; this is required to accurately pick the most frequently used ones. This parameter defines how many SQL queries are tracked in addition to the reported queries. It is expressed as a percentage of the maximum number of reported queries. Values greater than 100 are allowed. Default value: 100. Number of executions a query has to reach before being learned Defines the number of executions a query has to reach before it will be learned. Default value: 10. Inactivity timeout for query in the learning list Defines, in seconds, the period of inactivity, after which a candidate query is demoted from its candidate status, when the query list is full. 52

53 Chapter 7 Configuration Fine-Tuning NOTE Default value: 300 seconds. If the query list reaches its limit, queries are discarded using an algorithm of flushing and aging based on query activity. Tuning guidelines If you want a small and stable number of the most active queries, you need a large set of potential candidates. Use the following settings as guidelines to obtain this goal: Number of queries tracked in addition to the reported queries Set to a high value (for example, 500) to give you a large set of potential candidates. Number of executions a query has to reach before being learned Set to a high value (for example, a few thousand) to define a threshold that only a few queries will attain. Limit of queries that can be learned per server Set to a low value. If you want to explore queries (to get a large number of queries) and if you do not mind if they disappear after some time, adjust the parameters the opposite way: Number of queries tracked in addition to the reported queries Set to a low value. Number of executions a query has to reach before being learned Set to a low value. Limit of queries that can be learned per server Set to a high value. Query Diagnostics You can use the AMD console to obtain information on the performance of the query auto-learning engine. Log on to the AMD using an SSH connection and input the following command: rcon The AMD console program accepts commands for performing management and diagnostic functions. To assess how many queries are monitored, issue the rcon command: show status The following is an example of the relevant portion of the output produced by this command. SQL discovery status: srvc=1 count=0 cand=0 mcand=50 mem=0 mmem=50 added=0 lost=0 rm=0 prom=0 c-shuf=0 m-shuf=0 t-prot=0 c-prot=0 failed: 0 of 1 See Table 2. Status counters [p. 54] for the list of reported counters. 53

54 Chapter 7 Configuration Fine-Tuning Table 2. Status counters Counter count added lost rm cand mem prom c-shuf m-shuf t-prot c-prot Meaning Current number of defined automatic queries (monitored + candidates) Total number of occurrences when an observed query was added to monitoring Total number of occurrences when an observed query was not added to monitoring Total number of queries removed when flushing and aging the list Current number of observed candidates Current number of members (queries monitored) Number of times candidates were promoted to the list of monitored queries Number of times candidates were moved in the lists (shuffled) Number of shuffled members Number of timeout-protected candidates Number of use-protected candidates (candidates that are not deleted from the list because they are being used) To see the current cache content, use the command show queries from the AMD console. The command is either issued without any parameters (and then it displays all the information about all queries) or it can be supplied with the following three parameters, in this order: 1. SQL_Server_IP_address 2. software_service name 3. query For example: show queries " " "ORACLE" "SELECT NULL FROM DUAL FOR UPDATE NOWAIT" Query Aggregation SQL Query Normalization SQL query normalization means, for example, truncation of repeating patterns. Query normalization can be useful if you want to diminish the number of similar queries logged. If, for example, the queries differ only in the value of parameters, they can be truncated at a defined keyword and then only the unique queries will be processed and reported. You can choose an alternative way of gaining uniform, unique queries: mask literal values with a special symbol. You can configure the normalization methods globally, at the software service level or per individual static query, with the query level having the highest priority and the global setting, the lowest. To configure normalization methods, first open configuration settings for an AMD in RUM Console, and then choose how you want them to be configured: 54

55 Chapter 7 Configuration Fine-Tuning globally select Global Database Monitoring Query Aggregation. at the software service level in the Rule Configuration window, click the Query Aggregation tab; at the query level in the Query Monitoring tab, add or edit a query to open the Monitored Query dialog and choose Regular expression as Query type. SQL Query Sensitive Data Masks Literal values (text values) in SQL queries may be pieces of information that you do not want to be visible at any stage of database monitoring (for example, passwords or credit card numbers). If you know that such strings are part of your monitored database traffic, configure literal masking on the AMD using RUM Console. The table SQL query sensitive data masks must be filled with strings that will be used to mask literals in SQL queries. Such a string is a SQL query (or a significant fragment of a query) that unambiguously defines literals that have to be masked. The literal to be masked is represented by the \# symbol, which is an obligatory element in the defined mask, while the \% symbol stands for an ordinary literal. Note that a backslash in this format has a special meaning and must be escaped: to indicate a backslash, a double backslash ( \\ ) must be used. When literal masking is defined, all the matching classified data will be removed and will never be processed by the AMD. Example 1. Masking SQL Query Literals Adding the following mask: WHERE user=\% AND password=\# will result in masking the password literal, but will report user. Adding the following mask: INSERT INTO users (id, name, cardnumber) VALUES (\%, \%, \#) will result in reporting users' ID and name but will mask their card numbers. SQLQuery Normalization Method The panel SQL query normalization method enables you to choose: Queries will be cut before keyword set, values, where, ( Cutting means that queries will be truncated before a specific keyword: where, set, values, or (. If the resulting string is longer than 1024 bytes and the Log SQL Statements option is turned on, it will be further truncated so it does not exceed the default limit. All query literals will be masked using the? character All literal values will be replaced by the? symbol. This normalization method is useful for aggregation of queries - types of queries are grouped regardless of the literals they contain. Optionally, you can exclude certain literals from masking by adding them to SQL query literals to exclude from masking table. As in sensitive literals masking, you can use \% symbol as an obligatory element, and optionally \# symbol 55

56 Chapter 7 Configuration Fine-Tuning Example 2. Excluding Literals from Masking Adding following the pattern to the SQL query literals to exclude from masking table user=\% will result in masking all the literals except user. Note that, although only unique, normalized queries will be reported, the whole query with unmasked literals will still be recorded if the option Log Full Queries was set in the database monitoring global settings. Masking Sensitive Information in SQL Query Literals Use SQL query literal masks to protect confidential information that is part of a SQL query. Literal values (text values) in SQL queries may be pieces of information that you do not want to be visible at any stage of database monitoring (for example, passwords or credit card numbers). If you know that such strings are part of your monitored database traffic, configure literal masking on the AMD using RUM Console. To define masks in the configuration settings for an AMD, select Global Database Monitoring Query Aggregation. The table SQL query sensitive data masks must be filled with strings that will be used to mask literals in SQL queries. Such a string is a SQL query (or a significant fragment of a query) that unambiguously defines literals that have to be masked. The literal to be masked is represented by the \# symbol, while the \% symbol stands for an ordinary literal. Note that a backslash in this format has a special meaning and must be escaped: to indicate a backslash, a double backslash ( \\ ) must be used. When literal masking is defined, all the matching classified data will be removed and will never be processed by the AMD. Example 3. Masking SQL query literals WHERE user=\% AND password=\# INSERT INTO users (id, name, cardnumber) VALUES (\%, \%, \#) This kind of masking can also be set on each AMD (or a range of AMDs) and is defined per user-defined software service or individual query. For more information, see Configuring Database Monitoring for Individual Servers [p. 47] and Managing User-Defined Software Services [p. 42]. Configuring Database Availability To view information about database failures in DMI reports, access the Availability tab and configure the appearance of a specific failure type in a DMI report. You can configure database availability globally or at the software service level. For global configuration, open the AMD configuration and go to Global Database Monitoring Availability <DB Type>. For the software service level, select the Availability tab in the Edit Rule window. 56

57 Chapter 7 Configuration Fine-Tuning Transport Failures No response Disabled by default. Aborted response Enabled by default. Application Failures Database errors The number of database errors. Enabled by default. Database warnings The number of database warnings. Enabled by default. NOTE For DRDA, MySQL, Oracle, and Informix, any detected database errors are grouped together with warnings. As a result, the aggregated error and warning count for these database types appear in the Database Warnings column in a DMI report. For TDS databases, however, errors are differentiated from warnings. For more information, see TDS (Sybase and Microsoft SQL) Errors and Warnings [p. 57]. TDS (Sybase and Microsoft SQL) Errors and Warnings TDS error messages and warnings are interpreted differently depending if they are issued by Microsoft SQL or Sybase. Microsoft SQL An error class level of less than 11 is considered informational and not interpreted as an error or warning. An error class level between 11 and 16 is reported as a database warning. Anything greater than 16 is reported as a database error. Sybase TDS An error class level of less than 11 is considered informational and not interpreted as an error or warning. Anything greater than 10 is reported as a database warning. No errors are reported. Error Class Level Description Informational messages that return status information or report errors that are not severe. Informational messages that return status information or report errors that are not severe. Errors that can be corrected by the user. The given object or entity does not exist. A special severity for SQL statements that do not use locking because of special options. In some cases, read operations performed by these SQL statements could result in inconsistent data, because locks are not taken to guarantee consistency. 57

58 Chapter 7 Configuration Fine-Tuning Error Class Level Description Transaction deadlock errors. Security-related errors, such as permission denied. Syntax errors in the SQL statement. General errors that can be corrected by the user. Software errors that cannot be corrected by the user. These errors require system administrator action. The SQL statement caused the database server to run out of resources (such as memory, locks, or disk space for the database) or to exceed some limit set by the system administrator. There is a problem in the Database Engine software, but the SQL statement completes execution, and the connection to the instance of the Database Engine is maintained. System administrator action is required. A non-configurable Database Engine limit has been exceeded and the current SQL batch has been terminated. Error messages with a severity level of 19 or higher stop the execution of the current SQL batch. Severity level 19 errors are rare and can be corrected only by the system administrator. Error messages with a severity level from 19 through 25 are written to the error log. System problems have occurred. These are fatal errors, which means the Database Engine task that was executing a SQL batch is no longer running. The task records information about what occurred and then terminates. In most cases, the application connection to the instance of the Database Engine can also terminate. If this happens, depending on the problem, the application might not be able to reconnect. DRDA Errors and Warnings See the table listing the detected DRDA error messages and warnings. Table 3. DRDA/DB2 Error Messages for Database Errors (bucket 1): E 42 Invalid Application State Connection Exception SQL or Product Limit Exceeded Miscellaneous SQL or Product Error Resource Not Available or Operator Intervention System Error Dynamic SQL Error Invalid Connection Name Syntax Error or Access Rule Violation 58

59 Chapter 7 Configuration Fine-Tuning Table 4. DRDA/DB2 Error Messages for Database Warnings (bucket 2): 09 0A 0E 0F 0K 0W 0Z D 2F B 3C 3F Triggered Action Exception Feature Not Supported Invalid Schema Name List Specification Invalid Token Resignal When Handler Not Active Prohibited Statement Encountered During Trigger Diagnostics Exception Case Not Found for Case Statement Cardinality Violation Data Exception Constraint Violation Invalid Cursor State Invalid Transaction State Invalid SQL Statement Identifier Triggered Data Change Violation Invalid Transaction State Invalid Transaction Termination SQL Function Exception Invalid Cursor Name Invalid Transaction State Cursor Sensitivity Exception External Function Exception External Function Call Exception Savepoint Exception Ambiguous Cursor Name Invalid Schema (Collection) Name WITH CHECK OPTION Violation Java Errors Invalid Operand or Inconsistent Specification 55: Object Not in Prerequisite State Any values not defined will have a generic ": Database Error" message and get recorded Database Warnings. 59

60 Chapter 7 Configuration Fine-Tuning Oracle Errors and Warnings See the list of detected Oracle errors and warnings. Oracle Errors, Bucket 1 ORA ORA ORA ORA ORA-00108: ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA ORA TNS errors ORA ORA TNS errors ORA ORA All other TNS error codes not listed as informational are bucket 1 Oracle informational, not reported ORA TNS TNS TNS TNS TNS TNS TNS

61 Chapter 7 Configuration Fine-Tuning TNS TNS TNS TNS TNS TNS TNS TNS TNS TNS TNS TNS TNS TNS TNS OCI Oracle warnings, Bucket 2 All other ORA-, and OCI- error codes not listed in bucket 1 nor as informational. 61

62 Chapter 7 Configuration Fine-Tuning 62

63 CHAPTER 8 Monitoring SQL Sequence Transactions You can manage the sequence transactions (operation sequences) that are defined on an individual AMD or manage each transaction that is monitored by a group of AMDs. Viewing All Defined Sequence Transactions To view all transactions, select Reporting Configuration Sequence Transactions from the console top menu. Click in the Sequenced Transactions list and then type the first letters of a sequenced transaction name to find a sequenced transaction whose name matches what you have typed. Click the magnifying glass icon or press [Ctrl+F] to open a search box to limit the table view to only those rows that contain a match (in any column) to the search string. For each transaction, the following information is shown: Sequence Transaction Name The name of a transaction. Application The application that includes the listed transaction. Type The protocol used to define the listed transaction: ASYNC-HTTP, CERNER, CERNER-RTMS, HTTP, OF, SAP GUI, SQL or ML. Packaged Applications Whether the listed transaction is a packaged application whose transactions are recognized by the report server automatically. When you select a transaction by clicking it once, you can see the list of AMDs that monitor this transaction. Viewing Sequence Transactions Defined on an Individual AMD To view the defined transactions monitored by a single AMD, select Devices and Connections Manage Devices from the console top menu. Next, select Open configuration from the context menu for the AMD to access the AMD Configuration screen. Finally, select Configuration Sequence Transactions. 63

64 Chapter 8 Monitoring SQL Sequence Transactions The main Sequence Transactions table lists all of the currently defined transactions and their details: Sequence Transaction Name The name of a transaction. Application Name The application that includes the listed transaction. Type The protocol used to define the listed transaction. Steps The number of individual operations involved in the listed transaction. Priority The priority of the transaction. Possible values are 1 (highest priority), 2, and 3. Timeout The maximum time for the transaction to complete. Packaged Applications Identifies whether the listed transaction is a packaged application whose transactions are recognized by the report server automatically. Viewing Sequence Transaction Details On the Sequence Transactions screen, select Edit from the Actions menu for a given transaction to open the Edit Transaction window and examine the steps that make up the transaction. The listed details are as follow: Name, Application, Description: Timeout [s], Slow after [s], Priority URL, Timeout, Repetition. Managing Existing Sequence Transactions To manage all of the defined transactions, use the Sequence Transactions screen. To create new transactions, click Add Sequence Transaction. The Create Sequence Transaction screen appears, where you can select the AMD devices that will monitor this new transaction. To delete a transaction, select the check box for the transaction and click Delete. NOTE If the transaction you are deleting is monitored by more than one AMD, a new draft configuration must be published to all of the affected AMDs. To edit a transaction, from the Actions context menu for the transaction, select Edit. To copy a transaction to another device, from the Actions context menu for the transaction, select Copy. 64

65 Adding SQL Sequence Transactions You can add a sequence transaction to an individual AMD using the AMD Configuration window. To define a new transaction: 1. In the RUM Console, select Reporting Configuration Sequence Transactions. 2. Click Add Sequence Transaction. The Create Sequence Transaction pop-up window appears. 3. Enter the application and transaction names and a description. If you have configured the Dynatrace connection, click Browse to select a predefined application and a specific transaction within this application. For more information, see Configuring the BSM Connection in RUM Console in the Data Center Real User Monitoring Administration Guide. 4. From the Type list, choose SQL. 5. Select the devices that will monitor the transaction. When you publish the new configuration, it is only applied to these devices. 6. Click OK. On the screen, specify the configuration details for the transaction. 7. Provide the timing and priority values: Chapter 8 Monitoring SQL Sequence Transactions Timeout [s] The maximum time for the transaction to complete. Transactions must complete in this time to be logged as successful transactions. Slow after [s] If the transaction execution time exceeds this value, the transaction is classified as slow. Specify this threshold as fractions of seconds, for example: 0.5. Priority Determines which transaction is recorded if two or more transaction definitions match the transaction detected in the monitored traffic. The valid priority values are 1 (highest priority), 2, and 3. A multiple transaction match can happen if, for example, you first create a generic transaction definition that can match a number of more specific transactions and then you create another transaction definition that matches a particular sub-type of that generic transaction type. If an observed transaction is found to match the latter definition, it also matches the first (more generic) definition, and the system will need to determine under which transaction name to record the observed transaction instance. By increasing the priority of the second, more specific definition, you can count the occurrences of this particular transaction sub-type, which are then not counted in the statistics for the generic transaction type. So you can use this feature to increase the priority of specific customized transaction definitions that should take precedence over more generic transaction templates. 65

66 Chapter 8 Monitoring SQL Sequence Transactions 8. Specify queries comprising the transaction steps. Both SQL operation and query type can contain an optional wild-card character * to signify any number of any characters or a regular expression. You can use either of the two methods in one line. For example, you can use the regular expression based pattern for the query type, regex:rp[abc]$ and simple search pattern for the SQL operation, set*. For more information, see Reported Database Operation Types in the Data Center Real User Monitoring SAP Application Monitoring User Guide. To maintain the sequence of these operations, use the navigation buttons on the right. When using the regular expression in defining the HTTP, Oracle Forms and SQL transaction steps, you have to start the search string with the phrase regex: and follow it a valid regular expression which is applied to the URL, Oracle Forms, SQL operation or query type, for example: regex: For more information, see Using Wildcards in URLs in the Data Center Real User Monitoring Web Application Monitoring User Guide. Using the wildcard character *, you can signify any number of any characters. For more information, see Using Wildcards in URLs in the Data Center Real User Monitoring Web Application Monitoring User Guide. You can Add, Delete, Move Up, Move Down, or Copy the defined steps by selecting the step and clicking one of these actions. You can also make changes in the table itself: click in any of the column cells to edit the values. Using the table, you can determine whether the selected operation may be repeated within this transaction. 9. Click OK to add your transaction definition to a draft configuration. 10. On the Sequence Transactions screen, click Publish Configuration. What to Do Next You can also add a transaction using the Sequence Transaction Inspector. For more information, see Monitoring Sequence Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. Filters and Transaction Inspector for SQL Transactions The Edit SQL Transaction screen enables you to select individual steps and construct your own SQL transactions. The Transaction Inspector consists of two main areas: the Filters and the Transaction Inspector itself. Filters Transactions can be defined manually by entering each step, however the Filters area enables you to examine the Current Stream or Recent Data and select the detected steps to build a transaction. The transaction filter consists of two tabs: 66

67 Data Filter The Data Filter tab enables you to define your filter by selecting the source and range of data to be filtered. From the list, you can select the report server and create a user filter. Select User Name or User IP Address and either enter the data manually or click Browse to open the Select User window, where you can select the user identified in transaction traffic by the report server. You can highlight or search for the specific user or user IP address and filter the search query based on any of the columns in the transactions table. After the user or user IP address is selected, you can choose to either extract transactions from a Current Stream that is being monitored, or from Recent Data stored on the report server. The Recent Data option requires you to provide a Begin and End date for the time range to be processed. NOTE In the case of HTTP asynchronous transactions, you can only extract transactions from Recent Data and you cannot apply the user filters. Result Filter The Result Filter tab consists of a find field, a transaction detail field, and an interactive legend to filter transactions that have been classified as: Table 5. Result Filter Color Guide Chapter 8 Monitoring SQL Sequence Transactions The transaction was recognized and matched with the transaction currently being defined. One or more steps One or more steps An operation was in the transaction in the transactions recognized as an currently being for which an error already defined defined were not occurred. and saved completed. transaction definition. Excluded SQL operation which did not match any definition. By selecting and clearing the corresponding check boxes, you can filter the operations or queries from the data source. The color coding of the steps is based on your current transaction definition in the Transaction Definition area. To view the results and enable the filter to receive data, click, located on the right side of the Filters area. Transaction Inspector Transaction Inspector consists of two tables that display the SQL operations and Sequence Transactions detected in the data source defined in the Filters section. The Transaction Inspector enables you to select one or a number of detected steps and add them to your transaction definition. Select the check box corresponding to the SQL operation that you want to add and click the icon located just above the table. You can add the steps from both the SQL operations table and from the Transactions table. 67

68 Chapter 8 Monitoring SQL Sequence Transactions After the SQL operation is moved to the Transaction Definition table, you can modify it, position it within a sequence of other steps, clone it as another step, or delete it using the operation buttons to the right of the Transaction Definition table. Modifying, Deleting, and Cloning Transactions for a Single AMD Modifying a Sequence Transaction To modify the definition of an existing transaction: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. This opens the Sequence Transactions table, listing all of the defined transactions for this AMD. 3. Right-click the transaction to manage and select Open from the context menu. You can modify any of transaction details. For more information, see Adding Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. Deleting a Sequence Transaction To delete selected transactions: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 3. Click the transaction that you want to delete. To delete multiple transactions with one step, hold the [Ctrl] key as you click additional transactions. 4. Right-click and select Delete to remove the selected transactions from the list. Cloning a sequence transaction To clone selected transactions: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 3. Click the transaction that you want to clone. To clone multiple transactions with one step, hold the [Ctrl] key as you click additional transactions. 4. Right-click and select Clone to duplicate the selected transactions. A cloned transaction is indicated by the original transaction name with (Clone) appended to it. 68

69 Chapter 8 Monitoring SQL Sequence Transactions There are differences between cloning and copying. For more information, see Monitoring Sequence Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. 69

70 Chapter 8 Monitoring SQL Sequence Transactions 70

71 CHAPTER 9 Database Tiers A tier is a specific point where DC RUM collects performance data. It is a logical application layer, a representation of a fragment of your monitored environment. There is one default tier on the CAS that report database data: Database If you have defined software services based on database analyzers, the Database tier will automatically be displayed on the Tiers report. This tier shows measurements for database traffic monitored by AMD. 71

72 Chapter 9 Database Tiers 72

73 CHAPTER 10 Database Traffic on CAS Reports There exist several CAS reports that enable you to analyze database traffic by viewing various statistics. From the Reports menu, view the following reports: Applications: see the performance of applications for which data was detected on front-end tiers if you have defined database applications and transactions on the CAS. For more information, see Application and Transaction Management in the Data Center Real User Monitoring Administration Guide. Tiers: analyze statistics for Database, and RUM sequence transactions tiers. Top N View: view the most problematic software services, operations, and sites. Software Services: analyze statistics for database software services. Network Landing Page: analyze your network performance. User Activity: analyze traffic statistics for particular users. 73

74 Chapter 10 Database Traffic on CAS Reports 74

75 CHAPTER 11 Links to Advanced Diagnostics Server The links to Advanced Diagnostics Server reports are available only if your CAS installation is configured to use a link to related ADS reports, as explained in Report Server Clusters in the Data Center Real User Monitoring Administration Guide. NOTE For ADS reports to be available, the ADS installation has to be configured as a small website. If large Web site configuration is used, only some of the links described below will be available. This limitation has been put in place for performance reasons, since for a large website configuration, the data required to build a report may not always be available on the ADS and would have to be obtained from the raw monitoring data stored on the AMD. For more information, see ADS Basic Configuration Settings in the Data Center Real User Monitoring Administration Guide. Links to Slow Operation Cause Breakdown Report This report shows slow operation loads for a software service or a group of software services, classified by various reasons, plotted by time, and shows the total numbers of slow operations and percentage breakdown for various reasons. This is a starting point for investigating performance problems of software services and it helps you to narrow down areas to investigate, such as network, servers, or page design. For more information, see Slow Operation Cause Breakdown Report in the Data Center Real User Monitoring Central Analysis Server User Guide. How to Access the Report You can access the Slow Operation Cause Breakdown report through links provided in the Slow operations column on the Software Services report. 75

76 Chapter 11 Links to Advanced Diagnostics Server Example 4. Example Link to the Slow Operation Cause Breakdown Report 76

77 APPENDI A Diagnostics and Troubleshooting Report-Related Issues Central Analysis Server automatically detects a range of exceptions (anomalies) and notifies the report users. Exception notifications are displayed as yellow (warning) or red (error) triangle icons in the upper-left corner of the report window. To see the notification message, position the cursor over the triangle icon. The Slow Operation Load Sequence report is empty for an operation which is part of an ML transaction. Why and how do I fix this? For ML and SOAP, Operation Elements data is identical to Operation Analysis data, so, to avoid unnecessarily keeping the duplicates in the database, a VDATA_FILTER_MLSOAP filter is set to true by default. Keeping this filter set to true saves disk space but, because the ML and SOAP entries are filtered out, it makes reporting on the Operation Elements level (elements or headers) impossible. To change the value of VDATA_FILTER_MLSOAP property in userpropertiesadmin, type in the Web browser's Address bar and press [Enter], change the filter's property value, and click Set value to accept the change. To access this screen, you need to have administrative privileges for the report server. The yellow triangle displays AMDs produce no performance data. What do I do? The message AMDs produce no performance data means that AMDs connected to the report server do not produce any new data. To resolve this issue, you have to investigate the configuration of the AMDs and determine why they do not produce the performance data. The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? The report server has a built-in protection from simple configuration mistakes. One of the related problems is when data is incorrectly time stamped by AMD. This happens when the AMD is running with the system clock incorrectly set and is not being synchronized with the report server. If you see this notification, check the system time on the report server and on the AMD. Ensure the time synchronization option is turned on. 77

78 Appendix A Diagnostics and Troubleshooting To check the time synchronization: 1. Launch the RUM Console. 2. Select the AMD, right-click it and choose Open Configuration. The AMD Configuration window appears. 3. Select Global General. Check the IP address of the server authorized to set the AMD time. Make sure it is the same as the report server IP address. 4. Check the report server time setting. Do this by reading the time that is displayed at the bottom of the reports. Ensure the report server has the time zone set correctly. Figure 8. Example of the Report Time Stamp The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? Once a day the report server has to perform a database maintenance and memory cleanup. During that time, the data processing has to be suspended and you will see delayed data on reports. The daily maintenance is usually performed as the first task after midnight and it takes up to half an hour in installations with a large database. It is normal and expected to see this warning just after midnight. But if you see the message during the day, it can be a symptom of incorrect system configuration (check the time settings on the server) or of system overload. The yellow triangle displays No contact with the primary AMD. What do I do? This message indicates that the report server has lost contact with at least one primary AMD. If an AMD is marked as primary and the report server cannot communicate with this AMD, even if the performance data can be downloaded from the other AMDs, the system will wait until the communication with the primary AMD is restored. The yellow triangle displays No contact with any of the AMDs. What do I do? This message indicates that the communication link cannot be established with any of the attached AMDs. Check the network settings on the report server or the configuration of AMDs. 78

79 Appendix A Diagnostics and Troubleshooting The yellow triangle displays Delay in data processing. What do I do? If the last processed data is significantly behind the current time due to slow data processing or idle periods that occurred in the past, the report server displays the triangle icon with the message Delay in data processing. If the server had a delay, but now it is catching up, this message will not appear anymore. To confirm that delay is decreasing, inspect server.log and search for messages similar to this: T REC :10: zdata_43f47e58_5_t is being processed. Sample begin ts = :25. Sample delay 17 min. If the delay becomes smaller, the server is catching up. If the delay values are growing, it can indicate a system overload. The yellow triangle displays The AMD has not yet generated performance data. What do I do? This message indicates that some data files have already been generated on some AMDs, but not on the others. This may not be an indication of a problem and, when you refresh the reports after 30 to 60 seconds, this message may disappear. If necessary, verify the time synchronization among all the AMDs. See The yellow triangle displays Delay in data processing. What do I do? [p. 79]. The yellow triangle displays Data processing is being performed in the debug mode. What do I do? Data processing can be manually suspended and controlled by so-called debug mode, which can be enabled using Control Panel. Open Control Panel by typing: in the Address field of the web browser and clicking Go, then select Controlled data processing from the Configuration Management section. The red exclamation mark displays Data loading is in progress. Reports may be incomplete. What do I do? This message indicates that the report server is currently starting up. Because of this the information presented on reports may be incomplete. Depending on the database size, the startup process may take up to several minutes. If the server restart was not done manually or was not planned, inspect server.log or contact Customer Support. The red exclamation mark displays Low memory. The real-time cache will only be updated. What do I do? This message indicates that the report server has no free memory to process new entities such as software services, servers, and URLs. This message will be cleared when some resources are freed, this usually happens at midnight during the scheduled database maintenance (see The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? [p. 78]). All the metric values presented on reports (except user/client counters) will show correct values. However, the predefined tabular reports may not show all the entities they are intended to show. All the charts and DMI reports show correct data. The mechanism of updating the real-time cache, as described above, is a protection that allows the report server to continue the operation instead of closing down due to lack of memory resources. 79

80 Appendix A Diagnostics and Troubleshooting The red exclamation mark displays The number of servers has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored servers. If the number of observed servers reaches a defined limit, the report server will not accept any new servers and will drop the collected data for those servers. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red exclamation mark displays The number of clients has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored clients. If the number of registered clients (which also includes aggregated virtual clients such as Client from... ) reaches a defined limit, the report server will not accept any new clients and will drop the collected data for those clients. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red exclamation mark displays The number of sites has reached the defined limit. What do I do? The report server has a built-in limit of the number of automatically created sites. If the number of observed automatic sites reaches a defined limit, the report server will not create any new automatic sites and such traffic will be allocated to All Other. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The Sites report for a selected application is empty. Why? If the Sites report for a selected application is filtered for a client tier, such as Synthetic or RUM sequence transactions, it will not show any data. To see statistics for sites, drill down from the Applications report as follows: 1. Click the application name on the Applications report. 2. Click the client tier name on the Tiers report for a selected application. For the Synthetic tier, you will see the Overview Application Status report; for the RUM sequence transactions tier, the Sequence Transactions Log report. 3. Depending on the type of report, click the Overview Site Status or the Sites tab. I see gaps on the chart reports. Why are the charts incomplete? Gaps in reports mean that the report server missed some data and was not able to get it into the database on time. Your reports may resemble the example below. Figure 9. Gaps in a Graphical Report There are several reasons why the graphical reports may have incomplete data: 80

81 Appendix A Diagnostics and Troubleshooting The AMD was not able to detect any traffic from the monitored network, so it was not able to produce any valid data for the report server. To confirm that this was the reason, connect to the AMD using an SSH client and check whether the files named zdata_xxxxx_x_x are located in the /var/spool/adlex/rtm directory. Similar symptoms can be observed if the AMD has been down for some time and data files were not produced for that time. If data files are present and the viewed chart displays only a fragment of the monitored traffic, for example, for a specific server or site, it may indicate that a part of traffic, which was indented to be monitored, is missing. In this situation, the data files are much smaller than usual for the corresponding period of the day. Similar situations, that is, gaps only on some reports, may occur in a multi-amd installation when some AMD s were down or disconnected from the network. In the case when only one AMD is connected to the report server, communication problems do not cause data gaps. If the report server cannot communicate with the AMD, it will wait until the communication is restored and then will process all the data from the past. When there are multiple AMDs connected to the report server and there is a break in communication with only some of them, the report server processes the data from the available AMDs, so in this case, gaps can appear on some reports. If it is a critical issue and your network (or its parts) require continuous monitoring and you cannot miss the data from some AMDs, you have to mark the AMDs as primary. In this case, the report server will wait until the communication with primary AMDs is restored, even if other AMDs are available. Gaps in charts on some reports in multi-amd installations may be caused by unsynchronized AMDs. The reason for that may be that if the report server sees a data file for a specific time period on one of the AMDs, it will wait only 30 seconds for data files covering the same period of time from other AMDs. The 30 seconds are the server's tolerance for time synchronization issues. To verify that this situation occurred, compare the clock readings from AMDs and then check the time synchronization settings (see The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? [p. 77]). It may happen that a part of data will be missing. This will result in a significant decrease of the aggregated data, used to render the chart bars. Note that this effect relates to metrics that are calculated as sums, for example, number of operations, number of errors, number of users, or bandwidth utilization. Charts showing the averages (RTT, loss rate, operation time) will not be affected. I see gaps on the log-term data chart reports. Why are the charts incomplete? The report server aggregates the data collected during the day into daily (and monthly) rollups. This is a scheduled process. If this process is not triggered, you will see gaps in the daily rollups. The most frequent reasons for missing rollups are: 81

82 Appendix A Diagnostics and Troubleshooting The report server was down in the night; report data generation starts at 12:10 AM local time and if the report server was down at that time, no aggregate data for long-term reports will be generated. The report server was overloaded and it took too much time for other crucial tasks; report data generation for long-term reports was canceled. You can always re-generate data for long-term reports. Open Control Panel by typing: in the Address field of the web browser and click Go, then select Regenerate Reports from the System Management section. I created a report that consists of several charts but it loads very slowly. How can I improve its performance? If you are using exactly the same set of dimensions and filters for every chart but would like to show different metrics on separate charts, there are two ways of improving such a report. In this example, it is assumed that you want a report that shows Client bytes, Server bytes, and Total bytes on separate charts for the HTTP analyzer. First, the simplest and recommended method, is to define one section that contains all these three metrics. Figure 10. Creating One Section with Three Metrics Open the Chart settings panel and from the single chart per list select Metric. If you are using metrics with different units, you can select the Metric unit option instead. For more information, see Displaying Multiple Charts in the Data Center Real User Monitoring Data Mining Interface (DMI) User Guide. The second method requires changes on the Subject Data and Result Display tabs. 1. For each report section (chart), create the same set of metrics. To do this, for each chart add metrics that are displayed on the other charts. Note that the order of metrics must be the same in every section. For example, each section must contain the Client bytes, Server bytes, and Total bytes metrics listed exactly in the same order. 2. Disable showing unnecessary metrics for each chart. 82

83 Appendix A Diagnostics and Troubleshooting Go to the Result Display tab and disable showing the redundant metrics. For example, for chart that is going to show only the Client bytes metric, disable showing the Server bytes and Total bytes metrics. Figure 11. Selecting Metrics to Display on a Chart Application performance and availability data is missing from the tabular reports. How can I fix this? The missing data manifests itself as zero or a hyphen. The most frequent reason for this situation is the incorrect setting of business hours and holidays. Inspect the business hours and holiday settings by choosing Settings Report Settings Business Hours. The following configuration screen shows the current settings. Figure 12. Business Hours Configuration Screen To collect performance data seven days per week, including non-business days and holidays, clear the Holidays check box and select the check boxes for weekend days. In addition, you can collect performance data in 24/7 mode, but be aware that this results in a higher database growth rate and a larger database. To enable collecting data all the time, open the Control Panel by opening the following page: In the Control Panel, click Advanced Properties Editor from the Configuration Management section. Set ONLY_BUSS_HOUR_REPORTING to OFF. 83

84 Appendix A Diagnostics and Troubleshooting To see whether your holiday definition is correct, click View Holidays. Figure 13. Defined Holidays Screen The list of holidays is hard-coded and the default set is for the USA. To select a set, click the Choose holiday definition list. To see the content of the selected set, click Preview. To store the newly selected set, click Save. 84