Data Center Real User Monitoring

Transcription

1 Data Center Real User Monitoring Application Monitoring User Guide Release

2 Please direct questions about Data Center Real User Monitoring or comments on this document to: APM Customer Support FrontLine Support Login Page: Copyright 2012 Compuware Corporation. All rights reserved. Unpublished rights reserved under the Copyright Laws of the United States. U.S. GOVERNMENT RIGHTS-Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Compuware Corporation license agreement and as provided in DFARS (a) and (a) (1995), DFARS (c)(1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. Compuware Corporation. This product contains confidential information and trade secrets of Compuware Corporation. Disclosure is prohibited without the prior express written permission of Compuware Corporation. Use of this product is subject to the terms and conditions of the user's License Agreement with Compuware Corporation. Documentation may only be reproduced by Licensee for internal use. The content of this document may not be altered, modified or changed without the express written consent of Compuware Corporation. Compuware Corporation may change the content specified herein at any time, with or without notice. All current Compuware Corporation product documentation can be found at Compuware, FrontLine, Network Monitoring, Private Enterprise, Server Monitoring, Transaction Trace Analysis, Compuware APM, Vantage for Java and.net Monitoring, VantageView, Compuware APM, Real-User Monitoring First Mile, Gomez Performance Network, Data Center Real User Monitoring, dynatrace, and PurePath are trademarks or registered trademarks of Compuware Corporation. Cisco is a trademark or registered trademark of Cisco Systems, Inc. Adobe Reader is a registered trademark of Adobe Systems Incorporated in the United States and/or other countries. All other company and product names are trademarks or registered trademarks of their respective owners. Build: January 3, 2013, 23:22

3 Contents Contents Introduction Who Should Read This Guide Organization of the Guide Related Publications Accessing Customer Support Reporting a Problem Documentation Conventions Chapter 1 Monitoring Overview of Monitoring Chapter 2 Adding Basic DCRUM Devices Adding an AMD to the Devices List Adding Central Analysis Server to Devices List Configuring Connection Between CAS and AMD Chapter 3 Verification of Traffic Monitoring Quality Sniffing Point Diagnostics Sniffing Point Diagnostics Reports Network Interface General Statistics Network and Transport Protocol Information Services Detected in the Traffic Session-Related Statistics SSL Diagnostics Application Overview Using RUM Console to Identify Problems Related to Network Hardware Operation... Chapter 4 Basic Monitoring Configuration Configuring General Data Collector Settings Configuring Operation-Related Global Settings Chapter 5 Configuring AMD to Monitor User-Defined Software Services Defining or Editing User-Defined Software Services Creating User-Defined Software Services

4 Contents Creating User-Defined Software Services for a Range of AMDs Configuring Rules for User-Defined Software Services Excluding IP Ranges from AMD Client Analysis Managing User-Defined Software Services Chapter 6 SMTP Monitoring Configuring SMTP Monitoring Configuring SMTP User Identification Configuring SMTP Error Codes Reporting Creating a New Rule Chapter 7 Microsoft Exchange Monitoring Configuring Exchange Monitoring Monitoring Persistent TCP Sessions Configuring Exchange User Recognition Detecting Exchange Operations Chapter 8 MS Exchange Tiers Appendix A Regular Expression Fundamentals Testing Regular Expressions Best Practices for Regular Expressions Appendix B Diagnostics and Troubleshooting Report-related Issues Glossary Index

5 INTRODUCTION Who Should Read This Guide This manual is intended for users of Data Center Real User Monitoring who want to configure application monitoring. Organization of the Guide This guide is organized as follows: Overview of Monitoring [p. 9] guides you through the process of configuring monitoring of applications, explains how to view monitoring results on the reports and how to troubleshoot issues related to monitoring. Adding Basic DCRUM Devices [p. 13] - Describes how to add and configure the data sources and report servers using the RUM Console. Verification of Traffic Monitoring Quality [p. 17] - Describes how to verify sniffing points traffic detection quality before the actual monitoring begins. Basic Monitoring Configuration [p. 31] - Describes AMD general settings. Configuring AMD to Monitor User-Defined Software Services [p. 39] - Describes the creation and management of -based software services. SMTP Monitoring [p. 47] - Describes how to configure monitoring of a software service based on SMTP protocol. Microsoft Exchange Monitoring [p. 51] - Describes how to configure and monitor a Microsoft Exchange application. MS Exchange Tiers [p. 55] - Describes default tiers that present MS Exchange data. Regular Expression Fundamentals [p. 57] - Describes how to use regular expressions in CAS. Diagnostics and Troubleshooting [p. 63] - Lists common CAS support issues in the form of questions and answers. 5

6 Introduction Related Publications Documentation for your product is distributed on the product media. For DCRUM, it is located in the \Documentation directory. It can also be accessed from the Media Browser. You can also access online documentation for Compuware products via our FrontLine support site at FrontLine provides fast access to information about your Compuware products. You can download documentation and FAQs as well as browse, ask questions and get answers on user forums (requires subscription). The first time you access FrontLine, you are required to register and obtain a password. Registration is free. PDF files can be viewed with Adobe Reader, version 7 or later. If you do not have the Reader application installed, you can download the setup file from the Adobe Web site at Accessing Customer Support Corporate Web Site To access Compuware's site on the Web, go to The Compuware site provides a variety of product and support information. FrontLine Support Web Site You can access online customer support for Compuware products via our FrontLine support site at FrontLine provides fast access to critical information about your Compuware products. You can read or download documentation, frequently asked questions, and product fixes, or your questions or comments. The first time you access FrontLine, you are required to register and obtain a password. Registration is free. Contact Us The contact information to all local Compuware offices is provided on the Web site. All high-priority issues should be reported by phone. Reporting a Problem When contacting APM Customer Support, please provide as much information as possible about your environment and the circumstances that led to the difficulty. You should be ready to provide: Client number. This number is assigned to you by Compuware and is recorded on your sales contract. The version number of the AMD, report servers, and RUM Console with RUM Console Server. Report Server Use the report server GUI by selecting Help Product Information About, or Tools Diagnostics System Status. 6

7 Introduction AMD Scroll down to the Testing AMD section. At the bottom of the diagnostic data paragraph, look for Version ND-RTM v. ndw.ww.x.y.zz. RUM Console and RUM Console Server Use the RUM Console GUI by selecting Help About menu item. TCAM Use the TCAM GUI by selecting Help About menu item. Environment information, such as the operating system and release (including service pack level) on which the product (AMD, report server) is installed, memory, hardware/network specifications, and the names and releases of other applications that were running. Problem description, including screen captures. Exact error messages, if any (screen captures recommended). Whether or not the problem is reproducible. If it is, include a sequence of steps for problem recreation. If it is not, include a description of the actions taken before the problem occurred. A description of the actions that may have been taken to recover from the problem, and their results. Debug information for specific components obtained from RUM Console. Information about the RUM Console itself. To export all the information, navigate to Tools Export Diagnostic Information in the RUM Console menu. Information about the report servers. To export the information for a specific ADS or CAS, right-click the device on the Devices tab in the RUM Console and choose Export Diagnostic Information from the context menu. Include Collector Diag Option to include diagnostic information on data collectors attached to the report server. Save as Destination path and filename for the diagnostic package file. Information about the AMD. To export the information for a specific AMD, right-click the device on the Devices tab in the RUM Console and choose Export Diagnostic Information from the context menu. Include Data Files Option to include fragments of traffic data. Time range Time range of the monitoring data to be included with the diagnostics. Data File Filter (RegEx) Regular expression filter for monitoring data files generated during the defined time range. 7

8 Introduction Save as Destination path and filename for the diagnostic package file. Information from the TCAM System Event log of the machine where the TCAM is operating. TCAM logs which by default are stored in C:\ProgramData\Compuware\VTCAM for Windows Server 2008 and C:\Documents and Settings\All Users\Compuware\VTCAM for Windows Server NOTE Please compress all the files before sending them to Customer Support. Compuware values your comments and suggestions about the Compuware APM products and documentation. Your feedback is very important to us. If you have questions or suggestions for improvement, please let us know. Documentation Conventions The following font conventions are used throughout documentation: This font Bold Citation Documentation Conventions [p. 8] Fixed width Fixed width bold Fixed width italic Menu Item Screen Indicates Terms, commands, and references to names of screen controls and user interface elements. Emphasized text, inline citations, titles of external books or articles. Links to Internet resources and linked references to titles in Compuware documentation. Cited contents of text files, inline examples of code, command line inputs or system outputs. Also file and path names. User input in console commands. Place holders for values of strings, for example as in the command: cd directory_name Menu items. Text screen shots. Code block Blocks of code or fragments of text files. 8

9 CHAPTER 1 Monitoring monitoring refers to monitoring and analysis of SMTP and Exchange protocols. All the properties that SMTP and Exchange analyzers use are set in the RUM Console. You can turn on simple monitoring of these protocols by enabling monitoring of default software services. Then you should acquire basic data based on global settings common to all analyzers. To broaden or narrow the scope of the monitoring process, you need to create user-defined software services and a number of rules that will override default settings and will match your installations. See Basic Monitoring Configuration [p. 31] for information on how to: Use default and user-defined software services. Configure a particular aspect of monitoring. Configure settings common to all protocols (such as activating default service monitoring or defining a new user-defined service on a given IP address and port). Overview of Monitoring AMDs can be configured to monitor specific features of traffic, or to automatically recognize and analyze the users and operations performed by applications and clients. You can also configure many other aspects of database monitoring such as operation recognition or query masking. Before You Begin You should be familiar with DCRUM components and basic monitoring concepts. Refer to the Data Center Real User Monitoring Getting Started. You need to identify your monitoring goals. For more information, see Defining and Prioritizing Goals, Objectives, and Requirements in the Data Center Real User Monitoring Getting Started. You need to install the following DCRUM components: The latest version of AMD 9

10 Chapter 1 Monitoring Refer to the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. The latest version of RUM Console Refer to the Data Center Real User Monitoring RUM Console Installation Guide. The latest version of CAS Refer to the Data Center Real User Monitoring Central Analysis Server Installation Guide. Optionally: The latest version of ADS Refer to the Data Center Real User Monitoring Advanced Diagnostics Server Installation Guide. Make sure that default ports are available for communications between the individual DCRUM components. For more information, see Network Ports Opened for DCRUM in the Data Center Real User Monitoring Administration Guide. The following steps must be executed in order to begin monitoring the traffic using the DCRUM suite: Configuring devices 1. Adding Agentless Monitoring Device (AMD) AMD is the main data source (Data Collector) for DCRUM; it collects and presents the monitored data to DCRUM report servers for analysis and reporting. You need to add at least one AMD to the list of devices in RUM Console. For more information, see Adding an AMD to the Devices List [p. 13]. 2. Adding Central Analysis Server (CAS) CAS is the main report server for DCRUM. It uses data provided by the AMD and its monitoring and alerting mechanisms to identify, track, and report on issues affecting the security, performance, and reliability of your services. Add at least one CAS to the device list and configure its connection with the AMD. Adding a report server to a list of devices is similar to adding the AMD. For more information, see Adding Central Analysis Server to Devices List [p. 14]. 3. Verifying traffic monitoring quality and completeness You can verify traffic quality and completeness before the actual monitoring begins. Sniffing point diagnostics allows you to perform pre-monitoring tasks without the need of accessing the AMD console and executing a series of Linux commands which usually serve the purpose of validating AMD physical installation and connection. For more information, see Verification of Traffic Monitoring Quality [p. 17]. Configuring basic monitoring 4. Configuring general settings for your AMD Before you proceed to detailed monitoring rules, you need to define global settings that will be applied to all software services monitored by a given AMD. These global settings 10

11 Chapter 1 Monitoring include among others a monitoring interval and thresholds for basic metrics, and can be later overridden by more specific monitoring rules that you are going to define. For more information, see Configuring General Data Collector Settings [p. 31] and Configuring Operation-Related Global Settings [p. 35]. 5. Configuring support for WAN optimization All AMDs that monitor network traffic using WAN optimization must be configured before DCRUM can automatically recognize optimized WAN traffic. Otherwise, measurements collected from such an environment will be incorrect. For more information, please refer to Data Center Real User Monitoring WAN Optimization Getting Started. Customizing monitoring rules 6. Defining your own software services on specified ports and for specified IP addresses If you configure user-defined software services, you will be able to see data on many reports provided by DCRUM. 7. Displaying reports to review statistics for monitored traffic Determining the best possible configuration for your needs may be an iterative process, where you will fine-tune the configuration incrementally after viewing your report results. Troubleshooting 8. Troubleshooting problems You can review the answers to the most common questions and troubleshoot your setup and report configurations. For more information, see Diagnostics and Troubleshooting [p. 63]. 11

12 Chapter 1 Monitoring 12

13 CHAPTER 2 Adding Basic DCRUM Devices In DCRUM there are two types of devices: data collectors and report servers. To start using DCRUM, you need to add and configure one AMD, which is the main data source type for DCRUM, and one report server: Central Analysis Server. You manage these devices using the configuration tool called RUM Console. Adding an AMD to the Devices List Before you can monitor traffic with DCRUM, you have to add and configure an Agentless Monitoring Device using the RUM Console. To add an AMD: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click the table and select Add. The Add New Device window appears. 5. In the Description box, type a description of the device. TIP It is good practice to include the parent device name in the description of the device you are adding, and to add such names consistently. This enables you to sort the Devices table by the Description column and visually group the devices. 6. In the Device IP address box, type the device IP address. 7. In the Port box, type the port number for communication with this device. The standard port number used by AMD when communicating over HTTP is For a device communicating with the report server over HTTPS (secure HTTP), set the port number to Optional: In the Secondary IP address for this device field, type an alternative IP address for this device. 13

14 Chapter 2 Adding Basic DCRUM Devices 9. Select the Use secure connection check box if communication with this device should occur via a secure HTTP protocol. 10. Type the user name and password of the account that will be used for managing this device. If you are adding an AMD, the default user name is compuware with the password vantage. The credentials you enter here will be used by the RUM Configuration to communicate with the device and will also be passed to report servers so that they can collect monitoring data for processing. Note that the values used here for authentication are not equal to the values you use for logging in to the device via SSH or local console. 11. From the Device type menu, select AMD. 12. Click Test to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Optional: Click Next to go on to the SNMP configuration page. You may want to define the SNMP connection parameters so you can obtain more detailed health information about the device. To define SNMP connection parameters, select the Enable SNMP check box, type the read community name and port number. 14. Click Next to go on to the page for configuring connection settings for Guided Configuration. By default, the Guided Configuration connection is enabled when you add an AMD. However, for performance reasons, the number of AMDs with enabled Guided Configuration is limited to 15. Any additional AMDs will not feed data to the Guided Configuration perspective. For the user, this means that monitoring data from the additional AMDs will not be available for traffic trace capturing and for generating top Web traffic statistics in the Guided Configuration perspective. By default, the port number for communication between the Console Basic Analyzer Agent and the RUM Console Server is set to 9094 and the secure connection is enabled. In most cases, you will not need to modify this setting. If the default port number is already in use by other services, however, you must type your new port number in the Port number box. In this case, you will also have to manually change the port number setting on the Console Basic Analyzer Agent side. For more information, see Modifying Connection Settings for Guided Configuration in the Data Center Real User Monitoring Administration Guide. 15. Click Finish to save the configuration. The Devices tab in RUM Console presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Health, and Configuration. The Connection column indicates the basic connection status, where red in this column indicates a connection issue and green indicates that the connection is active and the attached device is available. The Configuration column highlights in violet configurations that the console manager detects as identical. Adding Central Analysis Server to Devices List To view reports based on data coming from your AMD, you need to add and configure a CAS using RUM Console. 14

15 Chapter 2 Adding Basic DCRUM Devices To add the device and configure its connection with the AMD: Adding Central Analysis Server 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click the table and select Add. The Add New Device window appears. 5. In the Description box, type a description of the device. TIP It is good practice to include the parent device name in the description of the device you are adding, and to add such names consistently. This enables you to sort the Devices table by the Description column and visually group the devices. 6. In the Device IP address box, type the device IP address. 7. In the Port box, type the port number for communication with this device. The standard port number used by CAS when communicating over HTTP is Optional: In the Secondary IP address for this device field, type an alternative IP address for this device. 9. Select the Use secure connection check box if communication with this device should occur via a secure HTTP protocol. 10. Type the user name and password of the account that will be used for managing this device. 11. From the Device type menu, select CAS. 12. Click Test to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Click Next to specify if the CAS you are adding should be used to get application and tiers configuration and should be a source for application hierarchy for other CASes. Note that if you are adding your first CAS in the RUM Console, this setting will already be configured for you. 14. Click Finish to save the configuration. Configuring the connection between CAS and AMD 15. In the Devices list, right-click your CAS and select Open Configuration from the context menu. 16. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 17. Right-click any row in the Assigned devices table and select Add from the context menu. 18. Select your AMD from the list and then click the button. 19. Click Finish to save the configuration. 15

16 Chapter 2 Adding Basic DCRUM Devices The Devices tab in RUM Console presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Health, and Configuration. What to Do Next It is important to keep the devices time-synchronized to avoid improper data interpretation. For more information, see Synchronizing Time Using NTP Server in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Time Synchronization Between AMD and Server in the Data Center Real User Monitoring Administration Guide. Configuring Connection Between CAS and AMD For CAS to be able to analyze and report data generated by the AMD, you need to configure a connection between CAS and the AMD. To configure a connection between CAS and the AMD: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. In the Devices list, right-click your CAS and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Right-click any row in the Assigned devices table and select Add from the context menu. 7. Select your AMD from the list and then click the button. 8. Click Finish to save the configuration. The Connection column indicates the basic connection status, where red in this column indicates a connection issue and green indicates that the connection is active and the attached device is available. The Configuration column highlights in violet configurations that the console manager detects as identical. 16

17 CHAPTER 3 Verification of Traffic Monitoring Quality With RUM Console you can verify traffic monitoring quality using two tightly connected solutions: Sniffing Point Diagnostics and Application Overview. It is highly recommended that you perform this step at the onset of your DCRUM deployment. This should help you to verify whether your hardware works properly and whether the applications you intend to monitor are detected. Verification can be repeated at any time and for any network conditions. You can verify test results and repeat them as needed on the Guided Configuration perspective of RUM Console. IMPORTANT All verification is based on a traffic recording, either manual or automatic. The outcome may not be representative if the target traffic is low at the time of recording or you are unable to capture a satisfactory number of complete sessions. You can choose between automatic and manual traffic recording, depending on whether you want to capture unfiltered or filtered traffic. However, you should enable automatic recording only during the configuration process and then disable it, because it may negatively affect the performance of the AMD during normal operations, especially if you are running a 32-bit AMD in a high-traffic environment or a 64-bit AMD with the native driver. Complete and most reliable set of statistics is available only if you use the 64-bit customized driver on the AMD. Verification of traffic monitoring quality is possible only for AMD 12.0 or later. Sniffing Point Diagnostics Sniffing Point Diagnostics is a type of hardware state analysis that enables you to perform pre-monitoring tasks without the need to access the AMD terminal. You can employ it as a powerful alternative for a series of UNIX or rcon commands executed to validate AMD the operation of sniffing ports. This step can be performed either at the DCRUM deployment stage or at any later time to see whether AMD performance is affected by malfunctioning hardware or external networking conditions. 17

18 Chapter 3 Verification of Traffic Monitoring Quality Typical issues that Sniffing Point Diagnostics highlights include: No traffic detected on sniffing interfaces. Interface or link overload. Poor quality of traffic due to mirrored ports on switching hardware configuration. Dropped packets (indicate AMD overload). Network conditions when unidirectional traffic prevails. Rejected packets, invalid packets, wrong check sums for packets. Missing packets (either lost or dropped). Missing bytes (how much traffic is lost in general). Conditions affecting AMD calculations, like: Duplicate traffic that cannot be handled by AMD. Incorrect choice of packet deduplication method. Incorrect settings for packet deduplication buffer. Incorrect settings for maximum packet size or huge packet size. Conditions affecting AMD performance, like: Duplicate traffic handled by AMD. Large percentage of non-ip traffic (noise). Large percentage of non-tcp or non-udp traffic (noise). Reordered sessions. Miscellaneous SSL problems: Unsuccessful decryption (in general). Uninitialized SSL cards unable to decrypt traffic. The ratio of encrypted and successfully decrypted traffic to encrypted and non-decrypted traffic. Incorrect or missing private keys. No match between the key and server certificate. Dropped or corrupted packets preventing decryption. Unsupported cipher methods (for example, Diffie-Hellman based key infrastructure) Unsupported SSL versions or features. Prerequisites and best practices To diagnose application detection and the connections of sniffing points, you first need to ensure that: All cables are connected correctly. 18

19 Chapter 3 Verification of Traffic Monitoring Quality AMD is properly installed and configured. This includes post-installation steps such as interface identification and network configuration. Traffic recording lasts long enough to capture a reasonable amount of traffic volume (for example, minutes of traffic). It is recommended that you not use any specific capture profiles when recording traffic. Always use the All available option for capture profiles when you do manual recording. It is recommended that you enable automatic trace recording to have access to regular and fresh snapshots of traffic traveling on your network. This, however, should happen only when you actually need to diagnose traffic or capture port problems. Sniffing Point Diagnostics Reports Sniffing Point Diagnostics is organized in several sections, each presenting a separate set of metrics related to either hardware or network traffic. You can progress through all necessary steps, passing from general information to details, and access all reports in sequence or as needed. Using Sniffing Point Diagnostics reports 1. Start either by looking at device health or from the reports section directly. If you enabled automatic trace recording, you can monitor the device state on the Device Information tab of the Entire Configuration console perspective. A separate set of statistics is provided for each AMD added to the console. If there are any alarm messages, switch to the Guided Configuration perspective and go to the Sniffing Point Diagnostics section. Inspect network interfaces in detail for a selected AMD. Open the Overview report to verify that the proper type of network driver is used (customized or native), that there is traffic detected, and check the number of dropped packets and other performance related issues. You can also verify that the NIC drivers are operational. For more information, see Network Interface General Statistics [p. 20]. 2. Switch to the Protocols section to inspect protocols. See whether network protocols are detected (IPv4 or IPv6) and verify detection of transport protocols (TCP or UDP). For more information, see Network and Transport Protocol Information [p. 22]. 3. Switch to the Services section to see the most active services. For more information, see Services Detected in the Traffic [p. 23]. 4. Depending on your goals, switch to the Sessions section either by selecting a particular service on the Services report to see session details or by choosing the Sessions section to see general statistics for all sessions. For more information, see Session-Related Statistics [p. 23]. 5. If you use SSL decryption, you can inspect whether there are problems detected for the currently used SSL engine or keys. For more information, see SSL Diagnostics [p. 25]. 19

20 Chapter 3 Verification of Traffic Monitoring Quality Network Interface General Statistics The Overview section of the Sniffing Points Diagnostics reports enable you to verify the general state of capture ports on a selected AMD. Information here is gathered directly from the NIC driver operating on the AMD. To obtain the most reliable results, 64-bit customized drivers must be used. Calculation of analyzed traffic The calculation of analyzed traffic is performed in several stages and depends on gradual excluding irrelevant statistics: 1. Overruns are excluded first. When received packets are counted overruns are omitted. 2. Received packets calculation depends on errors and filtered out packets subtraction. 3. Dropped packets start to be counted after filtered out packets are disregarded. 4. Analyzed packets is the amount of packets remaining after all previous categories are subtracted. Note that in default AMD installations non-tcp/udp packets are not part of this process and are never counted when analyzed packets number is given. Non-TCP/UDP traffic increases the amount of analyzed traffic only if you enable monitoring of default software services. 20

21 Chapter 3 Verification of Traffic Monitoring Quality Figure 1. Graphical explanation of analyzed traffic calculation for an AMD with 64-bit customized network interface driver All network packets Overruns Packets not received Received packets Errors and non-conditional filtering Errors: length or bad checksum; filtered out: non-ip Load balancing If active, fraction of the traffic Configuration filtering Based on defined software services Sampling and dropped packets Packets not analyzed due to performance issues Non-TCP, non-udp If default software services enabled Analyzed packets Interface operation-related metrics Some of the most important statistics presented on this screen include: Overruns Overruns may indicate link overload, usually too high traffic volume. This value may also indicate that the network interface or network interface driver cannot manage the amount of traffic received. Other hardware-related issues may also cause overruns. If you get a high overruns value, you should limit the traffic volume received by the card. Errors (length) Packets with erroneous length will be reported when they are too big (like jumbo frames) or are bigger than the maximum transmission unit (MTU). To avoid such problems, you can increase Maximum packet size in the Entire Configuration perspective. For more information, see Configuring General Data Collector Settings [p. 31]. Errors (bad checksum) Checksum-related errors in most cases may occur due to insufficient signal strength on an optical link. In other cases, checksum errors may indicate Ethernet distortion, such as 21

22 Chapter 3 Verification of Traffic Monitoring Quality duplex problems. In this case, checksum errors may result, for example, from a situation when the duplex auto-negotiation process fails. Check the host switch and AMD duplex settings. Filtered out (non-ip) Non-IP packets, such as ARP traffic. Even large numbers of such packets are generally considered harmless. They are not analyzed by the AMD software and are regarded as noise. You are advised, nevertheless, to reduce such traffic reaching AMD to reduce the possibility of performance degradation. Filtered out (load balancing) Applicable only in deployments with multiple AMDs where each device analyses only a given part of the same traffic. For more information, see Load Sharing Between a Number of AMDs in the Data Center Real User Monitoring Administration Guide. Filtered out (configuration) Additional filtering based on software service definitions. In default installations, where monitoring of default software services is turned off, the driver will limit the number of processed packets only to those relevant to the IP addresses included in user-defined software service definitions. Dropped (sampling) Sampling means dropping packets when driver performance is degraded. Packets are dropped in a controlled manner and always with care to preserve complete and therefore consistent sessions. Note that packet drops almost always mean that traffic is too heavy for a complete analysis. Also note that with packet drops the precision of Central Analysis Server reports is diminished. Sampling is active only with the customized 64-bit driver, and diagnostics always use this mechanism regardless of settings used in the general AMD configuration. Dropped (driver performance) Drops are always a symptom of problems, especially when SSL analysis is deployed. Drops occur when AMD software is unable to analyze all packets it receives from the driver. If you use 32-bit or native drivers, you may experience uncontrolled packet dropping. If you use the 64-bit customized driver, packet dropping may occur but in a software controlled manner with care for monitored data contingency. To avoid packet dropping, you should decrease the traffic volume your AMD analyzes or reduce the number of monitored software services. Non TCP/UDP Whether these statistics are classified as analyzed or not depends on default software services monitoring. Numbers in this section are mostly relevant if you enabled monitoring of default software services. In this case, ICMP traffic will also be analyzed. If monitoring of default software services is disabled and you still see a large percentage of non-tcp and non-udp traffic, it is possible that AMD performance will be affected. Network and Transport Protocol Information Use the Protocols report to check the ratio of supported transport or network protocols. Only supported protocols will be shown. In general, this report enables you to check whether traffic 22

23 Chapter 3 Verification of Traffic Monitoring Quality that makes sense (from the DCRUM perspective) is present, and heavy enough to give meaningful results for report servers. NOTE Because of the limited approximation algorithms used by native and 32-bit customized network interface drivers, there may be noticeable differences between the packet count in this and the Overview sections. To obtain the most reliable results, 64-bit customized drivers must be used. Problem detection Low traffic for IPv4 or IPv6 network protocols may indicate further monitoring problems. The presence solely of multicast or broadcast traffic is an indication that port mirroring is not enabled or inactive. Services Detected in the Traffic This overview report enables you to identify most active services in your network. You can see what their load is and what protocols are used by them. You can filter the results to display all data, monitored services, or unmonitored services. You can use filters to display statistics for all, monitored, or non-monitored services with additional protocol filtering. For each service, you are able to open the Sessions report and verify session-level statistics. NOTE Because of the limited approximation algorithms used by native and 32-bit customized network interface drivers, there may be noticeable differences between the packet count in this and the Overview sections. To obtain the most reliable results, 64-bit customized drivers must be used. Session-Related Statistics The Sessions section enables you to investigate more details on traffic quality. Some of the most important statistics presented on this screen include: Duplicates, Unhandled duplicates The value presented on the Sessions screen depends on the currently selected deduplication method in your AMD configuration. Packet duplicates may indicate incorrect configuration of mirroring ports. While this may be a sign of a problem, values of 10 to 20 percent typically are not reason for concern. AMD is capable of packet deduplication. Higher numbers of duplicate packets will degrade AMD performance and may negatively influence monitoring results. The diagnostics mechanism for duplicate detection and counting behind this report works with different settings than the network monitoring processes on AMD. Duplicate detection is performed using both methods of duplicate detection and with different settings (buffer 23

24 Chapter 3 Verification of Traffic Monitoring Quality and delay detection size). Based on these settings and calculations, Sniffing Point Diagnostics will give you suggestions concerning duplicate handling, such as increasing buffer size or changing the deduplication mechanism. It is recommended that you check whether there are unhandled duplicates detected, in which case it will be suggested that you switch the detection method in AMD general settings. For more information on packet deduplication mechanisms on AMD, see Configuring General Data Collector Settings [p. 31]. Unidirectional TCP sessions and UDP streams This may be a distinct problem related to incorrect configuration of mirroring ports. If the value of unidirectional traffic exceeds 90 percent, RUM Console will always mark it as an error. The numbers on the Sessions screen are the sums of many measurements; you are able to go deeper and analyze details for each server and check whether this is a problem related to a significant service or protocol. It may happen that insignificant traffic was recorded and included in general analysis, so you are encouraged to always check detailed reports when alarming numbers appear on the Sessions report. TCP sessions with missing packets Missing packets may result from interface or driver packet drops. If a session with missing packets is shown, the percentage value is counted with regard to all sessions. For example, if two percent of sessions have missing packets reported, this means that two out of a hundred sessions have missing packets. TCP sessions with missing packets and TCP bytes lost in missed packets may provide valuable insight into SSL decryption problems, especially in the case of long SSL sessions. TCP bytes lost in missing packets This is a complementary value to the TCP sessions with missing packets. You should verify the number of lost bytes with regard to missing packets to see whether the problem is serious (if there are large sums of missing bytes). This is useful additional information in the case of long TCP sessions; since only one lost packet classifies a session as having missing packets, the number here gives insight into the actual loss rate. TCP sessions with reordered packets Reordered packets are typically found when there is a WAN link enabled. Devices transferring WAN packets may affect packets order. The existence of reordered packets is not a problem in itself, because AMD software can restore original packet order, but an excessive number of such packets may cause performance degradation. NOTE Because of the limited approximation algorithms used by native and 32-bit customized network interface drivers, there may be noticeable differences between the packet count in this and the Overview sections. To obtain the most reliable results, 64-bit customized drivers must be used. 24

25 SSL Diagnostics Note that this portion of diagnostic information is highly dependent on capturing a large number of complete sessions. Incomplete sessions, missing packets, or missed handshakes will always result in a large number of errors reported and therefore unreliable reports. Always be sure to record traffic heavy enough and for long enough to allow you to capture complete sessions. Also note that a complete Statistics for encrypted traffic, SSL card and keys report is available only after traffic trace recording is finished. No partial statistics for SSL are provided for unfinished sessions. General statistics for encrypted traffic For a given time range (defined by the scope of recorded traffic traces), you can see the recognized SSL engine (for example, OpenSSL or ncipher) and the number of keys exchanged in the traffic. Detailed information on keys can be gathered from the remaining sections of this diagnostic report. You can also see the overall summary of the SSL traffic capture, whether there are errors prevailing or there are no alarming issues. The servers section gathers information on all SSL traffic captured during traffic trace recording. All detected encrypted protocols are listed together with matching keys (if seen in traffic). You are able to see whether key exchange was successful; matched keys are indicated by the icon. Key and certificate matching enables you to verify that certificates were found and were valid (no matching may indicate that certificates are out of date). SSL server status The Status column helps you instantly diagnose whether there are errors or whether erroneous sessions prevail. It may happen that the traffic capture does not contain session beginnings, or that handshakes are incomplete, or that there is no master session; such sessions are marked as ignored (indicated by the gray ( ) color bar). Sessions with errors are marked by the red ( ) color bar. The main error causes may be either missing packets or missing keys. Other error causes are listed in detail on the Detailed SSL Statistics for servers report. Detailed SSL statistics for servers Chapter 3 Verification of Traffic Monitoring Quality Detailed SSL statistics for servers is accessed from the Server or Status columns. This diagnostic report shows the percentage relation of sessions without errors to ignored or erroneous sessions. In case of errors or ignored sessions, you can see numbers for each problem in detail. If there are no problems you just see the number of decrypted sessions. Importantly, you are able to filter the results to display data for completed sessions (use Sessions finished as a filter setting) or for sessions that are still in progress (sessions that did not end before the traffic capture stopped; to see those session statistics, set the filter to Sessions in progress). 25

26 Chapter 3 Verification of Traffic Monitoring Quality Figure 2. An example of detailed SSL statistics for server, errors detected due to private key mismatch SSL keys information Because invalid or outdated keys are usually not removed from the SSL cards, the list of keys for which an error status will be indicated may be considerably long. In such cases, sort by the Status column to see keys correctly matched. Note that it may be necessary to format the SSL card storage area to refresh the key list. Application Overview The Application Overview screen enables you to answer several questions about your applications at the onset of your monitoring configuration. Are all my applications or servers detected? What applications or servers are detected? Can the detected applications or servers be successfully monitored? How heavy is the traffic for each application or server? What services are detected on each server? How heavy is the traffic for each detected service? Note that application detection happens only for complete sessions. If no session beginning has been recorded, no analysis is performed for the session. 26

27 Chapter 3 Verification of Traffic Monitoring Quality Figure 3. Example of the Application Overview screen with detected applications From this screen, you can start configuring software services using either Entire Configuration or Guided Configuration perspective. If it is possible to go through a step-by-step configuration, a wizard icon is displayed for the given protocol or service. Application detection mechanism Application detection is a three-stage process: 1. To provide the most accurate results, packet analysis is performed as a first step of application type detection. For SSL, HTTP, HTTPS, SOAP, and related protocols, packet analysis is performed. Application recognition is based on the first matching pattern found. This means that some services may not be properly classified if multiple protocols are used in one session. For example, if your application uses HTTP and SOAP over HTTP protocols, and plain HTTP communication opens a session, the application will be classified as HTTP. 2. Applications are also detected based on discovery of well-known ports. The default protocols definitions are stored on AMD and can be exported from the RUM Console. For more information, see Exporting AMD Configuration in the Data Center Real User Monitoring Administration Guide. Note that at times your applications may work on ports commonly used for other purposes, AMD is unaware of these circumstances and will report well-known protocol names. For example, if on of your Web application works on port 8080 and uses HTTP protocol for communication AMD will still report this as HTTP proxy. 3. If none of the conditions that are checked matches, the application is labeled as Unknown TCP or Unknown UDP. Server recognition in application detection is based on heuristic session analysis; results may vary depending on the type of network interface driver used. 27

28 Chapter 3 Verification of Traffic Monitoring Quality Using RUM Console to Identify Problems Related to Network Hardware Operation Typical configuration errors related to port mirroring can at times severely affect AMD software traffic analysis capabilities. Faulty hardware configuration may result in no data seen by the AMD, a large number of duplicate packets reaching the AMD, or only a limited portion of traffic visible for the monitoring software. You can use the Application Overview and Sniffing Point Diagnostics sections of the Guided Configuration perspective as tools that enable you to solve issues related to switching hardware configuration. The following list presents several common problems and suggestions on you can track the possible cause. No data seen by the AMD The cable is connected to the wrong physical port on the destination switch. This can be checked by physically tracing the cable directly to the switch and confirming the port ID. The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror the incorrect ports or incorrect destination. This can be resolved by logging onto the source switch and checking the mirroring ports configuration relevant to the requirements (see vendor-specific documentation for details). No data seen on Application Overview but non-tcp/udp traffic seen in interface statistics The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror the incorrect ports or incorrect destination. This can be resolved by logging onto the source switch and checking the mirroring ports configuration relevant to the requirements (see vendor-specific documentation for details). Application Overview does not show all expected data The port mirroring destination may be oversubscribed or dropping packets. Check by logging onto the switch and checking the SPAN or mirror destination interface. If it is recording many drops, review the configuration of source ports to understand the ratio of source interface bandwidth to the destination interface bandwidth. If the ratio is excessive (for example, greater than 4:1) consider reducing the number of source interfaces. If applicable, consider using device-specific filtering control to reduce the load on the destination interface (for example, VACL, Rx-only, or Tx-only sources). By design, port mirroring will not forward faulty frames. Check the source device interface statistics to ascertain the nature of the drops (refer to vendor-specific documentation for details). Check interface-related metrics. If there is a high rate of Errors (bad checksum), consider hard-configuring one end of the AMD SPAN connection to prevent auto negotiation from taking place. Session-related report shows a high rate of packet duplicates SPAN or mirror operates by copying frames from source interfaces and directing them to the destination interface. In effect, configurations often result in the same packet being 28

29 Chapter 3 Verification of Traffic Monitoring Quality copied twice. For example, if the source of a SPAN or mirror is set as a VLAN, any traffic that goes from one switch port to another switch port within the VLAN appears twice on the mirrored port. If the number of duplicates starts to affect AMD performance, consider reducing the number of source interfaces. If applicable, consider using device-specific filtering control to reduce the instances of copying the same frame more than once (for example, VACL, Rx-only, or Tx-only sources) or consider using tap technology as opposed to port mirroring to collect the data. Only unidirectional streams are seen on session-related overview If the AMD is connected via a SPAN or mirror, the configuration has been set incorrectly to send only one side of a Rx or Tx stream to the destination. Log on to the local source switch to check the configuration (refer to vendor-specific documentation for details). 29

30 Chapter 3 Verification of Traffic Monitoring Quality 30

31 CHAPTER 4 Basic Monitoring Configuration Many configuration settings can be defined globally for all software services for a given protocol and Data Collector or locally, for specific user-defined software services. If both types of settings are specified, the settings for a specific user-defined software service take precedence over the corresponding global settings. Basic monitoring configuration is performed in the RUM Console. It involves global settings for Data Collectors, operations, and the analyzer, as well as configuring Compuware APM to recognize WAN-optimized traffic. Configuration and recognition of optimized WAN traffic in Compuware APM is optional and it depends on whether WAN optimization technology is used in your network setup. Refer to the Data Center Real User Monitoring WAN Optimization Getting Started. NOTE If you make any significant changes in the configuration, like removing defined software services or operations, you are advised to restart the AMD. This will prevent a situation in which a persistent TCP session blocks the changes. Configuring General Data Collector Settings For any given data collector device, such as Agentless Monitoring Device, you can set a variety of options such as time thresholds. The general settings affect monitoring of default software services, as well as user-defined ones, and some of them can then be overridden by specific settings for a particular analyzer, software service, or URL. To define general settings for an AMD: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 31

32 Chapter 4 Basic Monitoring Configuration 6. Navigate to Configuration Global General to access the list of general configuration settings. While some of the options control only general AMD behavior, some options in the Advanced group affect more specific configurations in application monitoring. For example, if the Inherit from global settings option is selected in your other configurations while configuring user-defined software services, the global setting takes precedence over the specific monitoring configuration. Configuration options include: Monitoring interval The monitoring interval in minutes. Increasing this value reduces the number of chunks of data that need to be transferred and processed. Default: 5 minutes. Make sure that the monitoring interval is synchronized between the data collectors. Page load time/operation time threshold The number of seconds after which a page or operation is considered to be slow. The global threshold value depends on the analyzer. The threshold is used by following analyzers: Cerner Cerner over MQ Epic Generic with transactions HTTP MS Exchange over HTTP MS Exchange over HTTPS Oracle Applications over HTTP Oracle Applications over HTTPS SAP GUI SAP GUI over HTTP SAP GUI over HTTPS SMTP SSL SSL Decrypted Siebel over HTTP Siebel over HTTPS Server time threshold Server time threshold relates to the server time portion of a overall page load time. Server times above the threshold limit are considered to be slow due to poor datacenter performance. The threshold is used by following analyzers: HTTP Oracle Applications over HTTP Oracle Applications over HTTPS 32

33 Chapter 4 Basic Monitoring Configuration SAP GUI over HTTP SAP GUI over HTTPS IP address of the server authorized to set AMD time The IP address of the report server that has the authority to synchronize the time with this AMD. In an environment with a number of servers sharing the same AMD, it is good practice to use only one of these servers (a designated chosen time synchronization server) to make changes to AMD settings. Otherwise, the server used for time synchronization will change inadvertently every time you save an AMD configuration. Default analyzer The default setting for the TCP analyzer is Generic (with transactions). To change it, select another analyzer from the list. Client RST packet timeout to mark session as CLOSED If the time between the last ACK for data sent by the server and an RST packet sent by the client is greater than this value, the session is treated as closed instead of aborted. Huge packet size The upper size limit, in bytes, of an HTTP request to be processed successfully by the AMD. Maximum packet size AMD is capable of processing packets of up to bytes, besides the Ethernet standard MTU (Maximum Transmission Unit) of 1536 bytes. Choose one of the predefined values, 2048, 4096, 8192 or bytes to enable AMD to process non-standard MTU packets. When you have chosen the Maximum packet size value make sure that you also set the Huge packet size to an applicable value. Enabling AMD to process non standard MTU packets without extending RAM on the machine and leaving Packet buffer size (64-bit AMDs only) and Data memory limit unchanged can cause an exceeded packet loss. To avoid it, extend RAM and configure its usage as recommended in the tables below. For more information, see Setting Packet Buffer Size in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Setting Data Memory Limit in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. Table 1. Recommended RAM configuration for Maximum packet size values for 64-bit AMDs Maximum packet size RAM size 64-bit minimum (recommended) Data memory limit Packet buffer size Packet buffer count 1522 B 4 GB (32 GB) 1024 MB 1832 MB 1,200, B 5 GB (32 GB) 1024 MB 2289 MB 1,500,000 33

34 Chapter 4 Basic Monitoring Configuration Table 1. Recommended RAM configuration for Maximum packet size values for 64-bit AMDs (continued) Maximum packet size RAM size 64-bit minimum (recommended) Data memory limit Packet buffer size Packet buffer count 2048 B 6 GB (32 GB) 1024 MB 3022 MB 1,500, B 12 GB (32 GB) 1024 MB 5951 MB 1,500, B 22 GB (48 GB) 1024 MB MB 1,500, B 32 GB (64 GB) 1024 MB MB 1,000, B 48 GB (64 GB) 1024 MB MB 1,500,000 Sampling enabled Applicable when the 64-bit customized driver is used. The sampling mechanism is very beneficial when heavy traffic may negatively affect AMD performance and there is a risk of losing IP session consistency. When this option is enabled, the AMD tries to analyze the greatest possible portion of traffic, dropping packets in a controlled manner and with care to preserve complete and consistent sessions. Note that statistics for dropped packets will not be shown on the report server. If packets are dropped because of sampling, the Central Analysis Server shows notification messages. For percentages between 75 and 99, you will see a warning icon; for values below 75, the report server will issue error messages. When this option is disabled and network interface driver performance is degraded, random packets are dropped. Default: enabled. For more information, see Using Network Interfaces with Native Drivers in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Changing Driver Type for Network Interfaces in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. Deduplication method You can choose one of two methods for eliminating duplicate packets: Based on TCP checksum and IP ID Using this method, duplicate packets are detected based on their TCP checksum and IP ID. Based on TCP checksum with excluded SEQ and ACK numbers Using this (more complex, two-stage) method, duplicate packets are detected based on a modified packet TCP checksum (SEQ and ACK numbers are excluded) and IP ID. This method is useful if the AMD captures packets on various interfaces of the router, rewriting SEQ and ACK numbers. A packet is considered a duplicate when the modified checksum, IP ID, and SEQ and ACK numbers are identical. First, a packet checksum with SEQ and ACK numbers is created and compared to the packets stored in the detection buffer. 34

35 Chapter 4 Basic Monitoring Configuration If the comparison indicates that the packet is not a duplicate, it is checked to determine whether it matches current session. A packet matches the current session when its SEQ and ACK numbers are different from processed and cached numbers by the value defined in TCP duplicate window. If the difference exceeds the defined value, the AMD assumes the ACK and SEQ numbers were rewritten by the router and the packet is considered a duplicate. Default: Based on TCP checksum and IP ID. TCP duplicate window This setting is useful only if Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers. It is used for determining whether a packet, based on its SEQ and ACK numbers, belongs in the session. If a packet's SEQ and ACK numbers differ from current session's SEQ and ACK numbers by a value larger than TCP duplicate window, the packet is considered a duplicate. Default: Packet buffer size The number of packets to keep in the buffer for use as a basis for comparison in duplicate packet detection. Newly captured packets are sequentially compared to the packets in the buffer. A newly captured non-duplicate packet (all packets in the buffer are unique) is placed on the top of stack and the oldest is removed. Range: 6 to 24 packets. Default: 16. Reset duplicate detection time threshold Time of inactivity (in seconds) after which the duplicate packets elimination mechanism is reset. Especially, if Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers, the Reset duplicate detection time threshold should be greater than every response generation time (server time). NOTE In the case of Java/.NET Monitoring, the only available options are Monitoring interval and Page load time/operation time threshold. Using different database engine vendors for Java/.NET Monitoring Manager Devices connected to the same CAS is not supported. 7. Publish the draft configuration on the monitoring device. a. In the Entire Configuration perspective, select the Devices tab. b. Right-click the device and select Publish Draft Configuration from the context menu. Configuring Operation-Related Global Settings Global settings for operations enable you to define options that can apply to all monitored operations and take precedence over options defined for individual operations. To specify basic settings related to operations: 1. Start and log in to RUM Console. 35

36 Chapter 4 Basic Monitoring Configuration 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Select Configuration Global Operations to display general configuration settings. Options include: Operation load time threshold Number of seconds after which an operation is considered slow. You can set this value with a precision of one ten-thousandth of a second. Default: 0.5 seconds. The threshold is used by following analyzers: IBM over MQ JOLT MS Exchange Oracle Forms over HTTP Oracle Forms over HTTPS Oracle Forms over SSL Oracle Forms over TCP SOAP over HTTP SOAP over HTTPS XML XML over HTTP XML over HTTPS XML over MQ XML over SSL Max. operation duration Maximum number of seconds an operation can take. You can set this value with a precision of one ten-thousandth of a second. Default: 3600 seconds (1 hour). User abort threshold Minimum number of seconds between the beginning of a hit and TCP reset to count it as a user abort. Default value is seconds. You can set this value with a precision of one ten-thousandth of a second. Default: 1.6 seconds. ADS data generation settings Options in the ADS data generation settings section refer mainly to ways of handling various types of standalone hits, which are hits that cannot be automatically assigned to operations because reference information (such as correlating response, defined or auto-learned URL, no authorization, or orphaned redirects) is missing. By default, most standalone hits are not taken into account when generating operations data. 36

37 Chapter 4 Basic Monitoring Configuration Report data without monitored URL Select this option to report data for hits without a URL that has been explicitly defined in user-defined services or recorded through auto-learning. Report hits without response header Select this option to report data for discarded hits (hits without a correlating response header). Report hits not added to any operation Select this option to report data for other standalone hits caused by factors not covered by other options of this section. Report unauthorized hits Select this option to report data for hits with rejected authentication. Report orphaned redirects Select this option to report data for redirects to sites that are not being monitored or are not visible and therefore appear as orphaned redirects. Report filtered data This is a diagnostics option. When configuring content type monitoring, you can filter out pages based on the content of the URL. For more information, see Monitoring of Non-HTML Objects Based on Content Type in the Data Center Real User Monitoring Web Application Monitoring User Guide. If you select this option, the filtered out pages will not be reported, but will still be saved in the AMD data files. Ignored clients List of clients for which TCP setup time will always be ignored and all operations will start from the request packet. Right-click the list to open a menu of command options: Add, Edit, or Delete. 7. Publish the draft configuration on the monitoring device. a. In the Entire Configuration perspective, select the Devices tab. b. Right-click the device and select Publish Draft Configuration from the context menu. 37

38 Chapter 4 Basic Monitoring Configuration 38

39 CHAPTER 5 Configuring AMD to Monitor User-Defined Software Services If you require detailed analysis of traffic, you need to specify software services to be monitored on specific IP addresses and ports. Defining or Editing User-Defined Software Services If you require more detailed analysis of traffic or if software services you intend to monitor do not work on well-known ports, you can specify software services to be monitored on specific IP addresses and ports. For such software services, a much wider range of metrics is measured and detailed traffic analysis is performed. You can create new user-defined software services in several ways, depending on your system and on the traffic you want to monitor: To create a new software service based on HTTP, HTTPS, SOAP over HTTP, SOAP over HTTPS, XML over HTTP and XML over HTTPS, we recommend that you use Working with Guided Configuration Perspective in RUM Console in the Data Center Real User Monitoring Web Application Monitoring User Guide. To define software services based on other protocols, use one of the following procedures: Creating User-Defined Software Services [p. 39]: to define and manage software services on per-amd basis. If your system is using just one AMD, you may find it easiest to manage all of your software service definitions from within the AMD configuration. Creating User-Defined Software Services for a Range of AMDs [p. 40]: to create and manage software services independently of devices. You will always be able to assign newly created software services to one or more AMDs later. Note that you can define software services based on protocols other than HTTP/HTTPS only on an AMD and not for Java/.NET Monitoring. Creating User-Defined Software Services User-defined software services can be defined and managed on per-amd basis. 39

40 Chapter 5 Configuring AMD to Monitor User-Defined Software Services To define or edit a user-defined software service for an AMD: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Software Services User-Defined Software Services. 7. In the Software Services table, right-click to open the context menu and choose Add or Open. For a newly created item: in the Software Service dialog, provide the software service name, network protocol on which the software service is based, and the analyzer name used to parse the software service traffic. For an already existing item: you can change the name of the software service but not the type of the analyzer. You can add new rules or edit the existing ones. For more information, see Configuring Rules for User-Defined Software Services [p. 41]. Note that if you change the name of the software service, it will no longer be grouped together with other software services sharing the same name and identical rules in the Software Services tab in the main screen of RUM Console. Changing the name and rules to match another software service will group them together in the global software services tab. For more information, see Managing User-Defined Software Services [p. 45]. When you delete a software service, you also delete rules associated with this item. You will not be able to recover the rule sets or transfer them to another AMD. 8. Click OK to update the list of user-defined software services. 9. In the Software Services table select the name of software service to configure. 10. In the Rules table, right-click to open the context menu and choose Add or Open. The Rule Configuration window will open. 11. Complete the monitoring configuration by specifying additional rules as described in Configuring Rules for User-Defined Software Services [p. 41]. Creating User-Defined Software Services for a Range of AMDs User-defined software services can be created on more than one AMD at once. To define or add a software service for a range of AMDs: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Navigate to the User-Defined Software Services tab. 4. Right-click to open the context menu and choose Add New Software Service or Open. When adding a new software service: 40

41 Chapter 5 Configuring AMD to Monitor User-Defined Software Services a. In the Software service description panel of the Add Software Service dialog, provide a software service name and an analyzer type. b. Apply the definition to a selected number of AMDs. In the Apply to AMDs panel of the Add Software Service dialog, select the check box beside the addresses of each AMD that you want this service to be active on. When editing a software service, you can start adding, editing, or deleting rules immediately. The Edit Software Service dialog will open. 5. In the Edit Software Service dialog, specify rules for the newly defined software service. With this dialog, you can add a new set of rules, or delete or edit the existing ones. Right-click to open the context menu and choose Add, Open, or Delete. 6. Complete the monitoring configuration by specifying additional rules as described in Configuring Rules for User-Defined Software Services [p. 41]. 7. Click OK to save the new configuration. The new definition will be propagated to all the AMDs you selected when working on the software service configuration. Configuring Rules for User-Defined Software Services Each software service can have a number of specific rules that define what is to be monitored and what additional options are in effect. You can also assign each software service to existing or newly created tiers and applications. Before You Begin It is assumed for this task that you are already familiar with the concept of software services and that you know how to create and edit software services and how to open the Rules window. For more information, see Defining or Editing User-Defined Software Services [p. 39]. After a user-defined software service is created, you need to create a group of settings that comprise the rules for the software service. It is necessary to specify, at minimum, the IP addresses and port numbers for the software service. If you define software services for both AMD and Java/.NET Monitoring devices, when an AMD monitored software service applies to an IP address monitored by a Java/.NET Monitoring device, the software service rules are automatically transferred to Java/.NET Monitoring and need not to be configured separately on a Java/.NET Monitoring device. This rule does not apply to General Configuration Settings such as Monitoring interval which is set separately and must be kept synchronized across Data Collectors. For more information, see Configuring General Data Collector Settings [p. 31]. To configure rules for a user-defined software service: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 41

42 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 3. Navigate to the User-Defined Software Services tab. 4. Right-click a specific software service and select Open from the context menu. It will result in opening of the Edit Software Service window. 5. In the Rules table, right-click to open the context menu and choose Add or Open. The Rule Configuration window will open. 6. On the Services tab, select or clear Enabled to activate or de-activate the service definition. 7. In Rule description, type a brief description that will later help to identify the rule. The description you enter at this point will be shown in the Rules table, in the column Rule Name. If no text is entered here, the IP address specified later will be used as a description for this rule. 8. Right-click in the Services table and select Add or Open from the context menu. The Service Details window will pop up when adding or editing rules. 9. In the Service Details window, in the IP address(es) fields, enter the server IP address, or enter a range of IP addresses if you plan to monitor more than one server. 10. In the Port(s) fields, enter the port number of the monitored service. You can provide a range of port numbers if such a range of ports is used in your environment. Some software services may be active on a number of predefined ports or may change ports dynamically. To allow for this, you can specify a range of ports. Note, however, that specifying more than one port for a service prevents the port number from being reported for that service. If you define more than one port for a particular service name and server IP address (by either specifying a range of ports or by creating two or more distinct rules for the same service name and server IP address but with different port numbers), the AMD will report the port number for this service as 0, causing the port number to be ignored in traffic reports. NOTE You can define up to 5000 definitions containing a server and a port. Each association of a server and a port counts as a single definition. Specifying a range of ports counts as providing many individual definitions. On CAS, the number of processed server definitions is limited by the license. For more information, see Maximum Monitored Entities Licensing in the Data Center Real User Monitoring Administration Guide. 11. Optional: Select Client port(s) for reversed-direction protocols. This option makes sense only for protocols such as X-Window whose client-server meanings are reversed. If you are uncertain, leave this option cleared. 12. Optional: Enter a virtual IP address if your network uses a pool of virtual IP addresses. For more information, see Virtual IP address [p. 87]. 13. Optional: Enter the IP address of the server masking the addresses of monitored servers. If the servers you intend to monitor reside behind an appliance that masks and replaces the addresses of the target servers, you need to set NLB NAT masking IP address to the IP address of the masking server. 42

43 Chapter 5 Configuring AMD to Monitor User-Defined Software Services Without doing so, the AMD will see two unidirectional conversations instead of one bi-directional conversation between the servers and appliance: The conversation between the client and server is observed and recorded (IP address A talking to IP address B) When a response travels to the client, a different session (IP address C talking to IP address A) is recorded due to the server's IP address being replaced by the load balancer's IP address. Unless you account for this, CAS reports will return reports with ambiguously granulated data. Using the NLB NAT masking IP address option will ensure that the AMD monitors contiguous conversations. 14. Click OK to confirm your changes and close the Service Details window. 15. Go through all available tabs and fine-tune the monitoring conditions. The number of available configuration options depends on the analyzer. See the analyzer-specific section for more information. 16. Optional: In the Options tab, define analyzer-specific options. The following list describes all possible options. Depending on the analyzer, some may be unavailable: Operation load time threshold An operation that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. The global threshold value depends on the analyzer. Page load time/operation time threshold A page load that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, go to Global General and set the Page load time/operation time threshold. Server time threshold Server time threshold relates to the server time portion of a overall page load time. Server times above the threshold limit are considered to be slow due to the poor datacenter performance. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, go to Global General and set the Server time threshold. SQL query time threshold A database query that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, go to Global Database Monitoring General and set the SQL query time threshold. Enable monitoring of persistent TCP sessions When this is selected, TCP sessions not starting with SYN packets are monitored. Persistent TCP sessions, that is TCP sessions for which the start was not recorded, also referred to as non-syn sessions, can be included in the TCP statistics, based on 43

44 Chapter 5 Configuring AMD to Monitor User-Defined Software Services configuration properties you enable in RUM Console. The inclusion of such sessions may render the statistics somewhat inaccurate and must be undertaken with care. Generate transactions and ADS data Select this option to provide data to the report server consisting of, for example, lower-level protocol information: raw HTTP traffic data, enabling you to view the full HTTP request-response dialog. SQL Server uses dynamic ports This is a TDS analyzer specific option. Select this option if the database engine you intend to monitor does not have a static port number assigned (for example, a named instance). In this case SQL Server Browser Service (SSBS) will be used to discover the actual port of the service. The AMD will use additional UDP analysis of the SSBS to discover the actual port number for the service you intend to monitor. If you select this option make sure the connection details set in Services tab identify SQL Server Browser Service (use IP address of the server and the port number of the SSBS). Leave this option unselected if your SQL Server uses static ports. Convert the XML content URL-encoding This check box defines whether the XML URL-encoding content is enabled. When Inherit from global setting is selected, the global XML setting is used. URL parameter name that contains URL encoded XML document Provide the parameter name that contains a URL encoded XML document. If your leave the field empty, the AMD will not analyze XML documents sent in URL parameters. 17. Click OK to save the new configuration. 18. Publish the draft configuration on the monitoring device. a. In the Entire Configuration perspective, select the Devices tab. b. Right-click the device and select Publish Draft Configuration from the context menu. Excluding IP Ranges from AMD Client Analysis You can exclude particular client IP address ranges from AMD analysis. To narrow the range of monitored clients: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Global Advanced Excluded Client ranges. 7. Provide the start and end IP addresses for each range you want to exclude from AMD analysis. 44

45 Chapter 5 Configuring AMD to Monitor User-Defined Software Services Be sure not to filter everything out or you will have no data at all in your reports. 8. Publish the draft configuration on the monitoring device. a. In the Entire Configuration perspective, select the Devices tab. b. Right-click the device and select Publish Draft Configuration from the context menu. Managing User-Defined Software Services As a console user you have the ability to add, remove, and edit the properties of any software service defined on any of your AMDs. The User-Defined Software Services tab in the main screen of the RUM Console contains information about all user-defined software services created on all devices managed via this console. From here you can manage services, add new ones, delete existing ones, and copy them across AMDs. Note that default software services are specific to a single AMD and cannot be managed centrally. User-defined software services appear both in the User-Defined Software Services tab in the main RUM Console window and in Monitoring section of AMD Configuration. The first view presents you with all software services defined on any AMD in your Compuware APM deployment; the latter shows software services defined only on this particular AMD. For more information, see Defining or Editing User-Defined Software Services [p. 39]. Assigning software services to devices Software services mirrored across different AMDs (that is, having the same name and identical rules) are grouped together. The service name is the name for the group. Whenever you change rules on any of the AMDs in the group, the software services will be broken into single entries. After a software service is created on one of the AMDs in your network, you can copy it to another device. To do so, right-click the name of the software service and choose Copy. A dialog will appear where you can choose AMDs by selecting the check box beside their IP addresses. After you click OK, the software service definitions will be copied to the selected AMDs. You can remove AMDs from the group by right-clicking on one or several selected devices and using the Delete selected Software Services option. 45

46 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 46

47 CHAPTER 6 SMTP Monitoring The SMTP analyzer provides configuration settings to customize monitoring of software services based on SMTP. You can monitor SMTP traffic based on various configuration settings, such as user identification, collecting information on messages and attachments, and reported error codes. Configuring SMTP Monitoring You have to define a software service based on the SMTP analyzer to monitor SMTP traffic in your network. Assuming we have an SMTP server with IP address and port number 25. To create and configure a new software service based on the SMTP analyzer: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Software Services User-Defined Software Services. 7. In the Software Services table, right-click to open the context menu and choose Add. 8. In the Software Service dialog box, provide the service name, select TCP as the base protocol, and select the SMTP analyzer. 9. Click OK to add the new item to the list of user-defined software services. 10. In the Rules table, right-click to open the context menu and choose Add. For more information, see Configuring Rules for User-Defined Software Services [p. 41]. You have to create at least one rule for monitoring of this software service. After the rules are created, they all are displayed in the Rules table. 11. Publish the draft configuration on the monitoring device. a. In the Entire Configuration perspective, select the Devices tab. b. Right-click the device and select Publish Draft Configuration from the context menu. 47

48 Chapter 6 SMTP Monitoring Configuring SMTP User Identification To enable and configure SMTP user identification: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Configuration Global Other Protocols Monitoring SMTP 7. Set the Identification to ON. 8. Select the Derive user identification from domain of the FROM field checkbox. 9. Define an extended POSIX regular expression to extract the user identification from the FROM or TO field. To use a regular expression to extract the user identification from the FROM field, define the regular expression in the User identification extraction regular expression edit box and make sure that the identification method, determined in the Identification list is not Off; it can be Derived from the FROM and TO message fields or Limit to the FROM domain name. Then make sure that one of the following is true: The address of the sending client is on the list of monitored servers. The address of the receiving server is included in the software service definition and the sending client's IP address belongs to a range defined in the IP Address Ranges table. The address of the receiving server is included in the software service definition and the domain of the client matches one of the domains listed in the Domains of Interest list. To use a regular expression to extract the user identification from the TO field, define the regular expression in the User identification extraction regular expression field and set Identification to Derived from the FROM and TO message fields. The IP address of the receiving server should be included in the software service definition. For example, the following regex: (.*)@mycompany.com$ will match john.smith@mycompany and will extract john.smith You can test patterns that will be used by the AMD by means of Regular Expressions Test tool which is activated after you click the Test button located beside regular expression pattern field. For more information, see Testing Regular Expressions [p. 59]. 10. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. 48

49 Configuring SMTP Error Codes Reporting The error codes are reported for software services based on the SMTP analyzer. The default list of error codes can be freely modified according to your needs. Based on the lists, errors observed in monitored traffic are added to appropriate counters and reported accordingly. To modify the SMTP error code reporting list: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Configuration Global Other Protocols Monitoring SMTP 7. Define the error codes for Server, Authorization and Not Found categories. 8. Save or publish the configuration. Click Save to save your changes and continue with configuration. Chapter 6 SMTP Monitoring Click Save and Publish to immediately update the devices configuration. Creating a New Rule Rules are mandatory for any software service to be monitored. Here you will define a rule for an SMTP-based user-defined software service. Before You Begin It is assumed for this task that you are already familiar with the concept of software services and know how to create and edit software services and how to open the Rules window. For more information, see Defining or Editing User-Defined Software Services [p. 39] and Configuring Rules for User-Defined Software Services [p. 41]. To create a new rule for a software service based on the SMTP analyzer: 1. In the Rule Configuration dialog box, enter a rule name. In the Rule description edit box, type Rule 1 or you can type your own rule name. 2. Right-click the Services table and select Add from the context menu. The Service Details window will pop up. 3. In the Services window, type the server IP address, or type a range of IP addresses if you plan monitoring more than one server. We will create a simple rule to monitor one server (one IP address on the default port). In the Start IP address, type the IP address of the server you want to monitor. 4. Optional: Type a virtual IP address if your network uses a pool of virtual IP addresses. This rule will not use any virtual IP address. For more information, see Virtual IP address [p. 87]. 49

50 Chapter 6 SMTP Monitoring 5. Optional: Type the IP address of the load balancer if the monitored servers are behind a load balancer. This rule will not use any load balancer. A load balancer is likely to be installed in SMTP-heavy environments. 6. In the Port(s) fields, type the port number of the monitored service. By default, SMTP is served from port Optional: Select the box Client port(s) for reversed-direction protocols. This option makes sense only for protocols such as X-Window whose client-server meanings are reversed. If you are uncertain, leave this check box cleared. For this example, leave the box cleared. 8. Click OK to confirm your data and close the window. 9. Optional: Click the Options tab to review and modify common options and the ones specific to the SMTP analyzer. 50

51 CHAPTER 7 Microsoft Exchange Monitoring The main task for the AMD in Microsoft Exchange Server monitoring is to provide measurements of efficiency of the MS-RPC/ROEP network protocol. AMD also analyzes operations for which the users authenticate themselves in an Exchange server. The MS Exchange analyzer is used to detect and monitor Microsoft Exchange RPC-based operations and return metrics on user actions. In addition, the AMD gathers information on Microsoft Exchange operations and errors (a general distinction is made between server and protocol error types). To activate Exchange monitoring, create a dedicated user-defined software service. Configuring Exchange Monitoring You have to define a software service based on the Exchange analyzer to monitor Exchange traffic in your network. To create and configure a new software service based on the Exchange analyzer: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Software Services User-Defined Software Services. 7. In the Software Service dialog box, provide the service name, select TCP as the base protocol, and select the MS Exchagne analyzer. If your MS Exchange setup utilizes the remote procedure call (RPC) over HTTP or over HTTPS protocol, use the MS Exchange over HTTP or MS Exchange over HTTPS analyzer. 8. Click OK to add the new item to the list of user-defined software services. 9. In the Rules table, right-click to open the context menu and choose Add. For more information, see Configuring Rules for User-Defined Software Services [p. 41]. 51

52 Chapter 7 Microsoft Exchange Monitoring You have to create at least one rule for monitoring of this software service. After the rules are created, they all are displayed in the Rules table. 10. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. Monitoring Persistent TCP Sessions You can set up monitoring of persistent TCP sessions when creating a rule for the Microsoft Exchange dedicated user-defined software service. Persistent TCP sessions, that is TCP sessions for which the start was not recorded, also referred to as non-syn sessions, can be included in the TCP statistics, based on configuration properties you enable in RUM Console. The inclusion of such sessions may render the statistics somewhat inaccurate and must be undertaken with care. To set up monitoring of persistent TCP sessions for the Microsoft Exchange software service: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Software Services User-Defined Software Services. 7. In the Rules table, right-click a rule for which you want to modify the configuration and select Open from the context menu. The Rule Configuration pop-up window will appear. 8. Switch to the Options tab. 9. Select Enable Monitoring of Persistent TCP Sessions check box to monitor the persistent TCP sessions for Microsoft Exchange protocol. 10. Click OK to save the configuration. 11. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. Configuring Exchange User Recognition When monitoring the Microsoft Exchange protocol, you can have the AMD form the user name with a basic POSIX regular expression. To configure the user name recognition using the MS Exchange analyzer: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 52

53 Chapter 7 Microsoft Exchange Monitoring 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Configuration Global Front-End Monitoring MS Exchange to access the MS Exchange configuration settings for the device. The default regular expression to form a user name is \(.*\), which stands for any length string (all names are matched). In the case of multiple matches with the regular expression, only the first result will be displayed on CAS reports. Only the full username is guaranteed to be unique. Any partial results of the original user name are not guaranteed to be unique, so several users might be combined. See Regular Expression Fundamentals [p. 57] to learn more on using regular expressions in Data Center Real User Monitoring configuration tasks. NOTE If the Exchange user has turned on encryption in Outlook, all the user names will be presented as Default user. 7. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. Detecting Exchange Operations AMD can either detect every single operation or it can use its embedded intelligence to detect the sequence of operations as one. Some relatively simple Microsoft Exchange operations are in fact complex sequences of a large number of basic operations that may be irrelevant from the perspective of end user experience. You can let AMD detect and report complex sequences of operations as a single operation, for example OpenMessage. The following operation sequences are recognized by the MS Exchange analyzer: MAPI spooler Log on to mailbox Open folder Copy folder Create message Move folder Create a Strongly Typed Message Delete folder Open message Add an attachment Open embedded message Open attachment Copy message Delete attachment Move message Create appointment Delete message Move appointment Send a message Delete appointment Send a message to a remote recipient Resolved a recipient from an address book 53

54 Chapter 7 Microsoft Exchange Monitoring Find items in a folder that match search criteria Utilizing the Reporting Hierarchy within the RUM Console the default operations to tasks mapping can be overridden by importing a customized hierarchy defined within a CSV file. For more information, see Importing the Reporting Hierarchy in RUM Console in the Data Center Real User Monitoring Administration Guide. To enable automatic detection of operation sequences: 1. Start and log in to RUM Console. 2. Select the Entire Configuration perspective. 3. Click the Devices tab to display the current device list. 4. Right-click an AMD and select Open Configuration from the context menu. 5. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 6. Navigate to Configuration Global Front-End Monitoring MS Exchange to access the MS Exchange configuration settings for the device. 7. Select Detect operation sequences to detect the specific operation sequences as a single operation. If you wish to detect and report every single Microsoft Exchange operation, leave the check box unselected. 8. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. 54

55 CHAPTER 8 MS Exchange Tiers A tier is a specific point where DCRUM collects performance data. It is a logical application layer, a representation of a fragment of your monitored environment. There is one default tier on the CAS that report MS Exchange data: Exchange If you have defined software services based on MS Exchange analyzers, the Exchange tier will automatically be displayed on the Tiers report. This front-end tier shows measurements for MS Exchange traffic monitored by AMD. This includes the Outlook Anywhere (Remote Procedure Call over HTTP) Exchange feature. Properly configured AMD should classify and analyze such traffic as an Exchange type and have no impact on data in other reports such as Website reports. 55

56 Chapter 8 MS Exchange Tiers 56

57 APPENDIX A Regular Expression Fundamentals Regular expressions (regex) are logical formulas used for string pattern-matching in Data Center Real User Monitoring configuration tasks. The syntax of regular expressions is described. Regular expressions are used in a number of Data Center Real User Monitoring configuration tasks and therefore basic understanding of the concept is required before configuring certain features. This section attempts to explain the basic concept of regular expressions. For more exhaustive explanations, please refer to any of the numerous online or hard-copy publications available on the subject. NOTE Note that there are various flavors or standards of regular expressions. Data Center Real User Monitoring uses two of the standards: Basic POSIX and Extended POSIX. The basic difference between these is also explained below. The concept of regular expressions A regular expression is a logical formula enabling you to specify (match) a set of character strings and optionally extract sub-strings out of the found strings. It is usually used in the context of a larger set of character strings, out of which only certain ones fit (match) the specified regular expression or contain a substring that matches the expressions. Thus, for example, in a text file, a regular expression search enables you to find all the occurrences of a particular text pattern or all of the lines containing that pattern. Simple example of regular expression A single regular expression can match a wide range of very different text strings. For example, the expression. matches any single character and the expression.* matches any number of occurrences of any character, that is, in effect, it matches anything. Regular expressions can be used for finding particular text strings and then extracting certain parts of those text strings: the parts that match a sub-expression. The sub-expression is surrounded by round brackets ( ). For example, the expression a(b.), in the extended POSIX syntax, will find all strings composed of three characters, out of which the first one is a, the second one is b and the third one is any character. It will then extract the second and third character. 57

58 Appendix A Regular Expression Fundamentals Note, however, that the match has to be based on both the regular expression part outside the round brackets and also that inside the round brackets, that is the character a has to be in the found string, even though it is not extracted. Note that the same expression in the basic POSIX syntax would be written as a\(b.\), since the syntax requires special characters, such as round brackets, to be escaped using the backslash character. Most common regular expression symbols Some of the more common regular expression symbols: period. Matches any character. asterisk * Matches repetition of the previous character zero or more times. plus sign + Matches repetition of the previous character one or more times. Note: In basic regular expressions, it needs to be preceded by a backslash, to prevent it from being considered a normal character to match. caret symbol ^ This symbol can have a number of meanings, depending on the context: If it appears at the beginning of the expression, it means the beginning of the line or search string. If it appears as the first character in square brackets (see below), it means a negation. For example, [^@] means any character that is In other cases this character is considered a normal character and matches itself. the dollar sign $ Matches the end of the line or search string. square brackets [...] Group together symbols denoting a class of characters that is symbols that are to match a single character, for example, [a-z] stands for any lower case alphabetical character, [^@] means a character that is not symbol. hyphen - Is used to specify ranges of characters; see [ ] above. round brackets (...), escaped with backslashes in the basic syntax: \(...\) Select that part of the parsed string which we want to extract. Note: In basic regular expressions, round brackets need to be preceded by backslashes, to prevent them from being considered normal characters to match. Basic vs extended POSIX syntax In basic regular expressions, special characters also referred to as meta-characters, such as `?', `+', `{', ` ', `(', and `)', lose their special meaning. To achieve equivalent functionality, you need to precede (escape) these characters using the backslash character: `\?', `\+', `\{', `\ ', `\(', and `\)'. 58

59 Appendix A Regular Expression Fundamentals Walk-through example of a Basic POSIX regular expression For example, if an HTTP cookie name is defined as Pag and the cookie header line is as follows: Cookie: Pag=cf68603b@TXP293@ @D1R1wLLsMrjhw; the cookie value is: cf68603b@txp293@ @d1r1wllsmrjhw Assuming that the actual substring that we want to extract is positioned between the first and character in the cookie value string, it is: TXP293 The Basic POSIX regular expression can then be defined as: ^[^@]*@\([^@]\+\)@ In this particular case, the above regular expression can be understood as follows: ^ means find the beginning of the line. [^@]* means skip zero or more occurrences of any character that is \( means the string to extract is described by that part of the expression that is contained within round brackets. [^@] means that the first character we found above) must not \+ means that we want to extract this character and any other characters that follow it and that are also \) marks the end of the expression describing the string we want to means that the string to be extracted has to be terminated but we do not include the character in the extracted string because it is outside of the round brackets. Testing Regular Expressions Regular Expressions Test is a tool you can use to verify basic or extended regular expression output according to the rules that the AMD will apply to the data it processes. The tool is available for monitored URL configuration and several other configuration parameters. Before You Begin It is highly recommended that you make yourself familiar with Basic POSIX and Extended POSIX regular expression standards and with the rules for string pattern-matching in Data Center Real User Monitoring. Remember that when using a regular expression to specify a set of URLs to monitor, you must follow rules to enable the AMD to properly process those strings. Although this is not necessary for testing purposes, you are encouraged to learn these rules as described in Configuring URL Monitoring in the Data Center Real User Monitoring SAP Application Monitoring User Guide and to test realistic examples, not just portions of data. 59

60 Appendix A Regular Expression Fundamentals To access Regular Expressions Test, go to the RUM Console main menu and choose Tools AMD Regular Expression Test, or right-click an AMD name on the main Devices console tab. A Test button is also available for settings that are regular expression patterns; click Test to start the tool. You can use this tool to test regular expressions out of context and choose whether you want to apply extended or basic rules. Note that when you test a configuration environment, the extended or basic rules will already be selected depending on the AMD analyzer structure. Using Regular Expressions Test, you can check match patterns on the fly on any string of your choice. The window consists of the following fields: Regular Expression Type Optionally, this field is available when you invoke Regular Expressions Test for no specific configuration entry. Choose Basic or Extended, depending on the syntax type you intend to use. Pattern Use this field to enter the regular expression. Apply all necessary rules (such as grouping). When building a regular expression, click the? button to list metacharacters that you can use. Text to process Use this field to enter a portion of text that the regular expression processor will parse and search for excerpts matching the rules defined in the Pattern field. Click the icon above the text field to wrap long lines. Note that the test will be performed on a single line; if you paste a block of text that contains line ends, the test results will not be meaningful. Groups If you created one or more subexpressions in the Pattern, the tool will automatically show numbers for each bracket pair. Click the group number in this field to highlight the matched fragment of the processed text. AMD output This field presents the transformed text after regex rules were applied. If the pattern does not match any portion of the processed string, the Nothing matched message is displayed. To test regular expression matching: 1. Enter a pattern - a regular expression. When you click the Test button next to the field where you enter a regular expression, the testing utility opens with the Pattern field filled with the string you entered when creating the configuration. You can edit the Pattern field and verify the output immediately. This can be done safely because none of the values is transferred to configuration file until you confirm the changes. An example of a pattern: ( a. Optional: Verify grouping by clicking numbers in the Groups field. Each group will be highlighted after you click its number. 2. Enter text to process, such as a URL in the Text to process field. 60

61 Appendix A Regular Expression Fundamentals An example URL to which the matching rules will be applied: 3. Click Match to verify the output. The matched string will be displayed in the AMD output field. In this example, this will be: If the regular expression rule does not match any part of the processed text, the Nothing matched message is displayed. 4. Click Save to transfer the expression to the configuration field. Best Practices for Regular Expressions Use of regular expressions significantly increases the computing load on the AMD and may significantly affect performance, so unnecessary use of regular expressions or use of overly complicated regular expressions should be avoided. In particular, the following points should be considered when designing a regular expression: Do not use the match anything expression (a period followed by an asterisk.* ) unless necessary. This is sometimes used to skip a portion of an irrelevant pattern. When possible, use an alternative construction. For example, assume you have a URL of the form: To remove the session ID, you could write: ( But you would significantly reduce the processing load if you instead wrote: ( Use Unicode notation not only for URLs but for the entire HTTP header. For example, do not use spaces, but use %20 instead. This ensures that if a number of processing layers are involved, the special characters will be preserved intact through all interprocess conversions. Do not assume that the input stream ends with the end-of-line character. Take care of the end of line explicitly by closing patterns with %0d%0a. Example 1. An example of a well-designed regular expression Assume we have the following string contained in the HTTP header: ID=34fffff; REMOTE _ADDR: content type To extract the address, use the regular expression: %0d%0aREMOTE_ADDR:%20\([^%0d%0a]*\)%0d%0a 61

62 Appendix A Regular Expression Fundamentals 62

63 APPENDIX B Diagnostics and Troubleshooting Report-related Issues Central Analysis Server automatically detects a range of exceptions (anomalies) and notifies report users. Exception notifications are displayed as yellow (warning) or red (error) triangle icons placed in the upper-left corner of the report window. To see the notification message, place the mouse pointer over the triangle icon. Slow Page Load Sequence report is empty for operation which is part of an XML transaction. Why and how do I fix this? For XML and SOAP, Page Elements data is identical to Page Analysis data, so, to avoid unnecessarily keeping the duplicates in the database, a VDATA_FILTER_XMLSOAP filter is set to true by default. Keeping this filter set to true saves disk space but, because the XML and SOAP entries are filtered out, it makes reporting on the Page Elements level (elements or headers) impossible. To change the value of VDATA_FILTER_XMLSOAP property in userpropertiesadmin, type in the Web browser's Address bar and press [Enter], change the filter's property value, and click Set value to accept the change. To access this screen, you need to have administrative privileges for the report server. Values of typical metrics in historical reports are altered by adding a specific metric to such a report. Why and how can I avoid this? This deviation can be observed in reports based on monitoring reporting groups, transactions, applications (business units) and analyzing traffic monitored per Tiers. Metric value inconsistency occurs as a result of differently aggregated data used to calculate the same metric. This affects report with resolution set to 1 day or more. For example, a seven day report created in DMI containing Operations metric will base that metric value on data that is calculated and stored daily for the report's Time range. When HTTP errors metric is added to that report, the Operations metric will based on data that is calculated from the transactions analysis for the report's Time range. 63

64 Appendix B Diagnostics and Troubleshooting Example 2. Historical Comparison Reports The following sample report can be used to visualize the inconsistency in historical reports. Generate a simple report displaying the Reporting Group and Operations metrics: 1. Log in to CAS 2. Open Data Mining Interface by clicking Reports DMI Define Report 3. Select the Time range value to Last 7 days. 4. Make sure that Central Analysis Server Data Views is set to Monitored traffic. 5. Set Dimensions to Reporting group. 6. Set Filter for the reporting group that you would like to examine. 7. Add Operations metric to the list of metrics displayed on the report. 8. Click Display report button to view the report. To view the difference between the Operations metric, modify the historical report just created by adding the HTTP errors metric. 1. On the historical report, click the Refine button. 2. Duplicate the existing section by clicking the icon. 3. Add the HTTP errors metric in the newly duplicated section. 4. Click Display report to view both tables on the same report. The Operations metric value in such historical reports will differ since the first report is based on more general data source while the second report is based on more granulated data source. Since the list of Operations can be affected by URL aging, it is possible that the transactions that occurred in earlier days, are not detected in current traffic and thus, are not aggregated. This will cause the Operations metric value to be lower then in the first report. You can synchronize the reports and avoid this inconsistency by modifying the VAS_URL_AGING_CHECK_HIERARCHY property in Advanced Properties Editor. 64

65 Appendix B Diagnostics and Troubleshooting IMPORTANT Please be aware that, setting the following properties to ON may cause the report server performance to decrease if a large volume of monitored URLs are matching the defined URL mask. In order to synchronize the historical reports for future traffic monitoring, execute the following procedure: 1. As user with administrative rights, access Advanced Properties Editor in Control Panel. Type in your browser's Address field, and click Advanced Properties Editor from the Configuration Management section. 2. Set URL aging check mask property. Scroll down to URL Aging section of the console and set the VAS_URL_AGING_CHECK_MASK property to ON NOTE This setting change affects data aggregation for current and future traffic analysis. It will not synchronize past historical reports that are based on already aggregated data. 3. Make sure that the VAS_URL_AGING_CHECK_HIERARCHY is set to ON. This is a default setting however, your current configuration may have required this setting to be modified. 4. Verify that synchronization is active. Since this synchronization relates to historical reports, the synchronized reports will be available for traffic analyzed after this setting change. Once a desired number of days has passed, create a historical altered report as suggested in Example 2. Historical Comparison Reports [p. 64] example and compare the Operations metric. If the Operations metric value is the same in both tables, your historical reports are being synchronized. The following is a list of metrics that will alter the metrics in reports based on historical data: % of bad delay calls Operation attributes (1) % of bad jitter calls Operation attributes (2) % of bad lost packets calls Operation attributes (3) % of bad MOS calls Operation attributes (4) % of bad R-factor calls Operation attributes (5) Aborted page http server time Operation time - 2 stdv Aborted page image server time Operation time - stdv Aborted page redirect time Operation time + 2 stdv Aborted page request size Operation time + stdv 65

66 Appendix B Diagnostics and Troubleshooting Aborted page request time Operation time breakdown Aborted page server delay Operation time percentage breakdown Aborted page size in bytes Operation time with breakdown Aborted page SSL setup time Operations (with aborts) breakdown Aborted page TCP connect time Aborted page transfer time Aborts Attachments per Bad delay calls Bad jitter calls Bad lost packets calls Bad MOS calls Bad R-factor calls Calls Client ACK RTT + 2 stdv Client ACK RTT - 2 stdv Client ACK RTT - stdv Client ACK RTT + stdv Client loss rate - 2 stdv Client loss rate - stdv Client loss rate + 2 stdv Client loss rate + stdv Client RTT - 2 stdv Client RTT - stdv Client RTT + 2 stdv Client RTT + stdv Closed TCP connections Custom metric (1) (avg) Custom metric (1) (cnt) Custom metric (1) (sum) Custom metric (2) (avg) Custom metric (2) (cnt) Custom metric (2) (sum) Custom metric (3) (avg) Custom metric (3) (cnt) Custom metric (3) (sum) Custom metric (4) (avg) Operations (with aborts) percentage breakdown Operations with large server time Orphaned redirects Other SSL errors Page breakdown Page load time breakdown Page load time percentage breakdown Page percentage breakdown Page requests breakdown Page requests percentage breakdown Pages stopped Pages with large server time Pages with large server time per minute Pages with small server time Percentage of aborts Percentage of long aborts Percentage of optimized traffic (bytes) Percentage of optimized traffic (packets) Percentage of short aborts Percentage of stopped pages Percentage of TCP sessions w/errors Query time breakdown Query time percentage breakdown Redirect time Requests breakdown RPC Client error RPC Protocol error RPC Security error RPC Server error SAP GUI errors Server loss rate - 2 stdv Server loss rate - stdv 66

67 Appendix B Diagnostics and Troubleshooting Custom metric (4) (cnt) Custom metric (4) (sum) Server loss rate + 2 stdv Server loss rate + stdv Custom metric (5) (avg) Server operation size - 2 stdv Custom metric (5) (cnt) Server operation size - stdv Custom metric (5) (sum) Server operation size + 2 stdv DNS errors Server operation size + stdv DNS format errors Server realized bandwidth - 2 stdv DNS name errors Server realized bandwidth - stdv DNS not implemented errors Server realized bandwidth + 2 stdv DNS other errors Server realized bandwidth + stdv DNS refused errors Server RTT - 2 stdv DNS server failure errors Server RTT - stdv DNS timeouts Server RTT + 2 stdv transfer time breakdown Server RTT + stdv transfer time percentage Server think time breakdown Server time - 2 stdv Endpoints B Server time - stdv End-to-end RTT + stdv Server time + 2 stdv Errors Server time + stdv Errors Server time breakdown Errors Server time percentage breakdown Errors Short aborts Failed TCP connection attempts Short stopped pages Fast pages Slow pages FIX Client Messages Slow pages per minute FIX Client Resend Request Slow user sessions FIX Client Test Requests SMTP Command syntax error FIX Logons SMTP Errors FIX Logouts SMTP General system error FIX Not Parsed Client Bytes SMTP Mailbox not available FIX Not Parsed Server Bytes SSL conn. setup per operation FIX Server Messages SSL conn. setup per session FIX Server Resend Request SSL error 1 FIX Server Test Requests SSL error 2 Form submission time breakdown SSL errors Form submission time percentage SSL handshakes breakdown Stopped pages breakdown Hits (succeeded) 67

68 Appendix B Diagnostics and Troubleshooting HTTP availability HTTP client errors (4xx) Stopped pages percentage breakdown Successful TCP connection attempts HTTP client errors (4xx) percentage TCP + MS Exchange errors HTTP client errors (category 3) TCP connections attempts HTTP client errors (category 3) [%] Time to abort HTTP errors Total bytes compression HTTP not found errors (404) Total bytes on LAN side HTTP not found errors (404) [%] Total bytes on WAN side HTTP not found errors (404) [%] Total packets compression HTTP other client errors (4xx) Total packets on LAN side HTTP other client errors (4xx) percentage Total packets on WAN side HTTP other server errors (5xx) Transact. errors HTTP other server errors (5xx) Transact. rollbacks percentage Transact. rollbacks after timeout HTTP redirect time Transact. rollbacks before timeout HTTP response time Transact. service authentication errors HTTP server errors (5xx) Transact. service not found errors HTTP server errors (5xx) percentage Transaction (or network) throughput HTTP server errors (category 1) Transactional service errors HTTP server errors (category 1) [%] Unique client groups HTTP server errors (category 2) Unique page URLs HTTP server errors (category 2) [%] Unique servers HTTP server image time Unique services HTTP server time Unique sites HTTP unauthorized errors (401,407) User sessions LAN-WAN byte ratio VoIP delay LAN-WAN packet ratio VoIP delay for client-to-server traffic Long aborts VoIP delay for server-to-client traffic Long stopped pages VoIP Jitter Mail attachments VoIP Jitter for client-to-server traffic Mail server welcome msg. time VoIP Jitter for server-to-client traffic MQ appl. errors VoIP loss rate MQ appl. errors (1) VoIP loss rate for client-to-server traffic MQ appl. errors (2) VoIP loss rate for server-to-client traffic MQ appl. errors (3) VoIP MOS MQ appl. errors (4) VoIP MOS for client-to-server traffic MQ appl. errors (5) VoIP MOS for server-to-client traffic 68

69 Appendix B Diagnostics and Troubleshooting MQ client errors VoIP R-factor MQ Errors VoIP R-factor for client-to-server traffic MQ protocol errors MQ security errors VoIP RTCP Jitter MQ server errors VoIP R-factor for server-to-client traffic VoIP RTCP Jitter for client-to-server traffic MS Exchange errors VoIP RTCP Jitter for server-to-client traffic Number of hits in an aborted transaction Operation attributes Web errors The yellow triangle displays AMDs produce no performance data. What do I do? The message AMDs produce no performance data means that AMDs connected to the report server do not produce any new data. To resolve this issue, you have to investigate the configuration of the AMDs and determine why they do not produce the performance data. The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? The report server has a built-in protection against simple configuration mistakes. One of the problems of that kind is data incorrectly time stamped by AMD. It happens if the AMD runs with the system clock is incorrectly set and is not being synchronized with the report server. If you see such a notification, you have to check the system time on the report server and on the AMD. You have to ensure the time synchronization option is turned on. To check the time synchronization: 1. Launch the RUM Console. 2. Select the AMD, right-click it and choose Open Configuration. The AMD Configuration window will open up. 3. Navigate to Global General. Check the IP address of the server authorized to set the AMD time. It should be the same as the report server IP address. 4. Check the report server time setting. You can do it by reading the time that is displayed at the bottom of the reports. Ensure the report server has the time zone set correctly. 69

70 Appendix B Diagnostics and Troubleshooting Figure 6. An example of the report time stamp The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? Once a day the report server has to perform a database maintenance and memory cleanup. For that time, the data processing has to be suspended and you will see delayed data on reports. The daily maintenance is usually performed as the first task after midnight and it takes up to half an hour in installations with a large database. It is normal and expected to see this warning just after midnight. But if you see the message during the day, it can be an symptom of incorrect system configuration (check time settings on the server) or of system overload. The yellow triangle displays No contact with the primary AMD. What do I do? This message means that the report server lost contact with at least one primary AMD. If an AMD is marked as primary and the report server cannot get contact with this AMD, even if the performance data can be downloaded from the other AMDs, the system will wait until the communication with the primary AMD is restored. The yellow triangle displays No contact with any of the AMDs. What do I do? This message means that the communication link cannot be established with any of the attached AMDs. Check the network settings on the report server or the configuration of AMDs. The yellow triangle displays Delay in data processing. What do I do? If the last processed data is significantly behind the current time due to slow data processing or idle periods that occurred in the past, the report server displays the triangle icon with the message Delay in data processing. If the server had a delay, but now it is catching up, this message will not appear anymore. To confirm that delay is decreasing, you have to inspect server.log and search for messages similar to this: T REC :10: zdata_43f47e58_5_t is being processed. Sample begin ts = :25. Sample delay 17 min. If the delay becomes smaller, it means the server is catching up. If the delay values are growing, it can indicate the system overload. The yellow triangle displays The AMD has not yet generated performance data. What do I do? This message indicates that some data files have already been generated on some AMDs, but not on the others. This may not be an indication of a problem and, when you refresh the reports after 30 to 60 seconds, this message may disappear. All you can do is to verify the time synchronization among all the AMD s. See The yellow triangle displays Delay in data processing. What do I do? [p. 70]. 70

71 Appendix B Diagnostics and Troubleshooting The yellow triangle displays Data processing is being performed in the debug mode. What do I do? Data processing can be manually suspended and controlled by so-called debug mode, which can be enabled using Control Panel. Open Control Panel by typing: in the Address bar of your Web browser and clicking Go, then select Controlled data processing from the Configuration Management section. The red triangle displays Data loading is in progress. Reports may be incomplete. What do I do? This message indicates that the report server is currently starting up. Because of this the information presented on reports may be incomplete. Depending on the database size, the startup process may take up to several minutes. If the server restart was not done manually or was not planned, please inspect server.log or contact Customer Support. The red triangle displays Low memory. The real-time cache will only be updated. What do I do? This message indicates that the report server has no free memory to process new entities such as software services, servers, and URLs. This message will be cleared when some resources are freed, what usually happens at midnight during the scheduled database maintenance (see The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? [p. 70]). All the metric values presented on reports (except user/client counters) will show correct values. The only problem is that predefined tabular reports may not show all the entities they are intended to show. All the charts and DMI reports show correct data. The mechanism of updating the real-time cache, as described above, is a protection that allows the report server to continue the operation instead of closing down due to lack of memory resources. The red triangle displays The number of servers has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored servers. If the number of observed servers reaches a defined limit, the report server will not accept any new servers and will drop the collected data for those servers. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red triangle displays The number of clients has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored clients. If the number of registered clients (which also includes aggregated virtual clients such as Client from... ) reaches a defined limit, the report server will not accept any new clients and will drop the collected data for those clients. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. 71

72 Appendix B Diagnostics and Troubleshooting The red triangle displays The number of sites has reached the defined limit. What do I do? The report server has a built-in limit of the number of automatically created sites. If the number of observed automatic sites reaches a defined limit, the report server will not create any new automatic sites and such traffic will be allocated to All Other. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The Sites report for a selected application is empty. Why? If the Sites report for a selected application is filtered for a client tier, such as Synthetic or RUM sequence transactions, it will not show any data. To see statistics for sites, drill down from the Applications report as follows: 1. Click the application name on the Applications report. 2. Click the client tier name on the Tiers report for a selected application. For the Synthetic tier, you will see the Overview Application Status report; for the RUM sequence transactions tier, the Sequence Transactions Log report; and for the Cerner RTMS tier, the Cerner RTMS Log report. 3. Depending on the type of report, click the Overview Site Status or the Sites tab. I see gaps on the chart reports. Why are the charts incomplete? Gaps in reports mean that the report server missed some data and was not able to get it into the database on time. Your reports may resemble the example below. Figure 7. Gaps in a Central Analysis Server graphical report Figure 8. Gaps in an Advanced Diagnostics Server DMI-based graphical report There are several reasons why the graphical reports may have incomplete data: The AMD was not able to detect any traffic from the monitored network, thus it was not able to produce any valid data for the report server. To confirm that this was the 72

73 Appendix B Diagnostics and Troubleshooting reason, you have to connect to the AMD using an SSH client and check whether the files named zdata_xxxxx_x_x are located in the /var/spool/adlex/rtm directory. Similar symptoms can be observed if the AMD has been down for some time and data files were not produced for that time. If data files are present and the viewed chart displays only a fragment of the monitored traffic, for example, for a specific server or site, it may mean that a part of traffic, which was intended to be monitored, is missing. In such a situation, the data files are much smaller than usually for the corresponding period of the day. Similar situations, that is, gaps only on some reports, may occur in a multi-amd installation when some AMD s were down or disconnected from the network. In the case when only one AMD is connected to the report server, communication problems do not cause data gaps. If the report server cannot communicate with the AMD, it will wait until the communication is restored and then will process all the data from the past. When there are multiple AMD s connected to the report server and there is a break in communication with only some of them, the report server processes the data from the available AMDs, so in this case, gaps can appear on some reports. If it is a critical issue and your network (or its parts) need to be monitored continually and you cannot miss the data from some AMDs, you have to mark the AMDs as primary. In such a case, the report server will wait until the communication with primary AMDs is restored, even if other AMDs are available. Gaps in charts on some reports in multi-amd installations may be caused by unsynchronized AMDs. The reason for that may be that if the report server sees a data file for a specific time period on one of the AMDs, it will wait only 30 seconds for data files covering the same period of time from other AMDs. These 30 seconds is a server's tolerance for time synchronization issues. To verify that such a situation took place, compare the clock readings from AMDs and then check the time synchronization settings (see The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? [p. 69]). It may happen that a part of data will be missing. This will result in a significant decrease of the aggregated data, used to render the chart bars, as shown below: Figure 9. An example of partial lack of the performance data 73

74 Appendix B Diagnostics and Troubleshooting Note that such an effect relates to metrics that are calculated as sums, for example, number of page loads, number of errors, number of users, or bandwidth utilization. Charts showing the averages (RTT, loss rate, page load time) will not be affected. I see gaps on the long-term data chart reports. Why are the charts incomplete? The report server aggregates the data collected during the day into daily (and monthly) rollups. This is a scheduled process. If this process is not triggered, you will see gaps in the daily rollups as in the following example: Figure 10. An example of gaps in long-term graphical reports The most frequent reasons for missing rollups are: The report server was down in the night; report data generation starts at 12:10 AM local time and if the report server was down at that time, no aggregate data for long-term reports will be generated. The report server was overloaded and it took too much time for other crucial tasks; report data generation for long-term reports was canceled. You can always re-generate data for long-term reports. Open Control Panel by typing: in the Address bar of your Web browser and clicking Go, then select Regenerate Reports from the System Management section. Application performance and availability data is missing from the tabular reports. How can I fix this? The missing data manifests itself as shown below. Figure 11. Missing application performance and availability data 74

75 Appendix B Diagnostics and Troubleshooting The most frequent reason for such a situation is incorrect setting of business hours and holidays. Inspect the business hours and holiday settings by selecting the menu item Settings Report Settings Business Hours. The following configuration screen shows the current settings. Figure 12. The Business Hours Configuration screen If you want to collect performance data seven days per week, including non-business days and holidays, clear the Holidays check box and select the check boxes for weekend days. In addition, you may want to collect performance data in 24/7 mode, but be aware that this results in a higher database growth rate and a larger database. If you want to enable collecting data all the time, open the Control Panel by opening the following page: In the Control Panel, click Advanced Properties Editor from the Configuration Management section. Find ONLY_BUSS_HOUR_REPORTING and set it to OFF. To see whether your holiday definition is correct, click View Holidays. 75

76 Appendix B Diagnostics and Troubleshooting Figure 13. The Defined Holidays screen The list of holidays is hard-coded, and the default set is for the USA. To select a set, click the Choose holiday definition list. To see the content of the selected set, click Preview. To store the newly selected set, click Save. Why are SQL queries in reports are truncated even though full query logging is set on the AMD? By default, only the first 1024 bytes of a query are logged. This should be sufficient in most cases to log full queries. However, if you deploy queries that are longer than 1024 characters, you need to change the sql.query.length parameter in the rtm.config file. To edit this file: 1. Log in to the AMD as user root. 2. Navigate to /usr/adlex/config. 3. Open the file rtm.config in a text editor. 4. Append a new line with the query length property and its value. For example: 76