Data Center Real User Monitoring

Transcription

1 Data Center Real User Monitoring Cerner Application Monitoring User Guide Release 12.3

2 Please direct questions about Data Center Real User Monitoring or comments on this document to: Customer Support Copyright 2014 Compuware Corporation. All rights reserved. Unpublished rights reserved under the Copyright Laws of the United States. U.S. GOVERNMENT RIGHTS-Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Compuware Corporation license agreement and as provided in DFARS (a) and (a) (1995), DFARS (c)(1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. Compuware Corporation. This product contains confidential information and trade secrets of Compuware Corporation. Disclosure is prohibited without the prior express written permission of Compuware Corporation. Use of this product is subject to the terms and conditions of the user's License Agreement with Compuware Corporation. Documentation may only be reproduced by Licensee for internal use. The content of this document may not be altered, modified or changed without the express written consent of Compuware Corporation. Compuware Corporation may change the content specified herein at any time, with or without notice. All current Compuware Corporation product documentation can be found at Compuware, FrontLine, Network Monitoring, Enterprise Synthetic, Server Monitoring, Dynatrace Network Analyzer, Dynatrace, VantageView, Dynatrace, Real-User Monitoring First Mile, and Dynatrace Performance Network are trademarks or registered trademarks of Compuware Corporation. Cisco is a trademark or registered trademark of Cisco Systems, Inc. Internet Explorer, Outlook, SQL Server, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation. Firefox is a trademark or registered trademark of Mozilla Foundation. Red Hat and Red Hat Enterprise Linux are trademarks or registered trademarks of Red Hat, Inc. J2EE, Java, and JRE are trademarks or registered trademarks of Oracle Corporation. VMware is a trademark or registered trademark of VMware, Inc. SAP and SAP R/3 are trademarks or registered trademarks of SAP AG. Adobe Reader is a registered trademark of Adobe Systems Incorporated in the United States and/or other countries. All other company and product names are trademarks or registered trademarks of their respective owners. Local Build: December 8, 2014, 14:24

3 Contents Contents Introduction Who Should Read This Guide Organization of the Guide Related Publications Customer Support Information Reporting a Problem Documentation Conventions Chapter 1 Cerner Application Monitoring Overview Configuration Process Overview of Cerner Monitoring Chapter 2 Adding Basic DC RUM Devices Adding an AMD to the Devices List Adding a CAS to Devices List Adding ADS to Devices List Adding Business Service Management to RUM Console Chapter 3 Verification of Traffic Monitoring Quality Sniffing Point Diagnostics Sniffing Point Diagnostics Reports Network Interface General Statistics Network and Transport Protocol Information Services Detected in the Traffic Session-Related Statistics SSL Diagnostics Application Overview Using RUM Console to Identify Problems Related to Network Hardware Operation... Chapter 4 Basic Monitoring Configuration AMD General Configuration Settings Configuring Operation-Related Global Settings Chapter 5 Configuring AMD to Monitor User-Defined Software Services Creating Software Services Based on Cerner Analyzers

4 Contents Defining Oracle Forms Software Services Configuring Rules for User-Defined Software Services Example of Configuring Cerner Software Services Creating a New Rule for Cerner Software Service Managing User-Defined Software Services Chapter 6 Monitoring Sequence Transactions Adding Cerner Transactions Modifying, Deleting, and Cloning Transactions for a Single AMD Filters and Transaction Inspector for Cerner Transactions Chapter 7 Cerner Tiers Chapter 8 Cerner Traffic on CAS Reports Appendix A Diagnostics and Troubleshooting Report-Related Issues Appendix B Regular Expression Fundamentals Testing Regular Expressions Best Practices for Regular Expressions Appendix C Graphical Explanation of Network Performance Metrics Glossary Index

5 INTRODUCTION Who Should Read This Guide This manual is intended for users of Data Center Real User Monitoring who want to monitor the Cerner Millennium environment. Organization of the Guide This guide is organized as follows: Cerner Application Monitoring Overview [p. 9] - Describes capabilities of Cerner analyzers and provides an overview of the configuration process. Adding Basic DC RUM Devices [p. 15] - Describes how to add and configure the data sources and report servers using the RUM Console. Verification of Traffic Monitoring Quality [p. 21] - Describes how to verify sniffing points traffic detection quality before the actual monitoring begins. Basic Monitoring Configuration [p. 33] - Describes AMD general settings. Configuring AMD to Monitor User-Defined Software Services [p. 41] - Describes the creation and management of Cerner-based software services. Monitoring Sequence Transactions [p. 47] - Describes how to monitor Cerner transactions. Cerner Tiers [p. 53] - Describes tiers that present Cerner data. Cerner Traffic on CAS Reports [p. 55] - Lists CAS reports that display statistics for Cerner data. Diagnostics and Troubleshooting [p. 57] - Lists common CAS support issues in the form of questions and answers. Regular Expression Fundamentals [p. 67] - Describes how to use regular expressions in CAS. Graphical Explanation of Network Performance Metrics [p. 73] - Uses charts to illustrate network performance metrics. 5

6 Introduction Related Publications Documentation for your product is distributed on the product media. For Data Center RUM, it is located in the \Documentation directory. It can also be accessed from the Media Browser. Go online ( for fast access to information about your Dynatrace products. You can download documentation and FAQs as well as browse, ask questions and get answers on user forums (requires subscription). The first time you access FrontLine, you are required to register and obtain a password. Registration is free. PDF files can be viewed with Adobe Reader version 7 or later. If you do not have the Reader application installed, you can download the setup file from the Adobe Web site at Customer Support Information Dynatrace Community For product information, go to and click Support. You can review frequently asked questions, access the training resources in the APM University, and post a question or comment to the product forums. You must register and log in to access the Community. Corporate Website To access the corporate website, go to The Dynatrace site provides a variety of product and support information. Reporting a Problem Use these guidelines when contacting APM Customer Support. When submitting a problem, log on to the Dynatrace Support Portal at click the Open Ticket button and select Data Center Real User Monitoring from the Product list. Refer to the DC RUM FAQ article at to learn know how to provide accurate diagnostics data for your DC RUM components. Most of the required data can be retrieved using RUM Console. Documentation Conventions The following font conventions are used throughout documentation: This font Bold Citation Indicates Terms, commands, and references to names of screen controls and user interface elements. Emphasized text, inline citations, titles of external books or articles. 6

7 Introduction This font Documentation Conventions [p. 6] Fixed width Fixed width bold Fixed width italic Menu Item Screen Code block Indicates Links to Internet resources and linked references to titles in documentation. Cited contents of text files, inline examples of code, command line inputs or system outputs. Also file and path names. User input in console commands. Place holders for values of strings, for example as in the command: cd directory_name Menu items. Text screen shots. Blocks of code or fragments of text files. 7

8 Introduction 8

9 CHAPTER 1 Cerner Application Monitoring Overview Support for Cerner Millennium environment enables monitoring of the performance of Millennium operations seen on the network connection between Citrix Presentation/XenApp Servers and Cerner Nodes (physical application servers). The time tracking functionality integrated into Cerner Millennium enables the collection of real-time data from actual users of the system. Cerner monitoring provides information about the following: Usage of the Cerner applications Cerner service quality delivered to the end user Time consumed by slow performing Cerner applications Performance and availability delivered to key locations Correlation between Cerner RTMS and Cerner Decode Records To provide the maximum information about all running instances of a Cerner application on a Citrix server, data from RTMS and AMD are matched using two dimensions: Process ID and Citrix server IP address. AMD monitors sessions between the clients and the Citrix server (front-end) and between the Citrix server and the Cerner server (back-end). However, network traffic is not enough to match front-end and back-end sessions. To overcome this limitation, a special agent is installed on the Citrix server, called TCAM, which monitors connections established to and from the Citrix server and determines the Citrix user name of each session initiated on the Citrix server. The TCAM agent sends this information to AMD using several kinds of messages. Since a typical user can easily simultaneously start two instances of a given application (for example, Powerchart) with the same user name, correlation cannot be done using either the user name or the application name. This information needs to be extended with the process ID of the Cerner application running on the Citrix server. The process ID is received within the RTMS data and is unique for a given machine. This matching also allows for reporting real Cerner user names. 9

10 Chapter 1 Cerner Application Monitoring Overview Figure 1. Correlation between Cerner RTMS data and AMD data Citrix Presentation Server Front-end traffic (ICA protocol) Back-end traffic Users Process ID, client IP address Cerner application server Process ID, Citrix server IP address, user name, user IP address AMD BSM adaptor Record correlation CAS / ADS Configuration Process Overview of Cerner Monitoring Data Center Real User Monitoring (DC RUM) provides you with several ways of monitoring Cerner traffic. You can monitor software services, transactions, tiers, applications, and network. Before You Begin Before you start the configuration process: You should be familiar with DC RUM components and basic monitoring concepts. Refer to the Data Center Real User Monitoring Getting Started. You need to identify your monitoring goals. For more information, see Define and Prioritize Goals, Objectives, and Requirements in the Data Center Real User Monitoring Getting Started. You need to install the following DC RUM components: The latest version of AMD Refer to the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. The latest version of RUM Console Refer to the Data Center Real User Monitoring RUM Console Installation Guide. The latest version of CAS Refer to the Data Center Real User Monitoring Central Analysis Server Installation Guide. The latest version of ADS Refer to the Data Center Real User Monitoring Advanced Diagnostics Server Installation Guide. 10

11 Chapter 1 Cerner Application Monitoring Overview Make sure that default ports are available for communications between the individual DC RUM components. For more information, see Network Ports Opened for DC RUM in the Data Center Real User Monitoring Administration Guide. The following steps must be executed in order to begin monitoring the traffic using the DC RUM suite: Configuring Devices 1. Add Agentless Monitoring Device (AMD) AMD is the main data source (Data Collector) for DC RUM; it collects and presents the monitored data to DC RUM report servers for analysis and reporting. You need to add at least one AMD to the list of devices in RUM Console. For more information, see Adding an AMD to the Devices List [p. 15]. 2. Add Central Analysis Server (CAS) CAS is the main report server for DC RUM. It uses data provided by the AMD and its monitoring and alerting mechanisms to identify, track, and report on issues affecting the security, performance, and reliability of your services. Add at least one CAS to the device list and configure its connection with the AMD. Adding a report server to a list of devices is similar to adding the AMD. For more information, see Adding a CAS to Devices List [p. 17]. 3. Adding Advanced Diagnostics Server (ADS) ADS is a report server that performs detailed analysis of key transactional application protocols. It uses data provided by the AMD. Add at least one ADS to the device list and configure its connection with the AMD. For more information, see Adding ADS to Devices List [p. 18]. 4. Adding and configuring the Business Service Management using the RUM Console Establish connectivity between CAS and ADS and the Cerner RTMS Adaptor located on the Business Service Management server. For more information, see Adding Business Service Management to RUM Console [p. 19]. 5. Verify the traffic monitoring quality and completeness You can verify traffic quality and completeness before the actual monitoring begins. Sniffing point diagnostics allows you to perform pre-monitoring tasks without the need of accessing the AMD console and executing a series of Linux commands which usually serve the purpose of validating AMD physical installation and connection. For more information, see Verification of Traffic Monitoring Quality [p. 21]. Configuring Basic Monitoring 6. Configure general settings for your AMD Before you proceed to detailed monitoring rules, you need to define the global settings that are applied to all software services monitored by a given AMD. These global settings include a monitoring interval and thresholds for the basic metrics. These settings can be overridden at a later time with more specific monitoring rules that you can define. 11

12 Chapter 1 Cerner Application Monitoring Overview For more information, see Configuring General Data Collector Settings in the Data Center Real User Monitoring Web Application Monitoring User Guide and Configuring Operation-Related Global Settings [p. 37]. 7. Configure support for WAN optimization All AMDs that monitor network traffic using WAN optimization must be configured before DC RUM can automatically recognize the optimized WAN traffic. Otherwise, the measurements collected from such an environment will be incorrect. Refer to the Data Center Real User Monitoring WAN Optimization Getting Started. 8. Configuring global monitoring setting for Cerner Global settings are settings that affect monitoring of all services based on Cerner analyzers for a given AMD. It is crucial to configure these settings before defining specific Cerner monitoring rules. For more information, see Creating Software Services Based on Cerner Analyzers [p. 41]. Customizing Monitoring Rules 9. Define your own software services on specified ports and for specified IP addresses If you configure user-defined software services for Cerner, you will be able to see Cerner data on many CAS reports, such as Software Services, Tiers, or Network Status: Software Service View. For more information, see Defining Software Services Manually in the Data Center Real User Monitoring Web Application Monitoring User Guide and Defining Software Services with a Wizard in RUM Console in the Data Center Real User Monitoring Getting Started. For more information on CAS reports, please refer to Central Analysis Server - User Guide. 10. Display the reports to review statistics for monitored traffic For more information, see Cerner Traffic on CAS Reports [p. 55]. Fine-Tuning Reporting Configuration 11. Configure the sites, areas, and regions A site is a term for a group of users that are located in the same IP network or group of networks sharing similar routing properties. Sites can be grouped together into areas, which, in turn, can be grouped together into regions. The hierarchy of sites, areas, and regions provides an organized view of the monitored network on the reports. For more information, see Configuring Sites, Areas, and Regions in the Data Center Real User Monitoring Administration Guide. 12. Configure the transactions, applications, and reporting groups Transactions are sequences of information exchange that represent particular actions or functions performed by a human user or a client program. They are viewed as higher-level units of self-contained functionality and are tied to applications. For example, they may represent the procedure for an online purchase or ticket booking. AMD monitors traffic data and prepares it for transaction monitoring by an ADS and CAS. Some of the relevant 12

13 Chapter 1 Cerner Application Monitoring Overview configuration and processing is performed on the actual RUM Console and some is performed on the AMD. For more information, see Managing Business Units in the Data Center Real User Monitoring Administration Guide. 13. Configure the tiers A tier is a specific point where DC RUM collects performance data. You can have monitoring data reported based on the default tier configuration, or you can define tiers that fit your network architecture. For more information, see Cerner Tiers [p. 53]. 14. Configure the monitoring of sequence transactions DC RUM enables you to define and monitor transactions that are sequences of steps. For more information, see Adding Cerner Transactions [p. 48]. Troubleshooting 15. Troubleshoot problems You can review the answers to the most common questions and troubleshoot your setup and report configurations. For more information, see Diagnostics and Troubleshooting [p. 57]. 13

14 Chapter 1 Cerner Application Monitoring Overview 14

15 CHAPTER 2 Adding Basic DC RUM Devices In a DC RUM configuration, there are two device types: data collectors and report servers. To start using the product, add and configure at least one AMD data collector and one CAS report server. You manage these devices using a configuration tool called the RUM Console. Adding an AMD to the Devices List Before you can monitor traffic with DC RUM, you have to add and configure an Agentless Monitoring Device using the RUM Console. To add an AMD to the list: Adding an AMD 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type list, select AMD. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Information 5. In the Device IP address box, type the device IP address. 6. In the Port number box, type the port number for communication with this device. The standard port number used by AMD is Optional: Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. 15

16 Chapter 2 Adding Basic DC RUM Devices Providing the Authentication Details 8. Type the user name and password of the account that will be used for managing this device. By default, the AMD user is set to compuware and the password is set to vantage. The credentials entered here are used by the RUM Configuration to communicate with the device and are also passed to the report servers so that they can collect monitoring data for processing. Note that the values used here for authentication are not the same as the values you use for logging in to the device via SSH or local console. Configuring Advanced Settings 9. Select the Advanced options tab. 10. Optional: Under Secondary device connection, provide an alternative IP address for this device. 11. Optional: Enable SNMP connection. Optionally, you can define the SNMP connection parameters so that you can obtain more detailed health information about the device. To define SNMP connection parameters: a. Select SNMP Connection check box. b. Type the read community name and port number. 12. Enable Guided Configuration. By default, the Guided Configuration connection is enabled when you add an AMD. However, for performance reasons, the number of AMDs with enabled Guided Configuration is limited to 50. Any additional AMDs do not feed data to the Guided Configuration perspective. This means that the monitoring data from the additional AMDs is not available for generating the web traffic statistics or defining the web software services with a wizard. By default, the port number for communication between the Console Basic Analyzer Agent and the RUM Console Server is set to 9094 and the secure connection is enabled. In most cases, it is not necessary to modify this setting. If the default port number is already in use by other services, however, type the new port number in the Port number box. In this case, you also have to manually change the port number setting on the Console Basic Analyzer Agent side. For more information, see Modifying Connection Settings for Guided Configuration in the Data Center Real User Monitoring Administration Guide. 13. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 14. Click Finish to save the configuration. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. 16

17 Chapter 2 Adding Basic DC RUM Devices Adding a CAS to Devices List To view reports based on the data from the AMD, use the RUM Console to add and configure a CAS report server. Adding a CAS 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type menu, select CAS. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Details 5. In the Device IP address box, type the device IP address. 6. In the Port box, type the port number for communicating with this device. The standard port number used by the CAS when communicating over HTTP is Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. Providing the Authentication Details 8. Choose whether authentication should occur via CSS. 9. Type the user name and password of the account that will be used for managing this device. Configuring the Advanced Settings 10. Select the Advanced options tab. 11. Optional: Under Secondary device connection, provide an alternative IP address for this device. 12. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Click Finish to save the configuration. Configuring the CAS-AMD Connection 14. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 15. Select a report server from the list of devices. Click the server once to display the detailed information for the device. 16. Select the Data Sources tab. 17

18 Chapter 2 Adding Basic DC RUM Devices 17. Click Add Data Source. 18. Select your AMD from the list and then click the button. 19. Click Finish to save the configuration. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. What to Do Next It is important to keep the devices synchronized to avoid improper data interpretation. For more information, see Synchronizing Time Using the NTP Server in the Data Center Real User Monitoring Smart Packet Capture User Guide and Time Synchronization Between AMD and Server in the Data Center Real User Monitoring Administration Guide. Adding ADS to Devices List To view reports based on data from the AMD, use the RUM Console to add and configure at least one CAS report server. In addition, you can add one or more ADS report servers in a farm configuration. Adding an ADS 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type menu, select ADS. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Details 5. In the Device IP address box, type the device IP address. 6. In the Port number box, type the port number for communication with this device. The standard port number used by ADS when communicating over HTTP is Optional: Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. Providing the Authentication Details 8. Choose whether authentication should occur via CSS. 18

19 Chapter 2 Adding Basic DC RUM Devices 9. Type the user name and password of the account that will be used for managing this device. Configuring the Advanced Settings 10. Select the Advanced options tab. 11. Optional: Under Secondary device connection, provide an alternative IP address for this device. 12. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Click Finish to save the configuration. Configuring the ADS-AMD Connection 14. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 15. Select the ADS from the list of devices. Click in the row corresponding with your server to display details for the device. 16. Switch to the Data Sources tab. 17. Click Add Data Source. 18. Select your AMD from the list and then click the button. 19. Click Finish to save the configuration. 20. Configure the ADS and CAS to work together. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. Adding Business Service Management to RUM Console To gather Cerner data from Cerner Real-Time Management System (RTMS), transform it, and send to CAS, you must add the Business Service Management to RUM Console. Complete the following steps to gather Cerner data and send it to CAS for processing: 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type list, select BSM. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. 19

20 Chapter 2 Adding Basic DC RUM Devices Specifying the Connection Details 5. In the Device IP address box, type the device IP address. The device IP address should correspond to the IP address of the server that contains the Business Service Management with Cerner RTMS adaptor and Tomcat servlet. 6. In the Port number box, type the port number for communication with this device. Make sure the port selection corresponds to the port provided during the adaptor installation. The default port is 8090, but this can change depending on the ports in use. The Cerner adaptor installation displays the assigned ports in the final installation screens. 7. Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. Providing the Authentication Details 8. Type the user name and password of the account used to manage the device. A user name and password are required for authentication in RUM Console to access the servlet. The default user name and password can be found in the tomcat-users.xml file. The tomcat-users.xml file is located at:...\bsm_home\vsm\apache-tomcat \conf. 9. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. If the device fails to respond correctly, it may take several seconds before the test times out. 10. Click Finish to save the configuration. 20

21 CHAPTER 3 Verification of Traffic Monitoring Quality Use the RUM Console to verify the traffic monitoring quality using two tightly connected solutions: Sniffing Point Diagnostics and Application Overview. We highly recommend that you perform this step at the beginning of your DC RUM deployment to verify that your hardware is working properly and that the applications you intend to monitor are detected. You can verify the test results and repeat them as needed at any time and for any network conditions. IMPORTANT All verification is based on a traffic recording, either manual or automatic. The outcome may not be representative if the target traffic is low at the time of recording or if you are unable to capture a satisfactory number of complete sessions. Choose automatic or manual traffic recording to capture unfiltered or filtered traffic. Enable automatic recording only during the configuration process and then disable it. It can negatively affect the performance of the AMD during normal operations, especially if you are running a 32-bit AMD in a high-traffic environment or a 64-bit AMD with the native driver. For the most complete and reliable statistics, use the 64-bit customized driver on the AMD. The verification of traffic monitoring quality is possible only for AMD 11.7 or later. Sniffing Point Diagnostics Sniffing Point Diagnostics is a type of hardware state analysis that enables you to perform pre-monitoring tasks without the need to access the AMD terminal. You can use it to validate the operation of the sniffing points, instead of using a series of UNIX or rcon commands. This step can be performed at the DC RUM deployment stage or at any other time to determine if the AMD performance is affected by malfunctioning hardware or external networking conditions. The Sniffing Point Diagnostics analysis can detect issues, such as: No traffic detected on sniffing interfaces. Interface or link overload. 21

22 Chapter 3 Verification of Traffic Monitoring Quality Poor quality of traffic due to mirrored ports on switching hardware configuration. Dropped packets (indicates AMD overload). Network conditions when unidirectional traffic prevails. Rejected packets, invalid packets, wrong check sums for packets. Missing packets (either lost or dropped). Missing bytes (how much traffic is lost in general). Conditions affecting AMD calculations, such as: Duplicate traffic that cannot be handled by the AMD. Incorrect choice of packet deduplication method. Incorrect settings for packet deduplication buffer. Incorrect settings for maximum packet size or huge packet size. Conditions affecting AMD performance, such as: Duplicate traffic handled by the AMD. Large percentage of non-ip traffic (noise). Large percentage of non-tcp or non-udp traffic (noise). Reordered sessions. Miscellaneous SSL problems: Unsuccessful decryption (in general). Uninitialized SSL cards unable to decrypt traffic. The ratio of encrypted and successfully decrypted traffic to encrypted and non-decrypted traffic. Incorrect or missing private keys. No match between the key and server certificate. Dropped or corrupted packets preventing decryption. Unsupported cipher methods (for example, Diffie-Hellman based key infrastructure). Unsupported SSL versions or features. Prerequisites and Best Practices To diagnose application detection problems and sniffing point connection problems, ensure that: All cables are connected correctly. The AMD is properly installed and configured. This includes the post-installation steps, such as interface identification and network configuration. Traffic recording lasts long enough to capture a reasonable amount of traffic volume, for example, 20 to 30 minutes of traffic. 22

23 Chapter 3 Verification of Traffic Monitoring Quality Do not use specific capture profiles when recording traffic. Always use the All available option for capture profiles when you do manual recording. When you need to diagnose traffic or capture port problems, enable automatic trace recording. Trace recording provides access to regular and fresh snapshots of the traffic that is traveling on your network. Sniffing Point Diagnostics Reports Sniffing Point Diagnostics reports are organized into several sections, each representing a separate set of metrics related to either hardware or network traffic. This topic provides directions for viewing the reports, but you can follow each step or skip steps to view the only the information important to you. 1. Start either by looking at device health or from the reports section directly. If you enabled automatic trace recording, you can monitor the device state on the Device Status tab of the Devices screen. A separate set of statistics is provided for each AMD added to the console. If there are any alarm messages, go to Devices and Connections Verify quality of monitored traffic. Inspect network interfaces in detail for a selected AMD. Open the Overview report to verify that the proper type of network driver is being used (custom or native) and that traffic has been detected, and check the number of dropped packets and other performance related issues. You can also verify that the NIC drivers are operational. For more information, see Network Interface General Statistics [p. 23]. 2. Switch to the Protocols section to inspect protocols. See whether network protocols are detected (IPv4 or IPv6) and verify detection of transport protocols (TCP or UDP). For more information, see Network and Transport Protocol Information [p. 26]. 3. Switch to the Services section to see the most active services. For more information, see Services Detected in the Traffic [p. 26]. 4. Depending on your goals, switch to the Sessions section either by selecting a particular service on the Services report to see session details or by choosing the Sessions section to see general statistics for all sessions. For more information, see Session-Related Statistics [p. 26]. 5. If you use SSL decryption, you can inspect whether there are problems detected for the currently used SSL engine or keys. For more information, see SSL Diagnostics [p. 28]. Network Interface General Statistics The Overview section of the Sniffing Points Diagnostics reports enables you to verify the general state of capture ports on a selected AMD. The information in the Overview section is gathered directly from the NIC driver operating on the AMD. For the most reliable results, use the 64-bit customized drivers. 23

24 Chapter 3 Verification of Traffic Monitoring Quality Calculation of Analyzed Traffic The calculation of analyzed traffic is performed in several stages, gradually excluding the irrelevant statistics: 1. The overruns are excluded first. When the received packets are counted, the overruns are omitted. 2. The calculation of the received packets depends on the subtraction of errors and filtered-out packets. 3. The dropped packets are counted after the filtered packets are disregarded. 4. The number of analyzed packets is the count of packets remaining after all of the previous categories are subtracted. In default AMD installations, non-tcp/udp packets are not part of this process and are never counted when the number of analyzed packets is given. Non-TCP/UDP traffic increases the amount of analyzed traffic only if you enable the monitoring of the default software services. Figure 2. Graphical Explanation of Analyzed Traffic Calculation for an AMD with 64-bit Customized Network Interface Driver All network packets Overruns Packets not received Received packets Errors and non-conditional filtering Errors: length or bad checksum; filtered out: non-ip Load balancing If active, fraction of the traffic Configuration filtering Based on defined software services Sampling and dropped packets Packets not analyzed due to performance issues Non-TCP, non-udp If default software services enabled Analyzed packets 24

25 Chapter 3 Verification of Traffic Monitoring Quality Interface Operation-Related Metrics The statistics presented on this screen include: Overruns Overruns may indicate a link overload. The overload is typically caused by an exceptionally high traffic volume. This value may also indicate that the network interface or network interface driver cannot manage the amount of traffic received. Other hardware-related issues may also cause overruns. If a high overrun occurs, limit the traffic volume received by the card. Errors (length) Packets of erroneous length are reported when they are too big (such as jumbo frames) or are bigger than the maximum transmission unit (MTU). To avoid such problems, you can increase Maximum packet size in the Entire Configuration perspective. For more information, see Configuring General Data Collector Settings in the Data Center Real User Monitoring Web Application Monitoring User Guide. Errors (bad checksum) Checksum-related errors are typically caused by insufficient signal strength on an optical link. In other cases, checksum errors may indicate Ethernet distortion, such as duplex problems, where the checksum errors may result, for example, when the duplex auto-negotiation process fails. Check the host switch and AMD duplex settings. Filtered out (non-ip) Non-IP packets, such as ARP traffic. Even large numbers of such packets are generally considered harmless. They are not analyzed by the AMD software and are regarded as noise. Preventing such traffic from reaching the AMD may reduce the possibility of performance degradation. Filtered out (load balancing) This setting is only applicable in deployments with multiple AMDs where each device only analyzes a certain part of the same traffic. Filtered out (configuration) Provides additional filtering based on software service definitions. In default installations, where monitoring of the default software services is turned off, the driver limits the number of processed packets to only those that are relevant to the IP addresses included in user-defined software service definitions. Dropped (sampling) Sampling here means dropping packets when the driver performance is degraded. Packets are dropped in a controlled manner, and always with care, to preserve complete and consistent sessions. The packet drops almost always mean that traffic is too heavy for a complete analysis and that, with packet drops, the precision of CAS reports is diminished. Sampling is only active with the customized 64-bit driver and diagnostics always use this sampling mechanism regardless of the settings used in the general AMD configuration. Dropped (driver performance) Drops are always a symptom of problems, especially when SSL analysis is deployed. Drops occur when AMD software is unable to analyze all of the packets it receives from the driver. If you use 32-bit or native drivers, you may experience uncontrolled packet 25

26 Chapter 3 Verification of Traffic Monitoring Quality dropping. If you use the 64-bit customized driver, packet dropping may occur, but in a software-controlled manner with care for monitored data contingency. To avoid packet dropping, decrease the traffic volume that your AMD analyzes or reduce the number of monitored software services. Non TCP/UDP Whether these statistics are classified as analyzed or not depends on the default software service monitoring. The numbers in this section are mostly relevant if you enabled monitoring of default software services. In this case, ICMP traffic is also analyzed. If monitoring of the default software services is disabled and you still see a large percentage of non-tcp and non-udp traffic, it is possible that AMD performance will be affected. Network and Transport Protocol Information Use the Protocols report to check the ratio of supported transport or network protocols. Only supported protocols are shown. In general, this report enables you to check whether traffic that makes sense (from the DC RUM perspective) is present and is heavy enough to give meaningful results for report servers. NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. Problem Detection Low traffic for the IPv4 or IPv6 network protocols may indicate further monitoring problems. The presence solely of multicast or broadcast traffic is an indication that port mirroring is not enabled or inactive. Services Detected in the Traffic This overview report enables you to identify the most active services on your network. You can see what their load is and what protocols they use, and filter the results to display all data, monitored services, or unmonitored services. You can also use filters to display statistics for all, monitored, or unmonitored services with additional protocol filtering. For each service, you can open the Sessions report to verify session-level statistics. NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. Session-Related Statistics The Sessions section enables you to view detailed information about traffic quality. 26

27 Chapter 3 Verification of Traffic Monitoring Quality The statistics presented on this screen include: Duplicates, Unhandled duplicates The value presented on the Sessions screen depends on the currently selected deduplication method in your AMD configuration. Packet duplicates may indicate incorrect configuration of mirroring ports. While this may be a sign of a problem, values of 10 to 20 percent typically are no reason for concern. The AMD is capable of packet deduplication. Higher numbers of duplicate packets will degrade the AMD performance and may negatively influence the monitoring results. The diagnostics mechanism for duplicate detection and counting for this report works with different settings than the network monitoring processes on the AMD. Duplicate detection is performed using both methods of duplicate detection and with different settings (buffer and delay detection size). Based on these settings and calculations, Sniffing Point Diagnostics provides suggestions concerning duplicate handling, such as increasing buffer size or changing the deduplication mechanism. You should check whether there are unhandled duplicates detected, in which case it is suggested that you switch the detection method in the AMD general settings. For more information, see Configuring General Data Collector Settings in the Data Center Real User Monitoring Web Application Monitoring User Guide. Unidirectional TCP sessions and UDP streams This may indicate a problem related to incorrectly configured mirroring ports. If the value of unidirectional traffic exceeds 90 percent, the RUM Console always marks it as an error. The numbers on the Sessions screen are the sums of many measurements; you are able to go deeper and analyze details for each server and check whether this is a problem related to a significant service or protocol. Insignificant traffic may be recorded and included in the general analysis, so always check the detailed reports when you see alarming numbers on the Sessions report. TCP sessions with missing packets Missing packets may result from interface or driver packet drops. If a session with missing packets is shown, the percentage value is counted with regard to all sessions. For example, if two percent of sessions have missing packets reported, this means that two out of a hundred sessions have missing packets. TCP sessions with missing packets and TCP bytes lost in missed packets may provide valuable insight into SSL decryption problems, especially in the case of long SSL sessions. TCP bytes lost in missing packets This is a complementary value to the TCP sessions with missing packets. Verify the number of lost bytes with regard to missing packets to see whether the problem is serious (if there are large sums of missing bytes). This is useful additional information in the case of long TCP sessions; because one lost packet is enough to classify a session as having missing packets, the number here gives insight into the actual loss rate. TCP sessions with reordered packets Reordered packets are typically found when there is a WAN link enabled. Devices transferring WAN packets may affect the packet order. The existence of reordered packets is not a problem in itself, because the AMD software can restore original packet order, but an excessive number of such packets may cause performance degradation. 27

28 Chapter 3 Verification of Traffic Monitoring Quality NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. SSL Diagnostics The traffic for this report is dependent on capturing complete sessions. Incomplete sessions, missing packets, or missed handshakes cause a large number of errors and a large number of errors results in unreliable reports. Always be sure to record enough traffic for an adequate length of time to allow you to capture complete sessions. The Statistics for encrypted traffic, SSL card and keys report is only available after the traffic trace recording is finished. Partial statistics for SSL are not provided for unfinished sessions. General Statistics for Encrypted Traffic For a given time range, defined by the scope of the recorded traffic traces, you can see the recognized SSL engine (for example, OpenSSL or ncipher) and the number of keys exchanged in the traffic. The remaining sections of this diagnostic report show the detailed information about the keys, the overall summary of the captured SSL traffic, and whether there are errors. The servers section shows information for all SSL traffic captured during the traffic trace recording. All of the detected encrypted protocols are listed together with their matching keys, if they are seen in the traffic. You can see whether the key exchange was successful; the matched keys are indicated by the icon. Key and certificate matching enables you to verify that certificates were found and were valid. No matching may indicate that the certificates are out of date. SSL Server Status The Status column shows whether there are errors or whether erroneous sessions prevail. A traffic capture sometimes does not contain session beginnings, or it contains incomplete handshakes, or it has no master session; these sessions are marked as ignored, as indicated by the gray ( ) color bar. The sessions with errors are marked by a red ( ) color bar. The main causes of errors are missing packets or missing keys. Other causes of errors are listed in detail on the Detailed SSL Statistics for servers report. Detailed SSL Statistics for Servers Detailed SSL statistics for servers are accessed from the Server or Status columns. This report shows: The percentage of the sessions without error, with errors, or ignored. The counts of each problem, in detail, for the error or ignored sessions. The number of decrypted sessions if there are no problems. 28

29 Chapter 3 Verification of Traffic Monitoring Quality You can filter the results. Use Sessions finished to display the data for completed sessions. Use Sessions in progress to display the sessions that are still in progress (sessions that did not end before the traffic capture stopped; to see those session statistics). Figure 3. Example of Detailed SSL Statistics for Server, Errors Detected Due to Private Key Mismatch SSL Keys Because invalid or outdated keys are usually not removed from SSL cards, the list of keys for which an error status is indicated may be considerably long. In such cases, sort by the Status column to see keys correctly matched. Note that it may be necessary to format the SSL card storage area to refresh the key list. Application Overview The Application Overview screen enables you to answer several questions about your applications at the onset of your monitoring configuration. Are all my applications or servers detected? What applications or servers are detected? Can the detected applications or servers be successfully monitored? How heavy is the traffic for each application or server? What services are detected on each server? How heavy is the traffic for each detected service? Note that incomplete sessions are not analyzed. If no beginning is recorded for a session, that session is not analyzed. 29

30 Chapter 3 Verification of Traffic Monitoring Quality The Application Overview screen is an optional step towards defining new software services. To access it, select Software Services Add Software Service in the console top menu, then select By traffic lookup. Figure 4. Example of the Application Overview Screen Showing Detected Applications From this screen, you can configure software services either manually or by using the wizard. If it is possible to go through a step-by-step configuration, a wizard icon ( ) is displayed for the given protocol or service. Application Detection Mechanism Application detection is a three-stage process: 1. To provide the most accurate results, packet analysis for SSL, HTTP, HTTPS, SOAP, and related protocols is performed as a first step toward application type detection. Application recognition is based on the first matching pattern found. This means that some services may not be properly classified if multiple protocols are used in one session. For example, if your application uses HTTP and SOAP over HTTP protocols, and plain HTTP communication opens a session, the application is classified as HTTP. 2. Applications are also detected based on discovery of well-known ports. The default protocol definitions are stored on the AMD and can be exported from the RUM Console. For more information, see Exporting the AMD Configuration in the Data Center Real User Monitoring Administration Guide. At times applications may use ports commonly used for other purposes. The AMD is unaware of these circumstances and will report well-known protocol names. For example, if one of your web applications uses port 8080 and uses HTTP for communication, the AMD reports this as an HTTP proxy. 3. If none of the selected conditions matches, the application is labeled as Unknown TCP or Unknown UDP. 30

31 Chapter 3 Verification of Traffic Monitoring Quality Server recognition in application detection is based on heuristic session analysis; results may vary depending on the type of network interface driver used. Using RUM Console to Identify Problems Related to Network Hardware Operation Typical configuration errors related to port mirroring can, at times, severely affect the AMD software traffic analysis capabilities. Faulty hardware configuration may result in no data seen by the AMD, a large number of duplicate packets reaching the AMD, or only a limited portion of traffic visible to the monitoring software. Use the Application Overview and Sniffing Point Diagnostics sections as tools to solve issues related to the switching hardware configuration. The following list describes several common problems and some possible causes and solutions. No data seen by the AMD The cable is connected to the wrong physical port on the destination switch. This can be checked by physically tracing the cable directly to the switch and confirming the port ID. The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror incorrect ports or an incorrect destination. This can be resolved by logging on to the source switch and checking the mirroring ports configuration relevant to the requirements (see the vendor-specific documentation for details). No data seen on Application Overview but non-tcp/udp traffic seen in interface statistics The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror incorrect ports or an incorrect destination. This can be resolved by logging on to the source switch and checking the mirroring ports configuration relevant to the requirements (see the vendor-specific documentation for details). Application Overview does not show all expected data The port mirroring destination may be oversubscribed or dropping packets. Check this by logging on to the switch and checking the SPAN or mirror destination interface. If it is recording many drops, review the configuration of source ports to understand the ratio of source interface bandwidth to destination interface bandwidth. If the ratio is excessive (for example, greater than 4:1), consider reducing the number of source interfaces. If applicable, consider using device-specific filtering to reduce the load on the destination interface (for example, VACL, Rx-only, or Tx-only sources). By design, port mirroring does not forward faulty frames. Check the source device interface statistics to ascertain the nature of the drops (see the vendor-specific documentation for details). Check the interface-related metrics. If there is a high rate of Errors (bad checksum), consider hard-configuring one end of the AMD SPAN connection to prevent auto negotiation. Session-related report shows a high rate of packet duplicates A SPAN or mirror operates by copying frames from source interfaces and directing them to the destination interface. In effect, configurations often result in two copies of a packet. 31

32 Chapter 3 Verification of Traffic Monitoring Quality For example, if the source of a SPAN or mirror is set as a VLAN, any traffic that goes from one switch port to another switch port within the VLAN appears twice on the mirrored port. If the number of duplicates starts to affect AMD performance, consider reducing the number of source interfaces. If applicable, consider using a device-specific filtering control to reduce packet duplication (for example, VACL, receive-only, or transmit-only sources) or consider using tap technology as opposed to port mirroring to collect the data. Only unidirectional streams are seen on session-related overview If the AMD is connected via a SPAN or mirror, the configuration has been set incorrectly to send only one side of a receive or transmit stream to the destination. Log on to the local source switch to check the configuration (see the vendor-specific documentation for details). 32

33 CHAPTER 4 Basic Monitoring Configuration You can define many configuration settings globally for all software services for a given protocol and Data Collector, or locally for specific user-defined software services. If you specify both types of settings, the settings for a user-defined software service take precedence over the corresponding global settings. Monitoring software services For Cerner analyzers, there are no default software services defined. To monitor Cerner traffic, you have to define your own software services on specified ports and for specified IP addresses. To access global settings for a Data Collector, go to Global Front-End Monitoring Cerner MQ in the navigation tree of the RUM Console. To access settings for specific software services, you need to select and open the Rules configuration for that software service. For details on setting up and configuring monitoring for Cerner protocol, see Configuring AMD to Monitor User-Defined Software Services [p. 41] Monitoring transactions Transactions are sequences of information exchange that represent particular actions or functions performed by a human user or a client program. They are viewed as higher-level units of self-contained functionality and are tied to applications. For example, they may represent the procedure for an online purchase or ticket booking. AMD monitors traffic data and prepares it for transaction monitoring by an ADS and CAS. Some of the relevant configuration and processing is performed on the actual RUM Console and some is performed on the AMD. See Monitoring Sequence Transactions [p. 47] for information on how to configure and use transaction monitoring on the AMD. AMD General Configuration Settings For any given AMD, you can set a variety of options such as time thresholds. The general settings affect monitoring of software services, but they can be overridden by specific settings for a particular analyzer or software service. To define the general settings for an AMD: 33

34 Chapter 4 Basic Monitoring Configuration 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 4. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 5. Select Configuration Global General to access the list of general configuration settings. While some of the options control only general AMD behavior, some options in the Advanced group affect more specific configurations in application monitoring. For example, if Inherit from global settings is selected in your other configurations while configuring user-defined software services, the global setting takes precedence over the specific monitoring configuration. Configuration options include: Monitoring interval The monitoring interval in minutes. Increasing this value reduces the number of chunks of data that need to be transferred and processed. Default: 5 minutes. Operation time threshold The number of seconds after which an operation is considered to be slow. The global threshold value depends on the analyzer. This threshold is used by the following analyzers: Cerner Cerner over MQ Epic Generic with transactions HTTP MS Exchange over HTTP MS Exchange over HTTPS Oracle Applications over HTTP Oracle Applications over HTTPS SAP GUI SAP RFC SAP GUI over HTTP SAP GUI over HTTPS SMTP SSL SSL Decrypted Server time threshold The Server time threshold relates to the server time portion of an overall operation time. Server times above the threshold limit are considered to be slow due to poor datacenter performance. 34

35 Chapter 4 Basic Monitoring Configuration This threshold is used by the following analyzers: HTTP SAP GUI over HTTP SAP GUI over HTTPS IP address of the server authorized to set AMD time The IP address of the report server that has the authority to synchronize the time with this AMD. In an environment with a number of servers sharing the same AMD, it is good practice to designate only one of these servers as a time synchronization server to make changes to AMD settings. Otherwise, the server used for time synchronization will change inadvertently every time you save an AMD configuration. Default analyzer The default setting for the TCP analyzer is Generic (with transactions). To change it, select another analyzer from the list. Client RST packet timeout to mark session as CLOSED If the time between the last ACK for data sent by the server and an RST packet sent by the client is greater than this value, the session is treated as closed instead of aborted. Huge packet size The upper size limit, in bytes, of an HTTP request to be processed successfully by the AMD. Maximum packet size The AMD is capable of processing packets of up to bytes, besides the Ethernet standard MTU (Maximum Transmission Unit) of 1536 bytes. Choose one of the predefined values (2048, 4096, 8192, or bytes) to enable the AMD to process non-standard MTU packets. When you have chosen the Maximum packet size value, make sure that you also set the Huge packet size to an applicable value. Enabling theamd to process nonstandard MTU packets without extending RAM on the machine and leaving Packet buffer size (64-bit AMDs only) and Data memory limit unchanged can cause an excessive packet loss. To avoid this, extend RAM and configure its usage as recommended in the tables below. For more information, see Setting Packet Buffer Size in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Setting Data Memory Limit in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. NOTE Do not enable the processing of large packets for a Small AMD. These devices are not designed to process larger packets. For more information, see Small AMD in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. 35

36 Chapter 4 Basic Monitoring Configuration Table 1. Recommended RAM Configuration for Maximum Packet Size Values for AMDs Maximum packet size 8192 B or less 8192 B B Recommended RAM size for 64-bit platforms 64 GB 96 GB 128 GB Deduplication method You can choose one of four methods for eliminating duplicate packets: Based on TCP checksum and IP ID Using this method, duplicate packets are detected based on their TCP checksum and IP ID. Based on TCP checksum and IP ID (excluded SEQ and ACK numbers) Using this more complex, two-stage method, duplicate packets are detected based on a modified packet KCP checksum (SEQ and ACK numbers are excluded) and IP ID. This method is useful if the AMD captures packets on various interfaces of the router, rewriting SEQ and ACK numbers. A packet is considered a duplicate when the modified checksum, IP ID, and SEQ and ACK numbers are identical. First, a packet checksum with SEQ and ACK numbers is created and compared to the packets stored in the detection buffer. If the comparison indicates that the packet is not a duplicate, it is checked to determine whether it matches the current session. A packet matches the current session when its SEQ and ACK numbers are different from processed and cached numbers by the value defined in TCP duplicate window. If the difference exceeds the defined value, the AMD assumes the ACK and SEQ numbers were rewritten by the router and the packet is considered a duplicate. TCP checksum, IP ID and MAC address (excluded SEQ and ACK) Using this method, the deduplication process is similar to the one based on TCP checksum and IP ID (excluded SEQ and ACK numbers), but in addition to TCP checksum and IP ID, the source/destination MAC addresses are also taken into account for the calculation. TCP checksum, IP ID and MAC address Using this method, duplicate packets are identified based on their TCP checksum, IP ID and source/destination MAC addresses. TCP duplicate window This setting is useful only if Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers. It is used for determining whether a packet, based on its SEQ and ACK numbers, belongs in the session. If a packet's SEQ and ACK numbers differ from the current session's SEQ and ACK numbers by a value larger than TCP duplicate window, the packet is considered a duplicate. Default:

37 Chapter 4 Basic Monitoring Configuration Packet buffer size The number of packets to keep in the buffer for use as a basis for comparison in duplicate packet detection. Newly captured packets are sequentially compared to the packets in the buffer. A newly captured non-duplicate packet (all packets in the buffer are unique) is placed on the top of the stack and the oldest is removed. Range: 6 to 24 packets. Default: 16. Reset duplicate detection time threshold Time of inactivity (in seconds) after which the duplicate packets elimination mechanism is reset. If Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers or TCP checksum, IP ID and MAC address (excluded SEQ and ACK), and the Reset duplicate detection time threshold should be greater than every response generation time (server time). 6. Publish the draft configuration on the monitoring device. Configuring Operation-Related Global Settings The operation-related global settings enable you to define options that apply to all monitored operations. These settings take precedence over the options defined for individual operations. 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 4. Select Configuration Global Operations to display the general configuration settings. The options are: Operation load time threshold The number of seconds after which an operation is considered slow. You can set this value with a precision of one ten-thousandth of a second. Default: seconds. The threshold is used by following analyzers: IBM over MQ Jolt MS Exchange Oracle Forms over HTTP Oracle Forms over HTTPS Oracle Forms over SSL Oracle Forms over TCP SOAP over HTTP SOAP over HTTPS XML XML over HTTP 37

38 Chapter 4 Basic Monitoring Configuration XML over HTTPS XML over MQ XML over SSL Max. operation duration The maximum number of seconds an operation can take. You can set this value with a precision of one ten-thousandth of a second. Default: 3600 seconds (1 hour). User abort threshold The minimum number of seconds between the beginning of a hit and TCP reset to count it as a user abort. Default: seconds. (You can set this value with a precision of one ten-thousandth of a second.) ADS data generation settings The options in the ADS data generation settings section can be used to handle various types of standalone hits, which are hits that cannot be automatically assigned to operations because the reference information, such as correlating response, defined or auto-learned URL, no authorization, or orphaned redirects, is missing. By default, most standalone hits are not taken into account when generating operations data. Report data without monitored URL Select this option to report data for hits without a URL that has been explicitly defined in user-defined services or recorded through auto-learning. Report standalone hits without monitored URL Select this option to report data for standalone hits that at the same time do not refer to a monitored URL, as in Report data without monitored URL. Standalone hits are hits without a response header, unauthorized hits, orphaned redirects, or other hits missing the reference context. Report hits without response header Select this option to report data for discarded hits (hits without a correlating response header). Report hits not added to any operation Select this option to report data for other standalone hits caused by factors not covered by other options of this section. Report unauthorized hits Select this option to report data for hits with rejected authentication. Report orphaned redirects Select this option to report data for redirects to sites that are not being monitored or are not visible and therefore appear as orphaned redirects. Report filtered data This is a diagnostics option. When configuring content type monitoring, you can filter out pages based on the content of the URL. For more information, see Monitoring of Non-HTML Objects Based on Content Type in the Data Center Real User Monitoring Web Application Monitoring User Guide. If you select this option, the filtered out pages are not reported, but are saved in the AMD data files. 38

39 Chapter 4 Basic Monitoring Configuration Ignored clients A list of clients for which TCP setup time are ignored and all operations start from the request packet. Right-click the list to open a menu of command options: Add, Edit, or Delete. 5. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. 6. Close the AMD Configuration window. 39

40 Chapter 4 Basic Monitoring Configuration 40

41 CHAPTER 5 Configuring AMD to Monitor User-Defined Software Services If you require detailed analysis of traffic, you need to specify software services to be monitored on specific IP addresses and ports. Creating Software Services Based on Cerner Analyzers Support for Cerner Millennium environment enables you to monitor the performance of Millennium operations seen on the network connection between Citrix Presentation Servers and Cerner Nodes (physical application servers). To configure the monitoring of Cerner traffic, you need to create one or more software service definitions for the Cerner analyzers and assign the Business Service Management adaptor as a data source for CAS and ADS. Before You Begin For the full scope of Cerner monitoring, besides creating one or more Cerner software service definitions, Business Service Management Connection Manager should be added as a device to the RUM Console. Adding it as a device to the RUM Console enables you to assign it to CAS or ADS so that they can receive data imported by its input adaptor from Cerner Real-Time Management System (RTMS). For more information, see Adding and Editing Devices in a Report Server Configuration in the Data Center Real User Monitoring Smart Packet Capture User Guide. To define a Cerner software service: 1. Create a definition of a Cerner service. For the desired address IP and port, create a definition of a user-defined Cerner software service to be monitored. Depending on the protocol used for communication between Citrix Presentation Servers and Cerner Nodes, Cerner over TCP or Cerner over MQ, choose an applicable analyzer for the software service. Create a software service for each if both protocols are used. For details on how to create a new service, see Configuring User-Defined Software Services in the RUM Console Online Help. 41

42 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 2. Optional: Configure the rtm.config file to report the user name for Cerner over MQ instead of the user ID. To change the default reporting behavior to report the user name, set the cerner.showfullusername parameter to true in the AMD usr/adlex/config/rtm.config configuration file for each AMD in your environment. NOTE You must log on with the root account or account of the user who installed the AMD to make changes to the rtm.config file. After making changes to this file, restart the service by running the service rtm restart command. 3. Review report results and adjust the configuration as appropriate. Determining the best possible configuration for your needs may be an iterative process, where you fine-tune the configuration several times after viewing your report results. Defining Oracle Forms Software Services For software services you intend to monitor that do not work on a well-known port, you can use the specific IP address and port of the service when defining the software service configuration. For such software services, you can measure a wide range of metrics and perform detailed traffic analysis. To add a new software service: 1. Start and log on to RUM Console. 2. In the top menu, select Software Services Add Software Service. The Add Software Service pop-up window appears, listing all ways of adding a new service. 3. Select Manually as a method of adding a new software service definition. The Add Software Service window appears. 4. Specify basic information for your software service. Provide a software service name. Select appropriate analyzer to monitor the traffic. Using check boxes, select the devices that will monitor the new software service. When you later publish the software service definition, the new configuration will be applied only to the selected devices. 5. Click OK to proceed to monitoring rules configuration. 6. Right-click in the Rules table and select Add from the context menu. The Rule Configuration appears. 7. Proceed with the rules configuration.for more information, see Configuring Rules for User-Defined Software Services in the Data Center Real User Monitoring Oracle Forms Application Monitoring User Guide. 8. On the Software Services screen, click Publish Configuration. 42

43 Chapter 5 Configuring AMD to Monitor User-Defined Software Services Configuring Rules for User-Defined Software Services Each software service can have a number of specific rules that define what is to be monitored and what additional options are in effect. Before You Begin It is assumed for this task that you are already familiar with the concept of software services and that you know how to create and edit software services and how to open the Rules window. After a user-defined software service is created, create a group of settings that comprise the rules for the software service. It is necessary to specify, at minimum, the IP addresses and port numbers for the software service. To configure rules for a user-defined software service: 1. In the Rules table, right-click to open the context menu and choose Add or Open. The Rule Configuration window will open. 2. On the Services tab, select or clear Enabled to activate or de-activate the service definition. 3. In Rule description, type a brief description that will later help to identify the rule. The description you enter at this point will be shown in the Rules table, in the column Rule Name. If no text is entered here, the IP address specified later will be used as a description for this rule. 4. Right-click in the Services table and select Add or Open from the context menu. The Service Details window appears when adding or editing rules. 5. In the Service Details window, in the IP address(es) fields, enter the server IP address, or enter a range of IP addresses if you plan to monitor more than one server. 6. In the Port(s) fields, enter the port number of the monitored service. You can provide a range of port numbers if such a range of ports is used in your environment. Some software services may be active on a number of predefined ports or may change ports dynamically. To allow for this, you can specify a range of ports. Note, however, that specifying more than one port for a service prevents the port number from being reported for that service. If you define more than one port for a particular service name and server IP address (by either specifying a range of ports or by creating two or more distinct rules for the same service name and server IP address but with different port numbers), the AMD will report the port number for this service as 0, causing the port number to be ignored in traffic reports. NOTE You can define up to 5000 definitions containing a server and a port. Each association of a server and a port counts as a single definition. Specifying a range of ports counts as providing many individual definitions. On CAS, the number of processed server definitions is limited by the license. For more information, see Per-Measurement Licensing in the Data Center Real User Monitoring Administration Guide. 43

44 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 7. Optional: Select Client port(s) for reversed-direction protocols. This option makes sense only for protocols such as X-Window whose client-server meanings are reversed. If you are uncertain, leave this option cleared. 8. Optional: Enter a virtual IP address if your network uses a pool of virtual IP addresses. For more information, see virtual IP address [p. 84]. 9. Optional: Enter the IP address of the server masking the addresses of monitored servers. If the servers you intend to monitor reside behind an appliance that masks and replaces the addresses of the target servers, you need to set NLB NAT masking IP address to the IP address of the masking server. Without doing so, the AMD will see two unidirectional conversations instead of one bi-directional conversation between the servers and appliance: The conversation between the client and server is observed and recorded (IP address A talking to IP address B) When a response travels to the client, a different session (IP address C talking to IP address A) is recorded due to the server's IP address being replaced by the load balancer's IP address. Unless you account for this, CAS reports will return reports with ambiguously granulated data. Using the NLB NAT masking IP address option will ensure that the AMD monitors contiguous conversations. 10. Click OK to confirm your changes and close the Service Details window. 11. Go through all available tabs and fine-tune the monitoring conditions. The number of available configuration options depends on the analyzer. See the analyzer-specific section for more information. 12. Optional: On the Options tab, define analyzer-specific options. Enable monitoring of persistent TCP sessions When this is selected, TCP sessions that do not start with SYN packets are monitored. By default, this is selected. Persistent TCP sessions are TCP sessions for which the start was not recorded. They are also referred to as non-syn sessions. These sessions can be included in the TCP statistics, based on the configuration properties you enable in RUM Console. The inclusion of these sessions may render the statistics somewhat inaccurate and must be undertaken with care. Page load time threshold An operation that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, go to Global General and set the Operation time threshold. Generate sequenced transactions and ADS data Select this option to provide data to the report server that consists of low-level protocol information, such as raw HTTP traffic data, which enables you to view the full HTTP request-response dialog. 44

45 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 13. Click OK to save the configuration. 14. Publish the draft configuration on the monitoring device. Example of Configuring Cerner Software Services This example shows how to add and configure a user-defined software service based on the Cerner analyzer. Assuming we have a Cerner server with IP address and port number 1414, to create and configure a new software service based on the Cerner analyzer: 1. Start and log on to RUM Console. 2. In the top menu, select Software Services Add Software Service. The Add Software Service pop-up window appears, listing all ways of adding a new service. 3. Select Manually as a method of adding a new software service definition. The Add Software Service window appears. 4. In the Software Service dialog box: Provide a software service name (for example, Cerner). Select Cerner or Cerner over MQ (depending on your Cerner protocol version) as the analyzer used to parse the software service traffic. For this example, we choose Cerner over MQ. 5. Click OK to proceed to defining of monitoring rules for your software service. 6. In the Rules table, right-click to open the context menu and choose Add. For more information, see Creating a New Rule for Cerner Software Service [p. 45]. You have to create at least one rule for monitoring of this software service. After the rules are created, they all are displayed in the Rules table. Creating a New Rule for Cerner Software Service Rules are mandatory for any software service to be monitored. Here you will define a rule for a Cerner-based user-defined software service. Before You Begin It is assumed for this task that you are already familiar with the concept of software services and know how to create and edit software services and how to open the Rules window. For more information, see Configuring Rules for User-Defined Software Services [p. 43]. To create a new rule for a software service based on the Cerner analyzer: 1. In the Edit Rule dialog box, enter a rule name. In the Rule description edit box, type Rule 1 or you can type your own rule name. 2. Right-click the Services table and select Add from the context menu. The Service Details window appears. 45

46 Chapter 5 Configuring AMD to Monitor User-Defined Software Services 3. In the Services window, type the server IP address, or type a range of IP addresses if you plan monitoring more than one server. We will create a simple rule to monitor one server (one IP address on the default port). In the Start IP address, let's type In the Port(s) fields, type the port number of the monitored service. Type the standard MQ port. 5. Leave the Client port(s) check box cleared. 6. Click OK to confirm your data and close the window. 7. Optional: Click the Options tab to review and modify common options for the Cerner analyzer. Managing User-Defined Software Services As a console user you have the ability to add, remove, and edit the properties of any software service defined on any of the AMDs. You can view all user-defined software services on the Software Services screen in the RUM Console. To access this screen, select Software Services Manage Software Services from the top menu. This screen contains information about the user-defined software services that were created on all devices managed from this console. You can add new services, delete existing ones, and copy them to other AMDs. Note that the default software services are specific to a single AMD and cannot be managed centrally. To view user-defined software services monitored by a selected device, access the configuration for this device. Select Devices and Connections Manage devices from the top menu, a list of all devices managed by the console appears. Next, select Open configuration from the context menu for the device. Assigning Software Services to Devices Software services mirrored across different AMDs (that is, having the same name and identical rules) are grouped together. The service name is the name for the group. Whenever you change rules on any of the AMDs in the group, the software services are separated into single entries. After a software service is created on one of the AMDs in your network, you can copy it to another device. To do so, on the Software Services screen, select Copy from the Actions context menu for a given software service. A dialog box appears where you can choose the AMDs by selecting the check box next to their IP addresses. After you click OK, the software service definitions are copied to the selected AMDs. On the Software Services screen, select the Deployment tab and click Change Assignment to modify the list of devices that are monitoring a given software service. 46

47 CHAPTER 6 Monitoring Sequence Transactions You can manage the sequence transactions (operation sequences) that are defined on an individual AMD or manage each transaction that is monitored by a group of AMDs. To view the defined transactions monitored by a single AMD, select Devices and Connections Manage Devices from the console top menu. Next, select Open configuration from the context menu for the AMD to access the AMD Configuration screen. Finally, select Configuration Sequence Transactions. The main Sequence Transactions table lists all of the currently defined transactions and their details: Sequence Transaction Name The name of a transaction. Application Name The application that includes the listed transaction. Type The protocol used to define the listed transaction. Steps The number of individual operations involved in the listed transaction. Priority The priority of the transaction. Possible values are 1 (highest priority), 2, and 3. Timeout The maximum time for the transaction to complete. Packaged Applications Identifies whether the listed transaction is a packaged application whose transactions are recognized by the report server automatically. On the Sequence Transactions screen, select Edit from the Actions menu for a given transaction to open the Edit Transaction window and examine the steps that make up the transaction. The listed details are as follow: Name, Application, Description: Timeout [s], Slow after [s], Priority URL, Timeout, Repetition. 47

48 Chapter 6 Monitoring Sequence Transactions NOTE If the transaction you are deleting is monitored by more than one AMD, a new draft configuration must be published to all of the affected AMDs. To view all transactions, select Reporting Configuration Sequence Transactions from the console top menu. Sequence Transaction Name The name of a transaction. Application The application that includes the listed transaction. Type The protocol used to define the listed transaction: ASYNC-HTTP, CERNER, CERNER-RTMS, HTTP, OF, SAP GUI, SQL or XML. Packaged Applications Whether the listed transaction is a packaged application whose transactions are recognized by the report server automatically. When you select a transaction by clicking it once, you can see the list of AMDs that monitor this transaction. Adding Cerner Transactions You can add a transaction to either an individual AMD or a range of AMDs using the RUM Console. To define a new transaction: 1. In the RUM Console, select Reporting Configuration Sequence Transactions. 2. Click Add Sequence Transaction. The Create Sequence Transaction pop-up window appears. 3. Enter the application and transaction names and a description. If you have configured the Dynatrace connection, click Browse to select a predefined application and a specific transaction within this application. For more information, see Configuring the BSM Connection in RUM Console in the Data Center Real User Monitoring Administration Guide. 4. From the Type list, choose Cerner. 5. Select the devices that will monitor the transaction. When you publish the new configuration, it is only applied to these devices. 6. Click OK. On the screen, specify the configuration details for the transaction. 7. Provide the timing and priority values: Timeout [s] The maximum time for the transaction to complete. Transactions must complete in this time to be logged as successful transactions. 48

49 Chapter 6 Monitoring Sequence Transactions Slow after [s] If the transaction execution time exceeds this value, the transaction is classified as slow. Specify this threshold as fractions of seconds, for example: 0.5. Priority Determines which transaction is recorded if two or more transaction definitions match the transaction detected in the monitored traffic. The valid priority values are 1 (highest priority), 2, and 3. A multiple transaction match can happen if, for example, you first create a generic transaction definition that can match a number of more specific transactions and then you create another transaction definition that matches a particular sub-type of that generic transaction type. If an observed transaction is found to match the latter definition, it also matches the first (more generic) definition, and the system will need to determine under which transaction name to record the observed transaction instance. By increasing the priority of the second, more specific definition, you can count the occurrences of this particular transaction sub-type, which are then not counted in the statistics for the generic transaction type. So you can use this feature to increase the priority of specific customized transaction definitions that should take precedence over more generic transaction templates. 8. Specify Cerner operations comprising the transaction steps. The Cerner operation can contain an optional wildcard character *. Using the wildcard character *, you can signify any number of any characters. For more information, see Using Wildcards in URLs in the Data Center Real User Monitoring Web Application Monitoring User Guide. To maintain the sequence of these operations, use the navigation buttons on the right. You can Add, Delete, Move Up, Move Down, or Copy the defined steps by selecting the step and clicking one of these actions. 9. Click OK to add your transaction definition to a draft configuration. 10. On the Sequence Transactions screen, click Publish Configuration. What to Do Next You can also add a transaction using the Transaction Inspector. For more information, see Filters and Transaction Inspector for Cerner Transactions [p. 50]. Modifying, Deleting, and Cloning Transactions for a Single AMD Modifying a Sequence Transaction To modify the definition of an existing transaction: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 49

50 Chapter 6 Monitoring Sequence Transactions This opens the Sequence Transactions table, listing all of the defined transactions for this AMD. 3. Right-click the transaction to manage and select Open from the context menu. You can modify any of transaction details. For more information, see Adding Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. Deleting a Sequence Transaction To delete selected transactions: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 3. Click the transaction that you want to delete. To delete multiple transactions with one step, hold the [Ctrl] key as you click additional transactions. 4. Right-click and select Delete to remove the selected transactions from the list. Cloning a sequence transaction To clone selected transactions: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 3. Click the transaction that you want to clone. To clone multiple transactions with one step, hold the [Ctrl] key as you click additional transactions. 4. Right-click and select Clone to duplicate the selected transactions. A cloned transaction is indicated by the original transaction name with (Clone) appended to it. There are differences between cloning and copying. For more information, see Monitoring Sequence Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. Filters and Transaction Inspector for Cerner Transactions The Transaction Inspector enables you to select individual steps and construct your own transactions from live traffic or historical data. The Transaction Inspector consists of two main areas: the Filters and the Transaction Inspector itself. 50

51 Chapter 6 Monitoring Sequence Transactions Filters Transactions can be defined manually by entering each step, however, the Filter enables you to examine Recent Data and select the detected steps to build a transaction. The transaction filter consists of two tabs: Data Filter The Data Filter tab enables you to define your filter by selecting the source and range of data to be filtered. From the list, you can select the report server and create a user filter. Select User Name or User IP Address and either enter the data manually or click Browse to open the Select User window, where you can select the user identified in transaction traffic by the report server. You can highlight or search for the specific user or user IP address and filter the search query based on any of the columns in the transactions table. After the user or user IP address is selected, you can choose to extract transactions from Recent Data stored on the report server. You have to provide a Begin and End date for the time range to be processed. Result Filter The Result Filter tab consists of a find field, a transaction detail field, and an interactive legend to filter transactions that have been classified as: Table 2. Result Filter Color Guide The transaction was recognized and matched with the transaction currently being defined. One or more steps One or more steps A Step or a URL Excluded URLs in the transaction in the transactions was recognized as which did not currently being for which an an already defined match any defined were not HTTP error other and saved definition. completed. than 404 occurred. transaction definition. By selecting and clearing the corresponding check boxes, you can filter the URLs from the data source. The color coding of the steps is based on your current transaction definition in the Transaction Definition area. To view the results and enable the filter to receive data, click, located on the right side of the Filters area. Transaction Inspector Transaction Inspector consists of two tables that display the URLs and Sequenced Transactions detected in the data source that is defined in the Filters section. The Transaction Inspector enables you to select one or a number of detected steps and add them to your transaction definition. Select the check box corresponding to the URL that you want to add and then click located above the table. You can add the steps from both the URLs table and from the Transactions table. 51

52 Chapter 6 Monitoring Sequence Transactions After the URL is moved to the Transaction Definition table, you can modify it, position it within a sequence of other steps, clone it as another step, or delete it using the operation buttons to the right of the Transaction Definition table. 52

53 CHAPTER 7 Cerner Tiers A tier is a specific point where DC RUM collects performance data. It is a logical application layer, a representation of a fragment of your monitored environment. There are two tiers on CAS that report Cerner data: Cerner If you have defined software services based on Cerner analyzers, the Cerner tier will automatically be displayed on the Tiers report. This front-end tier shows measurements for Cerner traffic monitored by AMD. RUM sequence transactions If you have defined sequence transactions for Cerner monitoring, the RUM sequence transactions tier will automatically be displayed on the Tiers report in the Client tiers section. For more information, see Solution-Specific Tiers in the Data Center Real User Monitoring Administration Guide. 53

54 Chapter 7 Cerner Tiers 54

55 CHAPTER 8 Cerner Traffic on CAS Reports There exist several CAS reports that enable you to analyze Cerner traffic by viewing various statistics. From the Reports menu, view the following reports: Applications: see the performance of applications for which data was detected on front-end tiers if you have defined Cerner applications and transactions on the CAS. For more information, see Application and Transaction Management in the Data Center Real User Monitoring Administration Guide. Tiers: analyze statistics for Cerner RTMS, Cerner, and RUM sequence transactions tiers. Drill down from the Cerner RTMS tier to access Cerner-specific reports, such as the Cerner RTMS Log. Top N View: view the most problematic software services, operations, and sites. Software Services: analyze statistics for Cerner software services. Network Landing Page: analyze your network performance. User Activity: analyze traffic statistics for particular users. 55

56 Chapter 8 Cerner Traffic on CAS Reports 56

57 APPENDIX A Diagnostics and Troubleshooting Report-Related Issues Central Analysis Server automatically detects a range of exceptions (anomalies) and notifies the report users. Exception notifications are displayed as yellow (warning) or red (error) triangle icons in the upper-left corner of the report window. To see the notification message, position the cursor over the triangle icon. The Slow Operation Load Sequence report is empty for an operation which is part of an XML transaction. Why and how do I fix this? For XML and SOAP, Operation Elements data is identical to Operation Analysis data, so, to avoid unnecessarily keeping the duplicates in the database, a VDATA_FILTER_XMLSOAP filter is set to true by default. Keeping this filter set to true saves disk space but, because the XML and SOAP entries are filtered out, it makes reporting on the Operation Elements level (elements or headers) impossible. To change the value of VDATA_FILTER_XMLSOAP property in userpropertiesadmin, type in the Web browser's Address bar and press [Enter], change the filter's property value, and click Set value to accept the change. To access this screen, you need to have administrative privileges for the report server. The yellow triangle displays AMDs produce no performance data. What do I do? The message AMDs produce no performance data means that AMDs connected to the report server do not produce any new data. To resolve this issue, you have to investigate the configuration of the AMDs and determine why they do not produce the performance data. The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? The report server has a built-in protection from simple configuration mistakes. One of the related problems is when data is incorrectly time stamped by AMD. This happens when the AMD is running with the system clock incorrectly set and is not being synchronized with the report server. If you see this notification, check the system time on the report server and on the AMD. Ensure the time synchronization option is turned on. 57

58 Appendix A Diagnostics and Troubleshooting To check the time synchronization: 1. Launch the RUM Console. 2. Select the AMD, right-click it and choose Open Configuration. The AMD Configuration window appears. 3. Select Global General. Check the IP address of the server authorized to set the AMD time. Make sure it is the same as the report server IP address. 4. Check the report server time setting. Do this by reading the time that is displayed at the bottom of the reports. Ensure the report server has the time zone set correctly. Figure 5. Example of the Report Time Stamp The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? Once a day the report server has to perform a database maintenance and memory cleanup. During that time, the data processing has to be suspended and you will see delayed data on reports. The daily maintenance is usually performed as the first task after midnight and it takes up to half an hour in installations with a large database. It is normal and expected to see this warning just after midnight. But if you see the message during the day, it can be a symptom of incorrect system configuration (check the time settings on the server) or of system overload. The yellow triangle displays No contact with the primary AMD. What do I do? This message indicates that the report server has lost contact with at least one primary AMD. If an AMD is marked as primary and the report server cannot communicate with this AMD, even if the performance data can be downloaded from the other AMDs, the system will wait until the communication with the primary AMD is restored. The yellow triangle displays No contact with any of the AMDs. What do I do? This message indicates that the communication link cannot be established with any of the attached AMDs. Check the network settings on the report server or the configuration of AMDs. 58

59 Appendix A Diagnostics and Troubleshooting The yellow triangle displays Delay in data processing. What do I do? If the last processed data is significantly behind the current time due to slow data processing or idle periods that occurred in the past, the report server displays the triangle icon with the message Delay in data processing. If the server had a delay, but now it is catching up, this message will not appear anymore. To confirm that delay is decreasing, inspect server.log and search for messages similar to this: T REC :10: zdata_43f47e58_5_t is being processed. Sample begin ts = :25. Sample delay 17 min. If the delay becomes smaller, the server is catching up. If the delay values are growing, it can indicate a system overload. The yellow triangle displays The AMD has not yet generated performance data. What do I do? This message indicates that some data files have already been generated on some AMDs, but not on the others. This may not be an indication of a problem and, when you refresh the reports after 30 to 60 seconds, this message may disappear. If necessary, verify the time synchronization among all the AMDs. See The yellow triangle displays Delay in data processing. What do I do? [p. 59]. The yellow triangle displays Data processing is being performed in the debug mode. What do I do? Data processing can be manually suspended and controlled by so-called debug mode, which can be enabled using Control Panel. Open Control Panel by typing: in the Address field of the web browser and clicking Go, then select Controlled data processing from the Configuration Management section. The red exclamation mark displays Data loading is in progress. Reports may be incomplete. What do I do? This message indicates that the report server is currently starting up. Because of this the information presented on reports may be incomplete. Depending on the database size, the startup process may take up to several minutes. If the server restart was not done manually or was not planned, inspect server.log or contact Customer Support. The red exclamation mark displays Low memory. The real-time cache will only be updated. What do I do? This message indicates that the report server has no free memory to process new entities such as software services, servers, and URLs. This message will be cleared when some resources are freed, this usually happens at midnight during the scheduled database maintenance (see The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? [p. 58]). All the metric values presented on reports (except user/client counters) will show correct values. However, the predefined tabular reports may not show all the entities they are intended to show. All the charts and DMI reports show correct data. The mechanism of updating the real-time cache, as described above, is a protection that allows the report server to continue the operation instead of closing down due to lack of memory resources. 59

60 Appendix A Diagnostics and Troubleshooting The red exclamation mark displays The number of servers has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored servers. If the number of observed servers reaches a defined limit, the report server will not accept any new servers and will drop the collected data for those servers. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red exclamation mark displays The number of clients has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored clients. If the number of registered clients (which also includes aggregated virtual clients such as Client from... ) reaches a defined limit, the report server will not accept any new clients and will drop the collected data for those clients. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red exclamation mark displays The number of sites has reached the defined limit. What do I do? The report server has a built-in limit of the number of automatically created sites. If the number of observed automatic sites reaches a defined limit, the report server will not create any new automatic sites and such traffic will be allocated to All Other. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The Sites report for a selected application is empty. Why? If the Sites report for a selected application is filtered for a client tier, such as Synthetic or RUM sequence transactions, it will not show any data. To see statistics for sites, drill down from the Applications report as follows: 1. Click the application name on the Applications report. 2. Click the client tier name on the Tiers report for a selected application. For the Synthetic tier, you will see the Overview Application Status report; for the RUM sequence transactions tier, the Sequence Transactions Log report. 3. Depending on the type of report, click the Overview Site Status or the Sites tab. I see gaps on the chart reports. Why are the charts incomplete? Gaps in reports mean that the report server missed some data and was not able to get it into the database on time. Your reports may resemble the example below. Figure 6. Gaps in a Graphical Report There are several reasons why the graphical reports may have incomplete data: 60

61 Appendix A Diagnostics and Troubleshooting The AMD was not able to detect any traffic from the monitored network, so it was not able to produce any valid data for the report server. To confirm that this was the reason, connect to the AMD using an SSH client and check whether the files named zdata_xxxxx_x_x are located in the /var/spool/adlex/rtm directory. Similar symptoms can be observed if the AMD has been down for some time and data files were not produced for that time. If data files are present and the viewed chart displays only a fragment of the monitored traffic, for example, for a specific server or site, it may indicate that a part of traffic, which was indented to be monitored, is missing. In this situation, the data files are much smaller than usual for the corresponding period of the day. Similar situations, that is, gaps only on some reports, may occur in a multi-amd installation when some AMD s were down or disconnected from the network. In the case when only one AMD is connected to the report server, communication problems do not cause data gaps. If the report server cannot communicate with the AMD, it will wait until the communication is restored and then will process all the data from the past. When there are multiple AMDs connected to the report server and there is a break in communication with only some of them, the report server processes the data from the available AMDs, so in this case, gaps can appear on some reports. If it is a critical issue and your network (or its parts) require continuous monitoring and you cannot miss the data from some AMDs, you have to mark the AMDs as primary. In this case, the report server will wait until the communication with primary AMDs is restored, even if other AMDs are available. Gaps in charts on some reports in multi-amd installations may be caused by unsynchronized AMDs. The reason for that may be that if the report server sees a data file for a specific time period on one of the AMDs, it will wait only 30 seconds for data files covering the same period of time from other AMDs. The 30 seconds are the server's tolerance for time synchronization issues. To verify that this situation occurred, compare the clock readings from AMDs and then check the time synchronization settings (see The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? [p. 57]). It may happen that a part of data will be missing. This will result in a significant decrease of the aggregated data, used to render the chart bars. Note that this effect relates to metrics that are calculated as sums, for example, number of operations, number of errors, number of users, or bandwidth utilization. Charts showing the averages (RTT, loss rate, operation time) will not be affected. I see gaps on the log-term data chart reports. Why are the charts incomplete? The report server aggregates the data collected during the day into daily (and monthly) rollups. This is a scheduled process. If this process is not triggered, you will see gaps in the daily rollups. The most frequent reasons for missing rollups are: 61

62 Appendix A Diagnostics and Troubleshooting The report server was down in the night; report data generation starts at 12:10 AM local time and if the report server was down at that time, no aggregate data for long-term reports will be generated. The report server was overloaded and it took too much time for other crucial tasks; report data generation for long-term reports was canceled. You can always re-generate data for long-term reports. Open Control Panel by typing: in the Address field of the web browser and click Go, then select Regenerate Reports from the System Management section. I created a report that consists of several charts but it loads very slowly. How can I improve its performance? If you are using exactly the same set of dimensions and filters for every chart but would like to show different metrics on separate charts, there are two ways of improving such a report. In this example, it is assumed that you want a report that shows Client bytes, Server bytes, and Total bytes on separate charts for the HTTP analyzer. First, the simplest and recommended method, is to define one section that contains all these three metrics. Figure 7. Creating One Section with Three Metrics Open the Chart settings panel and from the single chart per list select Metric. If you are using metrics with different units, you can select the Metric unit option instead. For more information, see Displaying Multiple Charts in the Data Center Real User Monitoring Data Mining Interface (DMI) User Guide. The second method requires changes on the Subject Data and Result Display tabs. 1. For each report section (chart), create the same set of metrics. To do this, for each chart add metrics that are displayed on the other charts. Note that the order of metrics must be the same in every section. For example, each section must contain the Client bytes, Server bytes, and Total bytes metrics listed exactly in the same order. 2. Disable showing unnecessary metrics for each chart. 62

63 Appendix A Diagnostics and Troubleshooting Go to the Result Display tab and disable showing the redundant metrics. For example, for chart that is going to show only the Client bytes metric, disable showing the Server bytes and Total bytes metrics. Figure 8. Selecting Metrics to Display on a Chart Application performance and availability data is missing from the tabular reports. How can I fix this? The missing data manifests itself as zero or a hyphen. The most frequent reason for this situation is the incorrect setting of business hours and holidays. Inspect the business hours and holiday settings by choosing Settings Report Settings Business Hours. The following configuration screen shows the current settings. Figure 9. Business Hours Configuration Screen To collect performance data seven days per week, including non-business days and holidays, clear the Holidays check box and select the check boxes for weekend days. In addition, you can collect performance data in 24/7 mode, but be aware that this results in a higher database growth rate and a larger database. To enable collecting data all the time, open the Control Panel by opening the following page: In the Control Panel, click Advanced Properties Editor from the Configuration Management section. Set ONLY_BUSS_HOUR_REPORTING to OFF. 63

64 Appendix A Diagnostics and Troubleshooting To see whether your holiday definition is correct, click View Holidays. Figure 10. Defined Holidays Screen The list of holidays is hard-coded and the default set is for the USA. To select a set, click the Choose holiday definition list. To see the content of the selected set, click Preview. To store the newly selected set, click Save. 64