Data Center Real User Monitoring

Transcription

1 Data Center Real User Monitoring Oracle Forms Application Monitoring User Guide Release 12.3

2 Please direct questions about Data Center Real User Monitoring or comments on this document to: Customer Support Copyright 2015 Compuware Corporation. All rights reserved. Unpublished rights reserved under the Copyright Laws of the United States. U.S. GOVERNMENT RIGHTS-Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Compuware Corporation license agreement and as provided in DFARS (a) and (a) (1995), DFARS (c)(1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. Compuware Corporation. This product contains confidential information and trade secrets of Compuware Corporation. Disclosure is prohibited without the prior express written permission of Compuware Corporation. Use of this product is subject to the terms and conditions of the user's License Agreement with Compuware Corporation. Documentation may only be reproduced by Licensee for internal use. The content of this document may not be altered, modified or changed without the express written consent of Compuware Corporation. Compuware Corporation may change the content specified herein at any time, with or without notice. All current Compuware Corporation product documentation can be found at Compuware, FrontLine, Network Monitoring, Enterprise Synthetic, Server Monitoring, Dynatrace Network Analyzer, Dynatrace, VantageView, Dynatrace, Real-User Monitoring First Mile, and Dynatrace Performance Network are trademarks or registered trademarks of Compuware Corporation. Cisco is a trademark or registered trademark of Cisco Systems, Inc. Internet Explorer, Outlook, SQL Server, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation. Firefox is a trademark or registered trademark of Mozilla Foundation. Red Hat and Red Hat Enterprise Linux are trademarks or registered trademarks of Red Hat, Inc. J2EE, Java, and JRE are trademarks or registered trademarks of Oracle Corporation. VMware is a trademark or registered trademark of VMware, Inc. SAP and SAP R/3 are trademarks or registered trademarks of SAP AG. Adobe Reader is a registered trademark of Adobe Systems Incorporated in the United States and/or other countries. All other company and product names are trademarks or registered trademarks of their respective owners. Local Build: April 1, 2015, 12:53

3 Contents Contents Introduction Who Should Read This Guide Organization of the Guide Related Publications Customer Support Information Reporting a Problem Documentation Conventions Chapter 1 Oracle Forms Monitoring Configuration Process Overview of Oracle Forms Monitoring Chapter 2 Adding Basic DC RUM Devices Adding an AMD to the Devices List Adding a CAS to Devices List Adding ADS to Devices List Chapter 3 Verification of Traffic Monitoring Quality Sniffing Point Diagnostics Sniffing Point Diagnostics Reports Network Interface General Statistics Network and Transport Protocol Information Services Detected in the Traffic Session-Related Statistics SSL Diagnostics Application Overview Using RUM Console to Identify Problems Related to Network Hardware Operation... Chapter 4 Basic Monitoring Configuration AMD General Configuration Settings Configuring Operation-Related Global Settings Chapter 5 Oracle Forms Monitoring Configuration Global Settings for Oracle Forms Monitoring Configuring Oracle Forms Triggers

4 Contents Configuring Oracle Forms Availability Defining Oracle Forms Software Services Configuring Rules for User-Defined Software Services Configuring Monitoring of an Oracle Forms Software Service Managing User-Defined Software Services Chapter 6 Monitoring Sequence Transactions Adding Oracle Forms Transactions Modifying, Deleting, and Cloning Transactions for a Single AMD Filters and Transaction Inspector for Oracle Forms Transactions Chapter 7 Oracle Forms Tiers Chapter 8 ADS Support for Oracle Forms Appendix A Diagnostics and Troubleshooting Report-Related Issues Appendix B Regular Expression Fundamentals Testing Regular Expressions Best Practices for Regular Expressions Appendix C Graphical Explanation of Network Performance Metrics Appendix D Oracle Forms Error Categories and Messages Forms Client Errors Forms Application Errors Forms Server Errors Oracle Server Errors Oracle Applications Errors Glossary Index

5 INTRODUCTION Who Should Read This Guide This manual is intended for users of Data Center Real User Monitoring who want to monitor the Oracle Forms application. Organization of the Guide This guide is organized as follows: Oracle Forms Monitoring [p. 9] Describes capabilities of Oracle Forms analyzers and provides an overview of the configuration process. Adding Basic DC RUM Devices [p. 15] Describes how to add and configure the data sources and report servers using the RUM Console. Verification of Traffic Monitoring Quality [p. 21] Describes how to verify sniffing points traffic detection quality before the actual monitoring begins. Basic Monitoring Configuration [p. 33] Describes AMD general settings. Oracle Forms Monitoring Configuration [p. 41] Describes the creation and management of Oracle Forms-based software services. Monitoring Sequence Transactions [p. 57] Describes how to monitor Oracle Forms transactions. Oracle Forms Tiers [p. 63] Describes tiers that present Oracle Forms data. Diagnostics and Troubleshooting [p. 73] Lists common CAS support issues in the form of questions and answers. Regular Expression Fundamentals [p. 83] Describes how to use regular expressions in CAS. Graphical Explanation of Network Performance Metrics [p. 89] Uses charts to illustrate network performance metrics. 5

6 Introduction Related Publications Documentation for your product is distributed on the product media. For Data Center RUM, it is located in the \Documentation directory. It can also be accessed from the Media Browser. Go online ( for fast access to information about your Dynatrace products. You can download documentation and FAQs as well as browse, ask questions and get answers on user forums (requires subscription). The first time you access FrontLine, you are required to register and obtain a password. Registration is free. PDF files can be viewed with Adobe Reader version 7 or later. If you do not have the Reader application installed, you can download the setup file from the Adobe Web site at Customer Support Information Dynatrace Community For product information, go to and click Support. You can review frequently asked questions, access the training resources in the APM University, and post a question or comment to the product forums. You must register and log in to access the Community. Corporate Website To access the corporate website, go to The Dynatrace site provides a variety of product and support information. Reporting a Problem Use these guidelines when contacting APM Customer Support. When submitting a problem, log on to the Dynatrace Support Portal at click the Open Ticket button and select Data Center Real User Monitoring from the Product list. Refer to the DC RUM FAQ article at to learn know how to provide accurate diagnostics data for your DC RUM components. Most of the required data can be retrieved using RUM Console. Documentation Conventions The following font conventions are used throughout documentation: This font Bold Citation Indicates Terms, commands, and references to names of screen controls and user interface elements. Emphasized text, inline citations, titles of external books or articles. 6

7 Introduction This font Documentation Conventions [p. 6] Fixed width Fixed width bold Fixed width italic Menu Item Screen Code block Indicates Links to Internet resources and linked references to titles in documentation. Cited contents of text files, inline examples of code, command line inputs or system outputs. Also file and path names. User input in console commands. Place holders for values of strings, for example as in the command: cd directory_name Menu items. Text screen shots. Blocks of code or fragments of text files. 7

8 Introduction 8

9 CHAPTER 1 Oracle Forms Monitoring Monitoring of Oracle Forms operations and users enables Oracle Forms administrators to diagnose user problems and infrastructure issues related to Oracle Forms traffic. The presence of an Oracle Application Server on your network enables you to use Oracle Forms monitoring on the transaction level to help solve end-user experience issues. Use the AMD to decode the Oracle proprietary protocol and create dedicated software service definitions to achieve complete end-to-end instrumentation of the Oracle infrastructure from the client to the application server and down to the database server. The major Oracle Forms monitoring features in Data Center Real User Monitoring include: Monitoring Forms operations The Oracle Forms analyzer tracks Oracle Forms user activities and monitors single operations performed by each user. Forms operations are classified by Form name and user interface events such as a button press. You are able to determine which Form or user experienced delays or errors, and determine whether it was due to the Forms server or network conditions. Forms user recognition To provide Forms measurements in the context of named users, the user logon process to the Forms server (the HTTP-based logon, prior to Forms UI launch) is tracked. The user name extracted from the Forms HTTP server session (the JSession) or from Oracle Forms application internal login is reported with measurements of every subsequent Forms operation that particular user (the owner of the session) performs. User name tracking ensures that Oracle Forms can be monitored for users individually even when users are located behind a proxy server. Key measurements taken for Oracle Forms traffic include: Form name recognition. The AMD identifies, by name, operations that pertain to user actions such as button presses or clicking items in a list. Other operations are not identified by name. The number of Form submissions per named form. A submission is defined as an occurrence of one of several Forms events: ButtonItem 9

10 Chapter 1 Oracle Forms Monitoring IconicButtonItem TextFieldItem pressed LogonDialog action AlertDialog ListValuesDialog EditorDialog Server response time for operation request. Network parameters. TCP-level availability of Forms servers, including errors (for example, no response or connection refusal). User differentiation (based on the client's IP address, login name, Form's session cookie, or directly from Oracle Forms binary stream data). You can use one or two AMDs to listen both in front of the Oracle HTTP servers and in front of the Oracle database servers. Two separate AMDs should be used in configurations for large installations where AMDs are set up to monitor not only Oracle Forms, but also other enterprise applications and the whole enterprise network. Note that to analyze Oracle database traffic you must configure database monitoring options. For more information, see Database Monitoring in the Data Center Real User Monitoring Database Monitoring User Guide. When monitoring Oracle Forms operations and users in a load balancer setup, you have the choice of monitoring the traffic either in front of or behind a load balancer or of using two AMDs on both links. Because the Oracle Forms decoder cannot handle duplicate packets of the same jsessionid, it is impossible for one AMD to monitor traffic simultaneously in front of and behind a load balancer. Oracle Forms Support on the AMD Table 1. Analyzers and Supported Versions Analyzer Protocol Version Comments and example application Oracle Forms over HTTP Oracle Forms 6i, 9i, 10.1, 11g Oracle Forms 6i Oracle Forms over HTTPS Oracle Application Server 9i, 10i, Oracle Forms over TCP 10g R2, 11g Oracle Forms over SSL SSL support. Configuration Process Overview of Oracle Forms Monitoring DC RUM provides you with several ways of monitoring Oracle Forms traffic. You can monitor software services, transactions, tiers, applications, and your network. 10

11 Chapter 1 Oracle Forms Monitoring Before You Begin Before you start the configuration process: You should be familiar with DC RUM components and basic monitoring concepts. Refer to the Data Center Real User Monitoring Getting Started. You need to identify your monitoring goals. For more information, see Define and Prioritize Goals, Objectives, and Requirements in the Data Center Real User Monitoring Getting Started. You need to install the following DC RUM components: The latest version of AMD Refer to the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. The latest version of RUM Console Refer to the Data Center Real User Monitoring RUM Console Installation Guide. The latest version of CAS Refer to the Data Center Real User Monitoring Central Analysis Server Installation Guide. The latest version of ADS Refer to the Data Center Real User Monitoring Advanced Diagnostics Server Installation Guide. Make sure that default ports are available for communications between the individual DC RUM components. For more information, see Network Ports Opened for DC RUM in the Data Center Real User Monitoring Administration Guide. The following steps must be executed in order to begin monitoring the traffic using the DC RUM suite: Configuring Devices 1. Add Agentless Monitoring Device (AMD) AMD is the main data source (Data Collector) for DC RUM; it collects and presents the monitored data to DC RUM report servers for analysis and reporting. You need to add at least one AMD to the list of devices in RUM Console. For more information, see Adding an AMD to the Devices List [p. 15]. 2. Add Central Analysis Server (CAS) CAS is the main report server for DC RUM. It uses data provided by the AMD and its monitoring and alerting mechanisms to identify, track, and report on issues affecting the security, performance, and reliability of your services. Add at least one CAS to the device list and configure its connection with the AMD. Adding a report server to a list of devices is similar to adding the AMD. For more information, see Adding a CAS to Devices List [p. 17]. 11

12 Chapter 1 Oracle Forms Monitoring 3. Adding Advanced Diagnostics Server (ADS) ADS is a report server that performs detailed analysis of key transactional application protocols. It uses data provided by the AMD. Add at least one ADS to the device list and configure its connection with the AMD. For more information, see Adding ADS to Devices List [p. 18]. 4. Verify the traffic monitoring quality and completeness You can verify traffic quality and completeness before the actual monitoring begins. Sniffing point diagnostics allows you to perform pre-monitoring tasks without the need of accessing the AMD console and executing a series of Linux commands which usually serve the purpose of validating AMD physical installation and connection. For more information, see Verification of Traffic Monitoring Quality [p. 21]. Configuring Basic Monitoring 5. Configure general settings for your AMD Before you proceed to detailed monitoring rules, you need to define the global settings that are applied to all software services monitored by a given AMD. These global settings include a monitoring interval and thresholds for the basic metrics. These settings can be overridden at a later time with more specific monitoring rules that you can define. For more information, see Configuring General Data Collector Settings in the Data Center Real User Monitoring Web Application Monitoring User Guide and Configuring Operation-Related Global Settings [p. 37]. 6. Configuring global monitoring settings for Oracle Forms. Global settings are settings that affect monitoring of all services based on Oracle Forms analyzers for a given AMD. It is crucial to configure these settings before defining specific Oracle Forms monitoring rules. For more information, see Global Settings for Oracle Forms Monitoring [p. 41]. Customizing Monitoring Rules 7. Define your own software services on specified ports and for specified IP addresses If you configure user-defined software services, you will be able to see Oracle Forms data on your reports. For more information on software service configuration, see Oracle Forms Monitoring Configuration [p. 41] For more information on reports, please refer to Central Analysis Server - User Guide. 8. Display the reports to review statistics for monitored traffic Fine-Tuning Reporting Configuration 9. Configure the sites, areas, and regions A site is a term for a group of users that are located in the same IP network or group of networks sharing similar routing properties. Sites can be grouped together into areas, which, in turn, can be grouped together into regions. The hierarchy of sites, areas, and regions provides an organized view of the monitored network on the reports. 12

13 Chapter 1 Oracle Forms Monitoring For more information, see Configuring Sites, Areas, and Regions in the Data Center Real User Monitoring Administration Guide. 10. Configure the tiers A tier is a specific point where DC RUM collects performance data. You can have monitoring data reported based on the default tier configuration, or you can define tiers that fit your network architecture. For more information, see Oracle Forms Tiers [p. 63]. 11. Configure the transactions, applications, and reporting groups Transactions are sequences of information exchange that represent particular actions or functions performed by a human user or a client program. They are viewed as higher-level units of self-contained functionality and are tied to applications. For example, they may represent the procedure for an online purchase or ticket booking. AMD monitors traffic data and prepares it for transaction monitoring by an ADS and CAS. Some of the relevant configuration and processing is performed on the actual RUM Console and some is performed on the AMD. For more information, see Managing Business Units in the Data Center Real User Monitoring Administration Guide. 12. Configure the monitoring of sequence transactions DC RUM enables you to define and monitor transactions that are sequences of steps. For more information, see Adding Oracle Forms Transactions [p. 58]. Troubleshooting 13. Troubleshoot problems You can review the answers to the most common questions and troubleshoot your setup and report configurations. For more information, see Diagnostics and Troubleshooting [p. 73]. 13

14 Chapter 1 Oracle Forms Monitoring 14

15 CHAPTER 2 Adding Basic DC RUM Devices In a DC RUM configuration, there are two device types: data collectors and report servers. To start using the product, add and configure at least one AMD data collector and one CAS report server. You manage these devices using a configuration tool called the RUM Console. Adding an AMD to the Devices List Before you can monitor traffic with DC RUM, you have to add and configure an Agentless Monitoring Device using the RUM Console. To add an AMD to the list: Adding an AMD 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type list, select AMD. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Information 5. In the Device IP address box, type the device IP address. 6. In the Port number box, type the port number for communication with this device. The standard port number used by AMD is Optional: Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. 15

16 Chapter 2 Adding Basic DC RUM Devices Providing the Authentication Details 8. Type the user name and password of the account that will be used for managing this device. By default, the AMD user is set to compuware and the password is set to vantage. The credentials entered here are used by the RUM Configuration to communicate with the device and are also passed to the report servers so that they can collect monitoring data for processing. Note that the values used here for authentication are not the same as the values you use for logging in to the device via SSH or local console. Configuring Advanced Settings 9. Select the Advanced options tab. 10. Optional: Under Secondary device connection, provide an alternative IP address for this device. 11. Optional: Enable SNMP connection. Optionally, you can define the SNMP connection parameters so that you can obtain more detailed health information about the device. To define SNMP connection parameters: a. Select SNMP Connection check box. b. Type the read community name and port number. 12. Enable Guided Configuration. By default, the Guided Configuration connection is enabled when you add an AMD. However, for performance reasons, the number of AMDs with enabled Guided Configuration is limited to 50. Any additional AMDs do not feed data to the Guided Configuration perspective. This means that the monitoring data from the additional AMDs is not available for generating the web traffic statistics or defining the web software services with a wizard. By default, the port number for communication between the Console Basic Analyzer Agent and the RUM Console Server is set to 9094 and the secure connection is enabled. In most cases, it is not necessary to modify this setting. If the default port number is already in use by other services, however, type the new port number in the Port number box. In this case, you also have to manually change the port number setting on the Console Basic Analyzer Agent side. For more information, see Modifying Connection Settings for Guided Configuration in the Data Center Real User Monitoring Administration Guide. 13. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 14. Click Finish to save the configuration. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. 16

17 Chapter 2 Adding Basic DC RUM Devices Adding a CAS to Devices List To view reports based on the data from the AMD, use the RUM Console to add and configure a CAS report server. Adding a CAS 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type menu, select CAS. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Details 5. In the Device IP address box, type the device IP address. 6. In the Port box, type the port number for communicating with this device. The standard port number used by the CAS when communicating over HTTP is Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. Providing the Authentication Details 8. Choose whether authentication should occur via CSS. 9. Type the user name and password of the account that will be used for managing this device. Configuring the Advanced Settings 10. Select the Advanced options tab. 11. Optional: Under Secondary device connection, provide an alternative IP address for this device. 12. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Click Finish to save the configuration. Configuring the CAS-AMD Connection 14. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 15. Select a report server from the list of devices. Click the server once to display the detailed information for the device. 16. Select the Data Sources tab. 17

18 Chapter 2 Adding Basic DC RUM Devices 17. Click Add Data Source. 18. Select your AMD from the list and then click the button. 19. Click Finish to save the configuration. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. What to Do Next It is important to keep the devices synchronized to avoid improper data interpretation. For more information, see Synchronizing Time Using the NTP Server in the Data Center Real User Monitoring Smart Packet Capture User Guide and Time Synchronization Between AMD and Server in the Data Center Real User Monitoring Administration Guide. Adding ADS to Devices List To view reports based on data from the AMD, use the RUM Console to add and configure at least one CAS report server. In addition, you can add one or more ADS report servers in a farm configuration. Adding an ADS 1. Start and log on to RUM Console. 2. Select Devices and Connections Add device from the top menu. The Add Device pop-up window appears. 3. From the Device type menu, select ADS. 4. In the Description box, type a description of the device. TIP It is recommended that you include the parent device name in the description of each device you add and to add these names consistently. This enables you to easily find your device in the list. Specifying the Connection Details 5. In the Device IP address box, type the device IP address. 6. In the Port number box, type the port number for communication with this device. The standard port number used by ADS when communicating over HTTP is Optional: Select Use secure connection if you want to use HTTPS (secure HTTP) for communication between the console and the device you are adding. Providing the Authentication Details 8. Choose whether authentication should occur via CSS. 18

19 Chapter 2 Adding Basic DC RUM Devices 9. Type the user name and password of the account that will be used for managing this device. Configuring the Advanced Settings 10. Select the Advanced options tab. 11. Optional: Under Secondary device connection, provide an alternative IP address for this device. 12. Click Next to test your connection parameters. If your configuration fails the test, you can go back and adjust your settings. Note that if the device fails to respond correctly, it may take several seconds before the test times out. 13. Click Finish to save the configuration. Configuring the ADS-AMD Connection 14. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 15. Select the ADS from the list of devices. Click in the row corresponding with your server to display details for the device. 16. Switch to the Data Sources tab. 17. Click Add Data Source. 18. Select your AMD from the list and then click the button. 19. Click Finish to save the configuration. 20. Configure the ADS and CAS to work together. As a result, your device appears on the Devices list. To view the list, go to Devices and Connections Manage Devices in the top menu of the RUM Console. The Devices screen presents a comprehensive view of all the devices that you add, including their IP Address, Port, Description, Type, Version, Connection, Hardware Health, and Configuration. 19

20 Chapter 2 Adding Basic DC RUM Devices 20

21 CHAPTER 3 Verification of Traffic Monitoring Quality Use the RUM Console to verify the traffic monitoring quality using two tightly connected solutions: Sniffing Point Diagnostics and Application Overview. We highly recommend that you perform this step at the beginning of your DC RUM deployment to verify that your hardware is working properly and that the applications you intend to monitor are detected. You can verify the test results and repeat them as needed at any time and for any network conditions. IMPORTANT All verification is based on a traffic recording, either manual or automatic. The outcome may not be representative if the target traffic is low at the time of recording or if you are unable to capture a satisfactory number of complete sessions. Choose automatic or manual traffic recording to capture unfiltered or filtered traffic. Enable automatic recording only during the configuration process and then disable it. It can negatively affect the performance of the AMD during normal operations, especially if you are running a 32-bit AMD in a high-traffic environment or a 64-bit AMD with the native driver. For the most complete and reliable statistics, use the 64-bit customized driver on the AMD. The verification of traffic monitoring quality is possible only for AMD 11.7 or later. Sniffing Point Diagnostics Sniffing Point Diagnostics is a type of hardware state analysis that enables you to perform pre-monitoring tasks without the need to access the AMD terminal. You can use it to validate the operation of the sniffing points, instead of using a series of UNIX or rcon commands. This step can be performed at the DC RUM deployment stage or at any other time to determine if the AMD performance is affected by malfunctioning hardware or external networking conditions. The Sniffing Point Diagnostics analysis can detect issues, such as: No traffic detected on sniffing interfaces. Interface or link overload. 21

22 Chapter 3 Verification of Traffic Monitoring Quality Poor quality of traffic due to mirrored ports on switching hardware configuration. Dropped packets (indicates AMD overload). Network conditions when unidirectional traffic prevails. Rejected packets, invalid packets, wrong check sums for packets. Missing packets (either lost or dropped). Missing bytes (how much traffic is lost in general). Conditions affecting AMD calculations, such as: Duplicate traffic that cannot be handled by the AMD. Incorrect choice of packet deduplication method. Incorrect settings for packet deduplication buffer. Incorrect settings for maximum packet size or huge packet size. Conditions affecting AMD performance, such as: Duplicate traffic handled by the AMD. Large percentage of non-ip traffic (noise). Large percentage of non-tcp or non-udp traffic (noise). Reordered sessions. Miscellaneous SSL problems: Unsuccessful decryption (in general). Uninitialized SSL cards unable to decrypt traffic. The ratio of encrypted and successfully decrypted traffic to encrypted and non-decrypted traffic. Incorrect or missing private keys. No match between the key and server certificate. Dropped or corrupted packets preventing decryption. Unsupported cipher methods (for example, Diffie-Hellman based key infrastructure). Unsupported SSL versions or features. Prerequisites and Best Practices To diagnose application detection problems and sniffing point connection problems, ensure that: All cables are connected correctly. The AMD is properly installed and configured. This includes the post-installation steps, such as interface identification and network configuration. Traffic recording lasts long enough to capture a reasonable amount of traffic volume, for example, 20 to 30 minutes of traffic. 22

23 Chapter 3 Verification of Traffic Monitoring Quality Do not use specific capture profiles when recording traffic. Always use the All available option for capture profiles when you do manual recording. When you need to diagnose traffic or capture port problems, enable automatic trace recording. Trace recording provides access to regular and fresh snapshots of the traffic that is traveling on your network. Sniffing Point Diagnostics Reports Sniffing Point Diagnostics reports are organized into several sections, each representing a separate set of metrics related to either hardware or network traffic. This topic provides directions for viewing the reports, but you can follow each step or skip steps to view the only the information important to you. 1. Start either by looking at device health or from the reports section directly. If you enabled automatic trace recording, you can monitor the device state on the Device Status tab of the Devices screen. A separate set of statistics is provided for each AMD added to the console. If there are any alarm messages, go to Devices and Connections Verify quality of monitored traffic. Inspect network interfaces in detail for a selected AMD. Open the Overview report to verify that the proper type of network driver is being used (custom or native) and that traffic has been detected, and check the number of dropped packets and other performance related issues. You can also verify that the NIC drivers are operational. For more information, see Network Interface General Statistics [p. 23]. 2. Switch to the Protocols section to inspect protocols. See whether network protocols are detected (IPv4 or IPv6) and verify detection of transport protocols (TCP or UDP). For more information, see Network and Transport Protocol Information [p. 26]. 3. Switch to the Services section to see the most active services. For more information, see Services Detected in the Traffic [p. 26]. 4. Depending on your goals, switch to the Sessions section either by selecting a particular service on the Services report to see session details or by choosing the Sessions section to see general statistics for all sessions. For more information, see Session-Related Statistics [p. 26]. 5. If you use SSL decryption, you can inspect whether there are problems detected for the currently used SSL engine or keys. For more information, see SSL Diagnostics [p. 28]. Network Interface General Statistics The Overview section of the Sniffing Points Diagnostics reports enables you to verify the general state of capture ports on a selected AMD. The information in the Overview section is gathered directly from the NIC driver operating on the AMD. For the most reliable results, use the 64-bit customized drivers. 23

24 Chapter 3 Verification of Traffic Monitoring Quality Calculation of Analyzed Traffic The calculation of analyzed traffic is performed in several stages, gradually excluding the irrelevant statistics: 1. The overruns are excluded first. When the received packets are counted, the overruns are omitted. 2. The calculation of the received packets depends on the subtraction of errors and filtered-out packets. 3. The dropped packets are counted after the filtered packets are disregarded. 4. The number of analyzed packets is the count of packets remaining after all of the previous categories are subtracted. In default AMD installations, non-tcp/udp packets are not part of this process and are never counted when the number of analyzed packets is given. Non-TCP/UDP traffic increases the amount of analyzed traffic only if you enable the monitoring of the default software services. Figure 1. Graphical Explanation of Analyzed Traffic Calculation for an AMD with 64-bit Customized Network Interface Driver All network packets Overruns Packets not received Received packets Errors and non-conditional filtering Errors: length or bad checksum; filtered out: non-ip Load balancing If active, fraction of the traffic Configuration filtering Based on defined software services Sampling and dropped packets Packets not analyzed due to performance issues Non-TCP, non-udp If default software services enabled Analyzed packets 24

25 Chapter 3 Verification of Traffic Monitoring Quality Interface Operation-Related Metrics The statistics presented on this screen include: Overruns Overruns may indicate a link overload. The overload is typically caused by an exceptionally high traffic volume. This value may also indicate that the network interface or network interface driver cannot manage the amount of traffic received. Other hardware-related issues may also cause overruns. If a high overrun occurs, limit the traffic volume received by the card. Errors (length) Packets of erroneous length are reported when they are too big (such as jumbo frames) or are bigger than the maximum transmission unit (MTU). To avoid such problems, you can increase Maximum packet size in the Entire Configuration perspective. For more information, see Configuring General Data Collector Settings in the Data Center Real User Monitoring Web Application Monitoring User Guide. Errors (bad checksum) Checksum-related errors are typically caused by insufficient signal strength on an optical link. In other cases, checksum errors may indicate Ethernet distortion, such as duplex problems, where the checksum errors may result, for example, when the duplex auto-negotiation process fails. Check the host switch and AMD duplex settings. Filtered out (non-ip) Non-IP packets, such as ARP traffic. Even large numbers of such packets are generally considered harmless. They are not analyzed by the AMD software and are regarded as noise. Preventing such traffic from reaching the AMD may reduce the possibility of performance degradation. Filtered out (load balancing) This setting is only applicable in deployments with multiple AMDs where each device only analyzes a certain part of the same traffic. Filtered out (configuration) Provides additional filtering based on software service definitions. In default installations, where monitoring of the default software services is turned off, the driver limits the number of processed packets to only those that are relevant to the IP addresses included in user-defined software service definitions. Dropped (sampling) Sampling here means dropping packets when the driver performance is degraded. Packets are dropped in a controlled manner, and always with care, to preserve complete and consistent sessions. The packet drops almost always mean that traffic is too heavy for a complete analysis and that, with packet drops, the precision of CAS reports is diminished. Sampling is only active with the customized 64-bit driver and diagnostics always use this sampling mechanism regardless of the settings used in the general AMD configuration. Dropped (driver performance) Drops are always a symptom of problems, especially when SSL analysis is deployed. Drops occur when AMD software is unable to analyze all of the packets it receives from the driver. If you use 32-bit or native drivers, you may experience uncontrolled packet 25

26 Chapter 3 Verification of Traffic Monitoring Quality dropping. If you use the 64-bit customized driver, packet dropping may occur, but in a software-controlled manner with care for monitored data contingency. To avoid packet dropping, decrease the traffic volume that your AMD analyzes or reduce the number of monitored software services. Non TCP/UDP Whether these statistics are classified as analyzed or not depends on the default software service monitoring. The numbers in this section are mostly relevant if you enabled monitoring of default software services. In this case, ICMP traffic is also analyzed. If monitoring of the default software services is disabled and you still see a large percentage of non-tcp and non-udp traffic, it is possible that AMD performance will be affected. Network and Transport Protocol Information Use the Protocols report to check the ratio of supported transport or network protocols. Only supported protocols are shown. In general, this report enables you to check whether traffic that makes sense (from the DC RUM perspective) is present and is heavy enough to give meaningful results for report servers. NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. Problem Detection Low traffic for the IPv4 or IPv6 network protocols may indicate further monitoring problems. The presence solely of multicast or broadcast traffic is an indication that port mirroring is not enabled or inactive. Services Detected in the Traffic This overview report enables you to identify the most active services on your network. You can see what their load is and what protocols they use, and filter the results to display all data, monitored services, or unmonitored services. You can also use filters to display statistics for all, monitored, or unmonitored services with additional protocol filtering. For each service, you can open the Sessions report to verify session-level statistics. NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. Session-Related Statistics The Sessions section enables you to view detailed information about traffic quality. 26

27 Chapter 3 Verification of Traffic Monitoring Quality The statistics presented on this screen include: Duplicates, Unhandled duplicates The value presented on the Sessions screen depends on the currently selected deduplication method in your AMD configuration. Packet duplicates may indicate incorrect configuration of mirroring ports. While this may be a sign of a problem, values of 10 to 20 percent typically are no reason for concern. The AMD is capable of packet deduplication. Higher numbers of duplicate packets will degrade the AMD performance and may negatively influence the monitoring results. The diagnostics mechanism for duplicate detection and counting for this report works with different settings than the network monitoring processes on the AMD. Duplicate detection is performed using both methods of duplicate detection and with different settings (buffer and delay detection size). Based on these settings and calculations, Sniffing Point Diagnostics provides suggestions concerning duplicate handling, such as increasing buffer size or changing the deduplication mechanism. You should check whether there are unhandled duplicates detected, in which case it is suggested that you switch the detection method in the AMD general settings. For more information, see Configuring General Data Collector Settings in the Data Center Real User Monitoring Web Application Monitoring User Guide. Unidirectional TCP sessions and UDP streams This may indicate a problem related to incorrectly configured mirroring ports. If the value of unidirectional traffic exceeds 90 percent, the RUM Console always marks it as an error. The numbers on the Sessions screen are the sums of many measurements; you are able to go deeper and analyze details for each server and check whether this is a problem related to a significant service or protocol. Insignificant traffic may be recorded and included in the general analysis, so always check the detailed reports when you see alarming numbers on the Sessions report. TCP sessions with missing packets Missing packets may result from interface or driver packet drops. If a session with missing packets is shown, the percentage value is counted with regard to all sessions. For example, if two percent of sessions have missing packets reported, this means that two out of a hundred sessions have missing packets. TCP sessions with missing packets and TCP bytes lost in missed packets may provide valuable insight into SSL decryption problems, especially in the case of long SSL sessions. TCP bytes lost in missing packets This is a complementary value to the TCP sessions with missing packets. Verify the number of lost bytes with regard to missing packets to see whether the problem is serious (if there are large sums of missing bytes). This is useful additional information in the case of long TCP sessions; because one lost packet is enough to classify a session as having missing packets, the number here gives insight into the actual loss rate. TCP sessions with reordered packets Reordered packets are typically found when there is a WAN link enabled. Devices transferring WAN packets may affect the packet order. The existence of reordered packets is not a problem in itself, because the AMD software can restore original packet order, but an excessive number of such packets may cause performance degradation. 27

28 Chapter 3 Verification of Traffic Monitoring Quality NOTE To obtain the most reliable results, use 64-bit customized drivers. The limited approximation algorithms used by native and 32-bit customized network interface drivers may lead to differences between the packet count in this and the Overview sections. SSL Diagnostics The traffic for this report is dependent on capturing complete sessions. Incomplete sessions, missing packets, or missed handshakes cause a large number of errors and a large number of errors results in unreliable reports. Always be sure to record enough traffic for an adequate length of time to allow you to capture complete sessions. The Statistics for encrypted traffic, SSL card and keys report is only available after the traffic trace recording is finished. Partial statistics for SSL are not provided for unfinished sessions. General Statistics for Encrypted Traffic For a given time range, defined by the scope of the recorded traffic traces, you can see the recognized SSL engine (for example, OpenSSL or ncipher) and the number of keys exchanged in the traffic. The remaining sections of this diagnostic report show the detailed information about the keys, the overall summary of the captured SSL traffic, and whether there are errors. The servers section shows information for all SSL traffic captured during the traffic trace recording. All of the detected encrypted protocols are listed together with their matching keys, if they are seen in the traffic. You can see whether the key exchange was successful; the matched keys are indicated by the icon. Key and certificate matching enables you to verify that certificates were found and were valid. No matching may indicate that the certificates are out of date. SSL Server Status The Status column shows whether there are errors or whether erroneous sessions prevail. A traffic capture sometimes does not contain session beginnings, or it contains incomplete handshakes, or it has no master session; these sessions are marked as ignored, as indicated by the gray ( ) color bar. The sessions with errors are marked by a red ( ) color bar. The main causes of errors are missing packets or missing keys. Other causes of errors are listed in detail on the Detailed SSL Statistics for servers report. Detailed SSL Statistics for Servers Detailed SSL statistics for servers are accessed from the Server or Status columns. This report shows: The percentage of the sessions without error, with errors, or ignored. The counts of each problem, in detail, for the error or ignored sessions. The number of decrypted sessions if there are no problems. 28

29 Chapter 3 Verification of Traffic Monitoring Quality You can filter the results. Use Sessions finished to display the data for completed sessions. Use Sessions in progress to display the sessions that are still in progress (sessions that did not end before the traffic capture stopped; to see those session statistics). Figure 2. Example of Detailed SSL Statistics for Server, Errors Detected Due to Private Key Mismatch SSL Keys Because invalid or outdated keys are usually not removed from SSL cards, the list of keys for which an error status is indicated may be considerably long. In such cases, sort by the Status column to see keys correctly matched. Note that it may be necessary to format the SSL card storage area to refresh the key list. Application Overview The Application Overview screen enables you to answer several questions about your applications at the onset of your monitoring configuration. Are all my applications or servers detected? What applications or servers are detected? Can the detected applications or servers be successfully monitored? How heavy is the traffic for each application or server? What services are detected on each server? How heavy is the traffic for each detected service? Note that incomplete sessions are not analyzed. If no beginning is recorded for a session, that session is not analyzed. 29

30 Chapter 3 Verification of Traffic Monitoring Quality The Application Overview screen is an optional step towards defining new software services. To access it, select Software Services Add Software Service in the console top menu, then select By traffic lookup. Figure 3. Example of the Application Overview Screen Showing Detected Applications From this screen, you can configure software services either manually or by using the wizard. If it is possible to go through a step-by-step configuration, a wizard icon ( ) is displayed for the given protocol or service. Application Detection Mechanism Application detection is a three-stage process: 1. To provide the most accurate results, packet analysis for SSL, HTTP, HTTPS, SOAP, and related protocols is performed as a first step toward application type detection. Application recognition is based on the first matching pattern found. This means that some services may not be properly classified if multiple protocols are used in one session. For example, if your application uses HTTP and SOAP over HTTP protocols, and plain HTTP communication opens a session, the application is classified as HTTP. 2. Applications are also detected based on discovery of well-known ports. The default protocol definitions are stored on the AMD and can be exported from the RUM Console. For more information, see Exporting the AMD Configuration in the Data Center Real User Monitoring Administration Guide. At times applications may use ports commonly used for other purposes. The AMD is unaware of these circumstances and will report well-known protocol names. For example, if one of your web applications uses port 8080 and uses HTTP for communication, the AMD reports this as an HTTP proxy. 3. If none of the selected conditions matches, the application is labeled as Unknown TCP or Unknown UDP. 30

31 Chapter 3 Verification of Traffic Monitoring Quality Server recognition in application detection is based on heuristic session analysis; results may vary depending on the type of network interface driver used. Using RUM Console to Identify Problems Related to Network Hardware Operation Typical configuration errors related to port mirroring can, at times, severely affect the AMD software traffic analysis capabilities. Faulty hardware configuration may result in no data seen by the AMD, a large number of duplicate packets reaching the AMD, or only a limited portion of traffic visible to the monitoring software. Use the Application Overview and Sniffing Point Diagnostics sections as tools to solve issues related to the switching hardware configuration. The following list describes several common problems and some possible causes and solutions. No data seen by the AMD The cable is connected to the wrong physical port on the destination switch. This can be checked by physically tracing the cable directly to the switch and confirming the port ID. The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror incorrect ports or an incorrect destination. This can be resolved by logging on to the source switch and checking the mirroring ports configuration relevant to the requirements (see the vendor-specific documentation for details). No data seen on Application Overview but non-tcp/udp traffic seen in interface statistics The port mirroring configuration (for example, SPAN on Cisco hardware) has been set or changed to mirror incorrect ports or an incorrect destination. This can be resolved by logging on to the source switch and checking the mirroring ports configuration relevant to the requirements (see the vendor-specific documentation for details). Application Overview does not show all expected data The port mirroring destination may be oversubscribed or dropping packets. Check this by logging on to the switch and checking the SPAN or mirror destination interface. If it is recording many drops, review the configuration of source ports to understand the ratio of source interface bandwidth to destination interface bandwidth. If the ratio is excessive (for example, greater than 4:1), consider reducing the number of source interfaces. If applicable, consider using device-specific filtering to reduce the load on the destination interface (for example, VACL, Rx-only, or Tx-only sources). By design, port mirroring does not forward faulty frames. Check the source device interface statistics to ascertain the nature of the drops (see the vendor-specific documentation for details). Check the interface-related metrics. If there is a high rate of Errors (bad checksum), consider hard-configuring one end of the AMD SPAN connection to prevent auto negotiation. Session-related report shows a high rate of packet duplicates A SPAN or mirror operates by copying frames from source interfaces and directing them to the destination interface. In effect, configurations often result in two copies of a packet. 31

32 Chapter 3 Verification of Traffic Monitoring Quality For example, if the source of a SPAN or mirror is set as a VLAN, any traffic that goes from one switch port to another switch port within the VLAN appears twice on the mirrored port. If the number of duplicates starts to affect AMD performance, consider reducing the number of source interfaces. If applicable, consider using a device-specific filtering control to reduce packet duplication (for example, VACL, receive-only, or transmit-only sources) or consider using tap technology as opposed to port mirroring to collect the data. Only unidirectional streams are seen on session-related overview If the AMD is connected via a SPAN or mirror, the configuration has been set incorrectly to send only one side of a receive or transmit stream to the destination. Log on to the local source switch to check the configuration (see the vendor-specific documentation for details). 32

33 CHAPTER 4 Basic Monitoring Configuration You can define many configuration settings globally for all software services for a given protocol and Data Collector, or locally for specific user-defined software services. If you specify both types of settings, the settings for a user-defined software service take precedence over the corresponding global settings. Monitoring Software Services There are no default software services defined for Oracle Forms analyzers. To monitor Oracle Forms traffic, you have to define your own software services on specified ports and for specified IP addresses. To access global settings for a Data Collector, go to Global Front-End Monitoring Oracle Forms in the navigation tree of the RUM Console. To access settings for specific software services, you need to select and open the Rules configuration for that software service. For details on setting up and configuring monitoring for Oracle Forms protocol, see Oracle Forms Monitoring Configuration [p. 41]. Monitoring Transactions Transactions are sequences of information exchange that represent particular actions or functions performed by a human user or a client program. They are viewed as higher-level units of self-contained functionality and are tied to applications. For example, they may represent the procedure for an online purchase or ticket booking. AMD monitors traffic data and prepares it for transaction monitoring by an ADS and CAS. Some of the relevant configuration and processing is performed on the actual RUM Console and some is performed on the AMD. See Monitoring Sequence Transactions [p. 57] for information on how to configure and use transaction monitoring on the AMD. AMD General Configuration Settings For any given AMD, you can set a variety of options such as time thresholds. The general settings affect monitoring of software services, but they can be overridden by specific settings for a particular analyzer or software service. 33

34 Chapter 4 Basic Monitoring Configuration To define the general settings for an AMD: 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 4. Click Edit as Draft to set your configuration to draft mode (if you are not in draft mode already). 5. Select Configuration Global General to access the list of general configuration settings. While some of the options control only general AMD behavior, some options in the Advanced group affect more specific configurations in application monitoring. For example, if Inherit from global settings is selected in your other configurations while configuring user-defined software services, the global setting takes precedence over the specific monitoring configuration. Configuration options include: Monitoring interval The monitoring interval in minutes. Increasing this value reduces the number of chunks of data that need to be transferred and processed. Default: 5 minutes. Operation time threshold The number of seconds after which an operation is considered to be slow. The global threshold value depends on the analyzer. This threshold is used by the following analyzers: Cerner Cerner over MQ Epic Generic with transactions HTTP MS Exchange over HTTP MS Exchange over HTTPS Oracle Applications over HTTP Oracle Applications over HTTPS SAP GUI SAP RFC SAP GUI over HTTP SAP GUI over HTTPS SMTP SSL SSL Decrypted 34

35 Chapter 4 Basic Monitoring Configuration Server time threshold The Server time threshold relates to the server time portion of an overall operation time. Server times above the threshold limit are considered to be slow due to poor datacenter performance. This threshold is used by the following analyzers: HTTP SAP GUI over HTTP SAP GUI over HTTPS IP address of the server authorized to set AMD time The IP address of the report server that has the authority to synchronize the time with this AMD. In an environment with a number of servers sharing the same AMD, it is good practice to designate only one of these servers as a time synchronization server to make changes to AMD settings. Otherwise, the server used for time synchronization will change inadvertently every time you save an AMD configuration. Default analyzer The default setting for the TCP analyzer is Generic (with transactions). To change it, select another analyzer from the list. Client RST packet timeout to mark session as CLOSED If the time between the last ACK for data sent by the server and an RST packet sent by the client is greater than this value, the session is treated as closed instead of aborted. Huge packet size The upper size limit, in bytes, of an HTTP request to be processed successfully by the AMD. Maximum packet size The AMD is capable of processing packets of up to bytes, besides the Ethernet standard MTU (Maximum Transmission Unit) of 1536 bytes. Choose one of the predefined values (2048, 4096, 8192, or bytes) to enable the AMD to process non-standard MTU packets. When you have chosen the Maximum packet size value, make sure that you also set the Huge packet size to an applicable value. Enabling theamd to process nonstandard MTU packets without extending RAM on the machine and leaving Packet buffer size (64-bit AMDs only) and Data memory limit unchanged can cause an excessive packet loss. To avoid this, extend RAM and configure its usage as recommended in the tables below. For more information, see Setting Packet Buffer Size in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide and Setting Data Memory Limit in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. 35

36 Chapter 4 Basic Monitoring Configuration NOTE Do not enable the processing of large packets for a Small AMD. These devices are not designed to process larger packets. For more information, see Small AMD in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. Table 2. Recommended RAM Configuration for Maximum Packet Size Values for AMDs Maximum packet size 8192 B or less 8192 B B Recommended RAM size for 64-bit platforms 64 GB 96 GB 128 GB Deduplication method You can choose one of four methods for eliminating duplicate packets: Based on TCP checksum and IP ID Using this method, duplicate packets are detected based on their TCP checksum and IP ID. Based on TCP checksum and IP ID (excluded SEQ and ACK numbers) Using this more complex, two-stage method, duplicate packets are detected based on a modified packet KCP checksum (SEQ and ACK numbers are excluded) and IP ID. This method is useful if the AMD captures packets on various interfaces of the router, rewriting SEQ and ACK numbers. A packet is considered a duplicate when the modified checksum, IP ID, and SEQ and ACK numbers are identical. First, a packet checksum with SEQ and ACK numbers is created and compared to the packets stored in the detection buffer. If the comparison indicates that the packet is not a duplicate, it is checked to determine whether it matches the current session. A packet matches the current session when its SEQ and ACK numbers are different from processed and cached numbers by the value defined in TCP duplicate window. If the difference exceeds the defined value, the AMD assumes the ACK and SEQ numbers were rewritten by the router and the packet is considered a duplicate. TCP checksum, IP ID and MAC address (excluded SEQ and ACK) Using this method, the deduplication process is similar to the one based on TCP checksum and IP ID (excluded SEQ and ACK numbers), but in addition to TCP checksum and IP ID, the source/destination MAC addresses are also taken into account for the calculation. TCP checksum, IP ID and MAC address Using this method, duplicate packets are identified based on their TCP checksum, IP ID and source/destination MAC addresses. 36

37 Chapter 4 Basic Monitoring Configuration TCP duplicate window This setting is useful only if Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers. It is used for determining whether a packet, based on its SEQ and ACK numbers, belongs in the session. If a packet's SEQ and ACK numbers differ from the current session's SEQ and ACK numbers by a value larger than TCP duplicate window, the packet is considered a duplicate. Default: Packet buffer size The number of packets to keep in the buffer for use as a basis for comparison in duplicate packet detection. Newly captured packets are sequentially compared to the packets in the buffer. A newly captured non-duplicate packet (all packets in the buffer are unique) is placed on the top of the stack and the oldest is removed. Range: 6 to 24 packets. Default: 16. Reset duplicate detection time threshold Time of inactivity (in seconds) after which the duplicate packets elimination mechanism is reset. If Deduplication method is set to Based on TCP checksum with excluded SEQ and ACK numbers or TCP checksum, IP ID and MAC address (excluded SEQ and ACK), and the Reset duplicate detection time threshold should be greater than every response generation time (server time). 6. Publish the draft configuration on the monitoring device. Configuring Operation-Related Global Settings The operation-related global settings enable you to define options that apply to all monitored operations. These settings take precedence over the options defined for individual operations. 1. Start and log on to RUM Console. 2. Select Devices and Connections Manage Devices from the top menu, to display the current device list. 3. Select Open Configuration from the context menu for an AMD. The AMD Configuration window appears. 4. Select Configuration Global Operations to display the general configuration settings. The options are: Operation load time threshold The number of seconds after which an operation is considered slow. You can set this value with a precision of one ten-thousandth of a second. Default: seconds. The threshold is used by following analyzers: IBM over MQ Jolt MS Exchange Oracle Forms over HTTP 37

38 Chapter 4 Basic Monitoring Configuration Oracle Forms over HTTPS Oracle Forms over SSL Oracle Forms over TCP SOAP over HTTP SOAP over HTTPS XML XML over HTTP XML over HTTPS XML over MQ XML over SSL Max. operation duration The maximum number of seconds an operation can take. You can set this value with a precision of one ten-thousandth of a second. Default: 3600 seconds (1 hour). User abort threshold The minimum number of seconds between the beginning of a hit and TCP reset to count it as a user abort. Default: seconds. (You can set this value with a precision of one ten-thousandth of a second.) ADS data generation settings The options in the ADS data generation settings section can be used to handle various types of standalone hits, which are hits that cannot be automatically assigned to operations because the reference information, such as correlating response, defined or auto-learned URL, no authorization, or orphaned redirects, is missing. By default, most standalone hits are not taken into account when generating operations data. Report data without monitored URL Select this option to report data for hits without a URL that has been explicitly defined in user-defined services or recorded through auto-learning. Report standalone hits without monitored URL Select this option to report data for standalone hits that at the same time do not refer to a monitored URL, as in Report data without monitored URL. Standalone hits are hits without a response header, unauthorized hits, orphaned redirects, or other hits missing the reference context. Report hits without response header Select this option to report data for discarded hits (hits without a correlating response header). Report hits not added to any operation Select this option to report data for other standalone hits caused by factors not covered by other options of this section. Report unauthorized hits Select this option to report data for hits with rejected authentication. Report orphaned redirects Select this option to report data for redirects to sites that are not being monitored or are not visible and therefore appear as orphaned redirects. 38

39 Chapter 4 Basic Monitoring Configuration Report filtered data This is a diagnostics option. When configuring content type monitoring, you can filter out pages based on the content of the URL. For more information, see Monitoring of Non-HTML Objects Based on Content Type in the Data Center Real User Monitoring Web Application Monitoring User Guide. If you select this option, the filtered out pages are not reported, but are saved in the AMD data files. Ignored clients A list of clients for which TCP setup time are ignored and all operations start from the request packet. Right-click the list to open a menu of command options: Add, Edit, or Delete. 5. Save or publish the configuration. Click Save to save your changes and continue with configuration. Click Save and Publish to immediately update the devices configuration. 6. Close the AMD Configuration window. 39

40 Chapter 4 Basic Monitoring Configuration 40

41 CHAPTER 5 Oracle Forms Monitoring Configuration Support for the Oracle Forms environment makes it possible to monitor the performance of Oracle Forms applications communicating over SSL, TCP, HTTP, and HTTPS. To configure monitoring of Oracle Forms traffic, you need to create one or more software service definitions for the Oracle Forms analyzers and assign an AMD as a data source for the CAS and ADS. If you require detailed traffic analysis, you also need to specify software services to be monitored on specific IP addresses and ports. To define an Oracle Forms software service: 1. Configure global settings for Oracle Forms monitoring. The Oracle Forms analyzer provides configuration settings to customize the monitoring of software services based on Oracle Application Server. For more information, see Global Settings for Oracle Forms Monitoring [p. 41]. 2. Create a definition of an Oracle Forms service. For the desired address IP and port, create a definition of a user-defined Oracle Forms software service to be monitored. Depending on the protocol used for communication, choose an applicable analyzer for the software service. Create a software service for each if all of the protocols are used. For details on how to create a new service, see Configuring User-Defined Software Services in the RUM Console Online Help. 3. Review report results and adjust the configuration as appropriate. Determining the best possible configuration for your needs may be an iterative process, where you fine-tune the configuration several times after viewing your report results. Global Settings for Oracle Forms Monitoring The Oracle Forms analyzer provides configuration settings to customize the monitoring of software services based on Oracle Application Server. In the RUM Console, open the AMD Configuration window. Under Configuration, select Global Front-End Monitoring Oracle Forms General and tune the following Oracle Forms monitoring settings: 41

42 Chapter 5 Oracle Forms Monitoring Configuration Generate ADS data for Oracle Forms traffic Select this option to provide data to the report server that consists of low-level protocol information, such as raw HTTP traffic data, which enables you to view the full HTTP request-response dialog. Oracle Forms Identification Session identifier This property defines a pattern used for identification of the session ID, which is required for Oracle Forms session recognition and stability. You can find the session ID in the URL parameter of the POST request for a Forms servlet. Examples of a POST request for a Forms servlet include: POST/forms/lservlet; POST/forms/formsservlet; POST/forms/I90servlet; Typically the identifier name (note that this is not the ID value) required by the AMD is jsessionid or JServSessionIdforms, as shown in the following graphic: The default RUM Console setting is jsessionid=. User Recognition The following parameters listed under User Recognition enable you to use a login user name as the basis for reporting. Login method Specify the request method for the login URL using one of the following methods: GET or POST OFLogin Used for user login detection in web traffic. This setting defines an HTTP method in a request that contains a client username. The available values are: GET, POST, or no entry. Used for user login detection in an Oracle Forms binary stream. Select this option to configure user detection based on Oracle Forms internal login. Selecting OFLogin disables the Login URL parameter. ICXLoginUsed for user detection in an Oracle Forms 6i mixed environment where the user login exists in web traffic and the Oracle Forms session ID exists in a pure binary 42

43 Chapter 5 Oracle Forms Monitoring Configuration stream. To produce the user name of the Oracle Forms operation, add both the web server and all Oracle Forms servers when defining services in the software service rule configuration, as shown in the following graphic. The web traffic contains the user login and ICX Ticket data, and the Forms binary traffic contains a corresponding ICX ticket, which is used to associate the user login contained in the web traffic with the data in the Oracle Forms binary stream: NOTE Another method for identifying user names for traffic data records that do not have an associated user name is to configure the CAS to map the monitored user's name between analyzers for a given client IP address. For more information, see Overriding User Name Mappings for the Same Client IP (CAS Only) in the Data Center Real User Monitoring Administration Guide. Login URL Defines the URL in an HTTP request containing a client username. For an HTTP-based login, the login URL is the base URL where the username value is found in the trace file. For an Oracle Applications/EBS login, the login URL is typically /OA_HTML/OA.jsp/. This field is redundant when extracting the user name directly from Oracle Forms traffic, so it is disabled if the OFLogin method is selected. Login name parameter This property is used for searching for a user name in HTTP requests. For an HTTP-based login, this is the POST parameter key for the username found in the login URL. It typically has a value containing a username string. For an Oracle Applications/EBS login, the username key is typically usernamefield. For a Forms-based login, this is the label used for the username in the Forms login. Typically, this value is username. 43

44 Chapter 5 Oracle Forms Monitoring Configuration Identification cookie pattern This property helps to identify the cookie used for user assignment. Setting this properly helps AMD software match user names to TCP sessions. For an HTTP-based login, this is the cookie key for a Forms session. Typically, it follows the JSessionID cookie for the login URL in the capture file, and the same cookie key is also found in the POST request for the Forms servlet. For an Oracle Applications/EBS login, the key is usually customized. For a Forms-based login, this is the cookie key for a Forms session. Typically, it is JSessionID. Default: JServSessionIdroot The sample login URL show the configuration parameters for the login URL, request method, user name parameter, and the key for the HTTP cookie for the Forms session. Client IP Address Extraction In a load balancer environment, requesting an IP address does not return the actual client IP address. To get the real client IP address, you must extract the real client IP address from one of the HTTP header fields used by a load balancer to forward the real client IP address. It is usually X-Forwarded-For. In this case, select the Header field option and type in the name of HTTP header field containing the real client IP information. For example X-Forwared-For. Error Messages Use the Error Messages table to customize the configuration of error analysis and error message extraction in your environment. In this table, the ID column displays the error category ID and the Name column displays a regular expression that is applied against a recorded message to trigger an error count and save the error message for use in ADS. If there are multiple error strings per operation, per category, per monitoring interval, just the first string is reported. Multiple error message filters can be assigned to the same or different error buckets. 44

45 Chapter 5 Oracle Forms Monitoring Configuration NOTE Error message strings only appear in the ADS if the global option Generate ADS data for Oracle Forms traffic is selected and the Generate ADS data option is selected on the Options tab at the software service level. By default, the following five categories are provided to help you identify the specific error type and corresponding error message. Each category has a corresponding ID, which is used in the Error Messages table to identify the category. Table 3. Error Categories Error Category Forms client error Forms application error Forms server error Oracle server error Oracle Applications error Error Codes Provided by Default No No Yes Yes No For the Forms server error and Oracle server error categories, the actual error codes used to identify application availability problems are also specified by default in the Error Messages table, as shown in the following graphic: NOTE For the syntax in the Name column, the ^ symbol indicates the match must occur at the start of a new line, and the symbol is used as a delimiter between error codes. For the remaining default error categories, you have the option of adding specific error codes in any of the categories. For a listing of suggested error codes and recommended regular expression syntax, see Oracle Forms Error Categories and Messages [p. 93]. After configuring the Error Messages table, you must enable the availability of an error category on the Availability tab to view error message data in DMI reports. For more information, see Configuring Oracle Forms Availability [p. 48]. Thresholds When you configure Oracle Forms monitoring, you are able to define operation time thresholds on several levels starting from general settings of the analyzer, through per software service, to 45

46 Chapter 5 Oracle Forms Monitoring Configuration a per individual operation level. Note that global setting Page load time/operation threshold is ignored for Oracle Forms monitoring. Operation timeout The maximum time that a single operation should last. An operation with a longer execution time is regarded as unsuccessful or broken. Increase the value of this option if your Forms application response time is slow. Default value: 1 second. Session state timeout How long (in seconds) a session ID is regarded as valid and recognized by the analyzer. Default value: seconds (8 hours). Max. length of property The maximum length of a text string that is send in a binary Oracle Forms message. Default value: bytes. Operation Pattern Matching and Processing Operation pattern matching may be the key to understanding and debugging your Web applications. Setting all the parameters related to this issue will enable better presentation of the data by the report server. With them, you can see the actual relations between particular components of your Web application. Operation name pattern Use this field to influence how the content of a transaction name is formed. This string must contain keywords that will be replaced with real values. They can be used in any order and separated by any characters. Note that these keywords are case sensitive. The following keywords are supported: %parenttitle% - window title of the previously active window %resulttitle% - window title of the new window %controltitle% - the name of the control (for example, button's caption) %classname% - the type of the control (for example, ButtonItem) %propname% - action (for example, pressed) %modulename% - the name of the module in use Default: %parenttitle%\t%resulttitle%\t%classname%:%controltitle% Example 1. Sample Oracle Forms operation name pattern recorded on the AMD PT:Work Request Submission RT:Work Request Submission CT:Save CN:ButtonItem PN:pressed %modulename% Processing Use this setting to include a module's path or extension in reports. Note that this property is not part of the default configuration. %parenttitle% Processing These settings make it possible to transform a parent title. An extended POSIX regular expression (entered in the Regex Pattern column) matches the parent title and allows the 46

47 Chapter 5 Oracle Forms Monitoring Configuration selection of only a part or parts of the title for further processing (using grouping subexpressions). Output Pattern describes how to assemble the pieces matched by Regex Pattern. This is a string that may contain back references (backslash character \ followed by a non-zero decimal digit) to subexpressions matched and grouped in Regex Pattern. The following setting removes everything from the %parenttitle% from the ( character to the end: Regex Pattern: (.+)\( Output Pattern: \1 If the parent title were PT:Fiscal Report Submission (RNO ), after this transformation you would have PT:Fiscal Report Submission. Several instances of this property are allowed. Default: no action. %resulttitle% Processing Similarly to %parenttitle% Processing, this property makes it possible to transform the new window's title. An extended POSIX regular expression (entered in the Regex Pattern column) matches the parent title and allows the selection of only a part or parts of the title for further processing (using grouping subexpressions). Output Pattern describes how to assemble the pieces matched by Regex Pattern. This is a string that may contain back references (backslash character \ followed by a non-zero decimal digit) to subexpressions matched and grouped in Regex Pattern. The following setting regroups %resulttitle% components so that the requested strings appear first, although they are detected in a different order: Regex Pattern: ([^ ]*) ([^ ]*) ([^ ]*)(.*) Output Pattern: \3 \1 \2 \4 If the result title were RT:Work Request Submission Illegal x00af, after the transformation you would have RT:Illegal Work Request Submission x00af. Several instances of this property are allowed. Default: no action. See Regular Expression Fundamentals [p. 83] to learn more on using regular expressions in Data Center Real User Monitoring configuration tasks. Configuring Oracle Forms Triggers An Oracle Forms trigger defines properties that drive the transaction recognition algorithm. Oracle Forms triggers are a predefined set of mappings of GUI elements and their properties (state, actions, and behavior). This mapping is a definition of what is understood as a single operation. Control over triggers makes it possible to exclude insignificant GUI actions from being reported (for example, a window getting focus or a tooltip being displayed for a button). You can define whether setting particular properties for some classes is considered a user action or just setting the screen title. You can inspect the mappings in RUM Console in the AMD Configuration window. For this purpose, go to Global Front-End Monitoring Oracle Forms Triggers. Although each of the tables allows you to add, edit, or delete property mappings, we recommend that you use default values. 47

48 Chapter 5 Oracle Forms Monitoring Configuration The tables Class map and Property map should help you decipher the binary part of Oracle Forms traffic. The ID column contains the binary representation that is actually passed in network traffic. Its human-readable name is contained in the Name column. These names are later used and transformed by the AMD and finally the report server. Configuring Oracle Forms Availability To view information about Oracle Forms failures in DMI reports, access the Forms Availability tab and configure the appearance of a specific failure type in a DMI report. You can configure Oracle Forms availability globally or at the software service level. For global configuration, open the AMD configuration and go to Global Front-End Monitoring Oracle Forms Availability. For the software service level, select the Availability tab in the Edit Rule window. To view information about system, application, and environment failures in DMI reports, you must enable the appearance of a specific failure type, as described in the following list: Transport Failures No response Incomplete response. A hit with no response from a server. Disabled by default. Aborted response A response aborted by the user. Enabled by default. HTTP errors The AMD is able to deliver information on seven HTTP error groups ( categories ). HTTP client errors (4xx) HTTP server errors (5xx) HTTP unauthorized errors HTTP Not Found errors HTTP client errors (category 3) HTTP server errors (category 1) HTTP server errors (category 2) You can decide whether each of these should be taken into account when calculating (failures transport). Note that HTTP client errors (4xx), HTTP server errors (5xx), HTTP unauthorized errors, HTTP Not Found errors, and HTTP server errors (category 1) have configurable contents. For more information, see Assigning HTTP Error Codes to Error Categories in the Data Center Real User Monitoring Web Application Monitoring User Guide. Application Failures Forms client error Not enabled by default. For more information, see Forms Client Errors [p. 93]. Forms application error Not enabled by default. For more information, see Forms Application Errors [p. 94]. 48

49 Chapter 5 Oracle Forms Monitoring Configuration Forms server error Enabled by default. For more information, see Forms Server Errors [p. 104]. Oracle server error Enabled by default. For more information, see Oracle Server Errors [p. 114]. Oracle Applications error Not enabled by default. For more information, see Oracle Applications Errors [p. 114]. Defining Oracle Forms Software Services For software services you intend to monitor that do not work on a well-known port, you can use the specific IP address and port of the service when defining the software service configuration. For such software services, you can measure a wide range of metrics and perform detailed traffic analysis. To add a new software service: 1. Start and log on to RUM Console. 2. In the top menu, select Software Services Add Software Service. The Add Software Service pop-up window appears, listing all ways of adding a new service. 3. Select Manually as a method of adding a new software service definition. The Add Software Service window appears. 4. Specify basic information for your software service. Provide a software service name. Select appropriate analyzer to monitor the traffic. Using check boxes, select the devices that will monitor the new software service. When you later publish the software service definition, the new configuration will be applied only to the selected devices. 5. Click OK to proceed to monitoring rules configuration. 6. Right-click in the Rules table and select Add from the context menu. The Rule Configuration appears. 7. Proceed with the rules configuration.for more information, see Configuring Rules for User-Defined Software Services [p. 49]. 8. On the Software Services screen, click Publish Configuration. Configuring Rules for User-Defined Software Services Each software service can have a number of specific rules that define what is to be monitored and what additional options are in effect. Before You Begin It is assumed for this task that you are already familiar with the concept of software services and that you know how to create and edit software services and how to open the Rules window. 49

50 Chapter 5 Oracle Forms Monitoring Configuration After a user-defined software service is created, create a group of settings that comprise the rules for the software service. It is necessary to specify, at minimum, the IP addresses and port numbers for the software service. To configure rules for a user-defined software service: 1. In the Rules table, right-click to open the context menu and choose Add or Open. The Rule Configuration window will open. 2. On the Services tab, select or clear Enabled to activate or de-activate the service definition. 3. In Rule description, type a brief description that will later help to identify the rule. The description you enter at this point will be shown in the Rules table, in the column Rule Name. If no text is entered here, the IP address specified later will be used as a description for this rule. 4. Right-click in the Services table and select Add or Open from the context menu. The Service Details window appears when adding or editing rules. 5. In the Service Details window, in the IP address(es) fields, enter the server IP address, or enter a range of IP addresses if you plan to monitor more than one server. 6. In the Port(s) fields, enter the port number of the monitored service. You can provide a range of port numbers if such a range of ports is used in your environment. Some software services may be active on a number of predefined ports or may change ports dynamically. To allow for this, you can specify a range of ports. Note, however, that specifying more than one port for a service prevents the port number from being reported for that service. If you define more than one port for a particular service name and server IP address (by either specifying a range of ports or by creating two or more distinct rules for the same service name and server IP address but with different port numbers), the AMD will report the port number for this service as 0, causing the port number to be ignored in traffic reports. NOTE You can define up to 5000 definitions containing a server and a port. Each association of a server and a port counts as a single definition. Specifying a range of ports counts as providing many individual definitions. On CAS, the number of processed server definitions is limited by the license. For more information, see Per-Measurement Licensing in the Data Center Real User Monitoring Administration Guide. 7. Optional: Select Client port(s) for reversed-direction protocols. This option makes sense only for protocols such as X-Window whose client-server meanings are reversed. If you are uncertain, leave this option cleared. 8. Optional: Enter a virtual IP address if your network uses a pool of virtual IP addresses. For more information, see virtual IP address [p. 122]. 9. Optional: Enter the IP address of the server masking the addresses of monitored servers. 50

51 Chapter 5 Oracle Forms Monitoring Configuration If the servers you intend to monitor reside behind an appliance that masks and replaces the addresses of the target servers, you need to set NLB NAT masking IP address to the IP address of the masking server. Without doing so, the AMD will see two unidirectional conversations instead of one bi-directional conversation between the servers and appliance: The conversation between the client and server is observed and recorded (IP address A talking to IP address B) When a response travels to the client, a different session (IP address C talking to IP address A) is recorded due to the server's IP address being replaced by the load balancer's IP address. Unless you account for this, CAS reports will return reports with ambiguously granulated data. Using the NLB NAT masking IP address option will ensure that the AMD monitors contiguous conversations. 10. Click OK to confirm your changes and close the Service Details window. 11. Go through all available tabs and fine-tune the monitoring conditions. The number of available configuration options depends on the analyzer. See the analyzer-specific section for more information. 12. Optional: On the Options tab, define analyzer-specific options. Enable monitoring of persistent TCP sessions When this is selected, TCP sessions that do not start with SYN packets are monitored. By default, this is selected. Persistent TCP sessions are TCP sessions for which the start was not recorded. They are also referred to as non-syn sessions. These sessions can be included in the TCP statistics, based on the configuration properties you enable in RUM Console. The inclusion of these sessions may render the statistics somewhat inaccurate and must be undertaken with care. Page load time threshold An operation that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. To edit the global setting, open the AMD configuration, go to Global General and set the Operation time threshold. Generate sequenced transactions and ADS data Select this option to provide data to the report server that consists of low-level protocol information, such as raw HTTP traffic data, which enables you to view the full HTTP request-response dialog. 13. Click OK to save the configuration. 14. Publish the draft configuration on the monitoring device. 51

52 Chapter 5 Oracle Forms Monitoring Configuration Configuring Monitoring of an Oracle Forms Software Service If required, you can override the global settings for the Oracle Forms protocol analysis with specific settings for a particular user-defined software service. You have the choice of defining an Oracle Forms software service over HTTP, HTTPS, TCP, or SSL. Before You Begin It is assumed for this task that you have already created a user-defined software service for this protocol and have specified one or more rules containing the essential components such as the IP address and port of the software service to be monitored. For more information, see Configuring User-Defined Software Services in the RUM Console Online Help and Configuring Rules for User-Defined Software Services in the Data Center Real User Monitoring Citrix/Windows Terminal Services Monitoring User Guide. To specify protocol-specific settings: 1. Configure the Oracle Forms user identification for the service. Open the User Recognition tab on the rule screen for the service. If you wish to override the global settings, clear the Use global settings box, and then choose one of the following methods of user identification: Login method Specify the request method for the login URL using one of the following methods: GET or POST OFLogin Used for user login detection in web traffic. This setting defines an HTTP method in a request that contains a client username. The available values are: GET, POST, or no entry. Used for user login detection in an Oracle Forms binary stream. Select this option to configure user detection based on Oracle Forms internal login. Selecting OFLogin disables the Login URL parameter. ICXLoginUsed for user detection in an Oracle Forms 6i mixed environment where the user login exists in web traffic and the Oracle Forms session ID exists in a pure binary 52

53 Chapter 5 Oracle Forms Monitoring Configuration stream. To produce the user name of the Oracle Forms operation, add both the web server and all Oracle Forms servers when defining services in the software service rule configuration, as shown in the following graphic. The web traffic contains the user login and ICX Ticket data, and the Forms binary traffic contains a corresponding ICX ticket, which is used to associate the user login contained in the web traffic with the data in the Oracle Forms binary stream: NOTE Another method for identifying user names for traffic data records that do not have an associated user name is to configure the CAS to map the monitored user's name between analyzers for a given client IP address. For more information, see Overriding User Name Mappings for the Same Client IP (CAS Only) in the Data Center Real User Monitoring Administration Guide. Login URL Defines the URL in an HTTP request containing a client username. For an HTTP-based login, the login URL is the base URL where the username value is found in the trace file. For an Oracle Applications/EBS login, the login URL is typically /OA_HTML/OA.jsp/. This field is redundant when extracting the user name directly from Oracle Forms traffic, so it is disabled if the OFLogin method is selected. Login name parameter This property is used for searching for a user name in HTTP requests. For an HTTP-based login, this is the POST parameter key for the username found in the login URL. It typically has a value containing a username string. For an Oracle Applications/EBS login, the username key is typically usernamefield. For a Forms-based login, this is the label used for the username in the Forms login. Typically, this value is username. 53

54 Chapter 5 Oracle Forms Monitoring Configuration Identification cookie pattern This property helps to identify the cookie used for user assignment. Setting this properly helps AMD software match user names to TCP sessions. For an HTTP-based login, this is the cookie key for a Forms session. Typically, it follows the JSessionID cookie for the login URL in the capture file, and the same cookie key is also found in the POST request for the Forms servlet. For an Oracle Applications/EBS login, the key is usually customized. For a Forms-based login, this is the cookie key for a Forms session. Typically, it is JSessionID. Default: JServSessionIdroot 2. Specify an operation load time threshold per individual operation. Open the Oracle Form Monitoring tab on the rule screen for the service. Right-click the Oracle Form Definitions table and select Add or Open. In the Oracle Form Operation dialog, provide a regular expression to match the operation of your interest and its human readable name. If you wish to override the operation load time threshold for this particular service, deselect Inherit from rule setting and provide a desired value in seconds. 3. Open the Availability tab and choose the specific types of failures to display in DMI reports. For more information, see Configuring Oracle Forms Availability [p. 48]. 4. Open the Options tab on the rule screen for the service and select or clear the following check boxes and enter the required information, according to your monitoring needs. Enable monitoring of persistent TCP sessions When this is selected, TCP sessions that do not start with SYN packets are monitored. By default, this is selected. Persistent TCP sessions are TCP sessions for which the start was not recorded. They are also referred to as non-syn sessions. These sessions can be included in the TCP statistics, based on the configuration properties you enable in RUM Console. The inclusion of these sessions may render the statistics somewhat inaccurate and must be undertaken with care. Operation load time threshold An operation that takes more than this many seconds is considered slow. When Inherit from global setting is selected, the global setting is used. The global threshold value depends on the analyzer. Generate sequenced transactions and ADS data Select this option to provide data to the report server that consists of low-level protocol information, such as raw HTTP traffic data, which enables you to view the full HTTP request-response dialog. 5. Save or publish your changes. For more information, see Configuring Rules for User-Defined Software Services in the Data Center Real User Monitoring Citrix/Windows Terminal Services Monitoring User Guide. 54

55 Managing User-Defined Software Services As a console user you have the ability to add, remove, and edit the properties of any software service defined on any of the AMDs. You can view all user-defined software services on the Software Services screen in the RUM Console. To access this screen, select Software Services Manage Software Services from the top menu. This screen contains information about the user-defined software services that were created on all devices managed from this console. You can add new services, delete existing ones, and copy them to other AMDs. Note that the default software services are specific to a single AMD and cannot be managed centrally. To view user-defined software services monitored by a selected device, access the configuration for this device. Select Devices and Connections Manage devices from the top menu, a list of all devices managed by the console appears. Next, select Open configuration from the context menu for the device. Assigning Software Services to Devices Chapter 5 Oracle Forms Monitoring Configuration Software services mirrored across different AMDs (that is, having the same name and identical rules) are grouped together. The service name is the name for the group. Whenever you change rules on any of the AMDs in the group, the software services are separated into single entries. After a software service is created on one of the AMDs in your network, you can copy it to another device. To do so, on the Software Services screen, select Copy from the Actions context menu for a given software service. A dialog box appears where you can choose the AMDs by selecting the check box next to their IP addresses. After you click OK, the software service definitions are copied to the selected AMDs. On the Software Services screen, select the Deployment tab and click Change Assignment to modify the list of devices that are monitoring a given software service. 55

56 Chapter 5 Oracle Forms Monitoring Configuration 56

57 CHAPTER 6 Monitoring Sequence Transactions You can manage the sequence transactions (operation sequences) that are defined on an individual AMD or manage each transaction that is monitored by a group of AMDs. To view the defined transactions monitored by a single AMD, select Devices and Connections Manage Devices from the console top menu. Next, select Open configuration from the context menu for the AMD to access the AMD Configuration screen. Finally, select Configuration Sequence Transactions. The main Sequence Transactions table lists all of the currently defined transactions and their details: Sequence Transaction Name The name of a transaction. Application Name The application that includes the listed transaction. Type The protocol used to define the listed transaction. Steps The number of individual operations involved in the listed transaction. Priority The priority of the transaction. Possible values are 1 (highest priority), 2, and 3. Timeout The maximum time for the transaction to complete. Packaged Applications Identifies whether the listed transaction is a packaged application whose transactions are recognized by the report server automatically. On the Sequence Transactions screen, select Edit from the Actions menu for a given transaction to open the Edit Transaction window and examine the steps that make up the transaction. The listed details are as follow: Name, Application, Description: Timeout [s], Slow after [ms], Priority URL, Timeout, Repetition. 57

58 Chapter 6 Monitoring Sequence Transactions NOTE If the transaction you are deleting is monitored by more than one AMD, a new draft configuration must be published to all of the affected AMDs. To view all transactions, select Reporting Configuration Sequence Transactions from the console top menu. Sequence Transaction Name The name of a transaction. Application The application that includes the listed transaction. Type The protocol used to define the listed transaction: ASYNC-HTTP, CERNER, CERNER-RTMS, HTTP, OF, SAP GUI, SQL or XML. Packaged Applications Whether the listed transaction is a packaged application whose transactions are recognized by the report server automatically. When you select a transaction by clicking it once, you can see the list of AMDs that monitor this transaction. Adding Oracle Forms Transactions You can add a transaction to either an individual AMD or a range of AMDs using the RUM Console. To define a new transaction: 1. In the RUM Console, select Reporting Configuration Sequence Transactions. 2. Click Add Sequence Transaction. The Create Sequence Transaction pop-up window appears. 3. Enter the application and transaction names and a description. If you have configured the Dynatrace connection, click Browse to select a predefined application and a specific transaction within this application. For more information, see Configuring the BSM Connection in RUM Console in the Data Center Real User Monitoring Administration Guide. 4. From the Type list, choose OF. 5. Select the devices that will monitor the transaction. When you publish the new configuration, it is only applied to these devices. 6. Click OK. On the screen, specify the configuration details for the transaction. 7. Provide the timing and priority values: Timeout [s] The maximum time for the transaction to complete. Transactions must complete in this time to be logged as successful transactions. 58

59 Chapter 6 Monitoring Sequence Transactions Slow after [ms] If the transaction execution time exceeds this value, the transaction is classified as slow. Specify this threshold in seconds, for example: 500. Priority Determines which transaction is recorded if two or more transaction definitions match the transaction detected in the monitored traffic. The valid priority values are 1 (highest priority), 2, and 3. A multiple transaction match can happen if, for example, you first create a generic transaction definition that can match a number of more specific transactions and then you create another transaction definition that matches a particular sub-type of that generic transaction type. If an observed transaction is found to match the latter definition, it also matches the first (more generic) definition, and the system will need to determine under which transaction name to record the observed transaction instance. By increasing the priority of the second, more specific definition, you can count the occurrences of this particular transaction sub-type, which are then not counted in the statistics for the generic transaction type. So you can use this feature to increase the priority of specific customized transaction definitions that should take precedence over more generic transaction templates. 8. Specify Oracle Forms operations comprising the transaction steps. The Oracle Forms operation can contain an optional wildcard character *. Using the wildcard character *, you can signify any number of any characters. For more information, see Using Wildcards in URLs in the Data Center Real User Monitoring Web Application Monitoring User Guide. To maintain the sequence of these operations, use the navigation buttons on the right. You can Add, Delete, Move Up, Move Down, or Copy the defined steps by selecting the step and clicking one of these actions. 9. Click OK to add your transaction definition to a draft configuration. 10. On the Sequence Transactions screen, click Publish Configuration. What to Do Next You can also add a transaction using the Transaction Inspector. For more information, see Filters and Transaction Inspector for Oracle Forms Transactions [p. 60]. Modifying, Deleting, and Cloning Transactions for a Single AMD Modifying a Sequence Transaction To modify the definition of an existing transaction: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 59

60 Chapter 6 Monitoring Sequence Transactions This opens the Sequence Transactions table, listing all of the defined transactions for this AMD. 3. Right-click the transaction to manage and select Open from the context menu. You can modify any of transaction details. For more information, see Adding Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. Deleting a Sequence Transaction To delete selected transactions: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 3. Click the transaction that you want to delete. To delete multiple transactions with one step, hold the [Ctrl] key as you click additional transactions. 4. Right-click and select Delete to remove the selected transactions from the list. Cloning a sequence transaction To clone selected transactions: 1. Open AMD configuration and click Edit as Draft to switch to draft mode. 2. In the Configuration tree, select Sequence Transactions. 3. Click the transaction that you want to clone. To clone multiple transactions with one step, hold the [Ctrl] key as you click additional transactions. 4. Right-click and select Clone to duplicate the selected transactions. A cloned transaction is indicated by the original transaction name with (Clone) appended to it. There are differences between cloning and copying. For more information, see Monitoring Sequence Transactions in the Data Center Real User Monitoring Web Application Monitoring User Guide. Filters and Transaction Inspector for Oracle Forms Transactions The Transaction Inspector enables you to select individual steps and construct your own transactions from live traffic or historical data. The Transaction Inspector consists of two main areas: the Filters and the Transaction Inspector itself. 60

61 Chapter 6 Monitoring Sequence Transactions Filters Transactions can be defined manually by entering each step, however, the Filters area enables you to examine Recent Data and select the detected steps to build a transaction. The transaction filter consists of two tabs: Data Filter The Data Filter tab enables you to define your filter by selecting the source and range of data to be filtered. From the list, you can select the report server and create a user filter. Select User Name or User IP Address and either enter the data manually or click Browse to open the Select User window, where you can select the user identified in transaction traffic by the report server. You can highlight or search for the specific user or user IP address and filter the search query based on any of the columns in the transactions table. After the user or user IP address is selected, you can choose to extract transactions from Recent Data stored on the report server. You have to provide a Begin and End date for the time range to be processed. Result Filter The Result Filter tab consists of a find field, a transaction detail field, and an interactive legend to filter transactions that have been classified as: Table 4. Result Filter Color Guide The transaction was recognized and matched with the transaction currently being defined. One or more steps One or more steps A Step or a URL Excluded URLs in the transaction in the transactions was recognized as which did not currently being for which an an already defined match any defined were not HTTP error other and saved definition. completed. than 404 occurred. transaction definition. By selecting and clearing the corresponding check boxes, you can filter the URLs from the data source. The color coding of the steps is based on your current transaction definition in the Transaction Definition area. To view the results and enable the filter to receive data, click, located on the right side of the Filters area. While viewing the data in Transaction Inspector, at any time you can force the data to be recalculated using your current transaction definition by clicking or, you can stop the filter by clicking, located on the right side of the Filters area. Transaction Inspector Transaction Inspector consists of two tables that display the URLs and Sequenced Transactions detected in the data source that is defined in the Filters section. The Transaction Inspector enables you to select one or a number of detected steps and add them to your transaction definition. Select the check box corresponding to the URL that you want to add and then click located above the table. 61

62 Chapter 6 Monitoring Sequence Transactions You can add the steps from both the URLs table and from the Transactions table. After the URL is moved to the Transaction Definition table, you can modify it, position it within a sequence of other steps, clone it as another step, or delete it using the operation buttons to the right of the Transaction Definition table. 62

63 CHAPTER 7 Oracle Forms Tiers A tier is a specific point where DC RUM collects performance data. It is a logical application layer, a representation of a fragment of your monitored environment. There are two tiers on CAS that report Oracle Forms data: Oracle Forms The tier is by default marked as front-end and is automatically displayed on the Tiers report in the Data center tiers section. This tier shows measurements for Oracle Forms traffic monitored by AMD. RUM sequence transactions If you have defined sequence transactions for Oracle Forms monitoring, the tier will automatically be displayed on Tiers in the Client tiers section. For more information, see Solution-Specific Tiers in the Data Center Real User Monitoring Administration Guide. 63

64 Chapter 7 Oracle Forms Tiers 64

65 CHAPTER 8 ADS Support for Oracle Forms Using ADS for monitoring Oracle Forms traffic requires special setup that switches HTTP-related metrics into the Oracle Forms meaning. An AMD can be configured to monitor Oracle Forms traffic and pass on the traffic statistics to an installation of Advanced Diagnostics Server, where the data is displayed in all of the data views with the exception of Transactions. Oracle Forms statistics are not visible in the Transactions view and using Transaction configuration tools makes no sense for Oracle Forms traffic. In such a configuration, the meaning of ADS report columns and DMI dimensions and metrics is changed to include aspects specific to Oracle Forms: metrics and dimensions related to HTTP-specific concepts should therefore be interpreted as relating to the corresponding Oracle Forms concepts. In particular, operations refer to Oracle Forms transactions; and the concept of hits is obsolete, in that the number of hits per operation is always given as one. Metrics and dimensions that require additional explanation are listed below on a per-view basis. NOTE To view Oracle Forms statistics, you must change the default Web filter setting for the Analyzer group dimension. Unless the filtering condition includes Oracle Forms, the Oracle Forms data will not appear on the ADS reports. For more on filtering data refer to Dimension Filter in the Data Center Real User Monitoring Data Mining Interface (DMI) User Guide. Oracle Forms Traffic Analysis in the Operation data Data View The following Operation data dimensions have meanings specific to Oracle Forms: Operation begin time Oracle Forms transaction begin time. In addition to the time stamp of the measurement (the data package from the AMD), each transaction is separately stamped with its actual time of occurrence, with 1 millisecond accuracy. This dimension can be used to list transactions one-by-one, in the order of their occurrence, to reflect client or server activity. Operation Oracle Forms Transaction name. 65

66 Chapter 8 ADS Support for Oracle Forms HTTP response status Operation status Oracle Forms transaction status: Abort or No Abort. Root cause Root cause details Request method The following Operation data metrics have meanings specific to Oracle Forms: Operation size Transaction size: the number of bytes sent by the client and server during the transaction, without counting retransmitted bytes. HTTP request time Request time minus connection setup time. Operation load time Transaction time. Server time Server time of the Oracle Forms transaction: the time it took the server to produce a response for the given request. This time is an aggregated time of server times for all data exchanges that were part of the Oracle Forms transaction. It is not the request-response time difference which is typical for other analyzed protocols. Idle time HTTP server time Equal to server time: the time it took the server to produce a response for the given request. This time is an aggregated time of server times for all data exchanges that were part of the Oracle Forms transaction. It is not the request-response time difference which is typical for other analyzed protocols. Server think time Not applicable (always 0). Hits The number of Oracle Forms transactions. Hits per operation Not applicable (always equal to 1). HTTP errors and all metrics related to specific HTTP errors Percentage of slow operations due to DC Some data may appear for this metric. However, this data is calculated according to HTTP rules and the values rendered do not carry any meaning for Oracle Forms. 66

67 Chapter 8 ADS Support for Oracle Forms Transaction operation begin Slow operations (reason) Not applicable, for any value of reason, such as client delays and other. Some data may appear for these metrics. However, the data is calculated according to HTTP rules and the values rendered do not carry any meaning for Oracle Forms. Number of request cookies The number of key-value pairs in the Cookie field of the HTTP request header. Request header size Average length of the HTTP request header (sum for operation hits). Request body size Size of the HTTP request body (sum for operation hits). Avg redirect time Number of response cookies Response cookie size Response header size Max number of request cookies Max request cookie size Max request header size Max request body size Max redirect time Max number of response cookies Max response cookie size Max response header size Min number of request cookies Min request cookie size Min request header size 67

68 Chapter 8 ADS Support for Oracle Forms Min request body size Min redirect time Min number of response cookies Min response cookie size Min response header size Stdv number of request cookies Stdv request cookie size Std request header size Stdv request body size Redirect time Stdv number of response cookies Stdv response cookie size Std response header size Time resolution The remaining Operation data metrics retain their original meaning. Oracle Forms Traffic Analysis in the Operation element data Data View The following Operation element data dimensions gave meanings specific to Oracle Forms: Hit begin time Not applicable (equal to the Oracle Forms transaction begin time). Request method Operation Oracle Forms Transaction name. Component URL Oracle Forms Transaction name. HTTP response 68

69 Chapter 8 ADS Support for Oracle Forms Hit status Oracle Forms detailed transaction status. Values: No Abort no transaction abort Client abort client aborted transaction Dead Transaction error Response status Not applicable (always OK). Transaction status Not applicable (always Belongs to operation ). Content type The remaining Operation element data dimensions retain their original meaning: The following Operation element data metrics have meanings specific to Oracle Forms: Hits The number of Oracle Forms transactions. Component request begin Not applicable (always 0). Number of cookies Cookie total bytes Number of request cookies The number of key-value pairs in the Cookie field of the HTTP request header. Request header size Average length of the HTTP request header (sum for operation hits). Request body size Size of the HTTP request body (sum for operation hits). Avg redirect time Number of response cookies Response cookie size Response header size Max number of request cookies Max request cookie size 69

70 Chapter 8 ADS Support for Oracle Forms Max request header size Max request body size Max redirect time Max number of response cookies Max response cookie size Max response header size Min number of request cookies Min request cookie size Min request header size Min request body size Min redirect time Min number of response cookies Min response cookie size Min response header size Stdv number of request cookies Stdv request cookie size Std request header size Stdv request body size Redirect time Stdv number of response cookies Stdv response cookie size 70

71 Chapter 8 ADS Support for Oracle Forms Std response header size Time resolution The remaining Operation element data metrics retain their original meaning. 71

72 Chapter 8 ADS Support for Oracle Forms 72

73 APPENDIX A Diagnostics and Troubleshooting Report-Related Issues Central Analysis Server automatically detects a range of exceptions (anomalies) and notifies the report users. Exception notifications are displayed as yellow (warning) or red (error) triangle icons in the upper-left corner of the report window. To see the notification message, position the cursor over the triangle icon. The Slow Operation Load Sequence report is empty for an operation which is part of an XML transaction. Why and how do I fix this? For XML and SOAP, Operation Elements data is identical to Operation Analysis data, so, to avoid unnecessarily keeping the duplicates in the database, a VDATA_FILTER_XMLSOAP filter is set to true by default. Keeping this filter set to true saves disk space but, because the XML and SOAP entries are filtered out, it makes reporting on the Operation Elements level (elements or headers) impossible. To change the value of VDATA_FILTER_XMLSOAP property in userpropertiesadmin, type in the Web browser's Address bar and press [Enter], change the filter's property value, and click Set value to accept the change. To access this screen, you need to have administrative privileges for the report server. The yellow triangle displays AMDs produce no performance data. What do I do? The message AMDs produce no performance data means that AMDs connected to the report server do not produce any new data. To resolve this issue, you have to investigate the configuration of the AMDs and determine why they do not produce the performance data. The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? The report server has a built-in protection from simple configuration mistakes. One of the related problems is when data is incorrectly time stamped by AMD. This happens when the AMD is running with the system clock incorrectly set and is not being synchronized with the report server. If you see this notification, check the system time on the report server and on the AMD. Ensure the time synchronization option is turned on. 73

74 Appendix A Diagnostics and Troubleshooting To check the time synchronization: 1. Launch the RUM Console. 2. Select the AMD, right-click it and choose Open Configuration. The AMD Configuration window appears. 3. Select Global General. Check the IP address of the server authorized to set the AMD time. Make sure it is the same as the report server IP address. 4. Check the report server time setting. Do this by reading the time that is displayed at the bottom of the reports. Ensure the report server has the time zone set correctly. Figure 5. Example of the Report Time Stamp The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? Once a day the report server has to perform a database maintenance and memory cleanup. During that time, the data processing has to be suspended and you will see delayed data on reports. The daily maintenance is usually performed as the first task after midnight and it takes up to half an hour in installations with a large database. It is normal and expected to see this warning just after midnight. But if you see the message during the day, it can be a symptom of incorrect system configuration (check the time settings on the server) or of system overload. The yellow triangle displays No contact with the primary AMD. What do I do? This message indicates that the report server has lost contact with at least one primary AMD. If an AMD is marked as primary and the report server cannot communicate with this AMD, even if the performance data can be downloaded from the other AMDs, the system will wait until the communication with the primary AMD is restored. The yellow triangle displays No contact with any of the AMDs. What do I do? This message indicates that the communication link cannot be established with any of the attached AMDs. Check the network settings on the report server or the configuration of AMDs. 74

75 Appendix A Diagnostics and Troubleshooting The yellow triangle displays Delay in data processing. What do I do? If the last processed data is significantly behind the current time due to slow data processing or idle periods that occurred in the past, the report server displays the triangle icon with the message Delay in data processing. If the server had a delay, but now it is catching up, this message will not appear anymore. To confirm that delay is decreasing, inspect server.log and search for messages similar to this: T REC :10: zdata_43f47e58_5_t is being processed. Sample begin ts = :25. Sample delay 17 min. If the delay becomes smaller, the server is catching up. If the delay values are growing, it can indicate a system overload. The yellow triangle displays The AMD has not yet generated performance data. What do I do? This message indicates that some data files have already been generated on some AMDs, but not on the others. This may not be an indication of a problem and, when you refresh the reports after 30 to 60 seconds, this message may disappear. If necessary, verify the time synchronization among all the AMDs. See The yellow triangle displays Delay in data processing. What do I do? [p. 75]. The yellow triangle displays Data processing is being performed in the debug mode. What do I do? Data processing can be manually suspended and controlled by so-called debug mode, which can be enabled using Control Panel. Open Control Panel by typing: in the Address field of the web browser and clicking Go, then select Controlled data processing from the Configuration Management section. The red exclamation mark displays Data loading is in progress. Reports may be incomplete. What do I do? This message indicates that the report server is currently starting up. Because of this the information presented on reports may be incomplete. Depending on the database size, the startup process may take up to several minutes. If the server restart was not done manually or was not planned, inspect server.log or contact Customer Support. The red exclamation mark displays Low memory. The real-time cache will only be updated. What do I do? This message indicates that the report server has no free memory to process new entities such as software services, servers, and URLs. This message will be cleared when some resources are freed, this usually happens at midnight during the scheduled database maintenance (see The yellow triangle displays A daily maintenance task is in progress. Data processing suspended. What do I do? [p. 74]). All the metric values presented on reports (except user/client counters) will show correct values. However, the predefined tabular reports may not show all the entities they are intended to show. All the charts and DMI reports show correct data. The mechanism of updating the real-time cache, as described above, is a protection that allows the report server to continue the operation instead of closing down due to lack of memory resources. 75

76 Appendix A Diagnostics and Troubleshooting The red exclamation mark displays The number of servers has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored servers. If the number of observed servers reaches a defined limit, the report server will not accept any new servers and will drop the collected data for those servers. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red exclamation mark displays The number of clients has reached the defined limit. What do I do? The report server has a built-in limit of the number of monitored clients. If the number of registered clients (which also includes aggregated virtual clients such as Client from... ) reaches a defined limit, the report server will not accept any new clients and will drop the collected data for those clients. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The red exclamation mark displays The number of sites has reached the defined limit. What do I do? The report server has a built-in limit of the number of automatically created sites. If the number of observed automatic sites reaches a defined limit, the report server will not create any new automatic sites and such traffic will be allocated to All Other. The predefined value of the limit can be customized. However, the report server can automatically adjust the limit in low-resources situations. The Sites report for a selected application is empty. Why? If the Sites report for a selected application is filtered for a client tier, such as Synthetic or RUM sequence transactions, it will not show any data. To see statistics for sites, drill down from the Applications report as follows: 1. Click the application name on the Applications report. 2. Click the client tier name on the Tiers report for a selected application. For the Synthetic tier, you will see the Overview Application Status report; for the RUM sequence transactions tier, the Sequence Transactions Log report. 3. Depending on the type of report, click the Overview Site Status or the Sites tab. I see gaps on the chart reports. Why are the charts incomplete? Gaps in reports mean that the report server missed some data and was not able to get it into the database on time. Your reports may resemble the example below. Figure 6. Gaps in a Graphical Report There are several reasons why the graphical reports may have incomplete data: 76

77 Appendix A Diagnostics and Troubleshooting The AMD was not able to detect any traffic from the monitored network, so it was not able to produce any valid data for the report server. To confirm that this was the reason, connect to the AMD using an SSH client and check whether the files named zdata_xxxxx_x_x are located in the /var/spool/adlex/rtm directory. Similar symptoms can be observed if the AMD has been down for some time and data files were not produced for that time. If data files are present and the viewed chart displays only a fragment of the monitored traffic, for example, for a specific server or site, it may indicate that a part of traffic, which was indented to be monitored, is missing. In this situation, the data files are much smaller than usual for the corresponding period of the day. Similar situations, that is, gaps only on some reports, may occur in a multi-amd installation when some AMD s were down or disconnected from the network. In the case when only one AMD is connected to the report server, communication problems do not cause data gaps. If the report server cannot communicate with the AMD, it will wait until the communication is restored and then will process all the data from the past. When there are multiple AMDs connected to the report server and there is a break in communication with only some of them, the report server processes the data from the available AMDs, so in this case, gaps can appear on some reports. If it is a critical issue and your network (or its parts) require continuous monitoring and you cannot miss the data from some AMDs, you have to mark the AMDs as primary. In this case, the report server will wait until the communication with primary AMDs is restored, even if other AMDs are available. Gaps in charts on some reports in multi-amd installations may be caused by unsynchronized AMDs. The reason for that may be that if the report server sees a data file for a specific time period on one of the AMDs, it will wait only 30 seconds for data files covering the same period of time from other AMDs. The 30 seconds are the server's tolerance for time synchronization issues. To verify that this situation occurred, compare the clock readings from AMDs and then check the time synchronization settings (see The yellow triangle displays An AMD produces data stamped with a time from the future. What do I do? [p. 73]). It may happen that a part of data will be missing. This will result in a significant decrease of the aggregated data, used to render the chart bars. Note that this effect relates to metrics that are calculated as sums, for example, number of operations, number of errors, number of users, or bandwidth utilization. Charts showing the averages (RTT, loss rate, operation time) will not be affected. I see gaps on the log-term data chart reports. Why are the charts incomplete? The report server aggregates the data collected during the day into daily (and monthly) rollups. This is a scheduled process. If this process is not triggered, you will see gaps in the daily rollups. The most frequent reasons for missing rollups are: 77

78 Appendix A Diagnostics and Troubleshooting The report server was down in the night; report data generation starts at 12:10 AM local time and if the report server was down at that time, no aggregate data for long-term reports will be generated. The report server was overloaded and it took too much time for other crucial tasks; report data generation for long-term reports was canceled. You can always re-generate data for long-term reports. Open Control Panel by typing: in the Address field of the web browser and click Go, then select Regenerate Reports from the System Management section. I created a report that consists of several charts but it loads very slowly. How can I improve its performance? If you are using exactly the same set of dimensions and filters for every chart but would like to show different metrics on separate charts, there are two ways of improving such a report. In this example, it is assumed that you want a report that shows Client bytes, Server bytes, and Total bytes on separate charts for the HTTP analyzer. First, the simplest and recommended method, is to define one section that contains all these three metrics. Figure 7. Creating One Section with Three Metrics Open the Chart settings panel and from the single chart per list select Metric. If you are using metrics with different units, you can select the Metric unit option instead. For more information, see Displaying Multiple Charts in the Data Center Real User Monitoring Data Mining Interface (DMI) User Guide. The second method requires changes on the Subject Data and Result Display tabs. 1. For each report section (chart), create the same set of metrics. To do this, for each chart add metrics that are displayed on the other charts. Note that the order of metrics must be the same in every section. For example, each section must contain the Client bytes, Server bytes, and Total bytes metrics listed exactly in the same order. 2. Disable showing unnecessary metrics for each chart. 78

79 Appendix A Diagnostics and Troubleshooting Go to the Result Display tab and disable showing the redundant metrics. For example, for chart that is going to show only the Client bytes metric, disable showing the Server bytes and Total bytes metrics. Figure 8. Selecting Metrics to Display on a Chart Application performance and availability data is missing from the tabular reports. How can I fix this? The missing data manifests itself as zero or a hyphen. The most frequent reason for this situation is the incorrect setting of business hours and holidays. Inspect the business hours and holiday settings by choosing Settings Report Settings Business Hours. The following configuration screen shows the current settings. Figure 9. Business Hours Configuration Screen To collect performance data seven days per week, including non-business days and holidays, clear the Holidays check box and select the check boxes for weekend days. In addition, you can collect performance data in 24/7 mode, but be aware that this results in a higher database growth rate and a larger database. To enable collecting data all the time, open the Control Panel by opening the following page: In the Control Panel, click Advanced Properties Editor from the Configuration Management section. Set ONLY_BUSS_HOUR_REPORTING to OFF. 79

80 Appendix A Diagnostics and Troubleshooting To see whether your holiday definition is correct, click View Holidays. Figure 10. Defined Holidays Screen The list of holidays is hard-coded and the default set is for the USA. To select a set, click the Choose holiday definition list. To see the content of the selected set, click Preview. To store the newly selected set, click Save. 80