Monitoring Custom Applications User and Reference Guide

Transcription

1 Foglight Monitoring Custom Applications User and Reference Guide

2 2012 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser s personal use without the written permission of Quest Software, Inc. The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA legal@quest.com Refer to our Web site for regional and international office information. Trademarks Quest, Quest Software, the Quest Software logo, Foglight, IntelliProfile, PerformaSure, Spotlight, StealthCollect, TOAD, Tag and Follow, Vintela Single Sign-on for Java, and vfoglight are trademarks and registered trademarks of Quest Software, Inc in the United s of America and other countries. For a complete list of Quest Software s trademarks, please see Other trademarks and registered trademarks are property of their respective owners. Third Party Contributions Foglight contains some third party components. For a complete list, see the License Credits page in Foglight online help. User and Reference Guide February 2012 Version 5.8.0

3 Monitoring Custom Applications: User and Reference Guide 3 Table of Contents Introduction to this Guide...8 About Quest Software, Inc Contacting Quest Software... 8 Contacting Quest Support... 9 Monitoring Custom Applications...10 Selecting the Monitored Service Using the Custom Applications Monitor Dashboard Reviewing Alarms Managing Services Creating a New Service Examining the Structure of Custom Application Services Monitoring Systems and Servers...16 Monitoring Systems Viewing Systems Reviewing the Health of Your Systems Investigating System Component Health Monitoring Application Servers Viewing Servers Viewing JVM Data for Systems or Servers Monitoring Applications...28 Viewing Applications Monitoring Requests...32 Viewing Requests Managing Traces...36 Manually Collecting Single Traces Using a Rule Action to Collect Single Traces Using Groovy to Collect Single Traces Viewing Single Traces Traces View Trace Diagnosis View Selecting a Trace to Baseline or Compare Metrics... 48

4 Monitoring Custom Applications: User and Reference Guide 4 Table of Contents Using Object Tracking to Locate a Memory Leak Updating Object Tracking Agent Properties Calculating the LiveObjectLifespanLimit Value...52 Enabling Object Tracking Examining Object Tracking Data Reviewing Track Per Request Type Monitoring Methods Viewing Methods Collecting Metrics on Java Methods Java Common Rules JVM Deadlocked Threads Rule...59 JVM Garbage Collections Rule...60 JVM Heap Usage Rule Java EE Technologies Rules Application Request Rules Request Exceptional Exits Percent Rule Request Execution Time Trend Rule Request Incomplete Invocation Percent Rule Request Response Time Baseline Rule Request Response Time Threshold Rule Java EE Named Method Rules Java EE Named Method Group with Exceptional Exit Rule Java EE Named Method Group Execution Time Threshold JBoss Rules Deployments In Error Rule Deployments Not Started Rule...66 JBoss Application Instance Rule JBoss JCA Connection Pool Waiting Threads Rule...67 JBoss Server Rule JBoss System Thread Pool Queue Size Rule...68 JBoss Transaction Rollback Percent Rule JBoss Web Modules Rejected Sessions Rule JBoss Web Thread Pool Usage Rule JVM Memory Leak Detection by Object Count Rule...70 OracleAS Rules OAS Administrative Rollbacks Rule OAS Application Instance Rule OAS Application Rollbacks Rule OAS JCA Errors Rule... 73

5 Monitoring Custom Applications: User and Reference Guide 5 Table of Contents OAS JCA Waiters Rule OAS JDBC Waiters Rule OAS Java Message Service (JMS) Rules...74 OAS Pending Requests Rule OAS Resource Rollbacks Rule...76 OAS Server Rule OAS Timeout Rollbacks Rule OAS Web Site Rule Tomcat Rules Tomcat Application Instance Rule Tomcat Application Session Rejected Rule Tomcat Datasource Active Connections Rule...79 Tomcat Host Rule Tomcat Servlet/JSP Errors Rule Tomcat Thread Pool Errors Rule Tomcat Thread Pool Usage Rule WebLogic Rules WebLogic Application Instance Rule WebLogic Entity Bean Cache Thrashing Rule...82 WebLogic Entity Bean Pool Misses Rule WebLogic Execute Queue Thread Pool Usage Rule...84 WebLogic Health Rule WebLogic Invalid Login Attempts Rule WebLogic JDBC Connection Leak Count Rule...85 WebLogic JDBC Pool Usage Rule WebLogic JDBC Prepared ment Miss Percentage Rule...86 WebLogic JDBC Reconnect Failure Rule WebLogic JDBC Waiter Failure Rule WebLogic JMS Pending Bytes Rule WebLogic JMS Pending Messages Rule...89 WebLogic JMS Rule WebLogic JTA Rule WebLogic Locked Login Attempts Rule WebLogic Message Driven Bean Pool Misses Rule WebLogic Rollbacks - Application Rule WebLogic Rollbacks - Application (Named Transactions) Rule WebLogic Rollbacks - Resource Rule WebLogic Rollbacks - System Rule WebLogic Rollbacks - Timeout Rule WebLogic Server Rule WebLogic ful Session Bean Cache Thrashing Rule... 94

6 Monitoring Custom Applications: User and Reference Guide 6 Table of Contents WebLogic less Session Bean Pool Misses Rule...95 WebLogic Thread Pool Hogging Thread Rule...96 WebLogic Thread Pool Rule WebSphere Rules WebSphere Application Instance Rule WebSphere DynaCache Misses Rule WebSphere Entity Bean Pool Misses Rule WebSphere Message Driven Bean Pool Misses Rule WebSphere less Session Bean Pool Misses Rule WebSphere J2C Pool Usage Rule WebSphere Java Database Connectivity (JDBC) Rules WebSphere Transaction Rules WebSphere Server Rule WebSphere Message Listener Port Rule WebSphere Thread Pool Rules X-Agent Server Rule Custom Applications Monitor Views JVM View Method Groups View Request Types View Entity EJBs View Message Driven EJBs View ful Session EJBs View less Session EJBs View Deployed Applications View JSPs/Servlets Components View Resource Adapters Components View Web Applications Components View Web Services Components View JBoss Services Views JBoss JCA Resources View JBoss JMS Cache/Queues/Topics Views JBoss JTA View JBoss Thread Pools View Oracle Services Views OracleAS JCA Connection Pools View OracleAS JDBC Connection Pools View OracleAS JMS View OracleAS JTA View OracleAS Thread Pools

7 Monitoring Custom Applications: User and Reference Guide 7 Table of Contents Tomcat Services Views Tomcat Datasources Tomcat Thread Pools WebLogic Services Views WebLogic Execute Queues View WebLogic JDBC Data Sources View WebLogic JTA Runtime View WebLogic JMS Servers View WebLogic Security View WebLogic Self-Tuning Thread Pool View WebLogic Store and Forward View WebLogic Work Managers View WebSphere Services Views WebSphere Dynamic Cache View WebSphere JCA Factories View WebSphere JDBC Data Sources View WebSphere JTA Runtime View WebSphere Listener Ports View WebSphere SIB View WebSphere Thread Pools View JMX Views MBeans List View Index

8 Introduction to this Guide This Foglight Monitoring Custom Applications User and Reference Guide provides instructions and conceptual information about how to monitor and display Java EE Technologies metrics, and JMX metrics from JMX-enabled systems. In addition, this guide provides information about the Foglight Custom Applications Monitor dashboard, rules, and views. This guide is intended for any user who is interested in monitoring Java EE and JMX data with the Cartridge for Java EE Technologies or the Cartridge for JMX. About Quest Software, Inc. Quest Software (Nasdaq: QSFT) simplifies and reduces the cost of managing IT for more than 100,000 customers worldwide. Our innovative solutions make solving the toughest IT management problems easier, enabling customers to save time and money across physical, virtual and cloud environments. For more information about Quest solutions for administration and automation, data protection, development and optimization, identity and access management, migration and consolidation, and performance monitoring, go to Contacting Quest Software Mail Web site Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA USA Refer to our Web site for regional and international office information.

9 Monitoring Custom Applications: User and Reference Guide 9 Introduction to this Guide Contacting Quest Support Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to our support portal. Visit our support portal at support. From our support portal, you can do the following: Retrieve thousands of solutions from our Knowledge Base Download the latest releases and service packs Create, update, and review Support cases View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies, and procedures. The guide is available at: support.

10 1 Monitoring Custom Applications The Custom Applications Monitor dashboard provides a centralized location for monitoring your Java EE Technologies and JMX systems, servers, applications, requests, and methods. From this dashboard, you can view the health status of your monitored components, and investigate any alarms that occur. Note If you do not see the Custom Applications Monitor dashboard listed in the navigation panel, inform your Foglight System Administrator. Your Administrator can add your user account to one of the following role groups: Java EE Administrator, Java EE Read-only User, JMX Administrator, or JMX Read-only User. You must have one of these roles to be able to see the Custom Applications Monitor dashboard. To view the Custom Applications Monitor dashboard: On the navigation panel, under Homes, click Custom Applications Monitor. The Custom Applications Monitor dashboard opens. Note The tiles and views available on this dashboard depend on the cartridges you have installed. For example, the Applications, Requests, and Methods tiles are only available when you have installed the Cartridge for Java EE Technologies.

11 Monitoring Custom Applications: User and Reference Guide 11 Chapter 1 Monitoring Custom Applications Selecting the Monitored Service The Custom Applications Monitor dashboard displays services as defined by the default Custom Applications service category. The Custom Applications category contains all Java EE and JMX services. The first time you open the Custom Applications Monitor dashboard, you must select the service that you want to view. To select the monitored service to display: 1 On the Custom Applications Monitor dashboard, click the edit icon. A dialog box opens. 2 In the Service Selection dialog box, click the name of the service you want to display on the dashboard. Tip You may have to expand a service node to locate the service you want. The dialog box closes and the dashboard refreshes, displaying the selected service. You can select a different service to view at any time using the same procedure. You can also create your own services. For more information about Services, see the Foglight User Help. Using the Custom Applications Monitor Dashboard The Custom Applications Monitor dashboard organizes your Java EE Technologies and JMX monitoring data into views that are divided into five categories: Systems, Servers, Applications, Requests, and Methods. These categories are presented as tiles on the dashboard. Note The tiles available depend on the cartridges you have installed. For example, the Applications, Requests, and Methods tiles are not applicable to the Cartridge for JMX, and are therefore hidden when only JMX data is included in the selected service. Click a tile to access the views for that category. If you want to view only the components that have triggered alarms for that category, click the health status icon on the tile, as shown below.

12 Monitoring Custom Applications: User and Reference Guide 12 Chapter 1 Monitoring Custom Applications The health status icons act as filters for the category, allowing you to focus on trouble spots within your system. Show/Hide Table Columns This guide may refer to columns that are not visible by default. You can use the customizer menu to show all available columns, or to reduce the number of columns shown if the information is not pertinent to your monitoring needs. To show/hide columns: 1 Click the Customizer icon located at the top right-hand corner of any table. A list of the columns available to add to the table opens. 2 Select the columns you want to show or hide by selecting or clearing the appropriate check boxes. 3 Click Apply. Reviewing Alarms Regardless of which view you have selected on the Custom Applications Monitor dashboard, the upper portion of the dashboard always lists the details of the three most recent alarms, and a summary of current alarms, for the selected service. Use these links to access more detailed information about the alarms affecting your monitored services. To view the overall health summary: Click the large health status icon to the left of the service name. or Click any of the numbers in the Current Alarms list. The Health Summary popup opens. From here, you can acknowledge and clear alarms, or view details of an individual alarm.

13 Monitoring Custom Applications: User and Reference Guide 13 Chapter 1 Monitoring Custom Applications To review the details of one of the three most recent alarms: Click any part of the alarm information in the Recent Alarms list. The alarm details popup opens, allowing you to review and manage the alarm. For more information about managing alarms, see Viewing, Acknowledging, and Clearing Alarms in the Foglight User Help. Managing Services In Foglight, a service is defined as a grouping of one or more monitored components, such as Java Service Pages (JSPs), servlets, and Enterprise Java Beans (EJBs). Services can be nested within other services, each with their own monitored components. Custom Application (Java EE and JMX) services are organized by predefined types such as: all monitored WebLogic domains, WebSphere cells, OracleAS cluster topologies, JBoss partitions, Tomcat servers, or Web applications. Creating a New Service If the predefined services do not suit your monitoring needs, you can create one or more new services. For example, you may want to monitor only WebSphere cells, or only JBoss partitions. Note You must have the Foglight Advanced Operator role in order to create and edit services. To create a new service: 1 On the navigation panel, under Homes, click Custom Applications Monitor. 2 On the Custom Applications Monitor dashboard, click the edit icon. A dialog box opens. 3 In the Service Selection dialog box, click Add. A wizard opens. 4 In Step 1 of the Create Service from One or More Java Systems wizard, type a name and (optionally) a short or long description for the new service. 5 Click Next. 6 In Step 2, select one or more Java system components from the list. 7 In Step 3, on the same page of the wizard, select the service or service category that will contain the new service. 8 Click Finish. The wizard closes and the Service Selection dialog box refreshes. Click the name of the new service to display it in the Custom Applications Monitor dashboard. For more information about services, see Building a Service in the Foglight User Help.

14 Monitoring Custom Applications: User and Reference Guide 14 Chapter 1 Monitoring Custom Applications Examining the Structure of Custom Application Services The Custom Applications service category includes the All Java Systems service, which in turn can be comprised of one or more services. For example, the All Java Systems service could include a WebLogic service which monitors all WebLogic domains, a WebSphere server that monitors all web applications, an OracleAS service that monitors all OracleAS cluster topologies, a JBoss service that monitors all JBoss partitions, a Tomcat service that monitors all Tomcat servers, and a Java EE Web applications service that monitors all Java EE web applications. Each service has components or sub-components that are monitored by a rule. A component has its own alarms and Service Level Compliance (availability) status. The status of each component rolls up to provide an overall Service Level Compliance and alarm status for a service. From the Service Builder dashboard, you can explore how a service is built. You may want to examine this composition to see which components are contributing to the poor health. To examine how a service is built: 1 On the navigation panel, under Dashboards, click Services > Service Builder. The Service Builder dashboard appears. 2 Expand the Custom Applications service category. The grouping underneath the service category appears. In the example below, there are services for WebLogic_10, JBoss, and OracleAS. To see more detailed information: 1 Select the All Java Systems service. The All Java Systems Summary appears.

15 Monitoring Custom Applications: User and Reference Guide 15 Chapter 1 Monitoring Custom Applications The breakdown of the Java systems service is displayed with alarm information. From here you can investigate the Service Level Agreements (SLAs), examine the monitored components, and acknowledge and clear alarms. 2 Under the Explore heading, click Drilldown: Custom Applications Monitor to view the details of the service in the Custom Applications Monitor dashboard. For more information about Services and Service Level Agreements, see Monitoring Your Services in the Foglight User Help.

16 2 Monitoring Systems and Servers The Systems and Servers views of the Custom Applications Monitor dashboard are common to both the Cartridge for Java EE Technologies and the Cartridge for JMX. This chapter describes the features and functionality of these two views, and the related Java Virtual Machine (JVM) View. Monitoring Systems The Systems tile provides a high-level overview of the behavior of your systems. Use this view to review groups of monitored domains, cells, cluster topologies, and partitions. From here you can quickly see the number of servers and applications being monitored, their health, availability, and more. Select an object in the system summary table to view the components that belong to that system, such as servers and applications. When you hover over a component, a dwell displays all of its sub-components and any alarms associated with that system. From this view, you can perform the following actions: Review the number of monitored systems in your infrastructure Search for system types Identify the system aggregate state Review system servers total and unavailable counts Review system applications total and unavailable counts Collect Single Traces (from the action panel General tab, only when the Cartridge for Java EE Technologies is installed) Viewing Systems To view the Custom Application Systems: 1 On the navigation panel, under Homes, click Custom Applications Monitor. 2 Click the Systems tile.

17 Monitoring Custom Applications: User and Reference Guide 17 Chapter 2 Monitoring Systems and Servers Systems System Details Systems The Systems view displays the summary of all the monitored systems. Select one of the systems in the Systems table to populate the Servers Table and Applications Table in the System Details view. The System table contains a list of the systems running in your environment. This table also lists the total number of, and availability for, monitored servers and applications in your infrastructure. Use this table to quickly review the overall system health, and to determine if an error is occurring at the server or application level. Colored icons indicate the health status of the components: green checkmarks or upward arrows indicate a healthy state, while the red X or downward error indicates an alarm condition. A yellow caution triangle (not shown) indicates that the system is in a warning state and may become unhealthy. Select a system from the list to populate the System Details view on the right with data about the servers, applications, domains, cells, cluster topologies, or partitions included in the selected system. For example, in the image above, the health of the WebLogic_10 system is poor ( ), because one server and six applications are unavailable ( ). Click the status icon for an affected component (system, server, or application) to obtain information about the source. For more information, see Reviewing the Health of Your Systems on page 18. System Details The system details view displays details of the system selected in the System table. Depending on the type of system selected, this view may be divided into two tables: the Servers table (upper right), and the Applications table (lower right). For some systems, the Applications table is not applicable and is therefore hidden. To review the properties for the selected system, click Properties (located at the top right corner of the Systems Details view). The information available in this popup depends on the type of system selected.

18 Monitoring Custom Applications: User and Reference Guide 18 Chapter 2 Monitoring Systems and Servers Servers Table The Servers table lists the monitored servers for the selected domain, and provides you with at-aglance health information for the server and any monitored components. Click any entry to access a number of different drill-down views associated with the components. The columns in this table are dependent on the cartridges you have installed and the components you are monitoring. For example, for a system monitored by the JMX Agent, the Servers table includes MBeans and JVM status columns, and the Applications table is not available. Applications Table Note This table is only available with the Cartridge for Java EE Technologies. The Applications table lists all the applications running in a particular domain, cell, cluster topology, or partition and the stauts of related components such as EJBs, Web Applications, Web Services, and Resource Adapters. Click any entry to access drill-down views that provide detailed information about the selected component. For more information, see Reviewing the Health of Your Systems on page 18. Reviewing the Health of Your Systems You can obtain at-a-glance information about the health of a domain, cell, cluster topology, or partition, and quickly assess the health of components and sub-components, from the Systems view of the Custom Applications Monitor dashboard. To view the health of a system: Hover over any Health icon in the System table of the Systems view to display the Health Summary information for that system, domain, or application. The table lists all the alarms made for this system. Click any health icon to open the Health Summary dialog box, which lists the details of any alarms.

19 Monitoring Custom Applications: User and Reference Guide 19 Chapter 2 Monitoring Systems and Servers In this dialog box, select an alarm to view the full details, or to acknowledge or clear the alarm. Click any value related to an alarm to see more information. For example, click the agent name in the Agent column to review the agent health and related information. For more information about alarms, see the Foglight User Help. Example: Investigating Application Health from the Systems View In the following image, the medrec WebLogic Domain contains four applications, all of which are in poor health. To investigate: 1 In the Applications column of the Servers table, click the health status icon. A popup opens, displaying the health status of the applications broken down into categories (Deployed, Components, and EJBs), and sub-components (such as Web Applications and Web Services).

20 Monitoring Custom Applications: User and Reference Guide 20 Chapter 2 Monitoring Systems and Servers The status icon indicates that the problem is in the Deployed category. 2 Click the Deployed health status icon. The Health Summary view opens. 3 To review the details of an alarm, click the health status icon. The Alarm dialog box opens. Use this dialog box to review the alarm details, such as the full alarm message and its origin, and to acknowledge or clear the alarm. You can also drill down to more information about any named components (such as the host, agent, instance, or service) by clicking the component name. For more information about managing alarms in Foglight, see the Foglight User Help. For more information about the rules that trigger alarms in the Custom Applications Monitor dashboard, see Chapter 8, Java Common Rules and Chapter 9, Java EE Technologies Rules.

21 Monitoring Custom Applications: User and Reference Guide 21 Chapter 2 Monitoring Systems and Servers Investigating System Component Health The health of a system depends on the health of its servers, applications, and components (where applicable). Using the System view of the Custom Applications Monitor dashboard, you can drill down to investigate the health of an individual system component. For example, from the Servers table in the Systems view, you can see that a JBoss Java Virtual Machine (JVM) is experiencing a problem. To investigate: 1 Click the health status icon in the JVM column. A popup opens. Review the information in this popup. For example, check that the garbage collection rate and overhead shown in the GC Rate & Overhead chart are as expected. 2 Click the JVM name at the top of the popup. The Java Virtual Machine (JVM) View opens. In the JVM view, you can see that the warning health status is due to a warning level alarm. 3 Click the health status icon for the JVM to review the alarm details. For more information about the rules that generate alarms for the JVM, see Chapter 8, Java Common Rules. The JVM is just one example of a component for which there are drill-down views. Other examples include Web Applications, Web Services, EJBs, Requests, or Services. In all cases, you can access these drilldown views by clicking the name of an object or component in a health status popup, as described in the previous example. For more information about individual views, see Chapter 10, Custom Applications Monitor Views.

22 Monitoring Custom Applications: User and Reference Guide 22 Chapter 2 Monitoring Systems and Servers Monitoring Application Servers The Servers tile of the Custom Applications Monitor dashboard allows you to monitor specific application servers. From this view, you can investigate individual application servers in your internal and remote infrastructure and review the invoked operations used by these application servers. To learn more about the supported application servers, see the Managing Java EE Systems: Supported Platforms and Servers Guide or the Managing JMX Systems: User Guide. From this view, you can perform the following actions: Review the number of servers in your infrastructure Review the health of the servers Identify the application server type Review JVM Availability (average for the timeline range specified) Review JVM Heap Usage (average for the timeline range specified) Review JTA Rollback totals (Sum for the timeline range specified) Note JTA Rollback totals are available only for the Cartridge for Java EE Technologies. Review source metric charts in Quick View Collect Single Traces (from the action panel General tab, when the Cartridge for Java EE Technologies is installed) Viewing Servers To view the Custom Application Servers: 1 On the navigation panel, under Homes, click Custom Applications Monitor. 2 Click the Servers tile. Servers Server Details Quick View

23 Monitoring Custom Applications: User and Reference Guide 23 Chapter 2 Monitoring Systems and Servers Servers The Servers table lists all the application servers running in your environment. Use this table to review the health and version of any monitored application server, as well as the availability (up time), JVM Heap Usage, and JTA Rollbacks (if applicable). Select an application server in this table to display detailed information in the Server Details view on the right. Note In addition to the default columns, there are a number of columns you can add to this table. For information about how to select columns, see Show/Hide Table Columns on page 12. Server Details This view provides details about the server that you select in the Server table, including: Request Types, Method Groups, health of the JVM, Applications, any collected MBean components, EJBs, and Services. Note The details available in this view are relative to the type of server selected in the Servers table. Each monitored component in the server details view has a health status icon, and each component name is a link. To view the health summary for the component, click the status icon. For more information, see Reviewing the Health of Your Systems on page 18. To view more details about a component, click the name link. A popup or drilldown view opens, providing more information. For example, click JVM to open the Java Virtual Machine (JVM) View. Tip Use the breadcrumb trail at the top of any drilldown view to return to the Custom Applications Monitor dashboard. Quick View The Quick View option provides condensed charts that show the general state of a selected system. These charts answer questions such as what are the slowest three requests on server ABC, or what thread pools have the most waiters. Depending on the type of application server selected, the Quick View Selection menu will differ. The following table outlines the charts available for each application server type. JBoss Application Servers: Quick-View Chart Name Slowest Response Times Longest Wait Times Most Active Topics Highest Pool Usage % Request with longest response times on average. Queues with the longest wait times on average. Topics with largest incoming message rate on average. EJB pools with highest usage on average.

24 Monitoring Custom Applications: User and Reference Guide 24 Chapter 2 Monitoring Systems and Servers Quick-View Chart Name Slowest Service Times JSPs/Servlets with the lowest service times on average. OracleAS Application Servers: Quick-View Chart Slowest Response Times Longest Queues Longest Total Wait Times Slowest Service Times Requests with longest response times on average. Thread pools with largest number of requests in their thread pools on average. JSBC connection pols with the longest total time all requests spent waiting for a connection. JSPs/Servlets with the longest request times on average. Tomcat Application Servers: Quick-View Chart Requests - Slowest Response Times (ms) JVM - Heap Usage JSP/Servlets - Error Count (c) JSP/Servlets - Service Time (ms) Thread Pools - Highest Pool Usage Slowest response time for each request traced. Capacity and Used memory. JSP/Servlets with the highest number of errors. JSP/Servlets using the highest time on average. Thread pools which are being used most of the time. WebLogic Application Servers: Quick-View Chart Name Requests - Slowest Response Times (ms) JVM - Heap Usage Work Managers - Most Waiters EJBs - Highest Cache Miss % EJBs - Highest Pool Miss % JSPs/Servlets - Most Invocations Request with the longest response time on average. Capacity and Used memory. Work managers with the highest number of waiters in their queues on average. EJBs with the highest cache missed % on average. EJBs with the highest pool missed % on average. JSPs/Servlets with the most invocations.

25 Monitoring Custom Applications: User and Reference Guide 25 Chapter 2 Monitoring Systems and Servers Quick-View Chart Name Execute Queues - Most Pending Requests Local JMS Servers - Most Pending Messages JDBC Data Source - Most Connection Waiters Execute queues with the highest pending request count in their queues on average. JMS servers with the highest pending message count on average. JDBC data sources with the highest number of threads waiting for a connection on average. WebSphere Application Servers: Quick-View Chart Name Requests - Slowest Response Time (ms) JVM - Heap Usage Thread Pools - Highest Pool Usage % JDBC Data Source - Most Threads Waiting JSPs/Servlets - Most Popular JCA Pools - Most Waiters EJBs - Lowest Hit Ratio % Requests with longest response time on average. Capacity and Used memory. Thread pools with highest utilization on average. JDBC data sources with highest number of threads waiting for a collection. JSPs/Servlets accessed (requested) the most. JCA Pools with highest number of threads waiting on average. EJB pools with lowest hit ratios. To add or remove charts in Quick View: 1 Click Quick View, or the edit icon beside the text. The Quick View Selection dialog box opens. 2 Select any of the source metrics listed in the dialog box. 3 Click OK to close the dialog box. The Quick View refreshes.

26 Monitoring Custom Applications: User and Reference Guide 26 Chapter 2 Monitoring Systems and Servers Viewing JVM Data for Systems or Servers You can access Java Virtual Machine (JVM) details for an MBean Server from either the Systems view or the Servers view of the Custom Applications Monitor dashboard. To access JVM data from the Systems view: 1 Select a system from the System table on the left. 2 In the system details view on the right, click the icon in the JVM column. A popup appears, displaying Heap Usage and GC Rate & Overhead charts, as well as basic JVM information. 3 Click the JVM Name or the Heap Usage chart to open the Java Virtual Machine (JVM) View. Tip Use the breadcrumb trail at the top of the view to return to the Custom Applications Monitor dashboard. To access JVM data from the Servers tile: 1 Select a server in the Server table on the left. 2 In the server details view on the right, do one of the following: Hover your mouse over the JVM link in the server details area to open a dwell that displays the Heap Usage and GC Rate & Overhead charts. Click the Heap Usage chart in this dwell to open the JVM view. Click JVM in the server details view. The JVM view opens. Click Quick View, located just below the server details view, and select JVM - Heap Usage from the dialog box that opens. Click OK. The JVM Heap Usage chart appears in the Quick View area. Java Virtual Machine (JVM) View The top portion of this view lists the System, Server, and Health information. Click any status icon to review the health of that component. For more information about the rules that determine the Java Virtual Machine (JVM) health status, see Chapter 8, Java Common Rules.

27 Monitoring Custom Applications: User and Reference Guide 27 Chapter 2 Monitoring Systems and Servers The charts in the middle section give you a quick view of the JVM Heap Used and Committed, Garbage Collection (GC), and Process Metrics, including: the amount of memory used and available the number of garbage collection occurrences per second (rate) and the amount of time spent performing garbage collections compared to the interval that Foglight is reporting (overhead) the amount of CPU the threads are using In addition to viewing the heap usage of your JVM, it is important to understand how frequently garbage collection is being performed (GC Rate) and the percentage of time your JVM is spending performing garbage collection (GC Overhead). To see more details about the JVM, click one of the tabs in the JVM view below the charts. For more information, see JVM View on page 109.

28 3 Monitoring Applications Note Applications monitoring is available only for the Cartridge for Java EE Technologies. The Applications tile of the Custom Applications Monitor dashboard lists all the applications running in a monitored environment (domain, cell, cluster topology, or partition) and the related components, such as EJB, Web Applications, and the Web Services health status. From here you can drill down to explore the components in further detail. Use this view to investigate the applications running in your infrastructure. Select an application in the table to display a list of the various types of components that are running in a specific application and their deployment details. From this view, you can perform the following actions: Review the number of applications running in your infrastructure Identify the application type Identify the availability (average for the timeline range specified) Identify the application deployments (number of servers the application is deployed on) Select a health status icon for a component to display a dwell listing the various types of that component that are running on a specific application Drill down to explore the component type in further detail Review source metric charts in Quick View Collect Single Traces (from the action panel General tab) Viewing Applications To access the Custom Application Applications view: 1 On the navigation panel, under Homes, click Custom Applications Monitor. 2 Click the Applications tile.

29 Monitoring Custom Applications: User and Reference Guide 29 Chapter 3 Monitoring Applications Applications Table Components and Deployment View Quick View Applications Table The Applications table lists all monitored applications in your environment. Click the health status icon to review the health summary for an application. By default, only the Health, Name, and Deployments columns are visible. To add the Type and System columns, see Show/Hide Table Columns on page 12. Components and Deployment View The Components and Deployment view lists the availability and health for the selected application, its components, EJBs, and deployments. Click any object link in this view to access a health summary for that object. Some of the health summaries allow you to drill down to more detailed views by clicking the name of the object in the health summary popup you are viewing. For example, for a WebLogic Application that has less Session Beans, you can drill down to the instances by clicking the link less Session, and then clicking the name less Session on the health summary popup, as shown below. Tip Use the breadcrumb trail at the top of any drilldown view to return to the Custom Applications Monitor dashboard.

30 Monitoring Custom Applications: User and Reference Guide 30 Chapter 3 Monitoring Applications Properties You can also view the properties for the selected application by clicking the Properties link at the far right of the Components and Deployment view. The following server property information is available: Name Type Version Is Active Retire Time Retire Timeout Deployment Order Displays the type of application. Displays the version number of the application. Note n/a is displayed if the deployment descriptor element is not defined on the application server. An application server can have several versions of an application installed. A value of true indicates that the application is the active version (that is, the one that new sessions will use). Applications require upgrading to the latest version from time to time. This value indicates the retirement time of an application. (milliseconds) The amount of time an older application is given to retire (seconds). The number of applications depending on this application server. Quick View The Quick View option provides condensed charts that show the general state of a selected system. These charts answer questions such as what are the JSP/Servlets with the most invocations, or what is the average pool usage for an EJB instance. The Quick View menu options available depend on the type of application server selected. The following table outlines the charts available for each application type. JBoss Applications: Quick-View Chart EJBs - Highest Pool Usage % JSP/Servlets - Slowest Service Times Average JBoss EJB Instance pool misses for a specified period. JSP/Servlets with the longest request times on average. OracleAS Applications: Quick-View Chart JSP/Servlets - Slowest Service Times JSP/Servlets with the longest request times on average.

31 Monitoring Custom Applications: User and Reference Guide 31 Chapter 3 Monitoring Applications Tomcat Applications: Quick-View Chart JSP/Servlets - Error Count JSP/Servlets - Service Time (ms) JSP/Servlets with the highest number of errors. JSP/Servlets using the highest time on average. Application - Cache Hit Ratio WebLogic Applications: Quick-View Chart EJBs - Highest Pool Miss % JSP/Servlets - Most Popular Average WebLogic EJB Instance pool misses for a specified period. JSP/Servlets with the most invocations. WebSphere Applications: Quick-View Chart EJBs - Lowest Hit Ratio JSP/Servlets - Most Popular EJB pools with the lowest hit ratios. JSP/Servlets with the most invocations. To add or remove charts in Quick View: 1 Click Quick View, or the edit icon, located at the bottom right corner of the Components and Deployment view. The Quick View Selection dialog box opens. 2 Select any of the Source Metrics listed in the table. 3 Click OK to close the dialog box. The Quick View refreshes.

32 4 Monitoring Requests Note Request monitoring is available only for the Cartridge for Java EE Technologies. The Custom Applications Monitor dashboard includes predefined views for monitoring and investigating alarms associated with service requests made to your applications. Use the Requests view to investigate unique request types, such as requests made through HTTP, JMS, RMI, or JNDI, and to access full details of those request types. In addition, this interface allows you to investigate single traces in full detail or at component detail to assist you in localizing a potential trouble spot. Service requests arrive via HTTP, or RMI and JMS calls. The requests initiate a chain of events, potentially invoking numerous EJB methods and database calls, before returning a result. The entire related chain of events is defined as a service request in Foglight. Request data appears in the Custom Applications Monitor > Request view. You can also monitor requests and reports on response time through this dashboard. An alarm fires when a request falls outside of its predefined conditions, and the alarm appears in the Custom Applications Monitor dashboard > Requests view. The Requests view shows performance information about a selected service request. From this view, you can perform the following actions: Review all service requests coming from external clients, using HTTP, RMI or other communication protocols. Review Response Time/Call Count, Sampled Breakdowns, and Uncompleted/Exceptional Exits charts. Collect Single Traces (from the action panel General tab) Typical Request Issues that Generate Alarms Typical problems that show up as alarms include: Excessive execution times Unacceptable number of exceptions Unacceptable number of incomplete invocations

33 Monitoring Custom Applications: User and Reference Guide 33 Chapter 4 Monitoring Requests Viewing Requests Use the Requests view to review performance information about all of your unique request types. To view Custom Application Requests: 1 On the navigation panel, under Homes, click Custom Applications Monitor. 2 Click the Requests tile. Requests Table Chart Area Requests Table The Requests table lists all service requests coming from external clients, using HTTP, RMI or other communication protocols. For example, in the Medical Records sample application that ships with WebLogic server, the table displays HTTP Get methods for log in, log out, record requests, and administration activities. For each service request, the Requests table displays the following information. Data Item Name Health System Name of the service request. Health state of how the service requests are processing. You can click the icon in the Health column to obtain alarm information. System information including name, ID, and domain, cell, cluster topology, or partition type. Hover over or click the system name located at the top of the dwell to display the Domain details.

34 Monitoring Custom Applications: User and Reference Guide 34 Chapter 4 Monitoring Requests Data Item Context Path Completed Calls Incomplete Calls Incomplete % Current Response Time (s) Average, Max, and Total Response Time (s) Exceptional Exits, and Exceptional Exits % Traces The context path provides the Service request method and resource name, mapping, and context path. The total number of complete calls to a request. The total number of incomplete calls to a request. The percentage of incomplete calls to a request. The current column provides a quick history bar that shows the Current Response Time without requiring deeper diagnostics. Response time, including the average, minimum, maximum, and total response times. Percentage of requests that are incomplete and that resulted in an exception. The number of single traces that have been captured. You can click the data in the Traces column to view single trace details. For more information, see Managing Traces on page 36. Actions The Actions icon opens a menu with two options: Collect Traces - opens the Collect Traces dialog box, where you can configure single trace collection. Generate Report - allows you to generate a single trace report. Note Some of the columns described are not shown by default. To show or hide columns, see Show/ Hide Table Columns on page 12. The request you select from the Request Types table controls what data appears in the Charts area. By default, the first item in the Request Types table is selected. Chart Area The chart area contains graphs and tables of the execution time data for the selected request over the selected time period. Use the tabs to access more information about Response Time/Call Count Tab, Sampled Breakdowns Tab, and Incomplete/Exceptional Exits Tab. Response Time/Call Count Tab The graphs display the historical response time and call count of a request. For each selected request, the graph plots the average, maximum, and minimum values. Adjusting the timeline zonar causes the graphs to be updated with the data collected during that time period.

35 Monitoring Custom Applications: User and Reference Guide 35 Chapter 4 Monitoring Requests Sampled Breakdowns Tab There are four types of Sampled Breakdowns: Tier, Component Technology, Application, and Server. These types are described individually, below. The Sampled Tier Breakdown graph shows the distribution of the response time over different tiers, such as the application servers and database. The results are displayed as a stacked graph, with each tier represented by a different color. The Sampled Component Technology Breakdown graph shows the execution time data for the selected request broken down by component technology. The response time is displayed in segments and each segment represents the time spent in each Java EE Component Technology (for example, HTTP, Servlet, EJB, JDBC, JNDI, or Custom Components). This information helps you accurately determine where to focus your application performance-tuning efforts. The results are displayed as a stacked graph, with each component technology represented by a different color. The Sampled Application Breakdown is a stacked graph that shows the response time by the various applications that serviced the request. The Sampled Server Breakdown is a stacked graph that shows the distribution of response time over different application servers. These types of breakdowns are important because they show you if your application is spending more time in your application code versus your database. If a request passes between applications, the Sampled Application Breakdown graph shows you how much time is spent in each application. For example, if your application calls to another application for credit card validation, both applications are displayed in the Sampled Application Breakdown graph. Incomplete/Exceptional Exits Tab The Incomplete Requests chart displays the count of incomplete service requests that timed out during the interval spanned by the zonar. Foglight considers a request incomplete when the request does not complete processing within the timeout specified in the recording.config property file. The Exceptional Exits chart displays the count of requests that completed with an uncaught exception during the interval spanned by the zonar.