WebSphere Studio Application Monitor V3.2

Transcription

1 Front cover WebSphere Studio Application Monitor V3.2 Advanced Usage Guide Overall J2EE application management and monitoring Composite application monitoring with CICS and IMS Distributed and z/os platforms covered Budi Darmawan Carter Brooking Nichole Cargill Bart Jacob Jason Meiers Karel Michek Kimberly Nguyen-Theunissen ibm.com/redbooks

2

3 International Technical Support Organization WebSphere Studio Application Monitor V3.2 Advanced Usage Guide July 2005 SG

4 Note: Before using this information and the product it supports, read the information in Notices on page vii. First Edition (July 2005) This edition applies to Version 3, Release 2 of WebSphere Studio Application Monitor (product numbers 5724-K79, 5697-J18, 5697-J17, and 5697-J01). Copyright International Business Machines Corporation All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Notices vii Trademarks viii Preface ix The team that wrote this redbook ix Become a published author xi Comments welcome xii Chapter 1. Introducing application management Application management challenges Tivoli application management solution Project overview Project environment Chapter 2. WebSphere Studio Application Monitor V3.2 overview WebSphere Studio Application Monitor overview Architecture and interconnection The managing server Data collector Supported environment Monitoring features Administration Availability Problem determination Performance analysis New features in V Memory Leak Analysis Lock analysis WebSphere Portal Server metrics z/os data collector changes WebSphere Application Server 6 Support Trap and alert management Help system is now using Eclipse InfoCenter format Performance enhancement Chapter 3. Managing WebSphere Studio Application Monitor Overview Security considerations Node authentication default properties Copyright IBM Corp All rights reserved. iii

6 3.2.2 Users and roles Server management Server groups Data collector configuration Monitoring level considerations Sampling rate considerations Database management Migration from V Database backup Maintaining database performance Cleaning up the database Managing server parameters Component heap sizes JDBC driver properties Kernel host properties Port definitions Essential managing server tools Split server installation Data collector administration Configuration files for data collector Log-level settings Data collector log files cleanup Chapter 4. \Managing production environment Production environment definition An operator s typical day Reports Report types Performing capacity planning Report usage Performance tuning Development perspective Administration perspective Runtime perspective Notification Working with composite applications CICS composite application IMS composite application Chapter 5. Analyzing WebSphere application problems Problem scenarios Memory leaks Symptoms iv WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

7 5.2.2 Investigating a potential memory leak Narrowing the cause of a memory leak Determining the memory leak offender Resolution About this scenario Deadlock management Planning the problem resolution Performance analysis Lock Analysis preparation Lock analysis Historic lock information About this scenario Analyzing a slow-performing application Planning problem resolution for a slow application Performance analysis for a slow-performing application Identification of affected modules and servers Comparing system resources About the scenario Database performance Planning the problem resolution Analyzing the affected data source Re-creating the problem Application post-deployment activities About the scenario Chapter 6. Analyzing composite application problems Problem scenarios CICS composite transaction About this scenario IMS composite transaction About this scenario CICS resource enqueue Symptoms of resource enqueuing Investigating CICS resource enqueue About this scenario CICS transaction loops Symptoms of looping transaction About this scenario CICS DSA growth Monitoring storage growth About the scenario Appendix A. Sample scenario setup Contents v

8 Environment overview WebSphere applications used Memory leak sample Locking application sample Distributed Application Sample CICS application setup Trader CICS application Setting up CICS for TraderCICS Connecting WebSphere for TraderCICS IMS application setup Trader IMS application Setting up IMS for TraderIMS Connecting WebSphere for TraderIMS Putting in the load WebSphere Studio Workload Simulator Rational Robot Sample Java program Appendix B. Installation tips Embedded installation problems Composite transaction problems Appendix C. Additional material Locating the Web material Using the Web material System requirements for downloading the Web material How to use the Web material Abbreviations and acronyms Related publications IBM Redbooks Other publications Online resources How to get IBM Redbooks Help from IBM Index vi WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

9 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. Copyright IBM Corp All rights reserved. vii

10 Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX 5L AIX CrossWorlds CICS DB2 DB2 Universal Database IBM ibm.com IMS MVS NetView OMEGAMON Rational Redbooks Redbooks (logo) Tivoli Tivoli Enterprise Console TME WebSphere z/os zseries The following terms are trademarks of other companies: Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel is a registered trademark of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. viii WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

11 Preface The addition of WebSphere Studio Application Monitor (WSAM) to the Tivoli product portfolio enhanced Tivoli offerings for application management. WebSphere Studio Application Monitor enables comprehensive monitoring and management of J2EE-based applications with their back-end platforms. This IBM Redbook demonstrates advanced uses of WebSphere Studio Application Monitor to solve day-to-day operational problems. The book is intended for system administrators who want to use WebSphere Studio Application Monitor for monitoring Web-based applications. We do not discuss implementation issues in this book. We present WebSphere Studio Application Monitor concepts in Chapter 2, WebSphere Studio Application Monitor V3.2 overview on page 7, and offer usage details such as: System administration of day-to-day operations, from grouping and security design to maintenance tasks, in Chapter 3, Managing WebSphere Studio Application Monitor on page 33. Basic usage for a production environment in Chapter 4, \Managing production environment on page 67. Problem determination for WebSphere-based applications, demonstrated to assist with some common problems that may occur in your environment, in Chapter 5, Analyzing WebSphere application problems on page 99. Management and analysis of composite applications (which span multiple application servers and can be hard to manage) across J2EE servers, CICS, and IMS in Chapter 6, Analyzing composite application problems on page 145. The team that wrote this redbook This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center. Budi Darmawan is a Project Leader at the International Technical Support Organization, Austin Center. He writes extensively and teaches IBM classes worldwide about all areas of Tivoli systems management products. Before joining the ITSO six years ago, Budi worked in IBM Indonesia Integrated Technology Solution as lead solution architect and performer. His expertise ranges from Copyright IBM Corp All rights reserved. ix

12 distributed systems to mainframe solutions, and he has worked with various Tivoli Management Framework based solutions, z/os -based systems management solutions, DB2 in multiple platforms, and Web-based management solutions. His current interests are business service management, performance management, and Java programming. Carter Brooking is a Customer Solutions Engineer on the Tivoli SWAT team in Texas. He has five years of experience in the Composite Application Management field focusing on IBM Tivoli Monitoring for Transaction Performance and Java development. He holds a computer science degree from Rice University. His areas of expertise include Java programming and database design. Nichole Cargill is a Senior Accredited IT Specialist on the Americas, Tivoli Composite Application Management (ITCAM) technical support team in Illinois. She has worked at IBM for three years with a primary focus on installation and configuration of WebSphere Studio Application Monitor for WSAM Proof of Concepts. She holds a four-year degree in Business Management, two-year degree in Computerized Business Systems/Accounting, and a two-year degree in Data Processing Technology. Prior to her employment at IBM, she spent nine years employed with Caterpillar, Inc., assuming many different positions including: Enterprise Application Integration (EAI) standard, design, and implementation; WebSphere MQ and CrossWorlds (now known as WebSphere Business Integration) implementation and support; COBOL programming: system support; and operations support. Bart Jacob is a Senior Consulting IT Specialist in the IBM International Technical Support Organization, Austin Center. He has 25 years of experience providing technical support across a variety of IBM products and technologies, including communications, object-oriented software development, and systems management. He has more than 10 years of experience at the ITSO, where he has been writing IBM Redbooks and creating and teaching workshops around the world on a variety of topics. He holds a Masters degree in Numerical Analysis from Syracuse University. Jason Meiers is a Software Engineer on the Application Management Solutions team in San Francisco, California, with two years of experience in the J2EE Application Management field. Jason holds a degree in Information Systems from University of Applied Sciences Karlsruhe, Germany. His areas of expertise include J2EE and WebSphere. Jason has written extensively about describing advanced usage, configuration, and tuning of the managing server, data collector, and octigate database migration and configuration. Karel Michek is a Solution Architect at IBM Czech Republic. His areas of expertise include object oriented analysis and design with UML, enterprise Java programming, and database design. Karel has been focusing on the WebSphere x WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

13 family of products primarily in business integration area. He holds a Master s degree in computer science from Czech Technical university in Prague, and a number of IBM (WebSphere) and SUN (Java) certifications. Kimberly Nguyen-Theunissen is a senior software engineer at IBM working as a Customer Solutions Engineer on the Tivoli SWAT team in Austin. She has 11 years of experience in the software research and development field. Her areas of expertise include systems management and database administration and, currently, architecture and deployment of IBM Tivoli Monitoring product solutions. She holds a degree in Computer Science and Engineering from Louisiana State University. Thanks to the following people for their contributions to this project: Robert Haimowitz, Richard M. Conway, Joe DeCarlo, Betsy Thaggard International Technical Support Organization Alan Sampson, Arun Biligiri, Breet Bangle, Donna Martin, Jean-Jacques Heler, Nigel Williams, Rebecca Poole, Richard Mackler, Sanjay Sood, Sushanto Pandit, Tian Ming Pan Tivoli Application Management team Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You will team with IBM technical professionals, Business Partners, and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at the ibm.com Redbooks Web site: ibm.com/redbooks/residencies.html Preface xi

14 Comments welcome Your comments are important to us because we want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an to: Mail your comments to: IBM Corporation, International Technical Support Organization Dept. 0SJB Building Burnet Road Austin, Texas xii WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

15 1 Chapter 1. Introducing application management This chapter discusses application management in the overall system management context. The discussion is divided into the following sections: 1.1, Application management challenges on page 2 1.2, Tivoli application management solution on page 3 1.3, Project overview on page 4 Copyright IBM Corp All rights reserved. 1

16 1.1 Application management challenges The current open infrastructure with Web-enabled applications poses a challenge on monitoring and performance management. These applications are typically either new or evolved mainframe applications. A new application can be developed or purchased. This kind of application runs mainly as a Java-based Web solution residing on one or more Java 2 Enterprise Edition (J2EE) application servers with underlying relational database access. An evolved mainframe application is based on an existing mainframe-based application that runs on the CICS or IMS platform, and is converted to Web-enabled using various methods. This kind of application uses the business logic and data store from the existing application. As there are more components than for a typical mainframe application that runs on a mainframe machine or a cluster of mainframe machines (also called a sysplex or system complex), a different paradigm is needed to manage these Web-enabled or Web-based applications. Further complications arise from the need to correlate and analyze data from various application servers and platforms and present it in an integrated way so that the performance information can be analyzed. The performance analysis can range from simple regular monitoring to capacity planning to performance tracing of an application. Figure 1-1 shows this situation. J2EE application servers Backend servers Web servers Figure 1-1 Application management challenges 2 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

17 Managing Web-based and Web-enabled applications requires a deep analysis of Java performance in the applications. J2EE specifications provide some management capability such as JVMPI and JMX APIs, but these solutions must provide comprehensive information about the application. The application management solution must provide both snapshot information and performance history to efficiently analyze an application. Performance solutions should be able to use a snapshot to see which resource is mostly used by what module. A small memory leak is best analyzed using historical heap analysis. Composite application management requires an application management solution that can monitor multiple application servers, including mainframe applications, and correlate the composite application execution to show the complete end-to-end status of the application. The management of the application also must be able to monitor application availability from an end-user perspective and simulate the end-user experience for accessing the application. All of these application-management requirements can be solved using various Tivoli solutions, as we discuss in the next section. 1.2 Tivoli application management solution The IBM Tivoli composite application management solution helps you quickly isolate, diagnose, and fix business-critical transaction performance problems. Proactive monitoring of large-scale deployments and resolving bottlenecks are essential for On Demand Businesses and can increase availability for Web applications, portal solutions, and SOA-based solutions. You can visually follow entire transactions, end-to-end, to quickly pinpoint the source of problems. The root cause of a problem can then be determined using deep-dive, real-time diagnostic capabilities. Method-level diagnostics enable the specific code problem to be passed to the architect or developer for rapid correction. Business processes are increasingly dependent on multi-tier composite applications that use business logic and data from multiple resources: Web servers, J2EE application servers, integration middleware, and mainframe systems. However, building, running, and managing composite applications is increasingly difficult in such a complex environment. Using traditional tools, these problems are often not even identified until a customer complains. This solution helps resolve performance bottlenecks for multi-platform J2EE applications, WebSphere Application Server, WebSphere Portal, WebSphere Chapter 1. Introducing application management 3

18 MQ, WebSphere Business Integration Message Broker, WebSphere InterChange Server, BEA WebLogic, CICS, and IMS. The Tivoli application management solution consists of a complete end-to-end application management solution. It comprises management of Web-based applications, transaction monitors, ERP applications, and mainframe applications. Products that manage various application roles include: Web-based applications WebSphere Studio Application Monitor IBM Tivoli OMEGAMON XE for WebSphere ITM for Web Infrastructure Middleware IBM Tivoli OMEGAMON XE for WebSphere Business Integration Transaction monitors ITM for Transaction Performance: end-to-end transaction tracking that identifies and isolates problems for transaction requests ERP applications ITM for Applications Mainframe applications OMEGAMON XE for CICS OMEGAMON XE for IMS OMEGAMON XE for DB2 1.3 Project overview This redbook discusses management of J2EE-based application servers with back-end systems using WebSphere Studio Application Monitor V3.2. We assume that the reader is familiar with Web-based applications and basic Java programming terms. This book is not intended to cover basic installation and configuration aspects of WebSphere Studio Application Monitor. We demonstrate the use of WebSphere Studio Application Monitor on managing applications running on WebSphere Application Server, with some analysis with CICS or IMS back-end systems. Our objective is to demonstrate some advanced functions that you can perform using WebSphere Studio Application Monitor. 4 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

19 The chapters in this book are: Chapter 2, WebSphere Studio Application Monitor V3.2 overview on page 7 provides a brief introduction to the product and its functions for those who are not very familiar with WebSphere Studio Application Monitor. We also discuss new features that are available in Version 3.2. Chapter 3, Managing WebSphere Studio Application Monitor on page 33 shows the maintenance aspect of WebSphere Studio Application Monitor. It guides WebSphere Studio Application Monitor administrators to perform regular maintenance to keep the product running well for managing the environment. Chapter 4, \Managing production environment on page 67 serves as a baseline monitoring recommendation for application administrators and operators to perform regular monitoring of an application in a production environment. Here, we discuss screen recommendations, report selection, and other regular operation procedures. Chapter 5, Analyzing WebSphere application problems on page 99 builds on the regular operation to show some sample problem scenarios and ways to use WebSphere Studio Application Monitor to solve them. This chapter is for WebSphere Application Server based applications. Chapter 6, Analyzing composite application problems on page 145 also builds on the regular operation to show sample problem scenarios and ways to use WebSphere Studio Application Monitor to solve them. This chapter is for composite applications that access IMS or CICS on the back end. Our analysis discussions use scenarios that consist of applications that have problems implanted, which we find by using WebSphere Studio Application Monitor. We discuss these applications in Appendix A, Sample scenario setup on page 177. The application sources and samples are provided in Appendix C, Additional material on page 191. Appendix B, Installation tips on page 187 covers some common installation problems and tips. Chapter 1. Introducing application management 5

20 1.4 Project environment This section shows the various machines that we use in building the scenarios and performing tests. Figure 1-2 shows the overall environment. Network deployment environment khartoumnetwork Lima bandung Windows 2000 Server DB2 V8 WebSphere WSAM 3.2 MS Windows 2000 Server DB2 V8 WebSphere WSAM 3.2 Data Collector Santiago Windows 2000 Server DB2 V8 WebSphere WSAM 3.2 Data Collector peoria Windows 2000 Server DB2 V8 WebSphere WSAM 3.2 MS khartoum Windows 2000 Server DB2 V8 WebSphere ND WSAM 3.2 Data Collector Perth AIX 5.2 DB2 V8 WebSphere WSAM 3.2 Data Collector bdsr01a z/os WebSphere 5.1 WSAM 3.2 Data Collector SC58 wtsc58.itso.ibm.com SCSCPA2B z/os CICS TS 2.3 CICS TG v5 WSAM 3.2 Data Collector IMSB z/os IMS v8 IMS Connect 2.2 WSAM 3.2 Data Collector Figure 1-2 Project environment 6 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

21 2 Chapter 2. WebSphere Studio Application Monitor V3.2 overview This chapter provides an overview of the functions and facilities of WebSphere Studio Application Monitor V3.2. The discussion includes: 2.1, WebSphere Studio Application Monitor overview on page 8 2.2, Architecture and interconnection on page 8 2.4, Monitoring features on page , New features in V3.2 on page 23 For more information about installing WebSphere Studio Application Monitor, refer to WebSphere Studio Application Monitor Version 3.2 Installation Guide, SC , and Installing WebSphere Studio Application Monitor V3.1, SG We also provide some tips for common installation problems in Appendix B, Installation tips on page 187. Copyright IBM Corp All rights reserved. 7

22 2.1 WebSphere Studio Application Monitor overview WebSphere Studio Application Monitor (WSAM) observes and reports on the health of J2EE-based applications. It tracks the progress of applications as they traverse through J2EE application servers, middleware adapters and transports, and database calls and on to back-end systems such as CICS or IMS to extract business data or to invoke mainframe business processes. The tracking of applications produces request traces, where the events in a request s life are recorded and stored in a monitoring repository database. WebSphere Studio Application Monitor captures the CPU and the elapsed internal times when events are called and when they are exited, measuring as far down as the CPU times consumed and the elapsed internal times charged to individual methods in J2EE classes. The methods or events that take the most time are marked as an application s parts that deserve attention for runtime improvement studies and code optimizations. WebSphere Studio Application Monitor does not need modification of any J2EE or mainframe application code. Java Virtual Machine Tool Interface (JVMTI) interfaces and primitives, along with WebSphere Performance Management Interface (PMI) and z/os System Measurement Facility (SMF) 120 records, are WebSphere Studio Application Monitor s principal data sources. The monitoring data is collected and analyzed to offer a wealth of information about the health of J2EE applications and their servers. Many system-level performance metrics are collected and reported about J2EE application servers. The status of the servers and their resources, particularly at vital checkpoints such as CPU utilization, memory usage, and the status of internal components (database connection pools, JVM thread pools, EJB usage, request processing statistics) can be very important in locating real-time problems with J2EE applications. WebSphere Studio Application Monitor brings attention to these critical indicators with real-time, graphical displays of their values and their trends over spans of time. 2.2 Architecture and interconnection WebSphere Studio Application Monitor is a distributed performance monitoring application for application servers. Its components are connected through TCP/IP communication. The central component of WebSphere Studio Application Monitor is called the managing server. This is the heart and brain of WebSphere Studio Application Monitor. It collects and displays various performance information from application servers. 8 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

23 The application servers run a component of WebSphere Studio Application Monitor called the data collector. This is a collecting agent that runs in the application server and sends monitoring information to the management server. These data collectors operate independently from each other. Figure 2-1 shows the overall architecture of WebSphere Studio Application Monitor. IBM WebSphere Studio Application Monitor Managing Server Browser interface Web Server Application servers with WSAM Data collectors Figure 2-1 WebSphere Studio Application Monitor architecture A data collector agent runs on each monitored application server: J2EE, CICS or IMS. It communicates essential operational data to the managing server. Unique sampling algorithms maintain low CPU and network overhead while providing application-specific performance information. The managing server is composed of several Java-based components that provide the environment to collect and present management data. Chapter 2. WebSphere Studio Application Monitor V3.2 overview 9

24 2.2.1 The managing server The WebSphere Studio Application Monitor managing server controls and coordinates data collectors for J2EE, CICS, and IMS servers that run applications. The managing server uses the following software: The X-Windows Virtual Frame Buffer (Xvfb) graphics package Managing server database (DB2 UDB or Oracle on Sun Solaris) for relational data repository J2EE server to run the WebSphere Studio Application Monitor graphical console application An optional Apache Web server, such as IBM HTTP Server WebSphere Studio Application Monitor managing server overseer components, which are a set of Java-based components The WebSphere Studio Application Monitor overseer components are the controlling logic for the managing server. They are: Kernels control the managing server. There are always two copies of the kernels running on a WebSphere Studio Application Monitor managing server for redundancy and failover. The kernels register components as they join the managing server, periodically renew connections and registrations with components and data collectors, and collect server and component availability information. Publishing servers receive application and system event data from the data collectors, gather and compute request-level information about performance metrics such as response times, and implement the trap monitoring and alerts features. Archive agents receive monitoring data from the publish servers, and store the monitoring data in WebSphere Studio Application Monitor s repository. The global publishing server collects information from the publish servers and correlates all parts and pieces of multi-server requests, such as requests from J2EE servers to execute CICS or IMS programs. The message dispatcher is a conduit for messages from WebSphere Studio Application Monitor using and SNMP facilities. Polling agents collect data from Web Servers for Apache 2.0 and later. The visualization engine is a Web-based GUI with access to graphics, WebSphere Studio Application Monitor performance reports, real-time views of different slices of monitoring data, and access to WebSphere Studio Application Monitor internal commands and event-driven functions. 10 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

25 The visualization engine runs on a J2EE server, such as WebSphere Application Server. Figure 2-2 shows the conceptual relationship between the components. Global Publish Server (SAM) Publish traffic Snapshot traffic Message Dispatcher (MD) Publish Server (PS) Archive Agent (AA) Kernel (KL) Provide services on: - Lookup - Registration - Recovery - Configuration Visualization Engine Provide services on: -Administration -Availability -Problem Determination -Performance Management Polling Agent (PA) OCTIGATE database Figure 2-2 Kernel components At the managing server, monitoring data is prepared for real-time displays within the monitoring console and is inserted into the WebSphere Studio Application Monitor data repository. These are very resource-intensive operations; moving them to a standalone distributed server (or servers) isolates them from other enterprise activities, thus reducing WebSphere Studio Application Monitor s footprints in the monitored systems. This design also helps keep WebSphere Studio Application Monitor s processing overhead at levels low enough for 24x7 production system monitoring. Data from the data collectors is collected by publishing server and then stored in the OCTIGATE database by the archive agent. The Visualization Engine reads the database to present data through a Web console, and snapshot information such as lock analysis and in-flight transactions are retrieved directly from the data collectors Data collector The data collectors run inside the application servers. They use native system services, and they are tailored for the particular environments where they execute. The data collectors for z/os systems are written to take advantage of Chapter 2. WebSphere Studio Application Monitor V3.2 overview 11

26 services on z/os, such as MVS Cross-Memory Services and address space fencing, which are not available on distributed systems. Data collectors have two agents: Command agent The command agent fields requests from other components for information about EJB invocations, database connection pools, thread pools, stack traces, memory analyses, and heap dumps. Event agent The event agent provides data to the publish servers according to polling frequencies. This data includes system initialization data, application request-level data, and application method-level data. Collectively, these agents and other data collector routines unleash the probes, package the monitoring data into Java formats for the managing server, and deliver the data to the managing server. The data collectors send the probes into the application servers to analyze the performance of the applications. The probes collect monitoring data and feed it to transport routines that in turn route the data to the managing server. The managing server processes it for display in the WebSphere Studio Application Monitor console and for storage in the OCTIGATE repository. This relieves the processing burden of WebSphere Studio Application Monitor from the application servers as much as possible. The data collectors and probes are not designed to analyze or interpret data, but to collect it and route it as quickly as possible to the managing server where the analysis is performed. J2EE data collector The data sources employed by WebSphere Studio Application Monitor are: JVMTI garbage collector data, method trace, stack trace, CPU time, and heap dump JMX system resources SMF system resources (z/os only) PMI system resources Operating system services SCC, platform CPU, and its environment The data collector in J2EE server runs as a custom service called am in distributed and - for z/os. Figure 2-3 on page 13 shows the conceptual data collector structure. 12 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

27 WebSphere JVMTI JMX PMI am.dll JNI module Custom Service am Publish data Figure 2-3 J2EE data collector structure IMS data collector The IMS data collector installs itself as a standard IMS exit that taps transaction information. Two IMS exits are used: DFSYIOE0: the OTMA message exit DFSMSCE0: terminal message exit These exits turn the processing to the data collector JVM using a JNI routine. The data collector forwards the information to the managing server, as shown in Figure 2-4. IMS Control region OTMA message DFSYIOE0 OTMA edit routine OTMA message TM/MSC message DFSMSCE0 TM/MSC edit routine TM/MSC message libam_imsvv_zos.so JNI module Data collector JVM Publish data Figure 2-4 IMS data collector Chapter 2. WebSphere Studio Application Monitor V3.2 overview 13

28 CICS data collector The CICS data collector uses standard CICS calls that run in CICS to monitor transactions. The command agent installs itself as a CICS transaction SAMC that must be running during the length of the data collector. The event agent runs in a JVM in CICS to bridge the communication from CICS data collector to the managing server. Figure 2-5 shows the CICS data collector concept. CICS SAMC transaction libam_cics_zos.so JNI module Data collector JVM Publish data Figure 2-5 CICS data collector 2.3 Supported environment This section gives an overview table of the supported environment for WebSphere Studio Application Monitor V3.2 at the time we wrote this book. For the most up-to-date information on the supported platform, refer to WebSphere Studio Application Monitor Version 3.2 Release Notes, GI Table 2-1 shows the managing server requirement. Table 2-1 Managing server requirement Operating system Database Application server AIX 5L (5.1/5./5.3) RHEL 2.1 AS RHEL 3.0 AS DB2 8.1 ESE FP3 DB2 8.2 DB2 8.1 ESE FP3 DB2 8.2 Oracle 8i SE R Oracle 9i SE R Oracle 10g WebSphere AS WebSphere AS WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

29 Operating system Database Application server Solaris 8 Solaris 9 Solaris 8 Solaris 9 SUSE Linux 8.1 (on z/os) Windows 2000 Server SP4 Windows 2000 Advanced Server SP4 Windows 2003 Server Windows 2003 Enterprise Server DB2 8.1 ESE FP3 DB2 8.2 Oracle 8i SE R Oracle 9i SE R Oracle 10g Oracle 8i SE R Oracle 9i SE R Oracle 10g DB2 8.1 ESE FP3 DB2 8.2 DB2 8.1 ESE FP3 DB2 8.2 Oracle 8i SE R Oracle 9i SE R Oracle 10g WebSphere AS WebLogic 8.1 SP3 WebSphere AS WebSphere AS Note: Windows-based managing servers must have Microsoft Windows Services for UNIX 3.5 installed. For information about SFU, visit: Table 2-2 lists the data collector requirements. Table 2-2 Distributed data collector requirement Operating system AIX 5.1 AIX 5.2 Application server WebSphere , , 5.1.1, and 6.0 WebSphere Portal WebSphere , , 5.1.1, and 6.0 WebSphere Portal AIX 5.3 WebSphere and 6.0 Solaris 8 Solaris 9 WebSphere , , 5.1.1, and 6.0 WebSphere Portal WebLogic 7 SP5 WebLogic 8.1 SP3 HP-UX 11.iv1 WebSphere , , 5.1.1, and 6.0 WebLogic 7 SP5 WebLogic 8.1 SP3 Chapter 2. WebSphere Studio Application Monitor V3.2 overview 15

30 Operating system Application server RHEL 2.1 AS WebSphere , , WebSphere Portal WebLogic 7 SP5 WebLogic 8.1 SP3 RHEL 3.0 AS WebSphere and 6.0 WebSphere Portal WebLogic 8.1 SP3 SUSE Linux 8.1 on zseries WebSphere and 6.0 Windows 2000 SP4 Windows 2003 WebSphere , , and 6.0 WebSphere Portal z/os WebSphere with JDK WebSphere with JDK z/os z/os CICS Transaction Server V2 IMS TM/DB v7, v8, v9 2.4 Monitoring features The features and capabilities of WebSphere Studio Application Monitor are primarily exposed through the browser-based graphical user interface. The WebSphere Studio Application Monitor component that drives this user interface is referred to as the Visualization Engine. It runs as an application in a WebSphere Application Server environment. The visualization engine provides a very powerful and intuitive user interface that can be used by individuals performing different roles within the organization. At a high level, it is divided into specific functional areas (availability monitoring, problem determination, and planning and analysis). The interface is designed to provide an intuitive flow to greater levels of detail and to cross these functional boundaries as appropriate. After logging on, a WebSphere Studio Application Monitor user sees a display similar to Figure 2-6 on page 17. Individual users can alter the default home page to show a different level of monitoring. 16 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

31 Figure 2-6 Default home pane for WebSphere Studio Application Monitor user interface On the page in Figure 2-6, we can start to see how the WebSphere Studio Application Monitor features are organized. The primary menu bar near the top of the window shows the primary options in addition to LOGOUT and HELP: ADMINISTRATION AVAILABILITY PROBLEM DETERMINATION PERFORMANCE ANALYSIS We now look at each WebSphere Studio Application Monitor capability in detail Administration Administration functions target WebSphere Studio Application Monitor administrators. Its capabilities enable administrators to organize and maintain the data collectors (servers to be managed) and users. In addition it provides facilities for managing the WebSphere Studio Application Monitor server itself. Figure 2-7 shows available menu items under ADMINISTRATION. Figure 2-7 Administration menu Chapter 2. WebSphere Studio Application Monitor V3.2 overview 17

32 Account management Account management provides facilities for defining user profiles and roles. User profiles define the individual users who are allowed to log on to WebSphere Studio Application Monitor and perform various management functions. Each user has a defined role. Roles enable the administrator to limit the actions that are available to users. Three default roles are defined: Administrator, Operator, and User. Additional roles can be defined if required. Server management Server management provides facilities to group sets of servers. These groupings could be based on such criteria as location, associated business unit, operating platform, and so on. WebSphere Studio Application Monitor users can be associated with one or more groups and only those users will be able to perform management operations on the members of those groups. Server management also provides dialogs for setting the data collector configurations associated with the managed servers. Different servers, possibly running different application loads, may require different data collector configurations for optimal management. Under server management, an administrator can also define the Web servers that should be monitored for availability and throughput. Monitoring on Demand Three levels of monitoring can be used for individual application servers. At each level, the amount of detail gathered is increased and consequently there is also an increase in the overhead associated with the monitoring that could adversely affect the performance of the application server. Therefore, under normal operations, servers may be monitored at the first level, which provides a significant amount of information that can be used for availability and performance monitoring. However, in a testing environment or in a case where problems are occurring, you may wish to increase the monitoring level to provide more detail for problem determination or more specifics on the exact methods that may be affecting overall application performance. Monitoring on Demand enables an administrator (with the proper authority) to dynamically change the monitoring levels of individual servers. Additionally, schedules can be defined and enabled that allow for the monitoring level to be automatically changed at predefined times. Managing server Managing server options provide facilities to manage the WebSphere Studio Application Monitor server itself, to help ensure that the management 18 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

33 environment stays healthy so it can be used to ensure that the application environment can continue to be monitored and managed Availability The availability functions provided by WebSphere Studio Application Monitor are targeted for monitoring and maintaining the availability of application servers. Figure 2-8 shows the product features provided on the AVAILABILITY menu. Figure 2-8 Availability menu Systems overview Systems overview provides graphical representations of the availability and status of the managed environment. An administrator can easily view summary information at different levels: Enterprise: View the overall availability, throughput and recent average response times for each group of servers in the enterprise. From this screen, a WebSphere Studio Application Monitor user could drill down to the group level, which is described next. Group: View the overall availability, throughput, and recent average response times for each server within a group. From this screen, a WebSphere Studio Application Monitor user could drill down to the individual level described next. Server: View details on a specific server, including: Server information: server name, platform, IP address, and so on Server statistics: JVM CPU utilization, memory utilization, installed applications, and more Graphs representing activity over the last hour: throughput, response time, sessions/users Resource information: critical resource status such as thread pools and connection pools Portal: View status and usage information for configured portal servers Web Servers: View status and usage information for configured Web servers Chapter 2. WebSphere Studio Application Monitor V3.2 overview 19

34 Server statistics overview This option provides a tabular view of one or more groups of servers showing key statistics. These statistics can be configured by the WebSphere Studio Application Monitor user. In addition, thresholds for the various statistics can be set such that when a particular threshold is reached, the server and threshold will be highlighted to bring the status to the attention of the user. For example, a threshold might be set on average response time over a one-minute period. This view provides a powerful view of the monitored servers and can be used to launch additional tools to drill down into the details of a system that is not behaving as expected. Recent activity display This option enables the user to view and compare specific statistics for a server over the past 60 minutes or 48 hours. A wide variety of statistics relating to key metrics such as memory utilization, CPU utilization, garbage collection, response time, and so on can be viewed to identify trends or conditions that may affect overall system availability or performance. System resources This option provides a detailed representation of use of various resources that are critical to the operation and performance of an application server. Resource types for which detailed usage information is available include: EJBs JCA Connection Pools JVM/System DB Connection Pools JTA Transactions ORB Session Manager Thread Pools Web Applications SQL JCA-CICS MQI System resource comparison Though system resources as described above enables the WebSphere Studio Application Monitor user to see details for a specific server, it is often helpful to compare resource usage across multiple servers that may have a similar workload. System resource comparison provides this ability and can be an important tool to identify differences in the operation or performance of otherwise similar application servers. 20 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

35 2.4.3 Problem determination For those who are responsible for problem determination when problems arise in an enterprise, WebSphere Studio Application Monitor provides functions allowing for very detailed problem analysis. Users can start with higher-level views of the system and easily drill down to various levels of detail to identify the problematic method or resource that is the cause of the situation. Figure 2-9 shows the features on the PROBLEM DETERMINATION menu. Figure 2-9 Problem determination menu In-flight request search This facility provides information about active application requests in the system. Users can view all or search for specific method requests that are currently active. This facility helps identify requests that appear to be hung or are otherwise taking a long time to complete. Drill-downs enable users to identify the specific method (with stack traces) that may be causing the delayed response. Server activity display Server activity display provides detailed information regarding: Active requests: Similar to previously described in-flight requests. Recent requests: Displays information about requests that have completed recently. Though details such as a stack trace for individual methods are no longer available after the methods have completed, this list may help identify specific requests that the WebSphere Studio Application Monitor user would like to trace or otherwise investigate in the future. Lock contentions: This new feature in WebSphere Studio Application Monitor V3.2 provides information about any active lock contentions in the application server that could be affecting an application s performance or ability to respond to requests. Chapter 2. WebSphere Studio Application Monitor V3.2 overview 21

36 Memory diagnosis The memory diagnosis options provide a set of powerful functions for identifying problems associated with memory usage. Specifically, these features allow for detailed analysis of potential memory leaks with drill-down capabilities to the offending objects and methods. JVM thread display The JVM thread display feature provides facilities to monitor and track currently executing threads and to capture thread dumps if required. Trap and alert management All of the facilities described above can be very useful if the server or a specific request either has stopped or is performing so slowly that an administrator could explore the condition in real time and still capture the offending thread or method. In many cases, problems are more transient, intermittent or short lived such that capturing them through manual administrative actions might not be possible. In such cases, it is desirable to be able to set conditions under which the system will perform the necessary data capture at the time a problem occurs. By setting thresholds, a WebSphere Studio Application Monitor administrator can ask that critical data such as stack traces, thread dumps, and so forth occur when certain conditions take place. In addition, one or more individuals may need to be notified through s or SNMP traps being sent to other management systems. Software consistency check A technique often used in problem determination and isolation is to compare a system that is experiencing a problem to one that is not. Software consistency check enables a WebSphere Studio Application Monitor user to view the details of the installed binaries or the system runtime environment. In addition, these can be compared to other servers that should be configured similarly Performance analysis The last functional area that we discuss is performance analysis (Figure 2-10 on page 23). Previously we described availability functions that enable WebSphere Studio Application Monitor users to view and monitor the current operating status of the servers in the enterprise, and the problem determination functions that help individuals isolate and identify problems that have occurred. Performance analysis is for those who might be looking at longer-term trends or planning for new applications or changes in usage patterns so they can better predict future 22 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

37 requirements and avoid problems (especially those associated with resource utilization) before they occur. Figure 2-10 Performance analysis menu Basically, performance analysis consists of the ability to produce reports across a wide variety of metrics gathered from the day to day monitoring of running environment. The CREATE REPORT menu item enables the user to define and customize reports that are based on the performance and resource usage of the application servers in the managed environment. These reports can be analyzed further through drill-down capabilities, providing the user with data for a thorough analysis. The higher-level reports can be converted to PDF files for easier viewing and sharing among the user community. The data can also be exported to text files in comma-separated value (CSV) format for import to other analysis tools such as spreadsheets and custom report generators. After defining and customizing report definitions, you can save them and schedule them to occur on a periodic basis (daily, weekly, monthly, and so on) and then have them made available to interested parties. For environments that include z/os-based systems, the Daily Statistics option provides a facility for analyzing data coming from SMF records. 2.5 New features in V3.2 WebSphere Studio Application Monitor Version 3.2 includes several new and changed features, which we discuss briefly in this section. Improved WebSphere Portal Server monitoring that can provide better insight into key WebSphere portal metrics, such as pages and portlets Advanced J2EE memory diagnostics and lock contention analysis Expanded coverage for J2EE and composite application environments by delivering powerful new J2EE management capabilities to help subject matter Chapter 2. WebSphere Studio Application Monitor V3.2 overview 23

38 experts become more efficient at managing their J2EE applications and application servers: Support for WebSphere Application Server V6 Expanded support for BEA WebLogic V7 and V8 Broadened coverage of BEA WebLogic that includes the latest versions and fixes from BEA to help support mixed environments Support for z/os V1.6 and IBM zseries zaap processor support Monitoring of Java-optimized zaap processor metrics to help with smooth operation and detection of performance abnormalities Numerous back-end improvements, including new OMEGAMON data collector technology, which leverages OMEGAMON technologies such as instrumentation-based advanced memory diagnostics capabilities The enhancement for managing server includes: Additional platform support with WebSphere Studio Application Monitor managing server now running on Windows (a major addition) Embedded installer for installing WebSphere Application Server and DB2 Universal Database V8 (prerequisites that can be installed separately or within the managing server installation wizard) Extended language support, now consisting of English, Japanese, Korean, Simplified Chinese, Traditional Chinese, French, German, Italian, Spanish and Brazilian Portuguese The enhancement for data collectors includes: Improved portal monitoring to provide more insight into key WebSphere Portal Server metrics, such as pages and portlets Enhanced Java Heap Analysis to track object allocation and potential memory leaks, as well as identify allocating classes and methods New Lock Analysis capability identifies synchronized method invocations, as well as the number and duration of synchronized code lock waits Enhanced J2EE application server coverage Increased platform coverage 24 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

39 2.5.1 Memory Leak Analysis In previous releases, WebSphere Studio Application Monitor detected memory leak problems in two steps, but Version 3.2 has enhanced Memory Leak Analysis by adding a third step, as shown in Figure 2-11: Create a Memory Leak Confirmation Report: Confirm whether a memory may be occurring. Create a Memory Leak Candidate Finder: Find candidate object classes that could be causing the leak. Create Memory Leak Diagnosis Report: Identify the originating class, method, and line number that is causing the leak. Figure 2-11 Memory leak reporting Chapter 2. WebSphere Studio Application Monitor V3.2 overview 25

40 There are several new Memory Leak diagnosis error messages: CYNVE0745E This error message indicates that the data collector does not support heap analysis. It may be a WebSphere Studio Application Monitor 3.1 data collector, an IMS data collector, or a CICS data collector. CYNVE0746E This error message indicates that data collection is not enabled. This can be caused by either: the userbcm.xml parameter in bcm.properties file has an incorrect value; or there are no classes specified on the objectallocations tag in cyn.user.classes.xml. CYNVE0747E This error message indicates that the selected data collector has a monitoring level of L1 or L2. The memory analysis must be run on L3. For more information, see 5.2, Memory leaks on page Lock analysis Lock analysis is a new feature in WebSphere Studio Application Monitor 3.2 that enables tracking of synchronization calls, lock acquired and released and viewing of both in-flight and historic lock information. The in-flight lock contention information can be viewed from a Server Activity Display page, which has a new tab called Lock Contentions. Historic lock analysis events and requests can be viewed in the Performance Analysis Report. WebSphere Studio Application Monitor gets in-flight contention information directly from a WebSphere Studio Application Monitor J2EE data collector. By default, the Lock Contention result table is sorted on the Locked Object Class column. The Detailed Report page for Lock Analysis has a new Locks tab with two lock metric filters: Lock Acquisition Data Lock Contention Data Six new level 2 event types are passed from J2EE data collector to a WebSphere Studio Application Monitor Publish Server: Lock Acquisition Start: An application is trying to acquire a lock. Lock Acquisition End: An application has completed acquiring a lock. Lock Release Start: An application is starting to release a lock. Lock Release End: An application has completed releasing a lock. 26 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

41 Lock Contention Start: An application is trying to acquire a lock, but has to wait because the lock is owned by another thread. (Normally, this event follows a lock acquisition start event.) Lock Contention End: The thread that owns the lock has released it. (Normally, a lock acquisition end event follows this event.) A new trap type for Total Lock Acquisition Time is now available from PROBLEM DETERMINATION TRAP & ALERT MANAGEMENT using the Application Trap. Lock acquisition time is the amount of time taken to obtain a lock. The acquisition clock starts when the class method begins trying to acquire the lock and ends when the lock is acquired. The Total Lock Acquisition Time is not available for SQL traps. New lock analysis error messages: CYNVE0850E This error message indicates that the lock analysis tag cannot be found from the bcm.properties file. CYNVE0851E This error message indicates lock analysis data is not being collected because the selected server is at monitoring level L1. CYNVE0852E This error message indicates that the data collector does not support lock analysis because it is a WebSphere Studio Application Monitor 3.1 data collector, an IMS data collector, or a CICS data collector. CYNVE0500E This error message indicates that the data collector is a WebSphere Studio Application Monitor 3.1 data collector, an IMS data collector, or a CICS data collector that does not support Lock Contentions. For more about lock analysis, refer to 5.3, Deadlock management on page WebSphere Portal Server metrics WebSphere Portal Server support shows the contribution of each component to the total response time, such as: Overall response time (Gateway Servlet and Portal Pages) WebSphere Portal Server infrastructure (Authentication, Authorization, Page Loading, and Model Building) Applications (portlets) Chapter 2. WebSphere Studio Application Monitor V3.2 overview 27

42 WebSphere Studio Application Monitor V3.2 provides seven trend graphs on the Portal Overview page: Gateway Servlet Building Models Page Loading Authentication Authorization Portal Pages Portlets Only server groups that contain WebSphere Portal data collectors are in the Group pull-down list. If no WebSphere Portal data collector is monitored by the WebSphere Studio Application Monitor managing server, selecting the Portal tab produces a message. From the Portal Overview page, the link under Authorization leads to Portal Services: Credential Vault Page. The Response Time threshold values can be set using both Create Server Group and Modify Server Group pages. The new WebSphere Portal Overview error message is: CYNVE1082E z/os data collector changes This error message indicates that the server must be assigned to a group. WebSphere Studio Application Monitor 3.2 has enhanced z/os Date Collector support by providing zaap support. Four metrics of zaap-related information appear on the Server Statistics Overview page (Figure 2-12 on page 29): Delta zaap Time Delta Normal CP Time Delta zaap-eligible Time on CP Delta zaap-eligible Time 28 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

43 Figure 2-12 zaap statistic information By default, the four metrics are disabled. To view them, select the metrics in the Server Statistics Configuration (Figure 2-13) using the Customize button. Figure 2-13 Customizing zaap statistic display Chapter 2. WebSphere Studio Application Monitor V3.2 overview 29

44 2.5.5 WebSphere Application Server 6 Support WebSphere Studio Application Monitor 3.2 managing server monitors WebSphere Application Server 6 data collectors. From the WebSphere Studio Application Monitor console ADMINISTRATOR SERVER MANAGEMENT DATA COLLECTOR CONFIGURATION, the Configured Data Collector Overview page displays the WebSphere Application Server 6 server by identifying the WebSphere Application Server 6 server name followed by (default) in the Application Server column Trap and alert management In WebSphere Studio Application Monitor V3.2, a URL link to the Trap Action History page has been added in a trap . When a user clicks the URL link in the trap , the WebSphere Studio Application Monitor authenticates the user and the Trap Action History page is invoked. This enables users to click on all data actions that have been captured for the trap. An SMTP host can be set in the md.properties file of the WebSphere Studio Application Monitor Message Dispatcher component. To receive traps/alerts and WebSphere Studio Application Monitor performance analysis reports, a valid sender address must be specified in the ve.properties file. By default the HTTP port number in the trap or alert link is set to 80 in the WebSphere Studio Application Monitor Publish Server ve.port file. The HTTP port number must be changed if the WebSphere Studio Application Monitor Application Monitor HTTP port number is not 80. New Trap/Alert Management error ID message: CYNVE0691E This error indicates that the trap action history in the trap no longer exists in WebSphere Studio Application Monitor system Help system is now using Eclipse InfoCenter format In WebSphere Studio Application Monitor 3.2, the RoboHelp WebHelp system has been replaced by the Eclipse help system. As of Version 3.2, the glossary menu is part of the Help menu. This is shown in Figure 2-14 on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

45 Figure 2-14 WebSphere Studio Application Monitor help system Performance enhancement In WebSphere Studio Application Monitor 3.2, both DB2 and Oracle database performance tuning guides are provided in the WebSphere Studio Application Monitor Version 3.2 Installation Guide, SC The kernel has been redesigned to improve performance for data collector configuration tasks. The kernel process can configure a data collector, create and activate a trap on the data collector, and schedule a MOD level or sampling rate change on the data collector regardless of the number of data collectors the managing server is monitoring. There is a reduction in RMI calls to WebSphere Studio Application Monitor backend components. SQL statements in the performance analysis report have been tuned and WebSphere Studio Application Monitor provides the ability to utilize the DB2_SELECTIVITY parameter. This improves data distribution, which is a critical factor in determining whether the db2.selectivity parameter in the ve.properties file should be used to guide DB2 optimizer. Chapter 2. WebSphere Studio Application Monitor V3.2 overview 31

46 32 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

47 3 Chapter 3. Managing WebSphere Studio Application Monitor This chapter discusses the considerations and issues for running and operating the WebSphere Studio Application Monitor environment. The discussion in this chapter concerns only the WebSphere Studio Application Monitor environment, the managing server, and the data collectors. The topics are: 3.1, Overview on page , Security considerations on page , Server management on page , Database management on page , Managing server parameters on page , Data collector administration on page 62 Copyright IBM Corp All rights reserved. 33

48 3.1 Overview WebSphere Studio Application Monitor must be properly operated and maintained to keep it useful as a management application. This chapter explains the use of WebSphere Studio Application Monitor from the administrator perspective and discusses tasks related to maintaining the reliability, performance, and health of the application. We discuss regular administration to answer questions such as Why should this be created? or What should be defined? For answers to How to? questions, this section is meant to be used alongside the WebSphere Studio Application Monitor Version 3.2 User Guide, SC A WebSphere Studio Application Monitor administrator can use the information in 3.4, Database management on page 44 to maintain the database, from migrating Version 3.1 tables and making backups for database integrity issues to maintaining performance of the database. This chapter also contains information about maintaining the managing server and data collector. This includes property files management and reference, log file maintenance, and discussion about available utilities to manipulate them. 3.2 Security considerations When discussing security in WebSphere Studio Application Monitor, two topics should be considered: the security of the WebSphere Studio Application Monitor system itself against tampering and eavesdropping, and the user security that accesses the managing server using the Web console Node authentication default properties This section illustrates the design and implementation of node authentication and secure transmission among WebSphere Studio Application Monitor components. This addresses various security issues that may be encountered during client/server communication over TCP/IP. The node authentication effort is made to address the following issues: Eavesdropping: Privacy of information is compromised, although the information may remain intact. This could happen if the information on the wire is intercepted by a middleman. Eavesdropping is addressed by encrypting data before sending it and decrypting data after receiving it. 34 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

49 Tampering: Information may be modified or replaced while in transit, then relayed to the recipient instead of the original content. One solution is to fingerprint the data using a one-way hash and verify the fingerprint on the receiving end. Impersonation: Information is sent to a person who acts as the intended recipient. This takes two forms: Spoofing: A person or a computer pretends to be someone else. Misrepresentation: A person or computer misrepresents itself. A computer acts as a true representative of a site and accepts requests intended for the actual Web site. These problems can be handled using certificates and by performing both client-side and server-side authentication. Negotiation verifies the authenticity of the communicating ends. This way both ends can be sure that the data is being received by the correct system or person. All WebSphere Studio Application Monitor managing server and data collector components must communicate with each other in a reliable manner. To achieve this, WebSphere Studio Application Monitor components perform a handshake and authenticate on both ends before they establish a session, using the X.509 certificate-based authentication model. The X.509 certificate authentication is executed in this sequence: 1. A client sends a public certificate of the server to the server. The server verifies the certificate s authenticity with the public key of the certificate. Note: All certificates that are used are self-signed certificates. If the certificate is altered, its footprint will differ from the footprint generated via its public key. 2. If the verification passes, the certificate footprint is compared with the footprint stored on the server side. This confirms the authenticity of the certificate. 3. The certificate s expiration dates are checked to verify its validity. 4. The certificate s subject is checked to verify that the client is using the proper certificate. Example 3-1 on page 36 shows the portion of $AM_HOME/bin/setenv.sh that contains the certificate properties. This information must match the certificate definition in the data collectors and proxies. Chapter 3. Managing WebSphere Studio Application Monitor 35

50 Example 3-1 Certificate properties in setenv.sh CERT_PATH="$CYANEA_HOME_OS/etc/mgmttodc.cer;$CYANEA_HOME_OS/etc/mgmttoproxy.cer ;$CYANEA_HOME_OS/etc/mgmttomgmt.cer" KEYSTR_LOC=$CYANEA_HOME_OS/etc/CyaneaMgmtStore KEYSTR_PASS=cyanea94612 KEYSTR_KEYPASS=cyanea94612 NODE_USERID=mgmtcomp NODE_IP= Users and roles Users in WebSphere Studio Application Monitor are authenticated to the operating system. It uses the authenticate module in the bin directory to perform password check. The users are defined in WebSphere Studio Application Monitor database table. This definition associates the user with assigned role and server groups access. The same user ID can be used from multiple Web browsers, but for security purposes we recommend that you assign a separate ID for each user. These users must be defined twice: To the operating system, using the appropriate method for your platform, such as Computer Management in Windows, the smitty user command in AIX, or the adduser command in Linux. To the WebSphere Studio Application Monitor Web console using the menu ADMINISTRATION ACCOUNT MANAGEMENT USER PROFILES. Each user is associated with the following: The operating system user ID that the password will be authenticated to, which should be the same as the user name to avoid confusion. The actual address of the user to be able to send SMTP mail for a trap. The user role that determines which part of the menu can be accessed by the user. (See the Web console menu discussion in 2.4, Monitoring features on page 16.) Access to server groups that determine which servers the user can see and manage; this access is on server group level. A WebSphere Studio Application Monitor role is a set of predefined authorities that determine which WebSphere Studio Application Monitor function a user can access. There are three predefined roles: Administrator: all functions Operator: basic monitoring function and troubleshooting features User: only monitoring 36 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

51 We do not see an immediate need to define new roles. Unless you have determined that your environment must have a set of restrictions in addition to what WebSphere Studio Application Monitor already has, you do not need to add new roles. You cannot modify or delete the default roles. See WebSphere Studio Application Monitor Version 3.2 User Guide, SC , for detailed instruction about: Creating, modifying, and deleting a user Creating, modifying, and deleting roles 3.3 Server management As we discussed in 3.2.2, Users and roles on page 36, user access to servers is based on server groups. Several considerations must be made when designing optimum server groups and server configurations. The discussion is restricted to items that can be set for the data collector from the Web console Server groups Server groups are important to providing a meaningful grouping for servers. Server groups cannot include other server groups. There are several reasons for putting servers into a server groups: Aggregation: The servers in the servers group should be similar enough that the aggregate display will not mask an individual server s behavior. Comparison: Servers belonging to the same server group can be easily compared with each other. Function: Because the server group is the basic unit of assignment, it makes sense to put servers belonging to the same function or application in a server group to be easily assigned. Some recommendations for creating server groups: By application: group of servers for a certain application By server platform: servers running the same software level By authority group: servers owned by a department These server groups can then be assigned to the appropriate users. Chapter 3. Managing WebSphere Studio Application Monitor 37

52 3.3.2 Data collector configuration Apart from having the server groups, each server s data collector must be configured to be managed by WebSphere Studio Application Monitor. The system assigns a name to the data collector. On a non-z/os platform, the data collector s name is a combination of the administration server name and the application server name. For the z/os platform, the name is a combination of the sysplex name and the application server name. The name cannot be changed. In a z/os environment, a WebSphere server instance is represented by a data collector definition. It serves as a template for all data collectors in the server regions that belong to the same server instance. This means that although you may be configuring the data collector for a server instance, the configuration is actually used by all of the data collectors in the server regions when monitoring the applications. The configuration includes: Listing specific classes or transactions prevents them from being monitored; you do not want them monitored when they are provided by either the system or WebSphere Studio Application Monitor. Indicating what type of data collector is running Enabling or disabling a certain feature of the data collector, such as MQI collection When you install a data collector and start it for the first time, it reports in as an unconfigured data collector in the Data Collector Overview page. When you configure it, the system removes it from the unconfigured data collectors list and displays it in the configured data collectors list. Typically one of the default configurations (J2EE Default, CICS Default, or IMS Default) is adequate unless you are using MQ for composite transaction information. In this case, you must specify queue names, so you will have to create a new configuration. These configurations can be modified or you can create a new configuration to include or exclude other classes or queue names. If you decide to retire or redeploy an application server, you should unconfigure the data collector and the system will remove its configuration record from the WebSphere Studio Application Monitor database. When you unconfigure a data collector, the system removes it from the configured data collectors list and displays it with the unconfigured data collectors. When a data collector is retired, its data will be purged. If the data collector has reports associated with it, you are prompted to delete those reports before unconfiguring the data collector. 38 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

53 3.3.3 Monitoring level considerations When configuring a data collector, you must first set your desired monitoring level. When you install a new data collector, by it is set to a default monitoring schedule of level 1. There are three monitoring levels that can be set for each server or group of servers with data collectors installed on them: L1 (Production mode) provides Availability Management, System Resources, and basic request data. L2 (Problem Determination mode) monitors production level plus advanced request data, including CPU information, additional monitoring fields, and functions. L3 (Tracing mode) monitors everything in L2 plus method-level and SQL call level operations. Trap and Alert functions that are based on L3 events require that you set data collectors on L3. Default monitoring level The default monitoring level is defined at the managing server. (Select ADMINISTRATION MANAGING SERVER SYSTEM PROPERTIES.) Figure 3-1 shows the default settings for a newly installed data collector. When you install a new data collector, it takes on these default values of 2% sampling and L1, so we typically recommend that you leave the default or at least the conservative monitoring levels defined here. You then set schedules or override at the server or group level with Monitoring on Demand. Figure 3-1 Managing server system properties Chapter 3. Managing WebSphere Studio Application Monitor 39

54 Monitoring on Demand enables you to change the monitoring level and sampling rates. Monitoring level can be set based on a predefined time schedule, or you can change it on the fly. Note: Prior to WebSphere Studio Application Monitor V3.2, there was a value that had to be set, wsam_intent_trace, in the z/os data collector properties file. The default value was set to yes to enable you to change from level 1 to level 3 without restarting the application server your data collector resided on. However, enabling this function meant that Just In-Time (JIT) compilation would be bypassed, so many customers set this to no in production, which then required the application server that the data collector resides on to be restarted in order to change to monitoring level 3. The new architecture in WebSphere Studio Application Monitor V3.2 no longer uses this value or has this restriction. So now z/os is just like the distributed platforms in that you can change monitoring levels without restarting anything. Defining a monitoring schedule In some cases it makes sense to define a schedule for a particular server or servers within a particular group. For example, your application may be experiencing problems around 2:00 each morning, so you may want to schedule the data collector to go to level 3 Monitoring at 1:00 a.m., then return to level 1 monitoring at 3:00 a.m. to capture as much information as it can for that defined time. Another example is if developers use level 3 during the day while debugging and often forget to turn it back to level 1 monitoring at the end of the day, you may want to set up a schedule that every night sets the monitoring level to level 1 to cut down the network traffic between that data collector and the managing server. This automated approach does not require any manual intervention other than setting up and applying the schedule initially. These schedules can be modified or removed at any time. We walk through the next set of screens to define and apply a schedule to a server and server group to set monitoring to level 1 each evening at 7:00 p.m. This assumes that there is no override defined. (We will cover the override next.) 40 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

55 Figure 3-2 shows the Monitoring on Demand screen. (Select ADMINISTRATION MONITORING ON DEMAND.) This example shows that the lima1 server does not have a defined schedule, and santiago1 has a predefined schedule. This indicates that lima1 uses the default schedule defined in the group level. Figure 3-2 Monitoring on Demand Chapter 3. Managing WebSphere Studio Application Monitor 41

56 To define a schedule, click Create Schedule. We created a schedule called 7PM L1 Production Mode (Figure 3-3), which sets monitoring to level 1 at 7:00 p.m. on any day. Click Add to add a new scheduling rule, then click Save to store the schedule. Figure 3-3 Creating a schedule Select the MOD Console link to assign the schedule from the main Monitoring on Demand console. Here you can modify the assignment of the servers using the orange button in the Schedule Change/Override column. You can change the schedule for individual servers or a group of servers. Figure 3-4 on page 43 shows the schedule definition for a group. Note: The override monitoring level box takes precedence over the schedule selection, so to ensure that your schedule is activated, make sure the override monitoring box shows No Override. 42 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

57 Figure 3-4 Modify schedule at the group level After you select OK, you will see that the schedule has servers associated with it. There is no need to restart any components. This change will be picked up automatically and start taking affect. Changing the monitoring level on the fly You can also manually change the monitoring level on the fly. From the Monitoring on Demand screen shown in Figure 3-2 on page 41, you can change this level for a particular server or a group of servers. Select the orange box under the Schedule Change/Override column. From the Modify Server Settings screen, select a value from the Override Monitoring Level pull-down list. This value sets the monitoring level to 1, 2, 3, or no override. When you click OK, the override monitoring level takes affect. The override level has precedence over the monitoring level schedule. There is no need to restart any components. This change will be picked up automatically. Chapter 3. Managing WebSphere Studio Application Monitor 43

58 3.3.4 Sampling rate considerations The sampling rate determines what percentage of information from the data collector is stored into the OCTIGATE database. The sampling rate is processed by the publish server: For every completed request, the publish server generates a random number and compares it with the sampling rate. When the random number is less than the sampling rate, the request is queued to the archive agent for storage. This parameter is especially useful to tune the performance of your managing server, as the archive agent is one of the busiest tasks in WebSphere Studio Application Monitor and it constantly writes data to the database. The sampling rate is also useful to tune the size of your database. Too much information may cause your database to grow very quickly. This may slow down the managing server and the visualization engine considerably. You should maintain the sampling rate high enough to allow analysis and profiling of your application server load. When your sampling rate shows less than 50% of your transaction mix most of the time, your sampling rate is likely to be too low. 3.4 Database management This section discusses the management of databases for WebSphere Studio Application Monitor Migration from V3.1 In the event of a database migration from WebSphere Studio Application Monitor V3.1 to V3.2, WebSphere Studio Application Monitor Version 3.2 Installation Guide, SC , mentions that a database creation script is executed using user ID amuser, and it creates new tables with the creator AMUSER. (The V3.1 database contains tables with the creator CYANEA.) This leaves the database in an inconsistent state and sends errors at the Web management console startup. In migration, it is recommended that WebSphere Studio Application Monitor V3.2 installation and database migration are performed using the user ID cyanea. This ensures consistency of ownership of database tables and smooth startup of the managing server components. However, if you want to use the user ID amuser for consistency with the documentation, you should prepare your system using one of two approaches: Selectively export and re-import specific tables after the WebSphere Studio Application Monitor V3.2 is installed. Rename the table owners from CYANEA to AMUSER and perform the migration. 44 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

59 Selective export and import You can preserve the servers and users definition in WebSphere Studio Application Monitor by exporting and re-importing certain tables: 1. Create a temporary directory to store the temporary exported files. For our example the directory is called tmpdata. 2. Ensure that you have access to the DB2 command line environment. Source the environment from sqllib/db2profile from the instance owner home directory:. /home/db2inst1/sqllib/db2profile 3. Connect to the OCTIGATE database as the user cyanea: db2 connect to OCTIGATE user cyanea using pw 4. Export the necessary tables. The following tables are related to user, server groups, and role functions: map_gr_se servers map_us_gr groups users map_ro_fu roles The command to export these tables to an interchange format (IXF) is: export to /tmpdata/$tbname.ixf OF IXF messages /tmpdata/$tbname.mse select * from $tbname 5. Verify the message files in /tmpdata to ensure that the export process is successful. The content of the file should be similar to Example 3-2. Example 3-2 Exporting data SQL3104N The Export utility is beginning to export data to file "tmpdata/groups.ixf". SQL3105N The Export utility has finished exporting "1" rows. 6. Now that we have saved the necessary tables, we can drop the database, using this sequence of commands: db2 terminate db2 uncatalog database octigate db2 drop database octigate 7. Create user ID amuser and install WebSphere Studio Application Monitor 3.2. Chapter 3. Managing WebSphere Studio Application Monitor 45

60 8. Connect to the new OCTIGATE database as amuser. db2 connect to octigate user amuser using ampw 9. Restore data from previously exported files, a. Clean up the tables: delete from map_gr_se delete from servers delete from map_us_gr delete from groups delete from users delete from map_ro_fu delete from roles b. Import from the saved files: import from tmpdata/roles.ixf OF IXF messages tmpdata/roles.msi insert into roles import from tmpdata/map_ro_fu.ixf OF IXF messages tmpdata/map_ro_fu.msi insert into map_ro_fu import from tmpdata/users.ixf OF IXF messages tmpdata/users.msi insert into users import from tmpdata/groups.ixf OF IXF messages tmpdata/groups.msi insert into groups import from tmpdata/map_us_gr.ixf OF IXF messages tmpdata/map_us_gr.msi insert into map_us_gr import from tmpdata/servers.ixf OF IXF messages tmpdata/servers.msi insert into servers import from tmpdata/map_gr_se.ixf OF IXF messages tmpdata/map_gr_se.msi insert into map_gr_se c. Update the necessary references: update TABLEKEYS set NEXTKEY = (select count(*)+1 from roles) where TABLENAME = 'ROLES' update TABLEKEYS set NEXTKEY = (select count(*)+1 from users) where TABLENAME = 'USERS' update TABLEKEYS set NEXTKEY = (select count(*)+1 from servers) where TABLENAME = 'SERVERS' update TABLEKEYS set NEXTKEY = (select count(*)+1 from groups) where TABLENAME = 'GROUPS' 10.Verify message files created during import in the tmpdata directory and start the WebSphere Studio Application Monitor components and Web console server. 11.Update the Portal Section in the Fixed Response Time data in the Server Groups page for each group using the Web console. This is required for the new fields that do not exist in WebSphere Studio Application Monitor V When WebSphere Studio Application Monitor runs well, you can remove the /tmpdata directory and its related content. 46 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

61 Renaming the original tables You can perform a rename of the original tables using the following procedure: 1. Create a temporary directory to store the temporary exported files. For our example the directory is called tmpdata. 2. Log on to the database as the instance owner or any user with database administrator (dbadm) authority. Ensure that you have access to the DB2 command line environment. Source the environment from sqllib/db2profile from the instance owner home directory:. /home/db2inst1/sqllib/db2profile 3. Ensure that no application is using the database. (WebSphere Studio Application Monitor components and VE are down.) 4. From the tmpdata directory, run the export of all tables: db2move OCTIGATE export 5. Edit db2move.lst file to change all CYANEA to AMUSER. 6. Drop the OCTIGATE database and re-create a new blank OCTIGATE database: db2 drop db OCTIGATE db2 create db OCTIGATE 7. Import the data back to OCTIGATE: db2move OCTIGATE import 8. Now you can perform the migration using the WebSphere Studio Application Monitor installation utility Database backup To ensure that the data in WebSphere Studio Application Monitor is available for recovery, you must back up the database regularly. There are two types of backup in DB2: online and offline. An online backup enables backup to progress while the database is updated; however, this requires that the database is defined with the LOGRETAIN option, in which the transaction log is kept to understand (or map) the data status while the backup is being performed. WebSphere Studio Application Monitor database is created without the LOGRETAIN option by default, so only offline backup is available. To perform a successful offline backup, stop all applications that access it. This requires a user ID with DBADM authority; you can use the instance owner user ID or grant DBADM to the AMUSER to perform the process. To grant the DBADM authority, run the following commands as the instance owner: DB2 CONNECT TO OCTIGATE DB2 GRANT DBADM ON DATABASE TO AMUSER Chapter 3. Managing WebSphere Studio Application Monitor 47

62 Note: We recommend that you back up the OCTIGATE database at least weekly. If you are using the data trimmer, perform the backup before the data trimmer utility. Use the following procedure to back up the database: 1. Stop the visualization engine by running either: $WAS_HOME/stopServer.sh server_wsam or %WAS_HOME%\stopServer.bat server_wsam 2. Stop WebSphere Studio Application Monitor managing server components by using a Korn shell prompt: $AM_HOME/bin/am-stop.sh 3. Using the DB2 Command Line Processor, check whether any applications are active by running the following command: DB2 TERMINATE DB2 LIST APPLICATION Note: The DB2 TERMINATE command is necessary to ensure that your Command Line Processor is not connected to the database. 4. When no more applications are connected, you can invoke the backup process from the Command Line Processor using the following syntax: DB2 BACKUP DB OCTIGATE USER AMUSER TO <path> COMPRESS 5. When backup has completed, you can restart the WebSphere Studio Application Monitor managing server components using the Korn shell: $AM_HOME/bin/am-start.sh 6. Start the visualization engine using either: or $WAS_HOME/startServer.sh server_wsam %WAS_HOME%\startServer.bat server_wsam Maintaining database performance Database performance is key to the operation of WebSphere Studio Application Monitor. All WebSphere Studio Application Monitor processing is based on the OCTIGATE database. You may have to perform several steps. 48 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

63 Note: Depending on the activity of your database, you may want to perform the procedure daily. If you are running the data trimmer, you should perform the procedure after the data trimmer utility. Database performance is typically affected by these utilities: Reorganization: The REORG utility ensures that the database structure is organized correctly. Rows of data in the database are stored in a fixed set of pages to optimize disk operation. As data is inserted, updated, and deleted, some of the rows become unusable, moved, or unordered. The REORG process collects and sorts the data rows so that each page consists of neighboring rows (as indicated by their primary keys). It also ensures enough free space in the page for ease of insertion of new data when necessary. Run statistic: The RUNSTATS utility collects and populates table and index statistics, such as clustering percentage, row number, row distribution, and so on. These statistics are used to influence how DB2 gets to the data. As an example, if there are less than 10 rows in a table, contained in two pages, DB2 will not bother to retrieve an index for the search query; however, for a million-page search, using an index to search for a row is beneficial. A typical index-to-data ratio is 1 to 100, so an index search will read only 10,000 pages. Reorganization is a time-consuming process, so you may want to do it sparingly. The REORGCHK utility can help you decide whether a reorganization is necessary. REORGCHK can invoke RUNSTATS to collect table and index information for the REORG recommendation. You can use the command: DB2 CONNECT TO OCTIGATE USER AMUSER DB2 REORGCHK Example 3-3 shows REORGCHK output. An asterisk in the last column (as for CTG_SR_OVERVIEW) indicates that running REORG will be beneficial for that table or index. Example 3-3 Output of REORGCHK Doing RUNSTATS... Table statistics: F1: 100 * OVERFLOW / CARD < 5 F2: 100 * (Effective Space Utilization of Data Pages) > 70 F3: 100 * (Required Pages / Total Pages) > 80 SCHEMA NAME CARD OV NP FP ACTBLK TSIZE F1 F2 F3 REORG AMUSER CONFIG Chapter 3. Managing WebSphere Studio Application Monitor 49

64 AMUSER CTG_SR_OVERVIEW ** AMUSER DUAL AMUSER FGROUPS AMUSER FUNCTIONS AMUSER GC_DATA AMUSER GPSMASK AMUSER GPSREQUEST AMUSER GPSTOKEN AMUSER GPSTOKENINFO AMUSER GPSUOE AMUSER TRAP_STACK_TRACE AMUSER TRAP_THREAD_DUMP AMUSER USERS AMUSER VOLUMESTAT AMUSER WBI_REQUEST_OVERV> AMUSER WEBSERVERCHARTDATA Index statistics: F4: CLUSTERRATIO or normalized CLUSTERFACTOR > 80 F5: 100 * (KEYS * (ISIZE + 9) + (CARD - KEYS) * 5) / ((NLEAF - NUM EMPTY LEAFS) * INDEXPAGESIZE) > 50 F6: (100 - PCTFREE) * ((INDEXPAGESIZE - 96) / (ISIZE + 12)) ** (NLEVELS - 2) * (INDEXPAGESIZE - 96) / (KEYS * (ISIZE + 9) + (CARD - KEYS) * 5) < 100 F7: 100 * (NUMRIDS DELETED / (NUMRIDS DELETED + CARD)) < 20 F8: 100 * (NUM EMPTY LEAFS / NLEAF) < 20 SCHEMA NAME CARD LEAF ELEAF LVLS ISIZE NDEL KEYS F4 F5 F6 F7 F8 REORG Table: AMUSER.CONFIG SYSIBM SQL Table: AMUSER.CTG_SR_OVERVIEW AMUSER CSO_CT *-*- AMUSER CSO_PROBEID *-*- Table: AMUSER.FGROUPS SYSIBM SQL Table: AMUSER.FUNCTIONS SYSIBM SQL SYSIBM SQL Table: AMUSER.VOLUMESTAT AMUSER VOL_DOY AMUSER VOL_HOUR AMUSER VOL_MINUTE *---- AMUSER VOL_SERVID AMUSER VOL_TIME AMUSER VOL_YEAR WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

65 Table: AMUSER.WBI_REQUEST_OVERVIEW AMUSER WBI_R_BUSOBJ AMUSER WBI_R_PROBEID AMUSER WBI_R_VERB AMUSER WBI_R_VERB_BUSOBJ Table: AMUSER.WEBSERVERCHARTDATA AMUSER WS_SERVERID CLUSTERRATIO or normalized CLUSTERFACTOR (F4) will indicate REORG is necessary for indexes that are not in the same sequence as the base table. When multiple indexes are defined on a table, one or more indexes may be flagged as needing REORG. Specify the most important index for REORG sequencing. Tables defined using the ORGANIZE BY clause and the corresponding dimension indexes have a * suffix to their names. The cardinality of a dimension index is equal to the Active blocks statistic of the table. Run REORG for tables using this command: DB2 REORG TABLE <tblname> For reorganizing the index, use this command: DB2 REORG INDEXES ALL FOR TABLE <tblname> REORG takes a lot of time, so you may want to run it weekly, coinciding with the backup process in 3.4.2, Database backup on page 47. For performance reasons, you may want to back up the database after reorganization. We recommend that you perform RUNSTATS daily to ensure that the statistics are always current. It is recommended that such utilities access databases when workload is lowest. To run RUNSTATS from the DB2 Command Line Processor, use the following command: DB2 CONNECT TO OCTIGATE USER AMUSER DB2 RUNSTATS ON TABLE AMUSER.<tblname> WebSphere Studio Application Monitor provides a command file for performing statistics updates for its tables: $AM_HOME/bin/run_stat_cmds.sh. When you perform REORG and RUNSTATS, you must perform RUNSTATS after REORG, or all the collected statistics will be incorrect. Chapter 3. Managing WebSphere Studio Application Monitor 51

66 3.4.4 Cleaning up the database There are two utilities for trimming database size: datatrimmer.sh, which trims these tables in the OCTIGATE database: MEMORY_DATA GC_DATA MQI_QUEUEMGR_SR_OVERVIEW CTG_SR_OVERVIEW WEBSERVERCHARTDATA IMSTHREADS VOLUMESTAT PORTALSTATS PORTALOVERVIEW datatrim.sh, which allows a custom specification on the tables that you want to trim based on the content of $AM_HOME/etc/datatrim.xml. The default tables that it trims are: REQUEST METHOD SERVERSTATS PMISTATS VOLUMESTATS The Data Trimmer is used to trim old data that does not need to be maintained in the database any longer. It is provided as a command-line utility so that users can simply schedule a cron job to periodically trim appropriate data. For REQUEST and METHOD tables, this process has two parts: marking records to be deleted and deleting marked records. Be aware that running this command can take a long time depending on the amount of data being trimmed. This release of the Data Trimmer does not support more than one instance of the trimmer running concurrently. You should use the amuser account and make the properties file accessible to the amuser. You need the following files to use the Data Trimmer: $AM_HOME/lib/datatrim.jar $AM_HOME/etc/datatrim.xml $AM_HOME/etc/markdatadeleting.xml $AM_HOME/bin/datatrim.sh Data Trimmer produces $AM_HOME/logs/datatrim.log to store execution messages. The syntax of the datatrim command is: $AM_HOME/bin/datatrim.sh <dbname> <dbuser> <dbpassword> 52 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

67 3.5 Managing server parameters As discussed in 2.2.1, The managing server on page 10, the managing server has multiple components such as Kernel, Publishing Server, Archive Agent, Polling Agent, SAM, and so on. Each component has a different role in collecting, caching, massaging, distributing, and archiving data into the database. The managing server has several ways to set up its parameters, such as setenv.sh in the bin directory and various properties files in the etc directory. This section offers a comprehensive reference to set and change these parameters Component heap sizes Because the managing server components are Java processes, memory size is a big performance factor to consider. The memory size for a JVM is typically reflected in its heap size. The heap size for the managing server components are set in the setenv.sh. Example 3-4 shows the parameters in the setenv.sh file that enable configuration of the Java heap sizes for all components of the WebSphere Studio Application Monitor managing server. Example 3-4 Parameters in setenv.sh HEAP_MIN_SIZE_PS=256 HEAP_MAX_SIZE_PS=512 HEAP_MIN_SIZE_ARCHIVE_AGENT=256 HEAP_MAX_SIZE_ARCHIVE_AGENT=512 HEAP_MIN_SIZE_ =32 HEAP_MAX_SIZE_ =64 HEAP_MIN_SIZE_MESSAGE_DISPATCHER=32 HEAP_MAX_SIZE_MESSAGE_DISPATCHER=64 HEAP_MIN_SIZE_KERNEL=128 HEAP_MAX_SIZE_KERNEL=256 HEAP_MIN_SIZE_AVM=32 HEAP_MAX_SIZE_AVM=64 HEAP_MIN_SIZE_SAM=64 HEAP_MAX_SIZE_SAM=128 HEAP_MIN_SIZE_POLLING_AGENT=64 HEAP_MAX_SIZE_POLLING_AGENT=64 Example 3-4 shows that each component has a minimum and maximum heap size. The publishing server and archive agent initialize with the largest amount of memory, indicating that these processes use the largest memory resources. These components require heap size from MB to the maximum heap size of 1,665 MB. On top of this, we should consider the memory allocation for the virtualization engine, which has a default of an initial heap size of 512 MB and a maximum of 512MB as shown in Figure 3-5 on page 54. Chapter 3. Managing WebSphere Studio Application Monitor 53

68 Figure 3-5 Configuring virtualization engine heap size There are several reasons to modify the default heap size defined in the environment properties file, such as when the hardware of the managing server does not have enough memory. In this case there are two solutions to the problem: either add more memory or tune the components with lower heap size. Note: If any component is not configured with its default initial memory and maximum memory, then performance issues can occur. In a low-memory environment, the managing server that has been designed for testing may be downsized to use a lower heap size. Although this is not recommended for a production environment, it may have some benefit in testing or proof-of-concept implementation. You also may want to modify the visualization engine s initial heap size and maximum heap size. The settings are located in the WebSphere administration console of the application server. In the WebSphere administration console, select from the menu Servers Application Servers server_wsam Process Definition Java Virtual Machine (shown in Figure 3-5). There are 54 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

69 two parameters in this view called Initial Heap Size and Maximum Heap Size. Both values are set by default to 512 MB and can be set to a lower value such as 256 MB. This would free up memory resources on the managing server box. You must restart the application server instance after changing this JDBC driver properties The JDBC driver properties are limited to a set of four values for which the database is DB2 in this installation. The properties reference the JDBC driver name, driver URL, user, and password. These are the four properties: JDBC_DRIVER_NAME JDBC_DRIVER_URL JDBC_USER The driver class that WebSphere Studio Application Monitor uses to access the DB2 database. The driver URL is formatted as: <protocol>:<vendord>:<database name> Additionally, there is a parameter for a port number in case the configuration points to a remote database. The user is basically the schema name. JDBC_PASSWORD In DB2, you can test whether the user name is correct. Run these commands: db2 connect to oct_db32 user amuser using ampw db2 list tables There should be 59 tables and one view listed there. The password for the schema name. Example 3-5 shows the default settings after the installation using DB2 V parameter values for the JDBC driver settings. You can use other database versions and brands; check the certified versions table for more information. Example 3-5 JDBC driver parameters in setenv.sh JDBC_DRIVER_NAME=COM.ibm.db2.jdbc.app.DB2Driver JDBC_DRIVER_URL=jdbc:db2:OCT_DB32 JDBC_USER=amuser JDBC_PASSWORD=password You can also use the command line test on the database driver properties. This test is also performed when running the WebSphere Studio Application Monitor start script (./am-start.sh). To run the command line test before starting the managing server, run this command:./klctl.sh dbtest After the dbtest has run successfully, Example 3-6 on page 56 appears. Chapter 3. Managing WebSphere Studio Application Monitor 55

70 Example 3-6 Testing database connectivity $./klctl.sh dbtest Testing DB Connection with kernel properties... using the following properties to test database connection... JDBC DRIVER NAME =COM.ibm.db2.jdbc.app.DB2Driver DB URL =jdbc:db2:oct32 DB USER =amuser DB PASSWORD =XXXX Trying to establish connection to database Succesfully established connection Testing by executiong two sql calls... Trying to get first prepared statement from connection Successfully got prepared statement from connection Trying to execute prepared statement on connection Successfully executed prepared statement on connection Trying to close prepared statement Successfully closed prepared statement Trying to get second prepared statement from connection Successfully got prepared statement from connection Trying to execute prepared statement on connection Successfully executed prepared statement on connection Trying to close prepared statement Successfully closed prepared statement Trying to close database connection Successfully closed connection Test complete JDBC parameter problems The following sections show some sample problems with JDBC parameters. Wrong user name When you use a wrong user name, hence the database schema name is wrong, you may get the error shown in Example 3-7. Example 3-7 Error: Wrong user name $./klctl.sh dbtest Testing DB Connection with kernel properties... using the following properties to test database connection... JDBC DRIVER NAME =COM.ibm.db2.jdbc.app.DB2Driver DB URL =jdbc:db2:oct32 56 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

71 DB USER =aamuser DB PASSWORD =XXXX Trying to establish connection to database Exception in thread "main" java.sql.sqlexception: [IBM][CLI Driver] SQL30082N Attempt to establish connection failed with security reason "24" ("USERNAME AND/OR PASSWORD INVALID"). SQLSTATE=08001 at com.cyanea.common.database.dbaccess.getdbconn(dbaccess.java:104) at com.cyanea.common.dbtest.dbtest.testdbconnection(dbtest.java:36) at com.cyanea.common.dbtest.dbtest.main(dbtest.java:155) The user name is specified in the DB USER parameter. You can correct this problem by re-running the database creation script using this command:./db2install.sh db2inst1 /opt/wsam The first argument should be the DB2 instance owner and the second argument should be the absolute path of the WebSphere Studio Application Monitor installation directory. Database does not exist If the database does not exist or you specify the wrong database name, the error in Example 3-8 may happen. Example 3-8 Error: Database does not exist $./klctl.sh dbtest Testing DB Connection with kernel properties... using the following properties to test database connection... JDBC DRIVER NAME =COM.ibm.db2.jdbc.app.DB2Driver DB URL =jdbc:db2:oct382 DB USER =amuser DB PASSWORD =XXXX Trying to establish connection to database Exception in thread "main" java.sql.sqlexception: [IBM][CLI Driver] SQL1013N The database alias name or database name "OCT382" could not be found. SQLSTATE=42705 at com.cyanea.common.database.dbaccess.getdbconn(dbaccess.java:104) at com.cyanea.common.dbtest.dbtest.testdbconnection(dbtest.java:36) at com.cyanea.common.dbtest.dbtest.main(dbtest.java:155) The database name is specified in the third parameter of the DB URL. DB2 driver problem Another possible error is a missing database driver. The driver should be in the CLASSPATH if you have source db2profile. Otherwise the error in Example 3-9 on page 58 may happen. Chapter 3. Managing WebSphere Studio Application Monitor 57

72 Example 3-9 Error: Missing DB2 driver $./klctl.sh dbtest Testing DB Connection with kernel properties... using the following properties to test database connection... JDBC DRIVER NAME =COM.ibm.db2.jdbc.app.DB2Driver DB URL =jdbc:db2:oct32 DB USER =amuser DB PASSWORD =XXXX Trying to establish connection to database Exception in thread "main" java.sql.sqlexception: COM.ibm.db2.jdbc.app.DB2Driver at com.cyanea.common.database.dbaccess.getdbconn(dbaccess.java:104) at com.cyanea.common.dbtest.dbtest.testdbconnection(dbtest.java:36) at com.cyanea.common.dbtest.dbtest.main(dbtest.java:155) The DB2 driver file is in the db2java.zip file, which is typically located under the instance owner home directory, such as /home/db2inst1/sqllib/java12. Add this to the CLASSPATH environment variable of the managing server components Kernel host properties The kernel host properties are IP addresses of the servers where the kernel processes are running. There are two for each kernel process; if one kernel is killed or crashes then the second kicks in. These two processes are managed by the watchdog processes. Example 3-10 shows the portion of setenv.sh that lists the kernel hosts. Example 3-10 Kernel hosts KERNEL_HOST01= KERNEL_HOST02= To verify that a kernel is available and running healthily, run the./klctl.sh ping command (Example 3-11). This returns the component identification number and the date when the kernel was started. Example 3-11 Kernel is running $./klctl.sh ping Contacting Kernel... Kernel af45f8c3-6c3c b0ea116 up since Thu May 12 13:34:07 PDT 2005 If the kernel is not started, the message will be similar to Example 3-12 on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

73 Example 3-12 Kernel is not running $./klctl.sh ping Contacting Kernel... java.net.connectexception: Connection refused at java.net.plainsocketimpl.socketconnect(native Method) at java.net.plainsocketimpl.doconnect(plainsocketimpl.java:331) at java.net.plainsocketimpl.connecttoaddress(plainsocketimpl.java:196) at java.net.plainsocketimpl.connect(plainsocketimpl.java:183) at java.net.socket.connect(socket.java:478) at java.net.socket.connect(socket.java:428) at java.net.socket.<init>(socket.java:335) at java.net.socket.<init>(socket.java:150) at com.cyanea.kernel.util.kernelmanager.getkernel(kernelmanager.java:84) at com.cyanea.kernel.util.kernelmanager.<init>(kernelmanager.java:129) at com.cyanea.kernel.util.kernelmanager.main(kernelmanager.java:259) Kernel is not running Port definitions Port definitions are split into even and odd numbers by default. Even numbers are for the first components and odd numbers for the duplicate components. Each pair is configured to provide higher availability for required components of WebSphere Studio Application Monitor, such as for RMI, RFS, and CODEBASE. There is no benefit or penalty for changing the default port numbers, though this may be necessary due to firewall port requirements or port conflicts with other applications. Example 3-13 lists all of the ports in setenv.sh that you can change. Example 3-13 Ports definition PORT_KERNEL_CODEBASE01=9122 PORT_KERNEL_CODEBASE02=9123 PORT_KERNEL_RFS01=9120 PORT_KERNEL_RFS02=9121 PORT_KERNEL_RMI01=9118 PORT_KERNEL_RMI02=9119 PORT_ARCHIVE_AGENT1=9129 PORT_ARCHIVE_AGENT2=9130 PORT_MESSAGE_DISPATCHER=9106 PORT_PS=9103 PORT_PS2=9104 PORT_PA=9111 PORT_SAM=9126 Chapter 3. Managing WebSphere Studio Application Monitor 59

74 After changing the ports in setenv.sh, you must restart the component. For example, to restart the first kernel process, you can issue this commands:./klctl.sh kl1.properties stop./klctl.sh kl1.properties start Essential managing server tools Several tools are available under the bin directory of WebSphere Studio Application Monitor managing server. General managing server utilities These utilities are for the managing server components: am-check.sh amctl.sh am-start.sh am-stop.sh Verifies whether all settings are correct. This test can be run manually. The am-start.sh executes this check implicitly. This script can start, stop, ping, get status, and set logging level for all components. Starts the managing server components. Stops the managing server components. Database utilities Certain programs are used for managing the OCTIGATE database in the management server. This first set of programs is used for initializing or maintaining the database. This category includes: Database creation tools, depending on the platform and location of the database: db2configuration.sh db2configuration_lnxaix.sh db2configuration_remote.sh Database schema creation, when the database is already created: db2createschema.sh db2createschema_remote.sh Database installation scripts that are invoked when installing the management server: db2install.sh db2install_remote.bat db2install_remote.sh db2install_remote_int.bat 60 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

75 Database setting definitions for the existing database: db2settings.sh Uninstaller for the OCTIGATE database: db2uninstall.sh Database migration script: migration.sh (Also see 3.4.1, Migration from V3.1 on page 44.) Database utilities also provide some scripts for maintaining database performance: Updating the database statistics: run-stat-cmds.sh Trimming data: datatrimmer.sh and datatrim.sh Other utilities Other programs that are provided in the managing server are: authenticate rotate-apache-logs.sh Used to verify the user s account and password to the operating system for the Web console. Issues the apachectl command to rotate Apache logs. With WebSphere Application Server V5, the Web console must be able to produce graphic images. This can be achieved by using the Virtual Frame Buffer facility or adding a custom property in the visualization engine s JVM: java.awt.headless=true Split server installation When more memory and server resources are available in a distributed environment for installing the managing server, you can perform a split installation of the managing server on two or more servers. Follow these steps: 1. Install the managing server on a single server, following the installation guide. 2. Copy the /opt/ibm/wsam directory to the other server. 3. Configure the KERNEL_HOST1 and KERNEL_HOST2 properties in the setenv.sh files on both machines to point to the server that runs the kernel process. (Also see 3.5.3, Kernel host properties on page 58.) Tip: Keep the port definitions the same on both servers. (See 3.5.4, Port definitions on page 59.) Chapter 3. Managing WebSphere Studio Application Monitor 61

76 3.6 Data collector administration This section discusses administration issues to perform for managing a data collector system Configuration files for data collector Several files in the WebSphere Studio Application Monitor data collector etc directory configure the behavior of the data collector depending on its type. Distributed WebSphere data collector In the distributed WebSphere data collector, the etc directory of the data collector contains the following files: bcm.properties The main instrumentation property file, which governs how WebSphere Studio Application Monitor selects the Java classes to work on. BCMLogMessages.properties, LogMessages.properties List of BCM log messages formats, depending on the language-specific installation. CyaneaDCStore The data collector keystore certificate. CyaneaProxyStore The proxy certificate. cyn.dev.user.classes.xml, cyn.user.classes.xml List of instrumentation-specific classes filter. This file can be referred by the bcm.properties file. datacollector.policy JVM policy file for the data collector. datacollector.properties Common properties for the data collector. dctomgmt.cer Certificate for communication to the managing server using SSL. dctoproxy.cer Certificate for communication to a proxy using SSL. log.properties, LogMessages.properties Log setting for WebSphere Studio Application Monitor data collector. version.properties List of data collector code levels. *.xml Instrumentation files for specific functions that are referred by bcm.properties file. The following generated files are created when the data collector is started. These files may have to be deleted to be regenerated from the original source. 62 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

77 The file name is prefixed with the controller name, which is the WebSphere admin server name, and the application server instance name. <controller>.<server>.am.mod Pointer to the monitoring module dynamic library. <controller>.<server>.bcm.properties Generated from the bcm.properties file; this file must be deleted to activate changes in bcm.properties. <controller>.<server>.cyaneagpscounter.txt This is the persistent file that stores a composite token counter for an application server across restart. <controller>.<server>.datacollector.properties Generated from datacollector.properties file; this file must be deleted to activate changes in datacollector.properties. <controller>.<server>.id Unique ID storage to identify this instance of the data collector. When reinstalling or upgrading the data collector, keeping this file preserves the data collector information in the managing server. z/os WebSphere data collector In the z/os WebSphere data collector, the etc directory of the data collector contains the following files: BCMLogMessages.properties, LogMessages.properties List of BCM log messages formats, depending on the language-specific installation. bcm.properties The main instrumentation property file that governs how WebSphere Studio Application Monitor selects the Java classes to work on. cyn.dev.user.classes.xml, cyn.user.classes.xml List of instrumentation-specific classes filters. This file can be referred by the bcm.properties file. datacollector.env Environment settings for the data collector. datacollector.properties Common properties for the data collector. datacollectorservice.xml An ASCII XML file on z/os for the data collector s custom service information. jndi.properties Initial WebSphere Studio Application Monitor JNDI configuration. Chapter 3. Managing WebSphere Studio Application Monitor 63

78 version.properties *.xml List of data collector code levels. Instrumentation files for specific functions that are referred by bcm.properties file. The following generated files are created when the data collector is started. These files may have to be deleted to be regenerated from the original source. The file name is prefixed with the cell name, cell ID, node name, and application server name. <node>.<server>.bcm.properties Generated from the bcm.properties file. Must be deleted to activate changes in bcm.properties. <cell>.<lpar>.<node>.<server>.datacollector.properties Generated from the datacollector.properties file. Must be deleted to activate changes in datacollector.properties. <cell>.<lpar>.<node>.<server>.gpscounter.txt Global publish server counter for composite transactions. <cell>.<lpar>.<node>.<server>.id Unique ID storage to identify this instance of the data collector. When reinstalling or upgrading the data collector, keeping this file preserves the data collector information in the managing server. CICS and IMS data collector In the CICS and IMS data collector, the etc directory of the data collector contains the following files: LogMessages am.mod Log messages format; depends on the installed language. Pointer to the WebSphere Studio Application Monitor dynamic library. datacollector.env Environment settings for the data collector. datacollector.policy Data collector JVM policy file. datacollector.properties Main data collector configuration file. version.properties Data collector modules version list. The following generated files are created when the data collector is started. These files may have to be deleted to be regenerated from the original source. For IMS data collector, the file name is prefixed with the network name (defined in the datacollector.env am.ims.network parameter), sysplex name, sysplex ID, 64 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

79 z/os SMF ID, and IMS subsystem name. For CICS data collector, the file name is prefixed with sysplex name, sysplex ID, z/os SMF ID, and CICS application ID. <prefix>.datacollector.properties Generated from the datacollector.properties file; must be deleted to activate changes in datacollector.properties. <prefix>.gpscounter.txt Persistent file that stores a composite token counter for an application server across restart. <prefix>.id Log-level settings Unique ID storage to identify this instance of the data collector. When reinstalling or upgrading the data collector, keeping this file preserves the data collector information in the managing server. Data collector logging can be changed using the following methods: Change the datacollector.env file to include the am.debug property. This property can be set as no, yes, or trace, which corresponds to logging levels 0, 1, and 2. The change in datacollector.env is applied when the data collector is restarted. From the managing server, run the dcctl.sh command. This enables the logging level to be changed dynamically at run time. The argument of this command is the logging level, which can be info, debug, or trace. It then prompts for any connected data collector to change. Example 3-14 shows running this command. Example 3-14 Log-level configuration data collector $./dcctl.sh trace 0 BDCELL.A0B.BDNODEA:bdsr01a 812b6712-b4c8-d901-99d4-b48627d0a7cd ITSOAUS.PLEX58.A0B.SC58:IMSB a1346c86-54c2-d901-c291-f3fbac754b khartoum:khartoum e3c3-d901-2a0d-00096be9c perth:perth1 61e6b2aa-bdbd-d e > 2 In the example, the log file with the changes is in the data collector logs directory. The file is located at: $DC_HOME/logs/<ppe-controller>.<ppe-probe>.%%.datacollector.log When you change the logging level, you see the following message in the log: CYND100I Logging level changed from 0 to 2 Chapter 3. Managing WebSphere Studio Application Monitor 65

80 3.6.3 Data collector log files cleanup A data collector log file is created when you start the data collector. It contains useful information regarding the data collector itself. However, as this file is never deleted, it may fill up your disk so you must clean up the data collector log files regularly. For most installations you do not need to maintain a data collector log, as these log files contain only diagnostic information of the collector, not the application server. You can delete all log files in the $DC_HOME/logs except for the current data collector log. The currently running data collector has two log files: am.datacollector.log <ppe-controller>.<ppe-probe>.%%.datacollector.log You can use the rm $DC_HOME/logs/*.datacollector.log command to remove all log files in the logs directory. This removes all unused log files. 66 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

81 4 Chapter 4. \Managing production environment This chapter discusses the use of WebSphere Studio Application Monitor in a production operation environment. The discussion here consists of: 4.1, Production environment definition on page , An operator s typical day on page , Reports on page , Performance tuning on page , Notification on page , Working with composite applications on page 88 Copyright IBM Corp All rights reserved. 67

82 4.1 Production environment definition This chapter discusses tips and scenarios for managing a production environment. We expand on the basic premise that the production environment must be managed with minimal overhead to the environment. Moving beyond the L1 monitoring level, we investigate how WebSphere Studio Application Monitor is typically used in day-to-day monitoring. The primary discussion covers how an operator uses WebSphere Studio Application Monitor for regular monitoring in the environment in 4.2, An operator s typical day on page 68. Apart from online monitoring, some predefined reports may be generated automatically for review. These reports can be used to review a historical period for any performance anomaly that must be investigated further. Some reports may be run on a daily basis, some weekly, some monthly. This is discussed in 4.3, Reports on page 70. Regular monitoring cannot be separate from planning and improving the application environment, so we discuss performance tuning in 4.4, Performance tuning on page 78. Anomalies occur and support personnel must be notified about potential problems. This can be achieved through WebSphere Studio Application Monitor notification trap and alert management. We discuss this in 4.5, Notification on page 83. When things go wrong, further investigation is necessary. This is discussed in Chapter 5, Analyzing WebSphere application problems on page 99 and Chapter 6, Analyzing composite application problems on page An operator s typical day An operator uses WebSphere Studio Application Monitor to perform monitoring on an application environment. The environment has been set up for the operator s job by the WebSphere Studio Application Monitor administrator as discussed in Chapter 3, Managing WebSphere Studio Application Monitor on page 33. First, the operator collects and reads the scheduled reports that WebSphere Studio Application Monitor automatically generates in Adobe Portable Document Format (PDF). The reports, which we discuss in 4.3, Reports on page 70, are 68 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

83 evaluated to catch any trends and anomalies that may affect overall application health. Some important reports that should be evaluated are: Memory usage: for possible storage growth and out-of-memory conditions Response time report: for possible response time problems and trend analysis of application response time Transaction volume report: for trends that may affect server capacity Server statistics overview: for a glance at server statistics to spot any specific anomalies that may lead to problems The operator should also check the folder for WebSphere Studio Application Monitor trap notifications that may have been generated overnight. These trap and alert notifications are discussed in 4.5, Notification on page 83. Special notifications should show specific items that may be masked in the average report results, such as an individual transaction response time, sudden memory growth, or overnight deadlock conditions. After signing on to WebSphere Studio Application Monitor, the summary display appears. Depending on the job role, the operator may be assigned to a single server group or multiple server groups. If the operator is assigned to multiple server groups, the initial display should be the ENTERPRISE OVERVIEW. For a single server group, the initial display should be set to SERVER GROUP OVERVIEW. Typically an operator will not be assigned to a single server, but the operator can also have the SERVER OVERVIEW screen as the initial page if desired. These overview pages are typically a good initial page as: They give an overall indication of system health, showing at least the transaction volumes and response time averages. They are dynamically updated, so the operator does not need to refresh the page every time. Upon learning of a specific issue, such as response time peak or significant transaction rate changes (higher or lower), the operator should take action on it immediately by drilling down to the appropriate level to find the cause. It may be a normal system response caused by an unexpected load burst, or it may indicate a potential problem. Some typical cases are: Deadlock: A built-in transaction that has significant response time surges, sometimes with decreasing input load as most of the users are waiting for the transaction to complete. Remote application problem: A timeout caused by unavailable resources has a significant response time increase with a normal inbound rate. Chapter 4. \Managing production environment 69

84 Load burst: A sudden additional load followed by an increase of response time. If the increase is more than proportional to the load increase, then it may be an application problem that is not tuned properly to handle more load. Some special situations may occur for new or changed applications. These changes require the operator to take on additional monitoring tasks. This includes performance tuning for the new application, as discussed in 4.4, Performance tuning on page 78. Also, additional reports may be generated to understand the impact of the new or changed application in the production environment. 4.3 Reports Report types WebSphere Studio Application Monitor offers an extensive set of reports about collected data. These are accessible via PERFORMANCE ANALYSIS CREATE REPORTS. The data in these reports is from the database and offers the best look at historical data. Reports are useful for: Providing summary information about application server performance in an interval. This can also be scheduled regularly with PDF output. Showing long-term trends for growth of resource usage to allow preventative maintenance to occur. Assisting in capacity predictions of potential bottlenecks to allow for maintenance upgrades or preparation of new hardware. Comparing information from different time intervals, such as having a baseline for a good day to compare with a bad day. This section contains descriptions of the report types that are available with WebSphere Studio Application Monitor: application reports and server reports. Application reports Six reports are designed to monitor applications: Request/Transaction Compares requests over time and drills into specific requests and their transaction breakdown/flow. Useful for finding past errant behavior and examining the causes by going into detail using the drill-down feature. 70 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

85 Method/Program SQL MQI Lock Analysis Top Reports Similar to the Request/Transaction report but based on methods instead of requests to cover more data. Best used with the Request/Transaction name or Method/Program filters to narrow the amount of data displayed. Determines the frequency and response times of SQL statements. Best used to determine either slowest or most frequent SQL statements to investigate performance of specific SQL statements using the filtering (for example, response times of UPDATE calls to a specific table). Similar to the SQL report in that it determines slow or frequent MQ statements either through filtering or broadly without filtering. Lock data is collected only if Lock Analysis is turned on. (See 5.3, Deadlock management on page 112.) Lock Analysis reports help identify the frequency, source, and duration of locks. Basic table reports for finding slowest methods, slowest SQL call, and so on. No further drill-down is available, but the aggregate All Servers from a specific group option is useful to quickly see from a high level what is slow or frequently called across the monitored environment. Chapter 4. \Managing production environment 71

86 Figure 4-1 shows a sample application report comparing results from two days. Figure 4-1 Report comparison of two different days 72 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

87 Typically the application report is presented in a trend report with bar charts, and you can drill down into a decomposition report that provides a breakdown of each bar entry into a pie chart by transaction type or transaction name. Figure 4-2 shows the decomposition chart by transaction name. Figure 4-2 Request report decomposition Chapter 4. \Managing production environment 73

88 Drilling down from the decomposition, you can see additional detail, such as method trace, transaction stack, and SQL call details. Figure 4-3 shows transaction detail for a specific transaction. Figure 4-3 Request report details after drilling down Server reports WebSphere Studio Application Monitor has three server reports to monitor overall server information: System Resource Server Availability Capacity Analysis Reports on overall system resources such as CPU and memory with no drilldown offered. Good for judging memory growth, active sessions, and so on. Useful for SLAs, this report is based on every minute check of the server availability that WebSphere Studio Application Monitor does. Analyzes information and generates usage trends. This report plots either throughput per minute or number of users on the X-axis against system CPU, JVM/process CPU, JVM/process memory, thread pool, connection pool, or response time on the Y-axis. 74 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

89 4.3.2 Performing capacity planning Figure 4-4 shows capacity planning choices. Figure 4-4 Server metrics for capacity analysis Because throughput per minute is transient based on the number of requests, it is best gauged against relatively quickly changing metrics such as CPU and response time. On the other hand, users are on a server for longer and are better compared with fewer fluctuations metrics such as memory and thread pools or connection pools. As these are long term capacity reports, the time range should span a larger amount of time. At least a few days of data, if not a week or more, is best. Figure 4-5 on page 76 shows a generated report. Note: Because the capacity analysis reports plots every data point to the minute, generating these reports can take a fair amount of time on the server. Chapter 4. \Managing production environment 75

90 Figure 4-5 Capacity Analysis report showing memory against users The report in Figure 4-5 shows a typical graph with a trend line, which is a best fit line between the scattered data points. The trend lines can be used to plan future capacity by extrapolating future needs against currently available resources Report usage There are many WebSphere Studio Application Monitor Performance Analysis reports, and users will not use every report. Different users will find different reports more useful. Some reports are best scheduled as recurring reports. To generate recurring reports, the save as PDF feature must be enabled, otherwise you get error CYNVE0504E. For more information about enabling recurring reports, see the WebSphere Studio Application Monitor Version 3.2 Installation Guide, SC in Appendix F. System operators will use a broad selection of the reports, narrowing down problems so that subject matter experts can diagnosis exact causes of problems. A main focus area for day-to-day monitoring is Request/Transaction reports focusing on response time and throughput to learn transaction frequency and length. Another focus area is System Resource reports to ensure that capacity of memory, CPU, or pools is not approaching critical 76 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

91 levels. Availability reports chart service unavailability. A system operator will probably set up many reports to create daily recurring PDF reports for comparison analysis and to to other interested parties. Application Server administrators focus on application server performance and availability at a deeper level than system operators. In addition to the Request/Transaction reports, the Application Server administrators uses the Method/Program report to focus on specific problems in the Application Server. The System Resources reports are useful for tuning the Application Server s configuration by analyzing memory and connection/thread pool usage. Finally, the Lock Analysis report is useful for analyzing lock use with an application indicating deadlocks or prolonged wait times on locks. Database administrators will focus on the SQL reports to examine overall database request trends and data as well as specific poorly behaving SQL statements. In addition, the JDBC connection pool report in System Resources can provide insight into how an application server is utilizing the database. Capacity planners will focus on the Capacity Analysis reports to determine resource-against-usage trends to figure out which hardware and maintenance areas to focus on. Sample usages tips for WebSphere Studio Application Monitor reports include: Most of the reports offer Server Selection choices of All Groups or All Servers. These aggregate reports are useful for big-picture trends and high-level understanding, but averaging can mask problems when used for investigating specific issues. Many of the reports have common Report Filtering Options fields, similar to Figure 4-6. Request/Transaction Name refers to the first transaction in the transaction flow, often servlets or JSPs. Method/Program Name refers to specific Java methods that are invoked during the transaction flow. Figure 4-6 Typical Report Filtering Options Chapter 4. \Managing production environment 77

92 WebSphere Studio Application Monitor provides useful default date ranges, but note that the Previous 7 Days and Month to Date options do not include the current day s data. (They stop at the end of the previous day.) This is because today s information is not considered complete yet. In the Graphing Option section, the default X-axis configuration is Aggregate Hour of the Day (see Figure 4-7). The Aggregate options are useful for generating load profile (how the application performs in a cycle), and the time series is more useful to see trends and anomalies. The Time Series options create a linear graph, and the Aggregate options average all data points for that time range (from multiple intervals) into one data point. For example, the Time Series in Hour choice would have one data point for each hour in the Date Range. The Aggregate Hour of the Day would have 24 data points regardless of the Date Range, with each point representing the average value for all hours in that Date Range. Figure 4-7 Graphing option X-axis configuration choices If the reports seem slow, the database may have to be reorganized. See 3.4, Database management on page 44. Many of the reports have multiple levels of drill-down features. Navigating back and forth can be performed using the navigation menu on the left side of the screen. Saved reports in WebSphere Studio Application Monitor are the exact replica of the viewed report including the time range. So if a saved report s time range was configured for the previous 24 hours, when viewed again later it will show the original time range, not the current 24-hour period. 4.4 Performance tuning This section discusses some important performance tuning tips for the J2EE application server related to WebSphere Studio Application Monitor usage. The discussion is meant to be generic with some specific tips for ways WebSphere Studio Application Monitor can assist you in tuning your application server. 78 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

93 Performance testing of the J2EE application is an important stage in the development cycle, so J2EE application server performance tuning should be carried out before deployment in a production environment. This section is by no means a complete list of recommendations to tune up application servers; we explain common and most frequent performance problems administrators should understand when investigating their systems with WebSphere Studio Application Monitor. Performance tuning can be viewed from different perspectives: Development perspective Java programmers naturally produce design and coding errors. Fortunately WebSphere Studio Application Monitor is capable of identifying many of them. Administrators should understand the problem scope in order to analyze and report them correctly. Administration perspective J2EE applications strength is in declarative programming. Many application features are set during application assembly, others during deployment. Application server services must be properly configured, a task that involves another layer of application performance tuning. Runtime perspective Building a scalable application is key to enabling its growth when deployed in terms of throughput and load (number of users and transactions it can support). Scalable application components can then be distributed on specialized hardware by the load type thus increasing throughput and load Development perspective Development errors occur mostly in the design phase when selecting what Java components should run in what environment. Standard Java performance-related errors The most common standard Java errors involve memory usage. Fortunately WebSphere Studio Application Monitor allows memory leak analysis. (See 5.2, Memory leaks on page 100.) If WebSphere Studio Application Monitor indicates many very short-lived objects, consider using pools of preloaded objects and making sure objects get released back to the pool after use to avoid memory leaks. Storing objects in thread-safe collections is memory expensive; use lightweight collections such as ArrayList instead. Improperly managed collections are another source of frequent memory leaks. Chapter 4. \Managing production environment 79

94 Using Java reflection is CPU expensive; generally it should be avoided. WebSphere Studio Application Monitor helps to identify Java reflection problems in the Trace report for the selected request. If the Event Type column of this report shows an excessive number of classes from the java.lang.reflect package (especially the java.lang.reflect.method class), this might indicate poor Java code design style resulting in a performance penalty. Use standardized logging mechanisms such as log4j; using System.out is resource expensive. Enterprise Java performance-related errors Web component errors: Use small objects to save into HTTP session. These objects cannot be garbage collected. EJB component errors: Use Singleton and Service locator patterns to cache initial context references. Frequently, the synchronize keyword is used for factory lookups that may cause thread lock if designed improperly. WebSphere Studio Application Monitor provides extensive lock analysis. See 5.3, Deadlock management on page 112. Minimize remote calls for EJB; use Session facade and associated patterns such as Transfer object. Invoking fine-grained methods remotely severely hurts application performance. WebSphere Studio Application Monitor helps to identify these problems in the Servlet coverage and JNDI sections of the System Resources Overview page. Excessive remote calls and JNDI lookups per minute might indicate poor EJB distribution design or improper enterprise application configuration. For example, closely collaborating objects such as Order and Customer implemented as Entity Beans should be collocated within the same server. Avoid using statefull session beans. They are the slowest EJBs and often designed improperly. Data source errors: Release the SQL connection after use; not closing the connection means not returning it to the connection pool, so the application server is forced to maintain connections much longer than required. After getting the connection from the data source, creating statements, and executing and retrieving result sets, these should be closed in reverse order, connection last. All of the closing code must be in the final section of the try - catch block, so it is called regardless of the method result. Avoid creating the same dynamic SQL statements over and over. Use a cached PreparedStatement instead, then set the values to it as parameters. 80 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

95 Group the same SQL tasks (for example, INSERT) into a single task using Statement.executeBatch(). Avoid getting column values from the result set using get[columnname]. Use get[columnindex] instead. Avoid using an asterisk (*) in SELECT statements. Always specify only the required columns. WebSphere Studio Application Monitor support for identifying database problems is discussed in 5.5, Database performance on page Administration perspective Configuration errors may not be visible immediately after rolling out a new application, but with increasing load users experience response time problems, frequently due to an improperly customized environment: Adjusting System queues: WebSphere has a series of interrelated components that must be tuned to support the needs of large e-business applications. See the WebSphere Application Server documentation on adjusting EJB containers and data sources. Java memory tuning: Tune Garbage Collection after identifying overutilization of objects and memory leaks using WebSphere Studio Application Monitor. Java heap parameters should be adjusted too. See WebSphere Application Server documentation. Avoid memory thrashing at all costs. Relax auto reloads: After application resources such as Web and EJB components are fully deployed it is not necessary to aggressively reload these resources. Reload settings can be changed in WebSphere Application Server console. Verify whether component load is necessary at server startup and component reload interval. Component caching can greatly enhance application response time: Static content in the HTTP server Fragment caching in the application server Dynamic content in the application server: EJB caching can be modified; select Servers Application servers <server name> EJB container EJB Cache settings and check Deployment infrastructure. Data source settings: Let WebSphere Studio Application Monitor help you to right-size the connection pool size. If the situation described in 5.4, Analyzing a slow-performing application on page 124 occurs, Connection pool size should be increased in WebSphere Application Server 5.x console: Resources JDBC Providers <select JDBC provider> Data sources <select data source> Connection pool, and modify the Max Chapter 4. \Managing production environment 81

96 connections attribute as shown in Figure 4-8. The Max connections attribute specifies the maximum number of physical connections that can be created in this pool. After this number is reached, no new physical connections can be created. When modifying this attribute, consulted with the database administrator. Figure 4-8 Connection pool attributes in WebSphere administration console Runtime perspective Scalable applications should be composed of components supporting specialized tiers, with each tier supporting a specific type of load so that tuning hardware infrastructure can be based on it. Consider using dedicated servers by tier. Static Web server: A dedicated server should be used for reasons of security and type of load. Web servers must be tuned for the HTTP network. WebSphere Studio Application Monitor can display Apache Web server statistic based on the server status page. Web modules tiers: Servlets and JSPs must be tuned to produce dynamic content using template services for memory-intensive processing. A dedicated server might be required, and usually one JVM per server is the best choice. 82 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

97 EJB modules: The EJB tier serves the business logic. It is usually memory intensive and CPU intensive. A dedicated server required; usually one JVM per server is the best choice. Collocate EJB client and EJB server applications on the same application server if possible. Avoid splitting up EJB modules between different servers. Data access layer: Database interaction modules or JCA adapters. Dedicated servers might be required. For WebSphere Studio Application Monitor support for data sources, see 5.5, Database performance on page 134. Database servers: Dedicated servers are required. Consider clustering to distribute workload. WebSphere Studio Application Monitor provides a number of features to monitor applications residing in different nodes, such as comparing installed applications, comparing application behavior on different servers, and so on. Vertical clustering: Clone application server instances on the same server. Horizontal clustering: Clone application server instances to multiple servers. Tune the operating system. Tune TCP/IP settings. Tune page size (such as using 16 MB page for Power4). Introduce level 3 caching on Intel to benefit EJB modules. Use alternate thread library in Solaris. 4.5 Notification In the WebSphere Studio Application Monitor Web console menu, use PROBLEM DETERMINATION TRAP & ALERT MANAGEMENT for setting traps and alerts to monitor a single server or a group of servers. An action can be invoked when the system meets the conditions of the trap. The action can be: Sending notifications such as an SMTP mail or SNMP trap Collecting a stack trace Performing a component trace or method trace Generating a thread dump Use PROBLEM DETERMINATION TRAP & ALERT MANAGEMENT to set these functions. Figure 4-9 on page 84 shows the trap and alert management dialog. Chapter 4. \Managing production environment 83

98 Figure 4-9 Trap and alert management Notifications should be used when monitoring a group of servers or a selected server for the following events: Prevent disruptions in service by receiving alerts before problems arise. Gather data that helps pinpoint the root cause of problems that are difficult to reproduce. Alert immediately when servers, applications, components, or methods are not healthy, and obtain the data necessary for diagnosis. Note: Traps may add to the overhead used by your system, so use them sparingly. For sending SMTP (outgoing) , the SMTP host must be specified by editing the <WSAM_MS_HOME>/etc/md.properties file to specify the correct IP address or host name of the SMTP host, such as: smtp.host=xxx.xxx.xxx.xxx After the SMTP host has been configured in the md.properties file, a valid sender s address must be specified for receiving trap and alert s 84 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

99 and WebSphere Studio Application Monitor performance analysis reports. This is accomplished by editing the ve.framework. .sender value in the <WSAM_MS_HOME>/etc/ve.properties file to include a valid address: ve.framework. .sender=admin@wsamserver By default, the WebSphere Studio Application Monitor Web console port number is specified as port 80. This is specified in the publish server property files ps1.properties and ps2.properties. Modify these settings if necessary. The definition is shown in Example 4-1. Example 4-1 Publish server Visualization Engine properties ve.host=bandung.itsc.ibm.com ve.port=80 Example 4-2 shows a sample generated by a trap. Example 4-2 Sample trap Trap Test has been triggered Date : Mon Mar 28 14:39:38 PST 2005 AppServer : lima:websphere_portal Request : N/A Method : N/A SQL : N/A Resource : Avg Platform CPU Usage Trap Condition : Server Resource Condition Offending Content : N/A Threshold : 1 avg % MaxMin : Maximum Number of Hits : 1 Offending Value : 3.55 avg % Severity : Low Another notification option is to generate an SNMP event. This event can be send to an SNMP manager, such as IBM Tivoli NetView or to IBM Tivoli Enterprise Console with SNMP service. To send the event to Tivoli Enterprise Console, you can use either of these: Tivoli Enterprise Console non-tme SNMP Adapter Tivoli Enterprise Console Adapter Configuration Facility Tivoli Enterprise Console SNMP Adapter enables alerts from WebSphere Studio Application Monitor to appear in the Tivoli Enterprise Console. For more information about installing and configuring this SNMP adapter, see WebSphere Studio Application Monitor Version 3.2 Installation Guide, SC Chapter 4. \Managing production environment 85

100 Send the SNMP event to the Tivoli Enterprise Console SNMP adapter by configuring it from the menu ADMINISTRATION MANAGING SERVER SYSTEM PROPERTIES and clicking on the SNMP link. Figure 4-10 shows the SNMP configuration page. Figure 4-10 SNMP configuration page 86 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

101 Figure 4-11 shows a sample TEC event generated by WebSphere Studio Application Monitor. Figure 4-11 Sample TEC event from WebSphere Studio Application Monitor Identify a system as the SNMP adapter host, which accepts SNMP traps from WebSphere Studio Application Monitor Management Server, adapts them to Tivoli Enterprise Console Events, and forwards them to a Tivoli Enterprise Console Server. To avoid interference, we recommend that the system running the SNMP Adapter not run any other SNMP agent on the same UDP port. Chapter 4. \Managing production environment 87

102 4.6 Working with composite applications The composite application feature of WebSphere Studio Application Monitor enables correlation of a transaction started in a J2EE server with a back-end transaction in IMS or CICS. Table 4-1 shows the possible scenarios. Table 4-1 Composite application scenarios J2EE server Interface Back-end WebSphere z/os or UNIX CICS Transaction Gateway CICS WebSphere z/os or UNIX MQI CICS WebSphere z/os IMS connect IMS WebSphere z/os MQI IMS Note: The composite application correlation is not limited to a single hop application server. A chained application in multiple IMS or CICS systems is allowed. The IMS correlation is limited to the WebSphere on z/os platform, but the CICS can be from a distributed platform. For production environment, L1 monitoring of CICS and IMS yield only the basic information for the platforms. The composite application is shown in in-flight request search and application reports. The composite indicator is shown in the first column. In the subsections, we discuss these features for CICS and IMS. Find more information in Chapter 6, Analyzing composite application problems on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

103 4.6.1 CICS composite application CICS monitoring shows a system display similar to Figure Figure 4-12 CICS system summary Chapter 4. \Managing production environment 89

104 In level 1, the application report shows a transaction that runs composite transactions. From the decomposition pie chart, you can open the Servlet transactions. The composite transactions are indicated by the composite transaction icon on the left, as shown in Figure Figure 4-13 Servlet transaction with composite feature 90 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

105 Similarly, when you display the CICS transactions from the decomposition pie chart, you can see the correlated CICS transaction as shown in Figure Figure 4-14 CICS transaction with composite features Chapter 4. \Managing production environment 91

106 Going into more detail in the Servlet transaction mode, you can see the Servlet request detail, shown in Figure Figure 4-15 Servlet request detail 92 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

107 The Drilldown View tab shows the initial breakdown (Figure 4-16). Figure 4-16 Drilldown view of the servlet Chapter 4. \Managing production environment 93

108 Figure 4-17 shows the CICS transaction detail. Figure 4-17 CICS transaction detail Not much detail is available in the drilldown view or the flow view. 94 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

109 4.6.2 IMS composite application The composite IMS application generates a decomposition report in level 1 monitoring as shown in Figure Figure 4-18 IMS decomposition report Chapter 4. \Managing production environment 95

110 Clicking the IMS Transaction (TRADERBL in Figure 4-18 on page 95) presents a detail view as shown in Figure Figure 4-19 TRADERBL IMS transaction detail list 96 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

111 Further drilldown from the composite icon does not provide additional detail for level 1 monitoring, as shown in Figure The arrow contains the link to the J2EE transaction if available; otherwise it will display the text Event: INVOKE. Figure 4-20 Flow trace for IMS transaction in level 1 Chapter 4. \Managing production environment 97

112 The IMS transaction can also be seen in the Server Activity Display under PROBLEM DETERMINATION SERVER ACTIVITY DISPLAY. The recent transaction display is shown in Figure Figure 4-21 Recent transaction display 98 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

113 5 Chapter 5. Analyzing WebSphere application problems This chapter provides some sample scenarios for analyzing application problems in WebSphere Application Server. The chapter includes: 5.1, Problem scenarios on page , Memory leaks on page , Deadlock management on page , Analyzing a slow-performing application on page , Database performance on page 134 Copyright IBM Corp All rights reserved. 99

114 5.1 Problem scenarios This chapter discusses various problems on a WebSphere Application Server platform that can be resolved using WebSphere Studio Application Monitor. The scenarios presented here are by no means exhaustive but may help with solving other problems or a combination of problems. Subjects include: Memory leaks Deadlock management Slow-performing applications Database performance The scenarios that we chose to implement represent a subset of scenarios that we consider to be representative of most situations. We created and tested these scenarios in our testing machines; hence the scenarios are somewhat more simplified than real-life production environment problems. See 1.4, Project environment on page 6 for the environment that we used in these scenarios. The scenarios are presented in a problem-solving approach with performance tips when appropriate. At the end of the scenario we provide a description of the elements that are necessary to recreate the scenario. All sample programs are provided as additional material. See Appendix C, Additional material on page 191 for more information. 5.2 Memory leaks Symptoms These scenarios analyze memory leak situations that can affect daily operations. The typical memory leak problem includes the following tasks: Identifying the symptoms Investigating a potential memory leak Narrowing the cause of the memory leak Determining the memory leak offender During day-to-day monitoring of the application servers, memory usage is high or increasing, indicating a possible leak. However, these early indications do not signify a specific leak, and more investigation should be done. Indications of a potential memory leak can be acquired when the WebSphere Application Server memory usage report is evaluated. We select PERFORMANCE ANALYSIS CREATE REPORT SERVER REPORT SYSTEM RESOURCE and use Amount of Memory Used for the last month 100 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

115 with Time Series by Day as the X-axis option. Figure 5-1 shows a sample memory growth report. Figure 5-1 Memory growth report Investigating a potential memory leak When a memory leak situation is suspected, investigation begins. The investigation is performed in the following steps: Memory Leak Confirmation reports Trend reports for slower leaks Using traps to alert for low memory Memory Leak Confirmation reports WebSphere Studio Application Monitor has pre-built reports that are useful for determining whether memory growth is related to memory leak activity. To access these reports choose PROBLEM DETERMINATION MEMORY DIAGNOSIS MEMORY LEAK. Chapter 5. Analyzing WebSphere application problems 101

116 The Step 1 Memory Leak Confirmation window (Figure 5-2 on page 102) shows three reports. Each report is labeled by the question it answers and the data it presents. These reports first show the memory increase and the relationship of memory increase to number of users and volume of transactions. Figure 5-2 Three pre-built memory leak reports for memory leak confirmation The first option graphs the memory growth over time. Memory growth is only shown after Garbage Collection (GC) as this represents the real amount of used memory. However, memory growth can also occur because there are more users or more activities causing more Java objects to be allocated. Therefore the other two reports show useful comparisons for memory growth. 102 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

117 Figure 5-3 shows an upward memory growth trend even though the number of live sessions stays steady around 50 sessions. Such upward memory growth coupled with steady metrics is an indication of a memory leak. Figure 5-3 Memory growth with a constant average number of live sessions Similarly, the report for heap size after garbage collection compared to the number of requests helps indicate whether server load is affecting the memory growth. Actively executing Java methods adds objects to the heap to complete their tasks, but as soon as the requests are finished, the memory usage should be freed accordingly. Chapter 5. Analyzing WebSphere application problems 103

118 Figure 5-4 shows a slow upward growth in memory for the first 45 minutes and a corresponding large number of requests per minute. When those requests drop to zero in the last 15 minutes, the heap size flattens but does not drop as expected. This graph indicates that there could be a memory leak because the rise in memory occurs with a consistent average number of requests but does not fall when the number of requests falls. Figure 5-4 Heap size does not decrease when the number of requests drops Note: The three canned reports in the Memory Leak report are more useful than the three offered in the Memory Analysis report s JVM Heap Size section. The Memory Leak reports use the average heap size after garbage collection as their baseline, but the Memory Analysis canned reports use JVM Heap Size. When investigating memory leaks, doing a garbage collection beforehand is recommended so that no allocated but unused objects reside in memory for determining the memory size. By default, the three Memory Leak graphs show the past 60 minutes of data, but that time range might be too narrow so there is an option to change the range to the past 48 hours. You can also change any of the mapped metrics as well as the server and group to quickly compare a different server s memory growth. Trend reports for slower leaks The reports available in the MEMORY LEAK menu are useful for detecting leaks, but sometimes memory leak develops over more than the two-day period those reports show. In these cases, you can use the trend reports in the 104 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

119 PERFORMANCE ANALYSIS menu. In the System Resource section of the menu, the best metric to choose for leak detection is Amount of Memory Used (MB). This metric does not show the size of memory after garbage collection but over long time span; this does not matter. Note: You should not use the aggregate options in the trend reports. Choose a specific server to monitor, not all servers in a group, as small amounts of growth could be lost in the aggregation. Similarly, in the Date Range Page set the X-axis to Time Series in Hour or Time Series in Day depending on the range of data. The default of Aggregate Hour of the Day averages all common hours of the days, which smooths out any growth in memory. Using traps to alert for low memory Another option for being alerted of a memory leak is to define traps. When memory crosses a defined threshold, WebSphere Studio Application Monitor can send an or an SNMP alert. These traps do not definitely indicate a memory leak, but can alert to a low available memory situation that could be caused by a memory leak. These alerts can help prevent Out Of Memory Error and let you take preventative actions such as stopping some of the enterprise applications, increasing the heap size, or doing further memory leak investigations. To set up a trap, choose PROBLEM DETERMINATION TRAP & ALERT MANAGEMENT CREATE TRAP to get to Figure 5-5. Figure 5-5 Memory trap choices Chapter 5. Analyzing WebSphere application problems 105

120 In the System Resource Trap, select the target type. There are four options relating to memory as shown in Figure 5-5 on page 105: JVM Heap Size The JVM heap size approaches a certain percentage of the maximum memory available for the JVM. This indicates a low-memory situation. Garbage Collection Frequency Garbage collection runs too many times within a minute, which indicates that the garbage collection is not very successful. Garbage Collection Time Garbage collection takes too long, which indicates that many objects are being allocated that have to be examined. Average JVM Heap Size After Garbage Collection The JVM heap size exceeds a certain percentage of the maximum available memory but only after a garbage collection indicating the true amount of used memory. The most useful trap for potential memory leaks is Average JVM Heap Size After Garbage Collection because it counts only current objects or leaked objects. The next screen (Figure 5-6) is for configuring the trap. Put the Threshold here. A useful percentage number is highly system-dependant, but a valid range is probably between 80% and 95%. Figure 5-6 Configure the trap Narrowing the cause of a memory leak When a leak is suspected, WebSphere Studio Application Monitor has a Memory Leak Candidate Finder to help find leaking classes. The leak candidate report takes two snapshots of the suspected JVM, and compares the older snapshot to the newer one to determine which objects have increased its size in the time between the snapshots. 106 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

121 Configuring a memory leak candidate report The Memory Leak Candidate Finder configuration page is step 2 of the Memory Leak report section. As shown in Figure 5-7, specify the group server and the wait time, which indicates how long to wait between two snapshots of the JVM heap. This wait time depends on the rate of growth of the leak. If initial memory leak investigation indicates a noticeable growth in just an hour, then a wait time in as little as 5 or 10 minutes can be a good candidate. If the growth is on the order of a day or more, then a wait time of at least a few hours is recommended. Note: Ideally the snapshot should be taken when there is little or no load to ensure that all objects are in use. Figure 5-7 Configuring memory leak candidate finder Note: The monitoring level can be L1, L2, or L3 to do the snapshots. Analyzing a memory leak candidate report When the report has finished collecting both snapshots, clicking the server name shows the Analysis Report for the first snapshot, and the same report is available for the second snapshot. This report shows the objects that were in memory when the snapshot was made, as well as associated metrics such as the percentage of memory they took. The Comparison Data report is more useful because it shows only classes that have changed between the two snapshots. Figure 5-8 on page 108 shows the comparison report sorted by the change in the number of instances in memory. The top entries in this table are the most likely leaks because these are the Chapter 5. Analyzing WebSphere application problems 107

122 classes that have grown the most. We will use these classes to actually determine the memory leak. Figure 5-8 Memory Leak candidate comparison report Tip: By default, classes that are filtered out by the Monitoring configuration are excluded from the analysis. Though you may not want to monitor the basic classes such as Strings and Lists, these objects can often be a source of leaks and should be examined. To do this, clear the Exclude section of the Classname Filter Option and click Apply to see everything in the heap. (This is done dynamically without re-executing the report.) Then selectively exclude classes and packages that are not a concern. Finally, sort from most to least the column that shows change in the number of instances. The result will be a richer look at the objects that are growing in memory Determining the memory leak offender With potential leaking classes identified from step 2 of the Memory Leak report, determine which classes are causing the leak in step 3, the Memory Leak Diagnosis report. This report finds the most-leaked objects and shows the class, method, and line number of the allocating class of leaked objects to help the programmer determine the trouble spots of code. 108 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

123 Configuring the Memory Leak Diagnosis report To set up the Diagnosis report, reconfigure the data collector to monitor the growth classes found in 5.2.3, Narrowing the cause of a memory leak on page 106. The Memory Leak Diagnosis report is set up in an XML file on the data collector in the <am home>/etc directory. The bcm.properties file has a key/value pair called userbcm.xmlfilename that identifies which XML configuration file is used. Adjusting the bcm.properties files allows for multiple XML files to be swapped in and out. Note: WebSphere Studio Application Monitor generates a server-specific bcm.properties file called <host name>.<app server name>.bcm.properties. When changes are made to the original bcm.properties, the generated version must be deleted so that WebSphere Studio Application Monitor knows to regenerate the bcm.properties. The XML file is composed of multiple sections of selectclass tags that identify how to treat classes matching the mask pattern. The XML snippet in Example 5-1 shows the modification to monitor memory leaks. Example 5-1 XML for monitoring memory leak <selectclass> <mask>*</mask> <methods> <methodname>*</methodname> </methods> <ignoretrivial>true</ignoretrivial> <lockanalysis>true</lockanalysis> <objectallocations> <allocateclass>*</allocateclass> </objectallocations> <objectallocationtimes>true</objectallocationtimes> </selectclass> Only classes found inside the allocateclass tag are monitored as potential leaks. An asterisk in the tag will monitor all classes for leaks, but a better strategy is to use the top classes from step 2 (Figure 5-8 on page 108) for further analysis. If step 2 does not offer any good candidates, using the asterisk to turn it on allows for all objects to be monitored, but this has an adverse impact on performance. Setting the objectallocationtimes tag to true enables the Growth % and Growth Rate columns, which can help show the rate of growth in the objects. Important: The Diagnosis report is functional only in L3 mode. It can slow down a production system severely and should be used only in controlled test situations. Chapter 5. Analyzing WebSphere application problems 109

124 Using the Memory Leak Diagnosis report The memory leak diagnosis report is step 3 of the Memory Leak report. You select the group and server to get to the Report s table. The report result is a very wide table. Figure 5-9 shows the overall result. Figure 5-9 Memory leak diagnosis report 110 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

125 By default, the Growth Rate column is sorted so that the fastest-growing classes are shown first. The important columns are those that indicate the allocating class of the leaking object. These direct the programmer where to look and suggest why the classes might be leaking. Note: If objectallocationtimes is not set, there will be no Growth Rate data. Thus, the default sort will not be useful and another sort is needed, probably on Number of Objects Surviving Last GC column Resolution The Memory Leak Diagnosis report indicates the candidates that are growing over time and, often more important, the allocating classes for these leaks including the line number where the class is allocated. These columns are Allocating Class, Allocating Method, and Line Number. Using this detailed information about the source of the leak, the application developer can examine the source code to see why the class is being allocated and what is causing the class to leak About this scenario The memory leak scenario was created by having every access to a servlet add a String to a static Collection object. Hence, every refresh of the page would cause the leak to grow. The size of the String was set in a properties file so that different memory growth rates could be created. The URL is: The memory growth is specified in the piggybank.properties file, shown in Example 5-2. Example 5-2 The piggybank.properties file ### The below values are delays to be added in millisconds. Or if it is an ### exception than any postive value will do. To turn off ### comment out the line using a '#' # Amount of objects to leak display_account_servlet_leak=100 We used Rational Robot to generate the load to load the page once every 15 seconds. For more information about setting up the environment, see Appendix A, Sample scenario setup on page 177. Chapter 5. Analyzing WebSphere application problems 111

126 5.3 Deadlock management In this section we describe a scenario where Oscar, a systems administrator with InsuNet insurance company, receives complaints, plans the steps to resolve the problem using WebSphere Studio Application Monitor, and further escalates the problem until resolved Planning the problem resolution Oscar has just received a complaint from one of the myinsunetquote application users that a Web request has been hanging for a long time. After checking overall response time on the Enterprise overview page, from AVAILABILITY SYSTEM OVERVIEW ENTERPRISE, Oscar sees no bulk increase in response time, but rather occasional response time peaks. He decides to perform the following problem resolution steps: 1. Carry out performance analysis. 2. Identify the worst performers. 3. Report the problem to the component provider. 4. Prepare for lock analysis. 5. Perform lock analysis Performance analysis The steps in Oscar s performance analysis are: 1. Oscar produces a Trend report for current response time by selecting PERFORMANCE ANALYSIS CREATE REPORTS APPLICATION REPORTS REQUEST/TRANSACTION. 2. On the Report filtering options page he selects Response time for Metric and Request/Transaction type ALL, and clicks Next. 112 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

127 3. On the Date Range Settings page, he selects Previous 24 hour for the Preset and clicks View Report. Figure 5-10 shows the resulting report. Figure 5-10 Response time Trend report 4. The hanging application might be causing the response time peak, so Oscar clicks the highest graph column to drill down to the Decomposition report. Chapter 5. Analyzing WebSphere application problems 113

128 5. By sorting the report by Response time Oscar can identify that in the selected time period the worst response time produced a component called InsuNetWeb/quoteAction (Figure 5-11). Figure 5-11 Response time decomposition report 114 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

129 6. The Application/Transaction name presents a link to Request Report detail page so Oscar selects the InsuNetWeb/quoteAction with the highest response time (Figure 5-12). Figure 5-12 Request report detail for selected Application/Transaction In Figure 5-12, Oscar observes that the Locks tab on this report is disabled, probably because the monitoring level for this server is set too low for Lock Analysis. Note: A disabled tab does not necessarily mean that a particular function is not enabled. It can become disabled because showing data in a particular context is not applicable or data has not been collected at all. After inspecting the Request report detail page Oscar is sure that the quoteaction object (probably a page flow action object) is the worst performer, and that other objects in the same report have no performance problems at all; even quoteactions mostly behave correctly. Chapter 5. Analyzing WebSphere application problems 115

130 7. Oscar sorts the report by response time to get the worst quoteaction request, then selects the first request s link to drill down. This brings him to the Trace report shown in Figure Figure 5-13 Trace report for selected request When examining the report in Figure 5-13, Oscar immediately observes many references to Lock acquisitions and lock contentions, which might directly affect the response time. He decides to: Increase monitoring level for one of the servers. Enable lock analysis for appropriate data collector. Contact Telly, InsuNet Quote application provider s developer. Perform Lock analysis. Produce historic lock information report. 116 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

131 5.3.3 Lock Analysis preparation In order to perform Lock Analysis, Oscar has to increase the monitoring level. After resolving the problem he has to remember to set the level back to L1. 1. He finds the server by accessing ADMINISTRATION MONITORING ON DEMAND, and selects the link under Schedule change/override (Figure 5-14). Figure 5-14 Selecting server for monitoring level change 2. On the server settings modification page, (L2) Problem Determination Mode has to be selected as shown in Figure Figure 5-15 Changing monitoring level to L2 It may take a few seconds to refresh the monitoring level from L1 to L2. Browser refresh might be required so it is good practice to verify it before proceeding. 3. To enable lock analysis on the data collector, its property file cyn.user.classes.xml in the $DC_HOME/etc/ directory has to be modified. The lockanalysis tag indicates whether the class should be instrumented for the Lock Analysis feature (Example 5-3 on page 118). Chapter 5. Analyzing WebSphere application problems 117

132 Example 5-3 Enabling lock analysis <defineinstrumentation> <enablesignature>true</enablesignature> <userclasses> <selectclass> <mask>*</mask> <methods> <methodname>*</methodname> </methods> <ignoretrivial>true</ignoretrivial> <lockanalysis>true</lockanalysis> <objectallocations> <allocateclass>*</allocateclass> </objectallocations> <objectallocationtimes>true</objectallocationtimes> </selectclass> Data collector retrieves the name of the instrumentation file in bcm.properties in the etc directory (Example 5-4). If you are missing one of these two steps, you will get an error message: CYNVE0851E Lock analysis data is not being collected. Example 5-4 Sample bcm.properties userbcm.xmlfilename=cyn.user.classes.xml system.classloader.classnames=sun.misc.launcher*;com.ibm.ws.bootstrap.extclassl oader wls70=wls70_servlet.xml wls70_ejb.xml wls70_jndi.xml wls70_jdbc.xml wls70_jms.xml wls81=wls81_servlet.xml wls81_ejb.xml wls81_jndi.xml wls81_jdbc.xml wls81_jms.xml wls8130=wls81_servlet.xml wls8130_ejb.xml wls81_jndi.xml wls81_jdbc.xml wls81_jms.xml was51=was51_ejb.xml was51_jndi.xml was51_jdbc.xml was51_jmsq.xml was51_jmst.xml was51_jmsmqi.xml was51_jca.xml was51_request_ctg.xml was511=was51_ejb.xml was51_jndi.xml was51_jdbc.xml was51_jmsq.xml was51_jmst.xml was51_jmsmqi.xml was51_jca.xml was51_request_ctg.xml request_imsconnect.xml was502=was502_ejb.xml was502_jndi.xml was502_jdbc.xml was502_jmsq.xml was502_jmst.xml was502_jmsmqi.xml was502_jca.xml was502_request_ctg.xml was6=was6_servlet.xml was6_ejb.xml was6_jndi.xml was6_jdbc.xml was6_jmsq.xml was6_jmst.xml was6_jmsmqi.xml was6_jca.xml was6_request_ctg.xml 118 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

133 wps502=wps502_page.xml wps502_portlet.xml wps502_login.xml wps502_authorization.xml wps502_credentialvault.xml wps502_contentaccess.xml wps502_gateway.xml wps502_model_page.xml wbi=wbi_jdbc_requests.xml wbi_jdbc_sql.xml wbi_jdbc_pool.xml wbi_mqi.xml wbi_serialize.xml Note: The bcm.properties for the server-specific instance is generated in the $DC_HOME/etc directory Lock analysis Oscar can now ask Telly to join him in search for the malfunctioning component. 1. The next step is starting the actual Lock analysis using PROBLEM DETERMINATION SERVER ACTIVITY DISPLAY (Figure 5-16). Figure 5-16 Server activity display 2. On the Server Activity Display page, Telly can verify that: InsuNetWeb application is still hanging at the quoteaction component, resident and idle times show hours, and the thread status is waiting, so the thread has probably entered deadlock. Concrete Java class that probably caused the deadlock was insunet.web.actions.quoteaction. The same information can be confirmed on the Lock Contentions page, which shows many locks coming from the same component. Chapter 5. Analyzing WebSphere application problems 119

134 3. Telly selects Cancel Requests, causing the Lock contention to check the request detail page. Telly makes a note about available details on this page, then clicks OK to cancel the request. Figure 5-17 Cancel request for Lock contention 120 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

135 4. On the left-side navigator, Telly selects link Method/component trace to get a detailed Flow view(figure 5-18). On this page she can eventually identify what method in what class caused the Lock contention (insunet.web.action.quotetransaction class, synchronize update method). Figure 5-18 Flow view Chapter 5. Analyzing WebSphere application problems 121

136 5. Telly can further search through the method trace by clicking the Search tab and selecting a search type; for example, for methods that consume more than a given thread lock acquisition time (Figure 5-19). Figure 5-19 Search method trace The lock analysis has provided enough information for Telly to fix the malfunctioning code Historic lock information Oscar would like to know how long the previous lock has been occurring so maybe he could close some performance-related complaint records. 1. To create a Lock analysis report, he selects PERFORMANCE ANALYSIS CREATE REPORTS APPLICATION REPORTS LOCK ANALYSIS. 2. He selects No on Recurrence, selects the servers, uses report filtering options by Number of Lock contentions. 3. Oscar tries several Date range settings, and from the resulting graphs he can see that lock contention errors have been happening only for the past week (Figure 5-20 on page 123), which roughly corresponds to the installation of a new version of the InsuNetQuote application. Note: WebSphere Studio Application Monitor Lock analysis is available for J2EE transactions, but it is not available for CICS and IMS transactions. Lock analysis report data is historical. 122 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

137 Figure 5-20 Lock contention Trend report 4. By drilling down to the lock contention graph column, Oscar gets a Decomposition report similar to the pie chart in Figure 5-11 on page He drills down again for a Detail report (Figure 5-21). After inspecting the report, Oscar can correlate previous complaint records by comparing the time stamps and response times. Figure 5-21 Lock analysis Trend detail report Chapter 5. Analyzing WebSphere application problems 123

138 6. To check whether any other components were causing dead locks, Oscar selects the Locks tab and verifies that the previously identified QuoteTransaction class was the only one (Figure 5-22). Figure 5-22 Lock analysis Trend detail Locks tab About this scenario The Lock analysis scenario was produced using a QuoteServlet servlet in the Distributed2 Web application module. The QuoteServlet calls a remote stateless EJB (QuoteMediator) with synchronized methods and synchronized Java blocks. For more information about setting up the environment, see Appendix A, Sample scenario setup on page Analyzing a slow-performing application In this section we describe a scenario in which our systems administrator Oscar receives complaints about a slow InsuNet application, plans the resolution steps using WebSphere Studio Application Monitor, and further escalates the problem until it is resolved Planning problem resolution for a slow application Oscar has been asked by InsuNet Customer Service to improve the response time of the myinsunet application, particularly the Insurance Quotation module. myinsunet users complain that the Web quotes take too long to calculate, and the impatient ones just leave the quotes unfinished. Oscar understands that some part of the Quotes module is significantly slower than the others, so he decides to perform the following problem resolution steps: 1. Carry out performance analysis. 2. Identify the worst performers. 124 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

139 3. Report the problem to component provider and/or component owner. 4. Identify affected application modules and servers. 5. Compare system resources Performance analysis for a slow-performing application Oscar remembers that the Quotes module has been deployed in the Bandung server group. 1. He checks system overview by selecting AVAILABILITY SYSTEMS OVERVIEW SERVER. 2. In the group selection, he chooses the Santiago server, then the Lima server. The Santiago overview does not indicate any problem, but on the Lima server overview the average response time is 8 seconds (Figure 5-23). Tip: To visually compare two servers, it is probably better to use two browser windows, because it takes time to refresh all the performance sections for a selected server. Changing view to another server takes even more time. Figure 5-23 Lima server overview: Response time too high 3. After reviewing all the sections, he notices that although Response time and Throughput sections are all peaks, other statistics such as JVM CPU Chapter 5. Analyzing WebSphere application problems 125

140 Utilization and JVM Memory Utilization hover barely above minimum load. The first guess Oscar makes is that there are many requests waiting for some sort of network resources. To check these resources, Oscar selects AVAILABILITY SYSTEM RESOURCES. He goes directly to the Lima server System resource overview page, which shows very similar utilization statistics: underused CPU and memory, and hardly any data source or threading activity (Figure 5-24). Tip: When you move between pages, WebSphere Studio Application Monitor keeps the currently selected application server context for you so you do not have to select the group and server over and over again. It is faster to analyze a given server from several different views before changing to another server. Figure 5-24 Lima server system resource overview One thing looks suspicious on that page: the JNDI section, which shows 84 lookups per minute with average response time of 8 seconds. That statistic 126 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

141 means that most of the response time is spent waiting for a response from another server. Figure 5-25 shows the detail. Figure 5-25 Servlet coverage and JNDI sections 4. Having indicated JNDI as the bottleneck service on Lima, Oscar needs to find out what components use this service so extensively. Above the JNDI section in the Servlet/JSP section, he can hover his mouse over the bar graph to reveal the components involved statistics. The heaviest component, indicated in blue, is QuoteServlet2. Oscar can select the SERVLET/JSP COVERAGE link in Figure 5-25 to see the Web application summary page (Figure 5-26). Figure 5-26 Web application summary Chapter 5. Analyzing WebSphere application problems 127

142 5. In the Web application summary, Oscar can check all Web modules running on the Lima server and, most important, the worst five servlets and JSPs. The worst performer on Lima is a servlet called QuoteServlet2. On average, 7 seconds are being spent right in that class. Oscar can select the QuoteServlet2 link for the servlet summary (Figure 5-27). He makes a quick note of this page, where he identifies the slow application name as Distributed2.App (the RESOURCE section) on server Lima (APPLICATION SERVER NAME). The errant component that might be causing all the trouble is a servlet called QuoteServlet2 (Servlet Name column). This step concludes the Performance analysis phase. Figure 5-27 Servlet summary Identification of affected modules and servers At this point Oscar knows which component of what application on what server has been causing the Quote pages to be slow. Also, he knows that the root of the problem is associated with the JNDI service being called from the errant component. He asks Elmo, the myinsunet developer, to join him for deeper investigation of the problem. Elmo explains that myinsunet is a distributed application consisting of a Web module deployed on Application Server 1, Lima, and an EJB module deployed on Application Server 2, Santiago (Figure 5-28). Application Server 1 Application Server Web module JNDI JNDI EJB Module Figure 5-28 myinsunet distributed applications 128 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

143 The QuoteServlet2 servlet, which is deployed on Lima, communicates with EJB, deployed in Santiago using RMI-IIOP protocol, after looking up the bean in JNDI - Java Naming and Directory Interface. Thus JNDI lookup is the last activity WebSphere Studio Application Monitor has identified on Lima server. Oscar and Elmo know how to re-create the problem. They ask the Quote Web module for a complex quote and try to search for the current request by following PROBLEM DETERMINATION SERVER ACTIVITY DISPLAY to the server Lima (Figure 5-29). Figure 5-29 Lima Server Activity display On the Lima Server Activity display page Elmo can verify that: JVM CPU and memory are underutilized yet the average response time is very high. The affected method is QuoteServlet2.quoteTransaction, which is waiting for a response from QuoteMediator_Stub, a generated client interface for remote EJB QuoteMediator. The last known action was a JNDI request; obviously the problem is being caused on the other server. To find out the location of the remote EJB being called, Elmo selects the QuoteServlet2 link in Client Requests column, which opens the Request Details page. Oscar and Elmo cannot see more information on the detail page but there are links to more information in the left hand navigation panel. Chapter 5. Analyzing WebSphere application problems 129

144 They select Method/Component Trace, which produces the report in Figure On this page they can find out: Web module caller EJB module receiver EJB module network name Figure 5-30 Method/Component Trace for QuoteServlet2 The Complete Flow View page helps Oscar and Elmo understand that: QuoteServlet2 performs a lookup of remote QuoteMediator EJB in JNDI. QuoteMediator is being searched on the address that points to the Santiago server: cell/persistent/cell/nodes/santiago/servers/santiago1. QuoteServlet2 calls remote method QuoteMediator.quoteTransaction. Oscar is happy about finding out that the remote EJB sits on the Santiago server, because Santiago is included in a WebSphere Studio Application Monitor monitoring group called Bandung. They can return to Server Activity Display and select Santiago from the Server selection box. Because the request is still running, they can immediately see the quotetransaction request being processed on Santiago (Figure 5-31 on page 131). 130 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

145 Figure 5-31 Santiago Server Activity display By further investigating the problem they can identify the class and method that are causing the poor performance on the Santiago side of the system: It is QuoteMediatorBean.quoteTransaction. They can do the same as they did on the Lima server to find out more about the method stack. Elmo would like to trace how long this problem has been happening, so they produce reports. They select PERFORMANCE ANALYSIS CREATE REPORTS APPLICATION REPORTS METHOD/PROGRAM, choose No for Recurrence, and select Santiago from Server selection. For the report filtering options, they designate the following options before clicking Next: Metric:Response time (ms). Method/Program: Elmo types in quotetransaction (errant method on QuoteMediatorBean). Request/Transaction Type: EJB (QuoteMediatorBean is a stateless session EJB). For date range settings: Date range: The preset Month to date (because Elmo remembers some applications have been installed the current month). Graphing option: On the X-axis, Time series in day. (Elmo wants to track the quotetransaction method s response time for the past month.) Clicking View Report produces the report shown in Figure 5-32 on page 132. Chapter 5. Analyzing WebSphere application problems 131

146 Figure 5-32 Trend report: Average response time Looking at the Trend report, Elmo wonders why the response time of QuoteMediatorBean.quoteTransaction changed so abruptly in the middle of the current month. He has not installed a new version of the Quotes module for many months, but Oscar remembers that other applications have been installed on Santiago. They decide to compare system resources on Lima and Santiago. Note: To see this level of detail in Trend reports, you must set the appropriate Sampling Rate on the server settings page (ADMINISTRATION MONITORING ON DEMAND). Select Schedule change/override for the selected server and modify the Sampling Rate as shown in Figure Figure 5-33 Modify server settings Comparing system resources Oscar and Elmo have identified EJB QuoteMediatorBean on the Santiago server as the erring component, but then they realized that the same application module has been installed on the Perth server in a different WebSphere Studio Application Monitor group, without any performance problems observed. They decide to compare the two servers, first from the installed applications perspective. 132 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

147 They select the menus for PROBLEM DETERMINATION SOFTWARE CONSISTENCY CHECK INSTALLED BINARY COMPARISON. Comparing the Santiago and Perth servers, they see as in Figure 5-34 that the two servers have the same version of the myinsunet application installed (InsuNet.ear). Figure 5-34 Installed binary comparison Because there is no difference in the installed application instances, there might be a difference in the environment. They go to AVAILABILITY SYSTEM RESOURCE COMPARISON and select the resource Database Connection Pools as shown in Figure Figure 5-35 System resource comparison: database connection pools By comparing the same jdbc/quotesdatasource on Santiago and Perth, it is obvious that the one on Santiago has a smaller connection pool and is being overused. Probably another application on Santiago has been tuned up to work with a smaller data source. They must use the WebSphere Application Server Administration Console to modify it. The menu for this is under Resources Chapter 5. Analyzing WebSphere application problems 133

148 JDBC Providers; select JDBC provider; click Data sources; select the data source, and click Connection pool. This produces the dialog shown in Figure Note: Simply setting the Max connection parameter to a higher number does not solve the problem, and it might even worsen the situation. A simplified rule for this parameter is to set one database connection per Web container thread. Figure 5-36 Data source connection pool properties About the scenario This slow-application scenario was produced using a Web module (servlet) deployed on the Lima server, looking up the remote stateless session bean on the Santiago server. JNDI lookup in the servlet is performed using corbaloc:iiop:...santiago..., as outlined on Figure 5-28 on page 128. More information about the setup for this scenario is provided in Appendix A, Sample scenario setup on page Database performance In this section we describe a scenario where the systems administrator, Oscar, receives complaints about database performance on a quotes Web page, plans the steps to resolve it using WebSphere Studio Application Monitor, and further escalates the problem until resolved. 134 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

149 5.5.1 Planning the problem resolution Oscar has been receiving complaints from InsuNet Customer Service that Web customers leave the Web quotes flow unfinished because the last Web page takes too long to return from submit. On the Enterprise overview page, Oscar compares the load on the two groups of application servers being monitored, bandunggroup1 and santiagogroup1, shown in Figure Figure 5-37 Enterprise overview The throughput in both groups shows approximately the same increase, but response time increases rapidly in bandunggroup1. One of the servers in that group must have response-time problems. After identifying the problem in the bandung group, Oscar can drill down to that group by selecting the group name (bandunggroup1), which is the same as selecting the Group tab. Now he has to go server by server and compare the different statistics provided for each server, or use the System resource comparison feature of WebSphere Studio Application Monitor. The first server attribute to check is the Server statistics section, which lists application names. He has to verify that the malfunctioning application InsuNet is in that group. The application s impact on JVM CPU utilization and on JVM memory utilization is very low on both servers in the group, Lima and Santiago. The graph indicates that the slow response is not related to CPU or memory problems (Figure 5-38 on page 136). Chapter 5. Analyzing WebSphere application problems 135

150 Figure 5-38 Server statistics on Server overview, Server tab By comparing the servers, Oscar can see that QuotesDataSource is behaving differently between them. (He is using AVAILABILITY SYSTEM RESOURCE COMPARISON and comparing the database connection pool.) A pool called jdbc/quotesdatasource configured on Lima is being utilized 100%, but the same pool on Santiago is used only 40% (Figure 5-39). Figure 5-39 System resource comparison: JDBC 136 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

151 At this point Oscar is sure that the InsuNet application is having problems in jdbc/quotesdatasource on Lima. He needs to plan the steps to further identify the cause of this possibly database-related problem: 1. Create a new schedule for monitoring the Lima server. Using ADMINISTRATION MONITORING ON DEMAND, create a schedule to run monitoring level L2 on times when the load is highest, and click Save. Apply the new schedule to Lima on the Monitoring On Demand console page. As a result, Oscar should be able to collect more performance data about Lima to identify the malfunctioning component. 2. Analyze the long-term behavior of the affected data source. Check the system activity and system resources during the scheduled period on Lima, then create a performance analysis report. 3. Analyze short-term behavior of the affected application. Create a trap & alert function, then try to find the malfunctioning component on detail level using in-flight search. These tools could help Oscar provide a detailed problem description for the component providers. 4. Find out the differences between the servers. Why is the same InsuNet application behaving differently on Lima and Santiago? A software consistency check could be used to identify the differences. Oscar knows that the high peak for InsuNet is between 5 p.m. and 8 p.m. every day, so he decides to postpone the in-flight search until 6 pm, and starts step 2, data source analysis Analyzing the affected data source From the systems overview Oscar knows that the myinsunet application response problem is related to the database connection pool jdbc/quotesdatasource on Lima, so the first page he looks at to verify this is the Systems Resource Overview (AVAILABILITY SYSTEM RESOURCES). Selecting a server triggers data collection from the actual application server, which can take 5 to 10 seconds. The database connection pools still display the same 100% utilization, so the next source to check is the DB Connection pool. Figure 5-40 on page 138 shows this analysis. Chapter 5. Analyzing WebSphere application problems 137

152 Figure 5-40 Database connection pools summary General rules for a database connection pool are: Average waiting threads should be close to 0. If not, then there might a problem with the application itself, or connection pool size has reached its limit. In this case, HTTP requests probably have to wait because of the slow data source. Percent used must not be 100%. This complete utilization might indicate that the connection pool is configured incorrectly. Average waiting times correspond to average waiting threads. Connection pool faults indicate that there might be unfinished or failed database transactions. Oscar s problem investigation so far has been only a short-term analysis to capture a snapshot of the system behavior. To pass it on to a component producer, he must create an appropriate Trend report. Since he knows he is dealing with a database call related problem, he can create the report by selecting PERFORMANCE ANALYSIS CREATE REPORT APPLICATION REPORTS SQL, choosing the appropriate server group and server, then on Report filtering options designating SQL call as ALL. Figure 5-41 on page 139 shows a Trend report created using filtering options for Throughput per second and SQL call as ALL for all database tables. 138 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

153 Figure 5-41 SQL Trend report: Throughput seconds vs. Hour of the Day The throughput trend report makes Oscar think about redistributing free resources in bandunggroup, where InsuNet consumes almost nothing in the morning. The graph columns are links that enable drill-down to the SQL statements pie graph (Figure 5-42). Figure 5-42 Decomposition report filtered by SQL calls Chapter 5. Analyzing WebSphere application problems 139

154 The SQL statement with the highest throughput, count, and percentage is a good candidate for the final step: identifying the Java class and method name causing the problem. The SQL CALL statement UPDATE is a link that produces the final page, the SQL Report detail in Figure Figure 5-43 SQL Report detail Note: It is a good idea to sort this kind of long report by Response time. The top lines should identify the top problematic classes and methods candidates. Oscar can confirm his findings by scheduling the same report for the next day while selecting other monitoring options relevant to data source or SQL problems. Before sending the report to the component provider, the myinsunet application vendor, Oscar should try to validate it by re-creating the use case in question using the Insurance Quote Web pages Re-creating the problem After reporting the problem to the component provider, Oscar wants to be ready to prove that it was their fault when they contact him. A good way to prepare is to use the use case identified by the reports (InsuNet Quote Web pages) and start the in-flight search simultaneously (Figure 5-44 on page 141). Oscar has to do the test during the highest peak in order to simulate a real-time scenario. Using PROBLEM DETERMINATION IN-FLIGHT REQUEST SEARCH for the server indicated above should immediately display the request transaction in trouble. If not, Oscar probably has to try the test over and over to generate enough load. 140 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

155 Figure 5-44 In-flight request search The In-flight request search page does not point to a detail Java class yet, but it serves as confirmation that the right task is being analyzed. The Client Request/Transaction column points to the Web container and URL being invoked. Detail information is found under the Thread/Task ID link (Figure 5-45). Figure 5-45 In-flight search - Request detail Now the rogue Java class is revealed: insunet.ejb.quotemediatorbean EJB called from quoteaction.do Web URL. The last known method is getdatasource so whatever method called that method is causing the bean to enter Waiting Chapter 5. Analyzing WebSphere application problems 141

156 status. The Method/component trace in the left-side navigator panel offers even more detail (Figure 5-46). Figure 5-46 In-flight request search complete flow view On the Complete Flow View, all components in the current transaction are displayed, making it easier to identify the objects that are involved and their impact on request time and CPU time consumed. After demonstrating the reports and in-flight search results to the InsuNet application vendor, Oscar may help the developers to further debug the malfunctioning component using Traps: PROBLEM DETERMINATION TRAP & ALERT MANAGEMENT. This activity must be planned carefully because traps may add to system overhead. (See 4.5, Notification on page 83.) Application post-deployment activities Oscar has successfully identified the problem and provided all of the reports to the InsuNet application vendor. But why does InsuNet malfunction only on the Lima server? While scanning the Systems overview for all servers in the WebSphere Studio Application Monitor bandung group, Oscar also found InsuNet on the Santiago server, but there InsuNet was working without problems. Either the application developer used different versions of InsuNet, or maybe configured the application environment incorrectly. Oscar decides to compare InsuNet installations on Lima with Santiago. This activity should have been performed early after the application deployment. 142 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

157 To compare two applications on two servers, select PROBLEM DETERMINATION SOFTWARE CONSISTENCY CHECK INSTALLED BINARY COMPARISON. On the INSTALLED BINARY COMPARISON page select the two servers to compare (Figure 5-47), then select Next page. Figure 5-47 Select the servers to compare On the File selection page select the application InsuNet.ear and required file types (Figure 5-48), then select OK. Figure 5-48 Select sources to compare Chapter 5. Analyzing WebSphere application problems 143

158 From the resulting page it is obvious that there must be different versions of InsuNet installed on Lima and Santiago, because Full Match is 0 (Figure 5-49). Figure 5-49 Installed binary comparison result By further navigating the tree on the left side Oscar could identify that the two applications match only by name. The Unmatched Comparison only result page revealed that one of the.jar files is different. Oscar can conclude that there are different versions of myinsunet application running on the two servers. The version of QuoteMediatorBean deployed on Lima does not use DataSource properly About the scenario This database performance scenario was produced using a stateless session EJB that contains a very poor code: It never closes database connections, never returns them to the database connection pool, and does not use preparedstatement properly. More information about the setup for this scenario is provided in Appendix A, Sample scenario setup on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

159 6 Chapter 6. Analyzing composite application problems This chapter provides some sample scenarios for analyzing application problems when the transaction spans multiple platforms: 6.1, Problem scenarios on page , CICS composite transaction on page , IMS composite transaction on page , CICS resource enqueue on page , CICS transaction loops on page , CICS DSA growth on page 172 Copyright IBM Corp All rights reserved. 145

160 6.1 Problem scenarios This chapter discusses a set of composite applications with requests coming from a WebSphere J2EE application and invoking a CICS or IMS transaction. These types of transactions typically are harder to analyze as they span different platforms. The overall analysis must be performed by J2EE application experts and mainframe systems programmers well-versed in CICS or IMS. As they may not understand each other, the analysis involves getting these skills together. WebSphere Studio Application Monitor enables you to collect information from all of these components and present the result in a composite application view. This helps to speed up the recognition and analysis of composite application problems. This chapter explores some scenarios of composite application problems and presents possible solutions for them using WebSphere Studio Application Monitor. Chapter 2, WebSphere Studio Application Monitor V3.2 overview on page 7 discussed the mechanism and function of the composite application. This chapter should serve as additional debugging from the regular monitoring action laid out in Chapter 4, \Managing production environment on page 67, which should be performed regularly. First, we demonstrate the information that we can acquire for a composite application with CICS and IMS. 6.2 CICS composite transaction This chapter shows a CICS composite transaction behavior in WebSphere Studio Application Monitor. A WebSphere application invoking a CICS transaction through a CICS Transaction Gateway or MQ interface can be considered a composite transaction. This section shows the information that you can acquire for a level 3 trace with a CICS composite transaction. We use CICS Transaction Gateway. The composite transaction information can be seen in the In-flight request, Server Activity Display, and Transaction report under Performance Analysis. The transaction report gives the best view of the transaction, so we use it to illustrate the information in the composite application analysis. We generate the transaction report using the menu PERFORMANCE ANALYSIS CREATE REPORT TRANSACTION/REQUEST REPORT. We report from the CICS composite group and select the time span to run the test. We use time series in hours for the X-axis. 146 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

161 The report produces a bar chart showing each hour that we include in the report. We click a bar to generate a Decomposition report, which classifies the transactions into transaction types. Our Trader CICS transaction runs initially as a servlet, invoking a session EJB, which invokes a CICS transaction. We can view either a servlet or CICS transaction. When we select the transaction type, it shows the individual transactions that we can select. Figure 6-1 shows a summary of the servlet TraderCICSServlet. Figure 6-1 Servlet transaction TraderCICSServlet Chapter 6. Analyzing composite application problems 147

162 The Drilldown View, shown in Figure 6-2, gives the composite information. Figure 6-2 TraderCICSServlet flow view 148 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

163 More details about the transaction can be seen in the Flow View. This view shows all of the method calls during the life of the transaction, so it can be very long. Figure 6-3 shows parts of this flow view for the servlet transaction. Figure 6-3 TraderCICSServlet flow view This Flow View shows lines with INVOKE arrows. This indicates that the transaction is known to go to a back-end system. You can click these links to get to the related CICS transaction. Chapter 6. Analyzing composite application problems 149

164 Now we look at the information in CICS. Figure 6-4 shows the CICS transaction TRADERBL overview. Figure 6-4 TRADERBL overview 150 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

165 Figure 6-5 shows the Flow View of TRADERBL. Figure 6-5 TRADERBL flow view About this scenario This scenario was produced using the standard Trader CICS application with TRADERBL source and TraderCICS.jar. Chapter 6. Analyzing composite application problems 151

166 6.3 IMS composite transaction This section shows IMS composite transaction behavior in WebSphere Studio Application Monitor. IMS composite transactions are supported by WebSphere on z/os only when IMS transactions are accessed using IMS Connect on the MQ interface. In this section, we discuss the information that you can acquire for IMS composite transaction. This section uses IMS Connector for Java. Note: To correctly display IMS information: If your system is not running MSC, the DFSMSCE0 exit skips the transaction. You will also need an updated jar file for ppe.zcdn_impl.jar to correctly calculate the displacement of the Global Publish Server (GPS) token. The System Activity Display report shows IMS transactions being executed. IMS data collector is based on events collected by various IMS exits. These events are shown in the Recent Events display of the IMS transaction (Figure 6-6). Figure 6-6 IMS transaction events in level 3 monitoring 152 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

167 The list in Figure 6-6 on page 152 shows two IMS transactions: the IVTNO, which is executed using Commit Mode 0, and the TRADERBL transaction that is executed using WebSphere Application Server as the synchronization coordinator (Commit Mode 1). The Message Tag column shows an internal transaction number for WebSphere Studio Application Monitor. For each event, you can browse the data collected in the Hexadecimal dump format. The important message is the MESSAGE DATA segment, which represents the application data. Figure 6-7 shows the IVTNO application data. Figure 6-7 IVTNO application data In this figure, you can see the structure and content of the IMS communication area data for the IVTNO application. OTMA Output Edit Before and IMS Program Routing Before both contain the IMS transaction s response. Chapter 6. Analyzing composite application problems 153

168 Figure 6-8 shows another interesting field that is used by WebSphere Studio Application Monitor, the USER DATA segment. The GPS token that is used to identify the composite transaction is shown with line number 00B0 in the form of GTAGxxxx. Figure 6-8 USER DATA segment 154 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

169 We generate the transaction report (PERFORMANCE ANALYSIS CREATE REPORT TRANSACTION/REQUEST REPORT) from the IMS composite group, selecting the time span when we ran the test and using Time series in hours for the X-axis. The report shows a bar chart for each hour that we specified for the report. We click a bar to generate a Decomposition report (Figure 6-9), which classifies the transactions into transaction types. Our Trader IMS transaction runs initially as a servlet, invoking a session EJB, which invokes an IMS transaction. Figure 6-9 IMS Decomposition report Chapter 6. Analyzing composite application problems 155

170 We can view either a servlet or IMS transaction. Selecting the transaction type shows the individual transactions that we can select. Figure 6-10 shows the composite information for TRADERBL in the Flow View. Figure 6-10 TRADERBL flow view In this view, the composite application is indicated by the Event arrows. This indicates that the transaction is known to arrive from a front-end system. You can click the link in the arrow to go to the calling transaction. In the Flow View, you can expand each message and see the hexadecimal dump of the messages similar to the System Activity Display in Figure 6-7 on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

171 Figure 6-11 shows the Flow View for the WebSphere Servlet. Figure 6-11 Servlet composite application Chapter 6. Analyzing composite application problems 157

172 6.3.1 About this scenario This scenario was produced using the standard Trader IMS application with TRADERBL as the source and TraderIMS.jar. 6.4 CICS resource enqueue In this section, we show problem determination for a CICS enqueue problem Symptoms of resource enqueuing The problem is indicated by random transaction timeouts or high response times. This random timeout increases on a high load. WebSphere Studio Application Monitor reports the following load profile from the J2EE application servers. In this report, there are spikes of response time with almost no throughput for both the CICS system and WebSphere Application Server. Figure 6-12 Load profile 158 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

173 The report also shows low CPU usage for these transactions, as shown in Figure Figure 6-13 Transaction report Spikes of response time with low throughput and low CPU utilization are typical indicators of resource locking or a deadlock. Typically the user simply cancels the transaction when this happens. Chapter 6. Analyzing composite application problems 159

174 A CICS system programmer can also see the adverse impact of the transaction from the CICS command CEMT Inquire ENQueue. Figure 6-14 shows sample output from a deadlocked application. I ENQ STATUS: RESULTS Enq Uow(BD09EAC0F16D1D25) Tra(CSMI) Tas( ) Act Exe Own Dur( ) Res(VBD1 ) Rle(004) Enq Uow(BD09EAC24B526F2A) Tra(CSMI) Tas( ) Act Exe Wai Dur( ) Res(VBD1 ) Rle(004) Enq Uow(BD09EAC44ADC874A) Tra(CSMI) Tas( ) Act Exe Wai Dur( ) Res(VBD1 ) Rle(004) Enq Uow(BD09EAC6337CFA22) Tra(CSMI) Tas( ) Act Exe Wai Dur( ) Res(VBD1 ) Rle(004) Enq Uow(BD09EAC6337CFA22) Tra(CSMI) Tas( ) Act Exe Own Dur( ) Res(VBD2 ) Rle(004) Enq Uow(BD09EAC0F16D1D25) Tra(CSMI) Tas( ) Act Exe Wai Dur( ) Res(VBD2 ) Rle(004) SYSID=PA2B APPLID=SCSCPA2B RESPONSE: NORMAL TIME: DATE: PF 1 HELP 3 END 5 VAR 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 6-14 CICS inquire enqueue lock Investigating CICS resource enqueue When the load runs high, the in-flight transaction list reveals some transactions getting an extremely long response time. We first collect reports on which application is actually getting the erratic response time from both WebSphere Application Server and the CICS system. This is achieved from running PERFORMANCE ANALYSIS CREATE REPORT APPLICATION REPORT TOP REPORTS. We selected the reports for our CICS group to show the CICS-related transaction and its WebSphere Application Server, as shown in Figure 6-15 on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

175 Figure 6-15 Report selection for top report This produces the result for the indicative transactions, as shown in Figure Figure 6-16 Transaction list Chapter 6. Analyzing composite application problems 161

176 To collect more detailed information on these transactions, we run PERFORMANCE ANALYSIS CREATE REPORT APPLICATION REPORT REQUEST/TRANSACTION. We report for all servers in our new group and request the response time, as shown in Figure Figure 6-17 Report selection for application report Drilling down to the performance detail, we see the worst-performing transaction (hence the transaction that is in deadlock) as shown in Figure Figure 6-18 Transaction list for slowest response time 162 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

177 As the response time is high in both CICS and WebSphere Application Server, most likely the deadlock is in CICS. Further investigation into the report shows the method that has the high response time in CICS, as shown in Figure Figure 6-19 CICS calls response time Chapter 6. Analyzing composite application problems 163

178 This indicates that there is a potential ENQ (enqueueing) problem with a resource called VBD1VBD2, as other slow performers are typical system requests. Farther into the WebSphere Application Server method calls, the slow performer is shown in Figure Figure 6-20 WebSphere application methods This shows a problem with the buy subroutine and we may need to look at the source code of the program to determine the locking problem. If we do not have source code, we may have to collect additional information by adding the monitoring level of the data collector to collect the stack trace. As most of the locking problem happens only during peak periods, it is quite risky to increase the monitoring level on peak periods. 164 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

179 If you have the monitoring level set to level 3 (L3), you can get to PROBLEM DETERMINATION IN-FLIGHT REQUEST SEARCH and display the individual transaction. Figure 6-21 shows the In-flight request list. Figure 6-21 In-flight transaction list Chapter 6. Analyzing composite application problems 165

180 Figure 6-22 shows the CICS in-flight request. Figure 6-22 CICS transaction trace 166 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

181 6.4.3 About this scenario This scenario was produced using the sample program in member TRADERDL from the ITSO.COBOLSRC data set. The WebSphere environment was running the TraderCICS.ear file. Load was performed using the TraderCICSLocked loader program. The buy and sell program actually has a reverse order for locking the resource VBD1 and VBD2 as shown in the code snippet in Example 6-1. Example 6-1 CICS transaction enqueue BUY-SELL-BUY-FUNCTION SECTION. MOVE 'Entry for SELL' TO COMMENT-FIELD PERFORM WRITEQ-TS * Check whether we have any shares to sell EXEC CICS ENQ RESOURCE(VBD1) LENGTH(4) END-EXEC IF DEC-NO-SHARES IS GREATER THAN 75 THEN EXEC CICS ENQ RESOURCE(VBD2) LENGTH(4) END-EXEC END-IF... BUY-SELL-SELL-FUNCTION SECTION. MOVE 'Entry for SELL' TO COMMENT-FIELD PERFORM WRITEQ-TS * Check whether we have any shares to sell EXEC CICS ENQ RESOURCE(VBD2) LENGTH(4) END-EXEC IF DEC-NO-SHARES IS GREATER THAN 75 THEN EXEC CICS ENQ RESOURCE(VBD1) LENGTH(4) END-EXEC END-IF 6.5 CICS transaction loops Transaction loops in composite applications happen in the following situations: The calling program gets into a loop and calls the back-end transaction multiple times because it cannot decide whether the result is correct. The back-end transaction gets into a loop and eventually times out. A transaction occasionally shows high response time and users are complaining that certain transaction never finished. Chapter 6. Analyzing composite application problems 167

182 6.5.1 Symptoms of looping transaction A CICS transaction loop is usually identified by a significantly large response time and CPU time from a transaction (as opposed to a large response time with significantly small CPU time for a locking problem). The initial anomaly can be detected from the performance overview screen as shown in Figure Figure 6-23 Group overview 168 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

183 To collect information about the specifics of the application, we can request a transaction report by selecting PERFORMANCE ANALYSIS CREATE REPORT APPLICATION REPORTS REQUEST/TRANSACTION. This gives us a better idea about which transaction caused the anomaly and what the characteristic of the anomaly is, as shown in Figure Figure 6-24 Transaction report The invocation of CSMI is far larger that the servlet although they use the same sampling rate. This seems to indicate a loop in the CICS side. Investigating transaction loop This loop can be detected by employing an application trace method that can be provided with an L2 or L3 trace level. The looping transaction has a significant response time, so it can be easily found under the in-flight requests. Chapter 6. Analyzing composite application problems 169

184 Open PROBLEM DETERMINATION IN-FLIGHT REQUEST SEARCH and choose the topmost transaction with a large number of requests, as shown in Figure Figure 6-25 In flight request with a looping CICS transaction 170 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

185 As indicated in the transaction report in Figure 6-24 on page 169, the loop is suspected on the CICS side. This leads us to the stack trace in Figure Figure 6-26 CICS stack trace The stack trace indicates that the transaction CSMI is actually calling the TRADERBL program. As indicated by the method call in Figure 6-26, the loops run READ over and over again against the CUSTFILE. This warrants an investigation into the source file of the program to find when the CUSTFILE is accessed in an infinite loop. Chapter 6. Analyzing composite application problems 171

186 6.5.2 About this scenario This scenario was produced using the sample program in member TRADERLL from the ITSO.COBOLSRC data set. The WebSphere environment was running the TraderCICS.ear file. Load was performed using the TraderCICS loader program. The error was introduced by manually running a Sell transaction for a large amount of share using a Web browser. Going back to the application developer, this maps to the application loops when the number of stock sold is greater than the number of stock owned. The code snippet for this error is shown in Figure 6-2. Example 6-2 Loop error in TRADERBL 05 L PIC ERROR-SELL-NOT-ENOUGH-SHARE SECTION. PERFORM VARYING L FROM 1 BY 1 UNTIL L > 100 PERFORM READ-CUSTFILE END-PERFORM. ERROR-SELL-NOT-ENOUGH-EXIT. EXIT. This error is actually caught at compile time, but only gives a warning with a return code of 4. The warning message is: IGYPG3173-W The result of the comparison of operands L and 100 is known at compile time, based on the attributes of the operands. Unconditional code was generated. 6.6 CICS DSA growth This scenario tracks down DSA growth by a CICS application and analyzes it Monitoring storage growth CICS eventually can indicate that it has problems with storage by issuing a CICS Short on storage (SOS) message on the z/os console. This typically happens when the application allocates storage, either explicitly using EXEC CICS GETMAIN or implicitly within CICS calls. The console message that we see is: +DFHSM0133 SCSCPA2B CICS is under stress (short on storage above 16MB). 172 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

187 You can choose to monitor the system activity display and choose the servers that you are concerned with. Access the monitoring page (Figure 6-27) by selecting AVAILABILITY SERVER STATISTIC OVERVIEW. Figure 6-27 Server statistic overview The JVM/Region column for the CICS data collector indicates the size of DSA acquired, shown in two numbers: one is the total of DSA and the other is the total of EDSA. In this case it is similar to the result from using the CEMT Inqure DSA command from CICS (Example 6-3). Example 6-3 DSA size check in CICS I DSA STATUS: RESULTS - OVERTYPE TO MODIFY Sosstatus(Notsos) Dsalimit( ) Cdsasize( ) Chapter 6. Analyzing composite application problems 173

188 Rdsasize( ) Sdsasize( ) Udsasize( ) Edsalimit( ) Ecdsasize( ) Erdsasize( ) Esdsasize( ) Eudsasize( ) The display in the Server Statistics overview can be changed using the Customize button. This enables configuration of the displayed columns and the threshold to check on each column. Figure 6-28 shows the configuration dialog. Figure 6-28 Configuring statistic display 174 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

189 In this example, whenever the DSA exceeds 75 MB, it notifies you with change of color. Figure 6-29 shows the alerted display for the DSA. Figure 6-29 DSA alert for CICS When you see this alert in the Server Statistic Overview page, you can generate the recent activity display for CICS transactions to see the transactions that have run recently. (Use PROBLEM DETERMINATION SERVER ACTIVITY DISPLAY.) Figure 6-30 on page 176 shows CICS recent activities. Chapter 6. Analyzing composite application problems 175

190 Figure 6-30 CICS recent activities Based on this information, you may need to perform a further check using CICS tools or the OMEGAMON monitor to for each transaction to find who actually acquired memory without releasing it. Another approach is to collect level 3 trace to find GETMAIN macro calls About the scenario The scenario was performed using the standard TRADERBL transaction in CICS with TraderCICS.ear run on WebSphere Application Server using TraderCICSLoader. 176 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

191 A Appendix A. Sample scenario setup This appendix contains information about setting up the testing environment to run the sample scenarios provided in this book. For information about downloading and retrieving the samples, see Appendix C, Additional material on page 191. The topics in this appendix are: Environment overview on page 178 WebSphere applications used on page 178 CICS application setup on page 181 IMS application setup on page 183 Putting in the load on page 185 Copyright IBM Corp All rights reserved. 177

192 Environment overview The applications we use were developed and run on the following platforms: J2EE applications were developed using WebSphere Studio Application Development Integrated Edition (WSADIE) V5.1 and run on WebSphere Application Server V CICS applications were run on CICS Transaction Server V2.3 with CICS Transaction Gateway 5.1. IMS applications were run on IMS V8.1 with IMS Connect V2.2 WebSphere applications used Memory leak sample The applications discussed here are: Memory leak sample Locking application sample on page 179 Distributed Application Sample on page 180 The memory leak scenario is created using a modified piggybank sample program from WebSphere Version 5 Application Development Handbook, SG The additional material contains the piggybank.ear file that can be used to simulate that condition. The load is generated by accessing the page (shown in Figure A-1): Figure A-1 Piggybank servlet with leak 178 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

193 Locking application sample A locking application sample has been used to simulate the Locking application scenario in 5.3, Deadlock management on page 112. The application consists of a Web module that serves the Quote request page flow. The QuoteActionAction class uses QuoteTransaction to generate the lock using a number of calls to synchronized blocks in Java methods. The URL of the locking QuoteServlet is Figure A-2 shows the QuoteServlet page. Figure A-2 QuoteServlet page To install the Distributed Application Sample into WebSphere Studio Application Developer, use the menu File Import, choose the EAR file, and select InsuNetWebApp.ear. Importing this EAR file creates the correct file structure in WebSphere Studio Application Developer. Source code is included. Appendix A. Sample scenario setup 179

194 Distributed Application Sample This Distributed Application Sample is used to simulate the Slow application scenario. The application consists of a Web module deployed in Server1 and an EJB module deployed in Server2 as shown in Figure A-3. Application Server 1 Application Server Web module JNDI JNDI EJB Module Figure A-3 Application structure The Web module consists of a simple servlet and JSP page. Normally JNDI properties such as remote host URL and remote EJB home name are set using name space bindings (in WebSphere console: Environment Naming Namespace Binding) but in our example we set these parameters on the Web. The URL for the QuoteServlet is Figure A-4 on page 181 shows the QuoteServlet2 page. 180 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

195 Figure A-4 QuoteServlet2 page To install the Distributed Application Sample into WSAD, use File Import, choose the EAR file, select Distributed1.App.ear, and import the Distributed2.App.ear file. Importing these two EAR files creates the correct file structure in WebSphere Studio Application Developer. Source code is included. CICS application setup The CICS application is copied from WebSphere Version 5 Connectivity on z/os, SG This section discusses the implementation overview. Trader CICS application Trader CICS comes in the following files: TraderCICS.ear, TraderCICSDelay.ear, and TraderCICSlock.ear: The J2EE enterprise archive files TRADERCICS.xmi: The binary XMIT file for CICS source TraderCICSwksp.zip: Zipped workspaces for TraderCICS in WebSphere Studio Application Developer Integration Edition Appendix A. Sample scenario setup 181

196 Setting up CICS for TraderCICS To set up the CICS application, you need to have a good background in CICS setup and administration. These are the steps for CICS setup: 1. FTP TRADERCICS.xmi to your z/os as a Fixed Block (FB) 80 dataset. For example, the dataset name is TRADCICS.XMIT. 2. Receive into a data set using this TSO command: RECEIVE INDSN(TRADCICS.XMIT) Accept the data set name, which should end with TRADER.CICS. 3. From the TRADER.CICS data set, run the following jobs: a. COJCL: Creates company database. b. COREPRO: Populates company database. c. CUJCL: Creates customer database. d. MAPCOMP: Compiles CICS map for TRAD transaction. e. PROGCOMP: Compiles CICS programs TRADERBL and TRADERPL. 4. Define CICS resources using the CEDA transaction in CICS similar to the content of MYRDO member. Connecting WebSphere for TraderCICS You should have a running CICS Transaction Gateway and CICS system. Perform the following steps to enable WebSphere for TraderCICS: 1. Find the resource archive file cicseci.rar that contains the CICS ECI connector for WebSphere. 2. Open the WebSphere Administration Console. 3. Go to Resources Resource Adapters and click Install RAR. Find the RAR file and install the cicseci.rar with all default options. Save the configuration changes. 4. Return to Resources Resource Adapters, select the new ECI Resource Adapter, and go to J2C Connection Factories. 5. Click New to define a new connection factory. Supply the name and JNDI name for the new connection factory, using something that describes the CICS system you are accessing. Click Apply. 6. Edit the Custom Properties, then supply the values for: ServerName CICS application ID ConnectionURL Either tcp:hostname or local: PortNumber (remote only) CICS Transaction Gateway port 7. Save the configuration. 182 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

197 8. Go to Application Install new application and install the TraderCICS.ear file. Remember to update the JNDI name of the Resource name to the CICS connector connection factory that you specified in step 5 on page Save the configuration and restart the application server. TraderCICS should be ready. IMS application setup The IMS application is copied from WebSphere Version 5 Connectivity on z/os, SG This section discusses the implementation overview. We also use the sample IVTNO transaction from the IMS installation verification. Trader IMS application Trader IMS comes in the following files: TraderIMS.ear: The J2EE enterprise archive file TRADERIMS.xmi: The xmit binary file for IMS source TraderIMSwksp.zip: Zipped workspace for TraderIMS in WebSphere Studio Application Developer Integration Edition Setting up IMS for TraderIMS To perform this setup, you need to understand IMS system programming to be able to adapt the instructions for your environment. Follow these steps: 1. FTP TRADERIMS.xmi to your z/os as a Fixed Block (FB) 80 dataset. For example, the dataset name is TRADRIMS.XMIT. 2. Receive into a dataset using this TSO command: RECEIVE INDSN(TRADRIMS.XMIT) Accept the dataset name, which should end with IMS.TRADER.SOURCE. 3. From the IMS.TRADER.SOURCE, run the following jobs to define the TraderIMS applications: a. DBDLIB: Generates the database descriptions. b. PSBLIB: Generates the program specification block. c. ACBLIB: Uses the DBD and PSB that were generated to create a set of application control blocks. d. If Database Recovery Control (DBRC) is enforced, run the DBRCDEF job to define the databases to DBRC. e. LOADCUST: Defines new VSAM files for the database and loads the initial content of it. Appendix A. Sample scenario setup 183

198 f. DFSMDA: Defines the databases to IMS dynamic allocation. g. COMPILE: Compiles the COBOL in PGMSRC to a load module. 4. Update the IMS stage 1 generation deck with the content of IMS.TRADER.SOURCE(IMSSTG1). The default member for the IVP system can be found in INSTALIB(IV3C201T). 5. Perform IMS system generation. Typically the job is in INSTALIB(IV3C203J). This will generate Stage 2 job in INSTALIB(IV3C301J). You may have to run other generation jobs depending on your system needs. 6. Activate the new ACBLIB to the active ACB library (ACBLIBA or ACBLIBB) using the OLCUTL as in INSTALIB(IV3E316J). 7. Cold start IMS and check whether the databases and program are defined. Connecting WebSphere for TraderIMS You should have a running IMS Connect and IMS system. Perform the following steps to enable WebSphere for TraderIMS: 1. Find the resource archive file imsico.rar that contains the IMS Connect for Java connector for WebSphere. 2. Open the WebSphere Administration Console. 3. Go to Resources Resource Adapters and click Install RAR. Find the RAR file and install the imsico.rar with all default options. Save the configuration changes. 4. Return to Resources Resource Adapters, select the new IMS Connector for Java, and go to J2C Connection Factories. 5. Click New to define a new connection factory. Supply a name and JNDI name, using something that describes the IMS system you are accessing. Click Apply. 6. Edit the Custom Properties and supply the values for: IMSConnectName (local option) ID of the IMS connect in HWSCFG HostName (remote option) IMS Connect TCP/IP host name PortNumber (remote option) IMS Connect port number DataStoreName (both) IMS subsystem name 7. Save the configuration. 8. Go to Application Install new application and install the TraderIMS.ear file. Remember to update the JNDI name of the Resource name to the IMS connector connection factory that you specified in step Save the configuration and restart the application server. TraderIMS should be ready. 184 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

199 Putting in the load There are several tools that you can use to generate the load. We used these: WebSphere Studio Workload Simulator Rational Robot Sample Java program WebSphere Studio Workload Simulator Rational Robot Sample Java program WebSphere Studio Workload Simulator for z/os and OS/390 is a testing tool for creating virtual or simulated users for load and performance testing. WebSphere Workload Simulator can also be used to perform quality assurance on Web applications and to test the functionality of Web servers. For more information, refer to: IBM Rational Robot automates regression, functional, and configuration testing for e-commerce, client/server, and ERP applications. It is used to test applications based on a wide variety of user interface technologies, and is integrated with the IBM Rational TestManager solution to provide desktop management support for all testing activities. For more information, refer to: We also developed a custom Java program to perform simple application testing automation, and you can find directions for downloading it in Appendix C, Additional material on page 191. The program is provided in the loader directory. You can use this to test for the composite application scenarios. The application is executed using the loader.bat command. Appendix A. Sample scenario setup 185

200 186 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

201 B Appendix B. Installation tips This appendix contains a summary of installation tips for WebSphere Studio Application Monitor V3.2. The information in this appendix is provided as is; you should evaluate the actual product documentation and technical references. Copyright IBM Corp All rights reserved. 187

202 Embedded installation problems Table B-1 Installation tips Table B-1 lists some common embedded installation problems. Consider the following items: What is the purpose of an embedded installation? The goal of the DB2 and WebSphere Application Server embedded installations is to enable customers to get the product running quickly without requiring a separate effort of installing prerequisite software. This feature is intended for demos, Proof of Concepts, or for customers who want to get the WebSphere Studio Application Monitor managing server up without worrying about the details of the middleware. The embedded installations of DB2 and WebSphere Application Server make certain assumptions and take certain defaults. If the customer has a database administrator or webmaster who is interested in fully customizing the database or application server, the embedded installation option may not be the best. Where can I obtain the DB2 and WebSphere Application Server images? DB2 and WebSphere Application Server CD-ROM images are available in the WebSphere Studio Application Monitor V3.2 CD-ROM packet. To download images from the Web, contact your IBM sales representative to receive your user ID and password and Web site address for Passport Advantage of the IBM Software Group. Contact IBM Sales for more information. Symptom Circumvention Additional information DB2 version that WebSphere Studio Application Monitor V3.2 embeds caused installer to fail. Multiple processor machines do not work with embedded DB2 installation. JMS does not work with embedded WebSphere installation. Embedded installation from CD-ROM failed. DB2 installation from a downloaded image on Windows 2003 failed. Use DB2 UDB 8.1 FixPack 3 as required from the Release notes. Install DB2 manually. Do not use embedded installation. Install WebSphere manually. Do not use embedded installation. Copy installation images to disk before running embedded installation. Use the distribution CD-ROM. DB2 is installed with the fix pack not installed separately. Other DB2 levels may have a different installer program. Embedded installation is meant to be a specialized version of software installation when the customer does not want to get into the details of prerequisite software installation. This is a permission problem in the Windows 2003 file system. 188 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

203 Composite transaction problems Table B-2 lists some common composite transaction problems. Table B-2 Installation tips Symptom Circumvention Additional information WebSphere JVM restarting or generating javacore when running CICS transaction from WebSphere Application Server in Windows IMS 3270 transactions are not shown in VE for non-msc system. IMS Connect transaction is not correlated as composite transaction. Remove request_ctg.xml from bcm.properties file. Insert an MSC definition in IMS generation. Check GPS token; it should be around offset 180 in User Data Segment and should not be all 0s. CICS composite transaction is not supported from Windows data collector. Appendix B. Installation tips 189

204 190 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

205 C Appendix C. Additional material This book refers to additional material that can be downloaded from the Internet as described below. Locating the Web material The Web material associated with this book is available in softcopy on the Internet from the IBM Redbooks Web server. Point your Web browser to: ftp:// Alternatively, you can go to the IBM Redbooks Web site at: ibm.com/redbooks Select the Additional materials and open the directory that corresponds with the redbook form number, SG Using the Web material The Web material that accompanies this redbook includes this file: File name Description SG zip Zipped additional material Copyright IBM Corp All rights reserved. 191

206 System requirements for downloading the Web material The total size of the Web material is 4.3 MB. Distributed environment: Hard disk space: 15 MB of free space Operating system: Windows, Linux, AIX Memory: 1 GB or more WebSphere: V5.1.1 The z/os environment: Operating system: z/os v1.5 or later WebSphere: V5.1 CICS: CICS Transaction Server 2.3 CTG: CICS Transaction Gateway 5.1 IMS: IMS 8.1 IMS Connect: IMS Connect 2.2 How to use the Web material Create a subdirectory (folder) on your workstation and unzip the contents of the Web material zip file into this folder. For more information about the individual file setup, refer to Appendix A, Sample scenario setup on page WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

207 Abbreviations and acronyms ACB AIX Application Control Block Advanced Interactive Executive ASCII American Standard Code for Information Interchange BCM Byte Code Modification (in WSAM) CICS Customer Information Control System COBOL Common Business Oriented Language CPU Central Processing Unit CSV Comma Separated Values DB2 Database 2 DBD Database Descriptor DBRC Database Recovery Control (in IMS) DSA Dynamic Storage Area EAR Enterprise Archive EBCDIC Extended Binary Coded Decimal Instruction Code ECI External Call Interface (in CICS) EDSA Extended DSA EJB Enterprise JavaBean ERP Enterprise Resource Planning FTP File Transfer Protocol GPS Global Publish Server GUI Graphical User Interface HTTP Hypertext Transfer Protocol IBM International Business Machine Corp IMS Information Management System ITM ITSO IVP J2C J2EE JCA JDBC JMX JNDI JNI JSP JTA JVM JVMPI JVMTI MVS OTMA PMI PSB RAR RFS RMI SAD SLQ SMF SMTP SNMP IBM Tivoli Monitoring International Technical Support Organization Installation Verification Procedure Java 2 Connector Java 2 Enterprise Edition Java Connection Architecture Java Database Connectivity Java Management Extension Java Naming and Directory Interface Java Native Interface Java Server Page Java Transaction API Java Virtual Machine JVM Profiler Interface JVM Tool Interface Multiple Virtual Storage Open Transaction Manager Access Performance Management Interface Program Specification Block Resource Archive Remote File System Remote Method Invocation Server Activity Display (in WSAM) Service Level Objective System Measurement Facility Simple Mail Transfer Protocol Simple Network Management Protocol Copyright IBM Corp All rights reserved. 193

208 SOS SQL SSL TCP/IP TEC TSO UDB UDP UML URL VSAM WSAM XML Short on Storage (in CICS) Structured Query Language Secure Socket Layer Transmission Control Protocol/Internet Protocol Tivoli Enterprise Console Time Sharing Option Universal Database User Datagram Protocol Unified Modeling Language Uniform Resource Locator Virtual Storage Access Method WebSphere Studio Application Monitor Extensible Markup Language 194 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

209 Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook. IBM Redbooks For information on ordering these publications, see How to get IBM Redbooks on page 196. Note that some of the documents referenced here may be available in softcopy only. Installing WebSphere Studio Application Monitor V3.1, SG Monitoring WebSphere Application Performance on z/os, SG Planning for the Installation and Rollout of WebSphere Studio Application Monitor 3.1, SG WebSphere for z/os Connectivity Architectural Choices, SG Other publications These publications are also relevant as further information sources: WebSphere Studio Application Monitor Version 3.2 Installation Guide, SC WebSphere Studio Application Monitor Version 3.2 Release Notes, GI WebSphere Studio Application Monitor Version 3.2 User Guide, SC Online resources These Web sites and URLs are also relevant as further information sources: WebSphere Studio Workload Simulator page Rational Robot page Copyright IBM Corp All rights reserved. 195

210 Microsoft Windows Services for UNIX The itext, a free Java PDF library How to get IBM Redbooks You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications, and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site: ibm.com/redbooks Help from IBM IBM Support and downloads ibm.com/support IBM Global Services ibm.com/services 196 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

211 Index A ACB 184 account management 18 adduser 36 ADMINISTRATION menu 17 alert 69 am.mod 64 am-check.sh 60 amctl.sh 60 am-start.sh 55, 60 am-stop.sh 60 AMUSER 44 Apache Web server 10 apachectl 61 Application Control Block, see ACB archive agents 10 authenticate 36, 61 authentication 35 AVAILABILITY menu 19 B bcm.properties BCMLogMessages.properties C Capacity Analysis report 74 capacity planning 75 CEDA 182 CEMT 160, 173 certificate 35 CICS 10, 14, 27, 178 transaction SAMC 14 CICS transaction CEDA 182 CEMT 160, 173 CSMI 169, 171 CICS Transaction Gateway 178 CICS Transaction Server 178 get 81 commands adduser 36 am-start.sh 55 apachectl 61 authenticate 36 datatrim.sh 52 datatrimmer.sh 52 DB2 48 db2install.sh 57 dcctl.sh 65 klctl.sh 55, 58 run_stat_cmds.sh 51 smitty 36 comma-separated value, see CSV ConnectionURL 182 cross-memory services 12 CSMI 169, 171 CSV 23 CTG_SR_OVERVIEW table 52 CUSTFILE 171 CYANEA 44 CyaneaDCStore 62 CyaneaProxyStore 62 cyn.dev.user.classes.xml cyn.user.classes.xml CYNVE0500E 27 CYNVE0691E 30 CYNVE0745E 26 CYNVE0746E 26 CYNVE0747E 26 CYNVE0850E 27 CYNVE0851E 27 CYNVE0852E 27 CYNVE1082E 28 D Daily Statistics 23 data collector 10 CICS 10, 14, 27 IMS 10, 27 J2EE 10 technology 24 Database Recovery Control, see DBRC datacollector.env datacollector.log 65 datacollector.policy 62, 64 Copyright IBM Corp All rights reserved. 197

212 datacollector.properties DataStoreName 184 datatrim.sh 52 datatrimmer.sh 52 DB2 TERMINATE command 48 DB2 UDB 10 DB2 Universal Database V8 24 DB2_SELECTIVITY 31 db2install.sh 57 db2move.lst 47 DBADM 47 DBRC 183 DC 10 dcctl.sh 65 dctomgmt.cer 62 dctoproxy.cer 62 DFSMSCE0 13 DFSYIOE0 13 E eavesdropping 34 Eclipse InfoCenter 30 F FB 182 Fixed Block, see FB G Garbage Collection Frequency 106 Garbage Collection Time 106 Garbage Collection, see GC GC GC_DATA table 52 global publishing server 10 gpscounter.txt 65 I IBM HTTP Server 10 impersonation 35 important reports 69 IMS 10, 27, 178 IMS Connect 178 IMSConnectName 184 IMSTHREADS table 52 Installation Verification Procedure, see IVP IVP 184 J J2EE 2, 10 performance tuning 78 Java 2 Enterprise Edition, see J2EE Java Connection Architecture, see JCA Java Native Interface, see JNI Java Transaction API, see JTA Java Virtual Machine, see JVM java.awt.headless 61 JCA 20 JDBC_DRIVER_NAME 55 JDBC_DRIVER_URL 55 JDBC_PASSWORD 55 JDBC_USER 55 jndi.properties 63 JNI 13 JTA 20 JVM 13, 20 JVM Heap Size 106 K kernels 10 klctl.sh 55, 58 L L3 165 language support 24 level lightweight collections 79 lock analysis 26 Lock Analysis report 71 lock contention analysis 23 log.properties 62 LogMessages 64 LogMessages.properties M managing server 10 archive agents 10 global publishing server 10 kernels 10 message dispatcher 10 option 18 polling agents 10 publishing servers 10 watchdog 10 memory diagnostics WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

213 memory leak analysis 25 MEMORY_DATA table 52 message dispatcher 10 Message Queueing Interface, MQI Message Queuing Integrator, see MQI Method Program report 71 METHOD table 52 misrepresentation 35 monitoring level 39 level 1 39 level 2 39 level 3 39 Monitoring on Demand 18, 40 monitoring schedule 40 MQI 20, 71 MQI report 71 MQI_QUEUEMGR_SR_OVERVIEW table 52 MS 10 N node authentication 34 non-tme SNMP Adapter 85 notifications 69 O OCTIGATE database 11, 44 OMEGAMON 24 Open Transaction Manager Access, see OTMA Oracle 10 OTMA 13 P PDF 68 performance tuning 78 perspectives 79 PMISTATS table 52 polling agents 10 Portable Document Format, see PDF PORTALOVERVIEW table 52 PORTALSTATS table 52 PortNumber 182 publish server 26 publishing servers 10 R RECEIVE Remote File System, see RFS Remote Method Invocation, see RMI REORG utility 49 REORGCHK utility 49 report Capacity Analysis 74 Lock Analysis 71 Method/Program 71 MQI 71 Request/Transaction 70 Server Availability 74 SQL 71 System Resource 74 Top Reports 71 REQUEST table 52 Request Transaction report 70 RFS 59 RMI 59 RoboHelp WebHelp 30 rotate-apache-logs.sh 61 run_stat_cmds.sh 51 RUNSTATS utility 49 S SAD 26 SAMC 14 Server Activity Display, see SAD Server Availability report 74 server management 18 server statistics overview 20 Server Statistics Overview, see SSO ServerName 182 SERVERSTATS table 52 Services for UNIX, see SFU setenv.sh 61 SFU 15 Short on storage, see SOS Simple Mail Transfer Protocol, see SMTP Simple Network Management Protocol, see SNMP smitty 36 SMTP 36 SNMP 10 Solaris 15 SOS 172 spoofing 35 SQL 20 SQL report 71 SSO 28 Statement.executeBatch() 81 Index 199

214 Structured Query Language, see SQL Sun Solaris 10 System Resource report 74 system resources 20 systems overview 19 T table CTG_SR_OVERVIEW 52 GC_DATA 52 IMSTHREADS 52 MEMORY_DATA 52 METHOD 52 MQI_QUEUEMGR_SR_OVERVIEW 52 PMISTATS 52 PORTALOVERVIEW 52 PORTALSTATS 52 REQUEST 52 SERVERSTATS 52 VOLUMESTAT 52 VOLUMESTATS 52 WEBSERVERCHARTDATA 52 tampering 35 TEC 85 TERMINATE 48 Tivoli Enterprise Console, see TEC Top Reports 71 TRADERBL 171 trap 69 TSO command RECEIVE WebSphere Studio Application Development Integrated Edition, see WSADIE WebSphere Studio Application Monitor overview 8 Windows Services for UNIX 15 WPS 27 WSADIE 178 X X Xvfb 10 X-Windows 10 Z zaap processor 24, 28 V VE 16 ve.properties 31 version.properties 62, 64 Virtual Frame Buffer 10 Visualization Engine, see VE VOLUMESTAT table 52 VOLUMESTATS table 52 W watchdog 10 WebLogic 16 WEBSERVERCHARTDATA table 52 WebSphere Portal Server, see WPS WebSphere Studio Application Developer WebSphere Studio Application Monitor V3.2 Advanced Usage Guide

215 WebSphere Studio Application Monitor V3.2 Advanced Usage Guide (0.2 spine) 0.17 <-> <->249 pages

216

217

218 Back cover WebSphere Studio Application Monitor V3.2 Advanced Usage Guide Overall J2EE application management and monitoring Composite application monitoring with CICS and IMS Distributed and z/os platforms covered The addition of WebSphere Studio Application Monitor to the Tivoli product portfolio enhanced Tivoli offerings for application management. WebSphere Studio Application Monitor enables comprehensive monitoring and management of J2EE-based applications with their back-end platforms. This IBM Redbook demonstrates advanced uses of WebSphere Studio Application Monitor to solve day-to-day operational problems. The book is intended for system administrators who want to use WebSphere Studio Application Monitor for monitoring Web-based applications. We do not discuss implementation issues in this book. We present WebSphere Studio Application Monitor concepts and offer usage details such as: - System administration of day-to-day operations, from grouping and security design to maintenance tasks. - Problem determination for WebSphere-based applications to assist with common problems that may occur in your environment. - Management and analysis of composite applications, which span multiple application servers and can be hard to manage, with demonstrations of composite applications across J2EE servers, CICS, and IMS. INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment. For more information: ibm.com/redbooks SG ISBN