Enterprise PeopleTools 8.50 PeopleBook: PeopleSoft Performance Monitor

Transcription

1 Enterprise PeopleTools 8.50 PeopleBook: PeopleSoft Performance Monitor September 2009

2 Enterprise PeopleTools 8.50 PeopleBook: PeopleSoft Performance Monitor SKU pt850pbr0 Copyright 1988, 2009, Oracle and/or its affiliates. All rights reserved. Trademark Notice Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. License Restrictions Warranty/Consequential Damages Disclaimer This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. Warranty Disclaimer The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. Restricted Rights Notice If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR , Commercial Computer Software License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA Hazardous Applications Notice This software is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy and other measures to ensure the safe use of this software. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerous applications. Third Party Content, Products, and Services Disclaimer This software and documentation may provide access to or information on content, products and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third party content, products and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third party content, products or services.

3 Contents Preface Performance Monitor Preface... xi Performance Monitor... xi Chapter 1 Getting Started with Performance Monitor... 1 Performance Monitor Overview... 1 Performance Monitor Implementation... 1 Installation-Level Steps... 1 Post Installation-Level Steps... 2 Chapter 2 Understanding Performance Monitor... 3 Performance Monitor... 3 Performance Monitor Architecture... 4 The Monitored System... 6 The Monitoring System... 8 Implementation Options Self-Monitoring Monitoring System Production Monitoring System Performance Monitor Instrumentation Understanding PMUs Understanding Events Performance Integration with Third Party Systems Performance Monitor Data Scope of the Performance Monitor Chapter 3 Setting Up the Performance Monitor Configuration of the Performance Monitor Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. iii

4 Contents Enabling the Required Elements on the Monitoring System Setting Up the Performance Monitor Database Setting Up the Application Server Setting Up the Web Server for the Monitoring System Creating a PPMI User ID Specifying the PPMI URL and PPMI User ID Specifying the Integration Gateway URL Setting Up the Process Scheduler Server for the Monitoring System Enabling the Required Elements on the Monitored System Setting Up the Database of the Monitored System Specifying the Monitor URL Setting Up the Web Server of the Monitored System Setting Up the Application Server Setting Up the Process Scheduler Server Chapter 4 Administering the Performance Monitor Performance Monitor Administration Performance Monitor Administration Pages Used for Performance Monitor Administration Setting Global System Options Setting System Definition Defaults Modifying System Definitions Viewing Agent Definitions Setting Agent Filter Options Scheduling the Reaper Program Scheduling Performance Data Archiving Running the Performance Data Archiving Program Working with Aborted Program Runs Populating Performance Monitor Lookup Tables Working with Performance Monitor Tables Disabling Performance Monitor Agents Working with Performance Monitor Web Profile Properties Tracing Performance Monitor Agents Enabling Tracing on the Application Server or Process Scheduler Server Enabling Tracing on the Web Server Tracing the Monitor and PPMI Servlets Configuring Performance Monitoring Sampling Rate Viewing Monitor Servlet Diagnostics Monitoring Agents Monitoring PPMI Clients Setting Up Monitor Clusters iv Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

5 Contents Using Performance Monitor Data Mover Scripts Estimating Your Performance Database Size Estimating Your Performance Database Size Overview Estimating Space Requirements for Event Data Estimating Space Requirements for PMU Data Calculating Space Requirements Chapter 5 Working with the Performance Trace Understanding the Performance Trace Enabling a Performance Trace Running a Performance Trace Viewing Performance Trace Information Chapter 6 Monitoring System Performance System Performance Monitoring Monitoring System Performance Pages Used to Monitor System Performance Using the System Performance Home Page Viewing Web Server Performance Viewing Application Server Domain Performance Viewing Analytic Server Information Viewing Process Scheduler Server Performance Viewing Master Scheduler Performance Viewing Open PMUs Viewing Open PMU Trees Viewing Current User Sessions Viewing Recycled Processes Diagnosis Monitoring Resource Usage Working with Event 150 (JVM Status) Working with Event 151 (Network Status) Working with Event 200 (Resources Per Process) Working with Event 300 (Host Resource Status) Working with Event 301 (Tuxedo "pq Rows) Working with Event 302 (Tuxedo "psr" Rows) Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. v

6 Contents Chapter 7 Analyzing Historical Performance Data Historical Performance Data Understanding Historical Performance Data Pages Used to Analyze Historical Performance Data Viewing Completed PMUs Entering Search Criteria Working with Completed PMUs Completed PMU Status The PMU Details Page Viewing Event History Entering Search Criteria Working with Event History Information Viewing User Session History Entering Search Criteria Working with User History Information Chapter 8 Working with Performance Monitor Charts and Analytics Performance Monitor Charts and Analytics Understanding Performance Monitor Charts and Analytics Common Elements Used in This Chapter Pages Used to View Performance Monitor Charts Viewing User Request Analytics The User Requests Chart The Request Time vs. Think Time Chart User Request Details Viewing Component Trace Information Viewing Component Statistics Viewing Portal Statistics Viewing PIA Statistics Viewing Top Portal Content Requests Viewing Top PeopleCode Events Viewing Top PeopleCode Executions Viewing Top PeopleCode SQL Statements Viewing Top Components vi Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

7 Contents Chapter 9 Performance Monitor Security Considerations Setting Up PSPPMSRV Authentication Firewall Considerations Agent Communication with the Monitor Servlet PSPPMSRV Communication with the Monitor Servlet Monitor Cluster Members Setting Up SSL Setting Up SSL Between Agents and Performance Monitor Setting Up SSL Between Performance Monitor and PSPPMSRV and Monitor Cluster Members Setting Up SSL Client Authentication Protecting the Performance Monitor Servlets Chapter 10 Performance Monitor Meta-Data Performance Monitor Meta-Data Definitions Metric Definitions Event Definitions Context Definitions PMU Definitions Appendix A PMU Definition Reference PMU 100 Portal Request PMU 101 PIA Request PMU 102 Compression Request PMU 106 PIA Request From Portal PMU 108 User Session Ended PMU 109 User Session Began PMU 113 Authenticate PMU 114 Attach File Request PMU 115 JOLT Request PMU 116 Redirect after Login PMU 117 Get Content PMU 400 Tuxedo Service PCode and SQL PMU 401 ICPanel PMU 402 Modal Level Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. vii

8 Contents PMU 403 Modal Level PMU 404 PeopleCode Program Execution PMU 406 PeopleCode SQL Execute PMU 407 PeopleCode BuiltIn SQL Execute PMU 408 PeopleTools SQL Execute PMU 409 Explicit Commit PMU 410 ICScript PMU 411 ICQuery PMU 412 Tuxedo Service Summary PMU 413 GetCertificate PMU 414 SQL Fetch Summary PMU 415 PortalRegistry PMU 416 ICWorklist PMU 417 FTP Request PMU 418 PSBusComp PMU 419 BI GetInterlink PMU 420 BI Execute PMU 421 BI BulkExecute PMU 422 LDAP Bind PMU 423 LDAP Search PMU 424 Call AppEngine PMU 425 Implicit Commit PMU 426 RemoteCall Built-in PMU 427 Deserialization PMU 428 Serialization PMU 439 Workflow PCode Summary PMU 500 SavePreChange PCode Summary PMU 501 SavePostChange PCode Summary PMU 502 SaveEdit PCode Summary PMU 503 SaveEdit PCode Summary PMU 504 RowSelect PCode Summary PMU 505 RowInsert PCode Summary PMU 506 RowDelete PCode Summary PMU 507 FieldChange PCode Summary PMU 508 FieldEdit PCode Summary PMU 509 FieldDefault PCode Summary PMU 510 PrePopup PCode Summary PMU 511 ItemSelected PCode Summary PMU 512 SearchInit PCode Summary PMU 513 SearchSave PCode Summary PMU 514 Active PCode Summary PMU 515 PreBuild PCode Summary PMU 516 PostBuild PCode Summary PMU 700 PPMI Servlet PMU 800 Monitor Servlet Request viii Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

9 Contents PMU 801 Analytic Server Request Filter Level PMU 802 Synchronize/Read Data from Data Cache Summary PMU 803 Synchronize/Write Data to Data Cache Summary PMU 804 Synchronize/Read Data from Data Cache Details PMU 805 Synchronize/Write Data to Data Cache Details PMU 806 Load ACE Model Trees Summary PMU 807 Load ACE Model Trees Details PMU 808 Recalculate ACE Model Summary PMU 809 Recalculate ACE Model Details PMU 810 ACE Model Get Data PMU 811 ACE Model Set Data Appendix B Event Definition Reference Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event 703 PPMI Notify Cancel Interest Event 704 PPMI Notify Agent Change Event 705 PPMI Notify Monitor Change Event Event 802 Monitoring Server Client Lost Event 803 Monitoring Server Data Lost Event 900 Agent Buffer Overrun Event 901 Agent Init Notification Event 902 Agent Configuration Ack Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. ix

10 Contents Event 903 Agent Contact Notification Event 904 Analytic Server Status Event 905 Analytic Server Load Event Event 906 Out of Analytic Servers Event 907 Analytic Server Recycle Event Index x Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

11 Performance Monitor Preface This preface introduces the Performance Monitor. Performance Monitor Performance Monitor supports the monitoring of performance throughout your PeopleSoft systems. Readers of this PeopleBook are expected to be familiar with the following: PeopleSoft architecture. SQL. PeopleCode. Principles of performance tuning. PeopleSoft development tools, such as PeopleSoft Application Designer and PeopleSoft Application Engine. General PeopleSoft system administration. PeopleBooks and the Online PeopleSoft Library A companion PeopleBook called PeopleBooks and the Online PeopleSoft Library contains general information, including: Understanding the PeopleSoft online library and related documentation. How to send PeopleSoft documentation comments and suggestions to Oracle. How to access hosted PeopleBooks, downloadable HTML PeopleBooks, and downloadable PDF PeopleBooks as well as documentation updates. Understanding PeopleBook structure. Typographical conventions and visual cues used in PeopleBooks. ISO country codes and currency codes. PeopleBooks that are common across multiple applications. Common elements used in PeopleBooks. Navigating the PeopleBooks interface and searching the PeopleSoft online library. Displaying and printing screen shots and graphics in PeopleBooks. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. xi

12 Preface How to manage the PeopleSoft online library including full-text searching and configuring a reverse proxy server. Understanding documentation integration and how to integrate customized documentation into the library. Glossary of useful PeopleSoft terms that are used in PeopleBooks. You can find this companion PeopleBook in your PeopleSoft online library. xii Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

13 Chapter 1 Getting Started with Performance Monitor This chapter discusses: Performance Monitor overview. Performance Monitor implementation. Performance Monitor Overview The Performance Monitor enables you to view real-time and historical performance data of your PeopleSoft systems. The Performance Monitor provides the information that you need to solve immediate performance issues and analyze trends in system performance. By default, the Performance Monitor is disabled on a new PeopleSoft installation. To enable the Performance Monitor, follow the configuration instructions in this PeopleBook. When you have a PeopleSoft application environment running and you have set up the Performance Monitor, you can begin to capture and analyze PeopleSoft performance data. Performance Monitor Implementation Before using the Performance Monitor, you need to perform some implementation and configuration tasks, in addition to standard PeopleSoft installation and implementation tasks. See Also Chapter 3, "Setting Up the Performance Monitor," page 15 "Performance Monitor Preface," page xi Enterprise PeopleTools 8.50 PeopleBook: Getting Started with Enterprise PeopleTools Installation-Level Steps Complete the following steps during installation. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 1

14 Getting Started with Performance Monitor Chapter 1 Step Install Enterprise PeopleTools. Install one or more PeopleSoft applications. Reference See Enterprise PeopleTools 8.50 Installation for your platform. See Installation documentation for your applications. Post Installation-Level Steps To enable the Performance Monitor complete these steps. Step Configure appropriate access to the menu items that are under the PeopleTools, Performance Monitor menu. System administrators will need access to these pages to configure the system and view the pages displaying performance information. Configure the monitoring system. Configure one or more monitored systems. Reference See Enterprise PeopleTools 8.50 PeopleBook: Security Administration. See Chapter 3, "Setting Up the Performance Monitor," Enabling the Required Elements on the Monitoring System, page 15. See Chapter 3, "Setting Up the Performance Monitor," Enabling the Required Elements on the Monitored System, page Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

15 Chapter 2 Understanding Performance Monitor This chapter discusses: Performance Monitor. Performance Monitor architecture. Implementation options. Performance Monitor instrumentation. Performance Monitor integration with third party systems. Performance Monitor data. Scope of the Performance Monitor. Performance Monitor Performance Monitor helps system administrators monitor PeopleSoft performance data and, if wanted, share the data with third-party monitoring tools. Performance Monitor is a diagnostic utility for monitoring the performance of the main elements of your PeopleSoft system, such as web servers, application servers, and Process Scheduler servers. You can monitor real-time performance and analyze historical data. Performance Monitor reports: Durations and key metrics of PeopleTools runtime execution, such as SQL statements and PeopleCode events. Key resource metrics, such as host CPU use and web server execution threads. See Chapter 2, "Understanding Performance Monitor," Performance Monitor Instrumentation, page 11. The metrics that are provided by Performance Monitor enable system administrators to: Monitor real-time system performance. Identify poorly performing tiers, hosts, domains, servers, application code, and SQL in a PeopleSoft environment. Identify performance trends. Address and isolate performance bottlenecks. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 3

16 Understanding Performance Monitor Chapter 2 Note. Performance Monitor enables you to isolate and identify the causes of performance issues; however, it does not provide automatic solutions to performance issues. Performance Monitor Architecture The following diagram illustrates the main elements of Performance Monitor. 4 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

17 Chapter 2 Understanding Performance Monitor Performance Monitor architecture A logical distinction must be made between a monitored system and the monitoring system. A monitoring system can monitor one or more PeopleSoft application systems. However, the monitored system can be monitored by just one monitoring system. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 5

18 Understanding Performance Monitor Chapter 2 You can configure a self-monitoring system for development and testing. The Monitored System The monitored system comprises the following main elements: PeopleSoft application database. Application server. Web server. Process Scheduler server. Note. An application server is also known as a business logic server in some PeopleSoft materials. However, in this PeopleBook, only the term application server appears. Performance Monitor agents reside on each main element of the monitored system. Agents run on the servers to gather and report performance information. An agent is a small piece of code operating from within an existing process. It collects performance data and uses a dedicated thread to send the data to the monitoring system. Agent threads report data to the monitoring system's monitor servlet, which in turn reports XML formatted performance data to the PSPPMSRV server processes in the monitoring system's application server. Two types of agents exist: Process instrumentation agents that report data about the specific server activities. Domain resource agents that report resource usage statistics about the process and server, such as CPU usage. PeopleSoft Application Database The database being monitored is the database for your PeopleSoft application, such as the Human Capital Management database or the Customer Relationship Management database. Note. Performance Monitor does not monitor the performance of the RDBMS or the server on which it runs. Application Server The application server domain for the monitored system contains the following elements thta are related to Performance Monitor: 6 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

19 Chapter 2 Understanding Performance Monitor Performance Monitor Element Server process agents Description One agent exists for each of the following server processes running within an application server domain: PSAPPSRV PSQCKSRV PSQRYSRV PSMONITORSRV PSSAMSRV PSANALYTICSRV Domain Monitor (PSMONITORSRV) The domain monitor runs in the PSMONITORSRV application server process. It monitors: Host resource usage. Tuxedo performance metrics. Note. The domain monitor also monitors its own resource consumption. Web Server The web server contains the following elements that are related to the Performance Monitor: Performance Monitor Element Web server agents Domain Monitor Description Reports performance metrics from the instrumented code. The system starts a web server agent for a site after the site gets accessed for the first time. Reports resource events that related to the web server usage. One resource monitor exists for each system that is hosted in each web application on each web server. Process Scheduler Server The Process Scheduler for the monitored system contains the following elements that are related to Performance Monitor: Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 7

20 Understanding Performance Monitor Chapter 2 Component Server process agent Description On the Process Scheduler server, only the Master Scheduler process, PSMSTPRC, has a server process agent recording performance data. The PSMSTPRC agent reports batch summary metrics, such as the number of and types of queued tasks. Note. Performance information of this type is available only if a Master Scheduler is configured to run on the system. Domain Monitor (PSMONITORSRV) The domain monitor runs in the PSMONITORSRV server process. It monitors: Host resource usage. Tuxedo performance metrics. Note. The domain monitor also monitors its own resource consumption. The Monitoring System Like the monitored system, the monitoring system includes the following elements: Performance Monitor database. Application server. Web server. Process Scheduler server. Performance Monitor Database The database for the monitoring system requires at least the PeopleTools System database (PTSYS) that you create during an initial PeopleSoft installation. This database contains all of the necessary PeopleSoft objects and definitions that are required to store and display PeopleSoft performance data. Note. The space that is required to store performance data requires increased tablespace sizes, especially in PSIMGR and the associated index tablespace. Note. Any PeopleSoft system can act as a monitor for any other PeopleSoft system or for itself. See Chapter 2, "Understanding Performance Monitor," Implementation Options, page 10. Application Server The application server on the monitoring system contains the following elements that are related to Performance Monitor. 8 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

21 Chapter 2 Understanding Performance Monitor Performance Monitor Element PSAPPSRV Description The PSAPPSRV server process functions just as the PSAPPSRV process functions on any PeopleSoft application server. In addition, it advertises Performance Monitor services that are used by the Performance Monitor servlets to retrieve configuration and security data. At least two PSAPPSRV processes are required by the Performance Monitor. This ensures that: The system can publish notifications of configuration changes through the PeopleSoft Integration Broker gateway. The PPMI servlet can contact the application server to retrieve the configuration changes. PSPPMSRV The PSPPMSRV subscribes to performance data that is sent by the monitor servlet of the monitoring system and receives XML messages containing performance data, which it inserts into the monitoring database. Note. The PSPPMSRV does not receive any data directly from agents running on the monitored system. Web Server The web server on the monitoring system contains the following elements that are related to the Performance Monitor: Performance Monitor Element PIA (psc) and Portal (psp) servlets Monitor (monitor) servlet Performance Monitor Integration (ppmi) servlet Description These servlets are the main PeopleSoft servlets that handle browser requests. Administrators use PeopleSoft pages to view performance information or set configuration options for a monitoring system. The monitor servlet receives all of the data that is sent by each agent running on the monitored system and relays it to the PSPPMSRV processes. The PPMI servlet accepts and authenticates registration requests from PSPPMSRV processes to establish an HTTP pathway for receiving data from the monitor servlet. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 9

22 Understanding Performance Monitor Chapter 2 Performance Monitor Element Integration Broker gateway (PeopleSoftListeningConnector) servlet Description The gateway servlet is used to notify the monitor servlet of any configuration changes. The monitor servlet, in turn, notifies the agents that are running on the monitored system of any configuration changes the next time the agents interact with the monitor servlet. Note. This is the same servlet that is used by PeopleSoft Integration Broker to handle integrations. Process Scheduler Server The Process Scheduler server for the monitoring system has the following function: Performance Monitor Element PeopleSoft Application Engine Description You can schedule or invoke a variety of PeopleSoft Application Engine programs that maintain the data in your monitoring database. See Chapter 4, "Administering the Performance Monitor," page 23. Implementation Options Two options are available for implementing the Performance Monitor: Self-monitoring system. Production monitoring system. Self-Monitoring Monitoring System In a self-monitoring system, the same database that serves the PeopleSoft application is also being used to store the performance data. By default, all the elements that are required for the Performance Monitor are installed during the PeopleSoft server installations. So implementing the self-monitoring system requires no additional installation steps. While this system may require less hardware, the potential impact to your online applications must be considered. Keep in mind that the same database engine is being used to store and retrieve performance data, and the same web server and application server may be used to monitor and collate performance data. Note. The self-monitoring configuration is not supported for a production environment. The self-monitoring configuration is not designed for a production environment as it affects online transaction processing and overall system performance. Typically, the self-monitoring system is used in development, testing, or training. 10 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

23 Chapter 2 Understanding Performance Monitor Note. The Performance Monitor has been designed so that a monitoring system does not encounter an infinite loop in a self-monitoring configuration. Production Monitoring System With a production monitoring system, you configure a separate PeopleSoft system complete with web server, application server, and database server to act solely as a performance monitoring system. This is the recommended configuration for monitoring your production systems. Any PeopleTools system database that is delivered with PeopleSoft contains the appropriate database definitions to store and retrieve all Performance Monitor data. With the separate monitoring system, you can monitor a single PeopleSoft system, such as your PeopleSoft Human Capital Management system, or you can configure it to monitor multiple PeopleSoft systems. Note. Performance Monitor can monitor any PeopleSoft system running on PeopleTools 8.44 or above. Always consult PeopleTools Release Notes and PeopleBooks to become aware of any exceptions or unsupported features between release levels. Performance Monitor Instrumentation This section explains the terminology that is associated with the Performance Monitor instrumentation. Instrumentation refers to the modifications that are made to internal PeopleSoft programming code that enables the system to report performance data to a monitor. Select areas of PeopleTools runtime source code have been instrumented. When the system runs instrumented code, it generates time-stamped performance metrics that are relevant to that section of code. Agents running on the monitored system send the performance data to the Performance Monitor. Instrumentation generates performance measurement units (PMUs) and events. Understanding PMUs A PMU is a unit of measure that reflects the execution of a section of code. The system starts and stops a PMU at specific code locations, and the system may update a PMU anytime between the start and stop times. PeopleTools has defined a set of PMU types, and each type of PMU corresponds to the instrumentation at a specific code location, such as a SQL Execute in PSAPPSRV or a Jolt Request in the web server. Each PMU includes: PMU Type. Instance identifier (a unique identifier for a specific PMU instance). Start time. Stop time. Status. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 11

24 Understanding Performance Monitor Chapter 2 Metrics (such as number of SQL fetches or buffer size used in a Jolt response). See Chapter 10, "Performance Monitor Meta-Data," PMU Definitions, page 149. Open PMUs are those that are currently being processed. These are PMUs for which a "stop" has not yet been received. Completed PMUs are those for which a "stop" has been received. PMUs can assume a parent-child relationship. Child-PMUs start within a parent-pmu. Child-PMUs are linked to their parent by the parent instance ID. You can view PMUs within a tree structure that reveals the hierarchy of parent-child PMUs and indicates the processing times at each level. Understanding Events Events are notifications containing performance metrics that are different from PMUs in that they are not hierarchical and they do not have durations. Events relate to resource usage, such as CPU usage or memory allocation. PeopleTools has defined a set of event types, and each type of event is reported at a specific location in the instrumented code. Each event has: Type. Instance identifier (a unique identifier for a specific event instance). Timestamp. Severity. Metrics (such as CPU usage and memory allocation). See Chapter 10, "Performance Monitor Meta-Data," Event Definitions, page 148. Performance Integration with Third Party Systems Using web service technology, performance data that is generated by the Performance Monitor can be shared with third-party performance-monitoring tools by way of the Performance Monitor Integration (PPMI) API. The PPMI API uses the Simple Object Access Protocol (SOAP). Note. The PPMI API protocol is not documented in PeopleBooks. It is used only in products that are developed by PeopleSoft system and performance management integration partners. You can use the PPMI API only if you are licensed through PeopleSoft. Performance Monitor Data After the Performance Monitor is configured to retrieve and store data in the performance database tables, you can view the performance data using a variety of PeopleSoft pages that: Provide a summary of performance data. 12 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

25 Chapter 2 Understanding Performance Monitor Present raw performance data. Display information in charts. You can access all of the pages that are related to Performance Monitor using the PeopleTools Performance Monitor menu. Some performance information appears in grids. Viewing grids While other performance information appears in charts. Viewing Charts You have the option to view chart information in grids for sorting and downloading to Microsoft Excel. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 13

26 Understanding Performance Monitor Chapter 2 Scope of the Performance Monitor The Performance Monitor provides performance metrics for: PeopleSoft Application Server processes (including SQL, PeopleCode, and Cache management). Web server servlet performance. Host resource usage. Master Scheduler performance. Tuxedo performance. Recycled server processes in the application server domain. User request performance. Note. This list contains the only elements that are monitored by Performance Monitor. 14 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

27 Chapter 3 Setting Up the Performance Monitor This chapter provides an overview of the configuration of the Performance Monitor and discusses how to: Enable the required elements on the monitoring system. Enable the required elements on the monitored system. Configuration of the Performance Monitor The configuration of the Performance Monitor involves enabling the required elements on the monitored system and the monitoring system. When you have installed the PeopleTools CD and transferred the appropriate files to your web server, application server, and Process Scheduler server, all of the required elements are in place. You do not need to install additional modules for the Performance Monitor. Note. If you are setting up a self-monitoring system, the monitored system and the monitoring system reside on the same host machine. Enabling the Required Elements on the Monitoring System This section covers the configuration of your monitoring system, which includes: Setting up the Performance Monitor database. Setting up the application server. Setting up the web server for the monitoring system. Creating a PPMI user ID. Specifying the PPMI URL and PPMI user ID. Specifying the Integration Gateway URL. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 15

28 Setting Up the Performance Monitor Chapter 3 Setting Up the Performance Monitor Database The monitoring database requires the use of at least the PeopleTools System (PTSYS) database for the production monitoring system. Set this database up using the instructions in the PeopleTools Installation Guide. Note. For the self-monitoring configuration, use the application database for your system. The application must be running on at least PeopleTools See Also Enterprise PeopleTools 8.50 Installation for your platform Setting Up the Application Server The application server on the monitoring system requires that the following server processes be enabled: Typical application server processes included in a default configuration. PSPPMSRV. Default Server Processes A typical default application server configuration is suitable for Performance Monitor activity. The PSAPPSRV server process is the main server process of the PeopleSoft Application Server. At least two PSAPPSRV server processes must be enabled. This ensures that the monitoring system can publish notifications of configuration changes through the PeopleSoft Integration Broker gateway. You can enable more PSAPPSRVs to meet increased demand. PSPPMSRV The PSPPMSRV process receives the performance data from the monitor servlet and inserts it into the monitoring database. The following table describes the ways that you can enable this server process using the PeopleSoft Server Administration interface (PSADMIN). Method Quick Configure menu Description You can use the Quick Configure menu in PSADMIN to enable the PSPPMSRV server process. Enable the Perf Collator option. 16 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

29 Chapter 3 Setting Up the Performance Monitor Method Custom Configuration menu Description After you have scrolled through all of the configuration sections of the PSADMIN menu, the system prompts you to enable a variety of options. One of these prompts relates to enabling the PSPPMSRV server process. When prompted with the following: Do you want Performance Collators configured? (y/n) Enter y to indicate Yes. To add additional PSPPMSRV server processes, use the [PSPPMSRV] configuration section in PSADMIN and adjust the settings. The [PSPPMSRV] configuration section contains the following parameters. Min Instances (minimum instances) Enter the minimum number of PSPPMSRV server processes that you want to start when the application server boots. Max Instances (maximum instances) Enter the maximum number of PSPPMSRV server processes that you want to start when the application server boots. Note. For the PSPPMSRV server process, the values assigned to Min Instances and Max Instances must be equal. For example, if you want three PSPPMSRV server processes to start, set Min Instances=3 and Max Instances=3. Unlike PSAPPSRV, Tuxedo does not handle load balancing between PSPPMSRVs. Decide how many PSPPMSRV instances you need operating within the appropriate number of domains and boot all of them. If you have more than one PSPPMSRV configured, the monitor servlet automatically implements load balancing across the multiple PSPPMSRVs. Also, if you configure multiple application server domains, each with its own PSPPMSRVs, the monitor servlet automatically implements load balancing and failover across the domains. Setting Up the Web Server for the Monitoring System By completing the internet portion of the PeopleSoft Multi-Platform Installer program, you have installed all of the elements that Performance Monitor requires on the web server. No additional installation procedures are required. Performance Monitor servlets are not activated unless PSPPMSRVs register with them. Agent connections to a monitor are rejected until a PSPPMSRV on the monitoring system registers. The Performance Monitor servlets that run on the web server of the monitoring system are: Monitor servlet. PPMI servlet. Note. Make note of the URL of the web server for the monitoring system. On the monitored system, you need to specify the appropriate URL to which to send performance data. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 17

30 Setting Up the Performance Monitor Chapter 3 Note. If you elect to implement a production monitoring system, you should set up the monitoring system itself in self-monitoring mode so that you can detect alarm events that are generated from the monitoring web server. That is, while the monitoring system monitors one or more PeopleSoft application systems, it also monitors itself. If you need to configure multiple web servers for scalability and failover, you need to configure a cluster. Note. If you notice communication errors in the log files, even though the agents have registered successfully, check the keep-alive setting on the monitoring web server. The keep-alive timeout in the monitoring web server may need to be increased to 660 or higher to resolve these issues. After changing this setting, restart the web server. See Also Chapter 4, "Administering the Performance Monitor," Setting Global System Options, page 25 Creating a PPMI User ID The PSPPMSRV application server process acts as the PPMI client when you record PeopleSoft performance data. To register as a client, the PSPPMSRV requires the appropriate permissions in PeopleTools Security. Warning! Setting up PSPPMSRV authentication is required. To set up PPMI authentication: 1. Create a user profile with the name of your choice using PeopleTools Security. 2. Add the deliveredpeopletools Perfmon Client role to the user profile. The PeopleTools Perfmon Client role contains the PTPMCLNT permission list. 3. Select PeopleTools, Performance Monitor, Administration, Global Administration and enter the user profile that you created and the associated password in the PPMI User ID and PPMI Password text boxes. Specifying the PPMI URL and PPMI User ID PSPPMSRV processes must register with the PPMI servlet using the PPMI URL. The PSPPMSRV processes use the PPMI URL that is stored in the monitoring system's database to locate the PPMI servlet for registration. The PSPPMSRV servers periodically read the PPMI value to determine the PPMI URL. When it is set to NONE, the monitoring system is disabled, which means that the PSPPMSRVs do not insert performance data into the monitoring database. The PPMI user ID is required to authenticate the PSPPMSRV so that it can subscribe to performance data. To specify the PPMI URL and PPMI user ID: 1. Make sure that you have created a PPMI user ID with the appropriate permissions (as described in the previous section). 18 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

31 Chapter 3 Setting Up the Performance Monitor 2. Select PeopleTools, Performance Monitor, Administration, Global Administration. 3. On the Global Administration page, enter the appropriate value in the PPMI URL edit box. The format is: For example: Where psis the name of the PeopleSoft site. 4. Specify the PPMI user ID and password. 5. Click Save. 6. Click Ping Test to ensure that the URL that you entered is valid and the URL is available. See Also Chapter 9, "Performance Monitor Security Considerations," page 137 Specifying the Integration Gateway URL A gateway URL needs to be specified so that the monitoring system can notify the monitor servlet of configuration changes. The monitoring servlet in turn notifies the monitored agents of these changes when they next communicate with the servlet. This communication uses the gateway, which you specify in the Integration Broker administration pages. When specifying the gateway URL: Select LOCAL gateway. Enter this URL in the following format: See Also Enterprise PeopleTools 8.50 PeopleBook: PeopleSoft Integration Broker Administration, "Managing Integration Gateways," Defining Integration Gateways Setting Up the Process Scheduler Server for the Monitoring System To invoke Performance Monitor Application Engine programs to maintain the performance data, set up a Process Scheduler server that connects to your monitoring database. To set up a Process Scheduler server, follow the instructions that are provided in your Enterprise PeopleTools Installation Guide. See Enterprise PeopleTools 8.50 Installation for your platform The delivered PeopleSoft Application Engine programs that you schedule to run as part of Performance Monitor administration enable you to: Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 19

32 Setting Up the Performance Monitor Chapter 3 Maintain the current PMU table (the PSPM_REAPER program). Archive or delete historical performance data (the PSPM_ARCHIVE program). Manage the lookup tables in the Performance Monitor interface (the PSPM_LOOKUP program). See Also Enterprise PeopleTools 8.50 Installation for your platform Chapter 4, "Administering the Performance Monitor," page 23 Enabling the Required Elements on the Monitored System This section covers enabling the required elements on the monitored system, which is the system for which you intend to collect performance data. A monitored system requires the following items to be in place: PeopleSoft application database. Web server agent. Application server domain manager and agents. Process Scheduler server domain manager and agents. Valid monitor servlet URL value. Setting Up the Database of the Monitored System As long as your PeopleSoft application is running against your PeopleSoft database,no further database configuration is required for the Performance Monitor. Note. The database must be running on PeopleTools 8.44 or higher. Specifying the Monitor URL Specifying a monitor URL enables the Performance Monitor agents and identifies the monitor servlet. The agents on the monitored system check the monitor URL value periodically. When the monitor URL value is set to NONE, the Performance Monitor components on the monitored system are disabled. The monitor URL value must reflect the URL of the monitor servlet that is running on the monitoring system. To specify the monitor URL: 1. Select PeopleTools, Performance Monitor, Administration, Specify Monitor. 20 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

33 Chapter 3 Setting Up the Performance Monitor 2. On the Specify Monitor page, enter the appropriate value in the Monitor URL edit box. The format of the URL is http[s]://host[:port]/monitor/[site]/or NONE. For example, enter Where ps is the name of the PeopleSoft site. 3. Click Ping Test to ensure that you entered a valid URL and that the URL is available. 4. Click Save and Ping PPM to save the URL into the URL catalog, and to ensure that the associated agents are registered and able to share data with PPMI URL. If the URL you enter is different than the previous value, the system saves the new value to the URL catalog, performs registration, and displays status of the registered agents. In the case of a new registration, you must wait at least the duration of the Agent Heartbeat Interval. If the agents have successfully registered, the PPMConsole (a new browser window) displays a list of all registered agents along with the agent configuration information. If no agents have registered, no list appears. Note. In order for the new browser window (PPMConsole) to appear, the PPMConsole needs to be enabled on the Global Administration page. See Chapter 4, "Administering the Performance Monitor," Viewing Monitor Servlet Diagnostics, page 43. See Chapter 4, "Administering the Performance Monitor," Setting Global System Options, page 25. Setting Up the Web Server of the Monitored System No additional installation procedure is required. When the monitor URL value is supplied, agents on every PeopleSoft site automatically become active and begin relaying performance data to the monitoring system. Note. Ensure that the Enable PPM Agents option is selected on the General tab of the appropriate web profile. By default, this option is enabled. Note. If you start the web server in safe mode, then the monitoring agents in the web server are disabled. Setting Up the Application Server The following components run and are monitored on the application server of the monitored system: Domain monitor: PSMONITORSRV. PSAPPSRV agents. PSQCKSRV agents. PSQRYSRV agents. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 21

34 Setting Up the Performance Monitor Chapter 3 PSSAMSRV agents. PSANALYTICSRV. No installation or configuration procedures are required. When the monitored system is enabled, agents in every instrumented server process automatically become active and begin relaying performance data to the monitoring system. Note. Ensure that the EnablePPM Agent parameter in the [PSTOOLS] section of PSADMIN is set to 1 (enabled). By default, it is set to 1 (enabled) in the delivered domain templates. If the EnablePPM Agent parameter is commented out or set to 0, then the agents are disabled for this server domain. Setting Up the Process Scheduler Server The following components run and are monitored on the Process Scheduler server of the monitored system: Domain monitor (PSMONITORSRV). PSMSTPRC agents. No installation or configuration procedures are required. When the monitored system is enabled, agents in instrumented server processes automatically become active and begin relaying performance data to the monitoring system. Note. Ensure that the EnablePPM Agent parameter in the [PSTOOLS] section of PSADMIN is set to 1 (enabled). By default, it is set to 1 (enabled) in the delivered domain templates. If the EnablePPM Agent parameter is commented out or set to 0, then the agents are disabled for this server domain. 22 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

35 Chapter 4 Administering the Performance Monitor This chapter provides an overview of Performance Monitor and discusses how to: Set global system options. Set system definition defaults. Modify system definitions. View agent definitions. Set agent filter options. Schedule the reaper program. Schedule performance data archiving. Populate Performance Monitor lookup tables. Work with Performance Monitor tables. Disable Performance Monitor agents. Work with Performance Monitor web profile properties. Trace Performance Monitor agents. Trace the Monitor and PPMI servlets Configure performance monitoring sampling rate. View monitor servlet diagnostics. Set up monitor clusters. Use Performance Monitor data mover scripts. Estimate you performance database size. Performance Monitor Administration This section discusses Performance Monitor administration and lists the pages used for Performance Monitor administration. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 23

36 Administering the Performance Monitor Chapter 4 Performance Monitor Administration Performance Monitor administration includes: Specifying global settings. Viewing performance definitions such as those related to systems, agents, metrics, and so on. Setting system defaults. Scheduling batch programs that maintain performance data. Pages Used for Performance Monitor Administration Page Name Definition Name Navigation Usage Global Administration PSPMMONITORGBL PeopleTools, Performance Monitor, Administration, Global Administration System Defaults PSPMSYSDEFAULTS PeopleTools, Performance Monitor, Administration, System Defaults System Definitions PSPMSYSDEFN PeopleTools, Performance Monitor, Administration, System Definitions Agent Definitions PSPMAGENT PeopleTools, Performance Monitor, Administration, Agent Definitions Agent Filters PSPMAGENTFILTER PeopleTools, Performance Monitor, Administration, Agent Filters View and modify global administration settings, such as the PPMI URL value and monitor servlet clusters. Set global system defaults for all monitored systems. View and modify the system definition that is associated with each of the systems that are being monitored. For example, you can set archive, PMU timeout, and agent buffer size. View the definitions of the agents that are running on the monitored system's application server, web server, and Process Scheduler servers. It also enables you to disable the display of agent information. Set the filter levels for the agents that are running on the monitored system. The filter level determines the amount and granularity of the performance data that is reported by an agent. 24 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

37 Chapter 4 Administering the Performance Monitor Page Name Definition Name Navigation Usage Schedule Reaper PSPMREAPERRUNCNTL PeopleTools, Performance Monitor, Administration, Schedule Reaper Schedule Archive PSPMARCHIVERUNCNTL PeopleTools, Performance Monitor, Administration, Schedule Archive Schedule the PeopleSoft Application Engine reaper program that is designed to maintain the integrity of the performance data that is in your monitoring system. Schedule the PeopleSoft Application Engine archive program that is designed to archive or purge the performance data that is in your system. Schedule Lookup Maintenance PSPMLOOKUPRUNCNTL PeopleTools, Performance Monitor, Administration, Schedule Lookup Maintenance Schedule the PeopleSoft Application Engine program that is designed to maintain the lookup lists that are associated with the Performance Monitor interface. The lookup maintenance program ensures that the lookup lists are populated with current lookup options. Setting Global System Options Access the Global Administration page (PeopleTools, Performance Monitor, Administration, Global Administration). Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 25

38 Administering the Performance Monitor Chapter 4 Global Administration page 26 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

39 Chapter 4 Administering the Performance Monitor PPMI URL The PPMI URL that is stored in the monitoring system's database is used by PSPPMSRV processes to locate the PPMI servlet with which to register. The format of the URL is: http[s]://<host[:port]>/ppmi/<site_name>/ For example, After you specify this URL value, it is stored in the URL catalog under the ID PPM_PPMI. Note. If you select HTTPS, then the PSPPMSRVs use SSL encryption when sending the PPMI user ID and password to the PPMI servlet, but performance data sent from the PPMI servlet to PSPPMSRV is not encrypted. Because the data is not encrypted, the PSPPMSRV and the monitor servlet should reside in a secure network environment. Note. Use the Ping Test button to verify that you have entered a valid URL and that the URL is available. PPMI User ID The PSPPMSRV server processes pass this user ID and password to the PPMI servlet. The servlet verifies that the user ID and password are valid, and that the user has permission to access the PPMI servlet. The user ID that you specify must have a permission list with Performance Monitor PPMI Access selected in its PeopleTools permissions. PeopleTools recommends using the PTPMCLNT permission list, which is shipped expressly for this purpose. Note. The PPMI User ID and PPMI Password values are required. See Chapter 9, "Performance Monitor Security Considerations," page 137. PPMI Password The password associated with the PPMI user ID. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 27

40 Administering the Performance Monitor Chapter 4 Archive: Clear PMUs & Events Indicates to the archive program to delete all of the performance history data that is currently stored in the monitoring database. This is in the form of an unqualified DELETE on the history tables, or for those platforms for which it is supported, the tables are truncated. Note. If you select this option, the delivered archive program deletes all of the performance history data for every monitored system. Note. Deleting and truncating performance data may cause the current user count to appear artificially low. The system bases the count on events that are found in the PSPMEVENTHIST table, and rows from this table may be deleted during the archive process. Note. This option overrides any archive mode option that is set on the System Definition page. See Chapter 4, "Administering the Performance Monitor," Modifying System Definitions, page 30. Collator Row Limit Limits the amount of data that can be inserted into the Performance Monitor tables. Collators (PSPPMSRVs) periodically count the number of rows in each of the performance tables. If the total count of rows in PSPMTRANSHIST, PSPMTRANSCURR, and PSPMEVENTHIST is higher than this value, the PSPPMSRV does not insert any more rows into these tables until the number of rows deleted (by the archive program, the reaper program, manual intervention, or any combination of the three) brings the row count below this limit. If you set this value to 0, the PSPPMSRVs do not check for a row limit. Note. If the row limit is reached, the System Performance page will report that the agents on the monitored system have stale agent data. Search Row Limit Performance Monitor Cluster Limits the number of rows that are returned and displayed in searches that are initiated from Performance Monitor pages. The system imposes this limit on all users who access the monitoring system. To provide failover and scalability, performance monitor servlets on multiple web servers can be configured as a cluster. Entering the cluster information in the Performance Monitor Cluster grid enables the load-balanced monitor and PPMI servlets to share client registration information so that PSPPMSRV instances need not be aware of the cluster configuration. The format of the cluster member URL is where the host name is the actual web server machine. Click Save and Notify Cluster to notify the cluster if you have added or removed a cluster member. Click Ping Test to verify that you have entered a valid URL and that the URL is available. See "Setting up Monitoring Clusters" below. 28 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

41 Chapter 4 Administering the Performance Monitor PPMConsole Settings The PPMConsole settings control the display of the PPMConsole, which is an interface in a separate browser window that displays information related to servers, servlets, and agents. Click Enable PPMConsole to enable access to this interface. To enable password security for accessing the PPMConsole, enter a password in the Password field. If you have provided a password, any system administrator who would need to access the PPMConsole will need to be aware of the password. Note. After making any changes to the PPMConsole settings, you must restart the web server. See Chapter 3, "Setting Up the Performance Monitor," Specifying the Monitor URL, page 20. See Chapter 4, "Administering the Performance Monitor," Viewing Monitor Servlet Diagnostics, page 43. Setting System Definition Defaults Access the System Defaults page (PeopleTools, Performance Monitor, Administration, System Defaults). The System Defaults page enables you to set default values for all of the monitored systems. If you intend to monitor numerous systems, you can set the default values that you need for a system. When a new systems register with the monitoring system for the first time, the system adopts the default values that you have set. Using the System Defaults page enables you to set global values for each monitored system rather than modifying the values for each system separately. Note. Except for the following page elements, the System Defaults page is identical to the System Definitions page, which is documented in the following section. Agent Filter Level Apply to Current Systems Set the agent filter level for the agents of monitored systems. The default setting is 01 Standby, which means that the monitored system sends no performance information to the monitoring system. See Chapter 4, "Administering the Performance Monitor," Setting Agent Filter Options, page 35. Notifies the agents that are running in existing systems of the global configuration changes. When the agents of the existing monitored systems are notified, the existing systems adopt the new, default values. See Also Chapter 4, "Administering the Performance Monitor," Modifying System Definitions, page 30 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 29

42 Administering the Performance Monitor Chapter 4 Modifying System Definitions Access the System Definitions page (PeopleTools, Performance Monitor, Administration, System Definitions). System Definitions page A system refers to a particular monitored system. For example, you can monitor your PeopleSoft CRM system, PeopleSoft HCM system, and PeopleSoft Financials system. System definitions are created automatically when the first agent of a monitored system registers with the monitoring system. The database name and GUID (a PeopleSoft value used to uniquely identify a PeopleSoft system) are provided by the agent during its registration process. This section describes the properties and configuration options for each monitored system. System Identifier Database Name Identifies each monitored system. PeopleSoft automatically generates this value incrementally. The name of the PeopleSoft application database that is running on the monitored system. The monitoring system automatically inserts this value when it recognizes and creates a monitored system. 30 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

43 Chapter 4 Administering the Performance Monitor Unique Identifier Uniquely identifies a particular PeopleSoft system. PeopleSoft assigns a unique value, referred to as a GUID, to each PeopleSoft application installation. When an agent registers with the Performance Monitor, it provides this GUID. The first time the monitoring system receives information from a monitored system, it detects the GUID. For each new GUID that is detected, the monitoring system creates a new monitored system definition. Note. When copying databases, you should delete the GUID in the new (copied) database. If the new database is monitored by the same instance of the Performance Monitor that is monitoring the source database, the monitor assumes that the agents for both systems belong to the same system. This not only doubles the data that is being stored for a particular system, but also makes it unreliable. To resolve this, set the value of the GUID field in the PSOPTIONS table to <space> in the new database. The next time an application server connects to the database, the system generates a new, unique GUID. You insert the blank value in the PSOPTIONS table using the SQL tool at your site. Note. After cloning a database and before booting the web server and application server, you should set the PPMI and monitor URL to NONE in the cloned database. This will prevent the agents of the cloned database from reporting into the monitoring system that was used previously. To reset the URLs, update the URL field to NONE in the PSURLDEFN table where the URL_ID field is'ppm_monitor or PPM_PPMI. Description A description of the monitoring system to assist recognition. The default value is the database name. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 31

44 Administering the Performance Monitor Chapter 4 Archive Mode The archive mode that you set specifies how the archive program (PSPM_ARCHIVE) should process the performance data that is stored in the monitoring database. The default value is set to archive nothing after zero days. The options are: After N days. If you select Archive Data or Delete Data, then you must specify a retention period, which determines the number of days during which performance data remains in the tables of the online monitoring system. Performance data that is older than the specified value gets archived or deleted (depending on your selection) when the archive program runs. Delete Data. If this option is select, the next time the archive program runs, the program deletes performance history data that is older than the retention period. Archive Data. If this option is selected, the next time the archive program runs, the program archives performance history data that is older than the retention period. The archive program moves history tables (PSPMTRANSHIST and PSPMEVENTHIST) to the archive tables (PSPMTRANSARCH and PSPMEVENTARCH). Archive Nothing. Disables performance data archiving for this monitored system. Delete System. If this option is selected, the next time the archive program runs, it deletes the system definition and its performance data. Keep in mind that deleting a system does not prevent the monitored system from continuing to send performance data. You must set the Monitor URL value on the monitored system to NONE to disable monitoring. Note. If the Archive: Clear PMUs & Events option is enabled in the Global Administration page, the archive settings are ignored. See Schedule Archive section. Allow Performance Trace Enables authorized users who are connected to this monitored system to launch the Performance Trace Console. By default, this option is enabled on a new monitored system unless you adjust the system defaults on the System Defaults page. See Chapter 5, "Working with the Performance Trace," page Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

45 Chapter 4 Administering the Performance Monitor PMU Timeout Indicates the interval in which an open PMU is considered timed out. PMU timeouts are measured in days. The PMU Timeout value can't be set to zero. After the specified interval, the system assumes that a PMU that has not finished encountered an error and should no longer be considered open. The PeopleSoft Application Engine reaper program (PSPM_REAPER) moves timed out PMUs from the current PMU tables to the history PMU tables and sets the status to Reaper Timed Out. Note. When a PMU starts, the application server agents specify an override timeout that is equal to the Tuxedo service timeout of the domain. In such cases, the reaper program uses the override value instead of the PMU Timeout that is specified here. Agent Event Sample Rate Specifies the rate (in seconds) that agents collect resource metrics, such as CPU usage. The default is 300 seconds (five minutes). To disable event sampling, set the value to zero. Note. Smaller sampling rate intervals will provide more frequent updates of machine and process resource utilization on your monitored systems. However, consider that by decreasing the sampling rate interval, you increase the volume of data that is sent to and stored in the performance database. This increases the overall impact of performance monitoring. Agent Buffering Interval Specifies the rate (in seconds) at which an agent sends performance data to the monitoring system. This value can't be set to zero. A smaller interval decreases the delay between the time when the monitored system generates performance data and the time that it is displayed on the monitoring system. Larger intervals enable more efficient transmission of performance data across the network because the system can consolidate the data into packets. The larger the interval, the greater the Agent Max Buffer size should be set. The default is 10 seconds. Agent PMU Sample Rate Enables you to reduce the amount of PMU data that is generated by monitoring just 1 of every N server trips. Sampling does not affect PMUs that are initiated in a performance trace. See Chapter 4, "Administering the Performance Monitor," Configuring Performance Monitoring Sampling Rate, page 43. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 33

46 Administering the Performance Monitor Chapter 4 Agent Heartbeat Interval Agent Max Buffer Size Save and Notify Agents This is the interval at which agents that do not have any performance data to report or that are in standby mode connect to the monitor servlet to be notified of any configuration changes. If agents post data to the monitoring system before this interval expires, they will be notified of any configuration changes and the timer will be reset. The lower the number, the greater the responsiveness of the agents to configuration changes, but the greater the network traffic. This value can't be set to zero. The default value is 300. Determines the maximum size of the buffer containing performance data. This enables you to cap the amount of data that is being stored by the agent on the monitored system and the amount of data that is sent across your network. If this limit is reached, the agent automatically discards new performance data until the current data has been posted to the monitor servlet. The agent posts an alarm to the monitoring if the buffer size threshold is exceeded. The default is bytes (4 MB). The minimum must be bytes (10 KB). Notifies the monitor servlet of configuration changes made on this page. First, the system saves the changes to the database. Notification of the monitor servlets occurs through the PeopleSoft Integration Broker gateway. The monitor servlet notifies the agents of changes when the agents next communicate with the monitor. A delay occurs in publishing changes to the monitored systems. The maximum delay is the agent heartbeat interval. Note. The Integration Broker Gateway must be configured correctly; otherwise, you will see an error message indicating that the agents were not notified of system changes. Versions Enables you to view a history of the PeopleTools versions that are installed on a monitored system. The Tools Release column reveals the version numbers. The Valid From columns indicate the date and time that an agent on a particular version first communicated with the monitoring system. The Valid To columns indicate the date and time that an agent on a particular version last communicated with the monitoring system. Viewing Agent Definitions Access the Agent Definitions page (PeopleTools, Performance Monitor, Administration, Agent Definitions). Agent definitions enable you to view the details about the agents in monitored systems, and deactivate specific agents, if needed. 34 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

47 Chapter 4 Administering the Performance Monitor System ID Database Name Agent ID Domain Name Agent Type Domain Type Domain Monitor Server Instance Domain Host/Port Identifies each monitored system. The PeopleSoft system automatically generates this value incrementally. System definitions are created automatically when the first agent of a monitored system registers with the monitoring system. The name of the PeopleSoft application database that is running on the monitored system. The monitoring system automatically inserts this value when it recognizes and creates a monitored system. Uniquely identifies an agent within a system. This is automatically generated by the monitor the first time an agent registers with it. The name of the domain (application server or web server) in which the agent operates. Indicates the type of server process the agent is monitoring. Indicates whether the domain is an application server, a web server, or a Process Scheduler server domain. Displays as Yes or No. If Yes, then this agent is responsible for sending resource events for its host or domain to the monitor at the specified sampling rate for the monitored system. This number is specific to Tuxedo servers and corresponds to the Tuxedo instance number. The name or IP address of the server on which the domain resides, including the port number to which the domain listens for requests. Note. Web server agents register with both the HTTP and the HTTPS ports. The application server agents register with the Jolt Server Listener (JSL) port. Process Schedulers do not have ports. Domain Directory Inactive Agent The directory in which the domain is installed on the server. If this box is selected, the agent is considered inactive. That is, the agent's events and PMUs do not appear in the Performance Monitor pages showing current information. You can still view information about events and PMUs that are sent by inactive agents using the pages that display historical information. To reactivate an agent, clear the check box, and click Save. To deactivate an agent, select the check box, and click Save. Setting Agent Filter Options Access the Agent Filters page (PeopleTools, Performance Monitor, Administration, Agent Filters). Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 35

48 Administering the Performance Monitor Chapter 4 Agent Filters page Agent filters determine the amount of performance data that is generated and sent to the monitoring system. Depending on the situation, different levels of performance data may be needed to assist in your performancerelated decisions. The levels range from no information to extremely detailed information. Each type of PMU and event is associated with a filter level, which is the lowest level at which the system generates performance data for that PMU or event. System ID Database Name Reset All Filters Identifies each monitored system. The PeopleSoft system automatically generates this value incrementally. System definitions are created automatically when the first agent of a monitored system registers with the monitoring system. The name of the PeopleSoft application database that is running on the monitored system. The monitoring system automatically inserts this value when it recognizes and creates a monitored system. Enables you to uniformly adjust the agent filter levels across an entire system. Select the desired level and click Apply. 36 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

49 Chapter 4 Administering the Performance Monitor Agent Filters The Agent Type column displays the types of agents that are currently known to the monitoring system. The Last Update User ID column and Last Update Date/Time column display the user ID that last updated the filtering level and the date and time when the user made the last filtering level change. The Filter Level column contains a drop-down list box for setting the filtering level. You have the following options: 01 Standby: The monitor system sends no performance data to the monitor. Agents on this monitored system contact the monitor at an interval that is specified by the agent heartbeat interval. 02 Error: Only error events are reported to the monitoring system by the agents. 03 Warning: Only error and warning events are reported to the monitoring system by the agents. 04 Standard: This is the default level of monitoring that is adopted by agents when they first register with the monitor. In addition to errors and warnings, PMUs and events with a filter level of Standard are reported to the monitoring system. This is the recommended setting for monitoring production environments. It provides sufficient diagnostic information to isolate performance problems without inundating the network or monitoring database with performance data. 05 Verbose. In addition to errors and warnings, and standard PMUs and events, PMUs with a filter level of Verbose are reported to the monitoring system. This setting provides more detailed performance metrics and may, for some production systems, be worth the overhead that is associated with this monitoring level. 06 Debug: In addition to errors and warnings, standard and verbose PMUs and events, PMUs with a filter level of Debug are reported to the monitoring system. This setting provides extremely detailed performance metrics and is not suitable for monitoring production systems. It is intended only for development and test environments. Save and Notify Agents Notifies the PPMI servlet of configuration changes that are made on this page. First, the system saves the changes to the database. Notification occurs through the PeopleSoft Integration Broker gateway. The monitor servlet notifies the agents of changes when the agents next communicate with the monitor. A delay occurs in publishing changes to the monitored systems. The maximum delay is the agent heartbeat interval. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 37

50 Administering the Performance Monitor Chapter 4 Scheduling the Reaper Program The reaper program is a delivered PeopleSoft Application Engine program named PSPM_REAPER. The reaper program maintains the PeopleTools tables that the Performance Monitor uses to store performance data for current, real-time processing. When the PSPPMSRV gets notified that a PMU has finished (it receives a STOP for an open PMU), it: Flags the corresponding start and update rows in the current PMU table (PSPMTRANSCURR) for deletion. Inserts a row for the completed PMU in the PSPMTRANSHIST table. When the reaper program (PSPM_REAPER) runs, it: Deletes all rows in current PMU table (PSPMTRANSCURR) that are flagged for deletion. Sets the status to timed out for expired PMUs in the current tables. To run the reaper program: 1. Select PeopleTools, Performance Monitor, Administration, Schedule Reaper. 2. Select or add a run control ID. 3. Click Run. PeopleTools delivers a recurrence definition named PerfMon Reaper Recurrence, which is set to run every 15 minutes. Modify this recurrence definition, if necessary, and associate it with the PSPM_REAPER program to schedule the program to run at suitable intervals. Warning! If you do not schedule the reaper program to run often enough, the PSPMTRANSCURR table will grow very large over time, and it may contain many old, open PMUs. Scheduling Performance Data Archiving Performance data archiving options are set per system definition. So your HCM system and your CRM system may have different archiving modes. You define your archive settings in the Archive Mode group box on the System Definition page. The performance data archiving program is a PeopleSoft Application Engine program named PSPM_ARCHIVE. Note. The system overrides the archive options that are set in the System Definition page if you have selected the Clear PMUs & Events option on the Global Administration page. See Chapter 4, "Administering the Performance Monitor," Setting Global System Options, page 25. Note. PeopleSoft provides sample queries that demonstrate how to access data in the archive tables. Currently, no other delivered method of accessing the data is available in archive tables. 38 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

51 Chapter 4 Administering the Performance Monitor See Chapter 4, "Administering the Performance Monitor," Working with Performance Monitor Tables, page 40. Running the Performance Data Archiving Program To run the archiving program: 1. Select PeopleTools, Performance Monitor, Administration, Schedule Archive. 2. Select or add a run control ID. 3. On the Schedule Archive page: Determine whether you want to enable Run %UpdateStats at the end. If you enable this option, the system runs %UpdateStats meta-sql on both the history and archive tables after the archive program finishes successfully. Click Run to launch the archive program. Note. You should set up a recurrence definition in Process Scheduler so that the archiving program runs at regular intervals. This can help keep the performance history tables at more manageable sizes while containing the most relevant data. Working with Aborted Program Runs If the performance data archiving program does not finish successfully, the system automatically invokes the PeopleSoft Application Engine program named PSPM_ARCHREC. This program is designed to return the system to the state it was in before the archive program started. During an archive program run, the PSPPMSRVs redirect incoming PMU and event data to cloned history tables. When the archive program finishes, the system moves the data in the cloned tables to the history tables and the PSPPMSRVs resume inserting data directly into the history tables. If the archive program does not finish successfully, the PSPPMSRVs continue to insert data into the cloned tables. Populating Performance Monitor Lookup Tables On many of the pages that are used for viewing performance information, such as the Current PMUs page, you are prompted to enter either a user ID, a component name, or a performance trace name to narrow the search to relevant performance data. Unless you are self-monitoring, the monitored system components and User IDs differ from those of the monitoring system. Therefore, the performance monitor provides its own lookup tables, and the PSPM_LOOKUP PeopleSoft Application Engine program populates these lookup tables. To run the lookup program: 1. Select PeopleTools, Performance Monitor, Administration, Schedule Lookup Maintenance. 2. Select or enter a run control ID. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 39

52 Administering the Performance Monitor Chapter 4 3. On the Schedule Lookup page, click Run to launch the lookup program. Note. You should set up a recurrence definition in Process Scheduler so that the lookup program runs at regular intervals. Working with Performance Monitor Tables As with any PeopleTool or PeopleSoft application, the underlying application definitions and application data reside in a collection of database tables that are designed using Application Designer. Although most PeopleSoft applications provide data models that show the relationships between the database entities, typically, for PeopleTools, knowledge of the underlying database tables is not required. However, with the Performance Monitor, knowledge of the underlying database tables may be required. For example, the Performance Monitor interface provides numerous options to use when you are viewing performance data, such as viewing by time range, viewing by user, viewing by component, and so on. In some cases, you may want a more customized view of your performance data than what the interface offers. You can use PeopleSoft Query, or your SQL tool of choice, to build queries that run against the Performance Monitor tables and return the specific information that you require. To assist you in creating custom queries, the Performance Monitor data model appears in the form of an entity relationship diagram (ERD) that is posted on My Oracle Suppor. Refer to the Enterprise PeopleTools Release Notes for this release for the current location of the Performance Monitor ERD. See Performance Monitor Database Schema and Use Cases on My Oracle Supplrt Note. The Performance Monitor database schema may change in future releases. To view the results of sample queries running against the Performance Monitor tables, select PeopleTools, Performance Monitor, History, Sample Queries. To view the definitions and SQL of these sample queries, use PeopleSoft Query Manager. The sample queries attempt to show a realistic query while using all of the tables that you may want to include in similar queries. The sample query definitions are: Query PPM_COMP_BUILD_CACHE PPM_COMP_BUILD_CACHE_ARCH Description This query returns all application server requests for a specific system that had to retrieve metadata from the database as opposed to the cache. It also shows the file cache and memory cache for comparison. This query returns information from the PMU history table. This query is similar to PPM_COMP_BUILD_CACHE, except that it returns information from the PMU archive table. 40 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

53 Chapter 4 Administering the Performance Monitor Query PPM_TIMEOUT_SQL_REQ PPM_TIMEOUT_SQL_REQ_ARCH PPM_APPSRV_START_COUNTS PPM_APPSRV_START_COUNTS_ARCH Description This query returns information from the PMU history table while joining information that is stored in the event table. This query retrieves all PMU 400s (Tuxedo Service PCode and SQL) that were running SQL statements when an Event 500 (Jolt Service Exception) was received. It is assumed that this exception occurred because of a timeout, but it could also have been due to an application server outage or a Jolt error. This query is similar to PPM_TIMEOUT_SQL_REQ, except that it returns information from the PMU and Event archive tables. This query returns starting counts for different server processes over a period of time for a specific domain. This query is similar to PPM_APPSRV_START_COUNTS, except that it fetches information from the event archive table. Note. When you are running a sample query, the system prompts you to enter a date. The format for the date is MM/DD/YYYY HH:MM:SS AM/PM. For example, 09/03/ :00:01AM. Disabling Performance Monitor Agents In some cases, you may not want the Performance Monitor agents to run or to be a possible factor in your online system. For example, if you have ten application server domains running against the same database, you may want only one application server domain reporting information to the monitoring system. Agents in a domain whose monitor URL is "NONE" do not collect or transmit performance data. However, they periodically check the URL for changes. Disabling a domain prevents this small portion of Performance Monitor related processing from occurring. To prevent any information from being sent over the network, set the monitor URL to NONE and reboot all monitored domains. To completely disable monitor agents on your domains, deselect the Enable PPM Agents parameter and reboot all monitored domains. Application Server and Process Scheduler Domains Web Server For application server and Process Scheduler domains, you disable the monitor agents using the EnablePPM Agents parameter. This parameter is in the PSTOOLS section of PSADMIN. To disable the monitor agents, set the value to 0. To enable the monitor agents, set the value to 1. Reboot the application server domain for the change to take effect. When disabled, the Monitor URL is ignored by that domain. For the web server, you disable agents by deselecting the Enable PPM Agents option in the Web Profile interface. Reboot the web server for the change to take effect. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 41

54 Administering the Performance Monitor Chapter 4 Working with Performance Monitor Web Profile Properties This section alerts you to important web profile properties that are related to the Performance Monitor. Enable PPM Agent PPM Monitor Buffer Size Trace Monitoring Server Trace PPM Agent Enables a web server agent to be started on a monitored web server. Sets the maximum buffer size for the buffer that is used by the monitor servlet for incoming performance data. The default size is KB (50 MB). If you notice in servlet trace files or other warnings that you regularly see buffer overflows, you may consider increasing this value. Located on the Debugging tab. Enables you to trace the ppmi servlet and the monitor servlet. The system writes the trace information to the web server log file. Located on the Debugging tab. Enables you to trace web server agents on a monitored system. You enable this option on the web server of the monitored system. Tracing Performance Monitor Agents You can set up tracing on the: Application server and Process Scheduler servers. Web server. Enabling Tracing on the Application Server or Process Scheduler Server To enable tracing of the monitor agents on application server and Process Scheduler domains, use the TracePPM parameter in PSADMIN on the application server running on the monitored system. Set TracePPM to 1. To disable, set to 0. When enabled, the agents write debug information on monitored systems to a log file in the application server LOGS directory. To view the information, open TRACEPPM_mmdd.LOG. The LogFence setting for application server logs has no effect on this file. Error messages (such as those that are created when the monitor URL can't be reached) go directly to the APPSRV_mmdd.LOG. Enabling Tracing on the Web Server To enable tracing of the monitor agents on the web server, select Trace PPM Agent on the Debugging tab in the appropriate web profile on the monitored system. The agents write tracing information to the web server log file. 42 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

55 Chapter 4 Administering the Performance Monitor Tracing the Monitor and PPMI Servlets In some cases, you may want to view the activity of the monitor and PPMI servlets that are running on the web server on the monitoring system. To enable tracing for the Performance Monitor servlets, you select the Trace Monitoring Server option on the Debugging tab in the appropriate web profile. The system writes the trace results to the web server log file. Configuring Performance Monitoring Sampling Rate To reduce the overhead that is incurred by monitoring performance, you may not want to monitor every request that is submitted to your system. You can set a sampling rate for your monitored systems so that only one out of every N server requests generate PMUs. For the Nth request, all PMUs are generated at the filter level that is set for each agent type involved in processing the request. Examples of server requests would be browser requests to a web server or Application Designer requests to an application server (when running in a three-tier configuration). You set the sampling rate for PMUs using the Agent PMU Sample Rate option on the System Definitions page. Note. This sampling rate applies only to PMUs, not events. For example, if you set the sampling rate to 1/10, the system monitors the first PIA request, but does not monitor another request until the 11th request arrives at the system. Some PMUs are always monitored regardless of the sampling rate. The PMUs that are never ignored are those that have the Enable Sampling option cleared on the PMU Definitions page. Examples of such PMUs are those related to users signing on, signing off, and being redirected to other sites. Note. Setting the sampling rate to 0 (zero) disables sampling. The web server and the application server maintain separate counters. The web server counts all browser requests, and the application server counts all requests that are submitted directly to the application server, such as component interfaces or Microsoft Windows workstations running Application Designer. See Also Chapter 10, "Performance Monitor Meta-Data," PMU Definitions, page 149 Chapter 4, "Administering the Performance Monitor," Modifying System Definitions, page 30 Viewing Monitor Servlet Diagnostics You can view diagnostic information that is related to the monitor servlet by accessing the servlet using the following URLs. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 43

56 Administering the Performance Monitor Chapter 4 URL Description Reveals additional statistics about the agents sending data to the monitor servlet. Reveals additional statistics about the PPMI clients receiving data from the monitor servlet. Note. To provide access to this interface, you must enable the PPMConsole on the Global Administration page. See Chapter 4, "Administering the Performance Monitor," Setting Global System Options, page 25. Monitoring Agents Agents refers to the agents on various monitored systems that send performance data to the current monitor servlet. The system retrieves the agent information from the monitor's cache. If an administrator has changed any agent settings and clicked Save and Notify Agents on the System Definitions page, the agent information temporarily disappears in the Agents grid. Updated agent settings appear in the Agents grid after the agent communicates with monitor servlet. Note. You identify the monitored system, using the system ID (PeopleSoft GUID) appearing just above each grid. To identify the agents, you need to map the system GUID and agent ID with the definitions in the monitoring database. Note. Agents appear in the grid if they have successfully registered. The appearance of an agent does not imply that data from the agent is being successfully inserted into the monitoring database. The following information appears in the Agents grid. ID Last Comm Filter Buf-Size (buffer size) Send-Itvl (send interval) Heartbeat Sample-Itvl (sample interval) User Trace The agent ID that uniquely identifies an agent within a monitored system. The last time the agent contacted the monitor servlet. The current agent filter level. The current maximum buffer size for the agent that is specified in the system definition. The current agent buffering interval. The current agent heartbeat interval The current agent event sample rate. Indicates whether performance trace is allowed for this agent. 44 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

57 Chapter 4 Administering the Performance Monitor Sampling Rate Sampling Filter The current PMU sample rate. This column is reserved for future use. Monitoring PPMI Clients PPMI clients refer to the PSPPMSRV server processes that interact with the monitor servlet. The Clients grid shows all known PPMI clients. The following information appears in the Clients grid. Group ID URL Queue Length Estimated Queue Size Item Processed Estimated Bytes Processed Max Size Running Avg Size (running average size) Limit The system's unique identifier (PeopleSoft GUID). The internal ID assigned to this PSPPMSRV. The IP address of the PSPPMSRV. The number of PMUs and events not yet sent to this PSPPMSRV. The estimated size in bytes of the PMUs and events not yet sent to this PSPPMSRV. The number of PMUs and events sent across this PSPPMSRV connection. The estimated size in bytes of the PMUs and events sent across this PSPPMSRV connection. The maximum buffer size in bytes for the PMU and event queue reached in the lifetime of this connection. The running average of the estimated size in bytes of the PMUs and events not yet sent to this PSPPMSRV. The maximum buffer size in bytes for the PMU and event queue. Data is discarded when this limit is reached. Setting Up Monitor Clusters The following diagram depicts the relationship between the elements involved in a monitor cluster. In this diagram, the cluster contains two web servers. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 45

58 Administering the Performance Monitor Chapter 4 Monitor cluster elements When implementing a monitor cluster, keep these items in mind: A cluster must be accessed through an external third party load balancer. The web servers in a monitor cluster share PSPPMSRV registration information using HTTP/S. Each monitor servlet (one in each web server) load balances performance information across all PSPPMSRVs. The host and port of the PPMI URL for a clustered environment need to be set to reflect the host and port of the load balancer. 46 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

59 Chapter 4 Administering the Performance Monitor Note. External load balancers should ensure that performance information that is related to one agent is always sent to the same monitoring servlet. When sending performance data to the monitor, agents add their agent ID to the monitor URL. For example, for agent 8, the URL appears as The system administrator should set up a "sticky rule" on the load balancer so that requests from the same agent are always directed to the same web server, when available. If the sticky rule is not in place, a PMU stop time may be inserted into the monitoring database before the corresponding start time. This creates misleading open PMU information and more work for the reaper program. See Chapter 9, "Performance Monitor Security Considerations," page 137. Note. If a cluster member shuts down, all performance data that is currently queued on that cluster member for transmission to a PSPPMSRV is lost. To set up a performance monitor cluster: 1. In the monitoring system, use the host and port of the load balancer in the PPMI URL on the Global Administration page. 2. In the monitoring system, enter the URLs of each load-balanced host in the Performance Monitor Cluster grid on the Global Administration page. The format of the Member Servlet URL is: http[s]://host/ppmi/ps Where ps is the name of your PeopleSoft site, and host is the real host and port of the host on which your cluster member is running. Even though you enter ppmi as the servlet name, failover and scalability are implemented for both the PPMI and the monitor servlets from each site. 3. In the monitored system, use the host and port of the load balancer in the monitor URL on the Specify Monitor page. See Also Chapter 4, "Administering the Performance Monitor," Setting Global System Options, page 25 Chapter 3, "Setting Up the Performance Monitor," Specifying the Monitor URL, page 20 Using Performance Monitor Data Mover Scripts PeopleSoft delivers a set of PeopleSoft Data Mover scripts for use in the administration of the Performance Monitor. The scripts are located in the following directory: PS_HOME\scripts The delivered scripts are described below. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 47

60 Administering the Performance Monitor Chapter 4 perfmondataexport.dms Enables you to export data from your monitoring database. The data can be exported based on a specific system ID, between specific dates and times, or based on a specific performance trace. The default export is based on a specific system ID. The dat file that is created is named perfdata.dat. If you need to export performance data from a specific date and time, a performance trace, or information on all monitored systems, then open the script and edit the script as described in the comments within the script. perfmondataimport.dms Enables you to import data from perfdata.dat, which is created by the perfmondataexport.dms script, into your monitoring database. Warning! Do not run this file on a live monitoring system because current data may be lost. The script contains the REPLACE_DATA * command. PerfmonPurgeAll.dms Enables you to purge all Performance Monitor tables in the monitoring database. Note. This script deletes both system definitions and all performance data that are associated with any monitored system. Warning! Shut down the monitoring system before running this script. Estimating Your Performance Database Size This section provides an overview of estimating your performance database size overview and discusses how to: Estimate space requirements for event data. Estimate space requirements for PMU data. Calculate space requirements. Estimating Your Performance Database Size Overview Because performance monitoring can store a significant amount of data in your performance database, you may want to estimate the amount of data to be stored in your performance database so that you have, or are able to provide, the appropriate amount of space. Performance database sizing estimates are based on the sum of space requirements for events and performance measurement unit (PMU) performance data. Event data resides in the PSPMEVENTHIST table. PMU data resides in the PSPMTRANSHIST and PSPMTRANSCURR tables. This section presents formulas that you can use to estimate the potential size of your performance database. These formulas incorporate the following assumptions and considerations: Performance Monitor is set to Standard agent filter mode. 48 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

61 Chapter 4 Administering the Performance Monitor Estimates do not include space required for Verbose or Debug agent filter mode. Index overhead is included in the estimate. Performance history data is purged after a number of days (see parameter D in the following section). The archive mode of a monitored system is set to Delete Data after N days and the performance data archive program is scheduled to run daily. The reaper program is scheduled to run at least once a day. No performance data is stored in the archive tables (PSPMTRANSARCH and PSPMEVENTARCH). The calculation formulas use only the parameters that are presented in this section to calculate the estimates. In all cases, the numbers are conservative. For example, the exact formula may use (App 1) but we choose to round up and use App instead. Estimating Space Requirements for Event Data This section discusses two formulas that are used for estimating event data space requirements (in kilobytes). Standard formula: Use the standard formula if the application server domain configuration is based on PeopleSoft-delivered small, medium, or large template. Customized formula: Use the customized formula if the configuration is different from the templates. This table describes the variables that are used in the formulas. Notation Description Performance Monitor Default Value Navigation A Performance Monitor agent event sampling rate. 300 seconds PeopleTools, Performance Monitor, Administration, System Definitions N Number of PeopleSoft systems that are monitored by Performance Monitor. This is the number of PeopleSoft databases appearing on the System Definitions search page. NA PeopleTools, Performance Monitor, Administration, System Definitions D Performance history retention period in days. This is the value that is set for the After N days option. NA PeopleTools, Performance Monitor, Administration, System Definitions W Number of web server domains for a monitored system. This is the total number of web servers appearing on the System Performance page. NA PeopleTools, Performance Monitor, System Monitor, System Performance Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 49

62 Administering the Performance Monitor Chapter 4 Notation Description Performance Monitor Default Value Navigation P Number of application server domains for a monitored system. This is the total number of application servers appearing on the System Performance page. NA PeopleTools, Performance Monitor, System Monitor, System Performance App Number of server processes running in an application server domain for a monitored system. This is the number of program names appearing on for Server Status. Use the following number per domain template that you choose: Large = 60 Medium = 40 PSADMIN, Application Server, Administer a domain, Domain, Domain status menu, Server status Small = 20 S Number of monitored PeopleSoft Process Scheduler domains for a monitored system. This is the total number of Process Scheduler domains appearing on the System Performance page. NA PeopleTools, Performance Monitor, System Monitor, System Performance Prcs Number of server processes running in a PeopleSoft Process Scheduler domain for a monitored system. This is the number of program names appearing for Server status. 8 Increase this number if more than three Application Engine processes are configured. PSADMIN, Process Scheduler, Show Status of a Process Scheduler Server, Domain, Domain status menu, Server status MPrcs Number of Master Scheduler for a monitored system. 1 NA E Number of KB per event row in the table. Refer to the value in the following table for the target database. NA The following table helps you to determine the appropriate value for E (Number of KB per event row in the table). Parameter ANSI/Unicode Oracle Microsoft SQL Server DB2 UDB DB2/390 E ANSI E Unicode Using the Standard Formula The formula that you use differs depending on the template used in the application server configuration. 50 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

63 Chapter 4 Administering the Performance Monitor Large N x D x [8 x W x P + 16 x S + 1] x / A x E Medium Small N x D x [8 x W x P + 16 x S + 1] x / A x E N x D x [8 x W + 60 x P + 16 x S + 1] x / A x E Using the Customized Formula Use this formula if the application server configuration is different from the standard templates. N x D x [8 x W + 3 x P x App + 2 x S x Prcs + MPrcs] x / A x E Note. Eight events are reported per web server domain. Two events are reported per web server (JVM status and network status), one event per web site, and five events per web site for PeopleSoft servlets (psp, psc, cs, *, and Scheduler Transfer). If multiple systems are monitored and each is configured slightly differently, that is, the numbers of application server processes are different, then use the formula to estimate the requirement for each system separately. Estimating Space Requirements for PMU Data The total space requirements, in kilobytes, for PMU data that is stored in PSPMTRANSHIST and PSPMTRANSCURR tables can be estimated using the following formula: N x [D + 1] x L x R x M x T This table describes the variables that are used in this formula. Notation Description Performance Monitor Value Default Navigation N Number of PeopleSoft systems that are monitored by Performance Monitor. This is the number of PeopleSoft databases appearing on the System Definitions search page. NA PeopleTools, Performance Monitor, Administration, System Definitions D Performance history retention period in days. This is the value that is set for the After N days option. NA PeopleTools, Performance Monitor, Administration, System Definitions L Number of user sessions per day for a monitored system. A session means that a user signs on, performs a few transactions, and signs off. NA NA Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 51

64 Administering the Performance Monitor Chapter 4 Notation Description Performance Monitor Value Default Navigation R Number of user interactions per session. User interactions are anything that triggers a server trip, including clicking a button, clicking TAB, and so on. NA NA M Number of PMU rows that are captured per user interaction. 9 NA T Number of KB per PMU row in the table. Refer to the value in the following table for the target database. NA The following table helps you to determine the value for T. Parameter ANSI/Unicode Oracle Microsoft SQL Server DB2 UDB DB2/390 T ANSI T Unicode Calculating Space Requirements This section presents an example of using the formulas to estimate the performance database size for a fictitious organization. Company ABC uses Performance Monitor to monitor two PeopleSoft Enterprise Applications, Financials and HCM (N=2). Both applications use DB2 UDB Unicode databases. The company has decided that the performance history data will be kept for a 7-day period (D=7). Each system has two web server domains (W=2), two application server domains (P=2), and two PeopleSoft Process Scheduler domains (S=2). The implementation team decides to use the medium application server configuration for both domains. One Master Scheduler exists for each of the systems (MPrcs=1). It is estimated, on the average, that 10,000 user sessions (L=10000) will be logged per day in each of the systems. During each session, 50 user interactions (clicking buttons, tab to next field or page, and so on) will occur (R=50). This is the sample calculation for event data space (using the standard formula for a medium configuration): N x D x [8 x W x P + 16 x S + 1] x / A x E = 2 x 7 x [8 x x x 2 + 1] x / 300 x E = 1,165,248 rows x 0.9 KB per row = 1,024 MB This is the sample calculation for PMU data space: N x [D + 1] x L x R x M x T 52 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

65 Chapter 4 Administering the Performance Monitor = 2 x [7 + 1] x x 50 x 9 x T = 72,000,000 rows x 3.3 KB per row = 232,032 MB This is the formula for space requirement for storing performance data on a DB2 UDB Unicode database: 1,024 MB + 232,032 MB = 233,056 MB Company ABC decides to add a 1 TB disk. Business is going well for ABC Company. The demand for the Financial application increased by 50 percent. The IT department decided to add new web server, application server, and PeopleSoft Process Scheduler domains for the Financials application (N=1). According to the system administrator, when the application server domain is booted, a "22 processes started" message appears (App=22), and the PeopleSoft Process Scheduler domain shows a "12 processes started" message (Prcs=12). The IT department needs to estimate whether enough disk space is available to store additional performance data. Use the customized formula to calculate the space requirement for event data that is generated by the new configuration. N x D x [8 x W + 3 x P x App + 2 x S x Prcs + MPrcs] x / A x E = 1 x 7 x [8 x x 1 x x 1 x 12] x / 300 x E = 153,216 rows x 3.3 KB per row = 494 MB System usage increased by 50 percent for the Financials application, so the total space requirement is: 233,056 MB MB + [233,032 MB/2 x 50%] = 291,558 MB The IT department concludes that enough space is available to store the performance data. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 53

66

67 Chapter 5 Working with the Performance Trace This chapter provides an overview of the Performance Trace and discusses how to: Enable a performance trace. Run a performance trace. View performance trace information. Understanding the Performance Trace The Performance Monitor enables you to monitor and record performance information for all activity on PeopleSoft systems. However, times occur when you need to monitor the performance of a specific business process or the performance issues that are reported by a specific user. In these cases, you can use a performance trace. The Performance Trace enables you to: Group PMUs across server requests. Display PMUs from multiple systems. Override default agent filter levels. A user starts and stops a performance trace from the Performance Trace console. While a performance trace is in effect, the system associates the trace name with each PMU that is created during that user's session. The trace name may then be used in the Performance Monitor pages to search for performance data that is created during the performance trace. The trace can override the current agent filter level for PMUs that are created during the trace. The override applies to that user session only. Performance traces are effective across multiple systems that report to the same monitor. For example, if a portal displays content from another monitored system, traces that are started from the portal also apply to the displayed content. Note. The Performance Trace Console needs to be launched from an end user's browser. Note. A performance trace is not affected by the PMU sample rate. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 55

68 Working with the Performance Trace Chapter 5 Enabling a Performance Trace Systems that are configured to allow performance traces have a Performance Trace link appearing in the Universal Navigation Header on each PeopleSoft page. Note. In the following procedure, be sure to notice where each step needs to be completed: on the monitoring system or the monitored system. To enable a performance trace: 1. On the monitoring system, select PeopleTools, Performance Monitor, Administration, System Definitions, and then select the appropriate system definition. 2. On the monitoring system, make sure Allow Performance Trace is selected on the System Definitions page. Note. If a business process includes multiple systems, each system in the business process needs to have the system configured to allow performance traces. 3. On the monitoring system, click Save and Notify Agents. 4. On the monitored system, make sure the users who will run the performance trace have the WEBLIB_PPM web library specified in at least one of their permission lists. Note. If you don't want a user to see the Performance Trace link, you need to remove the web library WEBLIB_PPM from that user's permission lists. Without access to WEBLIB_PPM, the Performance Console link does not appear in the Universal Navigation Header. Running a Performance Trace The Performance Trace Console launches in a small, separate window. 56 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

69 Chapter 5 Working with the Performance Trace Performance Trace Console To run a performance trace: 1. If the Performance Trace link does not appear in the Universal Navigation Header, click Sign out to sign off the system, and then clear the browser's cache (remove temporary Internet files). Note. Clearing browser cache varies depending on which supported browser you use. Refer to your browser's documentation for details. 2. Sign on to the PeopleSoft system. 3. Navigate to the page where you want to begin the performance trace. 4. Click the Performance Trace link, which is located in the Universal Navigation Header. This launches the Performance Trace Console. 5. In the Performance Trace Name edit box, enter a name to identify the trace results. Remember the name that you enter because you need to enter this name in the search criteria when you intend to view the trace results. You can accept the default name, if one appears. 6. In the Filter Level group box, set the filter level options. You have the following options: Keep Current Settings Override With Standard The system continues to record performance information according to the current agent filter settings. Changes the current filter level to Standard for the current user's session. Override With Verbose Changes the current filter level to Verbose for the current user's session. Override With Debug Changes the current filter level to Debug for the current user's session. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 57

70 Working with the Performance Trace Chapter 5 7. When the appropriate filter levels have been set, click Start Trace. 8. Complete the business process that you need to monitor. 9. Click Stop Trace. Viewing Performance Trace Information To view performance trace information: 1. Make note of the performance trace name that was entered by the user. 2. Navigate to the Performance Monitor page that displays the desired performance information. You can use the performance trace name to view. Open PMUs. Completed PMUs. All of the performance charts. 3. In the search criteria for the page that you are on, enter the name of the performance that is trace in the Performance Trace Name edit box. 4. Enter any additional search criteria required. 5. Click Search. 58 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

71 Chapter 6 Monitoring System Performance This chapter provides an overview of system performance monitoring and discusses how to: Use the system performance home page. View web server performance. View application server domain performance. View Process Scheduler server performance. View Master Scheduler server performance. View open PMUs. View open PMU trees. View current user sessions. View recycled processes diagnosis. Understand the events that monitor resource usage. System Performance Monitoring This section contains an overview of monitoring system performance and lists the pages that are used to monitor system performance. Monitoring System Performance The activities that are related to monitoring system performance are primarily for viewing and analyzing the most recent performance data that is received from agents in a monitored system. PeopleSoft provides a collection of use cases in the form of flow charts for you to use as a framework for learning how to use Performance Monitor to detect performance issues. These flow charts do not appear in this PeopleBook; they are posted on My Oracle Support. Refer to the PeopleTools Release Notes for the current location of these flow charts. See Performance Monitor Database Schema and Use Cases on My Oracle Support Note. The information that is presented in the System Performance pages is as current as the last page refresh. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 59

72 Monitoring System Performance Chapter 6 Note. The Standard Deviation (Std. Dev.) is provided on many pages. The standard deviation is a statistic that tells you how tightly all the values that are used to compute the average are clustered around the average. Large standard deviations warn that the averages appearing in the chart are not a reliable indicator of response times that are experienced by individual users. Pages Used to Monitor System Performance Page Name Definition Name Navigation Usage System Performance PSPMSYSHEALTH PeopleTools, Performance Monitor, System Monitor, System Performance Web Server PSPMWEBDOM PeopleTools, Performance Monitor, System Monitor, Web Server Application Server PSPMAPPDOM PeopleTools, Performance Monitor, System Monitor, Application Server Process Scheduler Server PSPMPSCHEDDOM PeopleTools, Performance Monitor, System Monitor, Process Scheduler Server Provides a high-level view of the performance of a monitored system displaying performance indices as well as some critical event and PMU data for the servers in your monitored system. This page acts as a "home page" for monitoring system health. For example, from the System Performance page, you can access numerous related pages, such as the Web Server page, Application Server page, the Process Scheduler page, and so on. Enables you to drill down into the performance data that is related to your web server, such as JVM Status, network status, site performance, and servlet performance. Enables you to drill down into the status of an application server domain and the individual server processes that are running in that domain. Enables you to drill down into the status of a Process Scheduler domain, monitor the individual server processes within a domain, and monitor the resource usage. 60 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

73 Chapter 6 Monitoring System Performance Page Name Definition Name Navigation Usage Master Scheduler PSPMMASTSCHED PeopleTools, Performance Monitor, System Monitor, Master Scheduler Open PMUs PSPMTRANSCURR PeopleTools, Performance Monitor, System Monitor, Open PMUs Open PMU Trees PSPMTRANSUSER PeopleTools, Performance Monitor, System Monitor, Open PMU Trees Current User Sessions PSPMCURUSERS PeopleTools, Performance Monitor, System Health, Current User Sessions Enables you to view the status of the Master Scheduler distributing workload across multiple Process Schedulers. Provides search criteria by which you can search for open PMUs, which are PMUs that have not yet finished. Enables you to search open PMUs by user and display each open PMU in a tree format. Enables you to view the users who are currently signed on to the system as well as view an individual user's history. Recycled Processes Diagnosis PT_PM_RECPROC_DIAG PeopleTools, Performance Monitor, System Health, Recycled Processes Diagnosis Shows detail about application server crashes and last service request type. Using the System Performance Home Page Access the System Performance page (PeopleTools, Performance Monitor, System Monitor, System Performance). The System Performance page provides various health indices as well as a high-level view of the performance of your web servers, application servers, and Process Scheduler server domains. It acts as the "home page" for system performance. Note. A Status field appears at the top of the page only if the monitoring system detects stale data, which is data that is older than the specified event sampling rate. For example, if your event sample rate is 300 seconds, and no events have been received from a specific domain in over five minutes, then that domain is considered stale. Stale data could indicate an outage on the monitored system or that the monitored system's agent filters are set to 01 Standby. If a domain has been shut down permanently and you don't want it to appear with a stale data warning, go to the Agent Definitions page and set its domain monitor to inactive. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 61

74 Monitoring System Performance Chapter 6 Viewing Performance Indices User Sessions Displays the number of users who have signed onto the system within the last 12 hours and have not signed off. Shows all PMU 109s (User Session Began) that do not have an associated 108 PMU (User Session Ended). These PMUs are associated by the session ID (Context 1). The User Sessions value includes only users who have signed on in the last 12 hours and only those users that have signed on since the last time the web server agent sent an Event 901 (Agent Initialization). For details on individual users, click the Current User Sessions link. Note. If users do not click the Sign out link on a PeopleSoft page and instead close the browser or navigate to another site, then the system displays them as current users until the web-server time-out value is reached. Tuxedo Request Queued Displays the number of requests queued on all application servers on the monitored system. The value is the sum of all the metric 5s (Total pq) for Event 300s (Host Resource Status) that are sent by all active agents for the current system that are domain monitor agents of type application server. The value includes all Event 300 data that is generated within the defined sampling interval. PMUs in Past Hour Alarms in Past Hour Batch Jobs in Process Displays the number of PMUs that have run in the past hour. This is an indication of the load on the monitored system, the level of monitoring, and whether the monitored system agents are currently communicating with the monitor. The value is the sum of the rows that are inserted into the PSPMTRANSHIST table within the last hour. For more details, click the Open PMUs and the Completed PMUs link. Displays the number of error and warning events that were posted by the monitored system's active agents in the past hour. This is the count of all events with filter levels of error and warning inserted into the PSPMEVENTHIST table in the last hour. For more details, click the Alarm History link. Displays the number of Process Scheduler jobs that are currently running on all the Process Scheduler servers belonging to that monitored system. The value is the sum of all metric 1s for Event 350 (Master Scheduler Status). For more details, click the Master Scheduler link. Note. If a Master Scheduler is not configured for the monitored system, this value is zero. 62 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

75 Chapter 6 Monitoring System Performance Batch Jobs in Queue Displays the number of Process Scheduler jobs that are currently waiting to be assigned by the Master Scheduler. The value is the sum of all metric 2s for Event 350 (Master Scheduler Status). Note. If a Master Scheduler is not configured for the monitored system, this value is zero. Today's Averages The Today's Averages chart displays the average duration of various performance factors within your PeopleSoft system. The data applies only to the current day, which refers to all data that has been collected since midnight. If progress slows in the system anywhere, this chart enables you to identify which tier is affected and enables you to monitor Tuxedo queueing. Today's Averages chart User Response Displays the average amount of time that end users waited for server round trips to finish. A user is waiting for a server round trip to finish when Processing appears in the upper, right-hand corner of a PeopleSoft page. Note. This average does not include network latency. It is the average duration of all top-level PMU 101s (PIA Request) that the monitoring system has received since midnight. Jolt Request Application Server Displays the average and standard deviation of all PMU 115s (Jolt Request) received since midnight. Displays the average and standard deviation of the duration of all PMU 400s (Tuxedo Service PeopleCode and SQL Summary) received since midnight. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 63

76 Monitoring System Performance Chapter 6 SQL Displays the average and standard deviation for the time spent running SQL statements that were initiated from PeopleCode in a single application server request. This value includes the time that was required for SQL Prepare, Execute, Fetch, and Commit. PeopleCode SQL statements are run by application code that is written using PeopleCode and submitted to the database by way of SQLEXECs or SQL Objects. This value is derived from metrics 2, 3, and 4 from PMU 400 (Tuxedo Service PeopleCode and SQL Summary). Namely, this value includes PeopleCode SQL, PeopleCode Built-in SQL, and PeopleTools SQL. Note. This is the average time spent for running all SQL statements of these types in a single application server request. The system calculates this value using all data that has been received since midnight. Web Server Domains The Web Server Domains grid contains a row for each active domain-monitor web-server agent. The metric information is derived from Event 150 (JVM status) and Event 151 (Network Status). The system retrieves only the most recent event for each agent. If the most recent event's monitor date and time is older than the system's event sample rate, the Stale Data icon appears in the grid row and at the top of the page. Note. Web server agents do not report performance data until the first user connects to that web server. Name Host/Port Filter Level Displays the PeopleSoft web-server domain name that is being monitored. Click the name to go to the Web Server page for that web server. It's the same page that you access when you select PeopleTools, Performance Monitor, System Monitor, Web Server and choose that domain. Identifies the web-server domain host name and listening port numbers for HTTP and HTTPS. Appears on pages displaying current performance information that is related to servers, such as application servers, web servers, and so on. Displays the current agent filter level. The colors indicate the following filter levels: Blue: Standby, Error, and Warning. Green: Standard. Orange: Verbose. Red: Debug. Move the cursor over the icon to show a pop-up message displaying the agent filter level in text format. Sessions in Web App Displays the number of servlet sessions in the portal web application. 64 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

77 Chapter 6 Monitoring System Performance %JVM Memory Free Execute Threads Established Sockets Jolt Traffic A Java Virtual Machine (JVM) has a specific amount of memory allocated to it. This metric displays the percentage of the allocated memory that is available. The number of threads that are used by the web application to service incoming requests. The number of sockets in a connected state. This is a measure of the number of active users. Displays the per second amount (in bytes) of Jolt traffic between the web server and the application server. Application Server Domains/Process Scheduler Server Domains The application server and Process Scheduler Server grids contain a row for each active domain monitor agent that is running on the application server or Process Scheduler server. The metric information on each row is derived from Event 300 (Host Resource Status). The system retrieves only the most recent event for each agent. If the most recent event's monitor date or time is older than the system's event sample rate, the Stale Data icon appears in the grid row and at the top of the page. Note. When first booted, an active server may have a stale data warning until a full sampling interval has passed. Domain Name Host/Port Filter Level Displays the names of the domains that are being monitored. Click the domain name to access the Application Server or the Process Scheduler page for additional details on a particular domain. It's the same page that you access when you select PeopleTools, Performance Monitor, System Monitor, Application Server or Process Scheduler and choose that domain. Displays the server name or IP address and the Jolt listening port number. Appears on pages displaying current performance information that is related to servers, such as application servers, web servers, and so on. Displays the current agent filter level. The colors indicate the following levels: Blue: Standby, Error, and Warning. Green: Standard. Orange: Verbose. Red: Debug. Move the cursor over the icon to show a pop-up message displaying the agent filter level in text format. %CPU Used Displays the percentage of the CPU capacity that is being used on the host. This percentage includes the entire processing load on the host, not just that of PeopleSoft. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 65

78 Monitoring System Performance Chapter 6 %Memory Used Hard Page Faults/Second Total Tuxedo Connections Total Tuxedo Requests Queued Displays the percentage of physical memory that is used on the host. This percentage includes all memory used on the host, not just the memory used by PeopleSoft. This is the number of accesses to virtual memory in the last second that require disk reads on the host. This metric applies to all memory access on the host, not just the memory that is accessed by PeopleSoft. This is the number of connections to this domain, and it is equivalent to the number of rows that are returned by the Tuxedo command pclt (tmadmin), excluding JSH and WSH connections. This value does not apply to Process Scheduler. The number of Tuxedo requests that are queued on that domain. A high number indicates that insufficient server processes are configured, or that the load on the host machine is too high. This value does not apply to Process Scheduler. Ping Test PeopleSoft Ping is a diagnostic feature that enables you to troubleshoot systemic performance issues. If you enter the ping URLs for the monitored systems in the URL catalog, you can launch the ping test from the monitoring system instead of having to sign on to the monitored system. The following requirements must be in place: Single signon must be configured between the monitored and monitoring system. Current user ID must be a valid user ID in both the monitored and monitoring system. Current user ID must have permission to access the PSPing page in the monitored system. Any system that you intend to ping must have an entry in the URL catalog for the PSPing page. The URL must contain PTPERF_TEST for the URL to appear in the ping lists that are associated with the Performance Monitor. For example, Note. In a self-monitoring system, you must append the text "_newwin" to the end of the site name on which you are running the ping test. The "_newwin" indicates to the PeopleSoft system that the content can run in a new window at the same time that content from a different component runs in the previous window. URL Identifier Execute PSPing Select the URL of the system that you want toping. Click to launch the PeopleSoft Ping page at the URL appearing in the URL Identifier field. The page appears in a new window. If that system is monitored, when you run Ping the system reports the results that are displayed on the Ping page to the monitoring system in the form of Event 600s (PSPING). 66 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

79 Chapter 6 Monitoring System Performance View PSPing History Launches the Event History search page displaying all Event 600 (PSPING) data for the current day on the current system. Verify that you ping the appropriate system. PeopleSoft runs no internal verification to verify that the system you ping is the system which this page is monitoring. Viewing Web Server Performance Access the Web Server page (PeopleTools, Performance Monitor, System Monitor, Web Server). The Web Server page displays the most recent performance data that is received from a web server domain monitor. A domain corresponds to the domain that is specified during the PeopleSoft installation. Domain Name Domain Directory Filter Level A domain monitor starts when the first user connects to any site in a PeopleSoft web server domain. The domain name is the name of this site. The directory on the web server where the PeopleSoft site that triggered the domain monitor resides. Appears on pages displaying current performance information that is related to servers, such as application servers, web servers, and so on. Displays the current agent filter level. The colors indicate the following levels: Blue: Standby, Error, and Warning. Green: Standard. Orange: Verbose. Red: Debug. Move the cursor over the icon to show a pop-up message displaying the agent filter level in text format. Agent Date/Time The date and time according to the agent clock that the domain monitor used when it last sent performance data to the monitor. Note. The date and the time appearing on this page always apply to the system on which the agent runs. Keep this in mind if you are monitoring systems in other time zones. Status Host/Port Appears only if the system detects stale data. Performance data is stale if no status events have been reported in the last sample interval for the system. For example, Status shows stale data if the web server is not booted. The host of the web server and the listening ports for HTTP and HTTPS. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 67

80 Monitoring System Performance Chapter 6 Monitor Date/Time Last Page Refresh Web Server Snapshot The date and time that the monitoring system inserted the row of performance data into the monitoring database according to the database clock. Indicates the most recent time that the system refreshed the page either by loading the page into the browser or as a result of a user clicking the Refresh button. This link enables you to view historical information snapshots for the current agent. By clicking this link, you see a list of all the domain agent timestamps for the current domain monitor agent Event 150 (JVM Status). After you select a particular timestamp, the system displays the Web Server page containing web server performance information for that particular time. When you are viewing snapshot information, a message appears at the top of the page reminding you that the page contains historical performance data. You can view numerous snapshots using the same method. Note. While viewing snapshots, the Refresh button is not available. To view the most current web server information, you must access the Web Server page using the menu. System Performance This link accesses the System Performance page (the home page for system monitoring). JVM Status The JVM status applies to the status of the JVM in which the site runs. Multiple web sites within the same PeopleSoft domain can run within the same JVM. These values are derived from Event 150 (JVM Status). If the latest Event 150 monitor date and time is older than the event sample rate for the current agent's system, the Stale Data icon appears at the top of the page. %JVM Memory Free Max JVM Memory Available Sessions in Web-App Execute Threads Busy Threads Domain Count A JVM has a specific amount of memory allocated to it. This metric displays the percentage of the allocated memory that is available. Indicates the maximum amount of memory that is available for the JVM. This value depends on the Java Runtime Environment (JRE). The maximum JVM memory that is available equals maxmemory totalmemory + freememory. Displays the number of servlet sessions in the portal web application. The number of threads that are used by the web application to service incoming requests. The number of threads that are currently servicing requests. The number of domain monitors in this web server domain. The domain count will only be greater than 1 if a web server domain has sites belonging to different monitored systems. 68 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

81 Chapter 6 Monitoring System Performance The Events and PMUs tab contains links to the Event History, Open PMUs, and Completed PMUs components. When transferring to the Event History component, the system automatically displays all of the data for the current day's Event 150 (JVM Status) rows for the current agent and the current system. When transferring to the Open PMUs component, the system automatically displays all of the data for the current day's open PMUs for the current host/port and the current system. When transferring to the Completed PMUs component, the system automatically displays all of the data for the current day's completed PMUs for the current host/port and the current system. Network Status Network status data corresponds to Output from the domain monitor running the netstat a command in a shell on the monitored host. Counters that are maintained by the domain monitor. These values are derived from Event 151 (Network Status). If no Event 151 or the Event 151 has an agent date/time that does not match the agent date/time of the Event 150 (JVM Status) shown at the top portion of the page, this grid appears empty. Time Wait Sockets Close Wait Sockets Established Sockets Jolt Traffic The number of sockets in a TCP time wait state on the host on which the web server is running. A high count may mean that the time wait setting for the operating system of the server needs to be decreased, or it may just mean that a very high load is on that server. The number of sockets in a TCP close wait state on the host on which the web server is running. A high count means that TCP clients are not closing connections and may indicate network or software configuration issues. The number of sockets in a connected state. This is a measure of the number of users who are connected. Displays in bytes per second the amount of Jolt traffic between the web server and the application server. Note. The Jolt traffic value is zero unless compression is enabled. HTTP Indicates the HTTP traffic in bytes per second that is generated by this web server. The Events and PMUs tab contains links to the associated Event History, Open PMUs, and Completed PMUs. When transferring to the Event History component, the system automatically displays all of the data for the current day's Event 151 (Network Status) rows for the current agent and the current system. When transferring to the Open PMUs component, the system automatically displays all of the data for the current day's open PMUs for the current host/port and the current system. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 69

82 Monitoring System Performance Chapter 6 When transferring to the Completed PMUs component, the system automatically displays all of the data for the current day's completed PMUs for the current host/port and the current system. Sites The information on this grid applies to all sites on this web server even if they are not monitored sites. These values are derived from Event 152 (Web Site Status). If no Event 152 or the Event 152 rows have an agent date/time that does not match the agent date/time of the Event 150 (JVM Status), the grid appears empty. Site Path Requests to all Servlets Servlet Requests (last minute) Avg. Request Time (last minute) Time in all Servlets Current Sessions Indicates the directory on the web server where the site resides. Indicates the number of requests that have been submitted to all of the servlets running within the site. Indicates the number of servlet requests that were submitted to the site in the last minute. Indicates the average duration of the servlet requests that were submitted to the site in the last minute. Indicates the sum of durations of the servlet requests that were submitted to the site since the web server was last booted. The number of sessions that are active within the site. Active sessions are those that are currently waiting for a request to be processed. In most cases, this value is 0. However, if your site is running slowly or a transaction has a long duration, this value is greater than 0. The Events and PMUs tab contains links to the associated Event History, Open PMUs, and Completed PMUs. When transferring to the Event History component, the system automatically displays all of the data for the current day's Event 152 (Web Site Status) rows for the current agent and the current system. When transferring to the Open PMUs component, the system automatically displays all of the data for the current day's open PMUs for the current agent and the current system. When transferring to the Completed PMUs component, the system automatically displays all of the data for the current day's completed PMUs for the agent and the current system. Active Servlets Active servlets are all servlets that are running in the portal web application. These values are derived from Event 153 (Web Servlet Status). Servlet Name Requests to this Servlet Identifies the name of a particular servlet. Indicates the number of requests that were submitted to a particular servlet. 70 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

83 Chapter 6 Monitoring System Performance Servlet Requests (last minute) Avg. Request Time (last minute) Time in this Servlet Indicates the number of requests that were submitted to a particular servlet within the last minute. The average duration of the requests that were handled by the servlet in the last minute. The sum of durations of requests that were handled by this servlet since the web server was last booted. Viewing Application Server Domain Performance Access the Application Server Domain page (PeopleTools, Performance Monitor, System Monitor, Application Server). The Application Server Domain page displays the most recent performance data that was received from: An application server domain monitor. Each active application server agent within that application server domain. Domain Name Domain Directory Agent Date/Time Identifies the name of the current Tuxedo domain. Identifies the directory in which the domain is installed. The date and time according to the agent clock that the domain monitor used when it last sent performance data to the monitor. Note. The date and the time appearing on this page always applies to the system on which the agent runs. Keep this in mind if you are monitoring systems in other time zones. Status Host/Port Appears only if the system detects stale data. Data is stale if no status events have been reported in the last sample interval for the system. For example, Status shows stale data if the application server is not booted. Identifies the machine name and Jolt listening port of the application server domain. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 71

84 Monitoring System Performance Chapter 6 Application Server Snapshots This link enables you to view historical information snapshots for the current agent. By clicking this link, you see a list of all the domain agent timestamps for the current agent's Event 300 (Host Resource Status). After you select a particular timestamp, the system displays the Application Server page containing performance information for that particular time. When you are viewing snapshot information, a message appears at the top of the page reminding you that the page contains historical performance data. You can view numerous snapshots using the same method. Note. While viewing snapshots, the Refresh button is not available. To view the most current information, you must access the Application Server page using the menu. Monitor Date/Time Last Page Refresh The date and time that the monitoring system inserted the row of performance data into the monitoring database according to the database clock. Indicates the most recent time that the system refreshed the page either by loading the page into the browser or as a result of a user clicking the Refresh button. See Chapter 6, "Monitoring System Performance," Monitoring Resource Usage, page 86. Host Resource Status The Metrics tab presents the same values that also appear on the System Performance page. The metrics show current resource utilization on the entire host machine, not just the resources that PeopleSoft uses. The system derives these metrics from the most recent Event 300 (Host Resource Status) for the domain monitor process of that domain. If the latest Event 300 monitor date or time is older than the event sample rate for the current agent's system, the Stale Data icon appears at the top of the page. %CPU Used %Memory Used Hard Page Faults/Second Total Tuxedo Connections Total Tuxedo Requests Queued Displays the percentage of the CPU capacity being utilized on the host machine. Displays the percentage of physical memory being utilized on the host machine. This is the number of accesses to virtual memory in the last second that require disk reads. This is the number of connections to this domain, and it is equivalent to the number of rows that are returned by the Tuxedo command pclt (tmadmin), excluding JSH and WSH connections. The number of Tuxedo requests that are queued on that demand. A high number indicates that insufficient server processes are configured, or that the load on the host machine is too high. The Events and PMUs tab presents links to the Event History, Open PMUs, and Completed PMUs pages. 72 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

85 Chapter 6 Monitoring System Performance When transferring to the Event History component, the system displays all of the data for the current day's Event 300 (Host Resource Status) rows for the current agent and the current system. When transferring to the Open PMUs component, the system displays all of the data for the current day's open PMUs for the current host and port and the current system. When transferring to the Completed PMUs component, the system displays all of the data for the current day's completed PMUs for the current host and port and the current system. PQ Event Rows This section presents the results of the tmadmin pq command (print queue). It enables you to view information that is related to the Tuxedo queues that are used by the server processes running within a domain. The grid displays all Event 301 (Tuxedo "pq" Row) rows for the current agent with the same agent date or time appearing in the top portion of the page as the Event 300 (Host Resource Status). Queue Name Server Name Server Count Queue Length Identifies the Tuxedo queue that is associated with a server process. Indicates the server process that is servicing a particular queue. Indicates the number of a particular server process type that are currently running. For example, it indicates that three PSAPPSRV server processes are currently running. Indicates the current length of the queue, which is measured by the number of requests waiting to be processed. High queue lengths may indicate that more server processes need to be configured to run. PSR Event Rows This section presents the results of the tmadmin psr command (print server processes). This command enables you to view information that is related to all the server processes (monitored and unmonitored) running within a domain. The grid displays all Event 302 (Tuxedo "psr" Row) rows for the current agent with the same agent date or time appearing at the top of the page as the Event 300 (Host Resource Status). Server Name Server Instance ID PID Total Requests Indicates the name of the server process, such as PSAPPSRV, PSMONITORSRV, and so on. This is the instance ID that is assigned by Tuxedo to each server process. This number remains constant across recycles. Even if the PID changes, the instance ID remains constant. Indicates the operating system process ID on the server. Indicates the total number of requests that a server process has processed. Tuxedo continues to increment this number for a server instance even if the server recycles. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 73

86 Monitoring System Performance Chapter 6 Current Service Indicates whether the server process is idle or currently handling a request. Monitored Servers This section enables you to view information about the resources that are consumed by the monitored server processes running within a domain. The grid displays an entry for each active application server agent within the same system and same host and port, and domain directory as the domain monitor agent. For each agent, the system retrieves the latest Event 200 (Resources Per Process) and its metrics are displayed. If no Event 200 exists for a particular agent, the system displays zeros. The Metrics tab contains the following information. Server Server Instance PID Agent Date/Time Identifies the server process being monitored. This is the instance ID that is assigned by Tuxedo to each server process. This number remains constant across recycles. Even if the PID changes, the instance ID remains constant. The process ID that is assigned by the server operating system. The date and time according to the agent clock that the domain monitor used when it last sent performance data to the monitor. Note. The date and the time appearing on this page always applies to the system on which the agent runs. Keep this in mind if you are monitoring systems in other time zones. %CPU Used CPU Time VM Indicates the percentage of the CPU capacity that a particular server process is using. The amount of CPU time that the process has consumed. Indicates the amount of virtual memory that each server process is consuming. The Events and PMUs tab contains links to the Event History, Open PMUs, and Completed PMUs pages. When transferring to the Event History component, the system displays all data for the current day's Event 200 (Resources Per Process) rows for the current agent and the current system. When transferring to the Open PMUs component, the system displays all data for the current day's open PMUs for the current agent and the current system. When transferring to the Completed PMUs component, the system displays all data for the current day's completed PMUs for the agent and the current system. 74 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

87 Chapter 6 Monitoring System Performance Viewing Analytic Server Information The Analytic Server Summary section in the monitored Application Server Domain page provides generic performance and status information of the analytic Servers running in the application server domain. Total Configured Servers The current maximum number of configured analytic servers in the application server domain. Number Started Servers The current number of analytic servers, started up and loaded with analytic instances. Number Free Servers Number Loaded Instances Today Number Recycled Servers Number Out Servers Today The current number of analytic servers, started up and free. Today's total number of analytic instances loaded. Today's total number of analytic servers recycled for reuse. Today's total number of times that the monitored domain runs out of analytic servers. Each time a client asks to load an analytic instance, a free analytic server gets assigned and dedicated to the analytic instance. Viewing Process Scheduler Server Performance Access the Process Scheduler Server page (PeopleTools, Performance Monitor, System Monitor, Process Scheduler). The Process Scheduler Server page displays the most recent performance data that is received from: A Process Scheduler server domain agent. Each active Process Scheduler server agent within this Process Scheduler domain. Domain Name Domain Directory Agent Date/Time Identifies the name of the current domain. Identifies the directory in which the domain is installed. The date and time according to the agent clock that the domain monitor used when it last sent performance data to the monitor. Note. The date and the time appearing on this page always applies to the system on which the agent runs. Keep this in mind if you are monitoring systems in other time zones. Status Appears only if the system detects stale data. Data is stale if no status events have been reported in the last sample interval for the system. For example, Status shows stale data if the Process Scheduler is not booted. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 75

88 Monitoring System Performance Chapter 6 Host/Port Process Scheduler Snapshots Identifies the machine name of the Process Scheduler server domain. This link enables you to view historical information snapshots for the current agent. By clicking this link, you see a list of all the domain monitor timestamps for the current agent's Event 300 (Host Resource Status). After selecting a particular timestamp, the system displays the Process Scheduler Server page containing performance information for that particular time. When you are viewing snapshot information, a message appears at the top of the page reminding you that the page contains historical performance data. You can view numerous snapshots using the same method. Note. While you are viewing snapshots, the Refresh button is not available. To view the most current information, you must access the Process Scheduler Server page using the menu. Monitor Date/Time Last Page Refresh The date and time that the monitoring system inserted the row of performance data into the monitoring database according to the database clock. Indicates the most recent time that the system refreshed the page either by loading the page into the browser or as a result of a user clicking the Refresh button. See Chapter 6, "Monitoring System Performance," Monitoring Resource Usage, page 86. Host Resource Status The Metrics tab presents the values that also appear on the System Performance page. If the latest Event 300 (Host Resource Status) monitor date and time is older than the event sample rate for the current agent's system, the Stale Data icon appears at the top of the page. %CPU Used %Memory Used Hard Page Faults/Second Total Tuxedo Connections Total Tuxedo Requests Queued Displays the percentage of the Central Processing Unit capacity being utilized on the host machine. Displays the percentage of physical memory being utilized on the host machine. The number of accesses to virtual memory in the last second that require disk reads. This number is always zero for a Process Scheduler domain. The number of Tuxedo requests that are queued on that demand. A high number indicates that insufficient server processes are configured, or that the load on the host machine is too high. The Events and PMUs tab presents links to the Event History, Open PMUs, and Completed PMUs pages. When transferring to the Event History component, the system displays all of the data for the current day's Event 300 (Host Resource Status) rows for the current agent and the current system. 76 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

89 Chapter 6 Monitoring System Performance When transferring to the Open PMUs component, the system displays all of the data for the current day's open PMUs for the current host or port and the current system. When transferring to the Completed PMUs component, the system displays all of the data for the current day's completed PMUs for the current host port and the current system. PQ Event Rows This section presents the results of the tmadmin pq command (print queue). It enables you to view information that is related to the queues that are used by the server processes running within a domain. The performance data displays all Event 301 (Tuxedo "pq" Row) rows for the current agent with the same agent date and time appearing in the top portion of the page as the Event 300 (Host Resource Status). Queue Name Server Name Server Count Queue Length Identifies the Tuxedo queue that is associated with a server process. Indicates the server process that is servicing a particular queue. Indicates the number of a particular server process type that are currently running. For example, it indicates that three PSAESRV server processes are currently running. Indicates the current length of the queue, which is measured by the number of requests that are waiting to be processed. PSR Event Rows This section presents the results of the tmadmin psr command (print server processes). This enables you to view information that is related to all the server processes (monitored and unmonitored) running within a domain. The grid displays all Event 302 (Tuxedo "psr" Row) rows for the current agent with the same agent date and time appearing at the top of the page as the Event 300 (Host Resource Status). Server Name Server Instance ID PID Total Requests Current Service Indicates the name of the server process, such as PSAESRV, PSMONITORSRV, and so on. This is the instance ID that is assigned by Tuxedo to each server process. This number remains constant across recycles. Even if the PID changes, the instance ID remains constant. Indicates the operating system process ID on the server. Indicates the total number of requests that a server process has processed. Indicates whether the server process is idle or currently handling a request. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 77

90 Monitoring System Performance Chapter 6 Monitored Servers This section enables you to view information about the resources that are consumed by the monitored server processes running within a domain. Note. Currently, only PSMONITORSRV and PSMSTPRC are monitored. The grid displays an entry for each active application server agent within the same system and same host or port and domain directory as the domain monitor agent. For each agent, the system retrieves the latest Event 200 (Resources Per Process) and its metrics are displayed. If no Event 200 exists for a particular agent, the system displays zeros. The Metrics tab contains the following information. Server Server Instance PID Agent Date/Time Identifies the server process that is being monitored. This is the instance ID that is assigned by Tuxedo to each server process. This number remains constant across recycles. Even if the PID changes, the instance ID remains constant. The process ID that is assigned by the server operating system. The date and time according to the agent clock that the domain monitor used when it last sent performance data to the monitor. Note. The date and the time appearing on this page always apply to the system on which the agent runs. Keep this in mind if you are monitoring systems in other time zones. %CPU Used CPU Time VM Indicates the percentage of the CPU capacity that a particular server process is using. The amount of CPU time that the process has consumed. Indicates the amount of virtual memory that each server process is consuming. The Events and PMUs tab contains links to the Event History, Open PMUs, and Completed PMUs pages. When transferring to the Event History component, the system displays all data for the current day's Event 200 (Resources Per Process) rows for the current agent and the current system. When transferring to the Open PMUs component, the system displays all data for the current day's open PMUs for the current agent and the current system. When transferring to the Completed PMUs component, the system displays all data for the current day's completed PMUs for the agent and the current system. 78 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

91 Chapter 6 Monitoring System Performance Viewing Master Scheduler Performance Access the Master Scheduler page (PeopleTools, Performance Monitor, System Monitor, Master Scheduler). This page displays the most recent performance data received from a Master Scheduler agent. Note. The Performance Monitor displays Master Scheduler performance data only if a Master Scheduler has been configured. System ID Database Name Agent Date/Time Identifies each monitored system. The PeopleSoft system automatically generates this value incrementally. System definitions are created automatically when the first agent of a monitored system registers with the monitoring system. Reveals the name of the PeopleSoft application database that is running on the monitored system. The monitoring system automatically inserts this value when it recognizes and creates a monitored system. The date and time according to the agent clock that the domain monitor used when it last sent performance data to the monitor. Note. The date and the time appearing on this page always apply to the system on which the agent runs. Keep this in mind if you are monitoring systems in other time zones. Status Monitor Date/Time Last Page Refresh Appears only if the system detects stale data. Data is stale if it is older than the sample rate for that system. Data is stale if no status events have been reported in the last sample interval for the system. For example, Status shows stale data if the Process Scheduler is not booted. The date and time that the monitoring system inserted the row of performance data into the monitoring database according to the database clock Indicates the most recent time that the system refreshed the page either by loading the page into the browser or as a result of a user clicking the Refresh button. Process Summary This process displays the status of the processes that are being managed by the Master Scheduler running on the selected system. The data in this grid is derived from Event 350 (Master Scheduler Status). Active Processes The number of Process Scheduler jobs that are currently processing in this monitored system. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 79

92 Monitoring System Performance Chapter 6 Unused Process Slots Blocked Processes The number of available slots that this Master Scheduler can use to run process requests. A process can be blocked for two reasons: The process is "mutually exclusive" with another process, and the other process is currently running. The process is dependent on a file that another process has yet to create. Queued Processes The number of Process Scheduler jobs that are currently waiting to be handled by an available Process Scheduler server. Servers The Servers grid contains performance metrics for individual Process Scheduler servers that are running under the Master Scheduler. This grid displays all Event 351 (Master Scheduler Detail) rows for all active Process Scheduler servers under the current system with the same agent date and time as the latest Event 350 (Master Scheduler Status). The system groups the data by server name and the metrics are accumulated. Note. If the monitor date and time for an Event 351 is older than the current system's event sample rate, a Stale Data icon appears at the top of the page. Server Name Active Processes The name of the Process Scheduler server, such as PSNT, PSUNIX, and so on. The server name is a hyperlink that launches a Server Activity secondary page for that particular server type. The Server Activity page displays all of the individual Event 351 (Master Scheduler Detail) rows. Use the Refresh button on this page to refresh the Processes grid. The number of jobs that are currently being handled by a Process Scheduler server. Queue Details The Queue grid displays the status of the processes that are queued to be processed by the Master Scheduler. The data is generated from Event 354 (Batch Queue Details) rows for all active Process Scheduler agents in the current system with the same agent date and time as the latest Event 350 (Master Scheduler Status). Process Type Queued Processes Displays the type of process that are queued, such as SQR, Application Engine, COBOL, and so on. Displays the number of processes that are queued per process type. 80 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

93 Chapter 6 Monitoring System Performance Blocked Processes A process can be blocked for two reasons: The process is "mutually exclusive" with another process, and the other process is currently running. The process is dependent on a file that another process has yet to create. Viewing Open PMUs Access the Open PMUs page (PeopleTools, Performance Monitor, System Monitor, Open PMUs). An open PMU is a PMU that started but has not finished prior to an agent reporting performance metrics to the monitor. Information regarding open PMUs are stored in the PSPMTRANSCURR table. When the PMU finishes, the PSPPMSRV flags the PMU for deletion and inserts a matching row into the PSPMTRANSHIST table, which stores completed PMU data. See Chapter 4, "Administering the Performance Monitor," Scheduling the Reaper Program, page 38. If an end user is reporting that a PeopleSoft page is slow (displays the "Processing..." message), use the Open PMU page to see where the request has stalled. Search for that user's name to see the current state of that user's open PMUs. Before you view current PMU details, you need to enter search criteria. If you don't specify criteria, the page displays all of the open PMUs for the selected system. User ID Performance Trace Name Component If you want information per user, enter the user ID of the user for which you want to track current PMUs. The user ID prompts against the PSPMOPRDEFN table, which is populated by the Lookup Application Engine program. You enter the performance trace name to search on currently open PMUs running within a particular trace. If a user has launched a performance trace in the Performance Console, the system labels every PMU that is generated in a business process with a performance trace name. The performance trace name prompts against the PSPMPERFTRACE table, which is populated by the Lookup Application Engine program. If you want information per component, enter the name of the component for which you want to track completed PMUs. The component prompts against the PSPMPNLGRPDEFN table, which is populated by the Lookup Application Engine program. Note. Searching on component is equivalent to searching on context 1 for application server PMUs. Market Select the market that is associated with the component. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 81

94 Monitoring System Performance Chapter 6 PMU Set PMU ID Context 1, 2, 3 Top Instance Instance Identifier Duration >=/< Domain Name Domain Host/Port Agent ID Indicates the set to which the PMU definition belongs. A PMU definition set is similar to a message set. To view the complete set of PMU set definitions, select Performance Monitor, Utilities, PMU Definitions. See Chapter 10, "Performance Monitor Meta-Data," PMU Definitions, page 149. Identifies the PMU definition within a PMU definition set. To view the complete set of PMU definitions, select Performance Monitor, Utilities, PMU Definitions. PMU metrics contain data that is specific to that PMU. Context values, on the other hand, are common to the entire user request or a specific tier. For example, the component name is stored in a context for all PMU's that are generated by an application server while it is processing that component. The system uses contexts to "flatten" a PMU tree. For example, you do not have to navigate up from a SQL PMU to an ICPanel PMU to see what component generated that SQL statement. In some cases, a parent PMU determines the usage of a child PMU's context fields. In such cases, the context label for the child PMU type is Generic. Use the Context Help button to view the context definitions for a PMU set ID and ID that you've entered. See Chapter 10, "Performance Monitor Meta-Data," Context Definitions, page 149. Each PMU that is reported by an agent has a unique instance ID. The instance identifier of the first PMU generated by a particular user request is the top instance. The value of the top instance is then stored with every child PMU within a request. The top instance is the associating value for all PMUs that are generated by the same user request. Use this search field when you already know the top instance of the PMU in which you are interested. Each PMU reported by an agent has a unique instance ID. Specify criteria that is related to the duration of a PMU. The duration is the difference between the current monitor database time and the date and time that the monitor received the open PMU. The system searches for PMUs with a duration greater than or equal to the value that you enter, or less than the value that you enter. The value is in seconds. Select the application server, web server, or Process Scheduler domain of the agent process that is reporting the PMUs. Select the host name and port number of the application server, web server, or Process Scheduler domain from which the PMUs were reported. Select the agent that reported the PMUs. 82 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

95 Chapter 6 Monitoring System Performance Agent Type Select agent type, such as PSAPPSRV, PSQRYSRV, or PSQCKSRV, that reported the PMUs. Note. Some of the metrics for various PMUs don't have values until the PMU finishes, so in some cases, metrics may appear with no values. Open PMUs The following information appears on the Summary tab. PMU Duration PMU Details User ID Action Monitor Received Date/Time Agent Start Date/Time Displays the label for that PMU type. Displays the duration of the PMU. The string that is generated by concatenating all of the contexts and metrics that are selected for display in the PMU definition. Identifies the user whose request generated the PMU. The action could be Start or Update. Every PMU has a start, which refers to a request initiating the PMU. The update action applies to long running PMUs, which the Performance Monitor updates periodically with the latest metric values. Indicates when the monitor system received notification that a PMU had been started or updated. Indicates that time according to the agent clock that the last update was generated. The Identifiers tab presents the following additional information. PMU Set PMU ID Agent ID Domain Name Monitor Last Update Date/Time Indicates the set to which the PMU definition belongs. A PMU definition set is similar to a message set. To view the complete set of PMU set definitions, select PeopleTools, Performance Monitor, Administration, Meta-data, PMU Definitions. Identifies the PMU definition within a PMU definition set. See Chapter 10, "Performance Monitor Meta-Data," PMU Definitions, page 149. Identifies the agent that reported the PMU information. The domain from which the PMU was reported. The monitor date and time of the most recent event of a specific event type that was received from a specific agent. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 83

96 Monitoring System Performance Chapter 6 Agent Last Update Date/Time PID Instance Parent Instance Top Instance Indicates that time according to the agent clock that the last update was received. The operating system process ID of the process that generated the PMU. PMU instance ID. Each PMU has a unique instance ID. The instance ID of the parent PMU. Each PMU that is reported by an agent has a unique instance ID. The instance identifier of the first PMU that is generated by a particular user request is the top instance. The value of the top instance is then stored with every child PMU within a request. The top instance is the associating value for all PMUs that are generated by the same user request. The Contexts tab displays the context information that is associated with a particular PMU. The Metrics tab displays the metric information that is associated with a particular PMU. Viewing Open PMU Trees Access the Open PMU Trees page (PeopleTools, Performance Monitor, System Monitor, Open PMU Trees). Every user request generates a set of PMUs that you can display as a tree. The PMU with the top instance ID is the root of the tree and PMUs with no children are the leaves. A node represents each PMU or child PMU. Clicking a node reveals its details. The shaded text flags the node of the currently processing PMU. A search may return multiple PMU trees. For example, if a user ID is shared across multiple users, or a single user has initiated multiple browser sessions, then more than one user request may be currently processing, and therefore more than one PMU tree open. The data that is related to open PMUs is dynamic because the system is currently processing the PMU. Therefore, the composition of trees, and even their presence, is likely to change each time you click Refresh. If you have an open PMU tree that was captured using the standard agent filter mode, you can drill down to the detail of PMU 400 (Tuxedo Service PCode and SQL). The Current SQL statement shows what has been submitted to the application server. This can aid in troubleshooting long-running SQL or a hung query without increasing the agent filter and asking users to repeat the process. If the open PMU finishes, then the PMU row is marked for deletion by the reaper program. You can't access the current SQL statement if the PMU has finished. Note. So that the presentation of PMU information is readable and manageable, if a PMU is older than a day, the system displays the date that the PMU ran, not its duration in milliseconds. Note. For each PMU in the tree, the duration value represents the period of time that has elapsed since the monitoring system received the start timestamp for each PMU. In some cases, the display may indicate that a child PMU is "older" than its parent. For example, if the application server sent PMU information before the web server sent PMU information, the child PMUs that are running on the application server will display a smaller duration than the associated parent PMUs that are running on the web server. 84 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

97 Chapter 6 Monitoring System Performance User ID Total Trees received Currently displaying Enter the user ID of the user who is initiating the PMUs. You must specify a user ID to view open PMU trees. The number of currently open PMU trees for the user. Corresponds to the number of open requests for this user. Indicates the PMU tree that is currently displayed on the page, providing orientation in a list of returned PMU trees. Previous Tree/Next Tree Enables you to navigate within a list of PMU trees. Open PMU Tree You can expand or collapse PMU trees by using the folder icon to the left. Click a node to view details regarding that PMU. Use the Left and Right links to navigate within a single tree when the tree is 20 nodes deep. Viewing Current User Sessions Access the Current User Sessions page (PeopleTools, Performance Monitor, System Monitor, Current User Sessions). A row appears for every PMU 109 (user session began) that was received in the last 12 hours for which a matching PMU 108 (user session ended) has not been received. Note. If you recycle the web server during this time, the user session is considered closed. The Session tab contains the following information. User ID Session ID Monitor Received Date/Time Identifies the user. Indicates the open session on the web server that is associated with the user. The date and time that the monitoring system inserted the row of performance data into the monitoring database. User History Accesses the User History page. See Chapter 7, "Analyzing Historical Performance Data," Viewing User Session History, page 104. The Details tab contains the following additional information. The latency and user agent data is derived from PMU 116 (redirect after login) with the same session ID (context 1) as the associated PMU 109 (user session began). IP Address Indicates the IP address of the machine that the user is using to connect to the PeopleSoft system. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 85

98 Monitoring System Performance Chapter 6 User Agent Latency The Mozilla user-agent string specification for the user's browser. The duration of PMU 116 (redirect after login). The latency value enables you to infer the network latency for a user connection. Viewing Recycled Processes Diagnosis Access the Recycled Processes Diagnosis page (PeopleTools, Performance Monitor, System Monitor, Recycled Processes Diagnosis). The Recycled Processes Diagnosis page is used to determine the causes of application server recycles. Specifically, it can determine what kind of server request was received by application server processes before a restart occurred. Monitoring Resource Usage This section provides additional information that may help you to interpret the information that Performance Monitor provides that is related to the events reporting resource usage on host machines. These events are: Event 150 (JVM Status). Event 151 (Network Status). Event 200 (Resources Per Process). Event 300 (Host Resource Status). Event 301 (Tuxedo "pq" Row). Event 302 (Tuxedo "psr" Row). These events report the usage of machine resources (CPU, memory, network, and so on). These events, except for Event 150 (JVM Status), make calls to external APIs (often specific to the operating system) to retrieve metric information. The monitored system sends each event at the sampling interval that is specified for that system. Working with Event 150 (JVM Status) This event applies only to web servers. This event does not make calls to any operating system-specific API. Working with Event 151 (Network Status) This event applies only to web servers. 86 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

99 Chapter 6 Monitoring System Performance For Event 151 (Network Status) the system launches a separate executable from Java that invokes the "netstat n" command. On UNIX, the command runs in a separate shell. When the command finishes, the process ends. PeopleSoft does not run "netstat" with an interval argument. Warning! On some platforms the "netstat" command can require up to a minute (or more) to finish. If the sampling interval is shorter than the time required for the command to complete, "netstat" commands will be running continuously. Working with Event 200 (Resources Per Process) This event applies to the application server and the Process Scheduler server. The PeopleSoft system makes specific operating system calls to obtain metrics for %CPU that are used by the process, CPU time consumed, virtual memory size, and working set size. Operating systems have slightly different definitions for these quantities and different ways of reporting them. For instance, "working set" memory is a Windows term; "resident set" is the UNIX equivalent. PeopleSoft strives for consistency across platforms. For example, PeopleSoft expresses %CPU within a range from 0 to 100 on all machines even though some vendors scale to N*100% if multiple CPUs (N CPUs) exist. Microsoft Windows and Linux compute one or more resources as an average of the two measurements at the beginning and end of a sampling interval. On these platforms, the Performance Monitor does not report an Event 200 (Resources Per Process) until the second sampling interval after you boot a server. Process resource utilization is usually sampled by the operating system and written to a memory location. Windows writes to the registry, while UNIX writes to various files. The system reads the current values for the process, so events change only when the operating system updates the statistics. Most operating systems update these statistics at least once per second. PeopleSoft obtains all information using lightweight, C++ programmatic APIs. No additional processes or shell commands are run. Operating System Windows Description Performance Monitor uses Performance Data Helper (PDH) to read registry counters. The information is identical to the Windows Performance Monitor tool. When multiple copies of a process, such as PSAPPSRV, are running, registry counters are assigned arbitrarily. For example, counter 1 and counter 2 can reverse their process assignment when a process reboots. Performance Monitor corrects for this. CPU utilization is "% Processor Time," defined as the fraction of time that the process spends in kernel + user during the last PDH sampling interval (typically one second). Process time is kernel + user, accurate to 1 millisecond. Virtual memory is "Virtual Bytes." Working set memory is "Working Set," which is the same as "Mem Usage," displayed by the Windows Task Manager. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 87

100 Monitoring System Performance Chapter 6 Operating System AIX Description Performance Monitor reads the psinfo files, which is the same source of information that AIX uses for its "ps" command. CPU utilization is the same as "ps -o pcpu" or the %CPU from "ps v". AIX defines it as a lifetime average: total CPU time that is consumed by the process divided by total time that the process has been running. Process time is system + user, excluding children. Accurate to 10 milliseconds. Virtual memory is the same as "ps -o vsz", or the 1024 * SIZE field from "ps v". Resident set memory is the same as the 1024 * RSS field from "ps v". HPUX Performance Monitor reads pst_status using pstat_getproc. CPU utilization is the same as "top" divided by the number of CPUs (on HPUX "top" shows utilization of a single CPU). Process time is system + user, excluding children. Virtual memory is the same as the SIZE field from "top". Resident set memory is the same as the RES field from "top". Linux Performance Monitor reads ps information from /proc files. CPU utilization is the same as "top" divided by the number of CPUs (on Linux "top" shows utilization of a single CPU). It is the average utilization over the last sample period. Process time is system + user, excluding children, and accurate to 10 milliseconds. The times for all threads in the process are added together. Warning! Linux kernel 2.4 tracks only the process time of each thread. PeopleSoft searches the /proc directory to find all threads and report the total time of the process. However, on a production system with thousands of threads, accumulating this information may take up to a second of CPU time. Virtual memory is the same as "ps -o vsz" (not visible in "top"). According to the man pages, it counts just text, data, and stack. Resident set memory is the same as "ps -o rss" (not visible in "top"). This value includes just text, data, and stack. 88 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

101 Chapter 6 Monitoring System Performance Operating System Solaris Description Performance Monitor reads psinfo files, which is the same source of information that Solaris uses for its "ps" command. CPU utilization is the same as "top". Solaris computes a moving average with exponential weighting. Process time is system + user, excluding children, and accurate to 10 milliseconds. Virtual memory is the same as "ps -o vsz" or the SIZE field from "top". Resident set memory is the same as "ps -o rss" or the RES field from "top". On Solaris, shared libraries are counted in RES but not in SIZE. OS/390 The only metric that is supported in the current release is Process Time. Working with Event 300 (Host Resource Status) This event applies to the application server and the Process Scheduler server. Performance Monitor makes specific operating system calls to obtain metrics for %CPU use on the host machine, %Memory use, and the hard page fault rate. Operating systems have slightly different definitions for these quantities, and they have different ways of reporting them. In most cases, PeopleSoft expresses %Memory use to reflect utilization of physical memory. Performance Monitor programmatically queries the Tuxedo management information base (MIB) for total Jolt connections and total requests queued. All platforms compute one or more resources as an average of the two measurements at the beginning and end of a sampling interval. Performance Monitor does not report an Event 300 (Host Resource Status) until the second sampling interval after you boot the server. Process resource utilization is usually sampled by the operating system and written to a memory location. Windows writes to the registry, while UNIX writes to various files. The system reads the current values for the process, so events change only when the operating system updates the statistics. Most operating systems update these statistics at least once per second. PeopleSoft obtains all information using lightweight, C++ programmatic APIs. No additional processes or shell commands are run. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 89

102 Monitoring System Performance Chapter 6 Operating System Windows Description Performance Monitor uses Performance Data Helper (PDH) to read registry counters. The information is identical to the Windows Performance Monitor tool. CPU utilization is "% Processor Time (_Total)", defined as the fraction of time not run by the system idle thread in the last PDH sampling interval (typically one second). Memory utilization is "% Committed Bytes In Use" and will change if the paging file is extended by the system administrator. Page faults is "Pages / sec", which reports actual disk page fetches ("Page Faults /sec" reports soft faults). AIX Performance Monitor uses libperfstat API (a wrapper for knlist) to read kernel counters. CPU utilization is the same as "topas" and "vmstat". This is an average over the last sample period, but process CPU use is an average over process lifetime. Therefore, on AIX, it is possible for machine CPU use to be momentarily smaller than CPU use of a process that was previously CPU-intensive. Memory utilization is defined as "free real pages" / "total real memory pages". The "vmstat" command shows the number of free pages, but not the available total. The "topas" field, Real MEMORY, shows "real memory pages." The PAGING SPACE field shows only reserved pages, not free pages. Page faults is the same as "topas" and "vmstat", with the absolute difference averaged over the last sampling time. According to Linux information, this value includes pages faults that do not cause paging activity. HPUX Performance Monitor uses pstat_getdyamic (pstat) to read kernel counters. CPU utilization is the same as "top" and "vmstat". This is an average over the last sample period, but process CPU use is an average over a longer time period. Therefore, it is possible on HPUX for machine CPU use to be momentarily smaller than CPU use of a process that was previously CPU-intensive. Memory utilization is defined as "real memory + text pages / physical memory." Neither of these quantities are exposed with vmstat. Page faults is the same as "vmstat -s, zero fill page faults", with the absolute difference averaged to a rate over the last sampling time. 90 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

103 Chapter 6 Monitoring System Performance Operating System Linux Description Performance Monitor reads kernel statistics from files in /proc. CPU utilization is the same as "top", averaged over all processors. This is an average over the last sample period. Memory utilization is the same as "top", or the "free" command "Mem used / (av)ailable" field. It measures physical memory utilization. The "used" field is actually "available - free". The "M" and "K" units of "top" are 1024 * 1024 bytes and 1024 bytes, respectively. Page faults is the first "page" kernel counter from /proc/stat (pages swapped in from disk), with the absolute difference averaged to a rate over the last sampling time. Solaris Performance Monitor uses the Kernel Statistics API (kstat) to read kernel counters. CPU utilization is the same as "top, user + kernel". This is an average over the last sample period, but process CPU use is a weighted average over a longer time period. Therefore, it is possible on Solaris for machine CPU use to be momentarily smaller than CPU use of a process that was previously CPUintensive. Memory utilization is defined as the "used / used + available" fields from "swap -s". This is only an approximation because of allocated but unreserved RAM. See the Sun web site for more information. Page faults is the same as major page faults from "vmstat -s", with the absolute difference averaged to a rate over the last sampling time. OS/390 Performance Monitor uses the ERBSMFI and CVT APIs to report resource use on the logical partition. Higher priority jobs on other partitions can "steal" resources and not appear in these metrics. CPU utilization is the same as RMF "TOTAL/AVERAGE LPAR BUSY TIME PERC" field. Memory utilization is defined from the RMF "AVAILABLE" versus RMF "TOTAL FRAMES" fields. Page faults is not supported for this release. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 91

104 Monitoring System Performance Chapter 6 Working with Event 301 (Tuxedo "pq Rows) This event applies to the application server and the Process Scheduler server. The system programmatically queries the Tuxedo management information base (MIB) for the status of each queue. Working with Event 302 (Tuxedo "psr" Rows) This event applies to the application server and the Process Scheduler server. The system programmatically queries the Tuxedo management information base (MIB) for the status of each server. Only PeopleSoft servers appear as Performance Monitor events; the BBL is not reported. 92 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

105 Chapter 7 Analyzing Historical Performance Data This chapter provides an overview of historical performance data and discusses how to: View completed PMUs. View event history. View user session history. Historical Performance Data This section provides an overview of historical performance data and lists the pages used to analyze historical performance data. Understanding Historical Performance Data The system stores historical data from the moment a PMU finishes processing or an event occurs. When the archive program runs (PSPM_ARCHIVE), it either deletes the historical data or moves it to the archive tables. Analyzing historical data can help you: Identify trends. By comparing historical data, you can spot upward and downward performance trends. Investigate past user complaints. For example, a user may complain that performance was slow during the previous week. You search historical performance data to find the PMUs that were generated by the user at a specific time a week ago. Pages Used to Analyze Historical Performance Data Note. The charting pages appear under the History menu; however, those pages are covered in a separate chapter. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 93

106 Analyzing Historical Performance Data Chapter 7 Page Name Definition Name Navigation Usage Completed PMUs PSPMTRANSHIST PeopleTools, Performance Monitor, History, Completed PMUs Event History PSPMEVENTHIST PeopleTools, Performance Monitor, History, Event History User Session History PSPMHISTUSERS PeopleTools, Performance Monitor, History, User History Enables you to search for and display PMU details and trees. Enables you to search for and display event details. Enables you to view a user's signon and signout activity within a specified interval of time. See Also Chapter 8, "Working with Performance Monitor Charts and Analytics," page 107 Viewing Completed PMUs Access the Completed PMUs page (PeopleTools, Performance Monitor, History, Completed PMUs). Note. The search result of completed PMUs includes history information collected by agents even if they are currently marked inactive. Note. If a "clone table" warning appears at the top of the page, the archive program is either currently running or has abnormally terminated. If this is the case, all new incoming information that is being sent to the monitoring system is currently being inserted into the clone tables. New performance data is not inserted into the current data tables until the archive program finishes successfully. Entering Search Criteria Before you can view information about completed PMUs, you need to enter criteria so that you can locate the appropriate completed PMUs. Use the Advanced Search link to expose all search criteria. The maximum rows that are returned by the search depends on the value in the Search Row Limit field on the Global Administration page. See Chapter 4, "Administering the Performance Monitor," Setting Global System Options, page 25. System ID Identifies each monitored system. PeopleSoft automatically generates this value incrementally. System definitions are created automatically when the first agent of a monitored system registers with the monitoring system. 94 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

107 Chapter 7 Analyzing Historical Performance Data Database Name User ID Process ID Performance Trace Name Component The name of the PeopleSoft application database running on the monitored system. The monitoring system automatically inserts this value when it recognizes and creates a monitored system. If you want information for a particular user, enter the user ID of the user for which you want to track completed PMUs. This field prompts against the PSPMOPRDEFN table. Enables you to search completed PMUs by the process ID. If a user has launched a performance trace in the Performance Trace Console, the system labels every PMU that is generated in a business process with a performance trace name. Select the appropriate trace name to view its results. The Performance Trace Name field prompts against the PSPMPERFTRACE table, which the Lookup program (PSPM_LOOKUP) populates. If you want information for a particular component, enter the name of the component for which you want to track completed PMUs. This field prompts against the PSPMPNLGRPDEFN table. Note. Searching on a component is equivalent to searching on Context 1 for application server PMUs. Market PMU Definition Set PMU Identifier Context 1, 2, 3 PMU Status Enter the market that is associated with the component. Indicates the set to which the PMU definition belongs. A PMU definition set is similar to a message set. This is required if you want to generate metric and duration charts. Identifies the PMU definition within a PMU definition set. This is required if you want to generate metric and duration charts. PMU metrics contain data that is specific to that PMU. Context values, on the other hand, are common to the entire user request or a specific tier. For example, the component name is stored in a context. The system uses contexts to "flatten" a PMU tree. For example, you do not have to navigate up from a SQL PMU to an ICPanel PMU to see what component generated that SQL statement. In some cases, PMU types are shared across different user request types. In such cases, the label for the PMU will be generic. Use the Context Help button to view the context definitions for a PMU set ID and ID that you've entered. Displays the PMU status. See Chapter 7, "Analyzing Historical Performance Data," Completed PMU Status, page 100. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 95

108 Analyzing Historical Performance Data Chapter 7 Top Instance Statement Number From Date/Time To Date/Time Duration Domain Name Domain Host/Port Agent Identifier Agent Type Search Refresh Each PMU that is reported by an agent has a unique instance ID. The instance identifier of the first PMU generated by a particular user request is the top instance. The value of the top instance is then stored with every child PMU within a request. The top instance is the associating value for all PMUs that are generated by the same user request. This search field applies only when you are searching for PeopleCode SQL PMUs, which are PMU 406 (PeopleCode SQL Execute) and PMU 407 (PeopleCode BuiltIn SQL Execute). The Context 2 field contains the PeopleCode program name and Metric 2 contains the statement number in the PeopleCode where the SQL was executed. To search by statement number, you must enter the following search criteria: PMU Definition Set = 1, PMU ID = 406 or 407, and Context 2 must have the name of a PeopleCode program. Specify a range of time in which the system should search for completed PMUs. The system searches on the monitor date and time, not the agent date and time. Specify criteria that is related to the duration of a PMU in milliseconds. Select the application server domain to generate the PMUs. Select the name and port number of the server machine on which the domain that generated the PMU resides. Identify the specific agent that monitored a PMU. Identify the type of agent that monitored a PMU. The Search button performs a search based on the search criteria that is defined on the page. If you want to view the most current information, use the Refresh button after running the initial search. If you click Search repeatedly, the system returns the same results from the same time range for each search. Updates the value in the To Time field to equal the current time and performs a search. This enables you to view the most current results of a search. Working with Completed PMUs At the bottom of the Completed PMUs page, the Completed PMUs grid contains the information that is related to the PMUs that are retrieved by your search criteria. The tabs in the Completed PMUs page are discussed in the following sections. Summary PMU Indicates the name of the PMU. Click the PMU name to view the PMU Details page. 96 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.

109 Chapter 7 Analyzing Historical Performance Data Duration Complete Tree PMU Details User PMU Status Monitor Received Date/Time Agent Start Date/Time Domain Name Indicates the duration of the PMU from start to finish in milliseconds. Click the tree icon to view the complete PMU tree, including all ancestor and descendant PMUs. Provides further description of the PMU. The details comprise contexts and metrics that are helpful for identifying the PMU. Each type of PMU displays a different set of details. Indicates the user whose request triggered the PMU. Displays the PMU status. See Chapter 7, "Analyzing Historical Performance Data," Completed PMU Status, page 100. Indicates when the monitor system received notification that a PMU had been started or updated. Indicates that time according to the agent clock that the last update was received. Indicates the domain that handled the request that is associated with the PMU. Identifiers The Identifiers tab presents the following additional information. PMU Set PMU ID Agent ID Process ID Instance Parent Instance Top Instance Identifies the set to which the PMU ID belongs. Identifies a PMU definition within a PMU set. Identifies the agent that reported the PMU. The operating system process ID of the process that generated the PMU. A unique identifier that is generated by the system for each PMU. The PMU that produced the PMU in this grid row. Each PMU that is reported by an agent has a unique instance ID. The instance identifier of the first PMU that is generated by a particular user request is the top instance. The value of the top instance is then stored with every child PMU within a request. The top instance is the value that associates all PMUs that are generated by the same user request. Contexts The Contexts tab reveals the context definition and value that is associated with each PMU in the grid. To view the details of the contexts in a PMU, examine the context and PMU definitions. Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved. 97

110 Analyzing Historical Performance Data Chapter 7 This Description column shows data for the additional description value that is part of the incoming PMU. The description stores large character values and generally stores information that exceeds the 128 character limit of Metric 7 and the 254 character limit of the contexts. See Chapter 10, "Performance Monitor Meta-Data," Context Definitions, page 149. Metrics The Metrics tab presents the metric values for all of the metrics that are defined for each PMU in the grid. Metric information appears in a raw format. The values reflect exactly what is stored in the database. See Chapter 10, "Performance Monitor Meta-Data," Metric Definitions, page 146. Trees The Trees tab enables you to view the PMU within the context of a PMU tree. View Ancestors View Descendants Complete Tree Shows all parent PMUs for the selected PMU. This may be useful in determining the call tree of a child PMU. Only direct ancestors appear; sibling PMUs do not appear. Shows all child PMUs for the selected PMU. This may be useful in determining where the time is spent for a PMU. Displays the entire tree representing the user request. If verbose or debug filters are set, this tree may contain a large number of nodes and may take a significant amount of time to load. Metric Charts When your search criteria includes a PMU set and PMU identifier, all rows in the grid will be of the same type of PMU. In this case, the metric chart displays the metric values of the returned PMUs against one another. This enables you to track metrics over a period of time. Viewing a metric chart 98 Copyright 1988, 2009, Oracle and/or its affiliates. All Rights Reserved.