WebSphere Application Server V8.5.5 Serviceability and Performance Monitoring

Transcription

1 WebSphere Application Server V8.5.5 Serviceability and Performance Monitoring Vishal Charegaonkar WebSphere Serviceability Lead Lalitha V Pannala WAS Development Manager [email protected] 1

2 Agenda Quick Overview of WebSphere Application Server V855 Liberty Core Monitoring infrastructure in full-profile WAS v8.5.5 PMI TPV Request Metrics Monitoring in Liberty Core V8.5.5 Liberty Monitoring Serviceability in Liberty Core V855 Binary Logging in Liberty Timed Operations in Liberty Serviceability in Full Profile WAS V8.5.5 High Performance Extensible Logging (HPEL) XCT Performance Tooling in WAS ISA ISA DC 2

3 WebSphere Application Server - Liberty Profile

4 Monitoring Infrastructure in WebSphere Application Server Runtime FULL PROFILE WAS v

5 Performance Tools at a Glance PMI Server-side function that collects performance metrics from a running server General extensible framework for collecting performance data on running applications Family of products share interface; products may have different implementation TPV Display PMI data in a GUI viewer Request Metrics

6 PMI (Performance Monitoring Infrastructure) Monitor Overall System Health Web Applications Servlet/JSP resp time # Servlet/JSP requests EJBs EJB resp time # EJB requests Http Sessions Live sessions Invalidated sessions DB Conn Pools % pool in use Avg. JDBC time Dynamic Cache Memory cache size # of hits/misses Joe, in action! HTTP Client Trade servlets Trade JSPs JVM EJB Access Beans HTTP Session Trade Session EJB Profile CMP Holding CMP Account CMP Register CMP Quote CMP Session Entity Trade EJBs Trade Database EJB Persistence Web Services # of loaded services # of requests processed JVM Memory Free memory # of GCs System Data CPU utilization Memory usage Thread Pool Active threads % pool used

7 Servlet Counters (Example)

8 Levels in PMI Version 6 and later, PMI Levels None Basic (Default) Extended All Custom

9 PMI Clients Tivoli Performance Viewer (TPV) Uses calculations reported by the PMI Constructs graphical and tabular performance charts Java client Access the PMI data using JMX interface Java client can use either RMI/IIOP or SOAP You can build your own application. Performance Servlet Uses the Hypertext Transfer Protocol (HTTP) PMI metrics externalized as XML document

10 Performance Impact PMI implementation ensures efficient operation of WebSphere Application Server Data collected against the WebSphere Application Server must be maintained with a minimum overhead To accomplish this No significant calculations are performed on the server-side Client-side is responsible for holding and manipulating raw data that is collected against the application server Overhead cost for each counter Represented as a rating of low, medium, or high Appropriate for ongoing production monitoring

11 TPV Functionality A simple viewer for PMI data Display PMI data in chart view and table view Summary view for key performance data View refreshes automatically User can manually refresh to see real-time data Log PMI data as XML or binary format and replay logs Provide tuning advice based on analysis of buffered data (On Demand Performance Advisor)

12 TPV Summary Report Summary Views

13 TPV Graphic View

14 Request Metrics Request metrics measures these intervals for individual transactions Web Web Server Server Web Web Container Container EJB EJB Container Container Data Data Source Source JDBC Database Database Transaction A e.g. Response time for transaction A Track response time decomposition for individual transactions Correlators will be used to correlate the response time spent in each component Information can be saved to log files or sent to ARM agents ARM: Application Response Measurement Provide filtering capabilities Minimize performance impact by controlling what requests to monitor

15 An Example For Request Metrics Data HTTP request /trade/scenario 172 ms Servlet /trade/scenario > 130 ms EJB TradeEJB.getAccountData > 38 ms JDBC select > 7 ms A request sent to web server A hierarchal view for transaction flow by response time in following components: web server plug-in Web container EJB container JDBC driver (via RRA wrapper layer) Response time collected for each level includes the time spent at that level, plus the time spent in the lower levels. For example, the response time for the servlet, which is 130 milliseconds includes the times spent in EJB and JDBC Therefore, the servlet process itself contributes to =92ms

16 Monitoring Features in WebSphere Application Server Runtime LIBERTY CORE V

17 Key Concept Feature monitor-1.0 (introduced in WAS v85 Liberty Profile) - Collects data from various components JVM ThreadPool Web Applications Sessions ConnectionPool - Data reporting using MXBeans - Can be used with JConsole or any standard JDK client 17

18 Key Concept Liberty Monitoring vs twas Monitoring twas PMI - JSR 77 support - Use MBean and call getstats - Use Perf MBean for all stats - Return Type Statistics Objects TimeStatistics CountStatistics RangeStatistics Etc - Visualisation (TPV) Provided - Monitoring tools fetch PMI data by parsing Objects - Levels of PMI Liberty Monitoring - No JSR77 / MBeans - Byte Code Injection (If monitor enabled) - MXBeans for reporting (Instead of Statistics) ThreadPoolMXBean JvmMXBean ServletStatsMXBean - jconsole as visualization - REST Adapters to retrieve information from MXBeans - Apply filter at component level 18

19 Key Concept - JVM Monitoring Free Memory Used Memory Total Heap Size Process CPU Utilisation JVM Uptime in milliseconds GC Count Since JVM Started GC Time (Total time spent in GC) 19 JConsole Screenshot

20 Thread Pool ThreadPoolStats - Active Threads - PoolSize - Name of Thread Pool (Default Executor) JConsole Screenshot 20

21 Web Applications Servlet Stats (For each Servlet) - Request Count - Reponse Time (in nanoseconds) - Application Name - Servlet Name 21

22 ConnectionPool Monitoring ConnectionPoolStats(Per ConnectionPool) CreateCount DestroyCount ManagedConnectionCount WaitTime ConnectionHandleCount FreeConnectionCount 22

23 Sessions Monitoring SessionStats(Per WebApplication) Following data is available as part of SessionStats CreateCount LiveCount ActiveCount InvalidatedCount InvalidatedCountbyTimeout 23

24 Monitoring Multiple components Users can filter the components to be monitored using the monitor1-0 feature in the Liberty profile. Components to be monitored can be accomplished by specifying filter element in the server.xml of the specific server - Ex : <server description="new server"> <featuremanager> <feature>jsp-2.2</feature> <feature>jdbc-4.0</feature> <feature>monitor-1.0</feature> </ featuremanager> <monitor filter="jvm,threadpool,webcontainer,session,connectionpool"/> </server> 24

25 Serviceability Features in WebSphere Application Server Runtime LIBERTY CORE v

26 1. Binary Logging New way of handling log and trace content for Liberty Greatly improves speed of logging and tracing Helps you filter through logs and trace with a powerful command line tool Stores log and trace data in a binary format Provides a common solution for all supported platforms Includes an API to read from binary repositories programmatically Based on the High Performance Extensible Logging technology available in full profile WebSphere Application Server offerings Binary Logging 26

27 Binary Logging Log and Trace Repositories At runtime All WebSphere Application Server log content is written to the log data repository (Audit and above is also written to the console.log if it is enabled) System.out System.err java.util.logging log entries (Detail, Config, Audit, Warning, Severe, Fatal) OSGi logging service log entries (LOG_INFO, LOG_WARNING, LOG_ERROR) All WebSphere Application Server trace content is written to the trace data repository java.util.logging trace entries (Fine, Finer, Finest) OSGi logging service (LOG_DEBUG) The messages.log and trace.log files from the default Liberty log and trace framework are not written when binary logging is enabled Binary Logging 27

28 Binary Logging Viewing with binarylog command Viewing logs and trace, the binarylog command displays repository content in plain text provides options to filter the displayed content and to choose the output format used to monitor (tail) the repositories Binary Logging 28

29 Binary Logging - Performance Performance Measured using java.util.logging Six times faster than logging to messages.log when console.log was disabled Six times faster than tracing to trace.log Use binary logging to reduce performance impact of logs and traces on your production servers console.log must be disabled when superior logging performance is required (note that this is not a factor in trace performance) <logging consoleloglevel="off" /> Liberty Logging Performance Liberty Trace Performance events per second threads events per second Default (with console.log) Default (no console.log) HPEL (with console.log) HPEL (no console.log) Default HPEL Binary Logging threads 29

30 Binary Logging - Enablement To enable binary logging add this to your bootstrap.properties file in your server root directory: websphere.log.provider=binarylogging-1.0 To disable binary logging remove this line from your bootstrap.properties file A server restart is required to enable or disable binary logging Binary Logging 30

31 Binary Logging - Configuration Binary log and trace settings are set in child elements of the logging element in the server.xml Settings for the console log, trace specification, and log directory are inherited from the logging element <logging logdirectory="${server.output.dir}/logs" tracespecification="*=info" consoleloglevel="audit"> <binarylog purgemaxsize="50"/> <binarytrace purgemaxsize="50"/> </logging> Binary Logging 31

32 binarylog command Filtering Options for Viewing Logs and Trace The binarylog command is in the wlp/bin directory To view all binary logs and trace binarylog view servername To view all warnings and errors binarylog view servername --minlevel=warning To view all warnings and errors from thread 33 from the latest run of the server binarylog view servername -includethread=33 --includeinstance= latest To tail the content of the logs and trace as they are written by the server binarylog view servername --monitor To see all of the available options binarylog help view Binary Logging 32

33 2. Timed Operations Timed operations is a new feature that tracks the duration of JDBC operations running in the application server Logs a warning when operations take more or less time to execute than expected Periodically creates a report in the application server log, detailing which operations took longest to execute When running the server dump command, it will generate a report containing information about all operations it has tracked The information listed in these reports can be used to decide if anything has an unusual behavior Recommended for production environments to track slowdowns when they first occur Binary Logging 33

34 Timed Operations Sample logged message [3/14/13 14:01:25:960 CDT] TimedOperatio W TRAS0080W: Operation websphere.datasource.execute:jdbc/social:insert INTO DB2INST1.CommentTab (comment, extension, object, time, userid, mentionsid) VALUES (?,?,?,?,?,?) took ms to complete, which was longer than the expected duration of ms based on past observations. Sample automatically generated report in the log Binary Logging 34

35 Timed Operations Sample introspection file The description of this introspectable service: Information about the timed operations, grouped by type, and sorted within each group by expected duration Timed operations for: websphere.datasource.execute Expected duration (ms) Standard deviation (ms) Timed operation websphere.datasource.execute:jdbc/tradeds:update orderejb set holdingid =? where orderid =? websphere.datasource.execute:jdbc/tradeds:delete from orderejb websphere.datasource.execute:jdbc/tradeds:update orderejb set holdingid=null where holdingid =? websphere.datasource.execute:jdbc/tradeds:delete from accountejb websphere.datasource.execute:jdbc/tradeds:update orderejb set orderstatus =? where orderid =? websphere.datasource.execute:jdbc/tradeds:select * from orderejb o where o.orderstatus = 'closed' websphere.datasource.execute:jdbc/tradeds:delete from holdingejb where holdingid =? websphere.datasource.execute:jdbc/tradeds:update orderejb set holdingid =? where orderid =? websphere.datasource.execute:jdbc/tradeds:update accountejb set lastlogin=?, where userid=? websphere.datasource.execute:jdbc/tradeds:delete from orderejb where orderstatus='cancelled' websphere.datasource.execute:jdbc/tradeds:delete from accountprofileejb websphere.datasource.execute:jdbc/tradeds:update accountejb set balance =? where account =? websphere.datasource.execute:jdbc/tradeds:select * from orderejb o where o.orderid =? websphere.datasource.execute:jdbc/tradeds:select * from quoteejb q where q.symbol=? Binary Logging 35

36 Enabling Timed Operations To enable timed operations add the following feature to your server.xml file: <feature>timedoperations-1.0</feature> To disable timed operations remove this line from your server.xml file A server restart is not required to enable or disable timed operations Binary Logging 36

37 Configuring Timed Operations To disable the generation of the report to the logs <timedoperation enablereport="false"/> To change the frequency of the report, for example to once every 12 hours <timedoperation reportfrequency="12"/> To log a warning when the total number of timed operations reaches the specified value <timedoperation maxnumbertimedoperations="10000"/> Binary Logging 37

38 Serviceability Features in WebSphere Application Server Runtime FULL PROFILE V

39 High Performance Extensible Logging New way of handling log and trace content for Liberty Greatly improves speed of logging and tracing Helps you filter through logs and trace with a powerful command line tool Stores log and trace data in a binary format Provides a common solution for all supported platforms Includes an API to read from binary repositories programmatically Based on the High Performance Extensible Logging technology available in full profile WebSphere Application Server offerings Binary Logging 39

40 How do I get started? Admin Console 1. Expand Troubleshooting 2. Select Logs and trace 3. Select server1 4. Select Change log and trace mode 5. Click Switch to HPEL Mode 6. Save your configuration 7. Restart your server 40

41 What happened to my log/trace files? WAS V7.0 HPEL SystemOut.log SystemErr.log trace.log activity.log text: (messages, System.out) text: (System.err) text: (messages, System.out, trace) binary: (messages) logdata/ tracedata/ TextLog* binary: (messages, System.out, System.err) binary: (trace) text: (messages, System.out, System.err, [trace]) *TextLog is redundant (same content as logdata / tracedata) and can be disabled. 41

42 How do I use my logs/trace for problem determination? Command shell 1. Change to your WAS profile/bin dir 2. Use the logviewer command Admin Console 1. Expand Troubleshooting 2. Select Logs and trace 3. Select server1 4. Select View HPEL logs and trace 42

43 How do I configure it? Administrators can independently configure (as an immediate runtime change or a persistent config change): HPEL Log HPEL Trace HPEL Text log Log detail level setting is the same as previous releases. 43

44 Cross Component Trace Cross Component Trace (XCT) is a log/trace correlation technology. XCT enables you to determine which log/trace entries are part of each request. XCT can be used in any of three different modes: 1)Request ID mode 2)Request ID and correlation log record mode 3)Request ID, correlation log record, and data snapshot mode XCT works best in combination with High Performance Extensible Logging (HPEL). HPEL stores XCT Request IDs HPEL can filter log/trace records by XCT Request ID Cross Component Trace (XCT) 44

45 Cross Component Trace Logviewer IBM WebSphere Cross Component Trace Logviewer can be used to view files augmented with correlation log records. Available for the IBM Support Assistant Can load multiple files simultaneously Can show flat / hierarchical views Cross Component Trace (XCT) 45

46 Enabling XCT through Administrative Console The panel below is where XCT is enabled on the administrative console The panel can be found here: WebSphere Application Servers > SERVER_NAME > Change log detail levels Cross Component Trace (XCT) 46

47 Understanding XCT Correlation Log Records Correlation log records look like this example: [4/23/12 13:54:44:509 IST] e XCT I BEGIN AAADx/itMDz-AAAAAAAAAAA cccccccccc2 HTTPCF(InboundRequest /JMSApp/LocalMessageSend RemoteAddress( ) RequestContext( )) <Date>: The date and time when the log record was generated <Thread_ID>: The thread which generated this message <XCT_Logger_Name>: The XCT logger name is XCT. This logger is used to identify the XCT Records in the Log file <Message_Type>: Type of the log message <XCT_STATE>: Each XCT Record has a State, it can be BEGIN, END <XCT_ID>: A Unique ID generated for correlating the XCT Records <XCT_PARENT_ID>: The XCT_ID of the parent XCT context <XCT_MESSAGE>: The XCT message contains the information about the XCT record; this can contain some Associations and Annotations Cross Component Trace (XCT) 47

48 Tooling around the Application Server Runtime LIBERTY and FULL PROFILE 48

49 IBM Support Assistant IBM Support Assistant (ISA) is a free tool that serves as a single front-end to organize the resources and tools needed to resolve questions and problems with many IBM software products. Download from Over one hundred add-ons available for various IBM products across all IBM Software brands Three components: ISA version 5.0: Browser-based front-end plus shared server (new currently in Beta) ISA Data Collectors: tools and scripts to simplify diagnostic collection Available as standalone scripts (formerly ISA-Lite) or web-based (new) Installed with recent releases of WebSphere Application Server (V , V ) Key features: Collect and organize diagnostic data (logs, traces, etc.) Find and use Problem Determination tools Approximately 30 distinct PD tools currently offered in one central place Automatically discovers new tools and downloads updates to existing tools Search and browse support-related information about each product Receive step-by-step guidance for troubleshooting (Guided Troubleshooter / IBM Guided Activity Assistant) (phasing out) 49

50 IBM Support Assistant

51 IBM Support Assistant

52 IBM Support Assistant 5.0 New 52

53 IBM Support Assistant 5.0 New 53

54 IBM Support Assistant: 4.1 or 5.0 ISA 4.1 Workbench Single-user Individually installed on desktop Eclipse-based client ISA 5 Private Cloud Server-based model Install once Shared by many team members via browser Web 2.0 browser interface Remote execution of PD tools Off-load analysis processing Collaboration on PD Case Management Installation Manager All-in-one deployment EAR deployment Single-user option Tech Preview Dec 2011 Beta 1 May 2012 Beta 2 October 2012 Beta 3 May

55 Problem Determination Tools 55

56 Problem Determination Tools - WebSphere Tools managed in IBM Support Assistant 4.1 or 5.0 General Log analysis and Phase 1 Problem Determination Automated analysis integrated in ISA 5.0 Log Analyzer (LA) Configuration issues WebSphere Configuration Visualizer Profile Port Checker Visual Configuration Explorer (VCE) JVM-related issues Java Health Center (HC) Garbage Collection and Memory Visualizer (GCMV) IBM Pattern Modeling and Analysis Tool for Java GC (PMAT) IBM Thread and Monitor Dump Analyzer (TMDA / JCA) Memory Analyzer with DTFJ Adapter (MA) Heap Analyzer (HA) JVM and general dump analysis Interactive Diagnostic Data Explorer (IDDE) Dump Analyzer (DA) Specialized Modules for Dump Analyzer, Memory Analyzer, IDDE Trace and specialized analysis IBM WebSphere Cross-component Trace LogViewer (XCT) IBM Trace and Request Analyzer (TRA) IBM Database Connection Pool Analyzer (DCPA / JCP) IBM Web Server Plugin Analyzer (WSPA) Performance Performance Tuning Toolkit and many other tools for various functions and products Tools shipped/embedded with WSAS or stack products (e.g. WPS) Server Logs Viewer for WSAS (including HPEL) Version / installation history reporting tools Configuration Validator Classloader Viewer (CLV) Tivoli Performance Viewer (TPV) Server Logs Viewer in WebSphere Integration Developer Standalone products (IBM or third-party) Rational Tivoli Tools for standalone installation Diagnostic Tool for Java Garbage Collector Heaproots Trace Analyzer for WebSphere Application Server others Note: this list only includes tools that are generally available externally to IBM Clients 56

57 Most Commonly Used Tools in ISA Java Health Center (HC) Connects to a live JVM to investigate various performance and health issues Thread and Monitor Dump Analyzer (TMDA) Analyzes JVM thread dumps / javacores WebSphere Application Server Performance Tuning Toolkit (PTT) Analyzes and helps tune performance of a WebSphere Application Server, through PMI Log Analyzer (phasing out) Browses and analyzes log file content from wide range of IBM products Garbage Collection and Memory Visualizer (GCMV) Analyzes JVM verbosegc logs and other JVM performance logs Memory Analyzer (MA) Analyzes JVM heap dumps and system dumps to diagnose memory and other issues Dump Analyzer (DA) Analyzes JVM system dumps Heap Analyzer (HA) Analyzes JVM heap dumps to diagnose memory issues Memory Dump Diagnostic for Java (phasing out) Analyzes JVM heap dumps to diagnose memory issues 57

58 IBM Monitoring and Diagnostic Tools for Java A free suite of tools for understanding different aspects of Java applications Garbage Collection and Memory Visualizer (GCMV) Health Center Memory Analyzer (and IBM Extensions for Memory Analyzer) Interactive Diagnostic Data Explorer (IDDE) Provide visualisation, analysis and recommendations Fully supported Available in the IBM Support Assistant Provides the toolbox in which analysis tools reside Assist with opening PMRs and working with IBM Support Search feature to query IBM and non-ibm knowledge banks

59 Health Center Live monitoring tool with very low overhead Pre-installed in JDK and WebSphere Application Server Enable at startup: -Xhealthcenter Enable at runtime: java -jar healthcenter.jar ID=<pid> Understand how your application is behaving Monitor Class loading, File I/O, Environment settings, Garbage Collection, Method Profiling, Locking, Native memory use Diagnose potential problems, with recommendations Works at the JVM level no domain-specific (e.g. J2EE) information Suitable for all Java applications

60 IBM Support Assistant Data Collector (ISA DC) IBM Support Assistant Data Collector in WAS v8.5 Challenge To resolve software issues, analyzing key data is essential but it is often hard to locate and collect in a timely manner Shorten time to resolution Quickly collect diagnostic files or run traces that are predefined for WAS and WAS stack products ISA DC collections customized to specific WAS symptoms Built to match the MustGathers that L2 teams rely on Available in WAS bin / WAS profile bin directories to make data collection easier for our customers Provides a simplified interface to guide a user step-by-step through the data collection process Includes information about the customer environment (aka software inventory) and adds it to the PMR Increases the efficiency of getting PD data to IBM Support Built in capabilities to transfer data to IBM Ecurep in a secure way Facilitates WAS stack product data collections 60

61 IBM Support Assistant Data Collector Console (text) Interface isadc.bat or isadc.sh in WAS_HOME/bin Collection Archive Transfer to IBM for analysis with a PMR Silent response capability

62 Useful Links Problem Determination Tools IBM Support Assistant IBM Support Assistant /support/isa/ WAIT IBM Whole-system Analysis of Idle Time (WAIT) Cross Component Trace (XCT)