Java Application Performance Analysis and Tuning on IBM System i

IBM Systems & Technology Group Technical Conference 14 18 April, 2008, Sevilla, Spain Java Application Performance Analysis and Tuning on IBM System i iap02 Gottfried Schimunek Gottfried Schimunek Senior IT Architect Application Design IBM i5/os Enablement Program Manager IBM ISV Enablement 3605 Highway 52 North Rochester, MN 55901 Tel 507-253-2367 Fax 845-491-2347 schimu@us.ibm.com Abstract: Java Application Performance Analysis and Tuning on System i Speaker(s):Gottfried Schimunek IBM Corporation Java Applications are becoming more and more the norm in today's IT world; they are available on System i as part of a solution or as the complete solution. Although Java applications can be developed on another platform and easily deployed on System i, sometimes Performance issues can arise if the environment and application are not properly tuned. This presentation will demonstrate performance analysis on a Java application. We will show various tools and their output that lead to a resolution of particular performance problems. By the end of this session, attendees will be able to: 1.Do their own Java application performance analysis and perform tuning of their Java environment 1

Acknowledgements Thanks to: Scott Moore IBM Senior Technical Consultant Small and Medium Business Platform Jeremy Arnold IBM Java Performance Tools Development Agenda Introduction to Java Application Performance - Methodology How to detect and analyze a Java Application Performance issue? How to tune the Java Application environment on System i? 2

Network Client PC Backend Job DataBase HTTP Server Application Server Application Server Backend Job DataBase HTTP Server Application Server 3

Transaction with one second response time W here we spend our time 1.2 1 i5/os Seconds 0.8 0.6 0.4 Other Wait Java Wait Disk Wait CPU 0.2 0 Send on Network Network HTTP Server Application Server Backend Database Network Render on client Is the problem outside of the System i? Run WPM to determine if the problem is outside the System i Is the problem on the PC? Faster processor PC s can render complex web pages much quicker. Do some PCs seem to serve the web pages quicker? Use Task Manager to view the CPU usage on the PCs Amount of memory inadequate on PC? Is the problem on the network? From a MS DOS window, perform a tracert to the System i (Rough idea of route and times)* Analyze Network resources using Collection Services* Management Central* Turn on Apache Access log (or other HTTP server log) to find out how much time is spent on server. Perform a communications trace from i5/os to view when the request entered the System i* *- More detail will be provided in tools section 4

So, it looks like the issue resides on the System i If running a WebSphere application, run Web Performance Analyzer to make sure the system environment is set correctly Are the expectations of the system reasonable?* Older hardware or 1-Way Power5 systems with less then 3800 CPW will have longer response times for a single user or multiple users Machines without L3 cache will have less capacity then a machine with L3 cache. There are many flavors of a one-way iseries box. Java applications typically require more memory then other languages Is i5/os operating system setup properly? Try to run with the latest operating system version/release Latest PTFs applied to the system Common System Values Affecting Performance Memory pool Setup Job Setup 5

Is i5/os operating system setup properly? Try to run with the latest operating system version/release Throughput V5R2 WAS 5.0 V5R3 WAS 5.0 V5R3 WAS 5.1 V5R3 WAS 6.0 V5R4 WAS 6.0 V6R1 WAS 6.1 EJB Trade Mode JDBC Performance improvements in latest versions i5/os V6R1 has significant performance improvements over V5R4 i5/os V5R4 offers performance improvements over V5R3 and previous releases WebSphere Application Server 6.1 has significant performance improvements over 5.1, 5.0, and previous releases Is i5/os operating system setup properly? Latest PTFs applied to system PTF Name Latest CUMe PTFs Hiper PTFs Group PTFs DB2 UDB Group PTF Release All V5R2 V5R3 V5R4 All V5R2 V5R3 V5R4 Where to get it? IBM System i5 Support: Fixes SF99519: 520 Group Hiper SF99529: 530 Group Hiper SF99539: 540 Group Hiper Preventative Service Planning WebSite SF99502: 520 DB2 UDB for iseries SF99503: 530 DB2 UDB for iseries SF99504: 540 DB2 UDB for iseries WebSphere Application Server V6.1 PTFs Java Group PTF V5R3 V5R4 V5R2 V5R3 V5R4 SF99322: WebSphere Application Server SF99323: WebSphere Application Server SF99169: 520 JAVA PTF GROUP SF99269: 530 Java SF99291: 540 Java Other groups of interest: TCP/IP, HTTP Server, and Performance Tools 6

Is i5/os operating system setup properly? Common System Values Affecting Performance System Value Name Recommended Value Priority (1-High 5- Low) Notes QPRCMLTTSK See notes 1 On Power5, recommended value = 2 On Power3, recommended value = 1 Others before Power5, recommended value = 0 QQRYDEGREE See notes 2 If the system is CPU constrained, *NONE is the recommended value. If the system is DISK constrained, *OPTIMIZE is the recommended value If the system is NOT constrained, *OPTIMIZE is the recommended value QTHDRSCADJ On (1) 2 None QTHDRSCAFN Group=*NoGroup Level=*Normal 2 None QPFRADJ 0 (See Notes) 3 Java does not behave nicely when short of memory. QMAXACTLVL *NOMAX 2 None Is i5/os operating system setup properly? System Storage Pool Setup Recommendation is to put JVM in separate System Storage Pool Specify on the SubSystem Description to run in a separate pool. Using the WRKSYSSTS command look at the settings for your Storage Pool 1. Make sure Max Active is large enough to prevent jobs from entering ineligible state 2. Make sure the job has enough memory to prevent excessive Non- DB paging 7

Job Setup for Both JVMs Tune the garbage collection settings, especially the initial and max heap size* Make sure job priority is set to a low enough value (high priority) to get sufficient CPU Job Setup for Classic JVM only When running on V5R3 or later, we recommend the following 1. Set the Java compiler to run all classes with the JIT compiler Djava.compiler=jitc 2. Tune the garbage collection settings, especially the initial heap size (-Xms option) 3. For best possible throughput, do not run with max heap size set (default is *NOMAX)* * Discussed in separate presentation Where is the bottleneck? Network Processor Client PC Memory Disk Arms 8

Where is the bottleneck? It is likely that the problem is a combination of many different factors. The key is to focus on the most prevalent bottleneck for the largest gain. W here we spend our time From the chart to the right: Look at Database first, as this is the largest area for gains. Do not focus on areas which are taking less time. Seconds 1.2 1 0.8 0.6 0.4 0.2 0 Send on Network Network HTTP Server Application Server Backend Database Network Render on client Other Wait Java Wait DASD Wait CPU Is your problem with a single user or under load? If the problem resides with a single user, this is an easier problem to debug. Can be reproduced with a browser session and a stopwatch. If the problem occurs only under load, this is a tougher problem to debug. A LoadRunning test is VERY beneficial for solving this problem Certain problems ONLY occur under load Memory overloaded due to object leaks, for example Functional Timing problems Java locking issues 9

Are you CPU Bound? Is your problem with a single user or under load? Single User Under Load per second Java is Multithreaded, but one request usually uses one thread System i Is the bottleneck on the processor? HTTP Application Server Backend ILE Pgm Database Job 10

Is the bottleneck on the processor? If this is a single user The total CPU usage does not have to be 100% to be processor constrained. Depending on the hardware and number of processors, this result can vary Power5 processors are charged about 70-80% busy with a single thread of execution Of course, a 4-Way Power5 system will run at 20% busy, if one processor is in use. Other threads in a JVM will periodically use CPU Garbage Collection, Finalizer methods, etc I find the following to be useful when trying to find the performance issue quickly 1. Hit the webpage that you are having performance concerns 2. Press F10 on the WRKACTJOB screen monitoring the CPU usage 3. Once the request comes back stop pressing F10. 4. During the time that we were hitting the WRKACTJOB command, was a single CPU maximized? Is the bottleneck on the processor? If this is a multi-user test Use WRKACTJOB to determine how much CPU is being consumed. The total CPU usage should be around 90-100% to determine if CPU is the bottleneck. 11

Processor What is using the CPU? Using WRKACTJOB, determine which job(s) are using the most CPU Java Virtual Machine Database jobs (QSQSRVR and/or QZDASOINIT) Other Processor If the JVM is using most of the CPU Many different factors can cause your JVM to use excessive amounts of CPU Ensure the garbage collector is tuned correctly for your environment Ensure all tracing for your job is turned off Ensure you are not running with any debug options (JVMPI) Take a TPROF to determine where your job is spending it s time. This will spawn off new investigations. Use PRTPEXRPT to get a quick view Use PTDV to get a more detailed look at the PEX traces Looking into excessive object creations 12

Processor If the database jobs are consuming the CPU Perform Database performance tuning operations Use Database Monitor to drill down to individual queries Determine if any indexes are advised. Any Full opens being performed Use Visual Explain to determine why certain queries are more expensive Use Database tuning methods learned from SQL performance workshop Self Study: DB2 UDB for iseries: SQL Performance Basics http://www.ibm.com/servers/enable/site/education/ibp/4fa6/ Instructor led: DB2 for i5/os SQL Performance Workshop http://www.ibm.com/servers/eserver/iseries/service/igs/db2performance.html Is the bottleneck with the amount of memory? Java applications do not behave nicely when they run out of memory The Garbage Collector needs to touch the entire JVM heap looking for unused objects. If the heap does not fit in memory, it must page the heap from disk. This causes %age of disk being busy. Determine by WRKDSKSTS. The Garbage Collection cycle takes a longer period of time to complete. More objects are created and the heap grows even more. At this stage, the JVM could be in trouble, and a restart of the JVM might necessary. 13

Is the bottleneck with the amount of memory? If we have enough memory, the JVM Garbage Collector behaves nicely and can keep a manageable heap. 1 GB Of memory in pool Java Heap Java Heap Java Heap If the JVM takes more memory then available: GC cycles take longer, due to disk access More objects are created per GC cycle The heap will grow until it reaches Max. (Running Classic, it might now be set) Symptoms: High paging rates + high disk utilization Recovery: If this is just a highwater mark, the JVM could recover. In some cases, rebooting the JVM is your only option Is the bottleneck with the amount of memory? When running your workload Determine if you are excessively paging in the pool running the JVM (WRKSYSSTS) This is accompanied by high disk utilization (WRKDSKSTS) Before adding more memory, ensure the following: Make sure that there is not an object leak within your application. Make sure that you are not creating too many objects in a short period of time. Run your JVM in a separate pool, to ensure other jobs are not affecting the memory pool. Fine tune your Garbage Collector* If the JVM has not reached this state under load, you can guestimate how much memory is required for your application Start your workload in a pool with plenty of memory. Running the J9 JVM: Min Memory in Pool = At least the emporary storage used under the run attributes on Work with Job. Running the Classic JVM (use DMPJVM to get a rough idea of how much memory is needed in your pool ): Min Memory in Pool = Java_Heap_Size + ((JIT_Heap_Size + JVM Heap)/2) 14

Disk arms Is the bottleneck with the number of disk arms? As was mentioned earlier, if you have insufficient memory, the disk arm usage will be high in a Java environment This is a problem with the amount of memory, not the number of disk arms. Rule out a memory problem with the JVM first If there is database access from the JVM, this can affect the disk arm usage In some cases, lower values of faulting is perfectly normal. The smaller number of disk arms, the lower the acceptable rate of paging. It does not look like it is an issue with CPU, Memory, or disk arms What else can it be? 15

Another possibility: Java locks There could be a Java locking issue This is a scaling issue that only shows up when a load is applied to the system. A single user test usually will not show this problem Running IBM Technology for Java - J9 JVM: Use JVMTI or WRKJVMJOB to view Java locking issues. This will give lock information such as: When a java lock is requested stack information is gathered How long the thread has to wait for the java lock. Running Classic JVM: Run a PEX Java lock trace and analyze the results with PTDV. This will give lock information such as: When a java lock is requested stack information is gathered How long the thread has to wait for the java lock. Single threaded vs Multithreaded Java contention Single thread User thread Time User thread Multiple threads User thread User thread Start Mutual Exclusion Start Mutual Exclusion End Mutual Exclusion End Mutual Exclusion Start Mutual Exclusion Start Mutual Exclusion End Mutual Exclusion Start Mutual Exclusion End Mutual Exclusion Key running Waiting Running with lock 16

Is the bottleneck on the processor? Another possibility: Software Queuing (The correct way) System i HTTP Application Server Backend ILE Pgm Database Job Violation of Queuing theory System i Is the bottleneck on the processor? HTTP Application Server Backend ILE Pgm Database Job 17

Additional Resources (WhitePapers) WebSphere performance guide for developers on the System i platform http://www-03.ibm.com/servers/enable/site/education/wp/c572/c572.pdf Java on iseries performance guide for developers http://www-03.ibm.com/servers/enable/site/education/wp/8f56/8f56.pdf Collecting and analyzing a Java object creation trace on the IBM eserver iseries platform http://www-03.ibm.com/servers/enable/site/education/wp/8d46/8d46.pdf Collecting and analyzing an IBM iseries TPROF trace http://www-03.ibm.com/servers/enable/site/education/wp/9a1a/9a1a.pdf An IBM WebFacing Tool performance study http://www-03.ibm.com/servers/enable/site/education/wp/9ece/9ece.pdf Tuning garbage collection for Java and WebSphere running classic JVM on i5/os http://www-03.ibm.com/servers/enable/site/education/wp/c566/c566.pdf Conclusion This section has given a methodology to help determine where a performance issue lies Does the problem exist outside of System i or the iseries? PC or Network Do you have your system and job setup conducive for good performance? Does the bottleneck exist with the Processor, Memory or disk arms? Other examples of where bottlenecks can exist: Do you have Java locking issues within your application? Do you have correct queuing setup throughout your application layers? 18

Trademarks and Disclaimers 8 IBM Corporation 1994-2008. All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at http://www.ibm.com/legal/copytrade.shtml. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Prices are suggested U.S. list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography. Photographs shown may be engineering prototypes. Changes may be incorporated in production models. 19

Trademarks The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both. Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States. For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml: *, AS/400, e business(logo), DBE, ESCO, eserver, FICON, IBM, IBM (logo), iseries, MVS, OS/390, pseries, RS/6000, S/30, VM/ESA, VSE/ESA, WebSphere, xseries, z/os, zseries, z/vm, System i, System i5, System p, System p5, System x, System z, System z9, BladeCenter The following are trademarks or registered trademarks of other companies. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-ibm products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. 20