Discovering Performance Bottlenecks with the SAP JVM Profiler and SAP Memory Analyzer

Transcription

1 Discovering Performance Bottlenecks with the SAP JVM Profiler and SAP Memory Analyzer Boris Magocsi Regional Implementation Group (RIG) June 17, 2009

3 Background: Java Development Today Software Development Has Evolved Use of frameworks and tools, less development from scratch: Web Dynpro, composition tools, Web Services Heavy reuse of existing components and libraries Relying on Java memory management Manipulation of large data sets Typical problems Temporary large memory footprint Memory leaks High CPU utilization High load on the JVM s garbage collector (increasing the CPU utilization) SAP AG All rights reserved. / Page 3

4 SAP JVM Profiler and Memory Analyzer Tools Consequences Limited scalability applications cannot be used by many users in parallel Server breakdowns due to memory leaks Lessons Learned CPU and memory are scarce resources Every Java developer should minimize the resource consumption of applications Remedies Use the SAP JVM Profiler to analyze the memory footprint of your application identify runtime bottlenecks in your application assign the origin of performance or memory problems to users or utilized frameworks Use the SAP Memory Analyzer to perform static analysis and find memory leaks in your application SAP AG All rights reserved. / Page 4

6 SAP Memory Analyzer / Eclipse Memory Analyzer Eclipse-based application initially developed by SAP Donated as open source project Install Eclipse Memory Analyzer + NetWeaver Extensions Rich feature set takes you beyond heap walking Smart grouping / sorting functions Dominator tree for faster analysis Aware of NetWeaver HTTP Sessions, SCAs, Classloaders and Caches Production Analysis Support Heap dump parsing allows for efficient analysis on 32-bit platforms Sun, SAP, HP JDK support for >= 1.4.2_12 and 1.5.0_7 IBM JDK support via plug-in for SR 12+ NetWeaver 04, 7.0 and 7.1 releases SAP AG All rights reserved. / Page 6

9 What is a Leak? Leaks vs High Memory Consumption Objects inadvertently reachable from GC root Particularly problematic for long-running applications (server-based!) Most common cause of OutOfMemoryError: large data sets (as opposed to leak ) SAP AG All rights reserved. / Page 9

10 Heap Dump Content All Objects Class, fields, primitive values and references Potentially sensitive data All Classes Classloader, name, super class, static fields All Classloaders Defined classes Garbage Collection Roots Objects defined to be reachable by the JVM SAP AG All rights reserved. / Page 10

11 A Garbage Collection Root is an object which is defined to be reachable by the JVM: System Class: Class loaded by system class loader, e.g. java.lang.string Java Local: Local variable, i.e. method input parameters or locally created objects of methods still on the stack of a thread Busy Monitor: Everything you have called wait() or notify() on or you have synchronized on Thread Block: Started but not stopped threads JNI Local: Local variable in native code JNI Global: Global variable in native code Native Stack: In or out parameters in native code. Frequently seen as some methods have native parts and the objects handled as method parameters become GC roots, e.g. parameters used for file/network I/O methods or reflection SAP AG All rights reserved. / Page 11

13 Dominators and Dominator Tree An object X dominates an object Y if every path in the object graph from the start (or the root) node to Y must go through X The immediate dominator X of some object Y is the dominator closest to the object We build a dominator tree out of the object graph. In the dominator tree each object is the immediate dominator of its children The edges in the dominator tree do not directly correspond to object references from the object graph SAP AG All rights reserved. / Page 13

14 SAP 2008 / SAP TechEd 08 / COMP203, Page 14 DEMO Static Memory Analysis

15 Obtaining a Heap Dump Available in 1.4.2_12 and 5.0_ 7 and 6.0 upwards -XX:+HeapDumpOnOutOfMemoryError Alternatives to get it on demand -XX:+HeapDumpOnCtrlBreak jmap -dump:format=b,file=<filename.hprof> <pid> JConsole SAP Memory Analyzer / JVMMON / (MMC) SAP AG All rights reserved. / Page 15

16 Features I Want to Mention Aggregate views: group by class dealing with millions of Strings OQL for finding your data / user names / sessions SELECT * from java.lang.string s where tostring(s) like.*username.* Go in via HTTP session Show retained set by package (see retained heap of custom code!) Group by classloader to identify deployment unit SAP AG All rights reserved. / Page 16

17 Summary of Common Issues Very large data sets in production environments (sizing mismatch) Inadequate load testing Improper (aggressive) caching Large Web Dynpro model data in long-running applications Leaks are less common than the above SAP AG All rights reserved. / Page 17

21 SAP JVM Profiler and Memory Analyzer Tools Consequences Limited scalability applications cannot be used by many users in parallel Server breakdowns due to memory leaks Lessons Learned CPU and memory are scarce resources Every Java developer should minimize the resource consumption of his application Remedies Use the SAP JVM Profiler to analyze the memory footprint of your application identify runtime bottlenecks in your application assign the origin of performance or memory problems to users or utilized frameworks Use the SAP Memory Analyzer to perform static analysis and find memory leaks in your application SAP AG All rights reserved. / Page 21

22 SAP JVM Profiler Integration into Development Cycle When to search for resource problems? Find resource problems as early as possible You can easier relate resource consumption effects to changes in your application A from scratch analysis of a full business application can be overwhelming SAP 2008 / SAP TechEd 08 / COMP203, Page 22

23 SAP JVM Profiler Overview Unique SAP JVM Profiler Benefits Fully integrated with SAP NetWeaver Developer Studio 7.1 and Eclipse No prerequisites, configuration, or restart required just start it on demand No runtime overhead if profiling is switched off Moderate memory and CPU overhead when switched on Works reliably with large Java applications Debugging and profiling run in parallel, reuses existing debugging infrastructure Integrated with the SAP NetWeaver Application Server Java 7.1 Download the SAP JVM Profiler Now Previews available at: Everything else you need is a SAP JVM + a SAP NetWeaver Developer Studio or Eclipse Final product shipment with the next SAP NetWeaver release SAP 2008 / SAP TechEd 08 / COMP203, Page 23

24 SAP JVM Profiler Basic Terms (I) Performance Hotspot Analysis Samples in regular intervals the current execution stack of each Java application thread Provides statistics about the execution time of a Java method Self Time time spent in the method itself Total Time time spent in the method including all calls to other methods Statistical technique, overhead ~ % SAP 2008 / SAP TechEd 08 / COMP203, Page 24

25 SAP JVM Profiler Basic Terms (II) Allocation Analysis Tracks each object (= memory) allocation in a Java application Provides statistics about object allocations per class and per method Self Bytes / Self Objects objects allocated in the method itself Total Bytes / Total Objects objects allocated in the method including all calls to other methods Exact technique, overhead ~ % SAP 2008 / SAP TechEd 08 / COMP203, Page 25

26 SAP JVM Profiler Basic Terms (III) Remote and File Analysis Via a remote analysis a Profiler user can connect to a running Java VM on a local or remote computer Results of a remote analysis can directly be analyzed or saved in a file Via a file analysis a Profiler user can analyze a saved Profiler result file Snapshot Bundle of profiling information for a given timeframe To filter for specific entities (threads, user names, ) sub snapshots can be created SAP 2008 / SAP TechEd 08 / COMP203, Page 26

27 SAP 2008 / SAP TechEd 08 / COMP203, Page 27 DEMO Runtime Analysis

28 SAP JVM Profiler Performance Hotspot Analysis Procedure (I) Connect Warm-up application Turn analysis on Execute application Turn analysis off Check Check Method Statistic Inline Inline JDK JDK classes (adds (adds the the runtime runtime in in JDK JDK classes to to calling calling application classes) Bottom up up search search Sort Sort by by Self Self Time Time and and search search for for expensive methods For For expensive methods open open the thecalled CalledMethods statistic Sort Sort tree tree by by Total Total Time Time and and drill drill down down along along the the path path of of most most runtime Find Find methods suffering of of expensive methods via via the thecalling CallingMethods statistic statistic Sort Sort tree tree by by Total Total Time Time and and open open the the tree tree along along the the path path of of most most runtime Top Top down down search search Sort Sort by by Total Total Time Time and and search search for for expensive methods of of your your application Drill Drill down down via via the thecalled CalledMethods statistic to to find find reasons for for the the expense SAP 2008 / SAP TechEd 08 / COMP203, Page 28

29 SAP JVM Profiler Performance Hotspot Analysis Procedure (II) Typical Pitfalls Your application is not warmed up The runtime behavior in your measurement does not resemble a typical execution The runtime of your application is too short The Profiler could generate only a few samples (say, less than 100). Your analysis may not be significant You ran into a breakpoint The Profiler reuses the debugging connection to the SAP JVM. Active breakpoints are respected and may lead to unwanted delays and untypical runtime behavior You used the Performance Hotspot Analysis & the Allocation Analysis concurrently The overhead of the Allocation Analysis is high and therefore invalidates the performance measurement results SAP 2008 / SAP TechEd 08 / COMP203, Page 29

30 SAP JVM Profiler Object Allocation Dilemma Q: Why do I have to check the memory consumption of my application? Are object allocations evil in Java? A: No. Allocations are inevitable. A Java VM is highly optimized to make the allocation operation very cheap The used generational memory model is optimized for objects dying young There s no alternative in Java Own buffering infrastructures for Java objects reduce the effectiveness and decision options of the built-in garbage collector Means to control the lifetime of Java objects out of Java applications can be more expensive than just allocating (and garbage collecting) the object Q: So where is the problem? A: Excessive use of (temporary) objects! Does not scale with many users Objects needed only for the request cannot be kept in the young generation Increases the effort for performing a garbage collection There are no user specific limits: One user can consume all memory SAP 2008 / SAP TechEd 08 / COMP203, Page 30

31 SAP JVM Profiler Object Allocation Dilemma Ways Out Find out, how much memory your application really needs Interesting: How much memory is utilized in your Java methods? Often more interesting: How much memory is used elsewhere? Surrounding frameworks, utilized JDK classes, server runtime APIs, Some seemingly innocent methods can be memory hogs Cut off execution paths with extensive memory allocation Move code out of loops or out of frequently called methods Avoid external API calls or search for cheaper alternatives but keep maintainability Think about coding alternatives for your code Different data structures Lazy initialization of instance variables Cache objects locally (but only during a request) SAP 2008 / SAP TechEd 08 / COMP203, Page 31

32 SAP 2008 / SAP TechEd 08 / COMP203, Page 32 DEMO Allocation Analysis

33 SAP JVM Profiler Allocation Analysis Procedure (I) Connect Turn analysis on Execute application Turn analysis off Check Check Method Statistic Check Allocated Object Statistic Inline Inline JDK JDK classes Bottom up up search search Sort Sort by by column column Self Self Bytes Bytes and and search search for for expensive methods For For expensive methods open open the thecalled Methods statistic (tree) (tree) Sort Sort tree tree by by Total Total Bytes Bytes and and open open the the tree tree along along the the path path of of most most memory utilization Top Top down down search search Sort Sort by by column column Total Total Bytes Bytes and and search search for for methods belonging to to your your application Drill Drill down down via viacalled Methods statistic (tree) (tree) similar similar to to above above SAP 2008 / SAP TechEd 08 / COMP203, Page 33

34 SAP JVM Profiler Allocation Analysis Procedure (II) Connect Turn analysis on Execute application Turn analysis off Check Method Statistic Check Check Allocated Object Object Statistic Bottom up up search search Select Select the the classes with with the the biggest biggest memory consumption Drill Drill down down via via the theallocating Methods (from (from top) top) statistic Check Check if if you you find find methods from from your your application Plausibility check check Show Show the the Number of of Objects column column Check Check the the amount of of objects objects created created for for your your own own classes: Does Does this this meet meet your your expectations? SAP 2008 / SAP TechEd 08 / COMP203, Page 34

35 SAP JVM Profiler Tips & Tricks (I) Inline JDK classes The memory or runtime utilized in JDK classes is added to the calling application classes ( Self Bytes or Self Time columns) Shift click Pressing the shift key while clicking skips methods irrelevant regarding resource usage Show line numbers Click the green triangle to see, where you can find the method call in the source code SAP 2008 / SAP TechEd 08 / COMP203, Page 35

36 SAP JVM Profiler Tips & Tricks (II) Adaptive Allocation Trace Statistical approach for very long running applications (running several hours) Sub-snapshots for users & threads Filtered snapshots, helpful when other users were working on the system concurrently and you want to investigate only your application SAP 2008 / SAP TechEd 08 / COMP203, Page 36

37 Resources (Where to Get the Tools) Download the JVM Profiler from SDN Install the Memory Analyzer using instructions on SDN Memory Analyzer Forum on SDN (common issues discussed) SAP AG All rights reserved. / Page 37

40 Definition and Halftone Values of Colors SAP Blue SAP Gold SAP Light Gray SAP Gray SAP Dark Gray RGB 4/53/123 RGB 240/171/0 RGB 204/204/204 RGB 153/153/153 RGB 102/102/102 Primary color palette 100% Dove Petrol Warm Green Warm Red Violet/Mauve RGB 68/105/125 RGB 21/101/112 RGB 85/118/48 RGB 119/74/57 RGB 100/68/89 Secondary color palette 100% RGB 96/127/143 RGB 98/146/147 RGB 110/138/79 RGB 140/101/87 RGB 123/96/114 85% RGB 125/150/164 RGB 127/166/167 RGB 136/160/111 RGB 161/129/118 RGB 147/125/139 70% RGB 152/173/183 RGB 154/185/185 RGB 162/180/141 RGB 181/156/147 RGB 170/152/164 55% RGB 180/195/203 RGB 181/204/204 RGB 187/200/172 RGB 201/183/176 RGB 193/180/189 40% Cool Green Ocher Cool Red Warning Red RGB 73/108/96 RGB 129/110/44 RGB 132/76/84 Tertiary color palette 100% RGB 158/48/57 RGB 101/129/120 RGB 148/132/75 RGB 150/103/110 85% RGB 129/152/144 RGB 167/154/108 RGB 169/130/136 70% RGB 156/174/168 RGB 186/176/139 RGB 188/157/162 55% RGB 183/196/191 RGB 205/197/171 RGB 206/183/187 40% SAP AG All rights reserved. / Page 40

41 Copyright 2009 SAP AG All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iseries, pseries, xseries, zseries, eserver, z/vm, z/os, i5/os, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP Business ByDesign, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects S.A. in the United States and in other countries. Business Objects is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warrant. SAP AG All rights reserved. / Page 41