Java SE Performance Tuning Revision A

Transcription

1 Java SE Performance Tuning Revision A SEM-DTJ-380-LA

2 Copyright 2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Sun, Sun Microsystems, the Sun logo, the Duke logo, Java, JavaServer Pages, JSP, Java HotSpot, Java VisualVM, Sun Fire T1000, Sun Fire T2000, UltraSparc, NetBeans, NetBeans Profiler, Sun Studio, and JavaScript are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun's licensees who implement OPEN LOOK GUIs and otherwise comply with Sun's written license agreements. Federal Acquisitions: Commercial Software Government Users Subject to Standard License Terms and Conditions Export Laws. Products, Services, and technical data delivered by Sun may be subject to U.S. export controls or the trade laws of other countries. You will comply with all such laws and obtain all licenses to export, re-export, or import as may be required after delivery to You. You will not export or re-export to entities on the most current U.S. export exclusions lists or to any country subject to U.S. embargo or terrorist controls as specified in the U.S. export laws. You will not use or provide Products, Services, or technical data for nuclear, missile, or chemical biological weaponry end uses. DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. Export Control Classification Number (ECCN) assigned:ear99

3 Copyright 2008 Sun Microsystems Inc., 4150 Network Circle, Santa Clara, California 95054, Etats-Unis. Tous droits ré servé s. Ce produit ou document est proté gé par un copyright et distribué avec des licences qui en restreignent l'utilisation, la copie, la distribution, et la dé compilation. Aucune partie de ce produit ou document ne peut ê tre reproduite sous aucune forme, par quelque moyen que ce soit, sans l'autorisation pré alable et é crite de Sun et de ses bailleurs de licence, s'il y en a. Le logiciel dé tenu par des tiers, et qui comprend la technologie relative aux polices de caractè res, est proté gé par un copyright et licencié par des fournisseurs de Sun. Sun, Sun Microsystems, the Sun logo, the Duke logo, Java, JavaServer Pages, JSP, Java HotSpot, Java VisualVM, Sun Fire T1000, Sun Fire T2000, UltraSparc, NetBeans, NetBeans Profiler, Sun Studio et JavaScript sont des marques de fabrique ou des marques dé posé es de Sun Microsystems, Inc. aux Etats-Unis et dans d'autres pays. L'interfaces d'utilisation graphique OPEN LOOK et Sun a é té dé veloppé e par Sun Microsystems, Inc. pour ses utilisateurs et licencié s. Sun reconnaît les efforts de pionniers de Xerox pour larecherche et le dé veloppement du concept des interfaces d'utilisation visuelle ou graphique pour l'industrie de l'informatique. Sun dé tient une licence non exclusive de Xerox sur l'interface d'utilisation graphique Xerox, cette licence couvrant é galement les licencié s de Sun qui mettent en place l'interface d'utilisation graphique OPEN LOOK et qui en outre se conforment aux licences é crites de Sun. Lé gislation en matiè re dexportations. Les Produits, Services et donné es techniques livré s par Sun peuvent ê tre soumis aux contrô les amé ricains sur les exportations, ou à la lé gislation commerciale dautres pays. Nous nous conformerons à lensemble de ces textes et nous obtiendrons toutes licences dexportation, de ré -exportation ou dimportation susceptibles dê tre requises aprè s livraison à Vous. Vous nexporterez, ni ne ré -exporterez en aucun cas à des entité s figurant sur les listes amé ricaines dinterdiction dexportation les plus courantes, ni vers un quelconque pays soumis à embargo par les Etats-Unis, ou à des contrô les anti-terroristes, comme pré vu par la lé gislation amé ricaine en matiè re dexportations. Vous nutiliserez, ni ne fournirez les Produits, Services ou donné es techniques pour aucune utilisation finale lié e aux armes nuclé aires, chimiques ou biologiques ou aux missiles. LA DOCUMENTATION EST FOURNIE "EN L'ETAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L'APTITUDE A UNE UTILISATION PARTICULIERE OU A L'ABSENCE DE CONTREFAÇ ON.

4 Course Objectives Incorporate monitoring, profiling and tuning into the app Monitor the Operating System (OS) layer: Central Proce Monitor the Java Virtual Machine (JVM) and application Profile the OS, JVM and application layers Tune garbage collection (GC) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 4

5 Course Objectives Examine and manage the Just in Time (JIT) compiler Examine JVM ergonomics Examine 64 bit JVMs Tune the JVM for multi-core platforms Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 5

6 Course Overview: What to Expect Master Java SE performance monitoring by learning: > What and where to performance monitor > What to profile and what tools work the best for different use cases > Commonly observed patterns indicating performance issues > How Java HotSpot garbage collectors work and how to tune them > What you need to know about the JIT compiler > What is JVM ergonomics and how it works > What you need to know about 64-bit JVMs > How to tune the JVM for specific hardware platforms Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 6

8 Course Overview: The Nature of Perform Performance tuning is largely an art. There is no one approach that is always necessarily the There are performance issues which will require very sp Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 8

9 Course Module Outlines Module 1: Examining Performance Tuning > Distinguishing between monitoring, profiling and tunin > Incorporating monitoring, profiling and tuning into the Module 2: Monitoring the OS Layer > Monitoring CPU utilization > Monitoring network performance > Monitoring disk input output (I/O) > Monitoring memory utilization > Monitoring process behavior Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 9

10 Course Module Outlines Module 3: Monitoring the JVM and Application Layers > Examining generational collector architectures > Monitoring GC > Monitoring the JVM > Monitoring the application Module 4: Profiling the OS, JVM and Application Layers > Examining profiling tools > Profiling CPU usage > Profiling the heap and memory usage > Detecting lock contention Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 10

11 Course Module Outlines Module 5: Tuning GC > Tuning collector generation sizes > Selecting the collector that best fits application characteristics > Examining practices that negatively impact GC performance Module 6: Examining and Managing the JIT compiler > Examining choices of JIT compilers > Tuning the JIT compiler > Creating micro benchmarks Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 11

12 Course Module Outlines Module 7: Examining Ergonomics > Examining JVM ergonomics behavior Module 8: Using 64 bit JVMs > Examine the issues associated with using 64 bit JVMs > Identify application characteristics that suit 64 bit JVMs > Tuning 64 bit JVM for different application requirements Module 9: Optimize the JVM for Multi-core platforms > Examining JVM features that can leverage multi-core archite > Optimize the JVM for various multi-core architectures > Tuning the JVM for the Sun Fire T1000/T2000 platform Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 12

13 Module 1: Examining Performa SEM-DTJ-380-LA

16 Definitions: Performance Monitoring An act of non-intrusively collecting or observing perform In most cases, a preventative or proactive type of ac Can be performed in production, or qualification, or dev Helps identify or isolate potential issues without having Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 16

17 Definitions: Performance Monitoring Often times monitoring crosses over into trouble-shootin Training focus is on performance monitoring aspects ra Training is focused more on techniques and tools relate Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 17

18 Definitions: Performance Profiling An act of collecting or observing performance data from Usually more intrusive than monitoring. Usually a narrower focus than monitoring. In general a reactive type of activity. Could be a proactiv Seldom performed in production environments. Commonly done in qualification, testing or development Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 18

19 Definitions: Performance Tuning An act of changing tune-ables, source code and/or conf Usually results from monitoring and/or profiling activities Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 19

24 Objectives Monitor CPU usage Monitor network I/O Monitor disk I/O Monitor virtual memory usage Monitor processes including lock contention Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 24

25 What to Expect What level of the software stack to monitor > Operating system level > JVM level (covered in module 3) > Application level (covered in module 3) What information to monitor > What to monitor is covered per component at a given > Example: At OS level, monitor CPU usage What tools to use Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 25

26 What to Monitor in the OS Level CPU utilization Network traffic Disk I/O Virtual memory usage Processes and kernel locks Monitoring definition recap: An act of non-intrusively colle Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 26

27 Monitoring CPU Usage: Overview Rationale for monitoring CPU usage > Get big picture view of CPU demand > Get per process measurement of CPU utilization Measurements of CPU usage > User (usr) time > System (sys) time > Idle time > Voluntary context switching (VCX) > Involuntary context switching (ICX) Tools for monitoring CPU usage Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 27

28 CPU monitoring: What to look for High sys / kernel cpu time > High sys / kernel cpu time indicates a lot of cpu cycles are spent in the kernel. > A reduction in kernel cpu time will give more cpu time to the application. > Also, high sys cpu time could indicate shared resource contention, (in other words, locking). More on lock contention later. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 28

29 CPU monitoring: What to look for Idle cpu > On multi-threaded applications and multi-core systems, idle cpu can be an indicator of an application's inability to scale. > Combination of high sys or kernel CPU utilization and idle CPU could indicate shared resource contention as the scalability blocker. > Applicable to all operating systems, i.e. Windows, Linux and Solaris Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 29

32 CPU monitoring: What to look for High VCX > High voluntary context switching can be an indication an application is experiencing lock contention as a result of the JVM implements or supports Java locking such as synchronized methods or block, or Read/Write locks in the java.util.concurrent.locks package. High ICX > There are some applications (most often seen in OLTP / database systems) which can benefit by switching to the Solaris FX (fixed) scheduler as a means to reduce context switching. > For Solaris, use priocntl to set FX scheduling: > $ priocntl c FX -s <PID> [<PID>... ] for disc next week Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 32

33 Tools For Monitoring: CPU Usage Tools to monitor cpu utilization > vmstat (Solaris & Linux) > mpstat (Solaris) > prstat (Solaris) > top (Linux, prefer prstat on Solaris) > Task Manager (Windows) > Performance Monitor (Windows) > Windows Resource Manager (Windows Server) > xosview (Linux) > cpubar (Solaris Performance Tools CD) > iobar (Solaris Performance Tools CD) > dtrace (Solaris) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 33

34 Tools for Monitoring CPU Usage: vmstat Use vmstat to obtain summaries of CPU usage Data of interest: us user time; sy system time; id idle time Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 34

36 Tools For Monitoring: mpstat usr user time; sys system time; idl idle time csw context switches; icsw involuntary context switches Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 36

38 CPU monitoring: prstat -m Example of CPU utilization microstate information measured per process Data of interest: USR, SYS, VCX, ICX Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 38

39 Tools For Monitoring: prstat -Lm Example of CPU utilization including microstate information measured per light weight process. (USR, SYS, VCX, ICX) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 39

40 CPU Monitoring: Solaris - cpubar Data of interest: 0 CPU 0; 1 CPU 1 avg Average; --- Moving average; Green User; Red System; Blue Idle Available on Solaris Performance Tools 3.0 CD, or download from: Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 40

41 CPU monitoring: How to map Java Threads to Light Weight Processes (LWPs) Use HotSpot's jps command to find the process ids of all running Java processes on your machine. Use Solaris prstat -Lm, or prstat -Lmp <pid> to locate the LWP id(s) consuming the most cpu (usr or sys). Use HotSpot's jstack to find the executing threads taking the cpu (usr and sys) time. Map the LWPID to jstack's thread id. > LWPID is in the far right column of prstat. > Look for jstack's corresponding 'nid', reported in hex. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 41

43 CPU monitoring: Linux - vmstat Use vmstat to obtain summaries of CPU usage Data of interest: us user time; sy system time; id idle time Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 43

45 CPU Monitoring Using: pidstat %user - % of user level (application) task %system - % of system level (kernel) task %CPU total % of CPU time cswch/s total voluntary context switches/second nvcswhc/s total involuntary context switches/second Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 45

47 Monitoring Network I/O: Overview Data of interest > Network utilization in terms of Transaction Control Protocol (TCP) statistics and established connections Tools to monitor network I/O > netstat (Solaris & Linux) > Performance Monitor (Windows) > dtrace (Solaris) > nicstat (Solaris Performance Tools CD) > tcptop (Dtrace Toolkit) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 47

48 Monitoring Network I/O: Using tcptop tcptop can show per process TCP statistics The screen capture shows 'rcp' generating 115 kb of traffic Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 48

49 Monitoring Network I/O: Using nicstat nicstat displays network statistics Notice wavs, write average size, during four intervals is about 1420 bytes, the Maximum Transmission Unit (MTU) size. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 49

50 Monitoring Disk I/O: Overview Data of interest > Number of disk accesses > Latency and average latencies Tools to monitor disk I/O > iostat (Solaris & Linux) > iotop (Solaris & Linux) > pidstat (Linux) > Performance Monitor (Windows) > dtrace (Solaris) > iobar (Solaris Performance Tools CD) Disk caches Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 50

51 Disk I/O Monitoring Tools: iostat, iobar, iotop, pidstat iostat reports per disk, text output iobar reports per disk, gui output iotop reports per process statistics, text output pidstat reports per process statistics, text output Data of interest > number of disk accesses, latency, average latencies Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 51

52 Monitoring Disk I/O: iotop example iotop reporting at a 5 second interval DISKTIME reported in microseconds CMD find, is keeping disk cmdk0 busy almost 60% of time during the 5 second interval Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 52

54 Monitoring Disk IO: Disk Cache Why not enable disk cache? What's the risk? On some Sun branded systems, disk cache may be disabled by default. Linux & Windows systems usually have it enabled. Disk cache being disabled depends on the Sun branded model and how recent the model. Ask before disabling. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 54

55 Monitoring Virtual Memory: Overview Observe paging to identify swapping > Pages in (pi) > Pages out (po) > Scan rate (sr) Fixing the swapping problem Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 55

57 Monitoring Virtual Memory: Tools Tools to monitor memory paging & usage > vmstat (Solaris & Linux) > prstat (Solaris) > top (Linux prefer prstat on Solaris) > Performance Monitor (Windows) > dtrace (Solaris) > cpubar (Solaris Performance Tools CD) > meminfo (Solaris Performance Tools CD) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 57

59 Virtual Memory: Swapping Example Data of interest > pi pages in; po pages out; sr page scan rate > Watch for high scan rate (see rows 3 to 6), or increasing trend. Low scan rate is ok if they occur infrequently. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 59

60 Monitoring Virtual Memory : cpubar Data of interest > p/s - pages per second, sr - scan rate > Watch for high scan rate, or increasing trend. Low scan rate is ok if they occur infrequently. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 60

61 Virtual Memory: Fixing the Swapping Problem Smaller Java heap sizes Add physical memory Reduce number of applications running on the machine Any one, or any combination of the above will help Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 61

62 Monitoring Processes: Overview Data of interest > Footprint size > Number of threads and thread state > CPU usage > Runtime stack > Context switches > Lock contention Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 62

63 Monitoring Processes: Questions of Interest Why is footprint size, number of threads, thread state, lock contention and context switching important to monitor? What does lock contention and/or context switching look like on Solaris? How can you find the lock or locks causing problems? How can you address the thread context switching problem? Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 63

64 Monitoring Processes: Tools ps (Solaris & Linux) vmstat (Solaris & Linux) mpstat (Solaris) prstat (Solaris) pidstat (Linux) Performance Monitor (Windows) top (Linux prefer prstat on Solaris) dtrace (Solaris) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 64

65 Monitoring Processes: prstat -Lm Data of interest: Number of threads per process (sum of LWPIDs per PID); CPU usage (USR, SYS); Locks (LCK); Context switches (VCX and ICX); Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 65

67 Monitoring Processes: Kernel Data of interest > kernel cpu utilization, locks, system calls, interrupts, migrations, run queue depth Tools to monitor the kernel > vmstat (Linux & Solaris) > mpstat (Solaris) > lockstat & plockstat (Solaris) > Performance Monitor (Windows) > dtrace (Solaris) > intrstat (Solaris) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 67

68 Kernel Monitoring: What to look for Why are high sys / kernel cpu, run queue depth, lock contention, migrations and context switching important to monitor? > Discussion What do they indicate when each is observed? > Discussion How to do you address each of these problems? > Discussion Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 68

69 Kernel Monitoring Using: vmstat Data of interest: Kernel cpu utilization, run queue depth (r column represents number of runnable kernel threads) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 69

70 Kernel Monitoring Using: mpstat Data of interest: Kernel cpu utilization(sys); locks(smtx); system calls(syscl); interrupts(intr); migrations(migr); context switches (csw); involuntary context switches (iscw) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 70

71 Kernel Monitoring Using : prstat -Lm Data of interest: Kernel cpu utilization(sys), locks(lck), system calls(scl), voluntary context switches(vcx); involuntary context switches (icx). Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 71

72 Kernel Monitoring Using: pidstat %user - % of user level (application) task %system - % of system level (kernel) task %CPU total % of CPU time cswch/s total voluntary context switches/second nvcswhc/s total involuntary context switches/second Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 72

73 Module 3: Monitoring the JVM a SEM-DTJ-380-LA

74 Objectives Examine HotSpot generational garbage collectors Monitor the JVM > GC > JIT compiler Monitor the application > Application throughput > Application responsiveness Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 74

75 What to Expect Overview of generational GC How to monitoring the JVM > GC: Which tools to use and what to monitor > JIT compiler: Which tools to use and what to monitor How to monitoring the application > Application throughput > Application responsiveness Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 75

76 What to Monitor: JVM There are two major areas to monitor at the JVM level of the software stack. > Garbage collector >The portion of the JVM responsible for freeing memory no longer utilized by application logic. The magic that lets programmers not have to worry about managing memory. >Garbage collection involves traversing Java heap spaces where application objects are allocated and managed by the JVM's garbage collector. > JIT compilation >The portion of the JVM responsible for Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 76

77 Monitoring GC Examining the GC Basics > Young generation > Tenured generation > Permanent generation Data of interest > Frequency and duration of collections > Java heap usage > Number of application threads Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 77

79 GC Basics: The Major Spaces Young generation > Further divided into: > Eden >A from survivor space >A to survivor space > Java objects are allocated in eden Tenured (old) generation Permanent generation Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 79

80 GC Basics: Young Generation When eden space is full, minor garbage collection event occurs. Live objects in eden space are copied to a survivor space. Additionally, objects in the from survivor space are copied to survivor space. Each object which survives a garbage collection has its age incremented. Objects exceeding a JVM defined age threshold are promoted to the tenured (old) generation space. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 80

81 GC Basics: Young Generation If to survivor space is too small to hold surviving Eden and from survivor space objects, objects will be promoted to tenured space. > This is a situation which can potentially lead to performance issues. > Short lived objects getting promoted to old generation will require a Full GC (stop the world kind of event) or rely on a tenured spaced concurrent collector to remove them from the Java heap. Both choices have their own consequences. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 81

82 GC Basics: Young Generation Minor garbage collection events can be: > Stop the world single threaded events Stop the world single threaded events block all Java application threads to perform the garbage collection event. > Stop the world multi-threaded events Stop the world multi-threaded events block all Java application threads, but is multi-threaded. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 82

83 GC Basics: Young Generation Single threaded garbage collector is used with the default collector. Multi-threaded garbage collector, also called a parallel collector, can be used with either the throughput collector or concurrent collector. > -XX:+UseParallelGC, throughput collector > -XX:+UseParNewGC, concurrent collector >Throughput and concurrent collectors talked about in more detail later. Minor garbage collections are very quick compared to full garbage collection events. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 83

84 GC Basics: Tenured (old) Generation Terms old and tenured generation commonly used inter-changeably Contains objects which have survived minor collections and copied to tenured space > Some edge cases where objects are directly allocated in tenured generation. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 84

85 GC Basics: Tenured (old) Generation Objects are garbage collected by one of the following: > Stop the world single threaded full garbage collection > Stop the world multi-threaded full garbage collection > Stop the world means, all Java application threads are blocked for the duration of the garbage collection. > Single threaded and (mostly) concurrent garbage collection > Most of the garbage collection occurs concurrently while Java application threads run. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 85

86 GC Basics: Object Life Cycle Allocation through to garbage collection > Object is allocated in Eden space. > If object is in-use (reachable) at minor collection time, then object is copied to survivor space. If object is no longer in use (not reachable) it is garbage collected. > As subsequent minor garbage collections occur, the object is copied from a survivor space to the other survivor space. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 86

87 GC Basics: Object Life Cycle Allocation through garbage collection (continued) > If the object continues to be in-use at subsequent minor garbage collections, at some threshold age, the object will be copied (tenured) to the tenured space. If the object is no longer in use, it will be garbage collected. > Once in the tenured space, either a concurrent collection or full collection will be required to occur in order for the object to be collected when it is no longer in use. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 87

88 Module 3: Demo 1 88

89 GC Basics: Permanent Generation Contains meta-data (objects) required by the JVM to describe the objects used in the application such as class objects, and interned Strings. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 89

90 GC Basics: Permanent Generation Holds objects containing information which describe an application class's or Java SE library class's methods. Populated by the JVM at runtime based on classes in use by the application. Classes may get collected (unloaded) if JVM finds they are no longer needed and space may be needed for other classes. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 90

92 Monitoring GC: The Tools Tools for monitoring GC > -verbose:gc > -XX:+PrintGCTimeStamps > -XX:+PrintGCDetails > -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCApplicationConcurrentTime > jstat, jps > JConsole > Java VisualVM > VisualGC > dtrace (Java HotSpot JDK 6 contains samples) Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 92

93 Monitoring GC: Using -verbose:gc -verbose:gc > [GC 1884K->1299K(5056K), secs] Use -XX:+PrintGCTimeStamps with -verbose:gc > 3.791: [GC 1884K->1299K(5056K), secs] Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 93

94 Monitoring GC: Using -verbose:gc Use -XX:+PrintGCDateStamps (Java 6u4 and later) > T06:12: : [GC 10308K->2725K(101376K), secs] > Where: Date and time is GMT time. Format: YYYY-MM-DD:HH.MM.SS.mmm-ttttt where YYYY = year, MM=month, DD=day of month, HH=hour, MM=minute, SS=seconds, mmm=milliseconds, tttt=time zone offset. -XX:+PrintGCDateStamps and -XX:+PrintGCTimeStamps can be used together > T06:21: : 5.551:[GC 10377K- >2786K(101376K), secs] > PrintGCTimeStamps output after date & time. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 94

95 Monitoring GC: Using -verbose:gc Data of interest > Frequency and duration, heap usage Explain what pattern(s) indicate potential problems. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 95

97 Monitoring GC: Printing GC Details -XX:+PrintGCDetails [GC [DefNew: 490K->64K(960K), secs] 5470K->5151K(7884K), secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [Full GC (System) [Tenured: 5087K->5151K(6924K), secs] 6047K- >5151K(7884K), [Perm : 11178K->11178K(16384K)], secs] [Times: user=0.10 sys=0.01, real=0.10 secs] Data of interest > frequency and duration, heap usage Explain what patterns indicate potential problems? Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 97

99 Monitoring GC: Printing Pause Time -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime Helpful when tuning pause time sensitive applications Useful for identifying odd pause time issues when combined with GC timestamps and GC duration. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 99

101 Monitoring GC: Using jps jps > Included in the HotSpot JDK > Capable of local and remote monitoring > Command line utility to find running java processes. > jps [-q] [-mlvv] [<hostid> where <hostid> = <hostname>[:<port>] > Quick demo (local and remote monitoring) > See jps man page on java.sun.com for details on -q, -mlvv options Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 101

102 Monitoring GC: Using jstat jstat > included in the HotSpot JDK. > command line utility. > jstat -<option> [-t] [-h<lines>] <vmid> [<internal> [<count>]] > Garbage collection option(s): >-gc, -gccapacity, -gccause, -gcnew, - gcnewcapacity, -gcold, -gcoldcapacity, - gcpermcapacity, -gcutil > See jstat man page on java.sun.com for details on garbage collection options > Explain what patterns indicate potential Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 102

104 Monitoring GC: Using jstat Beware: When using the Concurrent Mark Sweep (CMS) collector (also known as concurrent collector), jstat reports two full gc events per CMS cycle, obviously misleading. But, young generation stats are accurate with CMS. Quick demo > Running jstat against Java2D demo. > Connect locally and remote > Remember, remote monitoring requires jstatd and a policy file. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 104

105 Monitoring GC: Using jconsole jconsole > Is a monitoring and management GUI console. > Is included in the HotSpot JDK. > Can attach local or remote. > Can monitor internals of a target JVM. > Can monitor multiple JVMs. > Explain what patterns indicate potential problems. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 105

106 Monitoring GC: Using jconsole Provides endless observability > MBean support for >JVM memory usage by memory pool / spaces >Class loading, JIT compilation, garbage collector, runtime, threading and logging >Thread monitor contention > Graphical view of heap memory, threads, cpu usage and class loading Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 106

108 Monitoring GC: Using VisualVM VisualVM: Background and capabilities > Packaged with JDK 6 update 7 > Open source project at > Integrates several existing JDK software tools, lightweight memory and CPU profiling capabilities. >JConsole >Subset of NetBeans Profiler > Includes performance analysis and troubleshooting abilities. >Thread deadlock detection Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 108

109 Monitoring GC: Using VisualVM VisualVM: Extendability > Can be further extended with specific functionality for target application through: additional plug-in or extending an existing plugin. >Possibilities include: GlassFish performance monitoring plugin JavaDB performance monitoring plug-in External vendors such as WebSphere performance monitoring plug-in. > Plugins, enhancements and updates delivered Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 109

112 Monitoring GC: Using VisualGC VisualGC > Standalone GUI or VisualVM plug-in. > Not included in HotSpot JDK. Separate download. > Visually observe garbage collection behavior. (A picture is worth a thousand words). > Also includes classloading and JIT compilation information. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 112

114 Garbage Collectors: GCHisto Currently standalone GUI, VisualVM plug-in under development. Open Source project, Not included in HotSpot JDK, Separate Graphical tool which summarizes GC activity obtained from GC logs Allows comparison of JVM tuning, such as heap sizes or collector types by comparing GC logs. Demo of GCHisto in Module 5. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 114

115 Monitoring JIT Compilation Tools for monitoring JIT Compilation > jstat > JConsole > VisualVM > VisualGC > -XX:+PrintCompilation (can be intrusive) > -XX:+LogCompilation (can be intrusive) > DTrace (HotSpot JDK 6 contains samples) Data of interest > frequency, duration, possible opt / de-opt cycles, failed compilations Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 115

116 Module 3: Demo

117 Monitoring JIT: Using PrintCompilation -XX:+PrintCompilation 1 java.util.properties$linereader::readline (452 bytes) 2 java.lang.string::hashcode (60 bytes) 3 java.lang.string::equals (88 bytes) 3 made not entrant (2) java.lang.string::equals (88 bytes) 4 java.lang.object::<init> (1 bytes) 5 java.lang.string::indexof (151 bytes) 6 java.lang.string::equals (88 bytes) < redoing 3 6 made not entrant (2) java.lang.string::equals (88 bytes) 7 java.lang.string::indexof (151 bytes) 8 java.lang.string::equals (88 bytes) < redoing 6 8 made not entrant (2) java.lang.string::equals (88 bytes) Data of interest > frequency, duration, possible opt / de-opt cycles, patterns of interest Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 117

118 Monitoring JIT: Using LogCompilation -XX:+LogCompilation > Beware, it can be intrusive Will probably need someone from JIT compiler team to analyze it. Data of interest > frequency, duration, possible opt / de-opt cycles Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 118

119 Monitoring JIT: Using.hotspot_compiler File What is the.hotspot_compiler file The.hotspot_compiler file format > exclude A/B/C/D methodname where >A.B.C.D is the fully qualified package and class name and methodname is the method name. > Example: To exclude java.util.hashmap.clear(), specify: >exclude java/util/hashmap clear The.hotspot_compiler file must be placed in the directory where the java command is launched. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 119 > Can also use -XX:CompileCommand=exclude,A/B/C/D,methodName

120 Tools For Monitoring : JIT When to use the.hotspot_compiler file > JIT compiler in an endless loop attempting a heroic optimization which will not converge > JIT compiler in a de-optimization reoptimization cycle > JIT compiler producing 'bad' code resulting in a core dump or other severe problem Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 120

121 Monitoring the Application: Terms Throughput sensitive applications > Have as highest priority the raw throughput of the information or data being processed > Maximize application throughput even at the expense of responsiveness > Will tolerate high pause times in order to maximize throughput > Example: Batch processing applications Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 121

122 Monitoring the Application: Terms Responsiveness sensitive applications > Have as highest priority the servicing of all requests within a predefined maximum time > Raw throughput of data or speed of processing requests are secondary to max response time goal > Are sensitive to GC pause time > Examples: >User input applications such as Web browser or GUI based applications >Financial trading applications >Telecommunication applications Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 122

123 Monitoring the Application: Tools and Data Tools > JConsole (using application Mbeans) > Extend VisualVM with a plug-in to gather Java application data of interest and monitor the application with VisualVM > Application log > GCHisto and btrace plugins for VisualVM > Specialized DTrace scripts Data of interest > Critical application information and instrumentation Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 123

124 Module 4: Performance Profil SEM-DTJ-380-LA

125 Objectives Examining and selecting profiling tools Examining profiling tips for: > CPU profiling > Heap profiling > Memory leak profiling > Lock contention detection Identifying anti-patterns in: > Heap profiles > Method profiles Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 125

126 What to Expect What tools to use to profile > Focus is on Sun Microsystems free and open source tools along with bundled JDK tools > No commercial or external vendor tools are covered An overview of how to use each profiling tool An examination of good fit use cases for each tool Tips for finding source of lock contention and memory leaks Commonly observed patterns in profiles and suggested ways to fix them Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 126

127 Tools For Profiling Java applications Free tools > NetBeans Profiler, subset in VisualVM > > Sun Studio Collector / Analyzer > > jmap / jhat >included in HotSpot JDK Commercial Profilers > Intel VTune > OptimizeIt > YourKit > Not covered in any detail here Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 127

128 Free Profilers: NetBeans Profiler Characteristics: > CPU performance profiling using byte code instrumentation > Low overhead profiling Capabilities: > Method profiling >Select all methods for profiling or specific method(s) > Note: You can limit everything but JDK classes > Memory profiling / heap profiling Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 128

129 Free Profilers: NetBeans Profiler Supported platforms: > Solaris (SPARC & x86) > Linux > Windows > Mac OS X Requirements: > Requires HotSpot JDK 5 or later Download: > Included out-of-the-box in NetBeans IDE 6.0 and later > Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 129

131 Free Profilers: Sun Studio Sun Studio Collector / Analyzer capabilities: > Statistical CPU profiling using JVMTI >Can specify sampling interval, default 1 sec >User and sys cpu time >Inclusive or exclusive method times > Time spent in locks > View Java byte code in User Mode and Machine Mode > View JIT compiler generated assembly code in Expert Mode > Supports specific CPU counter collection Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 131

132 Free Profilers: Sun Studio Sun Studio Collector / Analyzer characteristics: > Easily invoked with 'collect -j on' prefixed to Java command line. Supported platforms: > Solaris (SPARC & x86) and Linux Requirements: > Requires HotSpot JDK 5 or later Additional Information: > > > Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 132

135 Free Profilers: jmap and jhat Used in combination: > jmap produces heap profile > jhat reads and presents the data Additional information: > Shipped with JDK 5 and later > Command line tools > Heap memory profiling > Perm gen statistics > Finalizer statistics > Supported on all platforms Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 135

137 Profiling Tips CPU profiling tips Heap profiling tips Memory leak profiling tips Lock contention profiling tips Profiling tools selection tips Inlining effect Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 137

138 CPU Profiling Tips: Why and When Why perform CPU profiling? > CPU profiling provides information about where an application is spending most of its time. When is CPU profiling needed or beneficial? > Poor application throughput measured against a predetermined target > Saturated cpu utilization > High sys or kernel cpu utilization > High lock contention > To a lesser extent, idle cpu or poor application scalability Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 138

139 CPU Profiling Tips: Strategies Approaches which work best for CPU profiling > Start with holistic approach to isolate major cpu consumers or hot methods. >Look at methods with high usr and/or sys cpu usage. >Look at both inclusive and exclusive method times. >Looking at inclusive times may help identify a change in implementation or design could be a good corrective approach. >Looking at exclusive times focus on specific implementation details within a method. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 139

140 CPU Profiling Tips: Strategies Some profilers such as NetBeans Profiler allows you profile a subset of an application. > Approach can be useful when or if profiling the entire application is very intrusive or severely disturbs application's performance. > If holistic approach is not possible or painful, then profiling suspected subsets of an application is good approach. DTrace scripts can also be effective Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 140

141 CPU Profiling Tips: Which Product For profiling entire applications > Sun Studio Collector works well >1 second default sampling rate >Easy to setup, just prepend 'collect -j on' to java command line. >Can fine tune sampling rate. >Can direct output to specified file name. > DTrace scripting >Can customize to target specific areas. >May require DTrace scripting expertise to author the script. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 141

142 CPU Profiling Tips: Which Product For profiling portions of applications > NetBeans Profiler works very well >Can easily configure which classes or packages to profile, (include or!include). >Easy to setup if application is setup as a NetBeans Project. >Remote or local profiling >Can view profiling as application is running. >Can compare profile against another profile. > DTrace scripting >Customize to target specific portions. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 142

144 Heap Profiling Tips: Why and When Heap profiling provides information about the memory allocation footprint of an application. When is heap profiling needed or beneficial? > Observing frequent garbage collections > Application requires a large Java heap > Can be useful for obtaining better cpu utilization or application throughout and responsiveness > Less time allocating objects and/or collecting them means more cpu time spent running the application. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 144

145 Heap Profiling Tips: Strategies What approaches work best for heap profiling > Start with holistic approach to isolate major memory allocators. >Look at objects with large amount of bytes being allocated. >Look at objects with high number / count of object allocations. >Look at stack traces for locations where large amounts of bytes are being allocated. >Look at stack traces for locations where large number of objects being allocated Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 145

146 Heap Profiling Tips: Strategies If holistic approach is too intrusive, NetBeans Profiler can profile subsets of the application. > Hypothesize on packages or classes which might have a large memory allocation footprint. > Look at objects with large amount of bytes being allocated. > Look at objects with high number / count of object allocations. > Look at stack traces for locations where large amounts of bytes are being allocated. > Look at stack traces for locations where large number of objects are being allocated. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 146

147 Heap Profiling Tips: Strategies Cross reference cpu profiling with heap profiling > Look for objects which may have lengthy initialization times and allocate large amounts of memory. They are good candidates for caching. Look for alternative classes, objects and possibly caching approaches where high number / count of bytes are being allocated. Consider profiling while application is running to observe memory allocation patterns. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 147

148 Heap Profiling Tips: Complementary Tools (jmap / jhat) jmap and jhat can also capture heap profiles > Not as sophisticated as NetBeans Profiler > Limited to a snapshot at the time of jmap capture. (jmap captures the snapshot, jhat displays the data) > User interface not as polished as NetBeans Profiler > Easily view top memory consumer at time when snapshot was taken. > Look at stack traces for allocation location. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 148

149 Heap Profiling Tips: jmap / jhat Strategies Focus on large memory allocators > Consider alternative classes, objects and possibly caching approaches for large allocators. Capture several snapshots. Compare top memory allocators. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 149

150 Heap Profiling Tips: jmap / jhat Strategies Quick and easy to use > run jmap on the command line > run jhat on the command line > connect with a web browser Can be intrusive on the application to generate the snapshot. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 150

151 Memory Leak Profiling Tips: Why Memory leaks are situations where a reference to allocated object(s) remain unintentionally reachable and as a result cannot be garbage collected. Lead to poor application performance. Can lead to application failure. Can be hard to diagnose. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 151

152 Memory Leaks Profiling Tips: Tools Tools which help find memory leaks > NetBeans Profiler > VisualVM > jmap / jhat > Commercial offerings (not covered) >JProbe Memory >YourKit >SAP Memory Analyzer Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 152

153 Memory Leak Profiling Tips: Strategies NetBeans Profiler / VisualVM Strategies > View live heap profiling results while application is running. > Pay close attention to Surviving Generations. >Surviving Generations is the number of different object ages for a given class. >An increasing Surviving Generations over a period of time can be strong indicator of a source of a memory leak. > Use Heap Walker to traverse object references Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 153

155 Memory Leak Profiling Tips: Strategies jmap / jhat Strategies > Capture multiple heap profiles and compare footprints, (i.e. look for obvious memory usage increases). > -XX:+HeapDumpOnOutOfMemoryError >Use this JVM command line switch when launching application. Can be used with -XX:HeapDumpPath=<path>/<file> > Use jhat's Object Query Language (OQL) to query with interesting state information For example you can query live HTTP requests using: Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 155

157 Lock Contention Profiling Tips: Overview Use of Java synchronization can lead to highly contended locks. Observing high values of voluntary context switches can be an indication of lock contention. Collector / Analyzer is very good with identifying Java objects experiencing lock contention. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 157

159 Profiling Tips: Good Use Cases Collector / Analyzer > CPU profiling entire application > Sys cpu profiling or distinct usr vs sys profiling > Lock contention profiling > Integration with scripts, command files or batch files > Also view performance of JVM internals including methods > Want to see machine level assembly instructions > Narrow to specific window of sampling Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 159

160 Profiling Tips: Good Use Cases NetBeans Profiler > Profiling subset of application, for CPU profiling or heap profiling > Heap profiling > Finding memory leaks > Profiling an application using NetBeans IDE and/or NetBeans project > Remote profiling > Attach to running application > View profiling as application is running Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 160

161 Profiling Tips: Good Use Cases DTrace and DTrace scripts > Non-intrusive snapshots of running application > Command line utility > Can leverage existing public scripts >Heap profiling >Finding memory leaks >Monitor contention >JIT Compilation >Garbage collection activity >Method entry / exit >Java Native Interface entry / exit Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 161

162 Profiling Tips: Good Use Cases jmap / jhat > Heap profiling > Finding memory leaks > Simple command line utilities > Quick & easy snapshots of running application Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 162

163 Profiling Tips: Inlining effect If observing misleading or confusing results in cpu profiles, disable in-lining It is possible methods of particular interest are being in-lined and leading to misleading observations. To disable in-lining, add the following JVM command line switch to the JVM command line args: -XX:-Inline > Note: disabling in-lining may distort actual performance profile Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 163

164 Identifying Anti-patterns and Memory Leak Patterns Identifying anti-patterns in heap profiles Identifying memory leak patterns in heap profiles Identifying anti-patterns in method profiles Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 164

165 Anti-patterns in Heap Profile Large number of String or char[] allocations in heap profile > Possible over allocation of String > Possibly benefit from use of StringBuilder > Possible StringBuilder or StringBuffer resizing. > Possibly utilize ThreadLocal to cache char[] or StringBuilder or StringBuffer Reducing char[] and String allocations will likely reduce garbage collection frequency. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 165

166 Anti-patterns in Heap Profiles Observing StringBuffer in heap profile > Possible candidate for StringBuilder if synchronized access is not required. > Reducing char[] allocations on StringBuilder/StringBuffer on expansion of StringBuilder/StringBuffer size. Reducing char[] allocations will likely reduce garbage collection frequency. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 166

167 Anti-patterns in Heap Profiles Observing Hashtable in heap profile > Possible candidate for HashMap if synchronized access is not required. > Possible candidate for ConcurrentHashMap if synchronized access is required. > Further partitioning of data stored in Hashtable may lead to finer grained synchronized access and less contention. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 167

168 Anti-patterns in Heap Profiles Observing Vector in heap profile > Possible candidate for ArrayList if synchronized access is not required. > If synchronized access required and depending on its use, consider using: LinkedBlockingDeque, ArrayBlockingQueue, ConcurrentLinkedQueue, LinkedBlockingQueue or PriorityBlockingQueue. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 168

169 How to Reduce Lock Contention Approaches to reduce lock contention > Identify ways to partition the guarded data such that multiple locks can be integrated at a finer grained level as a result of partitioning. > Use a concurrent data structure first introduced in Java SE 5: >java.util.concurrent package. > If writes are much less frequent than reads: >Separate read lock from write lock by using a Java SE 5 ReentrantReadWriteLock. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 169

170 Concurrent Data Structures Versus Synchronized Collections: Tips Concurrent data structures might introduce additional cpu utilization overhead and might in some cases not provide as good of performance as a synchronized Collection. Compare the approaches with meaningful workloads. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 170

171 Concurrent Data Structures Versus Synchronized Collections: Tips HotSpot JVM biased locking may also improve synchronized collection performance. > -XX:+UseBiasedLocking introduced in JDK 5.0_06 > Improved in JDK 5.0_08 Must be explicitly enabled in JDK 5 versions. Enabled by default in JDK 6 versions. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 171

172 Anti-patterns in Heap Profiles Exception object allocations > Do not use exceptions for flow control > Use alternative flow control such as if / then / else, or switch flow control Generating stack traces are expensive operations Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 172

173 Memory Leak Patterns in Heap Profiles Monitoring for trends illustrating increasing surviving generations while heap profiling when application is running indicates strong memory leak candidate. > See section on Tools for Profiling. -XX:HeapDumpOnOutOfMemoryError can be used to capture heap dumps when out of memory errors occur. > Heap dumps can be analyzed by JHAT, NetBeans Profiler or VisualVM. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 173

174 Anti-patterns in Method Profiles Observing Map.containsKey(key) in profile. > If null keys are allowed in the Map, and null keys are not being used as valid keys in the Map. > Look at stack traces for unnecessary call flows which look like if (!map.containskey(key)) value = map.get(key); > value will be null if a key is not found via map.get(key) > Other use cases using Map methods such as put(key, value) or remove(key) may potentially Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 174

175 Anti-patterns in Method Profiles Observing high sys cpu times. > Look for monitor contention >Monitor contention and high sys cpu time have a strong correlation. >Consider alternatives to minimize monitor contention. > Look for opportunities to minimize number of system calls. >Example: read as much data as is ready to be read using non-blocking SocketChannels. > Reduction in sys cpu time will likely lead to better application throughput and response Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 175

178 What to Expect How to tune GC by setting GC generation sizes Compare different HotSpot garbage collectors Select HotSpot garbage collector based on application performance requirements Use tools to monitor GC and interpret the output Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 178

179 Garbage Collectors: An Overview The advantage of not having to deal with memory management issues as is the case with C / C++. Historically, garbage collection had often been the most common source attribute-able to poor Java application performance. It still can be a source of poor application performance. Just doesn't seem to get as much attention as it use to. To tune the JVM's garbage collectors, you need to understand the basics of how it works. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 179

180 Garbage Collectors: Sizing Java Heap Spaces -Xmx<n>m, max size of Java heap, (young generation + tenured generation) -Xms<n>m, initial size of Java heap (young generation + tenured generation) > Applications with emphasis on performance usually set -Xms and - Xmx to the same value. > When -Xmx!= -Xms, Java heap growth or shrink requires a full garbage collection -Xmn<n>m, size of young generation heap space Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 180

181 Garbage Collectors: Sizing Java Heap Spaces -XX:NewSize=<n>m, initial size of young generation space -XX:MaxNewSize=<n>m, max size of young generation space -XX:NewRatio=<n> ratio of young generation space to tenured space Applications with emphasis on performance tend to use - Xmn to size young generation since it combines use of - XX:MaxNewSize and -XX:NewSize Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 181

182 Garbage Collectors: Sizing Java Heap Spaces -XX:PermSize=<n>m, initial size of permanent generation space -XX:MaxPermSize=<n>m, max size of permanent generation space Applications with emphasis on performance almost always explicitly set -XX:PermSize and - XX:MaxPermSize to the same value. > Growing or shrinking permanent generation space requires a full garbage collection Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 182

183 Garbage Collectors: Sizing Spaces Tools to help identify heap sizes: > VisualVM's Monitor tab > JConsole's Memory tab > VisualGC's heap space sizes > GCHisto > jstat's gc options > -verbose:gc's heap space sizes > -XX:+PrintGCDetails heap space sizes Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 183

185 Garbage Collectors: The Choices Application throughput versus responsiveness > Throughput Emphasis on how much work can be done in a given interval of time with no concern of application pauses due to garbage collection > Responsiveness Emphasis on maintaining an elapsed time limit in which the application must respond to interactions and / or stimuli, or number times a time limit can be exceeded per some interval of time Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 185

186 Garbage Collectors: The Choices Choice of garbage collectors > Serial collector > Throughput collector > Concurrent collector > Low pause concurrent collector Note: For pause sensitive applications, if the serial collector (for small Java heaps), concurrent collector, or low pause concurrent collector are not able to meet application pause time requirements, consider using Java Real Time System (Java RTS) as an alternative. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 186

187 Garbage Collectors: Serial Collector An introduction > Enabled with -XX:+UseSerialGC > Single threaded young generation collector (stops all application threads) > Single threaded tenured generation collector (stops all application threads) Suitability > Well suited for single processor core machines > Well suited for configurations of one-to-one JVM to processor core configuration > Tends to work well for applications with small Java heaps, i.e. less than 128mb, may work well up to 256mb Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 187

188 Garbage Collectors: Serial Collector Comparing serial and throughput collectors > Serial collector can work better on applications with small young generation heaps versus parallel throughput collector. > Throughput collector's parallel GC threads may compete for work in small young generation heaps resulting in thrashing. > For small Java heaps, or small young generation Java heaps, try both, serial collector and parallel throughput collector. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 188

189 Garbage Collectors: Serial Collector Events which initiate a serial collector garbage collection > Eden space is unable to satisfy an object allocation request. Results in a minor garbage collection event. > Tenured generation space is unable to satisfy an object promotion coming from young generation. > An explicit invocation or call to System.gc(). Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 189

190 Garbage Collectors: Serial Collector Good throughput performance can be realized with the serial collector if: > Well tuned Java heap spaces on Java applications with small Java heaps, (i.e. smaller than 100mb heaps) > Target platform has small number of virtual processors > The number of JVMs deployed on a multi-core processor platform equals the number of multicore processors >Bind JVMs to processors or processor sets for best results Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 190

191 Garbage Collectors: Throughput Collector An introduction > Multi-threaded young generation space collectors enabled with -XX:+UseParallelGC > JDK 5.0_06 introduced multi-threaded tenured generation space collector enabled with - XX:+UseParallelOldGC (also enables - XX:+UseParallelGC) Suitability > Can significantly reduce garbage collection overhead on multi-core processor systems > Well suited for applications running on multiprocessor or multi-core systems Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 191

192 Garbage Collectors: Throughput Collector Managing collector threads > Number parallel throughput collector threads controlled by -XX:ParallelGCThreads=<N> >Defaults to Runtime.availableProcessors(). In a JDK 6 update release, 5/8ths available processors if > 8 >In multiple JVM per machine configurations, setting -XX:ParallelGCThreads=<N> lower will likely yield better results >Reducing number of parallel gc threads can also reduce fragmentation effect in promotion buffers in tenured space. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 192

193 Garbage Collectors: Throughput Collector Multiple JVM strategies > Bind JVMs to processor sets >Works well for JVMs which tend to have equal load since idle processors outside a processor set are not available to other JVMs outside the processor set >Note: Runtime.availableProcessors() reports number of processors in processor set > Create zones and run a JVM (or possible more than one JVM) per zone Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 193

194 Garbage Collectors: Throughput Collector Events which initiate a minor garbage collection > Eden space is unable to satisfy an object allocation request. Results in a minor garbage collection event. Events which might initiate a full garbage collection > Tenured generation space unable to satisfy an object promotion coming from young generation. > An explicit invocation or call to System.gc(). Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 194

195 Garbage Collectors: Throughput Collector Good throughput performance can be realized with the throughput collector if: > Pause time requirements are less important than throughput > Java heap spaces are well tuned > Application runs on multi-core system >Bind JVMs to processor sets in multiple JVM configurations for best results >Consider -XX:+BindGCTaskThreadsToCPUs or -XX:+AggressiveOpts when scaling across multiple cores > Young and tenured generations both use multi-threaded collectors, that is using -XX:+UseParallelOldGC and -XX:+UseParallelGC Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 195

196 Garbage Collectors: Throughput Collector Good throughput performance can be realized with the throughput collector if: > Use of -XX:+UseParallelGC, the multi-threaded young generation collector, and young generation heap is sized so that only long lived objects are tenured to old generation and Full GC events can be avoided. > If Full GC events cannot be avoided, a multi-threaded old generation collector can reduce length of Full GC events, -XX:+UseParallelOldGC. Note: -XX:+UseParallelOldGC also enables -XX:+UseParallelGC Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 196

197 Garbage Collectors: Concurrent Collector An Introduction > Single threaded tenured space collector which runs mostly concurrent with Java application threads and is enabled with - XX:+UseConcMarkSweepGC > Parallel, multi-threaded young generation collector enabled by default. Suitability > Well suited when application responsiveness is more important than application throughput > Consider using Concurrent incremental choice for applications with very sensitive pause time requirements Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 197

198 Garbage Collectors: Concurrent Collector Issues to be aware of > Cost of the concurrent collections is the additional overhead of more memory and CPU cycles > Concurrent mode failure can occur when objects are copied to the tenured space faster than the concurrent collector can collect them. ( loses the race ) > Concurrent mode failure can also occur from tenured space fragmentation. > Corrective action by the JVM is to perform a full garbage collection which will block all Java application threads. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 198

199 Garbage Collectors: Concurrent Collector Concurrent collector cycle contains the following phases > Initial mark > Concurrent mark > Remark > Concurrent sweep > Concurrent reset During a concurrent collector cycle, a Java application is paused during the 'initial mark' and 'remark' phases. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 199

200 Garbage Collectors: Concurrent Collector Initial mark phase > Objects in the tenured generation are marked as reachable including those objects which may be reachable from young generation. > Pause time is typically short in duration relative to minor collection pause times. Concurrent mark phase > Traverses the tenured generation object graph for reachable objects concurrently while Java application threads are executing. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 200

201 Garbage Collectors: Concurrent Collector Remark > Finds objects that were missed by the concurrent mark phase due to updates by Java application threads to objects after the concurrent collector had finished tracing that object. Concurrent sweep > Collects the objects identified as unreachable during marking phases. Concurrent reset > Prepares for next concurrent collection. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 201

202 Garbage Collectors: Concurrent Collector Additional heap space sizing considerations > Size young generation as large possible with the goal to only tenure long lived objects to old generation space and stay within pause time requirements. > If it is difficult to meet pause times requirements and have CPU available, try sizing young generation spaces smaller, (eden and survivor spaces). > Realize, sizing young gen smaller will put more pressure on the old generation concurrent collector since objects will be tenured to old generation at a faster rate. And, old generation space fragmentation may increase too. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 202

203 Garbage Collectors: Concurrent Collector Additional heap space sizing considerations > As much as 20% additional tenured generation space may be required for floating garbage > Floating garbage consists of objects which are found to be reachable by the concurrent garbage collector which may become unreachable by the time a concurrent garbage collection cycle finishes. > If unable to meet pause time requirements and cannot avoid Full GC events, consider heap profiling to reduce object allocations. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 203

204 Garbage Collectors: Concurrent Collector Additional heap space sizing considerations > Favor tuning survivor spaces rather than -XX:CMSInitiatingOccupancyFraction=# to reduce floating garbage and remark phase times. > To date, classes will not by default be unloaded from permanent generation when using the concurrent collector unless explicitly instructed to do so using -XX:+CMSClassUnloadingEnabled and -XX:+PermGenSweepingEnabled, (the 2nd switch is not needed in post HotSpot 6.0u4 JVMs). Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 204

205 Garbage Collectors: Concurrent Collector Additional heap space sizing considerations > If relying on explicit GC and want them to be concurrent, use: -XX:+ExplicitGCInvokesConcurrent (requires and later) - XX:+ExplicitGCInvokesConcurrentAndUnlo adsclasses for permanent generation concurrent collections and for class unloading, (requires JDK 1.6.0u4 or later). Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 205

206 Garbage Collectors: Concurrent Collector Additional heap space sizing considerations > For small Java heaps, pause time requirements may be achieved using the Serial collector. >Consider the Serial collector for Java heaps to 128mb and possibly up to 256m. >Serial collector is easier to tune than CMS. >What is learned from tuning the Serial collector is useful for initial CMS tuning. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 206

207 Garbage Collectors: Concurrent Collector Events initiating concurrent collection cycle > Ideally, the cycle needs to start early enough so that the collection finishes before tenured space becomes full. Because full garbage collections are expensive, the concurrent collector estimates the time remaining until tenured space will be used up and the amount of time needed to complete a concurrent collection cycle based on recent history. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 207

208 Garbage Collectors: Concurrent Collector Events initiating concurrent collection cycle (continued) > Concurrent collector will also start if the occupancy of the tenured space exceeds an initiating occupancy percentage threshold. Default value is 92% and may change from release to release. Tune -XX:CMSInitiatingOccupancyFraction=n where n is the % of the tenured space size. Note: Minor collection events occur as they do with throughput and serial collectors. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 208

209 Garbage Collectors: Concurrent Collector Good responsiveness performance can be realized with the concurrent collector if: > Pause time requirements are more important than throughput > Java heap spaces are well tuned >Will likely require larger tenured heap space sizing than other collectors due to heap fragmentation and floating garbage > By default a multi-threaded young generation collector, -XX:+UseParNewGC is enabled by default with the concurrent collector Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 209

210 Garbage Collectors: Concurrent Collector CMSIncrementalMode is an additional concurrent alternative for applications not able to meet pause time requirements with CMS and not wanting to use Java RTS. > CMSIncrementalMode enables the concurrent modes to be done incrementally. > Periodically gives additional processor back to the application resulting in better application responsiveness by doing the concurrent work in small chunks. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 210

211 Garbage Collectors: Concurrent Collector How to tune CMSIncremental mode > CMSIncrementalMode has a duty cycle that controls the amount of work the concurrent collector is allowed to do before giving up the processor. > Duty cycle is the % of time between minor collections the concurrent collector is allowed to run. > Duty cycle by default is automatically computed using what's called automatic pacing. > Both duty cycle and pacing can be fine tuned. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 211

212 Garbage Collectors: Concurrent Collector How to enable CMSIncremental mode > On JDK 6, recommend using the following two switches together: >-XX:+UseConcMarkSweepGC and >-XX:+CMSIncrementalMode > Or use: >-Xincgc Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 212

213 Garbage Collectors: Concurrent Collector How to enable CMSIncremental mode > On JDK 5 use all of the following switches together: >-XX:+UseConcmarkSweepGC >-XX:+CMSIncrementalMode >-XX:+CMSIncrementalPacing >-XX:CMSIncrementalDutyCycleMin=0 >-XX:CMSIncrementalDutyCycle=10 > JDK 5 settings mirror the default settings decided upon for JDK 6. > JDK 5's -Xincgc!= CMSIncrementalMode, it Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 213

214 Garbage Collectors: Concurrent Collector Fine tuning CMSIncremental mode > If full collections are still occurring, then: >Increase the safety factor using -XX:CMSIncrementalSafetyFactor=n The default value is 10. Increasing safety factor adds conservatism when computing the duty cycle. >Increase the minimum duty cycle using -XX:CMSIncrementalDutyCycleMin=n The default is 0 in JDK 6, 10 in JDK 5. >Disable automatic pacing and use a fixed duty cycle using -XX:-CMSIncrementalPacing Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 214

215 Garbage Collectors: Permanent Generation Why tune the permanent generation? > Some Java applications require fine tuning of the permanent generation space. > Applications that dynamically generate and load many classes such as commonly seen in web container or application server implementations, especially those utilizing JSPs need a larger permanent generation space than the provided default max size of 64m. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 215

216 Garbage Collectors: Permanent Generation Tuning the permanent generation > Use JConsole, VisualVM or jstat to observe the sizing behavior of permanent generation. > Use -XX:MaxPermSize=<n> to increase the maximum size. Also consider setting -XX:PermSize=<n> to the same value to avoid performance overhead of permanent generation space expansion. > Concurrent collector can be specified to collect permanent generation by using -XX:+CMSClassUnloadingEnabled and -XX:+PermGenSweepingEnabled (not required for HotSpot Java 6 JVMs.). Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 216

217 Garbage Collectors: Explicit GC Do not use System.gc() unless there is a specific use case or need to. Explicit invocations can be disabled using -XX:+DisableExplicitGC. A common use of explicit GC is RMI distributed garbage collection (dgc). -XX:+DisableExlipicitGC will disable RMI dgc. Consider tuning RMI dgc if needed rather than disabling explicit GC. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 217

218 Garbage Collectors: Explicit GC Default RMI distributed GC interval is once per minute, (60000 ms). > Use -Dsun.rmi.dgc.client.gcInterval and -Dsun.rmi.dgc.server.gcInterval to change. Max is Long.MAX_VALUE. > When using JDK 6 and the Concurrent collector also use - XX:+ExplicitGCInvokesConcurrent Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 218

219 Garbage Collectors: Reference Objects Got WeakReference, SoftReference or PhantomReference? Once a reference object is discovered by the garbage collector, it gets queued for reference processing which can extend the lifetime of a reference object until the reference processing is completed for that reference object. If there are lots of reference objects, the number of reference processing threads can have an impact on the latency of retiring the reference objects. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 219

220 Garbage Collectors: Reference Objects In addition, lots of reference objects also give the garbage collector more work to do since unreachable reference objects need to be discovered and queued during garbage collection. Reference object processing can extend the time it takes to perform garbage collections, especially if there are consistently many unreachable reference objects to process. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 220

221 Garbage Collectors: Soft References Soft References lifetimes are kept alive longer in HotSpot Server JVM. > -XX:SoftRefLRUPolicyMSPerMB=<n> to control clearing rate, default is 1000ms. > This specifies the number of ms a soft reference will be kept alive for each megabyte of free heap space after it is no longer strongly reachable. > Keep in mind soft references are cleared only during garbage collection which may not occur as frequently as the value set of SoftRefLRUPolicyMSPerMB. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 221

222 Garbage Collectors: Finalizers Use of finalizers can create performance issues. > Relying on garbage collection to manage resources other than memory is not a good idea. > For example, do not rely on a finalizer to close file descriptors. > Try to limit use of finalizer as safety net, use other mechanisms for releasing resources. > If using a finalizer cannot be avoided, try to keep the work being done as small as possible. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 222

223 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using serial collector Minor collection: > [GC [DefNew: 960K->64K(960K), secs] 3950K->3478K(5056K), secs] > DefNew is young generation space. > Old generation space not collected. > Permanent generation space not collected. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 223

224 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using serial collector(continued) Full collection: > [Full GC [Tenured: 4652K->4660K(8384K), secs] 4962K->4660K(9344K), [Perm : 1416K->1416K(12288K)], secs] > Tenured is old generation space. > Perm is permanent generation space. > Young generation stats reported. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 224

225 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using throughput collector Minor collection: > [GC [PSYoungGen: 13737K->1978K(14080K)] 17407K->6303K(43840K), secs] > Collected 13737K minus 1978K bytes of space collected in a 14080K sized young gen space. > 43840K overall heap size. > Took seconds to perform collection. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 225

226 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using throughput collector Full collection: > [Full GC [PSYoungGen: 32K->0K(12800K)] [PSOldGen: 5180K->5181K(29760K)] 5212K- >5181K(42560K) [PSPermGen: 10974K- >10974K(24832K)], secs] > PSYoungGen is young generation space > PSOldGen is old generation space > PSPermGen is permanent generation space Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 226

227 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using concurrent collector > Minor collections follow serial collector format. > [Full GC [CMS: 5994K->5992K(49152K), secs] 6834K->5992K(63936K), [CMS Perm : 10971K->10971K(18404K)], secs] > Note, CMS Perm indicates concurrent mark sweep collection activity in permanent generation space. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 227

228 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using concurrent collector > [GC [1 CMS-initial-mark: 13991K(20288K)] 14103K(22400K), secs] > CMS-initial-mark indicates the start of a concurrent collection cycle. > [CMS-concurrent-mark: 0.267/0.374 secs] > CMS-concurrent-mark indicates the end of the concurrent marking phase. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 228

229 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using concurrent collector > [CMS-concurrent-preclean: 0.044/0.064 secs] > CMS-concurrent-preclean indicates work done concurrently in preparation for the remark phase. > [GC [1 CMS-remark: 16090K(20288K)] 17242K(22400K), secs] > CMS-remark indicates remarking work > Note, a minor collection occurred concurrently with CMS-remark. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 229

230 Garbage Collectors: GC Output Understanding -XX:+PrintGCDetails using concurrent collector > [CMS-concurrent-sweep: 0.291/0.662 secs] > CMS-concurrent-sweep indicates the end of the concurrent sweeping phase. > [CMS-concurrent-reset: 0.016/0.016 secs] > CMS-concurrent-reset indicates work done to prepare for the next collection cycle. Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 230

231 Garbage Collectors: PrintGCStats PrintGCStats tool summarizes statistics from GC activity > Note: applicable only to 32-bit JVM's Script download location: turbo/printgcstats.zip Usage: PrintGCStats -v ncpu=<n> [-v interval=<seconds>] [-v verbose=1] <gc log file> > ncpu is number of cpus on the target machine > interval requires use of command line switch - XX:+PrintGCTimeStamps and reports statistics at each 'interval' > verbose provides more detailed output Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Revision A 231