Monitoring NonStop Server for Java memory

Transcription

1 Monitoring NonStop Server for Java memory Roland Lemoine HP GNSC November 16, 2015

2 Agenda NonStop JVM native memory usage Java heap layout and garbage collection refresh Out of Memory! Which one? Many tools available, which one should I use? Data to be captured Viewers Conclusion Appendix A: Version matrix 2

3 NonStop JVM native memory usage 3

4 NonStop JVM native memory usage NSJ 6 JVM native memory usage Within the 2 GB address space of the JVM most space is used by three competing areas: The native heap which includes the Java heap as well as native objects such as threads, etc. QIO, which may not be there if the Java code is not using any socket API or if it has been moved to KSEG2. Its size (by default 512Mb) may also be reduced and its start address changed. Flat segments may use a significant amount of space if the JVM is also a SQL/MX executor as it is the case when using the JDBC type 2 driver. 4

5 NonStop JVM native memory usage NSJ 7 32 bit JVM native memory usage In NSJ 7, two changes occurred. There is a dedicated space (a 256Mb flat segment) for those Java native objects called the JVM heap. Not to confuse with the Java heap and not to confuse with the JVM native heap. The Java heap and PermGen moved out of the Native heap into a flat segment as well. 5

6 NonStop JVM native memory usage Default max NSJ 7 Java heap in 32 bit, using QIO, PRIOR TO NSJ 7 update 1 is up to 124Mb: 6

7 NonStop JVM native memory usage Default max NSJ 7 Java heap in 32 bit, using QIO, with NSJ 7 update 1 is up to 316Mb: 7

8 NonStop JVM native memory usage NSJ 7 64 bit JVM native memory usage In 64 bit mode, the Java heap limit is lifted. 8

9 Java heap layout and garbage collection refresh 9

10 Java Heap layout and garbage collection refresh 10

11 Out of memory! Which one? 11

12 Out of memory! Which one? Java Heap and PermGen errors java.lang.outofmemoryerror: Java heap space java.lang.outofmemoryerror: PermGen space Java Heap and PermGen errors java.lang.outofmemoryerror: requested xxx bytes for Chunk::new. Out of swap space? java.lang.outofmemoryerror: requested <number> bytes for ChunkPool::allocate Exception in thread "CompileThread0" java.lang.outofmemoryerror: requested bytes Thread creation failed: Insufficient user memory 12

13 Selecting the appropriate options and tools 13

14 Selecting the appropriate options and tools Data to be captured: Garbage Collection data Heap dump using kill -3 (SIGQUIT) -verbosegc option -Xloggc option -Xverbosegc option Xverbosegc using kill -14 (SIGALRM) Data to be captured: Profiling data -agentlib:hprof -XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnCtrlBreak -XX:+HeapDump -XX:+PrintAllocStatistics -Xeprof Xeprof using kill -17 (SIGUSR2) 14

15 Data to be captured 15

16 Data to be captured Heap snapshot option: kill -3 <JVM PID> (or kill s QUIT <JVM PID>) Heap snapshot example: Output on JVM stdout 16

17 Data to be captured -verbosegc option of the JVM A minor GC where the GC freed about 500 Kbytes out of a total of 12 MB of heap: [GC 3328K->2854K(11904K), secs] A Full GC where the Full GC freed about 800 Kbytes out of 13 MB of heap: [Full GC 9508K->8740K(12928K), secs] Heap before GC Heap After GC Output on JVM stdout 17

18 Data to be captured -Xloggc 0.590: [GC 896K->278K(5056K), secs] 0.906: [GC 1174K->774K(5056K), secs] Timestamp Output on JVM stdout or named file 18

19 Data to be captured -Xverbosegc Log example: <GC: > <GC: > <GC: > <GC: >] Eden before and after GC Old Gen before and after GC Output on JVM stdout, or per PID named file 19

20 Data to be captured -Xverbosegc 1 Type of GC: 1: Scavenge (GC of New Generation only) 2: Old Generation GC or a Full GC 3: Complete background CMS GC 4: Incomplete background CMS GC 11: Ongoing CMS GC 2 Additional information based on GC type in field 1. 3 Program time at the beginning of the GC, in seconds. 4 GC invocation. Counts of background CMS GCs and other GCs are maintained separately. 5 Size of the object allocation request that forced the GC, in bytes. 6 Tenuring threshold determines how long the newborn object remains in the New Generation. 7 Eden Sub-space (within the New Generation) occupied before GC. 8 Eden Sub-space (within the New Generation) occupied after GC. 9 Eden Sub-space (within the New Generation) current capacity. 10 Survivor Sub-space (within the New Generation) occupied before GC. 11 Survivor Sub-space (within the New Generation) occupied after GC. 12 Survivor Sub-space (within the New Generation) current capacity. 13 Old Generation occupied before GC. 14 Old Generation occupied after GC. 15 Old Generation current capacity. 16 Permanent Generation (storage of Reflective Objects) occupied before GC. 17 Permanent Generation (storage of Reflective Objects) occupied after GC. 20

21 Data to be captured Zero-Preparation Xverbosegc : kill -14 <JVM PID> (or kill s ALRM <JVM PID>) NO JVM restart Output by default in java_<pid>.vgc 21

22 Data to be captured -agentlib:hprof java -agentlib:hprof=heap=sites <class> Generates ASCII file java.hprof.txt at program exit or when sending kill -3 <JVM PID>. Contains Objects created by method, Created Objects count and bytes java -agentlib:hprof=heap=dump <class> A lot more information is captured here (Several hundred Mb for a 64Mb java heap!) however allocation sites are not. Output by default in java.hprof.txt 22

23 Data to be captured XX:+HeapDumpOnOutOfMemoryError $ java -XX:+HeapDumpOnOutOfMemoryError MemLeak java.lang.outofmemoryerror: Java heap space Dumping heap to java_pid hprof... Heap dump file created [ bytes in secs] Exception in thread "main" java.lang.outofmemoryerror: Java heap space at java.util.hashtable.rehash(hashtable.java:428) at java.util.hashtable.put(hashtable.java:500) at MemLeak.main(MemLeak.java:17) Can also be enabled using JMX Output by default in java_pid<pid>.hprof 23

24 Data to be captured XX:+HeapDumpOnCtrlBreak Output by default in java_<pid>.hprof.militime XX:+HeapDump or _JAVA_HEAPDUMP=1 Output by default in java_<pid>_<date>_<time> _heapdump.hprof.txt 24

25 Data to be captured XX: +PrintAllocStatistics (Not supported on NonStop currently) Output by default on stdout or in java_<pid>.vgc when using kill

26 Data to be captured Xeprof Zero-Preparation Xeprof : kill -17 <JVM PID> (or kill s USR2 <JVM PID>) Output by default in java<pid>_<time>.eprof 26

27 Viewers 27

28 Viewers: HPjmeter Hpjmeter accepts: hprof eprof -Xverbosegc -Xloggc -XX:+HeapDump -XX:+HeapDumpOnCtrlBreak -XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnly -XX:+PrintAllocStatistics 28

29 Viewers: HPjmeter Note: When looking at hprof data, hprof data analysis may quickly exhaust HPJmeter default Java heap size of 512Mb depending on which metrics you look at. To Increase HPjmeter Java heap: cd C:\Program Files (x86)\hpjmeter\bin hpjmeter -Xmx1240m (with a 32bit jvm) But better would be: hpjmeter -Xmx4g with a 64bit jdk However, looking at Xverbosegc data or -XX:+PrintAllocStatistics does not require such a large heap. 29

30 Viewers: HPjmeter 30

31 Viewers: Hpjmeter connected to live JVMs Documentation: NonStop Server for Java 6.0 Programmer s reference: Chapter 6: Application profiling or NonStop Server for Java 7.0 Programmer s reference : Chapter 7: Application Tuning and profiling ; HPJmeter release notes and installation user guide ; HPjmeter 4.3 user s guide Installation: On the server side, the HPjmeter agents are shipped on the NSJ 6.0 or 7.0 product CD (product T0866H31). If not already extracted in /usr/tandem/hpjmeter, extract the pax file: pax rvf t0866pax A nodeagent process needs to be started: export JMETER_HOME=/usr/tandem/hpjmeter/ $JMETER_HOME/bin/nodeagent Setup application monitoring: The application being monitored must be started with the following option: -agentlib:jmeter and requires the following environment variable(s): In Java 6: export _RLD_LIB_PATH=$_RLD_LIB_PATH :$JMETER_HOME/lib/oss/ In Java 7: export _RLD_LIB_PATH=$JMETER_HOME/lib/oss32/(if 32-bit version of Java) export _RLD_LIB_PATH=$JMETER_HOME/lib/oss64/(if 64-bit version of Java) Unlike JMX no need to have one port per JVM that you want to monitor. 31

32 Viewers: Hpjmeter connected to live JVMs Dynamic attach feature now available: When used with: - HPjmeter 4.3 ( Server side T0866H31^AAD + Hpjmeter Console >= 4.3) - NS/Java 7 update 1 (T2766H70^ACJ or T2766H60^ACH) Add the following entry in /usr/tandem/hpjmeter/config/hpjmeter.conf JM_JAVA_HOME=/usr/tandem/nssjava/jdk170_h70 At this point there is no need to add -agentlib:jmeter to your application to be monitored anymore therefore no need for another JVM restart for each application server. 32

33 Viewers: HPjmeter 33

34 Viewers: HPjmeter 34

35 Viewers: Jconsole Add the following options to the JVM to be monitored: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false Or if using a port range: -Dcom.sun.management.jmxremote.port= Connect the jconsole GUI to the target JVM using the IP address and the jmx port. 35

36 Viewers: Jconsole 36

37 Viewers: Jconsole Mbeans tab HeapDumpOnOutOfMemoryError setvmoption 37

38 Viewers: Java VisualVM 38

39 Viewers: Java Heap Analysis tool (Jhat) 39

40 Viewers: Eclipse Memory Analyzer (MAT) 40

41 Viewers: Eclipse Memory Analyzer (MAT) 41

42 Take Away If the JVM is already running without any specific options, Zero-Preparation options allow you to trigger data capture without having to restart the JVM HPJmeter, provided for free by HP, is perfectly adapted to monitor multiple JVMs in a Pathway environment and is the only option for versions prior to NSJ H60^ACF where the JMX interface would be limited to use only when NUMSTATIC is 1. JMX provides out-of-the-box access to the JVM using various industry tools and allowing Java monitoring without HP NonStop platform specific knowledge. 42

43 Take Away In the specific task of identifying the root cause of memory leaks, hprof profiling data including site allocations appears to be the best option, while eprof profiling data is not useful for memory tuning specifically. Log file auto-naming, including the JVM PID also proves to be a practical feature in environments like HP NonStop where multiple JVMs are executing and sharing the same JVM arguments list. Hpjmeter 4.3 and NS/Java 7 update 1 provide a dynamic attach feature which allows to connect to a JVM without any downtime and without requiring configuration changes. 43

44 Appendix: Java memory monitoring matrix JVM stop Output Description Available since Garbage Collection data kill -3 <PID> No text on stdout Heap snapshot Jdk 1.2 -verbose:gc Yes text on stdout Garbage collection log Jdk 1.2 -Xloggc Yes text on stdout or file Garbage collection log, HP specific, can be fed to HPJmeter NSJ 1.2 -Xverbosegc Yes text on stdout or file Garbage collection log, HP specific, can be fed to HPJmeter NSJ 1.2 kill -14 <PID> (aka zero-preparation Xverbosegc) No Text on file Same output as Xverbosegc. File name auto-generated java_<pid>.vgc NSJ 6.0 ABP Profiling data -agentlib:hprof=heap=sites -agentlib:hprof=heap=dump Yes Text or binary in.hprof file. Industry standard profiling data (HPJmeter, JVisualVM, Jhat, Eclipse MAT). Provides memory leak root cause when capturing heap sites. NSJ 5 ( runhprof previously) -XX:+HeapDumpOnOutOfMemoryError Yes binary Heap dump captured in java_pid<pid>.hprof when OutOfMemoryError occurs NSJ 6 -XX:+HeapDumpOnCtrlBreak Yes binary Kill -3 generates hprof heap=dump NSJ 6 -XX:+HeapDump /_JAVA_HEAPDUMP=1 Yes Text by default Kill -3 generates thread dump on stdout and hprof in NSJ 5.1 java_<pid>_<date>_<time>_heapdump.hprof.txt -XX:+PrintAllocStatistics Yes Text on stdout or.vgc file kill -14 will generate site statistics and ALSO turn off/on -Xverbosegc NSJ 6.0 ABP -Xeprof -Xeprof:file=<name>.eprof Yes Not best for memory tuning. Generates <name><pid>.eprof kill -17 <PID> (zero-preparation eprof) No Text in.eprof file Not best for memory tuning. First signal starts profiling, second writes to java_<pid>_nn.eprof NSJ 5.1 Tools HPJmeter May GUI Accepts hprof, eprof and gc data or works connected when used with Server side T0866 node agent. NSJ 5.1 NSJ 6.0 ABR nsjps -h No OSS stdout List the *native* heap usage NSJ 6.0 ABP jconsole No GUI Uses JMX NSJ 6.0 ACF Jvisualvm No GUI Based on JMX or hprof data. NSJ 6.0 ACF Jhat No HTML Based on hprof data Jdk 1.6 Eclipse MAT No GUI Based on hprof data NSJ 5 44

45 Thank you 45