Production time profiling On-Demand with Java Flight Recorder

Transcription

1

2

3 Production time profiling On-Demand with Java Flight Recorder Using Java Mission Control & Java Flight Recorder Klara Ward Principal Software Developer Java Platform Group, Oracle Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential

4 About me Developer in the Oracle Java Mission Control team in Stockholm, Sweden Sometimes Celebrating Java 20 Years at JFokus

5 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 5

6 Agenda Overview of Java Mission Control Overview of Java Flight Recorder Demo Customization, Future, Links Q&A 6

7 Maurizio Cimadamore, Oracle (Java LangTools) Allan Thrane Andersen, Trygg I managed to do in one day what I've failed JMC is my main tool for getting insight into to do in 2+ weeks using <profiling tool> and the rhythm of a JVM and the running <another profiling tool>. applications. I have used recordings to resolve critical production issues caused by latency, memory-leaks or threading.

8 RebelLabs Developer Productivity Report 2015, Java Performance Survey

9 Java Mission Control profiling tool What do they mean? Probably: Data from Java Flight Recorder Visualized in Java Mission Control

10 Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview JDK JVM JFR Events Low overhead JFR Engine Control recordings start/stop/dump Recording data myrecording.jfr java -XX:StartFlightRecording JDK/bin/jcmd <pid> <cmd> JMC java -XX:+FlightRecorder JDK/bin/jmc or Eclipse plug-ins

11 Overview of Java Mission Control The graphical client 11

12 Java Mission Control Overview A tools suite for production use (fine in development too) Basic monitoring Production time profiling and diagnostics Free for development and evaluation Tool usage is free, data creation in production requires a commercial license tiny.cc/javalicense

13 History of Mission Control JRockit Flight Recorder Appeal(JRockit) -> BEA Systems -> Oracle <- Sun Best JRockit features -> HotSpot JVM JFR and JMC released with 7u40, September 2013

14 Java Mission Control Main Tools Two main tools: JMX Console Online monitoring Flight Recorder Offline low overhead profiler Control and visualization in JMC JRockit Mission Control also had the Memory Leak Analyzer

15 Experimental Plugins Downloadable from within Mission Control DTrace JFR style visualization of data produced by DTrace JOverflow Memory anti-pattern analysis from hprof dumps JMX Console plug-ins Java Flight Recorder plug-ins WLS JavaFX

16 JMC installation/startup <JDK>/bin/jmc Mac: (/usr/bin/) jmc Add if needed: consolelog debug ( more 2>&1 ) Eclipse plug-ins Install from update site on OTN: Eclipse Update Site Experimental plug-ins: Install from within the JMC app, or from Eclipse Experimental Update Site

17 Overview of Java Flight Recorder Low overhead profiling 17

18 Flight Recorder 101 High Performance Event Recorder Built into the JVM Already available runtime information Measuring the real behavior, doesn t disable JVM optimizations Binary recordings Self contained self describing chunks Very detailed information Extremely low overhead (~1-2%) Can keep it always on, dump when necessary

19 Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview JDK Java Events JFR JMX API JFR Java API Control recordings start/stop/dump java -XX:StartFlightRecording JDK/bin/jcmd <pid> <cmd> JVM JVM Events JFR Engine Recording data myrecording.jfr JMC java -XX:+FlightRecorder JDK/bin/jmc or Eclipse plug-ins

20 Data collected by JFR Java application behavior Threads/Locks I/O Exceptions JVM behavior (indirect Java application behavior) Garbage collection, allocation JIT Compiler Implemented by the different subsystem teams

21 Method sampling Sampling profiler Not displaying every single call to your method This is partly why we get the low overhead Detects hot methods Does not require threads to be at safepoints (Flags currently needed to give more accurate non-safepoint data) -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints (will be default in coming releases) Not sampling threads in native

22 RebelLabs Developer Productivity Report 2015, Java Performance Survey

23 RebelLabs Developer Productivity Report 2015, Java Performance Survey

24 Different Kinds of Events Instant Event - Exception Duration Event Thread.sleep Configurable threshold Requestable Event Method profiling sample Polled from separate thread Configurable period Period and threshold settings impact the performance overhead

25 Event settings Predefined settings default designed to get max information within <= 1 % overhead profile even more information, ~2 % overhead Enabling of event types, configuring periods and thresholds jre/lib/jfr/*.jfc Design your own from the Mission Control GUI

26 Different Kinds of Recordings Continuous Recordings Have no end time Must be explicitly dumped Example use case: Enable at startup, dump the last X minutes when needed Time Fixed Recordings ( profiling recordings ) Have a fixed time If started from Java Mission Control, opened automatically in the GUI Example use case: Performance testing under load, do a 1 minute recording Resulting *.jfr file also called Recording

27 Creating recordings More than one way 27

28 Preparations Start the JVM from which to get recordings with: -XX:+UnlockCommercialFeatures -XX:+FlightRecorder In 8u40 and later, possible to enable at runtime if needed Using JMC or jcmd On-demand

29 Creating Recordings Using Mission Control 1. Find a JVM to do a recording on in the JVM Browser 2. Double click the Flight Recorder node under the JVM 3. Follow the wizard NEW: No need for the JVM flags, automatic enablement from JMC

30 Creating Recordings Using Startup Flags -XX:+UnlockCommercialFeatures -XX:+FlightRecorder Time fixed -XX:StartFlightRecording= delay=20s,duration=60s, filename=c:\tmp\myrecording.jfr,settings=profile,name=javaland Continuous w/ dumponexit -XX:StartFlightRecording=settings=default -XX:FlightRecorderOptions=dumponexit=true, dumponexitpath=c:\tmp\myrecordings (Needed before 8u20: -XX:FlightRecorderOptions=defaultrecording=true ) See documentation for Java options (google java options )

31 Creating Recordings Using JCMD Useful for controlling JFR from the command line Usage: jcmd <pid> <command> Starting a recording: jcmd 7060 JFR.start name=myrecording settings=profile delay=20s duration=2m filename=c:\tmp\myrecording.jfr Dumping a recording: jcmd 7060 JFR.dump name=myrecording filename=c:\tmp\dump.jfr Unlocking commercial features (if JVM not started with the flag): jcmd 7060 VM.unlock_commercial_features Don t forget to try jcmd <pid> help to see what else jcmd can do

32 Creating Recordings Using JMX Console triggers When realtime monitoring your application Start JMC Connect a JMX Console to your application Configure and enable rules on the Triggers tab Recording will be started or dumped when the trigger occurs

33 Remote production systems No GUI, security restrictions Remote access either using -Dcom.sun.management.jmxremote, connect with JMC or start recording from local commandline using jcmd, transfer JFR to workstation Enabling JFR at startup some very small initiation overhead at startup, threads allocate small amount of extra memory dynamically 0% overhead at startup, initiation overhead happens at time of enabling, might cause some classes to be deoptimized etc. If you want to avoid restart

34 Analyzing recordings Using the graphical client 34

35 How to think about the information shown When analyzing Flight Recordings Only you know what your application is supposed to be doing Batch job, or real time trading? Do you want the CPU usage to be high or low? If you have a theory about what is wrong, you can find out why

36 How to think about the information shown Common pitfalls CPU load Low load method sampling not very interesting Full load latencies not very interesting/likely Event thresholds Keeping performance up but still detecting outliers Is the thread running normally during 30%? Hint: Default threshold is 10 ms.

37 DEMO! Flight Recorder startup and analysis 37

38 Customization Unsupported 38

39 import com.oracle.jrockit.jfr.*; public class Example { private final static String PRODUCER_URI = " private Producer myproducer; private EventToken mytoken; } Adding Your Own Events (unsupported) public Example() throws URISyntaxException, InvalidEventDefinitionException, InvalidValueException { myproducer = new Producer("Demo Producer", "A demo event producer.", PRODUCER_URI); mytoken = myproducer.addevent(myevent.class); name = "My Event", description="an event triggered by dostuff.", stacktrace=true, thread=true) private class MyEvent extends TimedEvent description="the logged important stuff.") private String text; public MyEvent(EventToken eventtoken) { super(eventtoken); } } public void settext(string text) { this.text = text; } public void dostuff() { MyEvent event = new MyEvent(myToken); event.begin(); String importantresultinstuff = ""; // Generate the string, then set it... event.settext(importantresultinstuff); event.end(); event.commit(); } 39

40 Adding Your Own Events (unsupported) Reusing the event (not thread safe)... private MyEvent event = new MyEvent(myToken); public void dostuffreuse() { event.reset(); event.begin(); String importantresultinstuff = ""; // Generate the string, then set it... event.settext(importantresultinstuff); event.end(); event.commit(); }... NB: If JFR is not enabled, the custom events gives no (0%) overhead 40

41 Parsing recordings (unsupported) Two unsupported options The JDK parser import oracle.jrockit.jfr.parser.*; SAX style parser The JMC parser import com.jrockit.mc.flightrecorder.flightrecording; DOM style parser 41

42 Built in GUI editor (unsupported) Show view -> Designer Customize the existing GUI or produce entirely new GUIs for events Export the created GUI to share it with others

43 Future 43

44 Future JFR - Supported API for adding your own JFR events JMC - Automatic analysis of Flight Recordings You have used a JVM flag that is not recommended Your application is doing a lot of GC, investigate more here...

45 Resources Homepage: (Click Discussion to see the forum) @hirt Blog: Facebook: ( for JMC tutorial)

46 Questions? Q: Can you repeat those JVM flags again? A: -XX:+UnlockCommercialFeatures XX:+FlightRecorder 46

47 Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 47

48

49