Production time profiling On-Demand with Java Flight Recorder Using Java Mission Control & Java Flight Recorder Klara Ward Principal Software Developer Java Platform Group, Oracle Copyright 2015, Oracle and/or its affiliates. All rights reserved. Oracle Confidential
About me Developer in the Oracle Java Mission Control team in Stockholm, Sweden Sometimes tweets @klaraward Celebrating Java 20 Years at JFokus
Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 5
Agenda 1 2 3 4 5 Overview of Java Mission Control Overview of Java Flight Recorder Demo Customization, Future, Links Q&A 6
Maurizio Cimadamore, Oracle (Java LangTools) Allan Thrane Andersen, Trygg I managed to do in one day what I've failed JMC is my main tool for getting insight into to do in 2+ weeks using <profiling tool> and the rhythm of a JVM and the running <another profiling tool>. applications. I have used recordings to resolve critical production issues caused by latency, memory-leaks or threading.
RebelLabs Developer Productivity Report 2015, Java Performance Survey
Java Mission Control profiling tool What do they mean? Probably: Data from Java Flight Recorder Visualized in Java Mission Control
Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview JDK JVM JFR Events Low overhead JFR Engine Control recordings start/stop/dump Recording data myrecording.jfr java -XX:StartFlightRecording JDK/bin/jcmd <pid> <cmd> JMC java -XX:+FlightRecorder JDK/bin/jmc or Eclipse plug-ins
Overview of Java Mission Control The graphical client 11
Java Mission Control Overview A tools suite for production use (fine in development too) Basic monitoring Production time profiling and diagnostics Free for development and evaluation Tool usage is free, data creation in production requires a commercial license tiny.cc/javalicense
History of Mission Control JRockit Flight Recorder Appeal(JRockit) -> BEA Systems -> Oracle <- Sun Best JRockit features -> HotSpot JVM JFR and JMC released with 7u40, September 2013
Java Mission Control Main Tools Two main tools: JMX Console Online monitoring Flight Recorder Offline low overhead profiler Control and visualization in JMC JRockit Mission Control also had the Memory Leak Analyzer
Experimental Plugins Downloadable from within Mission Control DTrace JFR style visualization of data produced by DTrace JOverflow Memory anti-pattern analysis from hprof dumps JMX Console plug-ins Java Flight Recorder plug-ins WLS JavaFX
JMC installation/startup <JDK>/bin/jmc Mac: (/usr/bin/) jmc Add if needed: consolelog debug ( more 2>&1 ) Eclipse plug-ins Install from update site on OTN: http://oracle.com/missioncontrol, Eclipse Update Site Experimental plug-ins: Install from within the JMC app, or from https://oracle.com/missioncontrol, Eclipse Experimental Update Site
Overview of Java Flight Recorder Low overhead profiling 17
Flight Recorder 101 High Performance Event Recorder Built into the JVM Already available runtime information Measuring the real behavior, doesn t disable JVM optimizations Binary recordings Self contained self describing chunks Very detailed information Extremely low overhead (~1-2%) Can keep it always on, dump when necessary
Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview JDK Java Events JFR JMX API JFR Java API Control recordings start/stop/dump java -XX:StartFlightRecording JDK/bin/jcmd <pid> <cmd> JVM JVM Events JFR Engine Recording data myrecording.jfr JMC java -XX:+FlightRecorder JDK/bin/jmc or Eclipse plug-ins
Data collected by JFR Java application behavior Threads/Locks I/O Exceptions JVM behavior (indirect Java application behavior) Garbage collection, allocation JIT Compiler Implemented by the different subsystem teams
Method sampling Sampling profiler Not displaying every single call to your method This is partly why we get the low overhead Detects hot methods Does not require threads to be at safepoints (Flags currently needed to give more accurate non-safepoint data) -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints (will be default in coming releases) Not sampling threads in native
RebelLabs Developer Productivity Report 2015, Java Performance Survey
RebelLabs Developer Productivity Report 2015, Java Performance Survey
Different Kinds of Events Instant Event - Exception Duration Event Thread.sleep Configurable threshold Requestable Event Method profiling sample Polled from separate thread Configurable period Period and threshold settings impact the performance overhead
Event settings Predefined settings default designed to get max information within <= 1 % overhead profile even more information, ~2 % overhead Enabling of event types, configuring periods and thresholds jre/lib/jfr/*.jfc Design your own from the Mission Control GUI
Different Kinds of Recordings Continuous Recordings Have no end time Must be explicitly dumped Example use case: Enable at startup, dump the last X minutes when needed Time Fixed Recordings ( profiling recordings ) Have a fixed time If started from Java Mission Control, opened automatically in the GUI Example use case: Performance testing under load, do a 1 minute recording Resulting *.jfr file also called Recording
Creating recordings More than one way 27
Preparations Start the JVM from which to get recordings with: -XX:+UnlockCommercialFeatures -XX:+FlightRecorder In 8u40 and later, possible to enable at runtime if needed Using JMC or jcmd On-demand
Creating Recordings Using Mission Control 1. Find a JVM to do a recording on in the JVM Browser 2. Double click the Flight Recorder node under the JVM 3. Follow the wizard NEW: No need for the JVM flags, automatic enablement from JMC
Creating Recordings Using Startup Flags -XX:+UnlockCommercialFeatures -XX:+FlightRecorder Time fixed -XX:StartFlightRecording= delay=20s,duration=60s, filename=c:\tmp\myrecording.jfr,settings=profile,name=javaland Continuous w/ dumponexit -XX:StartFlightRecording=settings=default -XX:FlightRecorderOptions=dumponexit=true, dumponexitpath=c:\tmp\myrecordings (Needed before 8u20: -XX:FlightRecorderOptions=defaultrecording=true ) See documentation for Java options (google java options )
Creating Recordings Using JCMD Useful for controlling JFR from the command line Usage: jcmd <pid> <command> Starting a recording: jcmd 7060 JFR.start name=myrecording settings=profile delay=20s duration=2m filename=c:\tmp\myrecording.jfr Dumping a recording: jcmd 7060 JFR.dump name=myrecording filename=c:\tmp\dump.jfr Unlocking commercial features (if JVM not started with the flag): jcmd 7060 VM.unlock_commercial_features Don t forget to try jcmd <pid> help to see what else jcmd can do
Creating Recordings Using JMX Console triggers When realtime monitoring your application Start JMC Connect a JMX Console to your application Configure and enable rules on the Triggers tab Recording will be started or dumped when the trigger occurs
Remote production systems No GUI, security restrictions Remote access either using -Dcom.sun.management.jmxremote, connect with JMC or start recording from local commandline using jcmd, transfer JFR to workstation Enabling JFR at startup some very small initiation overhead at startup, threads allocate small amount of extra memory dynamically 0% overhead at startup, initiation overhead happens at time of enabling, might cause some classes to be deoptimized etc. If you want to avoid restart
Analyzing recordings Using the graphical client 34
How to think about the information shown When analyzing Flight Recordings Only you know what your application is supposed to be doing Batch job, or real time trading? Do you want the CPU usage to be high or low? If you have a theory about what is wrong, you can find out why
How to think about the information shown Common pitfalls CPU load Low load method sampling not very interesting Full load latencies not very interesting/likely Event thresholds Keeping performance up but still detecting outliers Is the thread running normally during 30%? Hint: Default threshold is 10 ms.
DEMO! Flight Recorder startup and analysis 37
Customization Unsupported 38
import com.oracle.jrockit.jfr.*; public class Example { private final static String PRODUCER_URI = "http://www.example.com/demo/"; private Producer myproducer; private EventToken mytoken; } Adding Your Own Events (unsupported) public Example() throws URISyntaxException, InvalidEventDefinitionException, InvalidValueException { myproducer = new Producer("Demo Producer", "A demo event producer.", PRODUCER_URI); mytoken = myproducer.addevent(myevent.class); } @EventDefinition(path="demo/myevent", name = "My Event", description="an event triggered by dostuff.", stacktrace=true, thread=true) private class MyEvent extends TimedEvent { @ValueDefinition(name="Message", description="the logged important stuff.") private String text; public MyEvent(EventToken eventtoken) { super(eventtoken); } } public void settext(string text) { this.text = text; } public void dostuff() { MyEvent event = new MyEvent(myToken); event.begin(); String importantresultinstuff = ""; // Generate the string, then set it... event.settext(importantresultinstuff); event.end(); event.commit(); } 39
Adding Your Own Events (unsupported) Reusing the event (not thread safe)... private MyEvent event = new MyEvent(myToken); public void dostuffreuse() { event.reset(); event.begin(); String importantresultinstuff = ""; // Generate the string, then set it... event.settext(importantresultinstuff); event.end(); event.commit(); }... NB: If JFR is not enabled, the custom events gives no (0%) overhead 40
Parsing recordings (unsupported) Two unsupported options The JDK parser import oracle.jrockit.jfr.parser.*; SAX style parser The JMC parser import com.jrockit.mc.flightrecorder.flightrecording; DOM style parser http://hirt.se/blog/?p=446 41
Built in GUI editor (unsupported) Show view -> Designer Customize the existing GUI or produce entirely new GUIs for events Export the created GUI to share it with others
Future 43
Future JFR - Supported API for adding your own JFR events JMC - Automatic analysis of Flight Recordings You have used a JVM flag that is not recommended Your application is doing a lot of GC, investigate more here...
Resources Homepage: http://oracle.com/missioncontrol (Click Discussion to see the forum) http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/index.html Twitter: @javamissionctrl @klaraward @hirt Blog: http://hirt.se/blog? Facebook: https://www.facebook.com/javamissionctrl (http://hirt.se/blog/?p=611 for JMC tutorial)
Questions? Q: Can you repeat those JVM flags again? A: -XX:+UnlockCommercialFeatures XX:+FlightRecorder 46
Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 47