Copyright 2014 Splunk Inc. Splunking the JVM. Damien Dallimore. Dev Evangelist, CSO Office @ Splunk

Disclaimer During the course of this presentagon, we may make forward- looking statements regarding future events or the expected performance of the company. We caugon you that such statements reflect our current expectagons and esgmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentagon are being made as of the Gme and date of its live presentagon. If reviewed aser its live presentagon, this presentagon may not contain current or accurate informagon. We do not assume any obligagon to update any forward- looking statements we may make. In addigon, any informagon about our roadmap outlines our general product direcgon and is subject to change at any Gme without nogce. It is for informagonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligagon either to develop the features or funcgonality described or to include any such feature or funcgonality in a future release. 2

From Middle Earth Make Splunk Apps & Add- ons JVM background 3

apps.splunk.com github.com/damiendallimore 5

What is this JVM thing?

JVM = Java Virtual Machine First appeared in the early 90 s Now the dominant rungme for enterprise applicagons ApplicaGon Servers Enterprise Service Buses Databases NoSQL Distributed Big Data Web Servers Directory Servers Search Engines Build Systems Gaming Pladorms Trading Systems ReservaGon Systems Core Banking Messaging Infrastructure Proprietary Systems 7

Many JVM Variants Oracle Hotspot OpenJDK Oracle JRockit IBM J9 Azul Zing More or less the same. OpenJDK is reference impl for Java 7/8 Gehng merged into Hotspot Commercial alternagves 9

JVMs are here to stay It s not just about the J Big Data frameworks You should care about gehng insights into your JVM data 10

Gehng at the Data

Data Sources Logs Developers JMX (Java Management Extensions) InstrumentaGon Agents OperaGng System Distributed CommunicaGons 12

Log Data

Standard Log Files Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.catalina load INFO: IniGalizaGon processed in 1153 ms Oct 21, 2013 4:42:15 PM org.apache.catalina.core.standardservice startinternal INFO: StarGng service Catalina ApplicaGon logs that are part of the product Developer logs for any code that was deployed Wrisen to local disk or network storage 14

Garbage Collector Logs 54.736: [Full GC 54.737: [Tenured: 172798K- >18092K(174784K), 2.3792658 secs] 257598K- >18092K(259584K), [Perm : 20476K- >20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] Generated by way of arguments passed to the JVM at startup - verbose:gc - Xloggc:/home/damien/jvm_logs/gc.log - XX:+PrintGC - XX:+PrintGCTimeStamps - XX:+PrintGCDetails 15

SplunkJavaLogging SomeGmes you can t write to file or deploy a UF Appenders for Java UGl Logging, Log4J, Logback Simply add a logging appender to your logging configuragon file 16

LogBack Appender Example 17

Code 18

Beser ExcepGon Logging 19

Easier to Work With in Splunk 20

Splunk SDK for Java Use the SDK from any JVM Language, Java / Groovy / Scala etc. Send log events via REST, UDP or TCP directly to Splunk from your code Spring IntegraGon Adaptors available on Github SDK available from dev.splunk.com 21

JMX = Java Management Extensions Monitor JVM via MBean asributes, operagons and nogficagons JVM MBeans Vendor MBeans Custom Coded MBeans 25

Browse MBeans with JConsole 26

Gehng this Data into Splunk Runs on all supported Splunk pladorms Works with all main JVM variants 100% Free and Open Source 27

Simple to Configure 28

Many ConnecGvity OpGons Let s look at this Splunk App in acgon 29

30 This is great for monitoring a single JVM

But what if we have dozens, hundreds, even thousands? 31

MulGple jmxserver Elements Per ConfiguraGon File 32

MulGple ConfiguraGon Files Per Splunk Instance 33

But Stacking VerGcally Will Only Get You So Far 34

Go Horizontal to Achieve Real Scale Indexer Cluster UFs with JMX Mod Input Monitored JVMs 35

In my experience, the data sources shown so far will be thorough enough for most of your JVM monitoring use cases But what if you want to go deeper? 36

InstrumentaGon Agents

SplunkJavaAgent Dynamic Byte Code InjecGon Agent for JVMs No need to change any code to get APM level insights This is all you pass to the JVM at startup : - javaagent:splunkagent.jar 38

What Raw Metrics Can You Get? " Bytecode InjecGon class loading method execugon method Gmings method call stack caught & uncaught excepgons " JMX running in local in- memory mode " Binary HPROF Memory dumps decoded in text no more stagc hprof dumps and then loading into another Mem Analysis tool 39

Simple ConfiguraGon File 40

Raw Events Streamed into Splunk 42

Use Splunk Search to Deliver Insights 43

OperaGng System

The JVM Executes as a Process Let s see what the host OperaGng System can tell us : top External programs that provide addigonal JVM insights : jstat Index this data in Splunk and correlate 45

Top 46

jstat 48

Distributed CommunicaGons

Valuable Data Flows in & out of JVMs Messaging Capture packets off the wire 51

Build a Splunk App!

Opportunity Reuse some of the data collecgon approaches in this presentagon and build out a specific JVM app Publish it on apps.splunk.com - Make some money - Generate publicity for yourself / your company - Generate sales leads - Sell more Splunk licenses if you are also a partner - Do it because you love community collaboragon - Become my beer buddy for life 53

Some JVM App Ideas Splunk for JBoss Splunk for Tomcat Splunk for Solr Splunk for Cassandra I am here to help you 54

QuesGons?

Special Offer: Try Splunk MINT Express for Free! Splunk MINT offers a fast path to mobile intelligence. How fast? Find out with a 6- month trial* Register for your free trial: hsp://mint.splunk.com/conf2014offer Download the Splunk MINT SDKs Add the Splunk MINT line of SDK code and publish** Start gehng digital intelligence at your fingergps! *Offer valid for.conf2014 aiendees and coworkers of aiendees only. **Trial allows monitoring of up to 750,000 monthly acqve users (MAUs). 56

THANK YOU @damiendallimore ddallimore@splunk.com