Copyright 2014 Splunk Inc. Splunking the JVM. Damien Dallimore. Dev Evangelist, CSO Splunk

Transcription

1 Copyright 2014 Splunk Inc. Splunking the JVM Damien Dallimore Dev Evangelist, CSO Splunk

2 Disclaimer During the course of this presentagon, we may make forward- looking statements regarding future events or the expected performance of the company. We caugon you that such statements reflect our current expectagons and esgmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentagon are being made as of the Gme and date of its live presentagon. If reviewed aser its live presentagon, this presentagon may not contain current or accurate informagon. We do not assume any obligagon to update any forward- looking statements we may make. In addigon, any informagon about our roadmap outlines our general product direcgon and is subject to change at any Gme without nogce. It is for informagonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligagon either to develop the features or funcgonality described or to include any such feature or funcgonality in a future release. 2

3 From Middle Earth Make Splunk Apps & Add- ons JVM background 3

4 4

5 apps.splunk.com github.com/damiendallimore 5

6 What is this JVM thing?

7 JVM = Java Virtual Machine First appeared in the early 90 s Now the dominant rungme for enterprise applicagons ApplicaGon Servers Enterprise Service Buses Databases NoSQL Distributed Big Data Web Servers Directory Servers Search Engines Build Systems Gaming Pladorms Trading Systems ReservaGon Systems Core Banking Messaging Infrastructure Proprietary Systems 7

8 8

9 Many JVM Variants Oracle Hotspot OpenJDK Oracle JRockit IBM J9 Azul Zing More or less the same. OpenJDK is reference impl for Java 7/8 Gehng merged into Hotspot Commercial alternagves 9

10 JVMs are here to stay It s not just about the J Big Data frameworks You should care about gehng insights into your JVM data 10

11 Gehng at the Data

12 Data Sources Logs Developers JMX (Java Management Extensions) InstrumentaGon Agents OperaGng System Distributed CommunicaGons 12

13 Log Data

14 Standard Log Files Oct 21, :42:15 PM org.apache.catalina.startup.catalina load INFO: IniGalizaGon processed in 1153 ms Oct 21, :42:15 PM org.apache.catalina.core.standardservice startinternal INFO: StarGng service Catalina ApplicaGon logs that are part of the product Developer logs for any code that was deployed Wrisen to local disk or network storage 14

15 Garbage Collector Logs : [Full GC : [Tenured: K- >18092K(174784K), secs] K- >18092K(259584K), [Perm : 20476K- >20476K(20480K)], secs] [Times: user=0.56 sys=0.05, real=0.07 secs] Generated by way of arguments passed to the JVM at startup - verbose:gc - Xloggc:/home/damien/jvm_logs/gc.log - XX:+PrintGC - XX:+PrintGCTimeStamps - XX:+PrintGCDetails 15

16 SplunkJavaLogging SomeGmes you can t write to file or deploy a UF Appenders for Java UGl Logging, Log4J, Logback Simply add a logging appender to your logging configuragon file 16

17 LogBack Appender Example 17

18 Code 18

19 Beser ExcepGon Logging 19

20 Easier to Work With in Splunk 20

21 Splunk SDK for Java Use the SDK from any JVM Language, Java / Groovy / Scala etc. Send log events via REST, UDP or TCP directly to Splunk from your code Spring IntegraGon Adaptors available on Github SDK available from dev.splunk.com 21

22 22

23 23

24 JMX

25 JMX = Java Management Extensions Monitor JVM via MBean asributes, operagons and nogficagons JVM MBeans Vendor MBeans Custom Coded MBeans 25

26 Browse MBeans with JConsole 26

27 Gehng this Data into Splunk Runs on all supported Splunk pladorms Works with all main JVM variants 100% Free and Open Source 27

28 Simple to Configure 28

29 Many ConnecGvity OpGons Let s look at this Splunk App in acgon 29

30 30 This is great for monitoring a single JVM

31 But what if we have dozens, hundreds, even thousands? 31

32 MulGple jmxserver Elements Per ConfiguraGon File 32

33 MulGple ConfiguraGon Files Per Splunk Instance 33

34 But Stacking VerGcally Will Only Get You So Far 34

35 Go Horizontal to Achieve Real Scale Indexer Cluster UFs with JMX Mod Input Monitored JVMs 35

36 In my experience, the data sources shown so far will be thorough enough for most of your JVM monitoring use cases But what if you want to go deeper? 36

37 InstrumentaGon Agents

38 SplunkJavaAgent Dynamic Byte Code InjecGon Agent for JVMs No need to change any code to get APM level insights This is all you pass to the JVM at startup : - javaagent:splunkagent.jar 38

39 What Raw Metrics Can You Get? " Bytecode InjecGon class loading method execugon method Gmings method call stack caught & uncaught excepgons " JMX running in local in- memory mode " Binary HPROF Memory dumps decoded in text no more stagc hprof dumps and then loading into another Mem Analysis tool 39

40 Simple ConfiguraGon File 40

41 41

42 Raw Events Streamed into Splunk 42

43 Use Splunk Search to Deliver Insights 43

44 OperaGng System

45 The JVM Executes as a Process Let s see what the host OperaGng System can tell us : top External programs that provide addigonal JVM insights : jstat Index this data in Splunk and correlate 45

46 Top 46

47 47

48 jstat 48

49 49

50 Distributed CommunicaGons

51 Valuable Data Flows in & out of JVMs Messaging Capture packets off the wire 51

52 Build a Splunk App!

53 Opportunity Reuse some of the data collecgon approaches in this presentagon and build out a specific JVM app Publish it on apps.splunk.com - Make some money - Generate publicity for yourself / your company - Generate sales leads - Sell more Splunk licenses if you are also a partner - Do it because you love community collaboragon - Become my beer buddy for life 53

54 Some JVM App Ideas Splunk for JBoss Splunk for Tomcat Splunk for Solr Splunk for Cassandra I am here to help you 54

55 QuesGons?

56 Special Offer: Try Splunk MINT Express for Free! Splunk MINT offers a fast path to mobile intelligence. How fast? Find out with a 6- month trial* Register for your free trial: hsp://mint.splunk.com/conf2014offer Download the Splunk MINT SDKs Add the Splunk MINT line of SDK code and publish** Start gehng digital intelligence at your fingergps! *Offer valid for.conf2014 aiendees and coworkers of aiendees only. **Trial allows monitoring of up to 750,000 monthly acqve users (MAUs). 56

57 THANK