Java Monitoring Stuff You Can Get For Free (And Stuff You Can t) Paul Jasek Sales Engineer
A Bit About Me Current: Past: Pre-Sales Engineer (1997 present) WaveMaker Wily Persistence GemStone Application Developer (1988 1997) Hobbies Vinyl Dow Jones Dun & Bradstreet General Electric / RCA Lionel trains iphone Development 2 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Today s Agenda 1 Let s debug a Slow Application 2 Try out the Free Monitoring Tools 3 Why should I Pay for a Monitoring Tool? 4 The changing Application Landscape 5 How people are benefitting from APM? 3 Copyright AppDynamics. All rights reserved. "
App is running Slow AND It s running in Production Can t restart the servers Can t reproduce in non prod environment Data sets and load are not alike Just released a new version and it includes several refactoring, a new deployment architecture Revenue critical app, can t make it any slower by overloading it with intrusive monitors, additional logging Development team is at a different location and time zone 4 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Home grown - Logs, JMX Dashboards etc. When it works First level of defense Often added based on learning from outages and critical infrastructure components (pools, caches) Systemic issues, e.g. recurring error / warn messages When it doesn t Sheer volume, don t know where to start, after all logs are flat The issue is triggered by specific users, data, regions, etc. Logging libraries are often synchronous, and heavy on string operations, hence can lead to performance issues What if you forgot to add isdebugenabled() Time and Server correlation is not easy Server Time vs. Actual Outage time 5 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Let s try some free tools JConsole ships with all JVMs Visual VM Exclusive to Oracle (Sun) JVM Eclipse Memory Analyzer Heap Dump Analysis The Game Changer AppDynamics Lite 2.0 6 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Let s go classic - JConsole JVM Health 7 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Let s go classic - JConsole Thread Dumps Mbean Browser 8 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Let s go classic - JConsole Mbean Browser 9 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
JConsole Summary When it Works JVM needs tuning Get metrics for Memory, CPU, Threads and Garbage Collection Experiment with Memory Pools and GC Algorithms Systemic slowness, thread dumps point at areas Server MBean and Custom JMX provides immediate clues When it doesn t Metrics have no contextual information so no clue why thread is stuck on a socket! You don t know / own the source code and dev is not next to you It s not a JVM tuning issue? Passive, always have to be on watch Overhead 10 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
New Age - VisualVM How are my threads doing? 11 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
New Age - VisualVM Where are my Threads spending their time? A Thread dump to show what s behind the socket read 12 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
New Age - VisualVM What are my Threads munching on? My App Classes 13 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Visual VM Summary When it Works JVM Metrics overview : Memory, CPU, Threads, GC etc. Thread Summary View Wait, Sleep, Monitor, Run Time Sampling provides some initial clues to the problems Thread Sampling Where time is spent most often Memory Sampling What objects are created most often When it doesn t All the drawbacks of JConsole The profiler option (instruments all classes and methods) has way too much overhead even for trivial apps, so not an option Ships only for Sun (Oracle) JVM JRockit ships with a similar tool named JRA Nothing notable for IBM JVMs for free 14 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Out of Memory! - Eclipse Memory Analyzer Leak Suspects Retained Heap Table 15 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Eclipse Memory Analyzer - Summary When it works You have a Heap Dump generated though XX options Open up Leak Suspect Report and Check out the actual content of the Object (Map <Key, Value>) Trace the outgoing and incoming references on the object (Who is holding the map) It happens often and you know why it happens (you have some smart engineers working with you!) When it doesn t! Knowing what s big is less important, what you need is How it got big? Who is adding / putting elements Doesn t happen all the time and you can t associate with a specific event, it grows in lock step, not linear! 16 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
The Game Changer AD LITE 2.0
AppDynamics Lite 2.0 Application flow map Identifies key components Servlets, Struts, EJBs External calls HTTP, RMI, JMS, JDBC, THRIFT etc. Top Down View of Application Recognizable artifacts e.g. URI vs. Class Name Aggregated at the top, as experienced by real user Dynamic Thresholds Learns by itself what s slow for Login vs. Checkout Continuous visibility and Deep Dive when needed Every click is monitored, capture outliers Slow, Errors, Stalls Your very own dashboard App, BT, JMX Metrics Always ON, alerts when things go wrong Built for Production < 2% Overhead 18 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Shameless Plug http://www.appdynamics.com/ products-free-download.php 19 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
When does it make sense to Pay for a Monitoring Tool?
If your app Deployed on more than 5 JVMs Need to see data at one place, be able to navigate easily Request processing is multi threaded, distributed Need correlation across threads and services Agile releases, Now vs. Then Last 5 mins! Last Week! Last Quarter Custom Data Capture, Slicing and dicing of the data Capture Parameters, Select where session_id =, All slow SQLs Advanced Baselines and Policy Engine Static / Dynamic / Seasonal Baselines Making sense from the metrics deluge Dashboards, Business Metrics Pre built, custom widgets, live business metrics Moving towards Cloud Architecture 21 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
The Commercial Landscape The Big 5s IBM CA Compuware HP BMC The New Age Players AppDynamics New Relic 22 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
Must haves for your Paid Tool No professional services Plug and Play, no experts needed to configure and tune the tool You don t tell the Tool, the Tools tells you what you got Self adaptive learning Always ON in production with very low overhead Not a factor of how much traffic Easy to learn the tool and extend too Start with the basics and be an expert over time Easy to share and collaborate across the Organization Operation, Dev, QA, Performance, Architects Total Cost of Ownership include everything Software, Hardware, Deployment, Learning, Maintenance Continuous Innovation Cloud, NO SQL, Async Transactions 23 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
AppDynamics Pro Instrumentation Point API Data& Collec+on& API& Cloud& Connector& API& Application Server Agent (JVM/CLR) Business Metric Collection API Custom Header Correlation API Controller Alert& Integra+on& API&/&Scripts& Data&Feed& API& Machine Agent Custom Data Collection API / Scripts Custom Action API / Scripts
How is APM Helping Organizations?
AppDynamics in Action: Netflix A nice problem to have... http://techblog.netflix.com/2011/02/redesigning-netflix-api.html 37x Growth Jan 2010-Jan 2011 A Blockbuster in the Cloud World s largest Java public cloud deployment 26 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
27 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " 1ST CLICK: THE APP TOPOLOGY
2ND CLICK: REVIEW SLOW TRANSACTIONS 28 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
29 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " 3RD CLICK: TRIAGE LOCATION
30 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " 4TH CLICK: FIND THE LINE OF CODE
AppDynamics in Action: Karavel, France E Travel Site Tomcat, Active MQ, My SQL 200 Business Transactions 31 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "
32 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " CUSTOM DASHBOARDS
33 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " DRILL DOWN ON SLOW TRANSACTIONS
34 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " FIND THE CODE AND THE BAD SQL
35 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " VALIDATE THE FIX!
36 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. " FUTURE PROOF WITH ALERTS
Q&A Q&A 37 March 5, 2012 Copyright 2010 AppDynamics. All rights reserved. "