Glassbox: Open Source and Automated Application Troubleshooting. Ron Bodkin Glassbox Project Leader

Transcription

1 Glassbox: Open Source and Automated Application Troubleshooting Ron Bodkin Glassbox Project Leader

2 First a summary Glassbox is an open source automated troubleshooter for Java applications. It monitors to detect performance problems and failures It analyzes the data to pinpoint causes Troubleshooting at every phase is key especially integrating new tech like AJAX and ESB You don t bake it in. Drop glassbox.war into your app server, and it troubleshoots your existing apps with ~1% overhead. Glassbox provides the open source community with an easy, automated troubleshooter, consider adding it to your stack

3 Agenda Glassbox Intro and Demo Using Glassbox Extending Glassbox Conclusion

4 Glassbox Troubleshoots Java Glassbox provides automated diagnosis Discovers application problems Suggests causes Rules out incorrect hypotheses Uses across development cycle Dev, QA, production

5 Glassbox Key Features Drop-in installation. Up in minutes on existing apps with no source or build changes needed. One click problem diagnosis, focus on the 80% of common issues Glassbox learns the app, the user doesn t need to configure it Flags Service Level Agreement violations Clear descriptions with supporting evidence Low overhead: won t slow down production applications. Extensible: add application-specific logic to general-purpose performance and failure detection

6 Enterprise Monitoring Users Support CEM Conversion rates down Can t log in Order entry screen slow Traction Problem Prioritization Problem Blip on a graph Ops Bug in the code Worrying trend Dev System Monitors

7 Glassbox Live Installation & Demo Install Troubleshoot existing app Real-time results

8 Agenda Glassbox Intro and Demo Using Glassbox Extending Glassbox Conclusion

9 Performance Low overhead at runtime suitable for production <1% increase in end to end response times Focused data capture on slow operations Not heavy instrumentation Low frequency (100 ms) sampling Glassbox uses Load-time weaving, which has little effect on end to end speed ~40% slower for class loading (initialization only) ~20% memory overhead

10 Glassbox 2.0 Open Source Non-invasive data capture Captures data with Aspect-Oriented Programming Uses Java 5 and server JMX data Glassbox Troubleshooter analysis Automatic diagnosis of the 80% of common problems Correlates, compares, analyzes data from data capture & summary Exposed through an AJAX Web client Open Source LGPL License Supports Java 1.4 and later

11 What is in the Glassbox 80%? Slow database query Failing database query Bottleneck or thread contention Slow remote call (web service, EJB, Ajax ) Failing remote call Database connection failure or slowness Java Mail Issues Broken FTP Failing operation Slow Java

12 What Glassbox Won t Do Solve problems that haven t happened yet Having a problem in hand means alerts are actionable. Diagnose every problem 80% of your effort goes toward resolving common problems, those are the ones we focus on Provide low-level data crunching tools Automate a people process Clear data improves collaboration more than workflow automation. Glassbox doesn t (for example) locate the owner of code, or send an to someone s manager. Web-accessible HTML does enable collaboration

13 Common Application Problems

14 Glassbox Concepts Operations Service-Level Agreements Response API Plug-in API Clustered Monitoring

15 Operations Glassbox rolls up performance by operation: A single request processed on behalf of an action e.g. Spring Controller, Servlet, JSP, EJB, JAX-WS, DWR call Glassbox counts each associated request towards a single operation it prioritizes operations (Spring Controllers are higher priority than Servlets, etc) it picks parents over children if equal priority in future we d like to add requires new semantics This is more application-oriented than the traditional profiler call tree world view Glassbox uses operations to define service levels

16 Service-Level Agreements Glassbox service level agreements are based on the operation Real user issues are not subtle under load: failing or slow code sticks out like a sore thumb. Failing operations: Glassbox presents any failure as serious. Slow operations: base SLA is any operation slower than 1 sec 5% of the time. Can be configured. Goal: more granular thresholds by specifying a hierarchical SLA level based on Java packages, classes, and annotations

17 Response APIs Glassbox has a listener API to track meaningful responses to various requests The console statistics come from summarizing these responses There are more detailed log topics available (with more coming) This also supports alternative forms of summarization Events can be generated based on: Aspect monitors Event listeners Interceptors Annotations Directly in code Thread sampling is currently done as an auxiliary approach but will be merged in the future

18 Plug-In API Allows applications added flexibility: Add Custom Operations or new Frameworks Define Custom SLA (e.g., stale cache) Define your own Analysis process Create a Custom UI: Velocity reports for operations Add additional functionality like automatic connection discovery

19 Clustered Monitoring Glassbox allows monitoring multiple servers The console can connect to other VMs via JMX/Remote or Direct RMI Connections are saved to a configuration database Advanced configuration via properties and Spring Glassbox clients display data for each (or you can select subsets) You can also connect remotely with JMX to get lower-level statistics (e.g., via JConsole) This also allows connecting to non-web containers from outside (e.g., monitoring a Swing UI)

20 Glassbox Architecture

21 Glassbox Data in JConsole

22 Agenda Glassbox Intro and Demo Using Glassbox Extending Glassbox Conclusion

23 Developing Glassbox: Monitoring with Aspects Aspects run automatically at well-defined points at runtime No need to instrument code Allows low overhead tracking Easy to update monitoring policies Enable and disable, even sampling Standardized support Extensible with open source AspectJ language. Load-time weaving avoids changes to build process. Spring AOP allows coarse-grained components Flexibility Reuse open source monitors for common APIs Easy to extend for custom monitoring

24 Glassbox is easily extensible With Spring style with Java 5 Annotations XML Schema-based in Spring config file Avoids time and complexity of weaving many classes Allows instance-based configuration With AspectJ XML configuration for simple style with Java 5 Annotations AspectJ code style Directly via response and plug-in APIs

25 AOP Concepts Aspects a type that can crosscut other types Join points well-defined points in program flow Pointcuts join point queries that match and bind context Advice block of code executed at specified pointcuts

26 AspectJ Overview The original AOP implementation Language annotation, and XML definition options Java platform compatible Performance comparable to hand-written equivalent Tool support Compiler, linker, classloader-based weaving IDE support: Eclipse, JBuilder, JDeveloper, NetBeans Ant, Maven, ajdoc, Java debugger Open source:

27 AspectJ Mechanisms Runtime system Javac/ajc FooService (.java source) FooService (bytecode) ajc (weave) FooService (modified bytecode) Ajc/javac ServiceMonitor (.aj/.java-source) ServiceMonitor (bytecode) ServiceMonitor (original bytecode) Relies on bytecode modification of aspect-affected classes Weave can happen at compile, post-compile, or load time Can package as class files, jars, or in memory bytecodes

28 AspectJ XML customizing Glassbox <aspectj> <weaver> <include within="my.service.service+"/> </weaver> <aspects> <concrete-aspect name="serviceprocessingmonitor" extends="glassbox.monitor.methodmonitor"> <pointcut name="monitoredpublicmethods" expression="within(my.service.service+)"/> </concrete-aspect> </aspects> </aspectj>

29 Sample AspectJ Code public aspect Monitor extends AbstractMonitor { public pointcut monitorpoint(object message) : within(javax.mail.transport+) && execution(* javax.mail.transport.send*(..)) && args(message,..); public Serializable getkey(object message) { return "mail://"; } } public String getlayer() { return Response.RESOURCE_SERVICE; }

30 Agenda Glassbox Intro and Demo Using Glassbox Extending Glassbox Conclusion

31 Glassbox is Open Source Automated Troubleshooting Provides problem diagnosis for applications Development: save troubleshooting time QA: find bugs faster Production: monitor apps and triage failures easily Key Features One click problem diagnosis Provides clear descriptions with supporting evidence Glassbox learns the app, the user doesn t need to Flags SLA violations, so you know which problems matter Low overhead with minimal impact on production applications. Enabled by noninvasive monitoring. Up in minutes on any app with no source or build changes needed.

32 Troubleshooting Tools should be a part of every stack 1. Aggressive data filtering can reduce the data haystack early 2. Troubleshooting intelligence means users need not be experts in the tool or have written the system Business Benefits 3. Cleaner data and more accurate diagnoses streamline communication Fix problems faster and cheaper by streamlining communication & minimize meetings to decide an owner. Fix more problems by enabling non-experts find them. Early and often is key, get everyone involved using troubleshooters early and save both time and money

33 Final summary Glassbox is an open source automated troubleshooter for Java applications. It monitors to detect performance problems and failures It analyzes the data to pinpoint causes Troubleshooting at every phase is key especially integrating new tech like AJAX and ESB You don t have to bake it in. Drop glassbox.war into your app server, and it troubleshoots your existing apps with ~1% overhead. Glassbox provides the open source community with an easy, automated troubleshooter, consider adding it to your stack

34 References Glassbox sample code, questions and documentation: Information Page: Downloads: Community: Forums:

35 Thank you! The Glassbox project is open source and welcomes users, feedback, and help. Please leave a business card if you d like to talk more. Send a question : contact@glassbox.com Reach me directly : ron.bodkin@glassxbox.com