talent. technology. true business value

Transcription

1 March 26, 2008 JPMorganChase presents Java Debugging and Troubleshooting with No Source Code in Sight By Minoy Mathew talent. technology. true business value

2 The proliferation of the black and the gray boxes Buy Build Reuse Assemble Vendor products (black box) Open source products (gray box) Reusable components (gray box) Dependencies with other systems (black box)

3 Topics Information Visualization Information Dumps Debugging without Source Correlating Information Knowledge Information Data

4 Thinking like the programmer who wrote it! Patterns Globalized Industry Common mistakes and best practices She would have had to make it work with another black box too Some rules are ubiquitous

5 What s in a name Major Major Major

6 Leaking Map

7 Tools: Visual VM

8 Heap Dump

9 Homing in on the leak

10 JMX: Introspection into the JVM -- JConsole

11 Named your fear must be before banish it you can - Master Yoda (Star Wars)

12 Trends

13 Information Visualization: Squiggly Lines Profession Neurologist - EEG

14 Squiggly Lines Profession Java Developer

15 Garbage When Apps Panic

16 Garbage Finding Patterns

17 Garbage Findings GC panic was recoverable Indicative of an expiring cache usage? Activity persisted even with no load A background thread or process? Cyclical nature to recovery (~ 1 hour) Indicative that cache and background thread are related? Root Cause What Really Happened: A culprit background thread refreshing lookup data into an UNBOUNDED cache that had a TIMED EXPIRATION of an hour.

18 Garbage Problem Solved

19 Garbage and Squiggly Lines Tools and Learnings Know your GC algorithms and their repercussions java -Xms600m -Xmx600m -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled - XX:PermSize=64m -XX:MaxPermSize=256m -XX:NewSize=96m References: Increase your GC verbosity! Force the GCs and look for patterns Load Testing

20 Association, correlation and causal analysis

21

22 Java Process and Thread

23 Correlating it to DB info

24 Correlating it to DB info

25 Correlating it to DB info

26 More profiling

27 Always in motion is the future Master Yoda (Star Wars)

28 DEADLOCK!

29 Thread dump: kill -3

30 Java Thread Dumps Stuck in a Service Call "Portlet Runner Thread Pool Thread-1" daemon prio=5 tid=0x nid=0x109 runnable [96ffe fffc28] at java.net.socketinputstream.socketread0(native Method) at java.net.socketinputstream.read(socketinputstream.java:129) at java.net.socketinputstream.read(socketinputstream.java:182) at java.io.filterinputstream.read(filterinputstream.java:66) at java.io.pushbackinputstream.read(pushbackinputstream.java:120) at org.apache.commons.httpclient.httpparser.readrawline(httpparser.java:109) at org.apache.commons.httpclient.httpparser.readline(httpparser.java:135) at org.apache.commons.httpclient.httpconnection.readline(httpconnection.java:1086) at org.apache.commons.httpclient.httpmethodbase.readstatusline(httpmethodbase.java:2188) at org.apache.commons.httpclient.httpmethodbase.readresponse(httpmethodbase.java:1949) at org.apache.commons.httpclient.httpmethodbase.processrequest(httpmethodbase.java:2659) at org.apache.commons.httpclient.httpmethodbase.execute(httpmethodbase.java:1093) at com.mycompany.utils.gzipawarepostmethod.execute(gzipawarepostmethod.java:81) at org.apache.commons.httpclient.httpclient.executemethod(httpclient.java:675) at org.apache.commons.httpclient.httpclient.executemethod(httpclient.java:529) Then use Netstat or some other tool to find where the connection is being made to

31 Java Thread Dumps Waiting is a Good Thing, But Beware of the Sleepers! "Entitlement Thread Pool Thread-2" daemon prio=1 tid=0x00b08a28 nid=0xed in Object.wait() [980ff ffc28] at java.lang.object.wait(native Method) - waiting on <0xbc0236a0> (a com.mycompany.utils.threadpoolimpl$poolthread) at com.mycompany.utils.threadpoolimpl$poolthread.run(threadpoolimpl.java:122) - locked <0xbc0236a0> (a com.mycompany.utils.threadpoolimpl$poolthread) "Cache (LookupCache) Maintenance Thread" daemon prio=5 tid=0x0045e820 nid=0xa9 waiting on condition [986ff ffc28] at java.lang.thread.sleep(native Method) at com.mycompany.cache.jbosscache$maintenancethread.run(jbosscache.java:120)

32 Java Thread Dumps Not dependable when JVM in GC Panic "ExecuteThread: '2' for queue: 'weblogic.admin.rmi'" daemon prio=5 tid=0x002003e8 nid=0x49 runnable [9db7d000..9db7fc28] at java.net.socketoutputstream.socketwrite0(native Method) at java.net.socketoutputstream.socketwrite(socketoutputstream.java:92) at java.net.socketoutputstream.write(socketoutputstream.java:136) at com.sybase.jdbc2.tds.pduoutputformatter.doflush(pduoutputformatter.java:212) at com.sybase.jdbc2.tds.pduoutputformatter.flush(pduoutputformatter.java:155) at java.io.filteroutputstream.flush(filteroutputstream.java:123) at com.sybase.jdbc2.tds.tds.language(tds.java:698) at com.sybase.jdbc2.jdbc.sybstatement.sendquery(sybstatement.java:1451) at com.sybase.jdbc2.jdbc.sybpreparedstatement.sendquery(sybpreparedstatement.java:1158) at com.sybase.jdbc2.jdbc.sybstatement.executeupdate(sybstatement.java:1621) at com.sybase.jdbc2.jdbc.sybpreparedstatement.executeupdate(sybpreparedstatement.java:91)

33 Remote Debugging Inspection of all Variables Possible

34 Remote Debugging Don t Need Source to Set a Breakpoint

35 Remote Debugging Tools and Learnings Need to enable remote debugging at the command line: java -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=7010 Most modern IDEs have remote debugging capabilities built in: Fairly safe to use in disparate environments WHEN NEEDED! Targetted use of breakpoints can home in on race condition and deadlock errors.

36 Fallacies: Post hoc ergo propter hoc -- Correlation is not causation Judge me by my size, do you? Master Yoda (Star Wars)

37 Lots of Stuff Not Covered Java Scripting - BeanShell, Java 6 scripting Open the JVM so that you can connect to it and inspect anything! OS level tracing: prstat, pstack, pmap, truss, netstat Log Scraping Decompilation JAD injection of decompiled classes into an app Proxies Monitor the communication to and from an app