Performance Monitoring for the Java Virtual Machine (JVM ) Brian Doherty, Ben Kim, David Stoutamire J ava P erformance E ngineering S un Microsystems, Inc.
Presentation Goal L earn about non-intrusive performance monitoring and the tools available for monitoring production systems utilizing the J ava software stack. 2
Speaker s Qualifications Brian, Ben, and D avid are all members of the J ava P latform P erformance E ngineering group at S un Microsystems We represent expertise in performance measurement, analysis, and tuning of industry standard benchmarks, customer applications, the J ava class libraries, and the J ava HotS pot virtual machine 3
Performance Measurement A Product Lifecycle Process D on t stop monitoring your application just because it s been deployed! O bserve your production applications using minimally invasive tools that are becoming better integrated across the software stack. 4
Agenda W hat is performance monitoring Monitoring J ava applications J ava software stack D is tributed applications D emonstration T ools A nalysis T uning 5
Monitoring Goals Maintain and manage system health D etection of performance events D etection of application progress or liveness P erformance tuning D rilling down to find causes Identifying s olutions 6
Performance Management Process Start Analysis Design Code Benchmark Performance OK No Profile Monitor Yes Deploy 7
Profiling Vs. Monitoring P roduction vs. D evelopment P roduction systems have more restrictive meas urement needs P roduction loads vary over time P roduction loads difficult to simulate 8
Profiling Vs. Monitoring Profiling Monitoring Intrusion High Low D etail High Low A ccuracy High Low R equired K nowledge High Low 9
Complexities T he J ava S oftware S tack Application Application Server Virtual Machine Operating System Hardware 10
Complexities T he D eployment A rchitecture C lient T ier Web T ier J 2E E A pplication S erver T ier D atabase T ier 11
Monitoring Tools Network management systems O S monitoring tools J V M monitoring tools J 2E E monitoring tools 12
Demo
What We Saw S ymptoms E xces s ive memory utilization R estart T ools seen here R untime.freememory() gcore S erviceability A gent Identified s ource of memory cons umption 14
Other Tools P roduction -verbose:gc D evelopment -verbose:gc J V MP I/J V MD I profiling and debugging tools 15
Demo
What We Saw S ymptoms Inconsistent response times T ools seen here V isualg C Identified explicit S ys tem.gc() calls 17
Other Tools P roduction -verbose:gc mps tat identify s ingle threaded G C pstack print O S thread dump S IG INT print J ava thread dump D evelopment -X prof J V MP I/J V MD I profiling and debugging tools S erviceability A gent 18
Demo
What We Saw S ymptoms P oor system utilization L ow throughput T ools used Y A X (perfmeter) C P U only ~50% utilized S IG INT (thread dump) T hreads waiting on locks Identified s ource of lock contention 20
Other Tools P roduction B ytecode ins trumentation prs tat / top pstack D evelopment -X prof J V MP I/J V MD I profiling and debugging tools 21
What We Covered Monitoring the J ava S oftware S tack Application Application Server Virtual Machine Operating System Hardware 22
What We Covered Monitoring the J ava S oftware S tack Application Application Server Virtual Machine Operating System Hardware UNIX stat tools vmstat, iostat, mpstat, netstat kstat UNIX proc tools pstack, pmap, prstat/top, truss/strace Hardware counters cpustat, busstat Mis c. tools S E T oolkit 23
What We Covered Monitoring the J ava S oftware S tack Application Application Server Virtual Machine Operating System Hardware jvms tat tools C ommand line options -verbose:[gc class jni] -X prof -X X options S erviceability A gent J S R 174 interfaces 24
What We Covered Monitoring the J ava S oftware S tack Application Application Server Virtual Machine J MX interfaces S NMP, C IM/WBE M B ytecode ins trumentation V arious commercial tools J V MP I, J V MD I tools C ommercial and free Operating System Hardware 25
Summary P erformance measurement needs to be a product lifecycle proces s Not just a development process Non-intrusive tools are critical for meas urement of production applications T ools integrated across the software stack ease the measurement and analysis process 26
Resources HotS pot performance documentation http://java.s un.com/docs/performance jvms tat http://www.s un.com/developers/cools tuff S olaris performance documentation http://docs.s un.com J S R 174: Monitoring and Management S pecification for the J ava virtual machine 27
Q&A