All The Leaves Aren t Brown

Transcription

1 All The Leaves Aren t Brown Many Ways to Profile Your Application Code Chuck Ezell Senior Applications Tuner, datavail

2 Agenda Value in Profiling: the When, What & Why Profiling & Profilers: the right tool for the right job Server & Client Profiling Tools Summary 2

3 Value in Profiling 80% of unplanned outages are due to ill-planned changes made by operations or developers 60% of availability and performance errors are the result of mis-configurations Looking Ahead: through % of outages impacting mission-critical services will be caused by people and process issues. More that 50% will be caused by change/configuration/release integration and hand-off issues. 80% of incidents are caused by changes made to the IT environment including application code Source: Causes of Failure in Web Applications, Soila Pertet and Priya Narasimhan December 2005, Parallel Data Laboratory Carnegie Mellon University, Pittsburgh, PA

4 Tools Compared Questions We Have to Ask: How hard is it to Setup and Manage? Is it instrumented, a sampler or something else? What will it profile? Resources, Threading, Memory, Transactional Views, Single focused or Multi-faceted? What is the output? Will it keep history? How long? Is it useful as a proactive or reactive solution? Does it provide any analysis? Is it expensive? 4

5 Common Java Problems 1. Frequent or Long Garbage Collection App is spending too much time in GC Causes periods of extreme slowness across entire app 2. Application runs out of Heap Memory (Memory Leak) App runs out of memory Causes the application to crash 3. Thread Contention Thread locks on a resource for too long. Causes app to slow or lock and can impact entire app 4. Resource Contention (Pooled Resources) Not enough instances of a pooled resource (e.g. DB connections) Causes extreme slowness in app, can be wide spread or specific to certain transactions. 5

6 Profiling Tools Just to name a few jstat/jstack/jmap 6

7 Server Profiling Tools: jstack, jmap jstat Jstat, jstack & jmap Requires java 1.6 and up Only runs in the terminal Probably already installed Easy to use Does not retain any history unless you redirect it to a file No analysis provided just raw data Provides running threads & memory footprint Can show locked threads Does not require a kill -9 to the JVM for a dump Can be great in a pinch Setup: None jstat/jstack/jmap 7

8 Server Profiling Tools: jstack, jmap jstat You will need java 1.6 or higher to utilize these tools fully. To check your java version just enter java -version at the terminal. 8

9 Server Profiling Tools: jstack, jmap jstat Check your running JVMs by doing a quick ps ef grep Xmx 9

10 Server Profiling Tools: jstack, jmap jstat Using jps is much easier. Using the -l argument will give you the full package name. With this you can grep out JVMs. 10

11 Server Profiling Tools: jstack, jmap jstat Pass process id to jstack Using the l argument will give you locked threads jstack -l

12 Server Profiling Tools: jstack, jmap jstat What about long running threads or thread contention? Running a ps -el and grep for the JVM process id You get the lightweight process ids 12

13 Server Profiling Tools: jstack, jmap jstat Convert the binary lightweight thread id to hex. Then grep the output of jstack. You can find the lightweight process id for the thread in the thread stack after converting it into hex 13

14 Server Profiling Tools: jstack, jmap jstat Passing the same JVM process id to jmap will give you the memory footprint for this point and time. jmap -heap 3439 But how would we monitor the Garbage Collection (ongoing memory growth/shrink)? 14

15 Server Profiling Tools: jstack, jmap jstat Passing the same JVM process id to jstat will give you the garbage collection jstat gc jstat gcoldcapacity t

16 Server Profiling Tools: jstack, jmap jstat jstat gcnew -t Not a great proactive solution. 2. Great tools for reactive fire fighting. 3. Doesn t maintain history or reporting. 4. Output is essentially on screen and snapshot in time. 5. Will help answer questions and identify real problems now. 6. No Application Context & is not correlated with End User. 16

17 Server Profiling Tools: Visual VM VisualVM / Oracle Mission Control Both are free. Profiler much like the depricated jconsole Collects classes, threads, GC, CPU and more Capture Heap Dumps & Browse them Also built into NetBeans IDE (my IDE of choice) Has a plug-in architecture Setup: Easy 17

18 Server Profiling Tools: VisualVM You ll need JMX enabled JMX = Java Management Extensions Open a terminal and do a quick ps ef grep jmxremote Look for the JVM Process ID Then look for the port it s using for the JVM you want to monitor Might or might not need authentication 18

19 Server Profiling Tools: VisualVM 19

20 Server Profiling Tools: VisualVM 20

21 Server Profiling Tools: VisualVM JMX Console & JVM Browser 21

22 Server Profiling Tools: VisualVM Flight Recorder 22

23 Server Profiling Tools: VisualVM Flight Recorder 23

24 Server Profiling Tools: VisualVM 1. Not a great proactive solution. 2. Great visual tool for reactive fire fighting. 3. Maintains history in the form of independent snapshots. 4. Output is essentially on screen but can be called back up with the saved snapshots. 5. Will help answer questions and identify real problems now. 6. Is not correlated with End User Experience. 7. No transactional views. Hard to determine application context. 8. Does require JMXREMOTE settings to be added to JVMs. In addition to security & ports to attach remotely. Flight Recorder 24

25 Server Profiling Tools: Splunk Splunk There are often multiple JVMs and Apache instances running Often times we miss issues in the logs because there s too much to grep, wouldn t know what to look for or don t have the time Any log or flat file data can be forwarded Monitor code base for changes, network logs for drops, client machines The best results come from the best input so collect everything Create monitoring alerts, dashboards, configure transaction monitoring Plugin architecture for many different tools (BigIP, Hadoop, MySQL, etc ) Limited by time & creativity Splunk Forwarders On Each Server or write directly to server syslog. Data from the log is collected and forwarded on to Splunk where it s indexed and acted on. Setup: Moderate to Easy 25

26 Server Profiling Tools: Splunk We start with a basic SQL Script. And format the output into key/value pairs. SELECT 'sql_plan_timestamp=""' to_char(timestamp,'mm-dd-yyyy hh24:mi:ss') '", database=' b.name ', plan_hash_value=' plan_hash_value ', sql_id=' sql_id ', operation="' operation '", options=' options ', object_name=' object_name ', object_owner=' object_owner ', optimizer=' optimizer ', cost=' cost ', cardinality=' cardinality ', cpu_cost=' cpu_cost ', io_cost=' io_cost ', time=' time ', filter_predicates="' filter_predicates '"' as sql_plan_info FROM V$SQL_PLAN a, v$database b where (options like '%SKIP%' OR options like '%FULL%' OR options like '%PUSH%')--OR options like '%RANGE%') and object_owner!= 'SYS' and timestamp > (sysdate - 1) order by timestamp asc; Then add that SQL to a shell script and set cron to run it and produce the output. sql_plan_timestamp=" :51:01", database=oadev, plan_hash_value= , sql_id=696c56fyx1rwn, operation="index", options=full SCAN, object_name=gl_sets_of_books_u2, object_owner=gl, optimizer=, cost=1, cardinality=1, cpu_cost=7521, io_cost=1, time=1, filter_predicates="hoi2"."org_information1"=to_char("gsob"."set_of_books_id")" sql_plan_timestamp=" :51:01", database=oadev, plan_hash_value= , sql_id=36dx1vwfqmn2j, operation="table ACCESS", options=full, object_name=fnd_product_groups, object_owner=applsys, optimizer=, cost=2, cardinality=1, cpu_cost=7271, io_cost=2, time=1, filter_predicates="sql_plan_timestamp=" :51:01", database=oadev, plan_hash_value= , sql_id=7m21y56326kha, operation="index", options=full SCAN, object_name=gl_sets_of_books_u2, object_owner=gl, optimizer=, cost=1, cardinality=1, cpu_cost=7521, io_cost=1, time=1, filter_predicates="hoi2"."org_information1"=to_char("gsob"."set_of_books_id")" 26

27 Server Profiling Tools: Splunk 27

28 Server Profiling Tools: Splunk Client Tier Capture Opportunities Web Browser traffic Oracle Forms running in client side JVM Windows Event Logs CPU Load, Memory, Network Stats Active Directory Monitoring FND Diagnostics from Oracle Forms 28

29 Server Profiling Tools: Splunk Application Servers Concurrent processing server Application Tier Capture Opportunities Oracle JVM OACore JVM Threads JSP Request/Response Apache Requests (Access/Error) Forms Processes Linux/AIX Server Stats 29

30 Server Profiling Tools: Splunk Database Tier Capture Opportunities Oracle Alert Logs User Session Data Concurrent Requests SQL IDs, SQL Profiles & Outlines SMON/PMON DBWR/LGWR Hot Database Objects 30

31 Server Profiling Tools: Splunk 31

32 Server Profiling Tools: Splunk 32

33 Server Profiling Tools: Splunk 33

34 Server Profiling Tools: Splunk 34

35 Server Profiling Tools: jmap/jstack jstack & jmap can be setup to forward data to splunk. 35

36 Server Profiling Tools: jmap/jstack 1. Not out of the box solution requires some planning and skill. 2. Can be a great proactive solution. 3. Great visual tool for reactive AND proactive fire fighting. 4. Maintains history in the form of indexed data. 5. Output can be anything: alerts, notifications, pdf reports, charts, dashboards, html pages, etc 6. Will help answer questions and identify real problems now AND provide data to research patterns over time. 7. Does require setup and server to collect and maintain indexes. 8. Can provide the greatest flexibility for building what you want if you have vision, time, skill & creativity. 9. It is priced reasonably well based on the data indexed per day. 36

37 Server Profiling Tools: HP Diagnostics HP Diagnostics Integrates will with LoadRunner Full Application/Server Diagnostics Java Profilers can be configured to save results Will capture details for analysis later Is a bit confusing to use Older architecture utilizing some client tools still Profilers Can be server intensive Instrumented Setup: Involved 37

38 Server Profiling Tools: HP Diagnostics 38

39 Server Profiling Tools: HP Diagnostics 39

40 Server Profiling Tools: HP Diagnostics 1. Can be a great proactive solution. 2. Great visual tool for reactive and proactive fire fighting. 3. Maintains history in the form of snapshots and data. 4. Output is saved snapshots and can be recalled from history. 5. Will help answer questions and identify real problems now as well as help identify patterns. 6. Each java probe can be intensive (with more probes, etc ) 7. You don t know what you don t know. 8. Is not correlated with End User Experience. 9. No transactional views. Hard to determine application context. 10. Can be costly to purchase. 40

41 Server Profiling Tools: HP Diagnostics AppDynamics Captures Full Transaction Provides Monitoring History Full Profiler down to threads and exception stacks Auto Detects other devices and services such as other DBs Employs powerful and efficient Java Byte Code Injection Full Splunk Cross Integration Fast Root Cause Analysis Alert monitoring & configurable SLAs Scheduled & Automatic Diagnostic Sessions Modern architecture (just plain pretty) Very low server resource consumption (Netflix runs thousands) Setup: Easy 41

42 Server Profiling Tools: AppDynamics Dashboard Runtime application architecture discovery, modeling and display Number 2 on Gartner's 5 Dimensions of APM chart. 42

43 Server Profiling Tools: AppDynamics Not instrumented Auto detects all connections through Java Byte Code. 43

44 Server Profiling Tools: AppDynamics User-defined transaction profiling. This provides application context that is intuitive. Number 3 on Gartner's 5 Dimensions of APM chart. 44

45 Server Profiling Tools: AppDynamics 45

46 Server Profiling Tools: AppDynamics Component deep-dive monitoring in application context. - full call stacks and find bottlenecks. Number 4 on Gartner's 5 Dimensions of APM chart. 46

47 Server Profiling Tools: AppDynamics 47

48 Server Profiling Tools: AppDynamics 48

49 Server Profiling Tools: AppDynamics 49

50 Server Profiling Tools: AppDynamics 50

51 Server Profiling Tools: AppDynamics Health Rule Notifications JVM, SQL, Transaction Slowness, etc 51

52 Server Profiling Tools: AppDynamics 52

53 Server Profiling Tools: AppDynamics Monitor JVM Heap Segments and Ratios 53

54 Server Profiling Tools: AppDynamics End-user experience monitoring Number 1 on Gartner's 5 Dimensions of APM chart. 54

55 Server Profiling Tools: HP Diagnostics 1. Super Easy to configure and get started. 2. Great visual tool for reactive and proactive fire fighting. 3. Maintains full history in the form of diagnostic snapshots and data. 4. Output can be alerts, notifications, pdf reports, dashboards, etc. 5. Will help answer questions and identify real problems now as well as help identify patterns. 6. Very little overhead (often less than 2%). 7. You don t have to know what you don t know. Plug and play. 8. Can be costly to purchase if you re running multiple JVMs. 9. But there is a free version for single JVM setups. 55

56 Client Profiling Tools: Fiddler Setup: Easy Fiddler (HTTP Request Profiler) Easy setup just download and launch Create, save & recreate http requests Investigate & Debug http header problems Capture or Create form submits and re-run them anytime 56

57 Client Profiling Tools: Fiddler Review http request statistics 57

58 Client Profiling Tools: Fiddler Capture and Manipulate http requests for testing purposes 58

59 Client Profiling Tools: Fiddler Recreate http POST data requests for testing 59

60 Client Profiling Tools: Fiddler Simulate connection speeds, disable cache, measure response time 60

61 Tools Compared Threads Memory Transaction Views Maintains History Proactive Monitoring Analysis Setup Splunk If Logged If Logged If Logged YES YES No Moderate HP Diagnostics YES YES No YES YES Sort Of Involved AppDynamics YES YES YES YES YES YES Easy jstack/jmap YES YES No VisualVM YES YES No VisualStudio YES YES No Fiddler No No YES Firebug No YES YES Redirect To Flat File Limited Exports Limited Exports Limited Exports Limited Exports No No Easy YES Sort Of Easy No YES Easy No No Easy No No Easy 61

62 Questions? Profiling & Profilers: the right tool for the right job Server Profiling Tools Client Profiling Tools The Heat of the Moment: differentiating profiling tasks Chuck Ezell Senior Applications Tuner, datavail 62