Using idoctorjob Watcher to find out complex performance issues

Transcription

1 2011 IBM Power Systems Technical University October Fontainebleau Miami Beach Miami, FL Using idoctorjob Watcher to find out complex performance issues Gottfried Schimunek 3605 Highway 52 North Senior IT Architect Rochester, MN Application Design IBM STG Software Tel Development Fax Lab Services IBM ISV Enablement Materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.3 Acknowledgements Thanks to: Power Systems Technical University Dawn May, STSM Technical Lead for System SW Development 1

2 IBM idoctor for i Job Watcher a rapid snapshot taker that allows real-time, detailed analysis an IBM i exclusive! Patented technology! 3 Speaker Notes This presentation will feature the Job Watcher component of the IBM idoctor for i suite of tools and services for performance investigation. The first few slides are an executive overview of Job Watcher which includes a high level look at some of the ways data is graphically presented. This will be followed by a few in depth examples of using Job Watcher. 4 2

3 Job Watcher Performance Data Collector What is Job Watcher a snapshot approach a non-intrusive collector collects one or multiple jobs/tasks or all jobs/tasks harvests job details such as waits, call stacks by default optionally retrieve SQL, activation group, communication details analysis can start after just two snapshots More 5 Speaker Notes Job Watcher (JW) is a REAL TIME performance analysis tool, that was created, initially, to harvest and report on the V5R1 Job Wait Accounting statistics and quickly grew from there. The harvesting interval can be set "as fast as possible", which, for a single TDE, is approx. 50ms. Note that the shorter the sampling interval, the closer to a TRACE the data becomes. Performance data that is unconditionally gathered on each sample includes: - all the usual job/process/task data (on a per thread basis) that Collection Services gathers (SM DASD I/O counts, XPF's Logical I/O counters, CPU usage, current user profile, etc.) - call stack, including LIC pgms - 32 job wait accounting timer and count buckets - current LIC wait object, current wait object VA and name, current LIC block "enum", current holding TDE (if applicable) - Java Wait object, Java thread name, other Java heap/gc details Optional data includes: - Activation group details: size/number of heaps, names/sizes of pgm activation static MI frames - Current or last SQL stmt executed - IP and Socket info for all open file descriptors attached to a TCPENDPOINT (remote IP addr, frame sizes, performance info etc.) - Triggers can be defined to call a user program - Run in lurk mode, data is only dumped when a defined threshold is reached 6 3

4 Job Watcher Graphical User Interface What is Job Watcher cont Windows Graphical Client application wizard for constructing the collection options viewer for managing the collections viewer for analyzing collections (during and post) over 100 predefined graphs setup a monitor to have JW running continuously user defined query and user defined graph facilities user can create queries and graphs save, categorize and transfer data viewer can also be used against other DB files like collection services 7 Job Watcher idoctor vs. IBM i 6.1 Performance Tools When to use idoctor vs. the new tools? General positioning: Basic, high-level performance management use new tools Advanced, detailed analysis continue to use idoctor October 2008 IBM Systems Magazine article comparing idoctor vs. IBM i 6.1 Performance Tools 8 4

5 Job Watcher - easy to use and non-intrusive to jobs being watched. - allows real-time, summarized or detailed views of how jobs spend time What if the job's run/wait signature looked like... running waiting example #1 DASD example #2 Blocks example #3 CPUq 9 Speaker Notes Job Watcher is used to tackle (expose) the details that can make up the many components of wait time. Job Watcher is our first choice for several reasons: - It's easy to use and non-intrusive to the jobs being watched. - It allows real-time, summarized or detail views of the data collected. We don't have to wait for, nor expend the resources for, traces to be dumped and analyzed. The Job Watch analysis is nearly immediate while trace results can take hours. This chart depicts three possibilities that could explain the large wait component shown: 1. Synchronous DASD IO waits waiting for an IO to complete before the program can advance 2. Waits for Blocks things like file seizes, record locks, gate waits, etc. 3. CPU queuing waiting for a turn at the CPU processor This graphical representation is part of the Job Watcher's presentation scheme. 10 5

6 JW use rule/action and the lurk facility with history satisfied Job Watcher "rule" starts up DB recording start investigation JW coverage the problem occurs T I M E dump reduction start investigation 11 Speaker Notes 1. Lurk mode 2. Start a Job Watcher collection and PEX trace 3. Problem occurs, rule is satisfied 4. PEX trace is stopped 5. JW can be defined to dump n number of intervals prior to the rule being satisfied, then end or start a new collection 12 6

7 Job Watcher Job Watcher in a nutshell Job Watcher a real-time, high-performance, snapshot sampler created to fill the gap between Collection Services - and - the short duration, high data volumes, long dump/analysis times of PEX Trace 13 It all starts here. 14 7

8 15 Speaker Notes Job Watcher Startup Options: Definition, we can choose a previously created definition or after creating one we can save it for future reuse. Job Watch Name, the name that will be used when creating the member in the Job Watcher files. We can let the system name it or we can use our own name. Library, the library where we are going to store our collection. Don t worry if it does not exist, Job Watcher will prompt us to create the library. Interval duration, enter the wait between snapshots We can also collect as fast as possible, but be careful in using this option. Maximum data to collect, Job Watcher will stop when this value is reached. Job Watcher will also stop when the ASP threshold is reached. Click the Configure button to configure Data collection options. 16 8

9 17 Speaker Notes The default is to collect a call stack with every snap shot. 18 9

10 19 Speaker Notes SQL statements: None is the default Currently active on each interval is generally recommended. Last executed is also available Access groups: (not shown) Not generally used no graphs associated with the data Sockets/Communications: (not shown) Not generally used no graphs associated with the data Rule: (not shown) Define harvested value conditions under which we can start a collection or end a collection 20 10

11 21 Speaker Notes Job/Task option Default is to collect all jobs, threads, and system tasks For this demo we will are going to select specific jobs and tasks Collecting idle jobs in the 1 st interval is also available. This will enable Job Watcher to collect job information in the first interval regardless if they used CPU or not during the 1 st interval of the collection

12 23 Speaker Notes Jobs/Task selection Using the Select by drop down menu we have the option to narrow down our job/task selection. See the Redbook for a more detail a description of each

13 25 Speaker Notes Refreshing this window will show the filter results where we can select specific jobs to be watched then add them to our list. The Add Selected button adds the selected jobs to the list. Pressing Close will show us the summary of those jobs we have elected to watch

14 27 Speaker Notes Collections will end when a certain condition is met: Intervals (default = 100) Size (default = 1GB) Time 28 14

15 Job Watcher Data Analysis Object Lock Contention 29 Speaker Notes Collection Services data was showing a lot of object lock contention. However, collection services data does not contain sufficient information to be able to identify which object. Job Watcher data was collected, and the combination of call stacks, waiters, and holders provided sufficient information to identify and solve the problem. This was a 6.1 pre-ga problem in the Service Monitor function that was corrected

16 31 Speaker Notes We start by looking at the Collection Overview Time Signature. It is clear from the graph that Object Lock Contention is pervasive

17 33 Speaker Notes From the Collection Overview Time Signature chart, we can drill down into the Threads signatures ranked by Object Lock contention chart. Doing this allows us to identify what jobs are involved in the object lock contention 34 17

18 35 Speaker Notes We find only one job. QSCLICEV. Let's look at the wait time signature for this job

19 37 Speaker Notes We will drill down into Thread wait time signature for the QSCLICEV job

20 39 Speaker Notes Notice at interval 47 there is a small carat (^) under the time legend this symbol indicates that a holder was identified at that interval. So we will drill down and display the call stack for this job

21 41 Speaker Notes The call stack display tells us many things... The object the job is waiting on WATCHEVENSPACE The holding job QZRCSRVS An option to show the holder call stack information And the call stack itself This combination of the call stack, the object being waited on, along with the thread that is holding the lock is the true value of Job Watcher 42 21

22 Job Watcher Data Analysis 43 Customer supplied examples of the problem Job Job Date Time Date Time Elapse Name Num Started Started Ended Ended seconds FRGJOB /4/ :46:26 5/4/ :46:37 11 FRGJOB /4/ :06:07 5/4/ :06:13 6 FRHJOB /4/ :06:08 5/4/ :06:13 5 FRGJOB /4/ :44:16 5/4/ :44:21 5 FRGJOB /4/ :59:25 5/4/ :59:29 4 FRGJOB /4/ :54:03 5/4/ :54:06 3 FRHJOB /4/ :54:03 5/4/ :54:06 3 FRHJOB /4/ :45:20 5/4/ :45:23 3 FRGJOB /4/ :45:20 5/4/ :45:23 3 FRGJOB /4/ :45:10 5/4/ :45:12 2 FRGJOB /4/ :36:56 5/4/ :36:58 2 FRGJOB /4/ :33:20 5/4/ :33:22 2 Nightly processing log 05/04/

23 Speaker Notes The nightly processing log indicates that there is a of job that took 11 seconds which is about 2X the amount of time expected

24 Speaker Notes Job Watcher uses a parent/child tree structure, so available graphing options are organized based on the category of the parent folder. On the left side of the window is an initial list of folders that come standard with Job Watcher. On the right side of the window is the initial list of graphs that come standard with Job Watcher for the selected folder. There is also a folder that contains the server side files. These are all of the files created by Job Watcher. These files can be used for creating your own customized queries and graphs User defined queries and graphs are saved in a separate folder on the client PC. The scope of these queries can be All systems, Current system, and Current library. The Collection Overview Time Signature by interval is generally the initial graph to view when a performance issue surfaces. It will give an indication as to the overall health of the system by interval for the entire collection period

25 Speaker Notes Wait graphs by thread show threads ranked by various wait codes. Notice the interval range on the right hand side in parenthesis. When we use JW for the first time the default is the last 100 intervals. We will want to change this to a larger range, here we are using 20,000 so that we will not have to worry about missing intervals

26 Speaker Notes Select Preferences from the Edit menu on the tool bar to set the range of intervals to summarize the graphs over. Tip: Sometimes it is useful to set the range just over the intervals during which the problem occurred rather than over the entire collection. 51 Graph scope Available graphs Fast path to JW supplied graphs Collection management options Collection details, system info

27 Speaker Notes For the experienced user there is a fast path to the graph menu options. Management options: Search, more on this later Summarize, this is important to run after a collection has stopped. This option fills in the gaps that can be created when a job spends multiple intervals in a a long wait Copy, copy the entire collection to another library Delete, delete the collection Split, copy only certain intervals to a new collection/library. This is helpful for very large collections. Transfer to, FTP the collection to another system, FTP server, even to the IBM FTP site, testcase.boulder.ibm.com The next slide will look at the collection properties. 53 Start with a search for the job referenced 54 27

28 Speaker Notes When we are provided with details for specific jobs or tasks that are experiencing performance problems. The Search option allows us to quickly zoom in on those jobs or tasks. This can greatly reduce our analysis time in determining the source of the performance problem being seen. 55 Start with a search for the job referenced Enter job name and time range 56 28

29 Speaker Notes We can Simplify our analysis by using the Search function to search on a single user id, job name, all jobs within a subsystem, etc In this demo we know that job FRGJOB had an 11 second transaction. The search on this job name and the time stamp when the 11 second transaction occurred will query the data and pull out just the intervals of interest. 57 Select an interval of a candidate and go for a run/wait signature 58 29

30 Speaker Notes So that we can see more of what Job Watcher has to offer we will look at the job s run/wait signature. This will show us a graph representing the selected job starting with the interval chosen. 59 Waiter s run/wait signature 60 30

31 Speaker Notes We also notice that by clicking Seize contention in the legend we were able to drag it to the top of the legend. This changed the order in which the waits are displayed on the graph. The default color representing seize contention time is displayed as purple. Using a right mouse click on the Seize contention entry within the legend we have the option to change the color representing that wait within the graph. 61 Menu Buttons Interval Arrows 62 31

32 Speaker Notes Selecting any tab other than Quick view will show the General section in which there can be up to three menu buttons displayed. We chose the Wait buckets tab here: 1. Job information, this button is always available. 2. Object waited on, this button is available if there is a wait object and the following reports are available: Total occurrences Jobs with this occurrences Occurrences by interval 3. Holding job or task, this button is available when a holder has been identified. Notice the arrows next to the interval field? These arrows allow you to scroll through the intervals for this job. The outside arrows will refresh the window with each interval. The inside arrows will not refresh the window, this allows you to quickly scroll through multiple intervals. The button on the left of the interval field is a refresh which you can use after you have used the inside arrows and are at the interval of interest

33 Speaker Notes Clicking on the Holding job button allows us to quickly display the holder s run/wait signature without having to go back and use the search function to find the job. Also shown here is the wait bucket tab, under this tab we see a list of all the wait buckets for this interval along with the percentage of time spent and the count for each bucket. 65 Drill down for call stack, holder, etc. or go for a collection overview 66 33

34 Speaker Notes In this graph we are back looking at the run/wait signature for the job FRGJOB. Notice the carrot marks below the time stamp? These indicate that a holder was identified during this interval. Another way to view the holder is by doing a right mouse click on the interval where a holder has been identified and selecting Holder. We can choose to view the holder s call stack or run/wait signature

35 Speaker Notes This shows the holder s call stack which in this case is showing that this job is running a query performing a sort of a result set stored in a temporary table. We can also go to the holder s run/wait signature, if we wish, by clicking on the Job information button then selecting the desired option. 69 Who is using CPU? 70 35

36 Speaker Notes Let s see what jobs are using the CPU at the time our job of interest is spending a lot of time in CPU queuing. Through a right mouse click on the RED value in the interval of interest we can access the graph Thread signature ranked by Dispatched CPU, this will be just for the interval we selected. We can also take a look at the Collection Overview Time Signature by interval to see how the system was behaving during the time our job was in CPU queuing. CPU queuing is generally seen when the current workload is more than the processor can handle

37 Speaker Notes This graph ranks the jobs in descending order by CPU usage. This helps in highlighting any jobs that might be monopolizing the CPU. Or it just might indicate that there were a lot of jobs all requiring CPU. Also showing the run/wait signature for each job, 73 Why is job# burning so much CPU? 74 37

38 Speaker Notes In the next graph we will see the call stack for the first job listed

39 Speaker Notes Notice that this job is also doing a sort Job Watcher is also capable of listing other jobs that have a similar call stack. Either in this interval or over all intervals within the collection. A single call or multiple calls within the call stack can be used as the search argument. 77 List other jobs that are also engaged in QQQTSORT 78 39

40 Speaker Notes Now we have a list of the jobs and the intervals where they were running the selected call stack section which includes the module QQQTSORT. Scroll down to the interval where the long transaction took place. Notice the number of jobs running a sort during this time 3 in interval in interval in interval

41 Speaker Notes A right mouse click on one of the jobs listed will allow us to display its call stack. We can then select the SQL tab to see the SQL statement being run if we had chosen to collect them when we started the watch

42 Speaker Notes Using the Collection Overview Time Signature by interval, we can definitely see that there are CPU related issues at the time our job had issues. Notice the high degree of CPU queuing system wide and the 100% CPU utilization. This tells us that we have to identify and understand why so much CPU is being used at this time and possibly correct what may be causing this condition. We can right mouse click on the interval of interest to get to the job signature graph. There are various ways in which to get to any given graph. 83 Conclusion: Using Job Watcher we were able to determine that there were a number of jobs running sorts consuming high CPU. This caused jobs to wait excessively in CPUQ resulting in seizes to be held longer causing our job to wait for a seize much longer than expected. DBMON and/or Visual Explain should be used to review the SQL statements and the access methods being used in an effort to understand the reasons for the sort

43 In Summary The Job Watcher Approach allows real-time and post-watch inspection of run/wait components provides results for day to day use by coders, testers, and operations as well as for presentations in the executive boardroom requires less skill to get started in performance investigation and makes the skilled, experienced performance investigator more productive 85 More Information idoctor Forum Documentation Education Redbook Downloads FAQ Consulting Services 86 43

44 Additional Notes Additional information is available via the idoctor website idoctor Forum Documentation Education Redbook Downloads FAQ Consulting Services 87 Questions 44

45 Questions? Performance and Scalability Services The IBM i Performance and Scalability Services Center can provide services, facilities and hardware IN ROCHESTER to assist you in testing hardware or software changes Traditional benchmarks Release-to-release upgrades Assess application performance when migrating to a new release of IBM i Stress test your system Determine impact of application changes Proofs of Concept (e.g. HA alternatives; SSD analysis, external storage, etc.) Evaluate application scalability Capacity planning all with the availability of Lab Services IBM i experts and development personnel To request any of these services, submit at: Or Cindy Mestad at cindysm@us.ibm.com 90 45

46 91 Trademarks and Disclaimers 8 IBM Corporation All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Cell Broadband Engine and Cell/B.E. are trademarks of Sony Computer Entertainment, Inc., in the United States, other countries, or both and are used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Prices are suggested U.S. list prices and are subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography IBM Corporation IBM Corporation 46