Get Dirty with Diagnostics: Agile Performance Tuning at the Lowest Level

Transcription

1 Get Dirty with Diagnostics: Agile Performance Tuning at the Lowest Level Adam Reincke Performance Testing Consultant

2 My Background Adam Reincke Performance testing consultant at JDS Australia AIS and ASE certified in LoadRunner Recently worked on a large agile project using Diagnostics JDS Australia HP Platinum partner, providing consulting services in Performance Testing, Application Monitoring, etc. 3 JDS

3 Session Overview Test late? Test once? Typical challenges with performance testing and tuning. Diagnostics and Agile How we can combine the two, to get more out of both Test early! Test often! Benefits of rolling up your sleeves, and getting in early Get Dirty! Step by step process. An approach to performance tuning with agile Things to look for Common tuning opportunities to look out for with Diagnostics Beware Easy traps to fall into when performance tuning in an Agile environment Case Study How I ve recently implemented this model at Australia's largest telecommunications provider Follow up resources and questions 4 JDS

4 Test late? Test once? Performance testing/tuning doesn t typically align with the agile lifecycle Usually done once, at the end of a waterfall development cycle Can be a checkbox activity Don t worry, we ve allocated a week before go-live to run the performance tests When defects are found, it is usually too late, or very costly, to fix them Go live dates impending Pre-production environments are locked down Budgets are running dry Less access to developers Who wrote the faulty/slow piece of code? Are they available to debug the defect? When did they write it? 6 months ago? Restricted options to fix an issue under pressure Delay go-live Buy more hardware??? It ll do (revise requirements) 5 JDS

5 Diagnostics? What s That Again? HP tool standalone plus LoadRunner integration Provides code-level performance monitoring Java and.net applications (and SQL) Hooks into entry/exit points of specific methods via instrumentation Isolate bottlenecks by application layer, server request, objects and methods, resource utilisation etc. Server time only! Does not include client time, network etc. = 6 JDS

6 Agile?? Nope, I m Still Confused What do you think of when you hear the word Agile? Sprints, Stories, iterations Team, scrum, continuous delivery Iterative and incremental development and testing Splits tasks into small, functionally related areas (stories, modules, etc.) Each task has a specific owner Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan Fast and flexible. Responsive to change. Ideal when requirements are unclear Allows prototyping, experimenting Collaboration Teams working closely together Minimal documentation 7 JDS

7 You mean they work together!? Functional testing and tuning is done at the end of each iteration Why not performance testing and tuning? Focus on specific functionality Solve problems earlier! 8 JDS

8 Test Early! Test Often! Work closely with developers Single point of contact for defect resolution Developers are focused on performance as well as functionality Environment flexibility Development environments allow for fast deployments and fixes Less impact if environment is disrupted Build a robust performance test suite, ready for load testing. Shakeout scripts Validate data Give confidence that some performance defects have already been resolved Gives confidence in code if further load-related defects are found Further defects are most likely load-related Buy more hardware now? 9 JDS

9 So How Does It Work? Performance test each developed module as soon as possible 1. New module/story/functionality is developed 2. Script the new module in VuGen 3. Run the test and monitor with Diagnostics 10 JDS

10 So How Does It Work? (continued) 4. Identify inefficiencies 5. Work through inefficiencies with the developer 6. Developer applies a fix/patch 7. Re-test 11 JDS

11 12 JDS

12 Get Dirty Step 1 Diagnostics setup Install Diagnostics Probes on development environment servers Instrument developed code In-house developed code should be easily identified by its namespace e.g. com.hp.exampleclass.dosomething Balance your instrumentation. You do not want to create a massive overhead due to Diagnostics capturing every single method call. This will give you unrealistic results. [MyInstrumentation1] class = org.myapp.class1 method = hellowworld signature =!.* layer = MyLayer 13 JDS

13 Get Dirty (continued) Step 2 Script new module Script all functionality within the newly created module/page etc. Include as large a range of test data as possible Parameterize URLs, validation, login data etc. Anything that will need to change when you move between environments 14 JDS

14 Get Dirty (continued) Step 3 Run your test Run the test script in the LoadRunner controller Run with no pacing or think time to exercise a large range of test data, such as application users Only run a single user, to be sure there is no resource contention Do not run so much load that you stress the environment infrastructure. Ensure the Diagnostics Probe is capturing data Run the test for enough time to gain a good range of statistically significant response time measurements 15 JDS

15 Get Dirty (continued) Step 4 Diagnose Does anything stand out? Look for server requests with high response times Are any SLA s being breached? Look for patterns: Are there any method call structures that are repeated? Is any 1 method called a high number of times? Is any 1 method taking a long time to execute? Are there a large number of outbound calls? Where are these going? Is any SQL being executed? How long is it taking? Make a list of any suspected inefficiencies Save a Diagnostics snapshot of the test period for later reference 16 JDS

16 Get Dirty (continued) Step 5 Brainstorm, build, then do it all again. Sit down with the developer Run through your list of standout items Walkthrough method call tree snapshots Brainstorm possible fixes, alternatives Wait patiently for a fix to be deployed Work through the backlog? Re-script (if needed) Back to Step 3 17 JDS

17 Things to Look for - Patterns Are there any repeated blocks of code that look similar? Why are they called repeatedly? Do they need to be called that many times? Could results from a lookup method be saved in a variable? etc.. 18 JDS

18 Things to Look for Prevalent Methods Is there any 1 method that is used frequently throughout the call structure This may only be small (ms), but even a minor improvement to this method will have a big overall effect 19 JDS

19 Things to Look for Mystery Time Are there any large gaps in the call profile? Is the lowest level method in a call profile really the lowest level? Do you need to instrument more classes or methods? 20 JDS

20 Things to Look for Outbound Calls Do any methods with large transaction times end in a mystery outbound call? Where is this call going? Is it crossing to another VM or server? Is it executing a large SQL statement? Do you need more instrumentation? This is important to establish where a problem lies, and therefore who the owner is! 21 JDS

21 Beware! Traps to Avoid Use a range of data. Some methods may execute quickly with one data value, and take ages with another E.g. Searches Do not stress the infrastructure. You don t want to be telling developers their code is slow, when you re running production load in a scaled down environment. Stick to a single Vuser. It is what it is. Remember that Diagnostics is not reporting an end-user transaction time. This is time on the server only. Get involved! Avoid falling into a my turn your turn, way of working with developers. Get to know each other, and collaborate to find the best solutions. Don t get over confident. Just because you ve resolved some code-level defects, don t expect the test to run smoothly under full load. You ve found the low-hanging fruit, but not all the fruit. Do not over-instrument Diagnostics response times include the overhead caused by Probe instrumentation. Measure times (using LoadRunner) with Diagnostics enabled and disabled. 22 JDS

22 Case Study Largest Telecommunications provider in Australia SharePoint installation with 8000 daily users Multiple releases, constant development cycle Engaged JDS for performance testing on each of their SharePoint releases Agile development cycle Daily stand-ups (scrums) Continuous Integration Each developer owns a user story Multiple iterations (sprints) to a release New functionality is added to live production system 23 JDS

23 Case Study (continued) The problem: Performance testing services engaged shortly before releases were scheduled to go live Constantly struggling to complete performance testing within unrealistic timeframes Little or no time to resolve performance defects Lingering performance issues remaining unresolved throughout consecutive releases Code development was agile, performance testing was waterfall 24 JDS

24 Case Study (continued) The solution: We suggested the above approach. Test early! Test Often! Began working with the development team as part of their Agile development cycle Found multiple severity 1 defects Worked with development teams to tune and resolve defects Prevented performance defects from reaching pre-production (and production) environments + 25 JDS

25 Case Study (continued) The result: $$$ saved = Less risk to missing release deadlines = Less risk of defects reaching production = 26 JDS

26 Follow up Adam Reincke JDS Tech tips: Stuart Moncrieff myloadtest: 27 JDS

27 28 JDS

28 30 JDS