Aligning Correct and Realistic Performance Testing with the Agile Development Process

Transcription

1 Aligning Correct and Realistic Performance Testing with the Agile Development Process SIGIST Winter 2011 Conference Graham Parsons CEO, Reflective Solutions

2 Overview Introduction A major risk for Agile projects Why is non-functional testing ignored? Case Study: Capgemini Project Results: USA leading online company Sorry to disappoint; this is not a product sales pitch!

3 Introduction CEO and co-founder of Reflective Solutions (1998) Reflective Solutions assists organisations of all sizes to guarantee performance of key systems Credited with changing the face of non-functional testing by considerably simplifying the process Clients include some of the largest online presences in Europe and USA Focused on testing in Agile projects since 2009.

4 Business Benefits of the Agile Process Focus on quality Adaptable requirement changes are welcomed Reduces risk regular testing, regular business involvement Delivers business value business decides what will be built, changed, etc. Accelerates delivery the time to realise the business value.

5 The Agile Project Iteration/Sprint 1 Iteration 2 Iteration 3 Iteration N 2 4 weeks

6 A Single Iteration / Sprint

7 Release After Every Sprint Iteration 1 Iteration 2 Iteration 3 Iteration N Well that s the theory!

8 What About Non-Functional Testing? Nearly always left to the end So project can t really release at the end of each iteration Problems detected at end can cause weeks (or months) of rework If problems occur, negates most of the benefits of Agile No matter what process is used to deliver a system, performance problems kill positive user perception!

9 The Agile Project Reality Iteration 1 Iteration 2 Iteration 3 Iteration N

10 And if Problems are Discovered Iteration 1 Iteration 2 Iteration 3 Iteration N Fix exposes another performance problem. Extra Iteration

11 Barriers to Performance Testing in Agile Agile is: Fast Flexible Team members embrace different roles Traditionally, performance testing is: Slow (many weeks) Needs a code freeze Requires an environment for the testing Dedicated test tool experts It s all because of the complexity of performance testing tools!

12 New Breed of Tool Available Quick to use Can fit into Agile projects Quick enough to allow testing in every iteration / sprint Easy to learn No, or little, scripting Lots of in-tool guidance for occasional users The outcome: Many team members can configure performance tests Resource flexibility for the project manager.

13 What About the Test Environment? This is application performance testing Not testing of infrastructure Can use existing test environment Most application performance defects visible at relatively low load New tools require less hardware for load Easily simulate 100s of users on developer desktop.

14 How to Select the Correct Tool Check it works with your application Can it be afforded? How easy is it really? If you don t correctly simulate users, testing benefit is reduced Trial it to understand effort required Or ask the vendor to prove it! Get references from other Agile practitioners Testing in Agile is hard enough without fighting inadequate tools!

15 CASE STUDY: CAPGEMINI

16 Capgemini Recognised as Agile experts Many blue-chip clients Accelerated Delivery Centres Across UK, Europe and USA Benchmarked against worldwide projects (*) Typically 20% to 30% reduction in effort (vs. average) Typically 15% to 20% fewer defects At any one time: multiple projects, multiple clients Some Agile projects use Scrum Team members move between teams (*)

17 Foresaw the Potential Problem Functional testing every iteration / sprint Could release with minor functional errors However, projects often long-term Hidden risk Goal: performance testing in every iteration Always a concern about application performance No problems to date, but only a matter of time Realised they had to address the problem They wanted to test every iteration but without delaying the Agile project.

18 What They Had Tried Industry leading tools LoadRunner, etc. Many clients already had licenses the easy option Free / open source tools JMeter Grinder Etc.

19 The Problems They Found All the tools had the same problems: Complex script languages Considerable time required to become truly proficient Therefore skills only possessed by (too) few people Impossible to have many people trained Would have required too much effort So unavailability of resources would delay projects Too slow Even when used by tool experts!

20 Investigated New Tools New breed of performance testing tools which Claim to be quick and easy to learn Claim to be simple to configure Claim to be significantly quicker to use Up to 80% reduction in required effort Of the tools they evaluated Some did not live up to the vendors claims But some did!

21 Tool Adoption Approach Trained initial group of people Training took 2 days and was scheduled around projects These people, using online training backup, trained other colleagues many people now proficient Not only testers, developers too! Trial adoption into a critical Agile project Goal to performance test every sprint Project team would prioritise performance defects Project manager has flexibility over who does the performance testing (as many people knew the tool).

22 The New Agile Testing Process Iteration 1 Iteration 2 Iteration 3 Iteration N

23 Infrastructure Test is Still Required Iteration 1 Iteration 2 Iteration 3 Iteration N Application performance testing in every iteration Infrastructure test prior to release Any infrastructure performance problems require no code changes, typically are quick to fix, and hence little chance of major delays in release to user community.

24 USA ONLINE COMPANY: AGILE PERFORMANCE TESTING RESULTS

25 Iteration Comparison (*) StressTester Iteration Comparison Chart

26 Problem Identification (*) StressTester Problem Identification Results

27 Fixed in Next Iteration

28 StressTester has lowered the barrier to entry for performance testing for our sprint teams. It has also enabled us to put a performance testing capability in each Agile team, giving us performance testing capability within the sprint. Scott Davies, Engineering Director Capgemini UK

29 Conclusion From a functional quality perspective, Agile delivers Trouble is, non-functional testing is left until the end If problems occur, costly and significant delays Excuse for lack of performance testing: complex and slow tools New breed of tool means testing in every sprint is now possible Easy to learn lowers barrier to adoption Quick to use test in every iteration.

30 ANY QUESTIONS? COMMENTS? CHALLENGES? FREE WHITEPAPER: /agile