Combining Performance Testing and Modelling for easyjet.com Danny Quilton, COO, Capacitas Agenda Business Context easyjet s Goals Challenges with Agile How can we Reduce Risk? Performance Testing Performance Modelling What you need Costs Benefits Capacitas 2002-2009 2
About Capacitas UK-based Capacity and Performance Management specialists Training and professional services Hardware-vendor independent Tools independent Capacitas 2002-2009 3 Business Context Capacitas 2002-2009 4
Business Context Capacity planning service www.easyjet.com Business critical service 98% of revenue earned through the e-commerce service Highly competitive market Frequent innovation by adopting agile development Monthly software releases Legislative changes Consistent growth; typically 20% p.a. Low-cost operating model Requirement to manage risk Capacitas 2002-2009 5 easyjet s Goals Capacitas 2002-2009 6
easyjet s Goals Capacitas 2002-2009 7 Agile Development Process Product Backlog Scrum (Daily Meeting) OUT Sprint Planning Meeting IN Sprint Backlog Sprint (15-30 days) Sprint Review Pre-release Activities (e.g. Training, Documentation) LESSONS LEARNED Sprint Retrospective Meeting Release Capacitas 2002-2009 8
Challenges Associated with Agile Frequent change introduces risk Window available for performance testing is typically 1-2 weeks Final production candidate code often not available at the start of testing Multiple code releases within the testing window Increased probability of functional and environmental defects Frequent rework of code necessitates modifications to test scripts Cumulative effect on capacity from 12 releases per annum Narrow time window to resolve performance defects Capacitas 2002-2009 9 Challenges Associated with Agile Capacitas 2002-2009 10
Typical Causes of Degraded Service Performance Capacitas 2002-2009 11 Cause of degraded performance How Can We Reduce Risk? Mitigated by performance testing? Changes in user patterns No Yes Poor performance of 3 rd parties Physical capacity bottlenecks May be limited by environment May be limited by size of the environment Logical bottlenecks Yes If known Mitigated by modelling? Software configuration changes Yes Yes, if tested Yes Yes Increases in demand May be limited by size of the environment Yes Capacitas 2002-2009 12
Why Test? Performance testing is used to understand the software s performance characteristics Do unknown time and growth related defects exist? Memory leaks Poor database design/indexing Do unknown concurrency related defects exist? Database locking issues Memory locking issues Race conditions Obtain model inputs Capacitas 2002-2009 13 Why Model? Modelling uses our understanding of the software to predict end to end service performance under different scenarios Capacitas 2002-2009 14
Scenario Modelling Capacitas 2002-2009 15 Coverage of Testing and Modelling Testing Modelling Application Infrastructure Business Scenarios Technical Scenarios Capacitas 2002-2009 16
High Level Process Capacitas 2002-2009 17 Risk Assessment Capacitas 2002-2009 18
Risk Assessment Assess the potential performance impact on different elements of infrastructure Plan tests to help mitigate that impact Focus on areas of highest risk due to limited timeframes Description ID Web Risks DB Other Risk Mitigation Project X --- Medium Medium Low Test 1 Project Y --- Low High Medium Tests 2 & 3 Project Z --- Low Low Low None Capacitas 2002-2009 19 Performance Tests Employed Capacitas 2002-2009 20
Baseline Tests Used to determine the impact of the planned release Comparative performance between releases May be Load, Stress or Soak Tests Service times and response times are compared to each other Example service time comparison between two fictional baseline tests Service Time (ms) 1400 1200 1000 800 600 400 200 0 Database Server Service Time Web Server Service Time Test 1.1 Release X 752 1240 Test 1.1 Release Y 813 1192 % Difference 8% -4% Capacitas 2002-2009 21 Single Transaction Type Tests To determine the impact on the infrastructure of individual user transactions Every business process which is to be modelled will have a calculated service time for each hardware element Milliseconds Web CPU per page (fictional data) 328 132 233 221 156 122 Home Search Select Options Login Purchase Capacitas 2002-2009 22
Modelling Performance Test Results Capacitas 2002-2009 23 Performance Modelling Business Service Component Virtual Infrastructure Capacitas 2002-2009 24
Performance Modelling Aircraft Business Home Page Search Booking Service HTTP1 APP1 APP2 DB1 Component Virtual Infrastructure Capacitas 2002-2009 25 Performance Modelling 600 Business 100 80 5 Service 160 70 10 80 Component Virtual Infrastructure 3.5 transactions/s Capacitas 2002-2009 26
Model Scope What infrastructure to include? Major contributor to the performance experienced by end-user Highly utilised Critical Load growth Frequently reviewed Virtual Infrastructure Capacitas 2002-2009 27 Model Scope What transactions to include Business critical Importance to revenue High-volume High component cost Home Page Search Booking Change Password Service Capacitas 2002-2009 28
Demand Model Validation The model should be validated on an ongoing basis against actual values The first stage is to validate the demand part of the model to ensure that arrival rates are being calculated correctly Following this the true demand can be used to validate the utilisation calculations 600 100 80 5 610 80 80 6 Capacitas 2002-2009 29 Capacity and Performance Model Validation 100% Comparison of Actual Database Load to Predicted Database Load 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Actual Database CPU Predicted Database CPU Capacitas 2002-2009 30
What you need People Capacity and performance management skills: Demand forecasting Modelling Load testing A strong understanding of risk management A good understanding of the business model Flexibility Project management skills Capacitas 2002-2009 31 What you need Process Interface with the release team Participation in the agile development scrums Access to developers to understand application functionality Interface with the test environment management team for test planning Interface to the business to access business forecasts Interface to operations to access production measurements Capacitas 2002-2009 32
What you need Tools & Environment An appropriately sized test environment Production-like environment is not required Consider the ratio of capacity between presentation, business and data tiers Toolset Performance monitoring tools (production and test) Load generation tool Modelling tool Capacitas 2002-2009 33 What you need Environment Capacitas 2002-2009 34
What you need Data Software/hardware configuration (production and test) Performance measurements from production and test environments Benchmark data (Spec, TPC, etc.) Service times derived from performance test results SLAs Internal SLAs with third-parties Historical and forecast business demand Historical and forecast service demand Measured error of previous forecasts Capacitas 2002-2009 35 Historical Business Demand 180 160 Fleet Plan to April 2009 y = 0.0461x - 1673.1 R² = 0.9668 Number of Owned Aircraft 140 120 100 80 60 GB Airways acquisition 40 20 0 Mar-04 Jun-04 Sep-04 Dec-04 Mar-05 Jun-05 Sep-05 Dec-05 Mar-06 Jun-06 Sep-06 Dec-06 Delivery Date Mar-07 Jun-07 Sep-07 Dec-07 Mar-08 Jun-08 Sep-08 Dec-08 Mar-09 Capacitas 2002-2009 36
Historical Service Demand Capacitas 2002-2009 37 Costs Capacitas 2002-2009 38
Benefits Capacitas give us the confidence in making the correct release decision James Mackay, Test Manager, easyjet Test Phase Risk Assessment Performance Test Plan Targeted Performance and Stress Testing Regression Performance and Stress Testing All Defects: Closed Capacity Models Updated Status Complete Complete Complete Complete Complete Complete Capacity Plan updated and communicated Complete Capacitas 2002-2009 39 Benefits Testing and modelling complement one another The combined approach provides effective risk-mitigation in the context of: Frequent change Narrow performance test window Business growth The approach enables us to rapidly answer business and technical what if scenarios Costs are limited through the use of a scaled-down performance test environment and low-cost modelling tools Provides performance assurance in a highly competitive market place Capacitas 2002-2009 40
Questions? Presentation will be available for download at www.capacitas.co.uk Please visit us at our stand for any further questions Capacitas 2002-2009 41