27 th March 2015 Istanbul, Turkey Performance Testing Best Practice
Your Host.. Ian Molyneaux Leads the Intechnica performance team More years in IT than I care to remember Author of The Art of Application Performance Testing TestIstanbul 2015
27 th March 2015 Istanbul, Turkey Why Performance Matters
A knowledge gap
System complexity DATA CENTER INTERNAL USERS INTERNET CUSTOMERS Storage DB Servers Mainframe Middleware Servers Web Servers App Servers Load Balancers Network Major ISP Third- party/ Cloud Services Local ISP Content Delivery Networks Mobile Carriers
So what represents good performance? Unacceptable (10s) > Compe33ve advantage Tolerable (7s) > Average (6s)> Good (3s) > Excellence (Uneconomic) (1s)>
27 th March 2015 Istanbul, Turkey Performance Testing
What is performance testing? Testing to ensure that your application works under real-world loads before you release it to production. Works? - staying available and performant under conditions of peak load. Performant? - providing a good end user experience. The problem is defining what good means. It really comes down to the end users perception of performance.
Good performance definition refined.. A well-performing application is one that lets the end user carry out a given task without undue perceived delay or irritation. Performance really is in the eye of the beholder.
Important.. Performance testing is distinct from any other form of software testing. It is very much its own discipline.
27 th March 2015 Istanbul, Turkey Performance Testing Scope
Performance Testing helps identify EUE CAPACITY EUE NETWORK ISSUES CODE QUALITY CONFIG ISSUES CONTENTION EUE (No) SQL PERFORMANCE 13
Testing response time Local users Remote users Mobile users In reality this is server response time Plus the impact of latency
Testing response time Last-mile Service Providers Cloud Providers Content Delivery Networks Affiliates
Understand the technology Affiliates..
Testing capacity Finding the upper limits of Availability Acceptable response time
End to End The ability to remain available and performant As a complete integrated deployment In the presence of other applications and services
Soak testing Performance and availability over an extended duration No memory leaks No configuration hard-stops No disk space issues
Testing different configurations The What if? Useful for optimizing deployment Load balancing choices Web, App and DB configuration settings
Testing Disaster Recovery Don t forget your DR environment Hopefully you have one! Understand DR performance compared to production Failover testing under load
As part of procurement Try before you buy! Performance evaluation of potential suppliers Particularly relevant in a SaaS world
BAU Production Benchmarking Periodic non-destructive performance evaluation and benchmarking of live deployments Detecting performance and capacity regressions Performance impact of configuration changes Canary approach
27 th March 2015 Istanbul, Turkey Performance Testing Foundations
Built on 6 pillars Business knowledge Technical knowledge People Tooling Environments Process
27 th March 2015 Istanbul, Turkey Business Knowledge
Why are we performance testing? Pre-existing performance issues? Concern about an upcoming peak event? A desire to improve the quality of delivery? Fear of the unknown?
Understand the application What does the application do? Get familiar with the functionality Get familiar with the terminology Was it written in-house or is it a package? Is this a new release or a pre-existing application? Understand the size and distribution of end-user community
Understand the performance targets Typically.. Availability Concurrency Response time Consider.. How have they been derived? Are they realistic? Are they achievable?
Availability The application has to remain available under peak load. It cannot be seen to fail.
Concurrency Two types.. Application Automation Impacted by.. Session persistence Capacity constraints
Response time The end user experience.. From the perspective of the end user Service consumption Service SLA s Impacted by.. Performance by design Code quality Capacity constraints Design decisions Async rules!
Bad design
Use cases Selecting functionality to render as scripts Also called user journeys or transactions The building blocks of (performance) test scenarios Ideally represent high volume, high load activity Unusual to have more than 10
Use cases Several different types.. UI driven Service driven API driven Batch driven Device driven
Use cases Critical to get these correct.. Sub-divide into logical timings e.g. Login, Search, Add to Basket, Logout Organise into iterative and non-iterative steps e.g. Init, Body, End
Use cases
Load model Derived from.. Business intelligence data Application usage profile Web server logs Performance targets Use cases Test data
Load model input
Virtual user distribution
Load injection profile
Together represent performance test scenarios Virtual user distribution Load injection profile
27 th March 2015 Istanbul, Turkey Technical Knowledge
Understand the technology Application architecture Tech-stack Integrations Service consumption, internal and external Hosting infrastructure Servers Network
Determine the technical requirement Tooling options Automation effort and cost Instrumentation effort and cost Monitoring Tech-stack KPI s Hosting infrastructure KPI s
KPI s, a many layered approach.. Application Level Rendering Time TTFB Application Level Cache Performance HTTP Errors Application Level Connection Pool Use Garbage Collection Application Level SQL Query Performance Full Table Scans Business Level Response Time Availability Conversions Service Level Response Time Availability O/S Level CPU Loading Available Memory I/O Performance Free Disk Space Page File Usage APM Level Method Performance Business Transaction O/S Level CPU Loading Available Memory I/O Performance Free Disk Space Page File Usage APM Level Database Calls Service Calls O/S Level CPU Loading Available Memory I/O Performance Free Disk Space Page File Usage
The first deliverable Statement of Work Performance KPI s Tooling options Load Model Test Plan Time-scales Aim to achieve consensus and agreement on expectation
27 th March 2015 Istanbul, Turkey People
People The right people are essential to effective performance testing. Experienced performance testers are a rare commodity to be nurtured.
People They should ideally have.. Background knowledge of application design and deployment. An understanding of application architectures. Understand the importance of accurate requirement capture. The ability to use tooling and implement process.
People Promote a culture of performance Raise awareness of performance in IT Consider performance at project start. Test in Dev, QA and Production Be a performance warrior
27 th March 2015 Istanbul, Turkey (Automation) Tooling
The case for automation You cannot performance test effectively without automation.
Automation enables Scalability Consistency Correlation Repeatability
Tooling components Most automated performance testing tools have the following components.. Workbench for script recording Load injector to generate application traffic Controller to manage and execute tests Monitors to capture non-application KPI s Analysis to triage test results
Tooling components Load Web Server Tier Load Metrics Test Scenarios Load Test Controller App Server Tier Load Load Injectors Database Cluster
Tooling choice.. Performance Test Delivery JMeter Facilita Forecast Neotys Neoload Borland Silk Performer HP LoadRunner dynatrace APMaaS APM dynatrace Appdynamics New Relic Correlsense Sharepath HP Diagnostics Windows O/S and Application Monitoring Windows Performance Monitor Solar Winds Linux / Unix kernel tools
27 th March 2015 Istanbul, Turkey Tech-Stack Challenges
Tech-stack challenges Most modern performance testing requirements involve web Good support for Websockets, Async and similar current tech However you may also come across the following.. Thin-client Citrix, MSTC SAPGUI Oracle ebusiness BMC Remedy IT Service Management
Tech-stack challenges Tech-stack requirements other than web.. Tooling choice Non-web support is typically not OpenSource Licensing considerations other than performance tooling i.e Does the client have enough Citrix licenses? Specialist domain knowledge
27 th March 2015 Istanbul, Turkey Environments
Environments As close as practical to production in terms of.. Horizontal and Vertical scale Service consumption Data volume and quality Software Build, Deploy and Configuration Exclusive access during performance test execution Roll-back strategy
Environments Appropriate Instrumentation Application KPI s Infrastructure KPI s Appropriate monitoring Baked into performance test tooling Leverage existing tooling investments
27 th March 2015 Istanbul, Turkey Process
Process Make sure you follow a consistent approach.. Scoping Design Execution and Analysis Closure
Process Scoping.. NFR Capture Six Pillars approach Delivers a Statement of Work
Process Design.. Based on Statement of Work and Load Model Validate test environment Create scripts from use cases Create performance test scenarios Configure environment monitoring Ensure you are ready to execute
Process Execution and Analysis.. Generally an iterative process A lot of watchful waiting Feedback is critical Highly vulnerable to scope creep
Analysis - Concurrency versus response time
Analysis - Concurrency versus errors
Analysis - Concurrency versus hit rate
Process Closure Creation of engagement report Must report against performance targets and goals Clarity is key Workshop results with the client Follow up with quality review
27 th March 2015 Istanbul, Turkey Take Away
Built on six pillars Business knowledge Technical knowledge People Tooling Environments Process
Call to action Performance testing is a tactical deliverable but should be part of a strategic process Test early, test often, test effectively Be a performance warrior
27 th March 2015 Istanbul, Turkey Thank you Q & A