27 th March 2015 Istanbul, Turkey. Performance Testing Best Practice

Transcription

1 27 th March 2015 Istanbul, Turkey Performance Testing Best Practice

2 Your Host.. Ian Molyneaux Leads the Intechnica performance team More years in IT than I care to remember Author of The Art of Application Performance Testing TestIstanbul 2015

3 27 th March 2015 Istanbul, Turkey Why Performance Matters

4 A knowledge gap

5 System complexity DATA CENTER INTERNAL USERS INTERNET CUSTOMERS Storage DB Servers Mainframe Middleware Servers Web Servers App Servers Load Balancers Network Major ISP Third- party/ Cloud Services Local ISP Content Delivery Networks Mobile Carriers

6

7 So what represents good performance? Unacceptable (10s) > Compe33ve advantage Tolerable (7s) > Average (6s)> Good (3s) > Excellence (Uneconomic) (1s)>

8 27 th March 2015 Istanbul, Turkey Performance Testing

9 What is performance testing? Testing to ensure that your application works under real-world loads before you release it to production. Works? - staying available and performant under conditions of peak load. Performant? - providing a good end user experience. The problem is defining what good means. It really comes down to the end users perception of performance.

10 Good performance definition refined.. A well-performing application is one that lets the end user carry out a given task without undue perceived delay or irritation. Performance really is in the eye of the beholder.

11 Important.. Performance testing is distinct from any other form of software testing. It is very much its own discipline.

12 27 th March 2015 Istanbul, Turkey Performance Testing Scope

13 Performance Testing helps identify EUE CAPACITY EUE NETWORK ISSUES CODE QUALITY CONFIG ISSUES CONTENTION EUE (No) SQL PERFORMANCE 13

14 Testing response time Local users Remote users Mobile users In reality this is server response time Plus the impact of latency

15 Testing response time Last-mile Service Providers Cloud Providers Content Delivery Networks Affiliates

16 Understand the technology Affiliates..

17 Testing capacity Finding the upper limits of Availability Acceptable response time

18 End to End The ability to remain available and performant As a complete integrated deployment In the presence of other applications and services

19 Soak testing Performance and availability over an extended duration No memory leaks No configuration hard-stops No disk space issues

20 Testing different configurations The What if? Useful for optimizing deployment Load balancing choices Web, App and DB configuration settings

21 Testing Disaster Recovery Don t forget your DR environment Hopefully you have one! Understand DR performance compared to production Failover testing under load

22 As part of procurement Try before you buy! Performance evaluation of potential suppliers Particularly relevant in a SaaS world

23 BAU Production Benchmarking Periodic non-destructive performance evaluation and benchmarking of live deployments Detecting performance and capacity regressions Performance impact of configuration changes Canary approach

24 27 th March 2015 Istanbul, Turkey Performance Testing Foundations

25 Built on 6 pillars Business knowledge Technical knowledge People Tooling Environments Process

26 27 th March 2015 Istanbul, Turkey Business Knowledge

27 Why are we performance testing? Pre-existing performance issues? Concern about an upcoming peak event? A desire to improve the quality of delivery? Fear of the unknown?

28 Understand the application What does the application do? Get familiar with the functionality Get familiar with the terminology Was it written in-house or is it a package? Is this a new release or a pre-existing application? Understand the size and distribution of end-user community

29 Understand the performance targets Typically.. Availability Concurrency Response time Consider.. How have they been derived? Are they realistic? Are they achievable?

30 Availability The application has to remain available under peak load. It cannot be seen to fail.

31 Concurrency Two types.. Application Automation Impacted by.. Session persistence Capacity constraints

32 Response time The end user experience.. From the perspective of the end user Service consumption Service SLA s Impacted by.. Performance by design Code quality Capacity constraints Design decisions Async rules!

33 Bad design

34 Use cases Selecting functionality to render as scripts Also called user journeys or transactions The building blocks of (performance) test scenarios Ideally represent high volume, high load activity Unusual to have more than 10

35 Use cases Several different types.. UI driven Service driven API driven Batch driven Device driven

36 Use cases Critical to get these correct.. Sub-divide into logical timings e.g. Login, Search, Add to Basket, Logout Organise into iterative and non-iterative steps e.g. Init, Body, End

37 Use cases

38 Load model Derived from.. Business intelligence data Application usage profile Web server logs Performance targets Use cases Test data

39 Load model input

40 Virtual user distribution

41 Load injection profile

42 Together represent performance test scenarios Virtual user distribution Load injection profile

43 27 th March 2015 Istanbul, Turkey Technical Knowledge

44 Understand the technology Application architecture Tech-stack Integrations Service consumption, internal and external Hosting infrastructure Servers Network

45 Determine the technical requirement Tooling options Automation effort and cost Instrumentation effort and cost Monitoring Tech-stack KPI s Hosting infrastructure KPI s

46 KPI s, a many layered approach.. Application Level Rendering Time TTFB Application Level Cache Performance HTTP Errors Application Level Connection Pool Use Garbage Collection Application Level SQL Query Performance Full Table Scans Business Level Response Time Availability Conversions Service Level Response Time Availability O/S Level CPU Loading Available Memory I/O Performance Free Disk Space Page File Usage APM Level Method Performance Business Transaction O/S Level CPU Loading Available Memory I/O Performance Free Disk Space Page File Usage APM Level Database Calls Service Calls O/S Level CPU Loading Available Memory I/O Performance Free Disk Space Page File Usage

47 The first deliverable Statement of Work Performance KPI s Tooling options Load Model Test Plan Time-scales Aim to achieve consensus and agreement on expectation

48 27 th March 2015 Istanbul, Turkey People

49 People The right people are essential to effective performance testing. Experienced performance testers are a rare commodity to be nurtured.

50 People They should ideally have.. Background knowledge of application design and deployment. An understanding of application architectures. Understand the importance of accurate requirement capture. The ability to use tooling and implement process.

51 People Promote a culture of performance Raise awareness of performance in IT Consider performance at project start. Test in Dev, QA and Production Be a performance warrior

52 27 th March 2015 Istanbul, Turkey (Automation) Tooling

53 The case for automation You cannot performance test effectively without automation.

54 Automation enables Scalability Consistency Correlation Repeatability

55 Tooling components Most automated performance testing tools have the following components.. Workbench for script recording Load injector to generate application traffic Controller to manage and execute tests Monitors to capture non-application KPI s Analysis to triage test results

56 Tooling components Load Web Server Tier Load Metrics Test Scenarios Load Test Controller App Server Tier Load Load Injectors Database Cluster

57 Tooling choice.. Performance Test Delivery JMeter Facilita Forecast Neotys Neoload Borland Silk Performer HP LoadRunner dynatrace APMaaS APM dynatrace Appdynamics New Relic Correlsense Sharepath HP Diagnostics Windows O/S and Application Monitoring Windows Performance Monitor Solar Winds Linux / Unix kernel tools

58 27 th March 2015 Istanbul, Turkey Tech-Stack Challenges

59 Tech-stack challenges Most modern performance testing requirements involve web Good support for Websockets, Async and similar current tech However you may also come across the following.. Thin-client Citrix, MSTC SAPGUI Oracle ebusiness BMC Remedy IT Service Management

60 Tech-stack challenges Tech-stack requirements other than web.. Tooling choice Non-web support is typically not OpenSource Licensing considerations other than performance tooling i.e Does the client have enough Citrix licenses? Specialist domain knowledge

61 27 th March 2015 Istanbul, Turkey Environments

62 Environments As close as practical to production in terms of.. Horizontal and Vertical scale Service consumption Data volume and quality Software Build, Deploy and Configuration Exclusive access during performance test execution Roll-back strategy

63 Environments Appropriate Instrumentation Application KPI s Infrastructure KPI s Appropriate monitoring Baked into performance test tooling Leverage existing tooling investments

64 27 th March 2015 Istanbul, Turkey Process

65 Process Make sure you follow a consistent approach.. Scoping Design Execution and Analysis Closure

66 Process Scoping.. NFR Capture Six Pillars approach Delivers a Statement of Work

67 Process Design.. Based on Statement of Work and Load Model Validate test environment Create scripts from use cases Create performance test scenarios Configure environment monitoring Ensure you are ready to execute

68 Process Execution and Analysis.. Generally an iterative process A lot of watchful waiting Feedback is critical Highly vulnerable to scope creep

69 Analysis - Concurrency versus response time

70 Analysis - Concurrency versus errors

71 Analysis - Concurrency versus hit rate

72 Process Closure Creation of engagement report Must report against performance targets and goals Clarity is key Workshop results with the client Follow up with quality review

73 27 th March 2015 Istanbul, Turkey Take Away

74 Built on six pillars Business knowledge Technical knowledge People Tooling Environments Process

75 Call to action Performance testing is a tactical deliverable but should be part of a strategic process Test early, test often, test effectively Be a performance warrior

76 27 th March 2015 Istanbul, Turkey Thank you Q & A