Successful Factors for Performance Testing Projects NaveenKumar Namachivayam - Founder - QAInsights
Contents Introduction... 2 Planning... 3 Staffing... 5 Test Environments... 7 Support... 8 Intuitive Result Analysis... 9 Performance Testing Checklist... 10 Conclusion... 11 Author: NaveenKumar Namachivayam contact@qainsights.com 1
Introduction Performance matters for any products, solutions and services. If search results page takes more than 5 seconds to load, users will not use that search engine anymore. If shopping cart is taking more time to check out, users will avoid shopping in that portal. If tracking app in your mobile takes more time to locate, users will delete that app. Performance of the applications defines the organizations success and grasps clients. Following are the important types of testing which defines the quality of product: Functional, Performance and Security. Functional Testing validates the intent of the application under test (AUT) Performance Testing validates the non-functional aspects of the application Security Testing validates the loop-holes to prevent the cyber-attacks in the AUT Performance Testing is a super-set term; there are different types of performance testing such as Load, Stress, Volume, Spike, Endurance, and Availability. There are three main objectives in performance testing: Speed, Stability and Scalability In this article, we are going to discuss about the key successful factors for performance testing projects. Below mentioned points are gained by working with different clients and various types of protocols and applications. Author: NaveenKumar Namachivayam contact@qainsights.com 2
Planning Performance Test Planning is an inception and most critical phase in the project. Planning should answer following questions: What is the intent of performance testing? What tool will be used? How to generate test data? How many types of performance testing are required? What is the load to inject? What are the parameters to monitor? How to design workload model? How to identify the critical scenarios? How to analyze the test results and provide recommendations? If anyone is missing from the above questions, then it will be tough to address in later phases i.e. design, test execution and reporting. Below is the case study example for your better understanding. Case Study MyTechCube.com is an online portal where users can buy and sell mobile gadgets. Daily visitors are ranging from 10000 to 20000 during weekdays and 20000 to 30000 during weekends. Users spend most of the time in logging in, browsing mobile catalogs, searching, and viewing galleries. Client wants to validate the consistency in response time and identify the breaking point of the server. Client doesn t have a budget to afford a commercial tool. Now we will see how to address above questions for the effective planning: What is the intent of performance testing? o AUT is a growing online portal. The main objective of performance testing is to validate the speed and stability. What tool will be used? o Since client do not have a budget to afford a commercial tool. It is better to go for an open source tool e.g. Apache JMeter. How to generate test data? o For Login scenario, it is easy to generate the test data. There are many test data generators available online. For catalogs and search scenarios, keywords can be extracted easily. How to identify the critical scenarios? Author: NaveenKumar Namachivayam contact@qainsights.com 3
o In this case, client has provided the critical scenarios. If it is not available, from the server logs it is possible to extract the required data. How many types of performance testing are required? o To validate the response time, Load testing is required. o To identify the breaking point, Stress testing is required. What is the load to inject? o We will take maximum number of visitors, in this case 30000. So, 30000 / 7 (week days) = 4286 o 4286 visitors per day (24hrs) o 179 visitors per hour across the globe o For load testing, the load will be 180-200 o For stress testing, the load will be >180 What are the parameters to monitor? o Response Time, Throughput How to design workload model? o For load testing, steady state will be at least 1 hr. o For stress testing, the pattern will be like stair case. How to analyze the test results and provide recommendations? o Following graphs will be useful for the result analysis: Number of Virtual Users (VUsers) Vs Response Time Number of Virtual Users (VUsers) Vs Throughput Number of Virtual Users (VUsers) Vs Response Time Vs Time Number of Virtual Users (VUsers) Vs Throughput Vs Time Author: NaveenKumar Namachivayam contact@qainsights.com 4
Staffing Allocating skilled resources plays an important role in any projects. Performance testing projects requires more attention while staffing. Testers should have variety of skills when compared to functional testers. Testers should know at least one commercial tool and one open source tool. Well, it is not mandatory to know more than two tools; it is an added advantage to the management. Every organization must have tools lab setup. Relevant tools must be installed and it should be accessible to people who are interested to learn. Tools and training materials must be updated frequently. This will ensure that people are updating their knowledge on tools and its concept. Management should check the following factors when allocating testers to performance testing project: Experience, Tools, Skills, Protocols Knowledge, and Languages known. Experience: Testers should have at least 1 year of full-fledge performance testing is required for performance testing projects Tools: Testers should know at least one tool experience. All the performance testing tools concepts are same but the way to implement is different. It will be added-advantage if the testers know one commercial and one open source tool along with performance monitoring experience. Skills: Testers should have analytical skills, with statistics and mathematics interest. Performance engineering and performance monitoring will be added advantage. Testers should know about web servers, app servers, n-tier architectures, db servers, configuration details, designing workload models, monitoring etc. Protocols Knowledge: Most of the testers will have HTTP(S)/HTML protocol. Train them in Oracle, Siebel, SAP, AJAX, Flex, DCOM and other popular protocols. Languages: C and JavaScript are must languages to known for any performance testers. Testers who are interested in doing hardcore performance testing will learn the languages themselves. Communication: Performance Testing requires involvement from many people which includes Technical Architects, Managers, Developers, Team Leads, and Author: NaveenKumar Namachivayam contact@qainsights.com 5
other stakeholders. So, it is required to have good verbal, oral, and written communication skills. It is very much important to have article writing skills for the sign-off documents and test reports. Identify the right candidates who have passion in performance testing. I have seen many projects with testers who are new to performance testing and doing wrong things always. Often Management fail to train testers in performance testing tools and suffer once the project is in red status. I have seen many testers who do not know about Correlation but performs load testing and publish the results. But when the code moves into production, clients started facing issues in the response time. So it is much important to conduct an audit which will help to identify potential candidates to train and leverage testers in various tools and technologies. Once in a quarter, management should conduct a Webinar from the tool vendor which helps testers to keep updating their skills. Author: NaveenKumar Namachivayam contact@qainsights.com 6
Test Environments Preventive measures should be taken, because performance testing should not carry out in production environment. Production like preproduction environment should be in place for testing. If production environment has four boxes, preproduction should have four boxes. If client does not have budget to own servers, it can be rented or performance testing can be carried in off-business hours. Performance Testing Tools license should be properly deployed and maintained. This will avoid conflicts in using the protocol bundles. Apart from tools, monitoring agents should be deployed for performance monitoring. Performance monitoring tools like dynatrace should be installed on the server where AUT is present. Author: NaveenKumar Namachivayam contact@qainsights.com 7
Support Support should be there from DBAs, Architects, Test Data team, Environment Team, Developers, Leads and Managers to testers. If testers face any issues during scripting or execution, other teams should help testers to fix the issues. E.g. Developers should help testers about correlation ids, Environment team should support testers by shutting down or starting up relevant services. Author: NaveenKumar Namachivayam contact@qainsights.com 8
Intuitive Result Analysis Testers must have knowledge on how-to analyze the test results. Relevant graphs/charts should be generated for better presentation. By drilling down to the data points in the graphs might reveal the culprits of the performance. important to furnish your detailed analysis. Results must be compared with previous execution, and with benchmarks. Both client and server side parameters should be monitored and published. Most of the clients expect the root cause of the failures. It is very By having performance monitoring solution, it is simple and effective to drill down to the cause. CPU, Memory, Disk, Network, Host health details are the important metrics for the result analysis. Author: NaveenKumar Namachivayam contact@qainsights.com 9
Performance Testing Checklist Planning Objectives defined Signed off test plan Staffing Experienced candidates Technology knowledge Application knowledge Tools Communication skills Test Environment Production-like Servers Tools setup Monitoring setup Intuitive Result Analysis Parameters to be collected Graphs and Charts Correlation of graphs/charts Results Comparison Benchmarks Recommendations Author: NaveenKumar Namachivayam contact@qainsights.com 10
Conclusion By implementing proper planning, selecting right candidates for the exercise, having production-like pre-production infrastructure, support from other teams, and identifying bottlenecks thru intuitive results analysis leads to successful completion of performance testing projects. Images Credit: FreeDigitalPhotos.net Author: NaveenKumar Namachivayam contact@qainsights.com 11