SnEApproach for Reducing Cycle Time &Improving Quality of Performance Tests

Transcription

1 SnEApproach for Reducing Cycle Time &Improving Quality of Performance Tests QAI STC 2013 Parag Chandekar Performance Test Lead Eaton Technologies EON Free Zone, Cluster B, Wing 3,Plot no. 1, survey no. 77 MIDC, Kharadi Knowledge Park, Pune, Maharashtra411014, India

2 Introduction Software testing life cycle (STLC) consists of unit testing, system, integration, regression, user acceptance and Performance testing (PT). PT is often considered as the end of this cycle. There are occasions when the timeframe allocated for PTis way too less to accomplish any meaningful testing. This could be because of it being completely neglected during the planning phase, and being added much later, or due to planning flaws, or due to delays in phases prior to PT( like phases of software development, functional testing etc.)which pushes PT even further. This short duration is ofteninsufficient for thorough performance testing.many a time, adhoc approaches are followed to cover up test scenarios, and often there are possibilities of performance issues leaking into production. The short duration is insufficient for the execution of the current PT approach and it certainlyleaves no room for retesting of performance testing bugs or retesting post tuning changes to address and fix issues reported out of PT. If PT issues are reported, these too are required to be fixed at the same timeframe, putting excessive pressure on the resources to deliver it. The current performance testing life cycle (PTLC) as described in next sectionmight not fit in appropriately into such situations and requires modification in order to address these issues. This paper discusses a structured approach that can be followed while conducting PT in such short/stringent timelines. It additionally provides insights onpossibilities of reducing the duration in various phases, thus optimizingthe overall performance testing cycle time helping in early detection and fixing of issues.

3 Current PT Process The current PT approach which is widely followed in the industrymainly comprises of 6 steps. Requirement gathering Planning/scenario design Scripting Execution Analysis and reporting Tuning ( Application/Database/Operating system ) Figure1 : Current Performance Testing process Effort Distribution across Phases Effort distribution data was collected from previous performance testing projects within Eaton and even from the prior work experience with other companies. Percentage of effort spent across various PT phases and the below graph was plotted.

4 Figure 2: Effort/Time for various PT phases Looking at the percentage time spent across the various phases, one can see that requirement gathering and planning take around 26% of the overall efforts. Additionally, there is not much benefit in execution cycle time by optimizing the requirement gathering and test design preparation; as these can be done earlier in the project parallely with development and functional testing. Then comes the test scripting phase, where the in- scope call flows are required to be scripted. The scripting phase consumes close to 20-30% of the entire efforts of PT. If we do not consider test planning/requirement gathering time, the scripting and execution duration take 75% of the overall project time. These two phases do provide an opportunity to reduce the cycle time. Shortcomings of Current process Given shorter timelines for performance testing, the current approach would fail, as one might end up spending all the time in planning and scripting rather than actually executing tests. Adding to the woes, scripting phase also suffers from delays related to waiting for call flow details of scripts, until which, scripting cannot commence. Execution too sometimes gets delayed due to unavailability of volumetric confirmation and test data unavailability for some or all the scripts. Moreover, in the current process (in most of the cases) execution phase is initiated only once all the scripts are prepared. As the scripting phase itself is considerably long, it may be quite some time before the first test is run. And as we know, it s only the execution phase where one can find out defects related to performance and capacity.this delay in execution could have some really serious effects on the project.

5 One other pitfall of the current process is that the performance defects detected during the execution phase need to be fixed and retested. Sometimes these retesting efforts are not estimated and clearly it pushes the timelines further. Additionally, it is also important to allocate sufficient time for analysis. A huge amount of data can be returned when executing nonfunctional tests and analyzing it takes considerable time. As the analysis process is technical,it should not be rushed as it wouldincrease the risk of misdiagnosis and the resulting recommendations may be misleading. However, in the current performance testing process this activity is often compromised due to lack of time Opportunities for improvement However, there are a lot of places where the current process can be optimized.in the current process, issues get highlighted only during the execution phase. For example, issues related to load balancer (LB) configurations directed towards one server instead of being equally distributed across all the servers gets highlighted during execution. From static testing, which is done earlier in the PT lifecycle, we can find a lot of bugs in the system architecture affecting the performance; however, we can only assess the system performance once we start testing. Additionally, testers get a feel of the system only when they would have executed a few tests themselves. Moreover,in projects where performance testing is conducted for the first time, a lot of tests get added after seeing the results of a few tests and the tester gets a better understanding of the application while testing it. This additional effort puts an additional burden on the already loaded team to complete it in the stipulated time. PROPOSED SOLUTION An approach is being proposed whereinexecution starts as soon as the first script is ready.the philosophy, Script n Execute" would help in identifying issues on the system earlier before the actual planned execution cycle, where we run all the scripts together, begins.some restructuring of tests are suggested in the SnE approach which will help in finding defects quicker. The approach categorizestests into 3 different types. Type-A: Tests can be conducted with even a single working script; this will help in finding out issues much earlier. For instance, a single script execution finds out issues in login event, these issues will invariably be encountered by other scripts too. Early defect fixing will help in reducing the overall test cycle.in Type A tests formal reports might not be generated however statistics can be compiled in a sheet. Time spent in execution of single scripts in the SnE approach is limited to 20% of the time spent in making the script. Example of tests that fall under Type-A: Test Environment Configuration and limits tests LB configuration checks, test environment related issues get highlighted in the first day or Script &Execute phase Loadrunner server utilization checks. The benchmarking for load generation capacity of loadrunner servers can be done with a single script. Application Specific Tests Application configuration checks (e.g., threshold checks) Application capacity Breaking point of that call flow

6 Checking effects of caching and whether the response times rise or fall Tests to identify if a particular layer is not a bottleneck Benchmark response times for that call flow Test with gradual rampup for a very long time Concurrency tests(tests by placing rendezvous points on all the transaction checkpoints) End User Tests Tests with user load emulation with variousbandwidths. Memory Leaks Running long duration tests overnight will help in finding out potential issues, like memory leaks, disk space issues etc. Type-B: Tests are run with realistic volumes and all the scripts are required here. This emulates real traffic as one expects in production. These tests are designed in load test tool based on the inputs from Test type A, o Mixed tests having volumes of all the scripts/scenarios sent simultaneously o Tests with accurate, volumetric input from the test type-a on response times per call flow can helpin designing realistic load E.g.,if we run a breakpoint test with our first script, from the graphs we would know the concurrent user load at which the throughput begins to drop.type B test for finding out peak performance point can take this value and ramp it upby 20% more load. Example of tests that fall under Type-B: Mixed tests having volumes of all the scripts/scenarios sent simultaneously Mixed Normal load test Mixed Peak load Test Mixed Stress Test Mixed Soak Test Tests which require inputs from Type-A tests Type-C: Test which not planned and need to be done on the basis of inputs from Type-B Tests. Example of tests that fall under Type-C: Dynamic tests designed on basis of inputs from the system Tests which might require inputs from Type-B tests

7 Figure 3: Script n Execute Process Diagram Finalizing the order of script creation As we all know,risk Based Testing (RBT) helps in prioritizing the E2E call flows which will need performance testing. However,SnE will help in prioritization of order in which the scripts will be prepared to get the maximum out of this approach. Scripting and testing a call flow which has the most volume of transactions and concurrency makes more sense than one with less concurrency and volumes. Below is a simple flow chart that would help in selecting which call flows to script first. At run time during scripting phase, decision might need to be taken to script the call flow which has the highest priority as well as, which test data is available for test execution.

8 Figure 4: Script prioritization flowchart Benefits Below are various benefits of this approach: Defects are detected earlier: As can be seen from the diagram below, that defects can be found out earlier in the SnE approach as compared to the current process.with faster performance defect detections, they get raised earlier, which allows enough time for fixing and retesting them. Figure 5: Current and SnE process comparision

9 Confidence is gained as soon as tests are started &tests completed earlier: More than 50% confidence is gained with the SnE approach by the time all the scripts are ready for the mixed tests. The dynamic testing scenarios (Type-C) help in gaining that extra bit of confidence on the application as compared to the current process. Additionally as most of the tests get executed during the SnE phase itself, the duration of testing gets reduced by almost 25% percent. Figure 6: Confidence level comparision between current process and SnE approach Build enough data on performance test environment:these tests might also help in building of data or volumes on the servers before the actual type-b and type-c tests (for e.g., mixed load testing) are conducted on the system. The system monitoring all through the type-a test execution can help in identifying additional monitoring needs before conducting the realistic volume type-b tests. Test environment issues get highlighted earlier:issues like load balancer misconfigurations etc. can get highlighted earlier. Monitoring on the servers should be enabled before the SNE phase to get the maximum benefit from it. A lot performance data can be collected from the system under test during the SnE phase and this would help in finding issues related to memory, cpu disk etc. while Type-A tests are executed. Testers get more understanding of the environment:it is a known fact that the more you test, the more understanding you get of the test environment. While running type-a tests, the testers will be able to get further understanding on the test parameters. More time spent on tuning: If the issuesarehighlighted earlier, tuning can also start earlier.

10 Tests are done right the first time and time is not wasted on retests: The test type-a provide inputs for type-b and type-c tests reducing the chances of errors in configuring the tests, thus avoids retests and saves time. From the type-a tests, one can get a lot of insight into the response times for the various call flows. Unit testing the PTscripts: Moreover, this should also help us in finding out issues in performance testing earlier. We do not expect our type-b tests to fail due to memory leak issues, if any, in the scripts which get highlighted during the type-a test execution itself. Now this input can help in designing the complex type-b tests for accurate emulation of real life workload. This additionally saves a lot of effort incurred due to retests if the tests do not go as designed in first place,i.e, it helps to emulate realistic tests (using right pacing). Demonstration of SnE Approach Let s take an example of a type-a test and prove its benefits. Breaking pointtest:contrary to normal practice, the SnE puts the breaking point test in the type-a category as this can be executed using a single script. Additionally, executing this worst case scenario first will give higher confidence of the system and know its capacity upfront rather than on the last day of testing. The breaking point test is executed with an objective of finding the point at which the application produces noticeable exceptions instead of providing valid responses to requests. The coarse grain test was run with 40 concurrent-connections ramping up every five minutes for up to 480 concurrent-connections and then was run for five minutes before stopping all concurrent-connections simultaneously. Application breaking point Figure 7: Errors per Second Vs Concurrent Connections graph As seen in the graph above, Queue exceed maximum capacity of: '256' elements errors begin appearing at the 45- minute mark in the Performance center logs and continue through the remainder of the test. At this point, there are 400 concurrent-connections on the system. This load level is the Breaking Point of the application for this scenario.

11 At roughly 6mins into the test the throughput peaks up Figure 8: Transactions per Second Vs Concurrent Connnection graph In the Transaction per Second graph above, the TPS rate peaks early in the test (at six-minute mark) with 50 concurrent-connections. After that point, the level remains relatively steady until the breaking point when some drop in TPS was seen, however, remained constant after this for the remainder of the test. So from this test we know that the system cannot handle more than 480 concurrent user load. The actual figure with type-b test when run with all the scripts together will be very close to this figure. By executing this test first we have found out an approximate value of the system capacity with the first test itself. Additionally, this test has also provided us input on the rampup setting to be used for the type-bpeak performance point test. We roughly know that peak performance point lies somewhere between concurrent connections. However, a more granular type-b test needs to be done to prove it. Below is the result for the type-b Peak Point Test. The objective of the Peak point test is to find an accurate value for point of peak performance and SLA point. As we already know that the peak point roughly lies between concurrent connections we ramp up only 5 connections after every 5mins up to 100 concurrent-connections.

12 Figure 9 : Transaction per Second Vs Concurrent Connections graph The graph indicates that TPS increased with the number of concurrent-connections put in the system. At around 50 concurrent-connections (at 45-minute-mark) optimal point occurred, after this point the TPS remained constant in spite of the increase in concurrent connections. Type-A test inputs helped us to design the peak point test accurately and we were able to determine that 50 concurrent-connections is the optimal point the very first time we ran the test. Additionally, we were able to know the breaking point of the application with our first script itself, if this was one of the objectives of this testing exercise this would have also resulted in good test coverage. Prerequisites for implementing SnE approach successfully Listed below are few of the pre-requisites need to successfully implement SnE approach Test data needs to be available at-least for a few scripts to start with the test execution Volumetric information should be available as this is the basis of prioritizing the call flows which need to be scripted first Access to servers for monitoring during the scripting phase itself LB or infra /DBA support needs to be agreed before hand The system should be functionally stable while scripting or testing is being done on the system Testers should have the liberty to attempt and break the system and infra/dba support should be available to bring the system back in shape Shared environment cannot be used for performance testing and scripting

13 Overhead The SnE approach might have a bit of overhead on the phases prior to scripting. Below are the various phases and the additional tasks that might need to be done on it. Planning phase: While defining the strategy and making the test plan, some thought might need to be put in the kind of tests can be done as part of Test type-a&b. The prerequisites for SnE phase should be readied before the phase itself. Test data requirements too should be planned, finalized and requested in this phase itself. Test environment: The approach on script on one environment and test on the other cannot be used with this approach. That is we would be required a separate or isolated test environment for testing. Scripting phase: This approach can have a bit of overhead on the scripting process as the person would need to put some efforts in executing the scripts. There are few options available wherein this overhead can be reduced to around 20% of scripting effort so basically we are saving in on a maximum of 50% of the time in identifying the defects, but spending a little over 20% duration of scripting in getting it completed. If the PT project is being tracked for scripting progress then we might see a lesser completion percentage of scripting as compared to the current process implementation. However we can see the percentage progress in the execution during the scripting phase itself. Types of projects which can benefit from this approach Almost all projects can benefit from the SnE approach, however the below projects will have maximum benefits: Projects which have a very small execution window Projects where testing is attempted for the first time Projects where only a few resources are working for the entire PT cycle, eventually leading to elongated test scripting duration Conclusion SnE approach is the next level of Fail fast with Agility where in addition to prioritizing tests,tests are executed earlier to identify and fix defects as early as possible. I.e.Instead of waiting for all the scripts to be completed, performance testing is started in a structured way right from the time the first performance testing script is ready. This approach also realizes the power of one; i.e., things that can be achieved by a single performance script.

14 Author s biography Parag leads the Performance Testing vertical within QA-COE at Eaton Technologies. He has close to 10 years of extensive experience in performance consultancy, management, testing and tuning; and has worked on all major industry standard or open-source performance testing/engineering tools. Prior to Eaton he was working as project manager/consultant with Performance-Engineering group within Techmahindra and handled many critical telecom/banking projects for varied clients across the globe Parag leads the Performance. He completed his BE in computer science from RGPV University, Bhopal and has a Master s degree in Telecommunication and Software engineering from Birla Institute of Technology and Science, Pilani.