Performance Testing of a Large Wealth Management Product

Transcription

1 Performance Testing of a Large Wealth Management Product Meherphani Nori & Global Head Quality Assurance Krishna Kankipati & Vice President Mohan Pujari & Product Specialist Broadridge Financial Solutions (India) Private Limited

2 Abstract Performance testing is one of the non-functional testing types used to determine how a system performs in terms of responsiveness and stability under a particular work load. Often times before deploying an application in Production Performance testing is used to validate non-functional attributes of a system such as response time, scalability and resource usage. Financial Services industry has applications especially in certain functional areas that need to adhere to certain non-functional attributes in addition to having the required functional accuracy. This is more true in case of web based applications that are accessed by end customers as an unaddressed delay in response times as an example may lead to customer dissatisfaction. This paper discusses how one of the products in the Wealth Management space that had extremely stringent Performance requirements that included: Response time End of day batch processing limitations With preset SLA s and was tested for Performance. It details how the QA team went through the process of Performance testing the application, followed by tuning and retesting ultimately resulting in satisfied SLA s and client go-live.

3 INTRODUCTION As mentioned earlier, Performance testing is one of the non-functional testing types used to determine how a system performs in terms of responsiveness and stability under a particular work load. Various performance testing types are: Load Test: This is used to determine if a system can withstand the anticipated user loads. When the load placed on the system is raised beyond normal usage patterns in order to test the system s response at unusually high or peak loads, it is known as stress testing. Stress testing is usually done to identify the upper limits at which the system will break. Load testing helps to identify the maximum operating capacity of an application as well as any bottlenecks that might interfere with its operating capacity. Endurance Test: This is used to make sure the application can handle the expected workload over a long period of time. These tests are especially useful when trying to track down memory leaks or corruption. Spike Test: This is used to test the application reaction to sudden large spikes in the load generated by users. These tests are perfect for indicating whether system has the capability of handling unexpected loads of connections and users. Capacity Testing: This is used to determine how many users and/or transactions a given system will support and still meet performance goals When an application needs to be tested for performance which may include all or some of the above, the first step is to understand the performance requirements that the system should meet. Once these are clearly agreed then a tool that helps in performance testing needs to be leveraged and the application tested to measure performance. A number of tools are available in the market including open source that can be used for Performance testing. Some of the tools are: Load Runner, Webload, Rational Performance Tester, Visual Studio Team system, Neoload, Testing Anywhere, Appvance, WAPT, Httperf, Loadster, LoadUI, Apache Jmeter, LoadImpact, Open STA, Qengine and Loadstrom. Of the above LoadUI, Apache meter, LoadImpact, Open STA, Qengine and Loadstrom are open source. While different tools mentioned above offer unique benefits and a user should make an appropriate choice determined by the cost as well as the objective, it may be noted that as to how a tool can be chosen is not discussed in this paper. It is important to note that in addition to the above tools, a tool is also needed to monitor and measure performance test metrics. Some of the monitoring tools are: Gemfire Pulse, Pivotal VRP, Microsoft System center 2012 (SCOM) and CA Introscope.

4 PERFORMANCE TESTING METHODOLOGY This section discusses the Performance testing methodology that may be used when testing an application for Performance. Different steps in the process are mentioned with some detail as to what is involved at each step. Identifying Performance Requirements: When the business requirements are documented it is important to understand the performance requirements of the new application that is to be built. This should involve a series of interviews with the business users so that accurate SLA s are captured. It is important that Performance Test Engineers are involved in these discussions which many times are not done. Some of the important performance test requirements that need to be captured during this phase are: Transactions Per Second Transactions Rates: Peak and Average Maximum number of users Maximum number of concurrent users Throughput (Amount of Data Sent and Received per Screen) Data Volumes Workload Input & Hardware and Software configurations Response Times This data can also be used by the architects that would need to determine the architecture of the system that will be built as they will be involved in fine tuning along with the developers when performance testing is done. It is important that the culmination of the above exercise is to have an agreed acceptance performance test criteria with the customer. Tools and Environment: Selecting the tools and deciding on the environment is an important step. While on many occasions the environment is not given priority, it should be accorded top priority as it takes time to set this up and the environment set up is done by an independent department and generally not the QA team. Care should be taken that the test environment matches the production. Any differences should be understood including the impact on the testing and test results. Selecting the right tools for testing is a non-trivial process and one should take help from a knowledgeable experts/organization especially when not well versed with performance testing Performance Test Strategy: This is the most critical step in the testing process. The performance test strategy should detail the approach the testing team will take in using the selected tools in testing the application to ensure the Performance requirements are met. Following should be kept in mind while drafting the performance test strategy: Application Architecture/Design and Technology Platform Workload distribution to be used to exercise the application servers

5 User patterns to be modelled to include user Think times & Pacing Data management issues Tests to be performed Performance testing schedules Performance metrics to be collected Performance acceptance criteria Performance exit criteria Test Design and Test Execution: During the test design phase the following activities are carried out: Designing test scenarios and Generation of Test data Once the above are done, the performance test scenarios are executed. Each load test is executed as defined in planning phase such as user ramp-up rate, test duration and maximum user load simulated etc. While tests are in progress monitoring tools will collect the general system and application level metrics. To ensure the accuracy of the metrics collected, each test is repeated multiple times and average values of metrics should be captured from successful runs. These test results will be used to analyze the results/identify the bottlenecks & Memory issues if any. Few things to be kept in mind while executing tests are: Reboot all Load Generators, Application Servers, DB Servers and Web Servers before the test Don t run any Virtual users on the load testing tool machine (Controller in case of load runner) Make sure there is plenty of free disk space on Controller and Load Generator machines. Ensure that no other unnecessary applications and processes are running Analysis of Performance Test Results and Reporting: Performance test results are analyzed based on Performance under various loads, Transactions per second (TPS), Database throughput, Network latency, Think Time, etc. Based on the results analysis, a comprehensive Test Result report to all stakeholders that contains information & graphs, response times with user load, metrics related to App/Web/DB servers should be provided. The following Key Performance Test reports/graphs can be generated during the Test Result Analysis & Reporting phase

6 Application related graphs/metrics System related graphs/metrics Transaction response times Transactions per second Transaction performance summary Running Virtual users Hits per second http responses per second Throughput Error statistics by description Maximum number of connections Average Transaction response time Processor usage Memory usage Disk I/O rate Network Latency Committed bytes Available Mbytes Context switches/sec Pages/sec Pager faults Processor Queue length Table 1 Key Performance Test Report Graph Types Apart from counters mentioned in above table, Web/App/Database performance counters should also be included. It is recommended that the following data should be part of every test reporting: Response Times Transaction Performance Summary Number of Virtual Users Throughput Resource Utilizations (CPU & Memory) Network Utilization and Network latency Application specific counters When testing a specific application for performance while tweaking of the above process is possible, it is important to note that the following mistakes should be avoided: Not having Performance Goals Ignoring Performance measurement during design phase Performance testing is not conducted until Application is completely developed Focus on Single User Load Test Incorrect Extrapolation

7 CASE STUDY Above approach was used in testing a retail Wealth management Product that was built. This was large size application where the effort to build was more than 80 person years. The key performance requirements included a sub-second response time and End of Day (EOD) batch 5 times faster than that of the old system. An additional challenge was that this system was built using latest technologies like Green Plum and Gem Fire so studying the behavior/bottlenecks was being done by the team for the first time. In the rest of this section the key facets of the methodology detailed in section 3 will be discussed. 1. Identifying Performance Requirements: The QA team used a questionnaire to elicit the requirements. This questionnaire was circulated to key stakeholders to capture performance needs of the application/product. The key questions included identifying the protocols used in communication between the user desktop and server, user locations and number of users in each location, number of concurrent users, key business flows, details of test and production environment. 2. Based on the work carried out by the team it was decided to use Load runner, CA Introscope, Gemfire Pulse monitoring tool and Pivotal VRP. 3. Post the study of the requirements, key business flows, performance needs and tools selected a performance strategy was written. The strategy included the following standalone tests to be executed after loading the application database with 12.7 million accounts having 3.07 billion transactions: a. Load testing of User Interface, Web services b. Testing of extracts c. Testing of EOD batch process d. Endurance testing of web services In addition to the above standalone tests the following tests were also performed a. Combination of EOD batch and Web services b. Combination of User Interface and Web services load test c. Combination of extracts, web services and User Interface test 4. Test scenarios were developed to validate both stability and performance of User interface, extracts, batch loads and Web services. These were used to measure against the SLA s documented and feedback shared with the architecture/development team for tuning before another series of tests can be executed

8 5. Performance test results and Reporting: This section provides what is measured and the graphs and other reports used in the study Response Time Graph: This is used to check if the developed application is meeting the SLA s against response time. Response time is the time taken by the client from connecting to the application server under test until receiving the last byte of the response. In other words, it is the duration of the start of connection to the end of the response. 90 percentile values were considered in measurement. The below graph indicates webservice response times in 90 percentile. The team compared the response time with provided SLA s. The below graph shows the three webservice responses have exceeded SLAs. This was reported to the appropriate teams for correction. References & Appendix Graph 1 Response Times After fine-tuning of the code, QA team repeated the tests. Graph below indicates the webservice response times SLA s are met.

9 Graph 2 Response Times Virtual Users Graph: These graphs are used to understand if the application can withstand the load that it will be subjected to. In order to do this virtual users are created and load test is performed. During load test scenario execution, Virtual users generate data as if they perform transactions. Virtual user graphs can be leveraged to study the response. Below scenario have 50 concurrent virtual users. It can be seen that 26 started immediately followed by another 24 after 2 seconds. This helped us understand that a user load of 50 concurrent users can be taken by the application.

10 Virtual User Details Graph: (Active Users vs Elapsed Time) R a m p - U p R a m p - D o w n Graph 3 Virtual Users Throughput Graph: Throughput is the number of transactions produced over time during a test. Throughput is measured in bytes and represents the amount of data that Virtual users received from the server at any given second. It s also expressed as the amount of capacity that a website or application can handle. This graph shows the amount of data that went through the application when it was handled by 50 concurrent users Graph 4 Throughput

11 Transaction Summary Graph: This graph indicates the number of transactions that passed/failed during user action as detailed in the test scripts. QA team studied the failed transactions to identify the bottlenecks/errors in the application if any. Graph 5 Transaction Summary Errors Graph: Virtual users may not be able to complete all transactions successfully during load test scenario execution. Error graphs allow viewing information about the transactions that failed, stopped, or ended in errors. This graph displays the number of errors that occurred during the scenario execution, grouped by error description.

12 Error Statistics (by Description) Graph All the above graphs are analyzed and discrepancies reported. In the above graph a Gemfire Cache error was noticed as Out Of Memory issue. The below graphs captured from Pulse Monitoring Tool during the load test execution, show 44 JVMs were down and memory usage increased significantly.

13 Before Tuning Figure 1 Gemfire Cluster-1 Memory view After Tuning: As given below no JVMs were down and no Out Of Memory issue. Figure 2 Gemfire Cluster-1 Memory view

14 CPU Utilization: The application contains 48 segment servers and 2 master servers in the Database Architecture. While EOD Batch process was running one of the Green Plum segment server CPU utilization reached 100% and this was identified as an issue. Figure 3 GP Database Server CPU Utilization Below graph is post tuning and verifies that no Green Plum segment servers have reached 100% CPU utilization. Figure 4 GP Database Server CPU Utilization

15 CONCLUSION This paper discussed how a large product built in the Wealth Management space using latest technologies was tested for performance. The methodology used, tools and results have been presented. Subsequent user tests have been extremely satisfactory and the client migrated on to this platform in a shorter time window than was planned which is an ample demonstration of the results used in passing the system for user testing and acceptance.

16 Author Biography Meherphani Nori, Global Head Quality Assurance, Broadridge Financial Solutions has a Master s degree from the Indian Institute of Technology, Kanpur and Diploma in Finance. In his experience of about 23 years, he has played varied roles involving strategy, application development, quality assurance and execution. During his role in India Market Strategy was instrumental in signing first client for Broadridge in India. In the Year 2012 was awarded IT Professional of the Year by the Hyderabad Management Association (HMA) a premier association of Management Professionals in the state of Andhra Pradesh. Prior to Broadridge, was with Tata Consultancy Services playing key roles in their IT Consultancy and Account Management groups. Krishna Kankipati, Vice President, Broadridge Financial Solutions has Master s degree from IIT Mumbai has extensive experience in managing the offshore project teams to assure high quality of software being delivered to leading Broker-Dealers of US market. In his current role Krishna heads the Quality Assurance department covering all the products offered by Broadridge to US Equity, Fixed Income Securities Processing & wealth Management domain. Mohan Pujari Product Specialist has studied MCA (Master of Computer Applications). In his 12 years of experience in the field of testing he has worked in Performance Testing, Functional testing & Test Automation. His areas of expertise in Performance testing are monitoring & tuning of Web applications. He also has expertise in Load Runner and JMeter, Sitescope, Introscope, MoreVRP & Pulse tools. In test automation he has expertise in designing test automation frameworks suitable on various business models.

17 THANK YOU!