1 Performance Test Report: Unparalleled Scalability of a Consumer Information System (CIS) for Energy Utilities Performance Testing confirms that PowerOneData s Consumer Information System (CIS) is robust and can handle up to 4,000 concurrent users continuously for 12 hours, a figure corresponding to over 353,000 actual users. Performance Test Report Published: 1 st September 2012 Applies to: PowerOneData Consumer Information System (CIS) Application for energy utilities Copyright Reserved 2012, PowerOneData
2 Abstract In June 2012, Verifaya worked with PowerOneData at its Bangalore, India office to conduct a series of test scenarios on their Consumer Information System (CIS) version 1.0 for Energy Utilities running on Microsoft SQL Server R2 Enterprise Edition, with a goal to understand the application scalability under various loads. The test results overwhelmingly confirm that PowerOneData s CIS v1.0 is a highly scalable system, capable of supporting several thousand concurrent users under real-world conditions. The application was heavily tested under increasing load conditions to understand the behavior of application under stress. After conducting a series of tests and tuning every aspect of the application and database, we have found the application to be highly stable, scalable and capable of handling an increased number of concurrent users in the future without any error, meeting the requirements of major energy utility companies. In our view, energy utilities can be confident that PowerOneData s Customer Information System, when deployed in an up-to-date Microsoft Windows Environment will provide a stable, reliable, scalable, and manageable Customer Information System that will support even the largest and fastest growing bases of consumers.
3 Contents 1. Introduction What does an Energy Utility need and what do their Consumers want? Characteristics of good consumer web applications Application Studied Background of this application The reason for building such an application Need/necessity to test the application for Load/Soak/Stress Testing Reason to perform Load testing Reason to perform Stress testing Reason to perform Soak testing What is needed for an application to scale well? Development and Testing Environment What was the development process and technology used? Development Process Technology Used What was the testing process used? What was the software tools used for testing the application? Apache JMeter ANTS Memory Profiler Google Page Speed Windows Performance Counters ANTS Performance Profiler What hardware was used and how was the testing environment set up? Testing Process and Execution What were the key scenarios tested in the Application? What were the Key Performance Metrics that were measured by JMeter? Application Server Memory and CPU utilization snap shots during the Load Test Database Server Memory and CPU Utilization during the Load test? Inference and Results Does the Web application function properly under given load?... 15
4 1. Introduction 1.1 What does an Energy Utility need and what do their Consumers want? Many energy utilities in developed and developing nations still struggle to achieve a successful Smart Grid roll out. The challenges include hardware and infrastructure, but the most important elements for delivering the benefits of the Smart Grid is engaging customers. Without consumer engagement, the most advanced hardware and complex infrastructure will not be fully utilized, resulting in a large investment without optimal results. One of the most effective ways of engaging customers, rather than simply focusing their attention on the potential economic benefits, is by educating consumers and giving them the tools to save money by managing their own consumption. In a Utilities perspective, a web application not only serves to improve customer service and overall customer satisfaction but also helps in resolving issues based on consumer service requests and feedback. It also reduces the time required for solving billing issues hence reducing operational cost and increasing operational efficiency. There should also be real time consumption and billing analysis which helps in analyzing peak load consumption and billing patterns. In a Consumers perspective, a web application should be easy to use and navigate, should provide information on their energy usage and inform them about rate changes that have been put into effect from utilities. Web application should help consumers to reduce the peak load by viewing their energy consumption and pricing graphs. Should provide a payment gateway where they can pay his/her bill online hence saving time. 1.2 Characteristics of good consumer web applications Interactive web-based applications with large database back ends can be tricky to manage and maintain. Users of Web based applications change quickly, expect fast response times, and have diverse needs. Even though an energy utility may service only a defined geographical area, consumers expect to be able to check their accounts and usage from anywhere in the world, 24 hours a day, and 365 days a year. A highly available application, by necessity, must also be highly reliable and scalable. Every good Web based application benefits from several key attributes:. Speed of access, easy usability, ease of information location, user friendly navigation, and continuous availability. Since a utility in a large city may add hundreds of new consumers each week, the application must scale smoothly without slowing down or experiencing errors under increased load. 1.3 Application Studied We conducted a rigorous series of throughput and performance tests on PowerOneData s Consumer Information System (CIS) v1.0 for Energy Utilities Background of this application PowerOneData s CIS is a web-based Business-to-Consumer (B2C) application service with a database back end that allows a user to track his other electricity consumption. This module provides solution comprehensive view of usage patterns and payment transactions for the consumer. Once the consumer creates an account on the utility s customer portal, they can chart load profiles, analyze tariff plans, view bills, pay bills, and analyze their usage and payment history. If the utility chooses, their customers can also make requests for changes to their service online. In the future, the application may also be used by energy utilities to manage outages and to encourage consumers to conserve energy during periods of high demand (a strategy known in the industry as Demand Response ).
5 1.3.2 The reason for building such an application The electrical grid in many countries around the world, and especially in quickly developing areas, is under tremendous stress. Many energy utilities are unable to deliver all the energy that their customers need, leading to brown outs and unpredictable outages as utilities are forced to shed loads to protect their networks. To help avoid these unwanted and disruptive outages, energy utilities want consumers to actively participate in a Demand Response program, in which consumers use the information that the utility provides them in real-time about their consumption to turn off unnecessary equipment during peak hours and to shift their demand to off-peak hours.. A key component of any Demand Response system is the ability of consumers to get information from a utility through their Internet portal, such as PowerOneData s CIS application. 1.4 Need/necessity to test the application for Load/Soak/Stress Testing Testing is needed to verify and validate that the application meets its specifications. Testing ensures that what you get in the end is what you wanted to build. Testing enhances the integrity of a system by detecting deviations in design and errors in the system. Testing helps to identify error-prone areas and therefore limits the generation of new errors. Testing also adds value by ensuring that a product delivers on its promise to the customer by making sure advertised features actually work as claimed. The deployment of web technology in sophisticated software has created a major need for rapid, effective Quality Assurance (QA) testing solutions. Software needs to be tested for o o o o o o Functionality Usability Interface Compatibility Performance Security Of the above mentioned tests, this report outlines the results of Performance which was conducted under varying load conditions to understand the scalability of the application. The process involves simulating large numbers of virtual users, who use the web application in the same way as a real user would use it in the physical world. Each test is run multiple times at increasing levels of load so that the results from each test iteration can be used to make coding or architectural changes. The adjustments can often have a significant impact on the performance and scalability of the web application. Customers hate to wait while a web page is being loaded. As general rule, all pages must load in 15 seconds or less. This rule will of course depend on a particular application and the expectations of the customers using it. Since PowerOneData s CIS is a web based application and large number of consumers will be logging in at the same time to know there usage, performance testing is essential to ensure the application s scalability Reason to perform Load testing Goal: Find metrics for system performance under high load. The basic purpose of executing load and performance testing for an application and/or website is to accomplish the following:
6 Measure the effects on user experience as the total user load increases (i.e. more people log on) Measure the average page response time at both average and maximum user loads Measure the throughput of the server and server conditions Quantify the upper limit of the hardware being used by the application to determine whether the current system is capable of handling the anticipated future loads Reason to perform Stress testing Stress testing: An effective stress test will gradually increase the demands on an application until it goes beyond its specification limits. Web stress testing is performed to break the site under carefully controlled conditions to understand how the application reacts to stress and how the system recovers from crashes Reason to perform Soak testing In order to find system instabilities that occur over time, we need to conduct tests that run over a long period. That is what Soak Testing is for. Running Load Tests or even Baseline Tests over a long period of time demonstrates how the target environment handles system resources. Ideally, we should see no unwanted behavior occur regardless of how long an application has been running, such as: Memory leaks on the application servers Increased heap size utilization on the application servers Connection leaks on the database server Cursor leaks on the data base servers Response time (inconsistency or degradation) changes from the start of the load test to end of the load test Systems Resource (Memory, CPU, Network and Disk usage etc.) starvation or exhaustion from the beginning of the load test to end of the load test Application error occurrence over the test period 1.5 What is needed for an application to scale well? PowerOneData s Consumer Information System is a web based application. Generally utilities will have a large number of customers in their database and the number will keep growing as the population of the area increases. A successful application for a utility needs to handle customer growth smoothly, maintaining the same good customer experience for the millionth customers as well the one hundredth.. In order to handle the ever scalable consumers in future, this application should be in a state to handle the load and be stable. Hence PowerOneData s CIS should scale well.
8 2.2 What was the testing process used? Figure 2 Performance testing (Refer Figure 2) determines how a system performs in terms of responsiveness and stability under a particular workload. Performance testing is often afterthought and sometimes only done during or after system integration testing phase or User Accessibility Test phase of the life cycle. Verifaya s well-defined methodology and process covers end-to-end performance testing. The initial phase of performance testing is to prepare test plan based on performance test questionnaire, goals and objectives. Once the test plan is ready we establish the testing environment and install necessary tools for testing. Performance testing comprises load, soak, stress and capacity testing. Tuning an application for optimal performance involves considerable time and effort. During each phase of testing we draw results based on response time, break points, peak load, memory leaks and resource utilization. We analyze server, database, application and network reports to profile application performance. Throughout the testing process we are careful to recognize any unexpected or undesirable behavior that would indicate a business risk in the application.
9 2.3 What was the software tools used for testing the application? Apache JMeter Apache JMeter is a fully open-source, pure java tool designed for the purpose of load testing and measuring performance of web applications. JMeter can be effectively used to simulate heavy loads and analyze the overall performance under different load conditions including concurrent loads ANTS Memory Profiler ANTS Memory Profiler finds memory leaks within minutes. Memory leak includes excessive use of memory during the job execution and not releasing it when the job is done. Hence the tool helps in creating better performing, less resource-intensive applications. Intelligent analysis of memory profiler highlights the most likely causes of issues, often saving hours of problem hunting Google Page Speed PageSpeed Service is a browser based tool from Google to check the speed of each page in the site. PageSpeed Insights products will help you identify performance best practices that can be applied to the testing site Windows Performance Counters Counters are used to provide information on how long an application takes to execute certain tasks and how many system resources it consumes while doing those tasks. Performance counters can monitor system components such as processors, memory, and network I/O. The operating system, network and devices provide counter data that an application actually uses with a graphical view of how well the system is performing ANTS Performance Profiler ANTS Performance Profiler helps detect performance bottlenecks in a variety of applications. Performance Profiler can measure, track and analyze an application's performance variables (number of times a method gets called, execution time taken by a method) to tell exactly where bottlenecks exist.
10 2.4 What hardware was used and how was the testing environment set up? Figure 3 Figure 3 depicts the testing eco system which includes 2 groups, The Server Environment and Client Environment. This environment was created at the PowerOneData premises. Client Systems: One machine had a system configuration of Windows Server 2008 R2 Standard Edition, 64 Bit Intel Xeon, 2.93 GHz, 16 cores and other with Windows Server 2008 R2 Data Center Edition, 64 Bit Intel Xeon, 2.67 GHz, 32 cores were used to generate the virtual users (load) for accessing the application. Each machine was installed with the JMeter application, which has the capability to generate 2,000 virtual users, resulting in a total load of 4,000 virtual users. Server Systems: PowerOneData s CIS application was hosted on the App Server machine, which consisted of a Windows Server 2008 R2 Standard Edition, 64 Bit Intel Xeon, 2.93 GHz, 16 cores. ANTS memory profiler, ANTS performance Profiler, Google Page Speed and Windows counters were used to monitor the performance. Webserver IIS 7.0 was hosted on Application server. The database server was on a separate machine with system configuration Windows 7 Enterprise edition SP1, 64 bit, Intel I7, 2.93 GHz, 4 cores was used to store data and communicate directly with CIS application. ANTS memory profiler tool was installed in this machine in order to monitor the memory footprint of the Database.
11 3. Testing Process and Execution 3.1 What were the key scenarios tested in the Application? The testing process included 9 key working scenarios. The scenarios included 1. Visiting Home Page. 2. Logging into the application through Login Page. 3. Verifying Account details and updating it if necessary. Page. 4. Checking bill details in My Bill Page and Billing History Page. 5. Viewing the Unbilled Usage and Consumption Page. 6. My Payment Page to the pay bill. 7. Drilling down Consumption tunnel page to view Yearly, Monthly and Daily consumptions. 8. Payment Processing and Payment History Page. 9. Logging Out. 3.2 What were the Key Performance Metrics that were measured by JMeter? Hits per Second Graph: Figure 4 shows the number of web requests made by Virtual users to the Webserver during each second of the test run. This graph helps you evaluate the amount of load Virtual Users generate, in terms of the number of requests/second. In this case the hits reached a maximum of 1,800 requests/second, the highest recorded during the entire test. Figure 4
12 Web Response codes received every Second: Figure 5 confirms no application errors (HTTP error responses) were observed during the entire test and application functionality worked as expected. Red lines indicate successful serving of the page on each request, Pink indicates that pages were found on the server and Blue indicates that none of the pages were modified while serving. Web Page Response times of every page to the User: Figure 5 Figure 6 measures the web user experience as it indicates how long the user waits for the server to respond for his request. This is the time taken in milliseconds to receive full response from the server. It indicates the exact amount of seconds the user had to wait for the page requested. The graph suggests that all the requests have been served in a fairly less amount of time without any errors. Figure 6
13 3.3 Application Server Memory and CPU utilization snap shots during the Load Test Application Memory: Figure 7 displays the current amount of memory allocated to the application that cannot be shared with other processes. The graph shows the process w3wp takes a maximum of 0.5 GB (512mb) of private bytes for 4,000 virtual users when run for 12 hours. Notice that the graph a constant expansion and contraction of Memory usage, indicating that resources are released regularly to the heap when not in use. This is the indication that system is healthy and not consuming unnecessary resources when is not heavily loaded. Figure 7 Application server Memory Utilization: Private Byte s v/s Time Application Processor Usage: Figure 8 determines the percentage of time the processor is busy for the w3wp process. This counter provides a measure of how much time the processor actually spends working on productive threads and how often it was busy servicing requests. The server is configured with 16 core processors and the maximum CPU utilization per processor is 42.5% and average CPU utilization is 30% for 4000 Vusers when run for 12 hours. Figure 8 Application server CPU Utilization: % Processor Time v/s Time Note: Figure 8 shows maximum %processor time is 170% and an average %processor time is 120 %
14 3.4 Database Server Memory and CPU Utilization during the Load test? Database Server Memory Usage: Figure 9 displays the current number of bytes allocated to application that cannot be shared with other processes. The graph shows the SQL Server process takes a maximum of 1.5 GB (1600mb) of private bytes for 4000 Vusers when run for 12 hours. There has been a constant expansion and contraction of Memory usage thus indicating the release of resource to heap when not in use. Figure 9 Application server Memory Utilization: Private Byte s v/s Time Database Server Processor Usage: Figure 10 determines the percentage of time the processor is busy for SQL Server process. This measurement is the amount of processor utilization of the application. This counter provides a measure of how much time the processor actually spends working on productive threads and how often it was busy servicing requests. The server has 4 core processors; maximum and average CPU utilization per processor is 40% and 25% respectively for 4,000 Vusers when run for 12 hours. Figure 10 Database Server CPU Utilization Graphs: %processor Time v/s Time
15 4. Inference and Results 4.1 Does the Web application function properly under given load? The application was tested under the increasing load from JMeter. The load has been steadily increased from 5 to 4000 virtual users over a period of 12 hours of continuous testing. The average and the maximum response time of the application was well within the acceptable limit. Application doesn t seem to be resource hungry and tend to dispose the resources after the usage. Resources mainly Heap Memory is well within 500 MB during the peak load condition and the CPU usage by the worker process on an average is less than 30% during entire period of the soak testing and has increased to 40% during the peak load. The application does not have a hard ceiling; a proportional increase in server resources should support an ever increasing customer base. The performance of the four layer application scales linearly with increased hardware resources. The database server responded well under the load test, thanks to the fact that the application was designed to handle small transactions efficiently from a large number of customers. Each database object (e.g. Stored Procedures, Functions; SQL-CLR Procedures, etc.) completed its operation within milliseconds. We found no long-running database queries during Soak Testing. Like the application server, the database server handles system resources efficiently, and system resource usage remains steady despite an increase in load. The Memory footprint of the database is stable during peak load access and releases memory back to the heap when it is not in use. There were no memory leaks. The CPU is loaded to no more than 40% with the capacity to more than double the processor utilization when needed. Conclusion Over a period of two months, Verifaya performed a series of progressive performance tests on PowerOneData s Consumer Information System, gradually increasing the number of users and database transactions. The test results demonstrate that the application is stable under heavy loads, and provided there are adequate server resources available, the application scales smoothly as new users are added with no hard ceiling to the number of supported users. The application has been successfully tested up to 4,000 concurrent virtual users for 12 Hours, a figure corresponding to over 353,000 actual users. Application architecture is scalable and can easily accommodate an increase in the number of actual users in production Memory and CPU utilization at each application layer is optimal and predicable Application functionality under maximum load is stable and works as expected. No errors were observed during the 4,000 concurrent virtual users test. Ramana Reddy Chief Technology Officer Verifaya Technologies