Load Testing and Monitoring Web Applications in a Windows Environment

OpenDemand Systems, Inc. Load Testing and Monitoring Web Applications in a Windows Environment Introduction An often overlooked step in the development and deployment of Web applications on the Windows platform is load testing the application to ensure that it will perform as expected when accessed by the maximum number of authorized users in the production environment. This article emphasizes the importance of load testing as it relates to developing applications in a Windows environment and offers a few tips that will make the process easier to complete. The purpose of this article is to assist the experienced developer and IT professional in designing and implementing a complete load test scenario, in diagnosing the test results, and to recommend changes to correct deficiencies. Why Load Test My Application? Load testing should always be performed before deploying the application in a production environment. The basic purpose of load testing a Web-enabled application is to accomplish the following: To accurately measure the individual user experience as overall user load is increased on the system To determine the maximum capacity of the hardware utilized by the application and thus determine whether a hardware upgrade will be necessary before the deployment of the application in a production environment To define the acceptable performance threshold in terms of average page response time for the user of the application To ensure that the performance threshold remains at an acceptable level when the estimated maximum concurrent user load is placed on the system For most Web applications, the user experience is what matters most if the application is to be successful. However, there are many good reasons to put your application through the load testing process, including the following: Applications that work well during development can perform poorly when operating in a high load environment. For example, when Internet Information Server or SQL Server machines are being utilized by multiple applications simultaneously, adding your new application to the mix could interfere with, and maybe even break, existing applications if you haven't designed the application to operate correctly in this scenario. 1

Your customers will form their most important impression of your application the first few times they use it. If this impression is unfavorable because of load problems, it will be very difficult to change their minds, even after you have resolved the problems. On the other hand, by adequately load testing your applications before deployment, you establish a reputation with your customers as a developer who creates good, fast applications that work the way they are intended to work. The IT group responsible for deploying and maintaining your application will appreciate your load testing as much or more than your customers. They are on the front line and will be the first to hear opinions and observations. If you can reliably anticipate the scalability issues that will affect the IT group, the members of this group will be much more willing to assist you when you are in need of their skills. Potential business partners should also be included in the customer satisfaction matrix. They can significantly enhance the success of your application if they decide to include it as part of a larger application package. Preparing to Run a Load Test To determine best-case server analysis values, also known as a baseline, a staging or pilot system configuration should first be tested in a controlled environment. Next you should test in a simulated production environment to determine how the production environment configuration affects the results of the pilot system. In preparation for the load testing phase of your development cycle, you will need to focus on the following areas: Hardware and software configuration Server configuration Security configuration User load configuration Selecting the right load test tool Hardware and Software Configuration The pilot system should mirror the production system as much as possible. Hardware configurations for CPU, RAM, and network bandwidth are the most important areas that will be exercised during load testing. Every effort should be made to duplicate the software configurations, such as the appropriate Microsoft Windows version and service pack, the Internet Information Server configuration, and any other applications that will be running on the same machine in the actual production system. Be sure to configure your test Web site to run either in-process or out-of-process, whichever the final configuration will require. Selecting this option determines whether your Web application will run in the same address space as 2

IIS or in its own isolated address space. This configuration will have a major impact on the load tests that will be performed. In Windows NT 4.0, select the Run in separate memory space (isolated process) check box to run the Web application out-of-process. In Windows 2000/2003, select an Application Protection setting of Medium or High to run the Web application out-of-process. Server Configuration Configure Internet Information Server to mirror the production server. The Internet Services Manager properties page provides various tuning options available for IIS. Of particular importance are determining whether to enable logging (which can slow down the system considerably) and, under the Performance tab, selecting the number of expected hits per day. The data server is where the majority of load-related issues are likely to be resolved. For efficient query execution, proper normalization of the database design is imperative. Therefore, it is imperative that you load test with the actual database design your application is intended to work with, and you should ensure that the tables are populated with the maximum amount of data your application will generate. In addition, ensure that your test data server configuration options (most importantly, the locking and isolation levels and the optimization techniques employed for example, table indexing) match the configuration options of the production data server. Security Configuration Your application's security scheme can have a severe performance impact on your application under load, especially if the system includes an encryption technology. Therefore, you should configure your test system to use the same security scheme but not necessarily the same credentials. If possible, you should consider the use of roles to simplify and streamline the security authentication process and to enhance performance and stability. User Load Configuration First determine the maximum number of users you expect to access your application, and then double this number; a successful application will most likely be utilized by many more users than expected. In addition, calculate the time of day the majority of the users will need access and then determine the load that will be on the network during that time, which should be the time when you test your application. This strategy enables you to test the user load impact as well as the system-wide hardware configuration, to ensure that your application responds as expected during peak network periods. OpenLoad Tester The OpenLoad tester exercises the testing environment by realistically simulating multiple browsers requesting pages from a Web application and allows you to record a user scenario by accessing, from your browser, the pages you want to include in the test. This user scenario can then be saved and run from any location using your browser. 3

It is not necessary to have multiple client machines available as the OpenLoad tester is able to simulate hundreds or even thousands of clients from a single workstation or server. Running the Load Test After you've carefully planned out your testing strategy, actually running the tests is easy. The first task in performance testing is to use a tool, such as the OpenLoad tester, to apply load to the Web site and measure the maximum requests per second that the Web server can handle. This is a quantitative measurement. The second task is to determine which resource prevents the requests per second from going higher, such as CPU, memory, or back-end dependencies; this process is more an art than it is a precise measurement technique. Recommended Windows Performance Monitor Counters To effectively load test your Web application and correctly diagnose the results, a method of monitoring and recording operational statistics is of paramount importance. Performance Monitor, a tool that ships with Microsoft Windows NT, Microsoft Windows 2000 and Microsoft Windows 2003, is the best tool available for monitoring and recording these statistics, both in Internet Information Server and on your data server. When running your application inside Internet Information Server, you should monitor (using Performance Monitor) the following counters: Active Server Pages: requests per second, requests rejected, total queue length, and number of current sessions Inetinfo process: private bytes, virtual bytes, and number of open handles Processor: percent user time versus percent privileged time Note If your application is running in its own memory space in Windows NT 4.0 (or with a setting of Application Protection: High [Isolated] on the Load Properties page in Windows 2000/2003), you should monitor the mtx.exe (or the dllhost process in Windows 2000) rather than monitor the Inetinfo process. The Active Server Pages requests-per-second counter in Performance Monitor will display the actual throughput of your application. This statistic will enable you to diagnose the performance of Internet Information Server under load and will pinpoint the potential bottlenecks. This in turn will allow you to judge your application's ability to serve the maximum amount of users with an acceptable response time. A Web server running ASP technology assigns every page request a thread from a pool established at startup; if all threads are used, subsequent page requests are placed in a queue. By monitoring the total queue length with Performance Monitor, you can determine how many clients are waiting for a response from the server. 4

The two most common non-hardware load-related database problems are deadlocks and locking concurrency. On the data server, using the custom Performance Monitor counters that the data store will provide, you should monitor at a minimum the following: Lock requests Deadlocks per second Table lock escalations per second User connections Active transactions Your Web application should also be configured to take advantage of OLE DB resource pooling, which is managed automatically by the middle-tier OLE DB provider for Microsoft SQL Server. By creating the connection objects on a per-page basis and immediately releasing them, the database can handle thousands of concurrent users with a much smaller number of open database connections. This conserves database resources and provides much greater scalability. This performance enhancement should be monitored by tracking the number of user connections (using Performance Monitor) on the data server. As throughput increases, the number of user connections should remain stable as pooling controls the number of connections actually created on the data server. The process of tuning an application against the database is critical to achieving performance goals and must be factored into the development cycle. This should include optimizing the size of the allocated memory and distribution of the application on the disk drives and controllers. Give special consideration to eliminating the marshaling of data between processes whenever possible, because this is a very expensive operation. Run the load test for a period of time that is 50 percent longer than you expect the application to run in the actual user environment without interruption. Many problems, memory leaks in particular, do not manifest themselves until after the application has been running for an extended period of time. What to Look for During the Load Test The average value for each Performance Monitor counter is determined by your specific application and hardware configuration. Therefore, when the load tests are running, you should check each counter for any abnormal deviations from these average values. Monitoring Internet Information Server The most important areas to monitor on the Internet Information Server machine when looking for bottlenecks are the following: CPU utilization 5

Memory usage Throughput Optionally, depending on your projected application environment, there are other performance areas that you might want to track during load testing. Any of the following possible scenarios may indicate a problem with your application that should be repaired before the final release. CPU utilization A decrease in CPU utilization can point to a decrease in the performance of your application, possibly a thread contention issue. As you monitor the ratio of CPU time between user and kernel time, remember that the user time should be 80 to 90 percent of total CPU time as a rule. Therefore, kernel mode time in excess of 20 percent is an indication of contention for kernel level API calls. To get your money's worth out of your machines, you should register better than 50 percent processor utilization during peak load. A lower utilization value might indicate there are other bottlenecks in your system that need to be resolved. Memory usage A sudden jump or gradual increase in memory usage is another area that is frequently identified as a common problem for long-running server applications. Normally, this is where memory and resource leaks show up during the testing phase. Throughput Monitoring the Active Server Pages requests-per-second counter allows you to identify when or if your application begins to have performance problems. This counter will normally vary in a real-world application, but by carefully setting the number of threads and concurrent connections, you will be able to simulate a steady number of requests. A sudden decrease in this counter spells trouble. Optional testing areas The following are examples of other areas you might find well worth monitoring during load testing: Total queue length. Typically, the Total Queue Length counter in Performance Monitor goes up and down. Therefore, if total queue length never increases and you are running at low processor utilization, this 6

could indicate you have a smoothly running site with more capacity than you need for that load load. Or, if your queue length is going up and down but your processors are running below 50 percent, this could indicate some of your requests are blocking and may require further optimization. Browser response time. You can periodically access your Active Server Pages from a browser to monitor the response time and to ensure that the load test is running correctly and the Web site is still able to serve ASP pages correctly. It is recommended that you perform this test at least twice daily during load tests. Timeout errors. During browser testing, be on the lookout for timeout errors, which are returned by Internet Information Server; these errors might indicate too many users are accessing the application simultaneously. Monitoring the data server The internal processing of the various services with the data server and the formatting of data for display normally consume the majority of the available server resources dedicated to the Web application. Therefore, when load testing the application, it is imperative that special consideration be given to the performance of these components as it relates to the data access and data manipulation areas of the application. Database user connections, lock contention, and deadlocks are the primary candidates for monitoring on the data server. Periodically view the process information in the management console for your database (for example, on a machine running SQL Server, in the Current Activity area of SQL Enterprise Manager). Check for blocking server process IDs, a common cause of data queries that do not return a response. This is a contention problem and usually requires significant changes in either the database design or the application logic. Deadlocks can be identified in various ways. The most common method for identifying deadlocks is by using the Number of Deadlocks/sec custom counter in Performance Monitor. Your application should already be checking for deadlock problems and responding appropriately, because allowing the data server to designate the deadlock victim (that is, the user or session that will be canceled to resolve the deadlock) could cause problems with your application. The application should detect the deadlock condition when encountered and react accordingly. A common response to a deadlock situation is to wait for a few milliseconds and then retry the operation; normally, deadlocks are simply time-sensitive errors that will disappear when the operation is retried. Closing Summary The Internet opens your applications to many more potential users than traditional client-server applications. As more and more organizations look to the Web as a strategic part of their business strategy, they need to be sure the technology they choose can handle their demanding needs. Besides easy-to-use tools, organizations need an infrastructure to meet their user load requirements. Therefore, it is more important than ever that load testing be a fundamental part of your testing regime. 7

Note A preliminary requirement for operating successfully under load conditions is taking a best-practices approach during the development cycle. This means scheduling time both for performance testing under load and for tuning the application to achieve performance goals must be factored into the development process. The benefits of load testing and iteratively tuning your application under load are straightforward: You get the information you need to ensure superior throughput results for your application. You can accurately assess the scalability characteristics of your application so that you can adjust the application to achieve specified performance goals. You can discover early the design issues that degrade performance and throughput, and adjustments can be made before the application is deployed into production. Your applications will gain a reputation for high-performance dependability among your customers and business partners. OpenDemand and OpenLoad are trademarks or registered trademarks of OpenDemand Systems, Inc. *Other names and brands may be claimed as the property of others. 8