The Stress/Load Testing Simulator

Transcription

1 King Saud University College of Computer and Information Sciences Computer Science Departement MSC Project-1 The Stress/Load Testing Simulator Huda Bin Sadiq Under the supervision of Prof. Ghazy Assassa Stress/Load Testing Simulator 1

2 Revision History Date Document Changes 12/3/2007 Document creation 19/3/2007 Related Work 20/4/2007 Requirement specification, tools compression 29/5/2007 High Level architecture, literature review 31/5/2007 Literature reviwe, SLTS planning 2/6/2007 SLTS planning update 11/2/2007 Stress/Load testing process phases Stress/Load Testing Simulator 2

3 Table of Contents Chapter Introduction Performance Testing Overview Related Work (Literature Review)... 9 Chapter A Sample of Existing Tools Proxy Sniffer Apache Jmeter Hammer-Head Jcrawler Mercury LoadRunner Peassler Comparing the Tools Features SLTS Requirements Specification SLTS High-Level Architecture Description Chapter The Stress/Load Testing Process Stress/Load-Test Planning Analyzing the application: Identifying System Components Describing the System Configuration Analyzing the Usage Model Task Distribution Defining Stress/Load testing objectives Planning Stress/Load testing implementation: Stress/Load Testing Simulator 3

4 3.1 Defining the Scope of Performance Measurements Defining Virtual User (Vuser) Activities Selecting Virtual Users Choosing Testing Hardware/Software Chapter Metrics and Measurements SLTS Counter Metric Stress/Load Testing Simulator 4

5 Table of Figures Figure1: Types of performance testing... 7 Figure 2: Manual testing is problamatic... 8 Figure3: A single console controlling several thousand Vusers replaces manual testers... 9 Figure4. SLTS Architecture Figure5: The stress/load testing process Figure6: An online system accessed by many web users Figure7: Measuring end-to-end response time Figure8: Measuring N/W & server response times Figure9: Measuring GUI response time Figure10: Measuring server response time Figure11: Measuring MW-to-server response time Figure12: The stress/load testing process Stress/Load Testing Simulator 5

6 Chapter Introduction Software Testing is the process used to help identify the correctness, completeness, security, and quality of developed computer software. Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality related information about the product with respect to the context in which it is intended to operate; it includes the process of executing a program or application with the intent of finding errors. Testing a system is harder than building it in the face of new risks introduced by the everincreasing complexity of software and hardware because without programs that properly work, the system will never process information and produce the output for which it was designed, therefore testing procedures should be established, testing roles should be applied and testing data that can test the limits of the program should be created. But performing tests using the manual traditional way is a tedious, time consuming and costly job which has led to the development of testing tools. These tools may be the only practical way to conduct tests easily and efficiently. Also they can help in avoiding trouble, providing vital information, and they can enable an organization to take new opportunities with greater performance and strength. There are many types of software testing which include unit testing, integration testing, system testing, acceptance testing, functional testing and non-functional testing. Performance testing is a type of non-functional testing that includes other types of tests which are: 1. Component Testing: Find the behavior and performance of each tier. 2. Load Testing: Testing an application against a requested number of users (real-world load). The objective is to determine whether the site or system can sustain this requested number of users with acceptable response times. Stress/Load Testing Simulator 6

7 3. Capacity Testing: Testing to determine the maximum number of concurrent users an application can manage. The objective is to benchmark the maximum loads of concurrent users a site or a system can sustain before experiencing system failure. 4. Volume or Stress Testing: Load testing over an extended period of time to find stess points. It includes testing beyond normal operational capacity often to a breaking point to determine the stability of a given system or application. The objective is to validate an application s stability and reliability. Figure1: Types of performance testing 1.2 Performance Testing Overview Application load testing is the measure of an entire web application s ability to sustain a number of simultaneous users and/or transactions, while maintaining adequate response times. Because it is comprehensive, load testing is the only way to accurately test the end-to-end performance of a website or system prior to going live. Application stress/load testing enables developers to isolate bottlenecks and find errors in any component of the infrastructure. Two common methods for implementing this process are manual and automated testing. Manual testing has several built-in challenges, such as determining how to: Stress/Load Testing Simulator 7

8 Emulate hundreds of thousands of manual users that will interact with the application to generate load. Coordinate the operations of users. Measure response times. Repeat tests in a consistent way. Compare results. Because load testing is iterative in nature, the tester must identify performance problems, tune the system, and retest to ensure that tuning has had a positive impact countless times. For this reason, manual testing is not a very practical option. Figure 2: Manual testing is problamatic With automated stress/load testing tools, tests can be easily rerun and the results automatically measured. In this way, automated testing tools provide a more cost-effective and efficient solution than their manual counterparts. Plus, they minimize the risk of human error during testing. Today, automated stress/load testing is the preferred choice for stress/load testing a web application or a system. Most testing tools typically use three major components to execute a test. These include: A control console, which organizes, drives, and manages the stress/load test. Virtual users (Vusers), which are processes used to imitate the real user performing a business process on a client application. Stress/Load Testing Simulator 8

9 Load servers, used to run the virtual users requests. Figure3: A single console controlling several thousand Vusers replaces manual testers Using these components, automated stress/load testing tools can: Replace manual testers with automated virtual users (Vusers). Simultaneously run many virtual users on a single load-generating machine. Automatically measure transaction response times. Easily repeat load scenarios to validate design and performance changes. Easily generate reports. These advanced functionalities in turn allow an organization to save time and costly resources. 1.3 Related Work (Literature Review) Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results [14]. Although crucial to software quality and widely deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software [12], [13], [17]. The difficulty in software testing stems from the complexity of software. The purpose of testing can be quality assurance, verification and validation, or reliability estimation [16], [18], [23]. Software performance testing is a major area of software testing. Software testing is a trade-off between budget, time and quality [11], [22]. Stress/Load Testing Simulator 9

10 A search of the published literature using the on-line version of Science Citation Index, returned only two papers: "Performance Testing of Software Systems" [11] by Vokolos and Weyuker, and the other one is: "Deriving Workloads for Performance Testing" by Avritzer and Weyuker [2]. Similarly, using the ACM on-line search engine to search all available ACM papers, journals and conference proceedings returned only little research, Also the search facility for both the IEEE and IEE journals and conference papers uncovered only little research on the subject. There is significant literature describing work on software performance modeling [5], [7] [12], [17], [21] [6], [8], [10], software (functionality) testing, software testing strategies, software testing techniques [13] and automated testing [14], [20], [23], although all these subjects are related, this work does not directly consider how to establish or generate test sets for testing the performance of software, which is the primary goal of our research. A search of the internet for "software performance testing" uncovered lots of rising businesses organizations that offered some sort of tool to aid people doing software performance testing (Ex: Proxy Sniffer [36], Apache JMeter [37], Hammerhead2-Web Testing Tool [38], DBMonster [39], JCrawler [40], WAPT [41], Mercury Load Runner [42] and Peassler [43] ). Also their where workshops or seminars that included a section on the subject, but the search uncovered very little research papers. So, it appears from our search that there is neither an existing body of research literature nor an extensive collection of practical or experiential information available to help testers and researchers interested in software performance testing, and this is expected since performance testing is a relatively new field. We brief some of the scientific researches that we found in the following paragraphs. As we mentioned earlier performance testing papers are very rare in literature, although performance is what most people care about because it is an extremely significant issue for man large industrial projects. Often, the primary problems that projects report after field release are not system crashes or incorrect system responses, but rather system performance degradation or problems handling required system throughput, although the software system has gone through extensive functionality testing, it was never really tested to assess its expected performance. This paper "Performance Testing of Software Systems" by F.I. Vokolos and E.J. Weyuker [11], gives a discussion on approaches to software performance testing. A case study describing the experience of using these approaches for testing the performance of a system used as a gateway in a large industrial client/server transaction processing application is presented. Stress/Load Testing Simulator 10

11 In "Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study" by Vokolos and Weyuker [1], an examination of performance testing issues and description to an approach to address some of these issues for certain classes of software systems is addressed. This work is based on experience gathered while testing the performance of different industrial software systems. The paper classifies the objectives of performance testing and describes a technique for creating appropriate workloads needed for the testing. Also it looks at the role of requirements and specifications in successful performance testing. A case study is presented describing the results of some recent work that was done to test the performance of an existing system. The approach used drew on an earlier technique that was developed for performance testing, as described in "Deriving Workloads for Performance Testing" [2] paper. A. Avritzer and E.J. Weyuker paper: "Investigating Metrics for Architectural Assessment" has examined a substantial amount of data collected during architecture reviews. Their goal was to develop metrics to predict the likely success or failure of a project [4]. The goal of an architecture review is to assure that the architecture is complete and to help identify likely problems that can be easily addressed at this stage and may have negative impact on the software system that is ultimately produced and categorized according to the cause of the problem and the severity, but would be much more expensive to correct once implementation is in progress. Performance problems identified at this stage might include such things as the lack of performance estimates, the failure to have proposed plans for data collection, or the lack of a performance budget. Developing comprehensive performance testing strategies is essential. The issues that have to be addressed when doing performance testing differ in a number of ways from the issues that must be addressed when doing typical functionality testing. Briand [1], Labiche and Shousha in their paper "Stress Testing Real-Time Systems with Genetic Algorithms" suggest that during development, some performance analysis be performed to determine whether designed tasks will likely meet their deadlines. At design time, such performance analysis requires that task execution times be estimated (e.g., based on the expected number of lines of code) since the whole software is not fully developed they propose a Real- Time Scheduling Theory that helps designers determine whether a group of tasks whose individual execution times have been estimated, will meet their deadlines. Korel [2] and Al-Yami in their paper "Automated Regression Test Generation" talk about automating the regression testing process which involves testing the modified program with test Stress/Load Testing Simulator 11

12 cases in order to establish the confidence that the program will perform according to the (possibly modified) specification. During software maintenance the existing software is modified. There are three types of software maintenance: corrective maintenance is performed to correct an error that has been discovered in some part of the software. Adaptive maintenance is usually performed when the software is modified to ensure its compatibility with the new environment in which it will operate. Perfective maintenance is performed to add new features to the software or to improve performance of the software. Also Korel and Al-Yami proposed a regression test generation method to automatically generate test data that causes a modified program under test to behave incorrectly. Yang [3] and Pollock in there paper "Towards A Structural Load Testing Tool" introduce a testing tool for structural load testing which takes a program as input, and automatically determines whether that program needs to be load tested, and if so, automatically generates test data for structural load testing of the program. There exist many structural testing methods with the main goal of generating test data for executing all statements, branches, definition-use pairs, or paths of a program at least once to discover load sensitive faults, which are programming errors with which the program executes successfully if executed for a short time or with a small workload, but causes the program to fail when it is executed under a heavy load or over a long period of time. The paper "A Model-Based Approach for Testing the Performance of Web Applications" Shams, Krishnamurthy and Far studies the performance of Web-based systems. Poor performance can adversely impact the profitability of enterprises that rely on the web applications. As a result, effective performance testing techniques are essential for understanding whether a web-based system will meet its performance objectives when deployed in the real world. The workload of a web-based system has to be characterized in terms of sessions; a session being a sequence of inter-dependent requests submitted by a single user. Dependencies arise because some requests depend on the responses of earlier requests in a session. To exercise application functions in a representative manner, these dependencies should be reflected in the synthetic workloads used to test web-based systems. This makes performance testing a challenge for these systems. A modelbased approach to address this problem is proposed. The proposed model uses an application model that captures the dependencies for a Web-based system under study. Also the recent growth of the Internet and its increased heterogeneity has introduced new failure modes and added complexity to protocol design and testing. In addition, the advent of multicast Stress/Load Testing Simulator 12 [4] by

13 applications has introduced new challenges of qualitatively different nature than the traditional point-to-point protocols. A method for automatic synthesis of worst and best case scenarios for these new protocols was addressed by Helmy [5], Gupta and Estrin in "The STRESS Method for Boundary-Point Performance Analysis of End-to-End Multicast Timer-Suppression Mechanisms". The algorithm uses timing semantics to handle end-to-end delays and address performance criteria. Also The paper "Traffic-aware Stress Testing of Distributed Systems Based on UML Models" [6] by Garousi, Briand and Labiche studies a stress test methodology aimed at increasing chances of discovering faults related to network traffic in distributed systems. The paper "An Intentional Approach to the Specification of Test Cases for Database [7] Applications" by Willmor and Embury proposes a new approach in which the database states required for testing are specified intentionally, as constrained queries, that can be used to prepare the database for testing automatically. From our search we conclude that software testing is an art. Most of the testing methods and practices are not very different from 20 years ago [17], [20]. It is nowhere near maturity, although there are many tools and techniques available to use. Good testing requires a tester's creativity, experience and intuition, with proper techniques. Plus testing is more than just debugging. Testing is not only used to locate defects and correct them. It is also used in validation, verification process, and reliability issues that include performance measurements. Stress/Load Testing Simulator 13

14 Chapter A Sample of Existing Tools 1. Proxy Sniffer Proxy Sniffer is a web load/stress testing tool that measures the response time and the stability of web applications under real load conditions, by simulating hundreds or thousands of web users. This product is especially suited for testing the performance of: E-Banking applications. Web Portals. E-Commerce shops. 2. Apache Jmeter Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. 3. Hammer-Head2 Hammerhead-2 is a stress testing tool designed to test out any web server and web site. It can initiate multiple connections from IP aliases and simulated numerous (256+) users at any given time. It also can give numerous options for trying to create problems with a web site (so the developer can fix them). Hammerhead can destroy a web site very quickly so it should only be used to generate heavy loads on local web sites for testing purposes. 4. Jcrawler JCrawler is an open-source Stress-Testing Tool for web-applications. It comes with the crawling/exploratory feature. I tester can give JCrawler a set of starting URLs and it will begin Stress/Load Testing Simulator 14

15 crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. The load parameters (hits/sec) are configurable. 5. Mercury LoadRunner Mercury LoadRunner prevents costly performance problems in production by detecting bottlenecks before a new system or upgrade is deployed. A tester can verify that new or upgraded applications will deliver intended business outcomes before go-live, preventing over-spending on hardware and infrastructure. With LoadRunner, a tester can: Obtain an accurate picture of end-to-end system performance. Verify that new or upgraded applications meet specified performance requirements. Identify and eliminate performance bottlenecks during the development lifecycle. 6. Peassler Peassler is a web server performance, load, and stress-test tool, it is a powerful HTTPclient/server test application designed to pinpoint critical performance issues in a web site or web server that may prevent optimal experience for a site's visitors. By simulating the HTTP requests generated by hundreds or even thousands of simultaneous users, a tester can test a web server performance under normal and excessive loads to ensure that critical information and services are available at speeds the end-users expect. 2.2 Comparing the Tools Features Tool1: Proxy Sniffer Tool3: Hammer-Head2 Tool5: Mercury LoadRunner Tool2: Apache JMeter Too4: JCrawler Tool6: Peassler Stress/Load Testing Simulator 15

16 Feature Tool1 Tool2 Tool3 Tool4 Tool5 Tool6 Priority My System Justification 1. GUI support High 2. The generation of summary reports 3. Customizable PDF, Word reports 4. PDF Comparison Report (response-time comparisons between several test-runs) 5. The support of performance tests High X X X X Low X X X X X Medium High Due to limited time constraints 6. The support of load tests High 7. The support of stress tests High 8. The support of ramp tests X X X Medium 9. The support of regression testing 10. The generation of result diagrams X X High High 11. Test Scenarios generation Medium 12. Recording & playing back Medium 13. Simulating large numbers of simultaneous users 14. Specifying no. of current users 15. Simulate user load with minimum intrusion (shouldn't produce new problems). 16. Specifying Max. loops per user High High High Medium 17. The use of data feeders X X X X X X High 18. Data export of load test details (from files, excel sheets) Medium 19. Platform independent High Stress/Load Testing Simulator 16

17 20. The support of data driven testing ability to read and execute the same script Medium over a multiple set of data. 21. The support of test management activities like Test Preparation and Planning, Test Execution, Medium Test Results (Provide templates for test script preparation). 22. Maintain logs of all different runs and Medium executions. 23. Is the tool capable of integrating and interfacing High with different systems? 24. The ability to perform end to end testing with all interfaces and different protocols. Medium X 25. Does the tool support protocols such as TCP/IP, Low X SNA, X.25 and IFX 26. Can send last transaction status and details along with the new transaction X X X X Low X requests. 27. Touch screen support? X X X X X X Low X 28. Specific command reject message support? X X X X X Low X 29. Reversal transaction support? X X X X X Low X 30. The ability to encrypt data. Low X Due to limited time constraints we cannot include all protocols and different interfaces Due to limited time constraints Due to limited time constraints Inapplicable with current requirements Due to limited time constraints Due to limited time constraints Inapplicable with current Stress/Load Testing Simulator 17

18 requirements 31. Does it allow printing the transactions and requests? 32. Tool compliance with different formats and messages. X X X X X Low X Low X Inapplicable with current requirements Due to limited time constraints Stress/Load Testing Simulator 18

19 2.3 SLTS Requirements Specification The focus of this work will be the development of a Stress/Load Testing Simulator (SLTS). SLTS will provide the user with all the necessary means in which to adequately test his/her environment. 1. SLTS will be a tool that generates realistic heavy loads that typically results from the activity of many users. 2. A virtual environment is going to be built. 3. The virtual environment will have the capability to include as many users as needed to stress the system. 4. The virtual users' queries and resource utilization information from the different system servers, e.g. web server, application server and database server that are currently undergoing testing will be gathered and saved. 5. The testing results are going to be analyzed. 6. The analysis should provide precise performance measurements that will be collected during stress/load tests for further examination and analysis. 7. SLTS will provide a graphical real-time view graph of the system undergoing the stress/load test. 8. A summary report is going to be generated. 9. Different types of diagrams are going to be generated. 10. SLTS will operate within a three-tier client/server architecture. 11. The stress/load-testing concept and SLTS is going to be implemented using JAVA for multithreading and for making SLTS platform independent. Stress/Load Testing Simulator 19

20 2.4 SLTS High-Level Architecture Description Figure4. SLTS Architecture Stress/Load Testing Simulator 20

21 Chapter The Stress/Load Testing Process The stress/load testing process consists of several phases: 1. Plan the stress/load test: Define performance testing reqirments, ex: number of concurent users, transcations to ne performed and required response time. 2. Create the testing scripts: Capture the end-user activties into automated scripts. 3. Define a scenario. 4. Run the defined scenario. 5. Analyze the results. Terminology Figure5: The stress/load testing process 1. Vuser: Virtual users (Vusers) emulate the steps of real users. The steps that Vusers perform should be recorded in a Vuser script. 2. Vuser Scripts: The actions the Vusers performs during the scenario are described in a Vuser script. 3. Scenario: A scenario is a file that defines the scripts to execute, the number of Vusers to run, the goals of the test, the computer that will host the Vusers, and the conditions under which the stress/load test runs. Stress/Load Testing Simulator 21

22 4. Controller: The controller is the machine chosen to manage and maintain the scenarios and allows controlling all the Vusers of the scenario from a single location. All the elements of a scenario should be defined in the controller. 5. Transactions: To measure the performance of the server, a transaction is defined. A transaction is an end-to-end measurement of time elapsed in the execution of one or more steps in a business process that the tester is interested in measuring. 3.2 Stress/Load-Test Planning Stress/Load-Test planning is a three step process that includes: 1. Analyzing the application. 2. Defining Stress/Load testing objectives. 3. Planning Stress/Load testing implementation. 1. Analyzing the application: The first step to stress/load test planning is analyzing the application under test. The tester should become thoroughly familiar with the hardware and software components, the system configuration, and the typical usage model. This analysis ensures that the testing environment the tester creates using SLTS will accurately reflect the environment and configuration of the application under test, so the steps to be done can be summarized to: 1. Identifying system components. 2. Describing the system configuration. 3. Analyzing the usage model. 4. Task distribution. 1.1 Identifying System Components Draw a schematic diagram to illustrate the structure of the application. If possible, extract a schematic diagram from existing documentation. If the application under test is part of a larger network system, the tester should identify the component of the system to be tested. The diagram should include all system components, such as client machines, network, middleware, and servers. Stress/Load Testing Simulator 22

23 The following diagram illustrates an online system that is accessed by many web users. The web users each connect to the same database to get special data. The customers connect to the database server through the web, using multiple browsers. Figure6: An online system accessed by many web users 1.2 Describing the System Configuration Enhance the schematic diagram with more specific details. Describe the configuration of each system component. The tester should be able to answer the following questions: 1. How many users are anticipated to connect to the system? 2. What is the application client's machine configuration (hardware, memory, operating system, software, development tool, and so on)? 3. What types of database and Web servers are used (hardware, database type, operating system, file server, and so on)? 4. How does the server communicate with the application client? 5. What is the middleware configuration and application server between the front-end client and back-end server? 6. What other network components may affect response time (modems and so on)? 7. What is the throughput of the communications devices? How many concurrent users can each device handle? 1.3 Analyzing the Usage Model Define how the system is typically used, and decide which functions are important to test. Consider who uses the system, the number of each type of user, and each user's common tasks. In addition, consider any background load that might affect the system response time. Stress/Load Testing Simulator 23

24 For example, suppose 200 employees log on to the accounting system every morning, and the same office network has a constant background load of 50 users performing various word processing and printing tasks. The tester could create a SLTS scenario with 200 virtual users signing in to the accounting database, and check the server response time. To check how background load affects the response time, the tester could run the scenario on a network where he can simulate the load of employees performing word processing and printing activities. 1.4 Task Distribution In addition to defining the common user tasks, the distribution of these tasks needs to be examined. For example, suppose that an organization uses a central DB to serve clients across many states and time zones. The 250 application clients are located in two different time zones, all connecting to the same web server. There are 150 in Chicago and 100 in Detroit. Each begins their business day at 9:00 AM, but since they are in different time zones, there should never be more than 150 users signing in at any given time. The tester can analyze task distribution to determine when there is peak DB activity, and which activities typically occur during peak load time. 2. Defining Stress/Load testing objectives: Before beginning testing, the tester should define exactly what he wants to accomplish. The following table shows common application testing objectives that our proposed SLTS helps test: Objective Measuring end-user response time Defining optimal hardware configuration Checking reliability Checking hardware or software upgrades Evaluating new products Measuring system capacity Answers the Question How long does it take to complete a task? Which hardware configuration provides the best performance? How hard or long can the system work without errors or failures? Or How stable is the system under a heavy work load? How does the upgrade affect performance or reliability? Which server hardware or software should you choose? or What is the best server for 100 users? How much load can the system handle without significant Stress/Load Testing Simulator 24

25 performance degradation? Identifying bottlenecks Accept the system Which element is slowing down response time? Or What is the cause of degradation in performance? Is the system stable enough to go live? Or The system is not stable due to perfomance issues and an immediate action needs to be done? Also it is important to decide when to test. Stress/load testing is necessary throughout the product life cycle. The following table illustrates what types of tests are relevant for each phase of the product life cycle: Planning and Design Development Deployment Production Evolution Evaluate new products Measure response time Check reliability Measure response time Check HW or SW upgrades Measure response time Check optimal hardware configuration Measure response time Identify bottlenecks Measure system capacity Check HW or SW upgrades Measure system capacity Check reliability 3. Planning Stress/Load testing implementation: The next step is to decide how to use SLTS to achieve the testing goals, a tester should: 1. Define the scope of performance measurements. 2. Define virtual user activities. 3. Select the virtual users. 4. Choose the testing hardware/software. 3.1 Defining the Scope of Performance Measurements A tester can use our SLTS to measure response time at different points in the application. The tester should determine where to run the virtual users and which virtual users to run according to the test objectives: Measuring end-to-end response time: A tester can measure the response time that a typical user experiences by running a GUI Vuser. GUI Vusers emulate real users by submitting input to and receiving output from the client Stress/Load Testing Simulator 25

26 application. A tester can run the GUI Vusers at the front end to measure the response time across the entire network, including a terminal emulator or GUI front end, network, and server. Figure7: Measuring end-to-end response time Measuring network and server response times: A tester can measure network and server response time, excluding response time of the GUI front end, by running Vusers on the client machine. Virtual users emulate client calls to the server without the user interface. When a tester runs many virtual users from the client machine, the tester can measure how the load affects network and server response time. Figure8: Measuring N/W & server response times Measuring GUI response time: A tester can determine how the client application interface affects response time by subtracting the previous two measurements: GUI response time = (end-to-end) (network and server) Figure9: Measuring GUI response time Stress/Load Testing Simulator 26

27 Measuring server response time: A tester can measure the time it takes for the server to respond to a request without going across the network. When a tester runs virtual users on a machine directly connected to the server, he can measure server performance. Figure10: Measuring server response time Measuring middleware-to-server response time: A tester can measure response time from the server to middleware if he has access to the middleware and it's API (Application Programmable Interface). He can create virtual users with the middleware API and measure the middleware-server performance. Figure11: Measuring MW-to-server response time 3.2 Defining Virtual User (Vuser) Activities A tester should create Vuser scripts based on his analysis of Vuser types, their typical tasks and his test objectives. Since Vusers emulate the actions of a typical end-user, the Vuser scripts should include the typical end-user tasks. For example, to emulate an online accounting client, a tester should create a Vuser script that performs typical online accounting tasks. He would browse the pages that he normally visits to perform his tasks. The tester decides which tasks to measure based on his testing objectives and define transactions for these tasks. Transactions measure the time that it takes for the server to respond to tasks submitted by Vusers (end-to-end time). For example, to check the response time of a hospital web server supplying a search for a patients file, define a transaction for this task in the Vuser script. And to load the web server a tester can emulate peak activity by defining for example 100 users simultaneously updating or retrieving the patients files information. Stress/Load Testing Simulator 27

28 3.3 Selecting Virtual Users Before deciding on the hardware configuration to use for testing, a tester should determine the number and type of Vusers required. To decide how many Vusers and which types to run, the tester should look at the typical usage model, combined with the testing objectives. 3.4 Choosing Testing Hardware/Software The hardware and software should be powerful and fast enough to emulate the required number of virtual users. 3.3 Other Phases of the Stress/Load Testing Process After completing the planning phase, which defines the performance testing reqirments, for example the number of concurent users, transcations to be performed and the required response time the tester can move to the second phase which is creating the testing scripts to capture the end-user activties into automated scripts. After that the tester can go to phase three where he defines the testing scenarios. The next phase can be done where he runs the defined scenario. Finally the last phase where he can analyze the results. Figure12: The stress/load testing process Stress/Load Testing Simulator 28

29 Chapter Metrics and Measurements During Performance Testing, a tester should track the following metrics for each server within the topology: 1. SLTS Counter Metrics For each server utilized (including application servers, database servers, etc.), a tester can obtain the following metrics from monitoring the test: 1. Server Level Metrics: a. CPU Metrics i. % User CPU ii. % Privileged CPU (System CPU) iii. % Total CPU b. System Metrics i. System Calls/sec ii. Context Switches/sec c. Memory Metrics i. Pages /sec 2. SLTS Response Times: Monitor average transaction response times for each state. 3. Transactions per Seconds (TPS): Monitor average transactions per sec (Hits/Sec) for each state. 4. Throughput: Monitor average throughput (KB) for each state. 5. Error Rate: Monitor errors in detail for each state. Stress/Load Testing Simulator 29

30 References [1] [2] [3] [4] [5] [6] Vokolos and Weyuker, "Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study", IEEE transactions on software engineering, vol. 26, no. 12, December A. Avritzer and E.J. Weyuker, "Deriving Workloads for Performance Testing" Software- Practice and Experience, vol. 26, no. 6, pp , June A. Avritzer and E.J. Weyuker, "Monitoring Smoothly Degrading Systems for Increased Dependability", Empirical Software Eng. J., vol. 2, no. 1, pp , A. Avritzer and E.J. Weyuker, "Investigating Metrics for Architectural Assessment", Proc. IEEE Fifth Int'l Conf. Software Metrics (Metrics '98), pp. 4-10, Nov Beizer, Boris, Black-box Testing: techniques for functional testing of software and systems, New York: John Wiley & Sons, D. Ferrari, G. Serazzi, and A. Zeigner, Measurement and Tuning of Computer Systems. Prentice Hall, [7] E. Kit, Software Testing in the Real World. New York: Addison-Wesley, [8] M. Loukides, System Performance Tuning. O'Reilly & Associates, [9] B. Marick, The Craft of Software Testing. Englewood Cliffs, N.J.Prentice Hall, [10] C.U. Smith, Performance Engineering of Software Systems. New York, Addison-Wesley, [11] [12] F.I. Vokolos and E.J. Weyuker, "Performance Testing of Software Systems", Proc. ACM, Proceedings of the first international workshop on Software and performance, Oct 1998, Pages Victor R. Basili, Richard W. Selby, Jr. "Comparing the Effectiveness of Software Testing Strategies", Technical Report, Department of Computer Science, University of Maryland, College Park, [13] Boris Beizer, Software Testing Techniques. Second edition [14] Hetzel, William C., The Complete Guide to Software Testing, 2nd edition, Wellesley, Mass [15] William E. Howden. Functional program Testing and Analysis. McGraw-Hill, [16] Michael R. Lyu, Handbook of Software Reliability Engineering. McGraw-Hill publishing, [17] Myers, Glenford J., The art of software testing, Wiley [18] Software-reliability-engineered testing practice (tutorial); John D. Musa; Proceedings of the 1997 international conference on Software engineering, 1997, Pages Stress/Load Testing Simulator 30

31 [19] [20] [21] [22] [23] [24] Norman Parrington and Marc Roper, Understanding Software Testing, Published by John Willey & Sons, Progress toward automated software testing; Richard A. DeMillo; Proceedings of the 13 th international conference on Software Engineering, 1991, Page 180. Dick Hamlet; Foundations of software testing: dependability theory; Proceedings of the second ACM SIGSOFT symposium on Foundations of software engineering, 1994, Pages Smith, C. U. Performance Engineering of Software Systems. Addison-Wesley, Yang, M.C.K.; Chao, A. Reliability-estimation and stopping-rules for software testing, based on repeated appearances of bugs; IEEE Transactions on Reliability, vol.44, no.2, p , Joe W. Duran, Simeon C. Ntafos, "An Evaluation of Random Testing", IEEE Transactions on Software Engineering, Vol. SE-10, No. 4, July 1984, pp [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] Lionel C. Briand, Yvan Labiche, Marwa Shousha,June 2005, Proceedings of the 2005 conference on Genetic and evolutionary computation GECCO '05, ACM Press. Cheer-Sun D. Yang, Lori L. Pollock,May 1996, ACM SIGSOFT Software Engineering Notes, Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis ISSTA '96, Volume 21 Issue 3, ACM Press. Mahnaz Shams, Diwakar Krishnamurthy, Behrouz Far, November 2006, Proceedings of the 3rd international workshop on Software quality assurance SOQUA '06, ACM Press. Bogdan Korel, Ali M. Al-Yami,March 1998,ACM SIGSOFT Software Engineering Notes, Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis ISSTA '98, Volume 23 Issue 2, ACM Press. Ahmed Helmy, Sandeep Gupta, Deborah Estrin, February 2004, IEEE/ACM Transactions on Networking (TON), Volume 12 Issue 1, IEEE Press. Vahid Garousi, Lionel C. Briand, Yvan Labiche,May 2006, Proceeding of the 28th international conference on Software engineering ICSE '06, ACM Press. David Willmor, Suzanne M. Embury,May 2006, Proceeding of the 28th international conference on Software engineering ICSE '06, ACM Press. [36] Stress/Load Testing Simulator 31

32 [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] test-tools.technologyevaluation.com [49] [50] [52] [52] [53] [54] [55] W1IoCFRJxMAodO2ijgg [56] [57] Stress/Load Testing Simulator 32