Performance Testing Tools: A Comparative Analysis Shagun Bhardwaj Research Scholar Computer Science department Himachal Pradesh University Shimla Dr. Aman Kumar Sharma Associate Professor Computer Science department Himachal Pradesh University Shimla ABSTRACT Web Applications forms the important component of World Wide Web. They execute on web servers and end users access these applications through web browsers. They are critical for the success of e-business so there performance needs to be satisfactory for its users. Web users avoid waiting for long to get response of their request therefore we need to test a website for its performance. Testing web application performance involves testing its, throughput, responsiveness, reliability and scalability. Performance parameters are response time, latency and throughput. Testing tools help to achieve this purpose. There are many performance testing tools in a market. The comparative study of these tools helps in their promotion and usage. In this research comparative study of three performance testing tools is done on the basis of their response times. Keywords Software Testing, Testing Tools, Performance Testing, Web Application, Response Time. 1. INTRODUCTION 1.1 Software testing: Software testing is an important phase of software development life cycle. It is an important way of evaluate the software to test its quality [6]. It is the process of identifying and then correcting any ambiguity in a behavior of software from its actual expected specifications. Testing is executing a system in order to identify any kind of errors or missing requirements from the actual desired requirements. System can be tested for its success based on following parameters, functionality and performance [5]. Functionality Testing is a black box testing which aims at testing the software against its functional requirements [6]. It is a critical assessment for software quality control which is must for any software application as it is customer oriented. Customer oriented applications totally depend on customer satisfaction for their success. Performance testing refers to test a system against its non-functional requirements. 1.2 Performance testing: In today s scenario where there are many websites on internet it becomes necessary to test them for performance before launching. Performance testing is to check whether system meets its non-functional requirements identified in Software Requirement Specification (SRS) document [2] or not. Performance Testing is done to determine the response time and throughput of any web application [5]. For a website performance tests can check for its speed, reliability and load handling capacity. There are different types of performance tests: Stress test, load test, strength tests and volume test [4]. The performance testing involves monitoring and recording the performance levels during regular, low and peak hours for any website [5]. Performance testing tools are used to determine the time required to perform a task by the system. Testing tools simulate load of multiple virtual users at the same time on a site from a single client machine. Several issues have been observed related to tools when conducting performance testing such as tool s compatibility with the software under test, its installation, setup, flexibility in 100
testing[4]. In this study, three different performance testing tools are used for testing performance of the same web site and results are generated for different performance parameters at same time of the day. Similar test is performed at three different timings of the day and results are recorded to check any anomaly in the behavior of the tools. These tools are further compared and analyzed on the basis of response time. 1.3 Web application: It is software that runs in a browser and multiple users can access them simultaneously. A web application relieves the developer from the responsibility of building a client for a specific type of computer or a specific operating system as the client runs in a web browser. Therefore web application can run on any machine, running any operating system with a browser supporting the particular application [3].They form an important component of today s e-commerce business and needed to be tested thoroughly for its success on functional as well as non-functional aspects. Performance testing is one such aspect. This paper consists of 5 sections: Section 1 forms the basis of the study, Section 2 gives an overview of testing tools considered for study, Section 3 gives comparative study of the tools, Section 4 gives the result and discussion and Section 5 concludes the study along with scope for future work. 2. OVERVIEW OF TESTING TOOLS Testing tools are used to make testing easier and help simulate test environment for a system under testing. They automate testing process and make it easier. Manual testing is costlier, requires too much effort and is very time consuming. While automated testing with the help of tool reduces the cost, time and effort required. There are different types of testing tools like functional testing tools, black-box testing tools and white- box testing tools, bug tracking tools, performance testing tools and many others. Performance testing tools are used for different types of performance testing like load test, stress test, volume test and strength test. Testing tool enables the testers to create, manage and execute test in a particular environment, maintained for specific test for a particular application [2]. These tools can either be open source software or proprietary tools. For this research, three open source performance testing tools ApacheJmeter, Httprider and Fwptt have been selected. 2.1 ApacheJmeter: ApacheJmeter [10] is developed by Apache Software Foundation (ASF). It is open source performance testing tool. It is a Java desktop application with a graphical user interface can run on any environment / workstation accepting a Java virtual machine, for example: Windows, Linux and Mac. It can be used as a load testing tool for analyzing and measuring the performance of a variety of services mainly web applications [7]. It can also be used for unit testing for Java applications, LDAP, FTP, Web services, HTTP, generic TCP connections and Operating System Native processes. It is also used for functional testing. It can be used to simulate a heavy load on a server by generating multiple user threads at the same time to test its strength or to analyze overall performance under different load types. It also supports recoding browser session through proxy server and replays it to give different performance parameters like response time, throughput, latency, response bytes and load time. It also gives different representations of the results either as a tree or table or graphs. These views are also simultaneously available for use. Test plans can be stored in XML format and can be reused. 2.2 HttpRider: HttpRider [9] is a lightweight, open source testing tool for performance and stress testing of web sites. It is written in c#. It records http scenarios by creating http requests using Fiddler proxy and can create multiple user threads. It helps user to customize automation parameters 101
like query string and post parameters. It binds them to a customized data source in order to mimic non-identical requests. It records all the requests generated through the browser and then replay it with multiple concurrent users. Also requests can be edited as the convenience of the user. It also enable user to save scenarios to the disk. 2.3 Fwptt: Fast Web Performance Testing tool (FWPTT) program for performance testing of web applications [8]. It can record normal as well as Ajax requests. It enables the user to record browsing actions. It does so with the help of proxy server. After the recording, user can save the recorded data into an xml format for later use or can generate a c# class that will call all the http requests that the user has already recorded. Using the test runner the user will be able to run the code from this c# testing class. Because it uses a proxy server user can record browsing actions using any browser like IE, Firefox Opera and others. Also because the actual test is a C# class which can be modified. The class has functions for handling the query/post params which can be used to easily add more or delete some of the existing ones. It does not have any graphical viewer. 3. COMPARATIVE STUDY OF THE TOOLS This section represents the comparison of three performance testing tools ApacheJmeter, httprider, and fwptt along with their observed results. The comparison can help the researcher to determine the efficiency of test tool and use them according to their requirements. Website www.orbitz.com is tested for its performance using these tools on different parameters like response time, throughput and latency. Firstly the script is recorded with help of all these tools, then it is played back and results are recorded. 3.1 Environment used: All the tools are executed on an Intel Core i2, 2.10 GHz processor machine with 2GB RAM, Microsoft Windows 8 with the internet speed 100mbps. The comparison is made between three tools with the same script recorded with the tool itself. Testing of the tools requires it includes installation, setting up test environment, then selecting the parameters and analytical survey. The sample website www.orbitz.com is tested on the respective configured tools. 3.2 Approach Followed: All tests were conducted at the same instance of time at a same network speed. All the tools are executed simultaneously to record the requests through the browser with the help of a proxy server. This recording will act as a script for tests. The same script is run at three different times of a day to see how tool behaves and what differences are seen in the test results. These tests are performed on the same site which acts as an input to the tool. When the tests are performed, different parameters like response time, throughput and latency are retrieved. The performance of the tool was evaluated on the basis of performance parameter called response time. According to the response time best tool is selected from them. Table 1.Presents the comparison of the tools based on their technical requirements S.no Tool name Version used Programming OS Requirements language Support 1 ApacheJmeter ApacheJmeter 2.9 Java Windows, Linux, Mac Java 1.4 or higher virtual machine 2 Httprider Httprider 1.0 Asp.net Windows.net framework 3 Fwptt Fwptt 0.8 Asp.net Windows.net framework 102
These three tools vary from each other based on the technology they are developed in, the operating system in which they are supported and their requirements which need to be fulfilled before installation. ApacheJmeter is developed in Java while fwptt and httprider are developed in.net. ApacheJmeter can run on Windows, Linux and Mac while fwptt and httprider can run on windows. Table 2. Presents the values of different parameters as obtained by the tools at 1100 hrs: S.no Tool name Response Throughput Latency time(ms) 1 ApacheJmeter 2648 22.7/min 1295 2 Fwptt 9218.78 - - 3 Httprider 1583.34 - - In Table 2 results of a test conducted at 1100 hrs on 25 th march 2015 are presented. It can be observed from Table 2 that for httprider response time is minimum i.e. 1583.34 ms while fwptt behaves entirely different with response time of 9218.78 ms as compared to ApacheJmeter and httprider. While ApacheJmeter gives three parameters i.e. Response time, throughput and latency, fwptt and httprider only gives response time. Table 3. Presents the values of different parameters as obtained by the tools at 1500 hrs: S.no Tool name Response Throughput Latency time(ms) 1 ApacheJmeter 2100 28.6/min 832 2 Fwptt 9060.78 - - 3 Httprider 1161.56 - - Similar behavior of fwptt is seen when the test is performed at 1500 hrs on 25 th march 2015, again httprider gives the minimum response time 1161.56 ms while fwptt gives maximum response time of 9060.78 ms. Therefore it can be observed from table 2 and table 3 that httprider performs best out of three. Response time for all three tools is decreased for a test conducted at 1500 hrs as compared to the test conducted at 1100 hrs. Table 4. Presents the values of different parameters as obtained by the tools at 1800 hrs: S.no Tool name Response Throughput latency time(ms) 1 ApacheJmeter 2042 29.4/min 737 2 Fwptt 9843.2 - - 3 Httprider 1593.57 - - The same test is again conducted at 1800 hrs on 25 th march 2015 and results are recorded in Table 4. In this test also similar behavior can be observed. Here also fwptt gives maximum response time while httprider gives minimum response time and behaves best among others. In all three tests, same recording done with the help of proxy server is used as input to get the results. From Table 2, Table 3 and Table 4 it can be concluded that all tools are different in their internal structures and have used different algorithms to calculate these parameters which have resulted in the varying results. Also httprider has best performance in all three cases while fwptt performs worst with maximum response time. 103
4. RESULTS AND DISCUSSION Table 1 presents the comparison according to technical requirements of the tools, Table 2, Table 3 and Table 4 shows the results taken at 1100 hrs, 1500 hrs and 1800 hrs respectively. As it can be seen from these results that httprider has given minimum response times 1583.34, 1161.56 and 1593.57 but only gives one parameter i.e. response time. ApacheJmeter gives response times as 2648, 2100 and 2042. It also gives other parameters like latency and throughput which httprider and fwptt are unable to provide. Contrary to all others fwptt gives maximum response times as 9218.78, 9060.78 and 9843.2 which totally vary from ApacheJmeter and httprider. From Table 2, Table 3 and Table 4 average response time for a website www.orbitz.com can be calculated and is presented below in a table. Table 5.Presents average response time for website ApacheJmeter httprider fwptt 2263.33ms 1446.15ms 9374.25ms It can be observed that for fwptt response time is maximum while httprider gives minimum response time. Hence it can be concluded that httprider is best performing tool in terms of response time while fwptt is worst performing tool with maximum average response time. 5. CONCLUSIONS AND FUTURE WORK In this study performance testing tools are compared on the basis of response time parameter and an average response time is calculated for a website. From observed results it can be concluded that each tool has different internal structure and different algorithm to calculate these parameters which may have resulted in the difference observed in the results which form the basis of a comparison. From this study it is observed that fwptt has performed worst with maximum average response time of 9374.25 ms while httprider is the best performing tool with average response time of 1446.15 ms. This study can be extended with more tools to get more accurate conclusions and will further provide more appropriate analysis. Further we can improvise fwptt and httprider tools to give other parameters like latency and throughput. Also ApacheJmeter, fwptt and httprider can be used for load testing of web sites with multiple virtual users and load handling capacity of a website can be tested. REFERENCES 1. Arora A. and Sinha M., Web application Testing:A Review on Techniques, Tools and State of Art, International Journal of Science and Engineering Research, 2012. 2. Dr. Manju Kaushik, Pratibha Fageria, Performance testing tools: A comparative study, International Journal of Innovative Science, Engineering & Technology,2014. 3. Dr. S. M. Afroz, N. Elezabeth Rani and N. Indira Priyadarshini, Web application:the study on comparing software testing tools, International Journal of Computer Science and Telecommunications,2011. 4. Rina and Sanjay Tyagi, Comparative study of performance testing tools, International Journal of Advanced Research in Computer Science and Software Engineering, 2013. 5. S Sharmila and E. Ramadevi, Analysis of performance testing on web application, International Journal of Advanced Research in Computer and Communication Engineering, 2014. 6. Shivkumar Hasmukhrai Trivedi, Software testing techniques, International Journal of Advanced Research in Computer Science and Software Engineering, 2012. 104
7. Vaishali,Vinita Sharma and Bhawna, Performance Testing of On-line Recruitment web application Using JMeter Testing Tool, International Journal of Advance Research in Computer Science and Management Studies,2014. 8. www.fwptt.sourceforge.net, FWPTT, accessed on 25 January 2015 at 1100 hrs. 9. www.httprider.codeplex.com, HTTPRIDER, accessed on 18 January 2015 1500 hrs. 10. www.jakarta.apache.org/jmeter, APACHEJMETER, accessed on 22 January 2015 1300 hrs. 105