Automated Functional Testing of Web-based Applications

Automated Functional Testing of Web-based Applications Oliver Niese 1, Tiziana Margaria 1,2, Bernhard Steffen 2 1 METAFrame Technologies GmbH, Borussiastr. 112, 44149 Dortmund, Germany, {oniese,tmargaria}@metaframe.de 2 Chair of Programming Systems, University of Dortmund, Baroper Str. 301, 44221 Dortmund (Germany), {tiziana,steffen}@guiness.cs.uni-dortmund.de Keywords: Automated Functional Testing, Testing of Web-based Applications, Test-Tools, System- Level Test Environments Abstract Modern web-based applications are multitiered, distributed applications that typically run on heterogeneous platforms. Their correct operation depends increasingly on the interoperability of single software modules, rather than on their intrinsic algorithmic correctness. We present a scalable, integrated test methodology capable of granting adequate coverage of functional testing, yet still easy to use. It bases on our previous work on system level test of Computer Telephony Integrated applications, where our coordination-based coarse grain approach has been applied very successfully. The practicability of the approach is illustrated by testing complex role based functional features of our Online Conference Service, an advanced online management system for the organization of scientific conferences. 1 Dealing with Web-Based Applications Modern web-based applications are very complex multitiered, distributed applications that typically run on heterogeneous platforms. Their correct operation depends in steadily growing measure on the interoperability of single software modules, rather than on their intrinsic algorithmic correctness. This has a huge impact on the test and validation requirements: it is increasingly unrealistic to restrict the consideration to single units,

Client... Client Internet/ Intranet Test Coordinator Webserver Application Server Back-end Services Database Figure 1: Overview of the Test Setting e.g. the client side only, since complex subsystems affect each other. Instead, scalable, integrated test methodologies are needed that can handle the application in a whole. Figure 1 sketches a typical architecture of a web-based application. The browser, located on a client, plays the role of the classical application GUI and interacts with a Webserver. The Webserver itself communicates with an Application Server which builds dynamically the requested web pages through an interaction with a database and (several) Back-end services. Often staging servers, firewall solutions, load balancing and redundant architectures increase the speed and availability of the offered application, and increase the complexity of the considered test setting. So test methods are needed that can deal with this kind of complex distributed system. Due to the architecture of such a system it is obviously not sufficient to test the user interface only but it is of central importance to consider the server side as well (i.e. the web-server, application-server and the back-end services). This task exceeds today s commercial internet-testing tools, as marketed e.g. by Compuware [1] or Radview [9]. An adequate approach should on the one hand allow an easy, graphical design of test cases and on the other hand the execution of these complex test cases through a coordination of different, heterogeneous test tools. These test tools locally monitor and steer the behaviour of the software on the different clients/servers, which are seen as units under test. Depending on the considered application domain, different aspects of an application s functionality dominate the test purposes:

E-Commerce When testing E-Commerce applications (e.g. Web-shops) it is important to consider simple functional requirements like a user can check out the shopping cart after buying an item or security/administrative requirements like a user can be logged in only once. But also more complex tests must be performed, which take the whole distributed configuration of the service into account e.g. check the validity of a credit card. Here it is not sufficient anymore to perform tests that consider the client side only, since e.g. a faulty implementation of a back-end service may accept all credit cards without checking them correctly. (Safety critical) Web-services A good example for this special kind of Web-services is e.g. Online/Internet-Banking. It is of crucial importance that on the one hand the user-interface reacts as prescribed and that on the other hand the back-end services are interacting correctly with the clients, as in this scenario there are normally well established back-end services or legacy systems in use. In this special field of applications every fault (either functional errors or a system crash) can be very expensive in terms of indemnification, and, more importantly, of loss of trust. We propose an integrated test environment for web-based applications capable of addressing the distributed application in a whole and covering mostly the global functional aspects. To test these kind of complex systems, we add a Test Coordination layer driving the generation, execution, evaluation, and management of the system-level tests in the highly heterogeneous landscape, cf. figure 1. The coordination layer introduces the required flexibilization of the overall architecture of the test environment: it is a modular and open environment, so that diverse tools and units under test can be added at need. In the following we first give a rough overview of the test environment in Section 2. Afterwards we present in Section 3 our approach along a real life example: the test of a complex online conference service. Finally in Section 4 we draw some conclusions. 2 An Integrated Test Environment for the Web Complex subsystems affect each other in a variety of complex ways, so mastering today s testing scenarios for complex systems demands for an integrated, open and flexible approach to support the management of the overall test process, i.e. specification of tests, execution of tests and analysis of test results. To handle the structural complexity, our approach offers a coarse grained testing environment where test cases can be easily composed, test suites can be configured and initialized, critical consistency requirements including version compatibility and frame conditions for executability are easily formulated. The consistency of test cases is fully automatically enforced via lightweight formal methods. The heart of the environment is the Test Coordinator tool, built on top of METAFrame s Agent Building Center [10], which is a generic and flexible workflow management system.

In this application, we view system-level test cases of web-based application scenarios as executable workflows within the integrated test environment (which plays the role of an extended runtime environment). Building on the ABC s capabilities, test sequences can be graphically designed from palettes of subsystem-specific, generic test components, cf. figure 1(right). While the test components are developed by experts of the relative system, the test sequences (the actual workflows) can be designed graphically, at an intuitive level by application experts. A reporting tool concentrates the information coming from the observation points during execution of the test into reports. Through a CORBA/RMI-based implementation of the communication layer we are able to address and encapsulate a wide range of commercial test tools: this increases over time the reach and the capabilities of the resulting environment. This way the Test Coordinator executes integrated test cases by controlling several test tools, each managing its own subsystem. The extensibility of the environment by additional test tools is the key of the approach. So far the ITE has been successfully applied to the system level testing of complex system solutions like Web-based applications or CTI-Solutions [5, 6]. 3 Case Study: Testing the Online Conference Service We demonstrate the key features of the ITE on a real life example: the test of the Online Conference Service [2], a complex web-service that supports online the management of the scientific program of professional conferences. It proactively helps authors, Program Committee chairs, Program Committee members, and reviewers to cooperate efficiently during their collaborative handling of the composition of a conference program. The application provides a timely, transparent, and secure handling of the papers and of the related submission, review, report and decision management process. The online service is a powerful application, including in particular a role based access and rights management feature that is reconfigurable online, which accounts for a high flexibility and administration comfort, but which is responsible for potentially disruptive behaviour in connection with sensitive information. Because of the complexity of the underlying workflows (and of course therefore the complexity of the application itself) intensive testing is clearly unavoidable. The test process supported by the ITE is organized in the following main steps: 1. Identification of generic and reusable test blocks, 2. Graphical design of test cases, 3. Verification of the test cases against predefined properties, and finally 4. Execution of the test cases and reporting.

3.1 Identification of Generic and Reusable Test Blocks The first task when testing the conference service is to identify generic test actions which allow on the one hand to control the service via a browser and on the other hand to validate certain properties of the service state (e.g. via checking the current state of the GUI or via checking the entries of an underlying database). For each action a test block is prepared: a name and a class characterizing the block are specified and a set of formal parameters is defined to enable a more general usage of the block. In this way, for the Web-based application to be tested a library of test blocks has been incrementally defined. It includes test blocks representing and implementing, e.g. Common actions for the initialization of test tools, system components, test cases and general reporting functions, Common internet related actions e.g. starting/initialization of a browser, following a link, Application-specific actions: these are miscellaneous actions for operating a specific application via its graphical user interface, e.g., log-on/log-off of a user, or checking labels of GUI-elements. The library of test blocks grows dynamically whenever new actions are made available. 3.2 Graphical Design of Test Cases The design of test cases consists in the behaviour-oriented combination of test blocks. This combination is done graphically, i.e., icons representing test blocks are graphically stuck together to yield test graph structures that embody the test behaviour in terms of control. The test case shown in figure 2 tests wether an article, submitted by an author, is delegated correctly to a member of the program committee. From the submission to the start of the review process of an article at least three different roles are involved: the Author himself, the Conference Chair (PCChair), who must delegate the article to a member of the program committee, and finally the members of the program committee (PCMember), who are responsible for the reviews. So we need at least three concrete users to run this test, whereby each user is associated with a distinct role. At the beginning of the test case for every user a browser must be started to perform the required tasks. The single steps to be executed are here encapsulated in a macro, cf. Figure 2(left). Macros can be used as reusable atomic test actions in the design of new test cases. The submission workflow proceeds then as follows: 1. The Author submits an article.

(1) (2) (3) Figure 2: Example of a Test Case Definition, with Macros 2. If this operation has been successfully completed, the PCChair should delegate this newly submitted article to one of the program committee members. The test case continues in dependence of special conditions (which are checked in the two following test actions): (a) If the PCMember is a co-author of this article or has no access rights 1 to review this category of articles, the test case continues with step (2), where it will be checked that the PCMember has not yet been assigned the reviw of this paper (technically, the member is not in the current delegation list), (b) otherwise this PCMember is a legal potential reviewer for the paper, thus he must be in the delegation list of the PCChair. This will be checked in (3). 1 This information will be requested directly from the underlying database.

3. If the article can be delegated, in this case the PCChair delegates the review task for this article to the PCMember. 4. Finally after validating the corresponding result, i.e. that the PCMember has a new review task regarding this article, the test can be evaluated. Note that here it is e.g. insufficient to check just the number of articles in the task list in order to conclude a correct delegation: instead, the service must ensure that exactly the article submitted in step 1 is now in the task list. 5. The last action of every test is a special reset operation, which makes sure that the test environment is correctly reset (e.g. clean up the databases, log out every user), and that all the browser sessions on the clients are properly terminated. This test case is distributed, in the sense that it covers several subsystems involved in the application. In particular, test actions are executed on the clients marked with Author, PCChair and PCMember, as well as on the database and on the webserver in our test environment. Nevertheless the design of such a complex test is still easy and intuitive, since the tester (or test engineer) is not concerned with instantiation details like e.g. the concrete IP-Addresses of test clients. 3.3 Verification of the Test Cases In our environment, the design of test cases is constantly accompanied by online verification of the global correctness and consistency of the test cases control flow logic [7]. During the design phase, vital properties concerning the usage of parameters (local properties) and the interplay between the stimuli and verification actions of a test case (global properties) can be verified. Design decisions that conflict with the constraints and consistency conditions of the intended system are thus immediately detected. Systems developed with the Agent Building Center, like the Online Conference Service, enjoy a double bonus: the same constraints enforced at design time (detailed for this service e.g. in [11, 2]) can now be reused to validate the test cases, ensuring consistency of the test purposes with the original system specification. Local properties specified imperatively are used to check that the parameters are set correctly. Global properties concerning the interplay between arbitrarily distant test blocks of a test graph are expressed in a user-friendly specification language based on SLTL, the Semantic Linear-time Temporal Logic of [4], and are gathered in a constraint library accessed by the environment s model checker during verification. Typical additional constraints for the test of Web-based applications refer e.g. to the resource management: one can ensure that all used resources are correctly released after the execution of the test case 2, and this independently of the tests outcome. 2 E.g. every log-on for an user must be followed by a log-off for this user.

Figure 3: Test Execution: A Successful Execution Path and the Users Views If the model checker detects an inconsistency, a plain text explanation of the violated constraint appears. In addition, test blocks violating a local property as well as paths violating a global property are marked. 3.4 Execution of the Test Cases and Reporting Test cases can be executed immediately in the Test Coordinator by means of ABC s tracer. Starting at a dedicated test block of a test graph the tracer proceeds from test block to test block. The actions represented by a test block are performed, i.e., stimuli and inspection requests are sent to the corresponding system s component, responses are received, evaluated, and the evaluation result is used to select the relevant next test block. Figure 3 illustrates these features on a concrete snapshot of a test session. Here, an Author

submits an article that should be reviewed by a member of the program committee. Even this relatively simple scenario shows the potential complexity of such tests. In general, the implementation of this execution scheme requires two activities during set-up of the ITE: 1. The actions referenced via test blocks have to be implemented by means of test tools. This task is performed by testers which are familiar with test tools, their handling and programming. For each action, the tester has to specify instructions to be executed by the test tool determined to support the specific action, e.g., via recording GUI-activities. In case of the conference service the Rational Robot is used on the clients to control an internet browser. 2. Specific tracer code has to be developed, that is assigned to the action s test block and that will be executed by the tracer. Experience with the system level testing of CTI systems shows that this code can be generated automatically for most actions. Finally, when executing a test graph, a detailed protocol is prepared. For each test block executed by the tracer, all relevant data (its execution time, its name, the version of the files associated with the test block, the block s parameter values, and the processed data) are written to the protocol. 4 Conclusion The state of the art of web application testing tools is still dominated by static approaches, focussing on a posteriori structure reconstruction and interpretation [8, 3] rather than on functional aspects directly related to the applications design. As it has happened in the areas of telecommunications and CTI, this evolution is necessary in order to test applications also for their intention, rather than just for environmental accidents. We have presented a methodology capable of granting adequate coverage of functional testing, yet still easy to use. It bases on the experience we gained in the last two years in the area of system level test of Computer Telephony Integrated applications, where our coordination-based coarse grain approach has been applied very successfully ([6]). We have shown so far the practicability of the approach by successfully testing complex role based functional features of the Online Conference Service, an advanced online management system for the organization of scientific conferences.

References [1] Compuware Inc. http://www.compuware.com. [2] B. Lindner, T. Margaria, and B. Steffen. Ein personalisierter Internetdienst für wissenschaftliche Begutachtungsprozesse. In GI-VOI-BITKOM-OCG-TeleTrusT Konferenz on Elektronische Geschäftsprozesse (ebusiness Processes), Universität Klagenfurt, September 2001. [3] C.H. Liu, D.C. Kung, P. Hsia, and C.T. Hsu. Structural testing of web applications. In Proc. of Int. Symposium on software reliability engineering (ISSRE 2000), pp. 84 96, 2000. [4] T. Margaria and B. Steffen. Backtracking-free design planning by automatic synthesis in METAFrame. In Proc. of the 1st Int. Conf. on Fundamental Approaches to Software Engineering (FASE 1998), volume 1382 of Lecture Notes in Computer Science, pp. 188 204. Springer Verlag, 1998. [5] O. Niese, T. Margaria, A. Hagerer, M. Nagelmann, B. Steffen, G. Brune, and H. Ide. An automated testing environment for CTI systems using concepts for specification and verification of workflows. Annual Review of Communication, 54, 2000. [6] O. Niese, T. Margaria, A. Hagerer, B. Steffen, G. Brune, and H. Ide. Automated regression testing of CTI-systems. In Proc. of IEEE European Test Workshop (ETW 2001), 2001. [7] O. Niese, B. Steffen, T. Margaria, A. Hagerer, G. Brune, and H. Ide. Library-based design and consistency checks of system-level industrial test cases. In H. Hußmann, editor, Proc. of the 4th Int. Conf. on Fundamental Approaches to Software Engineering (FASE 2001), volume 2029 of Lecture Notes in Computer Science, pp. 233 248. Springer Verlag, 2001. [8] F. Ricca and P. Tonella. Building a tool for the analysis and testing of web applications: Problems and solutions. In T.Margaria and W.Yi, editors, Proc. of the 7th Int. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2001), volume 2031 of Lecture Notes in Computer Science, pp. 372 388. Springer Verlag, 2001. [9] Radview Software. http://www.radview.com. [10] B. Steffen and T. Margaria. METAFrame in Practice: Design of Intelligent Network Services, volume 1710 of Lecture Notes in Computer Science, pp. 390 415. Springer Verlag, 1999. [11] B. Steffen, T. Margaria, and V. Braun. Coarse-granular model checking in practice. In M.B. Dwyer, editor, Model Checking Software - 8th International SPIN Workshop, volume 2057 of Lecture Notes in Computer Science, pp. 304 312. Springer Verlag, 2001.