Research into Testing Service Oriented Architectures: Preliminary Report, November 2006 Prepared For: Contract Monitor: Avaya, Inc Dave Weiss Principal Investigator: Jeff Offutt George Mason University November 16, 2006 EXECUTIVE SUMMARY This report presents results for the Avaya Inc. sponsored project on testing service oriented architectures (including web applications and web services), which started August, 2006. The purpose of this project is to identify improved methods to test software built with service oriented architectures (SOA). Avaya Research Labs software technology research department is currently building SOA software. Several specific problems were identified in the SOW: (1) How to save and reuse tests of the web and voice interactions with people, particularly in automating test scripts for regression testing; (2) How to validate correct behavior of web service components with regards to requirements; (3) How to detect failures when the incorrect behavior is only reflected in server-side data stores, such as databases and files; (4) Is it possible for external users to subvert web service components and violate security protocols; and (5) How to ensure that the system will recover correctly after loss of connectivity or failure of one or more components. The GMU project is working on aspects of problems 2-5, as specified in the SOW, but not on problem 1. The significant steps accomplished so far are: Creation of hand-generated ad-hoc tests for VIA Deploying AutoBypass on Avaya s server Selecting Selenium as a tool to automate web application tests Construction of bypass-style tests by hand (still in process)
1 TASK LIST The following specifically task list was initially proposed by Offutt on 9 August. Tasks 1-7 were originally designated for the graduate research assistant, Qingxiang Wang (QW) to be carried out at Avaya, and tasks 8-13 were originally designated for the graduate research assistant and PI to carry out after the GRA returned to GMU in Fairfax. The task list was quickly amended at Avaya, when the GRA was asked to hand-generate tests for Diamond/Via. Thus, his work on the tasks listed below started after his return to GMU. 1. Become familiar with AutoBypass. Give demo to Joanne, others. (QW) - week of 14 Aug. 2. Gain experience with JUnit. (QW) - by 18 Aug. 3. Gain experience with HTTPUnit. (QW) - by 18 Aug. 4. Evaluate Javascript unit testing tools. Determine which will work best for us. (QW) - by 18 Aug. 5. Understand the string-based communication between contexts. Uses a singleton class in the shared class loader. Find the schemas that define the strings. (See task 11 below.) (QW) - by 18 Aug. 6. Study the JavaScript-generated VIA HTML pages. Work with Hamilton to understand how some of them work. (See task 10 below.) (QW) - by 25 Aug. 7. Run AutoBypass on VIA pages. (a) Install AutoBypass at Avaya. (b) Faults found? (c) How to check results? (d) Can it find JS-generated form elements? (QW) - by 25 Aug. *** Qingxiang back to Fairfax *** 8. Construct bypass-style hand-generated tests for VIA pages. Write as HTTPUnit tests. (QW), (AJO) - by 29 Sept. 9. Use Input Space Partitioning (ISP) to generate tests for screen interactions. 31 screens : 14 user screens, 3 user utilities, 7 account screens, 3 debug, 4 popups Analyze form elements (characteristics) Partition into values (select boxes, radio boxes, text fields,...) Choose base elements Choose combination criterion Combine to form tests Write as J,HTTP,JSUnit tests (QW), (AJO) - by 13 Oct. 2
10. Design ways to test JavaScript in VIA HTML pages. How can coverage be assured? Do tests have to be created and run by hand? Can we use HTTPUnit or JSUnit? (QW), (AJO) - by 27 Oct. 11. Analyze string-based communication between contexts. Study schemas. Construct tests based on XML modification rules (Wuzhi s research). Write as HTTPUnit tests. (QW), (AJO) - by 10 Nov. 12. Deliver all tests to Avaya. (QW), (AJO) - by 17 Nov. 13. Write paper as case study of testing web application. Every step and decision must be documented!!! (QW), (AJO), (JOrd),... - Spring 2007. 3
2 SUMMARY OF PROGRESS ON TASKS 2.1 Become familiar with AutoBypass This task was significantly delayed by the travel schedule of the author of AutoBypass, Vasileios Papadimitriou. We have successfully deployed AutoBypass on Qingxiang s computer, then Avaya s server, and finally Offutt s ISE server. The following issues were identified. When deploying AutoBypass tool on any server, we must set two global variables in the Java source file FormAnalysis.java. The first one is String localpathtoresultsfolder. The AutoBypass tool will store the test cases generated and the results of running test cases in this directory. For example, we can set localpathtoresultsfolder = /home/faculty/offutt/j2ee/jsp/autobypass/results/ ; The second variable that must be set is String URLtoResultsFolder. This variable determines the link style and how test cases will be accessed, and how the test cases are displayed via the http protocol within a browser. For example we can set URLtoResultsFolder = http://ise.gmu.edu:8080/offutt/jsp/autobypass/results/ ; If these two variables are not set correctly, the test results could not be seen. 2.1.1 Deploying on ISE Server-an older version of Tomcat Two problems appeared when deploying the AutoBypass tool on the ISE sever. The first is that AutoBypass sets the TEMP environment variable to be the directory of /usr/local/jakarta-tomcat- 4.1.8/temp, but there was no such directory. Thus, an exception was thrown while deploying the jar files that AutoBypass needed. The second problem is that the servlets of the tool use RequestDispatcher to include the header and the footer files when processing requests. With Tomcat version 4.1.8, this raises an Illegalstate error. This error did not occur with tomcat version 5. The solution was to remove these calls, which makes the user interfaces look worse but does not effect the functionality of the tool. 2.2 Gain experience with JUnit Done. 2.3 Gain experience with HTTPUnit Done. 2.4 Evaluate Javascript unit testing tools Qingxiang spent several weeks exploring various tools. The VIA HTML pages make extensive use of JavaScripting, not just to provide feedback and checking in response to user events, but to dynamically redraw the screens during use, and almost completely change forms by introducing new form fields and deleting others. We have evaluated JUnit, HttpUnit, HtmlUnit and Selenium to identify the tool that is best suited to test the VIA project. Finally, we have determined that that Selenium is the most effective tool for automating tests for the VIA project. Details are below. HttpUnit is a suite of Java classes to test Web applications over HTTP. It is written in Java and it emulates the essential portions of a browser s behavior, including form submission, JavaScript, basic HTTP authentication, cookies and automatic page redirection. It also allows Java methods to examine pages returned from the server as containers of forms, tables, and links. HttpUnit can also be combined with a JUnit testing framework and it is fairly easy to write tests 4
that very quickly verify the functioning of a web site. The problem for our application is that HttpUnit does not support all features of the JavaScript, while many VIA web pages rely heavily on JavaScript. HtmlUnit is similar to HttpUnit in that it works within a JUnit framework to automate Web Application tests. The biggest difference between HtmlUnit and HttpUnit is the style of writing test cases. HttpUnit centers itself on the protocol, making it easy to create requests and send them to the server independently of whether the technology is HTML, servlet, or JSP pages. HtmlUnit knows more about HTML; testers can ask an HTML Page for HTML anchors and virtually click them or set the values for form fields such as text inputs and select box, then submit the inputs. With more research, we determined that both HtmlUnit had HttpUnit use an open-source implementation of JavaScript called Rhino. Rhino is written entirely in Java and is designed to be embedded into Java applications to provide scripting to end users. Unfortunately, Rhino does not support all features of JavaScript, and HtmlUnit has the same limitations with VIA as HttpUnit has. Finally, we found the tool Selenium. Selenium tests run directly in a browser, just as real users enter data. The tests run in several browsers, including Mozilla Firefox on Windows, Linux, and Macintosh, and Internet Explorer. Thus, Selenium can be used to test application s conformance to different browsers and operating systems. Selenium also has builtin features for regression tests. Technologically, Selenium uses JavaScript and Iframes to embed test automation engines into browsers. Theoretically, this technique should work with any JavaScript-enabled browser. Selenium s ability to support all JavaScript features makes it the best choice for the VIA project. It can be combined with HttpUnit and HtmlUnit by using HttpUnit or HtmlUnit to retrieve controls from web pages, analyze their features and create Selenium style test cases. Selenium can be accessed from its website: http://www.openqa.org/selenium/. A good reference can be found at: http://release.openqa.org/selenium-core/0.8.0/reference.html. The following steps can be used to run Selenium tests on Avaya s network: 1. Open the link: http://ovid.research.avayalabs.com:8000/selenium/core/testrunner.html 2. Enter../avaya/AvayaTestSuite.html in the text box shown in the page and then click button Go 3. Run tests (a) Click the button All on the right side of the page to run all test cases (b) Or select one test from left side of the page and then click selected to run the test case selected 4. The results should be displayed. 2.5 Understand the string-based communication between contexts. Uses a singleton class in the shared class loader This task is on hold for now. It is probable that Qingxiang will need to physically be at Avaya Labs for this. 2.6 Study the JavaScript-generated VIA HTML pages. Work with Hamilton to understand how some of them work This task is on hold for now. It is almost certain that Qingxiang will need to physically be at Avaya Labs for this. 5
2.7 Run AutoBypass on VIA pages We are not actively working on this task at the moment. AutoBypass has been installed on Avaya s server and is available for Avaya testers. The heavy reliance on JavaScript will reduce the usefulness of AutoByass for the VIA application. 2.8 Construct bypass-style hand-generated tests for VIA pages. Write as HTTPUnit tests This has consumed a significant amount of time thus far. After selecting Selenium as our test automation tool, we have proceeded to generate tests for VIA pages. Tests for three pages have been delivered (TestLogin.html, TestPointsOfContact.html, and TestPointsOfContactButton.html), and more are in production. 2.9 Use Input Space Partitioning (ISP) to generate tests for screen interactions We have recently started this task. A preliminary input partition model (IPM) has been created for one page (points of contact), and is being evaluated. 2.10 Design ways to test JavaScript in VIA HTML pages Not started as yet. 2.11 Analyze string-based communication between contexts Not started as yet. 2.12 Deliver all tests to Avaya Tests are being delivered as created. 2.13 Write paper as case study of testing web application Not started as yet. 6