}w!"#$%&'()+,-./012345<ya

Transcription

1 }w!"#$%&'()+,-./012345<ya MASARYK UNIVERSITY FACULTY OF INFORMATICS Automation of regression testing of web applications DIPLOMA THESIS Bc. Dávid Chmurčiak Brno, spring 2013

2 Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Bc. Dávid Chmurčiak Advisor: Ing. RNDr. Barbora Bühnová, Ph.D. ii

3 Acknowledgement I would like to thank my supervisor Ing. RNDr. Barbora Bühnová, Ph.D. for her patience and helpfulness. My thanks also belong to Tomas Novotny the manager of the Xythos server quality assurance team for his invaluable advice and practical experiences. I also have to thank to my colleagues Miroslav Supol and Bronek Houdek for their support and contribution to this project. iii

4 Abstract The aim of this thesis is to study problems related to the automation of regression testing of web applications. It describes advantages and disadvantages of automated testing with the focus on testing user interfaces of webbased applications. Further it addresses different problems that are associated with this testing task and discusses possible solutions. The work also contains methodology for the deployment of automated regression testing of the web-based application developed by Blackboard Czech s.r.o. company within the Xythos document management server project. iv

5 Keywords Regression testing, test automation, user interface, web application, continuous integration, Selenium, WebDriver, TestNG. v

6 Contents 1 Introduction Software Testing Software errors, faults and failures The difficulty of testing Software testing approaches Black vs. White box testing Static vs. Dynamic testing Automated vs. manual testing Testing dependent on the product development method Testing within traditional methods of development Agile testing Regression testing Summary Tools for automated testing of web applications Unit Test Frameworks JUnit TestNG ReportNG Tools Comparison Tools for automated testing of web-based user interfaces Criteria for tools comparison Selenium Selenium IDE Selenium Remote Control (RC) Selenium 2 (WebDriver) Selenium Grid Tellurium Tellurium UI model Plugin (TrUMP) WebAii Framework Telerik Test Studio WebTest Xebium FitNesse Tools Comparison Other useful tools Firebug FirePath vi

7 3.3.2 Hudson Blackboard Xythos Document management system Application Description Product Development and Quality Assurance Product Development Phase Quality Assurance Phase Xythos Test Automation Project Test Automation Motivation and Requirements Selection of Technology and Tools Test plan Project team members and responsibilities Framework Description Layer 1 - Webdriver wrapper Layer 2 - Access modules for application pages Layer 3 - Test cases Test case structure Reporting and Results Analysis Test Execution and Continuous integration Manual test execution Automatic test execution Framework Optimization Test case Optimization Brittle automated tests Modules optimization Conclusion Appendix 70 A Application under test description and test-case example A.1 User Web-based Interface Analysis A.1.1 Admin UI A.1.2 WebUI A.1.3 MobileUI A.2 Test case example B Framework source code B.1 Content of electronic appendixes vii

8 1 Introduction A Zero-bug programmer is an oxymoron like an all-bug detecting test. This statement hanging on the wall in my office is a perfect example of why software testing is an important and inseparable part of software development. The increasing amount of an application s source code and the increasing number of developers participating on the development of software products causes growing numbers of errors in applications. One of the basic functions of software testing is to detect these errors and to detect them as fast as possible. The rate at which it is possible to identify these errors is entirely dependent on the testing approach used to perform the tests, the quality assurance team size and experience and also on the tools that are utilized in the process of software testing. These tools are very important in the process of test automation and they can make testing easier and more effective. Effective test automation is one of the most important practices for rapidly and repeatedly delivering high quality software. The aim of this thesis is to study the problems related to the automation of regression testing of web applications, to present and analyze tools available on the market and to implement and deploy automated regression tests for a web application developed by Blackboard Czech s.r.o. company by using appropriate tools. The thesis is divided into three parts. The first part (Chapter 2) addresses theoretical basics used in the field of software testing. It contains basic axioms of testing, their significance in the development process and approaches used to perform testing. Emphasis is put on automated testing and its application in different software development methods and different phases of application development cycles. Automated testing is compared to manual testing and its advantages and disadvantages are discussed. The second part (Chapter 3) deals with the tools that are an essential part of the development and deployment of automated tests. An emphasis is put on tools and frameworks used for unit testing and functional testing of web-based user interfaces. In addition to basic tools this chapter also presents tools that can be advantageously used in the process of creating and executing automated tests. This chapter concludes with a summary table comparing criteria of individual tools. This table can serve as a foothold in the process of selecting tools suitable for test automation projects. The practical part of this thesis (Chapters 4 and 5) presents a concrete implementation and deployment of the framework for building automated regression tests used for testing the Xythos Enterprise Document Manage- 1

9 1. INTRODUCTION ment System (EDMS) web application. Chapter 4 is a brief introduction that presents the application under test as well as development and testing methods used by the Xythos server team to develop it. The results from this and previous chapters are then used as the basis for the actual implementation of the automated test framework that is discussed in the last chapter of the thesis (Chapter 5). Reasons and requirements that led to the decision to replace manual testing with automated testing are discussed at the beginning of the chapter. Based on the analysis this and the preceding chapter the most appropriate tools are selected, a test plan is proposed and then the automated tests are implemented and deployed. This part solely describes the framework and also discusses problems and solutions that were solved during the project. As a conclusion to this work, results are reviewed and proposals for future development are given. Given the fact that this thesis is a solution created for a commercially sold product, not all parts of the thesis could be made available in the public version of the thesis. The public version of the thesis includes a complete theoretical part and parts of the practical implementation that do not contain any proprietary information. These are only complete source code and scripts attached as the appendix B. All other parts of the work including the complete text of the work and its conclusion are publicly available without any restraint. 2

10 2 Software Testing Software testing is a wide engineering field, which has an irreplaceable position in the process of software development. According to the definition proposed by IEEE Std [7] we can define software testing as: The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component. The process of analyzing a software item to detect the differences between existing and required conditions (that is, a bug) and to evaluate the features of the software items. The main task of this process is to discover and to remove the errors from the software product as soon as possible. This chapter will deal with the basic concepts of software testing as an error, why errors emerge, types of software testing used in different software development stages. 2.1 Software errors, faults and failures In the field of software testing there are many different terms used to describe the situation involving the application s error state. For a situation described as a software failure a various terms as issue, anomaly, mistake, error, failure, defect, inconsistence and many more can be used. Terms as fault, failure or defect usually refer to a situation which is in some way serious or even dangerous. Anomaly and inconsistence are usually considered as less severe situations. Issue and error are terms with very wide meaning. It is therefore appropriate to properly define the notion of fault. According to definition proposed by Ron Patton[9] we talk about the software error when at least one of the following conditions is fulfilled: 1. The software does not do what it should according to the product specification. 2. The software does something what it should not according to the product specification. 3. The software does something that the product specification does not mention. 4. The software does something that the product specification does not mention, but it should refer to. 3

11 2. SOFTWARE TESTING 5. The software is too complicated, difficult to operate, too slow or according to QA 1 opinion the customer will not consider it to be suitable. According to various studies the most common source of software failures is product specification, mostly from its absence. Another source is product design, which may not be sufficiently detailed or there are constant changes in it. Errors present in the program code are involved to a lesser extent. Those failures can be caused by unintentional programmers mistakes, by the inadequate or misunderstood design documentation, by errors in data selection or by lack of time due to deadline. Figure 2.1: Causes of faults[9] 2.2 The difficulty of testing It is necessary to take into account that the software testing is a complex process, which takes up a considerable portion of the development time and even at the end we cannot be sure that the developed application is adequately tested. It is so because even a simple application contains large number of combinations of values and conditions that makes its complete testing practically impossible. This implies the fact proposed by Edsger W.Dijkstra[3]: 1. Quality Assurance team 4

12 2. SOFTWARE TESTING "Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence." Testing therefore can not prove that there are no defects in the application. It can only prove the presence of variety of software errors. The good testing strategy and the testing scenario is the one that has a high probability to find yet undetected errors. The relation between the testing cost and number of undetected errors was published in the book Software Testing[9] and it is described on the following figure. Missed bugs Cost of testing Optimal amount of testing Quantity Not enough tests executed Too much tests executed Number of executed tests Figure 2.2: Optimal level of testing of a software product [9] The percentage of detected errors is rising with the growing number of executed tests. At the same time the amount of resources needed to be invested into the process of testing is growing. If we pass the point where those two lines cut across the testing cost will go up dramatically. It is therefore appropriate to find the limit, when testing of the application is still effective enough with respect to amount of non-detected errors and resources invested to find them. In the perspective of the small organization, it is common that the amount testing is below optimal level. One of the way how to increase the amount of executed tests with almost the same amount of resources spent can be use of appropriate tools and technologies for the automated testing. 5

13 2. SOFTWARE TESTING 2.3 Software testing approaches In order to test the whole software application sufficiently it is necessary to determine what types of tests, when and to what extent shall be performed. There is many ways how to test a software application. They are based on several aspects and they are divided into different categories. Not every category of tests is suitable for testing of each specific application. Incorrectly chosen testing approach may result in a failure to detect large number of software errors or errors will be detected too late Black vs. White box testing These approaches are based on two opposing concepts. First concept ignores internal mechanism of the application or its components and focuses entirely on the outputs generated as a response to executed commands. This approach is called "Black box testing" or functionality testing defined in Galin s book[4] as: Black box testing identifies bugs only according to software malfunctions as they are revealed in its erroneous outputs. In cases that the outputs are found to be correct, black box disregards the internal path of calculations and processing performed. From the tester s point of view black box testing is testing when he does not need to know how software works internally. On other hand "White box testing" or structural testing relies just on internal structure of software and its calculations. It examines internal calculation paths in order to identify bugs. Although the term "white" is meant to emphasize the contrast between this method and black box testing, the method s other name - "glass box testing" - better express its basic characteristic, that of investigating the correctness of code structure[4]. Both mentioned approaches has some advantages and also disadvantages which are often solved by using method which combines both of them called "Gray box testing". In the first step of testing using gray box tests are developed with emphasis on functional aspects of the application. This means that the application will be tested according to its real utilization. In the second step procedures utilized in the process of white box testing are used. They main goal is to increase effective coverage of the application code Static vs. Dynamic testing Another approach how to test the software is static or dynamic testing. As the name suggests static testing takes place on the application that is not running. Tested application is only reviewed and revised. On the other 6

14 2. SOFTWARE TESTING hand dynamic testing is performed on the running application where user s actions and system s reactions are analyzed. With the respect to previously defined approach (Black vs. White Box) and this approach we can define this four combined approaches[9] : Static testing of the black box: In this case testing is executed as revision of documents which belongs to the software specification. This documents can come from different sources as: usability studies or marketing requirements. Static testing of the white box: In this case the application design and code is reviewed. This is process of careful examination of the design, architecture and application code on the static (not running) system. Dynamic testing of the black box: This is testing of the application without any knowledge of the application code. This is an inspection or correct transformation of inputs to expected results. Dynamic testing of the white box: This type of testing is similar to debugging but has different goal. The goal of this type of testing is to detect errors while main debugging goal is to fix detected errors. It is obvious that this two processes overlay and difference is only in the approach from side of testers and programmers Automated vs. manual testing Another way how to distinguish approaches for software testing is based on who or what executes tests. Manual testing can be described as a process where a person initiates each test, interacts with it, and interprets, analyzes and reports the results. Software testing is automated when there is a mechanism for tester-free running of test cases.[5] Fully automated testing should be able to run specified test cases or its subset without any intervention after tests are launched. It should be able to capture relevant results, compare them with expected results, flag differences and report pass/fail for all executed tests. Manual testing is suitable to be used in cases where tests require execution and evaluation of complex tasks. Contrary to this when tests need to execute a large amount of repetitive tasks or tests generate large amounts of data then it is better to use computers to execute them. In the principle, we could say that the deployment of test automation is appropriate where manual testing is a tedious repetitive activity. On the other hand, it is necessary to remember that automatic tests can not replace 7

15 2. SOFTWARE TESTING manual tests, because even very similar tests performed manually and automatically have different value for testing. It is important to automate only tests that are simple and straightforward. Too complicated tests can be very inefficient and their maintenance costs can overwhelm their benefits. Components of the test automation Tests that are designed to automate the particular test scenario do not consist only from steps that are automatically executed but also from another components. These components enable automated test cases to be effectively reused. Keith Stobie and Mark Bergman have described these components using term SEARCH[12]. Setup is effort to bring the software to a point where the actual test operation is ready for execution. Execution is core of the test. The specific steps necessary to verify functionality, error handling or some other relevant task. Analysis is the process of determining whether the test passes or fails. This is the most important and the most complicated step of the test. Reporting includes display and dissemination of the analysis, for example, log files, database or other generated files. Cleanup is the phase that returns the software to a known state so that the next test can be executed. Help enables maintenance and robustness of the test case throughout its life. It is good to notice that manual and automated tests can be used simultaneously in each of the component. It is possible to have automated setup, execution, reporting and cleanup but to manually analyze test results or to prepare the help. Advantages and disadvantages of automated tests As it was mentioned before manual and automate testing are two different testing approaches and cannot substitute each other. Automated tests should be used together with manual tests to increase benefits resulting from their advantages. This section presents qualitative comparison of manual and automated testing displayed as listing of advantages and disadvantages of automated tests. Main advantages of automated test are[4]: 8

16 2. SOFTWARE TESTING Accuracy and completeness of the performance. Computerized testing guarantees that test cases are carried out completely and accurately. Manual testing suffers from periods of tester weariness or low concentration, that induce omissions and errors in the test execution. Accuracy of results logs and summary reports. Comprehensiveness of the test information. Data gathered from automated tests results stored in the database can be easier queried and reported than it would be after manual tests. Few manpower resources required to perform tests. Manual testing in comparison is a major consumer of manpower resources. Automation frees people to do more important work. Shorter duration and reusability of tests. The duration of automated test is usually shorter than that of manual tests. In addition, automated tests in contrast to manual tests can be executed 24 hours a day, seven days a week. Therefore automated tests give feedback, early and often. Performance of complete regressions tests. Automation allows to broaden the range of regression tests and increases chance of detecting errors introduced by fixing previously found issues. Performance of test classes beyond the scope of manual testing. Computerization enables testers to perform for instance load or performance tests on medium or large scale system. This is almost impossible by using only manual tests. The main disadvantages are: High investments required in package purchasing and training High test development initial investment costs High manpower requirements for test preparation Considerable testing area uncovered. Automated testing tools do not cover the entire variety of development tools and either applications so manual testing is still needed. Problems related to test automation When creating and deploying automated tests problems that can be associated with this process need to be taken into the account. Some of these problems are[6]: 9

17 2. SOFTWARE TESTING Automated tests are poorly designed. Automated tests must be designed differently from manual tests. Much of the power of automated testing comes from using a computer to do things a person cannot do. When designing manual tests, you are not likely to consider tests that apply repetitive operations over thousands of files. On the other side automated tests are best in executing repetitive operations. Designing tests without a good understanding of automation possibilities may overlook some of the most valuable opportunities for automation. Difficult test results analysis and debugging. In case that there is a failure detected after automated tests are executed. Analysis of failed tests must be as easy as possible. Therefore is necessary to include some form of test recording, result logging into the test automation project. This can be done for instance by saving a detailed test log and screen-shots of the user interface. This screen-shots can help even in case of debugging automated tests. False alarms and Silent misses may be common. False alarms occur in case of test failure due to errors in tests not error in tested application. Silent misses are serious issue because this means that there is a bug in the application which was not detected by automated tests but it should be detected. Improperly defined element locators. Element locators should be defined as robust as possible to be immune to changes in the application. For that reason it is not proper to bind element locators to internal structure of the web page. Instead of this element locators should be bound to unique identifiers associated with the element. Prevent tests from interfering with each other. This means to design all tests to be independent. Interfering tests may cause lots of problems when executed in parallel. 2.4 Testing dependent on the product development method There are many different methods used for the software development. Each method is characterized by the different approach to software development and testing. Next section will discuss traditional and agile software development methods and corresponding testing practices. 10

18 2.4.1 Testing within traditional methods of development 2. SOFTWARE TESTING Traditional methods of software development are characterized by the development process divided into stages that are closely linked together with emphasis laid down on the final product specification. Among traditional methods belong also so-called "V-model"[6]. This model is an extension of the traditional "waterfall" model. Testing of applications developed using "V-model" takes place on multiple levels, each of this levels is bound to a certain stage of the application development process and can be done only in a particular time during the development process. This model can be represented as depicted on the Figure 2.3. Customer Requirements Technical Requirements High Level Design Detailed Design Integration Testing Unit Testing Acceptance Testing System Testing Implementation Figure 2.3: The "V" - Model Each of these levels requires different approach to testing and focuses on the different application functionality. The levels can be defined as follows: Unit testing: Testing of methods, classes and objects as isolated components. Focus is put on testing inner logic of particular functions, methods and procedures. Typically, unit testing is performed with the support of IDE 2, unit test frameworks and debugging tools. Practically this is a type of testing executed by the programmer who wrote the code and issues found during unit testing are fixed immediately without being reported. 2. Integrated Development Environment 11

19 2. SOFTWARE TESTING Integration testing: Testing of interfaces, cross component interactions and interactions with different parts of the system as databases, file systems or interaction between the application and the operating system. Both functional and structural approaches can be used. System testing: Testing is concerned with the behavior of the application as a whole system that will be shipped to the customer. System testing includes the user interface testing, load testing, performance testing, et cetera. Acceptance testing: Acceptance testing is mainly focused on the compliance with customer demands placed on the application. The goal in acceptance testing is to establish customer trust in the product. Finding bugs is not main goal in this type of testing Agile testing Agile development methods are alternative to traditional methods. They were created to eliminate their drawbacks. The agile development means that the development process is iterative and incremental. An iteration may be as short as one week, or as long as a month. The fully functional increment of the application is delivered at the end of each iteration. Comparing to traditional development methods emphasis are laid down on individuals and interactions over processes and tools. Working software over comprehensive documentation. Customer collaboration over contract negotiation. Responding to changes over following the plan[2]. Agile testing is different to testing in the above mentioned "V-model" in a way that the process of testing is done in parallel to development of application during each particular iteration. The development team is responsible for unit and integration tests that are executed after each build of the product. Testing scenarios for acceptance testing are outcome from analysis of the requirements which need to be met in each iteration. Each iteration in addition add more regression tests. Maintaining tests coverage for the product developed using agile development methods can be long term sustainable only with use of test automation on each level of testing. Besides the fact that automated tests help to increase test coverage they can also supply the role of the product documentation because they reflect current state of the product. The agile test automation pyramid on Figure 2.4 shows three different layers of automated tests. The lowest tier of the pyramid is the foundation 12

20 2. SOFTWARE TESTING that support all of the rest. It is mainly made up by unit and component tests. These tests are quick and least expensive to write. They provide best and quickest feedback to programmers. The middle tier includes functional and acceptance tests that operates on the API 3 level and therefore bypass presentation layer of the application. They are less expensive to write and to maintain as GUI 4 automated tests. On the top tier lay GUI tests they represent what should be the smallest automation effort. This is because automated GUI tests are brittle, expensive to write and to maintain. GUI tests are written after the code is completed, and so they are written to critique the product and they go directly to the regression suite. Manual tests on the top of the pyramid represent exploratory and acceptance testing that is better to perform manually[2]. Manual Tests GUI Tests Acceptance Tests (API layer) Unit Tests Figure 2.4: The automation pyramid[2] Regression testing According to IEEE is regression testing defined as: Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements[7]. 3. Application Programming Interface 4. Graphical User Interface 13

21 2. SOFTWARE TESTING Regression testing is specific kind of testing which can be performed at all levels of testing mentioned in previous sections. This implies that not only testers are involved in this kind of testing but also programmers and customers. Basic purpose for regression tests is to show that some defect was not correctly fixed, to show that an old fix was broken or that proper fix for some issue brought in some new issue. This kind of testing also carries some weakness and those are: Poor at finding new bugs. Anything which is not covered in the regression suite may be broken. Maintenance of the regression test suite can be extremely costly. Most defects are found during the test creation process. If we talk about the test automation. Regression testing is one of the best candidate for test automation. It can make good use of automation s advantages as reusability, repeatability of test cases. On the other hand it needs to be taken into account that automated regression tests find minority of bugs. Projects with significant well designed automated regression suite can find only about 15 percent of total bugs reported by using test automation[6]. Regression test automators usually find more bug during the test development than when they execute test later. However big opportunity to increase number of found bugs is to use automated regression tests to test different environment configurations (e.g. different platforms, databases, web browsers, etc.) Summary This chapter has discussed different software development approaches and testing methods. Following table brings summary of all mentioned testing levels, methods usually used to perform them, person by whom tests are typically executed and how important the test automation is for the particular test level. Testing level Method Executed by Automation Unit testing white box programmers mandatory Integration testing black/white box programmers required System testing black box testers required Acceptance testing black box customers optional Regression testing black/white box testers necessary 14

22 3 Tools for automated testing of web applications Currently there is number of tools available on the market designed to enable automated testing. Those tools can be used in various stages of the software development and testing process to support creation of automated unit, integration, system and acceptance test suites. These tools are also very useful for regression testing and user interface testing. To select appropriate tools different technical, organizational and financial aspects which are associated with the introduction of tools must be necessary taken into the account. Technical aspects include the support for programming and scripting languages, different platforms support, integrability with another tools or user-friendliness. The second aspect is a financial difficulty to obtain the tool and related license terms and conditions. And last but not least there are initial costs in terms of time and organizational demands that also must be taken into consideration. In following sections tools for automated testing of web applications will be presented and compared. Emphasis will be put particularly on tools useful for unit testing, regression and user interface testing based on Java 1 platform, because the tested application is written in Java and our development team consists only from Java programmers. At the end of the chapter comparison that may help in selecting the right tool will be provided. 3.1 Unit Test Frameworks Choice of tools available for unit testing is dependent on the platform on which the product is developed. On the market there is variety of tools for different platforms. Each of these frameworks 2 should meet the basic requirements and those are: Support for the integration of most widespread software development environments such as Eclipse, NetBeans or Microsoft Visual Studio. Support for software build tools such as Ant 3 or Maven. Java annotations 4 support or similar support for different program- 1. Java is object-oriented programming language 2. Software framework is a reusable set of libraries or classes designed to support development of another software application 3. Apache Ant is a Java library and command-line tool used to build Java applications 4. Annotation is a form of syntactic metadata that can be added to a Java class, method or variable. The feature was introduced in Java 6. 15

23 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS ming languages. Annotations are crucial for the test automation as metadata added using annotations allows configuration and execution of automated tests. Ensure initialization and release of resources before and after the test. Test that your source code is multithread-safe. Support for the test clustering. Transparent and well arranged test results. In following subsections basic frameworks for unit testing in Java are introduced and described JUnit JUnit is a testing framework written in Java. It is one of the most successful frameworks from the xunit 5 family. The older set of libraries called JUnit 3.x that was used for earlier projects is no longer developed, but still remains maintained for the purpose of backward compatibility. A new set JUnit 4.x offers several enhancements compared to older version and uses all options available in Java 5 and later. One advantage from JUnit 3 is removed dependency to use the name of the test method with the prefix test. The test created in JUnit is represented by a class having public methods marked with annotation. One class represents a test suite designed to test the programming unit. Before the individual test is performed it can be initialized by using method annotated This method is executed always before the start of each test and can prepare environment needed for the test execution. After the test is finished a method annotated with is called. This method releases all resources acquired by test (e.g. delete temporary data, restore default settings). For tests that require complex and time-consuming setup or cleanup a method marked with and is called. Methods annotated this way are executed only once, before the start or after the finish of all tests. To verify results of the tested functionality JUnit uses methods with name prefix "assert" (e.g. asserttrue). Simple JUnit test is described on the following example. 5. x from the tool name stands for shortcut of the programming language in which is framework developed, e.g CppUnit(C++) 16

24 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Listing 3.1: Simple JUnit test import org. j u n i t. Test ; import s t a t i c org. j u n i t. Assert. ; public c l a s s ApplicationUnitTest { public void testconcatenate ( ) { ApplicationUnit ApplicationUnit = new ApplicationUnit ( ) ; S t r i n g r e s u l t = ApplicationUnit. concatenate ( "one", "two" ) ; a s s e r t E q u a l s ( "onetwo", r e s u l t ) ; } JUnit is primarily oriented on unit testing. The main idea is to isolate individual tests. Such structured tests have one big advantage and that the failure of one of tests does not affect test results for other tests. JUnit therefore does not provide the functionality to write complex tests TestNG TestNG is an open-source testing framework written in Java and inspired by JUnit and NUnit. The framework was created as a response to problems in the framework JUnit 3.x. It was created at a time when JUnit 4 which addresses those deficiencies was not released yet. TestNG contains improvements[1] over the JUnit as: Wider set of annotations Support for parametric and data-driven testing (@DataProvider annotation). Powerful execution model (no more TestSuite). The ability to run tests in parallel and in multiple threads Greater possibility of the configuration during the execution of tests (e.g. the ability to configure different test groups and run them under different conditions, test dependencies) Creating TestNG tests consist typically from three parts: Create business logic to be tested and annotate test methods. Add configuration information (e.g. names of classes or groups to be executed) in to the testng.xml or the testng.xml configuration file. 17

25 Execute tests. 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS TestNG is available as a plugin to all of the most widely used software development environments from which tests can be easily configured and executed. Very important is also functionality to produce structured reports using TestNG Reporter API ReportNG RerportNG is a simple plugin for TestNG framework, that has been developed as a replacement for proprietary reporting tool supplied with TestNG. It provides a simple and well arranged view of testing results. Compared with basic TestNG tool test reports are more transparent and easier to read which is probably the biggest advantage. In addition to conventional HTML report ReportNG is able to generate XML output, which allows it to be integrated with tools for CI 6 such as Hudson Tools Comparison TestNG and JUnit are similar tools, but they have also some differences, which are described in following table: Functionality JUnit TestNG Test initialization Yes Yes Test suite initialization Yes (using static Yes helper method) Support for creating test groups Limited Support Yes Support for data-driven testing Yes (using plugin) Yes Parametrized test Limited Support Yes Support for parallel test execution Yes Yes Support for Ant and Maven Yes Yes Support for IDE Yes Support for test reporting Yes Yes Test dependencies Yes Yes Table 3.1: Unit test frameworks comparison 6. Continuous Integration 18

26 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS 3.2 Tools for automated testing of web-based user interfaces Currently, there there are several tools enabling automated testing of user interfaces. With regard to their similar purpose they also offer an equivalent functionality. However, there are also differences among them. In the next chapter various test frameworks will be discussed and compared in terms of technical parameters, input costs and provided technical support Criteria for tools comparison Individual tools can be compared according to various criteria, as they are used for same purpose they will also have similar characteristics. Basic criteria designated for comparison and evaluation of the available tools are described in the Table 3.2. Criterion Description Supported platforms Support for most widely used operating systems and their different versions. Important is also support for mobile operating systems. Supported programming Support for different programming and languages scripting languages Supported web browsers Does the tool support testing of web-based applications for the most common web browsers? Important are both desktop and mobile web browsers GUI for the test development Does it include applications or web browser plugins, which can be used to record test scenarios? Parallel test execution Possibility to trigger automated tests in parallel and in distributed environment. Integrability with other Possibility to integrate the framework with automation tools other automated testing tools Technical support User support availability and method. Availability and frequency of updates. Licensing and price Price of the tool and licensing requirements for commercial use. Table 3.2: Comparison criteria 19

27 3.2.2 Selenium 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Selenium is a set of different software tools designed to automate testing of web applications. The suite is developed by the company ThoughtWorks 7 since year All of these tools together provide a rich set of testing functionality, compatible with all the most widespread software platforms and web browsers. In addition to standard browsers Selenium also supports virtual browser HtmlUnit 8, implemented in Java. This browser does not have a graphical interface and servers solely as a JavaScript interpreter to simulate the real web page. All components of the project are released under opensource license Apache Under this license the source code of Selenium is freely available to be used for commercial purposes. With regard to the fact that Selenium is open-source project is the technical support provided by a large community of volunteers and it is available in form of user discussion groups[11]. Another way of the Selenium support is the Selenium Bug Report[11] system, that offers simple way to report found issues. There is also possibility of the commercial support, which is currently provided by more than 20 companies worldwide Selenium IDE The Selenium IDE is a prototyping tool for creating automated tests, that is implemented as a plugin for the web browser Mozilla Firefox from the version 3. It provides simple interface that contains the record functionality. By using this functionality the system can directly record steps performed by a user when using the web application. User s actions are recorded as triplets of values: Command - representing performed action Target - identifying element for which the command is performed. Element can be identified by the name, the attribute or using the XPath 10 expression Value - set by the command to target. It may also be empty. Each such triplet is one command for the Selenium IDE. These commands are called Selenese 11 and they are used to perform tests. Selenium Language used for navigation and query in XML document

28 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS offers a wide range of these commands that create whole test language. Commands can be divided into three groups: Actions - commands used to manipulate the state of the application (e.g. Click on the button) Accessors - used to detect application s state and to store results in a variable (e.g. storetitle). Accessors are also used to automatically generate assertions. Assertions - used to compare current state of the application to the expected state (e.g. asserttext) Commands generated this way are stored in the form of a HTML table with three columns as shown in the following table: gobackandwait verifytextpresent Welcome to my page. type id=phone type id=address Table 3.3: Selenium IDE test definition With the basic knowledge of the Selenese HTML syntax and the Selenium IDE it is possible to create automated tests very fast and easily, without closer programming skills. For completeness, specific example of a simple test recorded in the Selenium IDE is shown on Figure 3.1. Example is showing the simple test of logging into the web application. Uploaded test is for proper functioning supplemented by an assertion. At the bottom part of the figure is notation using the Selenese syntax. Vertical lines in the notation represent particular table cells. From tests created by this method the Selenium IDE can also generate scripts in programming languages C#, Java, Python, Perl. The Selenium IDE is also able to generate scripts for other unit test frameworks. Supported frameworks are JUnit, NUnit(.Net), RSpec(Ruby), unittest(python) or TestNG. The disadvantage is that precisely because of the simplicity and associated small set of useful functions the Selenium IDE does not allow to use of conditions and cycles required to create complex test scenarios. At the same time it is quite difficult to reduce duplicate code and this disadvantage is causing great difficulty in tests maintenance. The tool is therefore characterized by a good level of productivity for a small number of tests, but produc- 21

29 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Figure 3.1: Selenium IDE tivity and usability decreases with their widening number. Because of these drawbacks Selenium IDE is intended solely as a quick prototyping tool to support the development of automated tests by using other tools from the Selenium family Selenium Remote Control (RC) It is the older version of the Selenium API, that has the server as the main Selenium project until it was merged with the Selenium WebDriver. It is still supported for backward compatibility, but is no longer actively developed. However the Selenium RC still provides some features that are not yet im- 22

30 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS plemented in Selenium WebDriver, such as the support for older versions of web browsers. It consist of two basic components: Selenium Server - The server side takes care of the administration of browser windows in which tests are performed. It is also responsible for sending and executing the commands for the web application and also acts as a proxy between the tested application and the web browser. Server is written in Java and can by run just like any other Java application, therefore requires no installation. Client libraries - The client part contains libraries that are used to create tests in Java, C#, Perl, PHP, Python and Ruby. The primary object used to create tests is the browser, represented by an instance of a class which implements interface Selenium. Via the browser object various features (e.g. type or click) can be called and also the current state of the application (e.g. gettext) can be queried. For the determination of the attribute with which the method should work text identifier is used. The identifier is determined by name in the DOM 12 model or by using XPath expressions. Listing 3.2: Simple Selenium RC test public c l a s s simpleseleniumrctest { public void l o g i n T e s t ( ) { DefaultSelenium browser = new DefaultSelenium ( " localhost", 4444, "*firefox,", " ) ; browser. s t a r t ( ) ; browser. open ( "/" ) ; browser. type ( "XY_username", "testuser" ) ; browser. type ( "XY_password", "secret" ) ; browser. c l i c k ( "id=okbutton" ) ; } } Selenium 2 (WebDriver) Selenium 2 is the latest framework from the Selenium family, which has been created as a combination of the Selenium RC and the project Web- Driver. It is also called Selenium WebDriver[11]. WebDriver has been designed as a simple and accurate API that repaired some of the shortcomings 12. Document Object Model is a cross-platform and language-independent convention for representing and interacting with objects in HTML, XHTML and XML documents 23

31 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS and limitations of the Selenium RC. It has a better support for dynamically generated web pages, in which the previous version lacked support. The main difference is in the WebDriver approach to the browser. Each of supported browsers is using a special controller, using that WebDriver communicates directly with the web browser. It does not send commands using the JavaScript as the Selenium RC does but instead of this it communicates directly by using browser native bindings for the test automation support. Another difference with the Selenium RC is that there is no need to start the Selenium Server in case that tests are going to be executed locally on the same computer and they will only used WebDriver API. There are drivers available for all most common operating systems and related browsers starting with versions: Google Chrome 12+, Internet Explorer 6+ (both 32 and 64-bit version), Opera 11.5+, Mozilla Firefox , Safari 5.1+, HtmlUnit 2.9. Support for two most prevalent mobile platforms has also been recently added and WebDriver now supports: Android ( the physical device and emulator), ios 3+ for smartphones and ios 3.2+ for tablets (the physical device and emulator) WebDriver API is more comprehensive, more intuitive and the resulting code is compared to the Selenium RC much easier to read. In this case is as browser used an instance of a class inherited from the interface WebDriver. Initialization of the browser is easier in this case due the parameterless constructor. This class ensures all the functionality needed to access and cooperate with the web browser (e.g. gettitle or findelement). Another useful interface is WebElement, which is used to create objects representing individual HTML elements located on the web page and to work with them. As a mechanism to locate elements on the web page acts class inherited from interface By. This interface offers methods that locates elements using their names, ids, tags, XPaths and many more. On the next figure is shown a simple example of the automated test completed using the Selenium Web- Driver. Listing 3.3: Simple Selenium WebDriver test public c l a s s simpleseleniumwdtest { public void l o g i n T e s t ( ) { WebDriver br = new F i r e f o x D r i v e r ( ) ; br. get ( " ) ; WebElement name = br. findelement ( By. name("xy_username " ) ) ; WebElement pass = br. findelement ( By. name("xy_username " ) ) ; 24

32 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS } WebElement ok = browser. findelement ( By. id ( "okbutton" ) ) name. sendkeys ( "testuser" ) ; pass. sendkeys ( "secret" ) ; ok. c l i c k ( ) ; } Selenium Grid Selenium Grid is a tool enabling parallel execution of various automated tests using multiple remote test environments with different web browsers. The tool is suitable for faster and more effective testing of large and time consuming test suites that must be executed on a variety of test environment configurations. Grid consists of so-called Hub and one or more Nodes. The Selenium Server is running independently on each of these nodes. The Hub maintains configuration information for each authorized node and it is responsible for routing of test requirements to the Selenium RC node with the correct configuration (e.g. Widows 7 with Firefox web browser). If we want to take advantage of the Selenium Grid all automated tests must be able to run in parallel by using features for the parallel test execution delivered by for example the TestNG framework. Using the Selenium Grid and one of the frameworks for unit testing can significantly reduce time for executing the entire automated test suite because there is no need to wait until the single test is completed and tests can be distributed on several test environments Tellurium It is a tool for automated testing of web applications based on the Selenium RC framework. The tool has been developing since 2008 and it is regularly updated. Similar to the Selenium Tellurium is open-source project licensed under Apache 2.0 license and freely available for commercial use. The motivation to develop this tool was to remove the lack of robustness emerged by using other frameworks, especially the Selenium. It uses different philosophy in the approach to each element on the website. While the Selenium accesses each DOM element like buttons or text fields as a separate object. Tellurium treats all user interface elements as one unit grouped into so-called UI models. Each element is defined by the unique identifier (uid). By using this uid elements are referred when called by a function. To define the UI modules and functions for different types of elements (e.g. button 25

33 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS and corresponding action click) scripting language Groovy 13 is used. By the separation of elements definition and the test code Tellurium increases the robustness of developed tests. The framework is thus able to adapt on minor changes of elements and their attributes during the development phase of the application, because element locators are not fixed, but dynamically generated at the test run time. These locators can have different formats including JQuery or XPath expressions. Among the basic functions provided by the Tellurium belongs: Uses DSL 14 for the UI definition, action and testing. Supports JavaScript. Supports Data Driven testing. Supports Selenium Grid. Allows integration with JUnit and TestNG. Allows integration with Ant and Maven Tellurium UI model Plugin (TrUMP) It is similar to the Selenium IDE development environment implemented as a web browser plugin for the Mozilla Firefox. Currently is this plugin compatible with the Firefox version 2+ and 3+. However it is not available for later versions. Similar to the plugin for Selenium it has the recording function, but unlike it primarily uses this feature to create UI modules described in the previous text. This will create the UI module and then test itself and thus ensure that the description of user interface elements and the test code is separated. The following figure shows a simple test recorded by using TrUMP WebAii Framework WebAii is a framework for automated testing of web applications, originally developed by the company ArtOfTest later acquired by the company Telerik 15. The framework continues to be actively developed, supported completely free of charge, but it is not open-source. The professional technical support is also available 16. Since this is the platform closely associated 13. Groovy is scripting language based upon strength of Java but with additional features inspired by languages like Python, Ruby and Smalltalk 14. Domain Specific Language

34 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Figure 3.2: Telurium IDE with the operating system Microsoft Windows it supports creating tests in programming languages C# and Visual Basic.Net. Officially supports integration with frameworks for unit testing as NUnit, MSUnit, XUnit, MbUnit. To use this frameworks just add the reference of the tool library into the automation project development environment. Basic building blocks of the application being developed using WebAii framework are classes Browser and Manager. The class Browser represents an instance of the web browser, in which automated tests are executed, provides access to elements and control them on the website. The class Manager provides the life cycle management of browser instances and also manages communication between the framework and various opened browser instances. Supported web browsers are Internet Explorer, Firefox, Chrome, Safari (only version for the operating system Windows). Creating a test is similar to Selenium WebDriver. The method for search and identification of user interface elements is also similar and allows to search for the element identification as well as search using XPath. Unlike 27

35 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS the Selenium it allows using of query language LINQ 17 expressions. Another difference is the approach to the element identification. In the Web- Driver framework every element is descendant of the class implementing the interface WebElement. In case of the WebAii a separate class exists for each particular element. For example the element <a> has corresponding class <HtmlAnchor>. Listing 3.4: Simple WebAii test //Launch web browser and open page f o r t e s t i n g Manager. LaunchNewBrowser ( BrowserType. I n t e r n e t E x p l o r e r ) ; ActiveBrowser. NavigateTo ( " :8080" ) ; //find element by name Element username = ActiveBrowser. Find. ByName( "XY_username " ) ; //find element <a> containing t e x t HtmlAnchor l i n k = ActiveBrowser. Find. ByContent <HtmlAnchor >("Jump to content" ) ; The tool also offers options for testing web applications using technologies as AJAX 18 or MS Silverlight 19. One of the disadvantages is limited support for the parallel test execution. Overall, WebAii is comprehensive tool for creating automated tests intended mainly for users preferring programming in C# and.net Telerik Test Studio In addition to the freely available framework described in previous subsection the company Telerik develops an application for testing and test automation called the Telerik Test Studio. It is the comprehensive commercial system based on the WebAii framework, and therefore provides support for the same web browsers and frameworks for unit testing. However, it also comes with many features that are available to users through a transparent GUI. It also provides support for recording test scripts as a plugin for the Internet Explorer. It also has support for automated testing of applications for mobile devices. Currently there is available solution for Apple ios which is used by mobile phones (iphone), tablets (ipad) and music players 17. Language Integrated Query - query language for platform.net 18. Asynchronous JavaScript and XML - general name for a set of technologies that are used by interactive web applications, also referred to as the Web Silverlight is Microsoft platform dedicated to develop business and multimedia web applications 28

36 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS (ipod). This solution not only enables testing of web applications, but also testing of native mobile applications. Since Telerik Test Studio is not only the framework but the complete system for application testing it offers following additional features: Support for testing desktop applications based on WPF 20. Performance testing. Load tests - test a large number of concurrent accesses to the web application. Storing and managing test suites. Since it is a commercially available solution (single user license costs 2499 $) the technical customer support is on professional level and the product is regularly updated WebTest WebTest is a simple framework for automated testing of web applications, written in Java. Test scenarios are executed under a virtual interpreter HtmlUnit included in the distribution. The main advantage of the HtmlUnit is in speed of the test execution, because test are not executed in the real graphical user interface of the browser. This virtual browser simulates manner in which is JavaScript processed in the Internet Explorer and Firefox web browser. On the other hand, this solution imposes constraints in the form of problems associated with processing poorly formatted HTML code. Tests are created either as XML scripts very similar to scripts used for Ant or as Groovy scripts. WebTest directly supports integration with Ant. A very interesting part of the project is a reporting package that provides the transparent presentation of test results. WebTest contains an extension for the Firefox web browser that provides an easy way to create test scenarios just like by using Selenium IDE Xebium Xebium is a tool for creating automated integration, acceptance and regression tests. It can be used as a testing framework for functional testing of the user interface of web applications. Xebium is a combination of three frameworks. These are Selenium IDE, Selenium WebDriver and FitNesse. 20. Windows Presentation Foundation - desktop application written in XAML language for platform.net 29

37 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Frameworks from Selenium family were further discussed in the previous section. Third tool will be introduced in the next subsection FitNesse FitNesse is an open-source project implemented in Java and based on the FIT 21 framework. It is designed to support creation and maintenance of the acceptance test suite. It is created in order to enable software developers, testers and customers collaboratively develop tests for web applications. It therefore supports features that make it possible to easily create readable tests without any programming skills. Originally it was developed for the Java language, but currently also programming languages C++, Python, Ruby, Delphi and C# are supported. FitNesse tool can be used for multiple test purposes. First of all it is tool used for the collaborative software development, but it has also features to create and organize automated acceptance and regression tests in form of Wiki web sites. All these parts are closely interrelated and provide comprehensive tool for the software and the automated tests development. The main difference compared to other available frameworks for test automation is a method of creating and storing tests. FitNesse tests are expressed as tables with input data and expected outputs. DivisionComponentTest numerator denominator quotient? Table 3.4: FitNesse test table These tables are in FitNesse called Decision Tables and represent a complete test scenario. The scenario described in the Table 3.4 is an example of a simple division test. The first row is the name of the class (Division Component Test), which will process the test called Fixture. Columns Numerator and Denominator are input titles and value assigned to them, while column quotient? is the expected outcome of the test. Tables like this are created and stored as individual tests using the FitNesse wiki that can be subsequently easily executed. If the test passes the color of the table cell marked by? is changed to green otherwise it is colored red. These tables are in the FitNesse

38 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS wiki represented as shown in the next figure. The corresponding fixture is listed on the Figure 3.5 DivisionComponentTest numerator denominator quotient? Table 3.5: FitNesse test Listing 3.5: FitNesse fixture public c l a s s DivisionComponentTest { p r i v a t e double numerator, denominator ; public void setnumerator ( double numerator ) { t h i s. numerator = numerator ; } public void setdenominator ( double denominator } { t h i s. denominator = denominator ; } public double quotient ( ) { return numerator/denominator ; } } FitNesse uses a notation very similar to Selenese scripts and therefore can be easily integrated with the Selenium IDE framework. Xebium coupled with Selenium therefore creates a tool that harnesses the best features from both mentioned frameworks. Advantage of Selenium is great ability for testing a wide range of web browsers, while FitNesse has ability to easily create, store and maintain very readable tests. Tests created is Selenium IDE can be easily stored and launched by FitNesse and executed by Selenium WebDriver Tools Comparison The Table 3.6 clearly shows comparison of previously discussed frameworks. Considering the fact that all mentioned frameworks are very similar tools offering similar functionality it is difficult to determine the best choice. Selection of the well suited tool is determined by subjective preferences and needs of everyone who want to work with these tools. 31

39 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Supported platforms Supported programming languages Supported browsers GUI for test development Selenium Tellurium WebAii Telerik Test Studio Windows, Mac OS X, Unix Java, C#, PHP, Ruby, Python, Perl IE, Safari, Firefox, Chrome, Opera, HtmlUnit, Android, ios, Blackberry proprietary browsers. Selenium IDE for Firefox 2+ Windows, Mac OS X, Unix Windows Windows, ios 4+ WebTest Xebium Windows, Mac OS X, Unix Windows, Mac OS X, Unix Java, Groovy C#,.Net C#,.Net Java, Java, C++, Groovy Ruby, Python, Delphi, C# HtmlUnit IE, Safari, IE, Safari, Firefox, Chrome, Opera, HtmlUnit, Android, ios, Blackberry proprietary browsers. TrUMP plugin for Firefox 2+/3+ IE, Firefox, Chrome, Safari (only windows version) IE, Firefox, Chrome, Safari (only windows version), ios No support Complex GUI for test development + plugin for IE WebTest Recorder plugin for Firefox) Firefox, Chrome, Opera, HtmlUnit, Android, ios, Blackberry proprietary browsers. Selenium IDE for Firefox 2+ 32

40 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS Parallel test execution Itegrability with another autom. tools Technical Support Licensing and Price Selenium Tellurium WebAii Telerik Test Studio Selenium Grid Support for Selenium Limited Limited Support GRID Support Bromine, JUnit, RSpec, TestNG, Test::Unit, unittest, Ant, Maven Community and professional support, javadoc doc, updates every 2-3 weeks Open-source (Apache 2.0) JUnit, TestNG, Ant, Maven Community support, updates once a year Open-source (Apache 2.0) NUnit, MbUnit, XUnit Community and professional support, good doc, updates tests Telerik online and community support, regular updates regularly Free for commercial use (not opensource) NUnit, MbUnit, XUnit, WPF, performance Commercial product, 2499$ for a single license Table 3.6: Tools Comparison WebTest Xebium No Support Selenium Grid Ant Fitnesse limited support, online doc, last update in 2009 Community support, regular updates Opensource (Apache 2.0) Opensource 33

41 3.3 Other useful tools 3. TOOLS FOR AUTOMATED TESTING OF WEB APPLICATIONS These tools can be used in process of creating, debugging and deploying automated tests. They offer variety of features such as simple localization and selection of the elements present on the web page. This feature is especially beneficial if it is necessary to obtain identifier of the element needed for creation of automate tests. Second category of tools mentioned here are systems for continuous integration designed to create, maintain and monitor repeated tasks Firebug Firebug is a plugin to the Mozilla Firefox web browser. It is very popular tool used to develop and debug websites. It is the open-source project licensed under BSD license. The basic functions are[8]: Review and modify HTML. Review and edit CSS styles. Network monitor. Debugger for JavaScript. Review DOM model of the site. Interactive debugging in a GUI FirePath Is a Firebug extension that adds the development tool designed to generate, edit and review of XPath expressions, CSS 3 and JQuery selectors[13] Hudson Test automation requires introduction of the system for continuous integration (CI). System for CI should support source code revision systems and it also should contain some kind of the task scheduler for the automatic task execution. Among tools like that belong also Hudson 22. This system supports the project version control, application build tools (e.g. Ant), scheduling, different execution mechanisms and straightforward GUI that enables users to monitor executed tasks and review tasks reports. The tool provides many different plugins that can be used to enable integration with another tools as for instance the plugin for the TestNG or Selenium framework

42 4 Blackboard Xythos Document management system Xythos is a document and file management system that empowers organizations to securely access, manage, and share content over the Web. Xythos family of software tools manages document creation, facilitate collaboration, and manages access to ensure that the right people can view and modify paper and electronic documents. The Company was founded in 1999 and acquired by Blackboard inc. in Today Xythos products have over 400 customers and over 3 million users worldwide. Xythos key markets include higher education, scientific research and government. Xythos offers wide variety of products, but this thesis will focus on Xythos EDMS (Enterprise Document Management Suite) and Xythos DL (Document Locker) server with the web-based interface. The only difference between these two products is only in features included. Xythos EDMS contains full-scale of features while the Xythos DL is lightweight version that offers only basic DMS functionality. This chapter describes the architecture of web application interfaces chosen to be automated and the process of development and testing of the Xythos application is discussed. 4.1 Application Description Xythos server is an web-based application with multiple user interfaces. Supported browsers are Internet Explorer, Mozilla Firefox, Google Chrome and Safari. The system is developed using Java and supports Oracle, Postgres and Microsoft SQL Server databases. Web based interfaces that are part of the server side application Xythos document management system are described in more detail in the Appendix A Product Development and Quality Assurance Xythos server team is dealing with the continuous development of a single product, the maximum capacity of programmers and quality assurance engineers is used in the first place to correct errors detected in the previous version, and secondly to the development of new features and new test definitions. The product development team currently consists from 8 programmers, five QA engineers and 1 person maintaining documentation. It implies that entire development process of the new version must be finished in few months. This process is divided into two parts : Product Development Phase 35

43 4. BLACKBOARD XYTHOS DOCUMENT MANAGEMENT SYSTEM Quality Assurance Phase Each of those phases consists from different tasks that must be finished correctly before the new version is released Product Development Phase During product development phase developers work according to classic development methodology similar to waterfall. So the development of the new version is only one iteration in which new features are implemented and old issue found in previous versions are fixed. During this phase the QA team works on the analysis and the development of new test scenarios and newly the implementation of automated test cases. From the perspective of testing phase of development new version of product responds to unit testing. This automated tests are managed and executed by the developer team and the QA team plays no role in this test execution Quality Assurance Phase The QA phase or the test phase for the new product version is initiated when all the new functionality and at the same time all issues from previous versions scheduled for the new version of the product are fixed. This phase of the product development consists of several parts, which together last for approx. 4 months. In first part all newly added functionality is tested. Second stage is the bug verification which together with regression testing using predefined test scenarios continuously follows the first stage. In the bug verification phase all old issues fixed in the new version are verified. That means all issues were correctly fixed with relation to the specification. In the phase of regression testing the QA team verifies that error corrections have not introduced another issues. This means that no regressions were introduced by bug fixes. These two parts are executed in two cycles to improve robustness of tests and to enhance the test coverage. Regression testing is performed manually on basis of prearranged test scenarios. Each scenario corresponds to the part of the functionality featured by the Xythos server product. To facilitate the work with test scenarios each scenario contains document that includes information as : Objective - the test scenario description and the info about functionality that is covered by this test case Not Included in this Test Case - functionality not covered by the test case 36

44 4. BLACKBOARD XYTHOS DOCUMENT MANAGEMENT SYSTEM Dependencies - reliance on other test scenarios Included in Test Case - this part contains description of the individual use case and test definitions included in them. Each test definition has defined title, description and expected result. This document is just outline of the test case and does not include exact steps needed to perform the particular test definition. It contains only basic information that facilitate work with test cases. Complete set of test definitions is maintained using Silk Central 1. The regression test phase performed Figure 4.1: example of the outline for test case Copy_EDMS manually using test scenarios described above is the most time-consuming phase. Currently our test suite includes more than 100 test scenarios and more than 2800 test definitions. One cycle of manual regression testing takes 4 testers and 20 days of the work for one test configuration. In an optimal case all of regression test cases should by executed on all possible configuration of the test environment. Each of test environments consists from the version of operation system, the database and the web browser. From the above it follows that it is not in ability of a small QA team to carry out all manual pre-prepared test definitions on all possible system configurations during 2 months of the testing phase. This fact was one of reason why it was decided to create the automation test framework and automated test cases. 1. Silk Central is a test management tool used to create, store and execute test cases for manual testing 37

45 5 Xythos Test Automation Project The aim of this thesis is among other things creation of the framework that will be consequently used for the development, execution and evaluation of automated testing of the Xythos application s interfaces described in the previous section. The actual implementation should be preceded by a more detailed analysis of the requirements and risks that can affect the project. First off all, based on theoretical knowledge from the first chapter, we need to determine what kind of testing will be performed. In this case it is a form of regression testing, whose main purpose is periodically verify that the current behavior and functionality of the application is correct also in the case of changes within another part of the system. Correct processing and displaying of data will be verified and also whether the application meet customer s expectation will be validated. All tests shall be performed on the running system therefore it is dynamic testing. Testers will act as an application user without any knowledge of application source code, but since they will use HTML code of the page to write tests, it is testing of the gray box. 5.1 Test Automation Motivation and Requirements Motivation to supersede the manual regression test suite by automated tests is: to reduce time that QA team spends on the execution of boring and time-consuming tests and to increase time spent on more important work as for example exploratory testing; to provide faster and frequent feedback for programmers. This will be ensured by deployment of automated test on the system for CI. Time needed to perform regression tests on application should be reduced from months to weeks or ideally days; to improve re-usability of tests and efficiency of the test execution; to increase the test coverage for different test environments; to set up liable process for the test case update and documentation. Before these goals can be fulfilled an appropriate analysis which reveals strengths, weaknesses of the project as well as conditions for the successful implementation and deployment of tests should be performed. For this purpose the SWOT analysis [10] was used. In case of the Xythos test automation 38

46 5. XYTHOS TEST AUTOMATION PROJECT project, factors described in the Table 5.1. could affect the development and the successful release of automated tests. Strengths Weaknesses Opportunities Threats Factors affecting the project the stable user interface that remains almost unchanged, the application covered by unit tests, Team members with previous experience with test automation, Support from the team of programmers small QA team, not all members of QA team have programming skills, high initial costs, existing regression test suite is outdated and contains many duplicity so test cases must be reworked before they will be automated reduce the amount of manual tests and use capacities for different QA tasks, extend the test coverage, extend number of test executions on different environments. the automated test framework can be used to create tests also for developers or to verify hot patches delivered by the Xythos Server Team, faster feedback for programmers as the automated regression suite can be executed faster and more often by using the CI possibility of big changes in user interfaces in future, incompatibilities and internal defects of tools used for the automation Table 5.1: SWOT analysis The development and deployment of automated tests is a difficult process, that entails various risks. Risks may vary depending on the project however there are some risks that are typical for the deployment of automated tests of user interfaces. These risks include[6]: Test running too long. This is a price for complex testing of user interfaces. Possible solution for this problem is to minimize run-time of tests by using parallel test execution. Tests can be parallelized by using appropriate tools and the appropriate test case design. Individual tests should be short and independent to each other. Inability to test the desired functionality. Each of used tools has some limitations that must be identified prior to the test development to be able to respond to them in a timely manner. 39

47 5. XYTHOS TEST AUTOMATION PROJECT Problematic test modification and maintenance. Automated tests for user interfaces are usually very brittle and they must be modified every time the tested user interface is changed. A solution to this problem is to create a test framework that allows to react to this modifications quickly and easy. Many of mentioned problems can be minimized by using appropriate tools as well as by application of appropriate techniques for creating and executing of automated tests. The following section will therefore deal with the selection of tools that will be subsequently used for the implementation of automated tests within the scope of the test automation project. 5.2 Selection of Technology and Tools The process of selecting particular tools is dependent on various factors that influence the final selection. The selection must be adapted to needs of the particular project as well as to the team involved in the test automation development. Risks mentioned in the previous section should be taken into the account and appropriate tools that can overcome these risks should be selected. In the case of the Xythos test automation project following conditions that must be met by selected tools were chosen: Selected tools must be easily integrated with existing testing and development infrastructure. That especially means that tools must be based on the Java platform as the automation team has Java programming skills and the product for which automated tests are developed is also written in Java. Selected tools must be easily integrable into the continuous build process. Selected tools must support testing of web applications that use the AJAX technology. Selected tools must be free to use for commercial purposes. Ideally open-source. Selected tools must be continuously developed and updated. Selected tthe ools also should have good and accessible documentation. Selected tools must support the parallel test execution to reduce the risk of long running tests. 40

48 5. XYTHOS TEST AUTOMATION PROJECT According to these conditions it was decided that the most appropriate tools that meet mentioned criteria will be the combination of tools: Selenium (Webdriver, Selenium Grid) + TestNG with ReportNG plugin + Perforce 1 + Ant + Hudson. Tools FireBug and FirePath was used to simplify work with the DOM of web pages. This combination met all specified conditions although using WebDriver has some limitations as: WebDriver cannot handle Java applets. it is not possible to control browser dialogs (e.g. Save File dialog). It was decided that these limitations are not a big issue to the automation project although they cause that it will not be able to automate some functionality. Despite these limitations, by combining selected tools we get the comprehensive framework that can be used for creating and executing automated tests. Tests are created by using Selenium and executed by TestNG. Ant is used to build the framework independently on the development environment an to deploy it to different testing environments where tests can be automatically build, performed and monitored by Hudson. Automated tests and the whole testing framework is stored and maintained in Perforce. 5.3 Test plan Prior to the development and deployment of automated testing it is necessary to determine the extent to which the application will be covered by automated tests. 100% tests automation is not necessary, and in some cases not even feasible, so it must be decided which functionality will be covered and which will not. As described in the Chapter 4 the application under test consists of three web-based user interfaces (AdminUI, WebUI, MobileUI). Each of these interfaces offers different functionality and it is used by different amount of users. It was therefore necessary to decide whether and how they will be covered. AdminUI - This interface is used only by administrators and therefore it was decided not to cover this interface by automated tests. 1. Perforce is a commercial, proprietary revision control system developed by Perforce Software, Inc. 41

49 5. XYTHOS TEST AUTOMATION PROJECT However the automated control of the interface must be implemented in the automation framework, because the functionality will be needed for automated test setups. This means that it will be used for automated creation of test users, import of test data or to set up system parameters for purpose of automated testing. WebUI - From automated tests perspective this is the most important interface. The functionality of the interface will gradually be covered with automated regression tests. Since this is a complex interface with large number of features the automation process will be divided into steps that will be described later in this section. This interface should be covered by the regression test suite as far as possible. MobileUI - This interface is intended to be accessed from mobile devices and therefore offers only the basic functionality. Automated user interface testing has already been used during the implementation phase of the first version of this web interface. The automated user interface tests were used as acceptance tests. Thus created test suite was smoothly transformed to the regression test suite. This interface will be fully covered by the automated tests. Because of the scope the project cannot be completed at once and therefore the project was divided into phases. This phases are mainly performed during the product development phase described in the previous section. Phase 1 - survey of available tools, tools selection, redesign manual test cases for test automation, framework design and implementation of the basic functionality for AdminUI and WebUI, creation of the sanity test suite 2, creation of upgrade tests 3, implementation of the manual test execution (using scripts) Phase 2 - implementation of the full functionality for AdminUI, WebUI and MobileUI, implementation of tests for full regression suite (this will be completed in phase 3), creation of the environment for automated test execution, deployment of the system for continuous integration. Phase 3 - completion the full regression test suite, implementation of the Master Dump Approach to automated testing. This approach is based on the creating database dump that contains all data needed 2. This test suite contains tests that verify only basic functionality of the product. 3. This tests covers regression testing of the application after upgrade from the previous version. They verify that all setting, users and data remain same after system is upgraded 42

50 5. XYTHOS TEST AUTOMATION PROJECT to test set of issues reported by customers. This dump will be subsequently used to perform automated test suite based on the same customer issues. This thesis deals only with the first and second phase of the automation project. 5.4 Project team members and responsibilities The test automation project have involved members of the QA team, one member of the development team and member of the IT department. It was however not necessary to involve all QA team members from the beginning of the project as they were not expected to have profound Java programming skills. Therefore there was two members of the QA team that participated on the design and implementation of the framework since earliest stages of the project. These team members was managed by one programmer, whose main responsibility was to help to design framework and to review the implemented code. The QA team manager was responsible for the whole project management. Other members of QA team joined the project later. They participated in the process of rewriting old manual tests that will be automated, in the process of writing project documentation, on further development of the framework and on writing automated tests themselves. It was also necessary to create an environment that will be used for the test execution. This part of the project was covered by QA team members with cooperation of the IT department. Personally I participated on the framework design and implementation. My primary tasks were to implement functionality for the Admin user interface, part of the functionality of WebUI interface and also automated test cases themselves. I created part of utility scripts used to install and configure application under test. I also participated in process of creating and configuration of the environment that will be used for the test execution. 43

51 5.5 Framework Description 5. XYTHOS TEST AUTOMATION PROJECT Tools described in the previous part were used to have the flexibility for customization that was required for the application under test. The automated test framework includes 3 layers as shown on the Figure 5.1. Layer 1 - Wrapper for WebDriver Layer 2 - Page Access Layer Layer 3 - Test Layer Figure 5.1: Framework description The first layer includes classes that are used as a wrapper for the Selenium Webdriver (WebDriver, WebElement, Alerts). This layer also contains classes such as loggers or classes used to process configuration files (server and client configuration files). The second layer, the page access layer, contains code used to access individual web pages. For example the tested web application contains a log-in page, a create user page, a copy file page etc. Classes written in java contain the code that could perform functions in the tested application, such as a method to log-in to the application, a method to create user, or a method to copy files. These classes do not contain any data needed for testing. This means that the class used to log-in users does not know particular user name and password. For the purposes of this project these classes are called modules. The third and top layer is the test layer. Basically this layer contains all test cases with all data needed to perform tests. The Layer 3 methods call Layer 2 methods, which in turned called Layer 1 methods. For example the actual test would call logintogrid(username, password) and pass the username testuser1 and the password secret. This means that many different data sets can be fed in easily. Functional tests use Java and Selenium to control the DOM of the web browser and can access almost all objects on the page. 44

52 5. XYTHOS TEST AUTOMATION PROJECT Figure 5.2: Detailed framework structure On the figure are described connections between particular framework layers. Test cases on the third layer use BaseTestCase to initialize WebTestSesion as described on the diagram by using blue arrow. The WebTestSession is than used to instantiate all modules that are used directly in test cases to perform action on the tested web page. Actions implemented in modules than directly call methods from WebTestSession located at the first layer. Black arrows symbolize inheritance of the modules structure. The connection between third layer and second layer is not depicted in the diagram to keep diagram simple and straightforward. The WebTestSession implements the interface ModuleManager this connection is depicted on the diagram as the yellow arrow Layer 1 - Webdriver wrapper Classes belonging to the Layer 1 are the core of the framework. They are located in the package common. The most important class is the WebTest- Session that serves primarily as a wrapper for API methods from package Selenium WebDriver and WebElement. All of basic methods needed to control and operate web browsers are implemented here. One of the methods is described on the following example. 45

53 5. XYTHOS TEST AUTOMATION PROJECT Listing 5.1: wrapped WebDriver method protected WebElement findelementby ( By p_by ) { t r y { return m_webdriver. findelement ( p_by ) ; } catch ( NoSuchElementException l_ex ) { // FindElement throw exception NoSuchElementException when element i s not found. return n u l l ; } } WebTestSession also serves as the module manager that instantiates modules used in a particular test case to perform actions needed for testing. The modules will be discussed closely in the next section. Another important part is BaseTestModule this class is the parent of all modules and provides and extends basic operation from WebTestSession as shown on the next example. Listing 5.2: wrapped WebElement method public void c l i c k ( By p_by ) { WebElement l_element = getelementbywait ( p_by ) ; Assert. asserttrue ( l_element. isenabled ( ) && l_element. isdisplayed ( ), "Element " + p_by + " is NOT enabled or displayed ( cannot click)!" ) ; l_element. c l i c k ( ) ; Log. l o g I n f o ( "Clicking on element " + p_by + "." ) ; } Main classes and relations between them are described on the Figure 5.1. In addition this package contains classes needed to log actions and results collected during the test session, class that takes screen-shots of the browser page that was opened in the moment of the test failure (TestListener) and other utility classes needed for testing purposes. 46

54 5. XYTHOS TEST AUTOMATION PROJECT common WebTestSession - filemanager : FileManager - webdriver : WebDriver - confparams : ConfParams - store : Store + WebTestSession(ConfParams : confparams, WebDriver : web- Driver, Store : store) + getfilemanager() : FileManager + getserverconf() : ServerConf + getbrowserconf() : BrowserConf # findelementby(by : by) : Web- Element # findelementbywait(by : by) : WebElement # rightclick(by : by) # quit() # open(string : url) # close() <<uses>> <<interface>> ModuleManager + getauthentication() : Authentication + getfilemanager() : FileManager + getfilesystem() : FileSystem <<abstract>> BaseTestModule - webtestsession : WebTest- Session # getmodulemanager() : ModuleManager # typetoelement(by : where, String : text) # iselementpresent(by : where): boolean # getelementvalue(by : by): String Figure 5.3: Layer Layer 2 - Access modules for application pages Basic operations implemented in BaseTestModule are subsequently used to perform more complex operations that are provided by classes on the Layer 2. These classes are called modules. Each module offers functionality that is used to test particular feature in the Xythos document management server. For instance the module Authentication provides methods to log-in and logout user from WebUI. This methods are shown as the Listing 5.3. Methods typetoelement and clickonbutton are provided by the parent base module. 47

55 5. XYTHOS TEST AUTOMATION PROJECT <<abstract>> BaseAdminModule # navigatetoadmin- Menu(AdminMenu : which) FileSystem + addtld(string : tld, String : vs, String : owner) <<abstract>> BaseFileManager # getentriescount() FileManager - wizzard : Wizzard - dialogop : DailogOperations - search : SearchManager - bookmarks : BookmarkManager + docopy(string : dest) <<abstract>> BaseTestModule - webtestsession : WebTest- Session # typetoelement(by : where, String : text) <<abstract>> BaseWebuiModule # getactualdirectory() <<abstract>> BaseBrowserModule - registerbrowserwin : Map # closebrowserwindow(string : pagename) SearchManager + doquicksearch(string : arg, String msg) Figure 5.4: Layer 2 CommonUtils + getinputvalue(by : by) <<abstract>> BaseMobileuiModule # opencontextmenu() MobileDetails + createnewdefaultticket() Authentication + logintogrid(principal : user) + logout() Dropbox + addsubmitters(principal : submitters) 48

56 5. XYTHOS TEST AUTOMATION PROJECT Listing 5.3: Layer 2 method for the user log-in public void login ( S t r i n g p_username, S t r i n g p_password ) { typetoelement (USERNAME_INPUT, p_username ) ; typetoelement (PASSWORD_INPUT, p_password ) ; clickonbutton ( CommonButton.LOGIN_PAGE_OK_BUTTON) ; Log. logpass ( "User \"" + p_username + "\" is logged into WFS." ) ; } Modules are initiated by each particular test case when functionality provided by the module is needed to be performed in the test case. UML diagram on the Figure 5.4 shows how modules and base module abstract classes are inherited Layer 3 - Test cases On the top of the implemented framework lie classes whose main purpose is to perform automated tests itself. Each test case is represented by class from package testcase that is a descendant of the super class BaseTestCase. This class provides common methods and settings for the test case execution using the TestNG framework. This class contains methods that are annotated using annotations from This is not complete list of annotations that can be used, complete list can be found in TestNG documentation[1]. Listing 5.4: start and termination of the test ( { "defaultbrowsertype", "browsersfile", " serverpropertyfile", "wfsversion", " url" } ( alwaysrun=true ) public f i n a l void beforeclass ( S t r i n g S t r i n g p_browsersfile, S t r i n g ( DEFAULT_WFS_VERSION_TO_REPLACE) S t r i n g ( "" ) S t r i n g p_ url ) {... ( alwaysrun=true ) public f i n a l void a f t e r S u i t e ( ) {.. } Methods with are executed always before (after) the test case is executed and are used to create WebTestSession 49

57 5. XYTHOS TEST AUTOMATION PROJECT and setup the configuration for test cases. Methods shown on the Listing 5.4 use also the that allows to read parameters defined in the testng.xml configuration file. This file serves as the main configuration file in which all parameters needed for the test execution are defined. This class also provides access to ModuleManager that is used to access Modules from Test Case classes as shown on the Figure 5.5, where basic relations among TestCase, WebTestSession and Modules are described Test case structure Test cases are main classes that represent automated tests for execution. As mentioned in Chapter every automated test consists from components setup, execution, analysis, reporting, cleanup and help. Test cases were designed to conform to this definition as far as it is possible. Each test case therefore consists of: Setup Setup is the first action before actual test definition can be executed. It can be divided into two stages: modules setup enables all modules needed to perform test definitions. test case setup creates all data needed for testing (e.g. test users, test data, system settings) Modules are initialized as shown on the Listing 5.7 Listing 5.5: Module initialization P r i v a t e Authentication m_authentication = getdefaultmodulemanager. getauthentication ( ) ; Execution Each test case consists from atleast one method marked with the TestNG These methods perform testing as shown on the Listing 5.8 Listing 5.6: automated test definition import com. xythos. q a t e s t. common. P r i n c i p a l ; public s t a t i c f i n a l P r i n c i p a l USER1 = P r i n c i p a l. createuser ( "testuser" ) ( groups = { "sanity" } ) 50

58 5. XYTHOS TEST AUTOMATION PROJECT public void testcopyfileintosamefolder ( ) { Log. logtd ( "Copy and verify default name of file copied into the same folder" ) ; m_ browserutils. openwebuipage ( ) ; m_authentication. logintogrid (USER1) ; m_filemanager. docopyandverifymessage ( FILE_COPY, COPY_DEST) ; asserttrue ( m_filemanager. i s E n t r y P r e s e n t ( FILE_COPY1 ), " Entry is NOT present!" ) ; } The test performs actions provided by modules (in this case module Authentication and FileManager). The first action is logtd(...) that ensures proper log of the test definition present in the test result. Another methods perform openwebuipage(), logintogrid(...) and docopyandverifymessage(... ) actions. At the end we have to verify that copy of the file is present in the file manager grid. This action is ensured by the assertion method that is implemented in BaseTestModule. Each test can be assigned to a group. Grouping can be used for instance to divide tests included in the test case to different test suites. Reporting and Result Analysis in the Section 5.7. This part will be discussed in more detail Cleanup After all tests are finished (succeeded or failed) the test cleanup must be done to ensure that all data created by the test are deleted and the system is restored into its default state. This must be done after the finish of each test case to guarantee that tests will not interfere with each other. Help As it was discussed in Section each test case used by the QA team has defined outline. This document contains information about test definitions and the test case itself. Originally outlines were stored in Silk Central, after the test case is automated we no longer maintain copy of the test case outline there. Instead of that the outline is moved to the automated test case and transformed into the JavaDoc documentation. By using this approach we ensure that the test case can be easily maintained and updated. Example of the test case with attached outline is shown in the attachment A.2. The example also shows complete automated test case with all components discussed in this section. 51

59 5. XYTHOS TEST AUTOMATION PROJECT testcase <<abstract>> BaseTestCase - webtestsession : WebTestSession - webtestsessionlist : Collectio<WebTestSession> - serverconf : ServerConf - browsertype : BrowserType - browserconfmap : Map<BrowserType, BrowserConf> + beforeclass(string : param...) + afterclass() + beforesuite(string : runtimefile) + aftersuite() # getdefaultmodulemanager() : ModuleManager # getnewmodulemanager (browsertype : BrowserType) : ModuleManager # quitmodulemanager (modulemanager : ModuleManager) - createnewwebtestsession (browserconfig : BrowserConf) TestCase - module : Module + classcleanup( + classsetup() + method- Setup(method : Method) + method- Cleanup(method : Method) + modulessetup() + testcasesetup() + testcasecleanup() + UC_usecaseName() FileManager <<interface>> ModuleManager Authentication FileSystem instantiate 0..1 initiate 1..* WebTestSession Figure 5.5: Layer 3 52

60 5.6 Reporting and Results Analysis 5. XYTHOS TEST AUTOMATION PROJECT Each execution of the test case produces a report from the test case execution. TestNG with ReportNG is being used for test case reporting. Currently two kinds of reports are implemented: detailed and simple. Here is the example of both kinds of supported reports. The reports belong to a Demo test case that is performing activities described on the Listing 5.7. Listing 5.7: steps performed in demo testcase Setup: Create user Create r o l e f o r Dropbox and Wiki Test D e f i n i t i o n "demo" : Create f o l d e r AAA Create tag "Demo Tag" f o r f o l d e r AAA Create f o l d e r F1 Copy f o l d e r F1 i n t o f o l d e r AAA Verify b a s i c permissions on f o l d e r AAA f o r owner and global groups Create wiki1 Check i f f o l d e r BBB i s a v a i l a b l e > a s s e r t generated because f o l d e r BBB does not e x i s t Test D e f i n i t i o n "createfolder" : c r e a t e f o l d e r Folder1 Turn Logging On. Create Comment1 and Comment2 on the folder, s e t storage quota value f o r Folder1 Test D e f i n i t i o n "copyfolder" Copy Folder1 i n t o home l o c a t i o n as Folder2 Cleanup: Delete user created, d e l e t e r o l e created The simple variant of the report provides the log of activities corresponding to the description of the test case provided on Listing 5.7. The format of the log should correspond to steps as they are stored in manual test cases in Silk Central. This variant of the report should be used within the process of execution of automated test cases. Based on the report the potential issue can be reported with minimal effort needed in term of the investigation of the source of the issue. The report should always contain a screen-shot of the user interface from the moment of the test failure. Simple report shown on the Figure 5.6 contains all logged steps performed during the test. In case of the test failure it contains the assertion error that has caused the test failure. 53

61 5. XYTHOS TEST AUTOMATION PROJECT Figure 5.6: Simple report 54

62 5. XYTHOS TEST AUTOMATION PROJECT The detailed variant of the report provides same logged activities as the simple variant. In addition detailed information about basic actions performed by the framework are logged. This information are logged mainly for the purpose of the detailed analysis performed in phase of the test case implementation. An example of the detailed test report is shown on the Figure 5.7. Additional information that is logged is mainly information about elements that were clicked or used during the test case execution. Logging of the additional info can be configured by parameters passed to the test execution in the run-time configuration file. This file and its usage is described in the next section. In case that some test fails it is necessary to analyze the cause and to perform manual testing of the failed test scenario. This must be done to determine actual reason of the test failure due to fact that automated tests are characterized by high occurrence of false alarms. In case of the confirmation that the issue is real this must be reported. Figure 5.7: Detailed report 5.7 Test Execution and Continuous integration Automated test executions are not possible without the proper configuration and build of the whole test project. This tasks are ensured by TestNG 55

63 5. XYTHOS TEST AUTOMATION PROJECT framework and Ant. TestNG uses the file testng.xml where all tests and test suites intended for the test execution are defined. This file also contains additional parameters needed to perform tests. In the case of this project, TestNG configuration files contains parameters: runtimefilename - this parameter specifies the file that contains run-time properties for the given test suite. defaultbrowsertype - this parameter specifies the browser type needed to create proper instance of the WebDriver. browserfile - this parameter specifies the external configuration file that contains browsers that can be used for testing. It also defines where tests will be performed. It can be locally on the same instance as server resides or remotely using Selenium Grid as shown on the Listing 5.7. serverpropertyfile - this parameter specifies the file that defines the server instance where tests will be executed as shown on the Listing url - address of the server used for testing. automated tests and test suites to execute. Listing 5.8: browser file <browsers> <browser type="ff" drivertype="remote" u r l ="http: // :4000/wd/hub" localdatapath="t:\ Automation\" /> <browser type="ie" drivertype="remote" u r l ="http: // :4000/wd/hub" localdatapath="t:\ Automation\" /> </browsers> Listing 5.9: server property file hostname = qas aut wfs. brno. l o c a l os = linux ip = db = postgres u r l = h t t p : //qas aut wfs. brno. l o c a l : / testdatadirpath = /usr/ l o c a l /xythos/testdata/automation/ logdirpath = /tmp/ indexingdirpath = xythoslogdirpath = 56

64 5. XYTHOS TEST AUTOMATION PROJECT An example of the textng.xml file is shown on the Listing 5.9. The example shows all mentioned parameters and also another parameters that are important for the test execution. Listing 5.10: example of testng.xml <! DOCTYPE s u i t e SYSTEM " > < s u i t e name="test Case" preserve order="true" p a r a l l e l =" false" thread count="3"> < l i s t e n e r s > < l i s t e n e r c l a s s name="org.uncommons.reportng. HTMLReporter"/> < l i s t e n e r c l a s s name="org.uncommons.reportng. JUnitXMLReporter"/> </ l i s t e n e r s > <parameter name="runtimefilename" <! The r e l a t i v e path f o r f i l e with runtim parameters. I. E. value="/runtime.properties" > < t e s t name="@tc@"> <parameter name="defaultbrowsertype" value="@browser@ "/> <! Browser type s t r i n g > <parameter name="browsersfile" value="@browsersfile@" /> <! The r e l a t i v e path f o r f i l e with browser configuraton. I. E. value="/testngconfig/clients/ localbrowsers.xml" > <parameter name="serverpropertyfile" <! The r e l a t i v e path f o r f i l e with server configuraton. I. E. value="/testngconfig/servers/ aut72sp5.prop" > <parameter name=" url" value=" brno.local/qamail"/> < c l a s s e s > < c l a s s name="com.xythos.qatest.testcase.@tc@"/> <! The peckage and TestCase name I. E. webui. DCM_FS_NewFolder_DL > </ c l a s s e s > </ t e s t > </ s u i t e > 57

65 5. XYTHOS TEST AUTOMATION PROJECT The tool Ant is used for the automated compilation and build of the test project. Build process is configured using the XML file build.xml. This file contains definitions that determine the path to the directory structure for the project and to include required libraries. To perform a complete build process of the project it is necessary to preform particular steps. These steps are called "targets" and they perform all tasks needed for the project build. Listing 5.11: example of testng.xml <?xml version="1.0" encoding="utf-8"?> < p r o j e c t name="qa Automated Tests" basedir="." d e f a u l t =" TestNG"> <! Parameters from command l i n e > <prop erty name="testng.xml.file" value="${param. testngxml}"/>... <! Parameters > <prop erty name="testng.configfiles.dir" value=" testngconfig"/>.... < t a s k d e f name="testng" classname="org.testng. TestNGAntTask"> <! Directory path of compiled c l a s s e s ( i. e. c l a s s ) and necessary l i b r a r i e s > <path id="classes"> <! Clean build d i r e c t o r y > < t a r g e t name="clean"> <! Compile t e s t automation p r o j e c t > < t a r g e t name="compile" depends="clean"> <! Create t e s t n g. xml> < t a r g e t name="createxml"> <! Run t e s t case > < t a r g e t name="testcase" depends="createxml"> <! Update information in t e s t n g. xml f i l e > < t a r g e t name="updatexml"> < t a r g e t name="testng" depends="updatexml"> <! One minor modification. with l i n k to r e s u l t > < t a r g e t name="modifyresults" depends="testng"> <! Task to send n o t i f i c a t i o n a f t e r the t e s t i s f i n i s h e d > < t a r g e t name="send " depends="modifyresults"> </ p r o j e c t > The build.xml file used to build the Xythos Test Automation project contains targets to compile project, run tests and send s as shown on the Listing 58

66 5. XYTHOS TEST AUTOMATION PROJECT Besides these targets it also contains the target that creates and updates the testng.xml file according to parameters inserted by command line scripts used to manually execute automated test suites. These scripts are described in more detail in the next section Manual test execution To enable the test execution on different test instances it was necessary to develop scripts that are used to download, build and execute tests. These scripts can be used on both Unix and Windows based systems. The test cases can be executed by using these scripts: getsource - download the source code from Perforce compilesource - compile downloaded source code runsuite - run tests specified in XML configuration file runsuiteandsend - run tests and send results runtc - run specified test case. The script runsuite is used to execute the whole test suite that is defined in a TestNG XML configuration file. The XML file is passed to the script as one of the parameters. Other parameters used by this script are the server property file, the browser file, the run-time property file and the browser type on which tests will be executed. All other scripts are utilized similarly. By using these scripts automated test cases and test suites can be manually executed on test instances. Each test instance used for testing must contain installed Ant tool and configured Perforce client Automatic test execution Deployment of the continuous integration system into the process of test automation is very important step, because it allows central access to automated test execution and result analysis to the whole development team. The tool used to accomplish this goal within the Xythos test automation project is the Hudson server. All previously mentioned parts of the project are used to create Hudson tasks that perform testing and send information about test results. All test results can be then easily analyzed directly by using Hudson s graphical user interface. One of the possible architecture of the continuous delivery system is described on the Figure

67 5. XYTHOS TEST AUTOMATION PROJECT Figure 5.8: Automated test execution environment architecture This architecture uses different virtual machines to enhance performance and to simplify maintenance of the system. The source code repository is located outside of the testing environment. Hudson tasks use only connection into repository to download all parts of the project needed for the test execution e.g. the automation framework, automated tests, scripts. The project is then build using Ant and executed by Selenium. Selenium Grid is used to perform test remotely on different client environment configurations. This configurations include different browsers and theirs versions 60

68 5. XYTHOS TEST AUTOMATION PROJECT run on different operation systems. Supported browsers are Internet Explorer, Mozilla Firefox, Google Chrome, Safari, proprietary mobile operating system browsers - ios and Android. We also need to test different server configuration of the system under test. This configuration includes: The operating system on which the application under test is running (supported/tested operation systems are Windows, Unix, Solaris) The database system (supported are MSSQL, Postgres, Oracle) The version of the Xythos document management system whether to use Xythos EDMS or Xythos DL Each Hudson task is parameterized and can be performed on different test system configurations manually or automatically according to given schedule. 5.8 Framework Optimization During the process of creating the test automation framework and automated tests, I ran into several problems that can negatively influence tests run-time and their maintainability. These issues are related to techniques used to design and create tests. In the next section I would like to address these problems and propose possible solutions. These so far unsolved problems can also be seen as tasks for work that will be an continuation to this thesis Test case Optimization As a part of this thesis, portion of manual test scenarios have been reworked into automated tests. Number of reworked tests corresponds to about onetenth of test scenarios used for manual testing. Total number of automated test scenarios implemented as the part of this work is 182. This test scenarios form regression test suite that was deployed into process of automated testing. Run-time for all tests is 55 minutes for one test configuration and all tests are executed sequentially. However this time is not the total time needed to run all tests because each test contains setup and cleanup that takes a significant part of the run-time. In the case that all 1800 manual test definitions will be automated the test automation will no longer be effective. Run-time for whole test suite will be too long and effective integration into process of continuous integration will not be possible. To solve 61

69 5. XYTHOS TEST AUTOMATION PROJECT this problem the way how automated tests are created must be changed. Automated tests must be reworked so they can be executed in parallel to reduce tests run-time. The effective tests parallelization is possible only if tests are designed and created as a large set of short running test definitions instead of long running tests as they are created now. This means that the individual test definition should test only a narrow set of the application functionality and it should not contain more than 30 steps - user actions on the website. This value is only approximation that defines the boundary where it is possible to test the functionality of certain part of the application and test definitions do not take long time to be executed. Large set of shortly running test definitions can be run parallel in case that all test definitions are independent and do not interfere with each other. Tools used to run parallel test definitions can be TestNG and Selenium Grid. These tools offer functionality for parallel test executions that can be used with many advantages for automated regression testing. The current status of created tests is that each test case represented by one TestNG class is divided into use cases. Each use-case in represented by the method annotated Each usecase contains 1-14 test definitions performed usually by the same test user. Use-cases are independent in manner that different use-cases do not access same data. This means that the smallest test unit (method annotated annotation)is not on the level of the test definition but on the level of the use-case. The longest running use-case takes more than 8 minutes to run and contains more than 10 test definitions. All use-cases are performed sequentially and are not ordered. Figure 5.9 shows described test case design. By using this test design the time needed to perform whole test case equals to the sum of run-time for individual use-cases. This is negligible for test suites containing only small number of tests but increasing amount of automated tests will cause problems with long running test suites. Tests designed this way also cause different problem. If one of test definitions fails all other test definitions will not be executed and must be re-runned after the analysis of the failed test definition is performed. All this problems can be solved in several ways and I try to propose two alternatives as a solution to his problems. The structure of test cases remains as it is now, but we rewrite them to ensure complete independence of different use-cases. This means that each use-case will be performed by different set of test users. This allows all use-cases to be executed in parallel and run-time of test case will be reduced to the run-time of the longest use-case contained in the test case. By using this approach the run-time of 62

70 5. XYTHOS TEST AUTOMATION PROJECT Figure 5.9: Test-case design currently deployed regression test suite can be reduced from 55 minutes to 18 minutes. Disadvantage of this approach is that the test setup will have to be more complex and we will need to create more test data as before. The problem that wills not by solved by this approach is that there will be many not executed test definitions in failed use-cases. The structure of test cases will be changed so that created use cases will be further divided into particular test definitions. Each test definition will be represented as a method marked annotation. These test definitions will be independent tests. Alternatively dependencies among tests can be solved by using proper techniques offered by the TestNG parameter). This allows all test definitions to be executed in parallel and the run-time of whole test case can be in optimal case reduced to the run-time of the longest test definition included in the test case. Time saved by using this design can be even greater than in previous case. However the development costs spent on creating and deploying test cases would be much higher due to its complexity. 63

71 5. XYTHOS TEST AUTOMATION PROJECT Another way how to reduce run-time of the automated test cases is to reduce time spent on setup and cleanup of the data needed for automated tests. Currently all setups and cleanups are performed in the same way as tests themselves by using GUI of the tested application. These setups and cleanups occupying a significant portion of the total test case run-time. However run-time of this part of test cases can by automated by using directly application s API instead of GUI to setup and cleanup tests. This approach can also reduce the test case run-time Brittle automated tests GUI automated tests are in general characterized by the tendency to break down in case that the application interface is changed during the application development. During the framework development, I have encountered similar problems associated with even small changes of the user interface between versions of the tested application. Several changes on application pages claimed many changes of element locators used by methods in the framework. Selenium Webdriver framework offers several options for the web page elements location. It is possible to locate elements by using the element id, name, link text, tag name, CSS locator or using the XPath expression. The last mentioned option was used in the automation framework. Usage of XPath expressions ensures complex access to the elements displayed on the website, but it does require knowledge and skills in creating XPaths. The automation framework has been designed to minimize duplicity in XPath locators used within tests. This means that each locator that is present on the page of the web application will be created in the framework only once. This should facilitate subsequent maintenance and expansion of the framework. However in many cases it was necessary to use complex XPaths that were excessively bounded to the inner structure of the website. So even small changes on the site caused that the framework was not able to return correct element for the given locator. Creation of complex XPath expressions was caused by inadequate or missing unique element identifiers. Solution to this problem is close cooperation with programmers in process of the user interface development and respect for the principles and best practices for the development of web-based applications. Each element used on websites should have unique identifier. This may in the future facilitate the creation and maintenance of automated tests. Using unique element identifiers can simplify XPath locators and therefore can make test more robust because locators does not to be excessively bounded to the the inner structure of the website. 64

72 5. XYTHOS TEST AUTOMATION PROJECT Modules optimization The automated framework is designed as described in the Section 5.5. All modules that provide functionality to control different parts of the tested web application are located on the Layer 2. This modules contain three different types of methods. First are actions that can by performed by the test user on the page. Second are methods that can access different elements on the page and return their values and third type are methods used for the page layout verification. These methods however was not unified so they are implemented in many different ways. Many of them are applicable only to positive testing. For instance the method logintogrid() assumes that the log-in action will be successful. Methods implemented this way are therefore possible to use only in one case. If negative testing of the logging action is needed we must implement another method that will be used for negative testing. This causes that modules contain many very similar actions that are used only once. With the large number of very similar actions clarity and maintenance of the framework became more difficult. This problem can be solved by unification that will be performed on all modules by using these principles: Modules can contain only a single method for each action (e.g login- ToGrid(username, password)) that can be used for both positive and negative testing. This can be done by the introduction of the rule that each method that performs an action on the web page must be able to recognize the action result. It can simply return true in case that action was successful or false if some error has occurred. Module methods should also know how to handle the error messages that the application generates. For example, when invalid log-in name and password is passed to log-in page, the module that handles log-in should detect the error and then it should pass the problem back to the test. This means that same actions on Layer 2 can be used for positive and negative testing. In negative case Layer 3 test expects Layer 2 to return failure. The Layer 3 action then checks if the test failed for the correct reason and evaluates if the test should pass or fail. Any assertions should be present only on the test layer - If changes proposed in previous text will be implemented all assertions present in module methods that perform actions on the website must be moved to tests. No method on Layer 2 should make assertions. Assertion should be present only in tests or in verification methods. 65

73 5. XYTHOS TEST AUTOMATION PROJECT This ensures that only in tests we can decide if they succeeded or failed. Verification methods should be removed from modules and they should be placed in separate classes used exclusively for this purpose. This changes will ensure that modules will be smaller and easier maintainable objects that will contain only necessary methods for testing. 66

74 6 Conclusion The aim of this diploma thesis was to describe procedures and objectives related to automated testing of web applications based on a Java platform. The main task was to compare and evaluate tools designed for test automation that are available on the market and subsequently, based on this analysis, to design, implement and deploy automated tests for the Xythos document management server. The theoretical part discusses important concepts in the field of software testing such as basic axioms and definitions, categories of tests or techniques used for software testing. Special attention is devoted to automated testing and its difference from manual testing. The emphasis was on evaluating the advantages and disadvantages of automated testing as well as problems that are associated with this testing approach. At this point it should be emphasized that automated testing can significantly increase efficiency of testing, but it is important to consider the extent and type of automated testing that will be used. If development and deployment of automated tests are not properly evaluated it can bring more problems than benefits to the project. The final section of this chapter summarizes the testing approaches and the test automation suitability for each testing approach. Another task was to examine and compare tools designed for automated testing that are currently available on the market. These tools were then described and compared on the basis of criteria defined at the beginning of the chapter. Tools were divided into two groups. The first group was unit test frameworks and the second group was frameworks for automatic testing of web-based user interfaces. Each tool was described according to comparative criteria. The final section of the chapter summarizes the criteria for evaluation in the form of a clearly arranged table. After the final evaluation of tools, I came to the conclusion that it is not possible to clearly determine which of the compared tools is most suitable for automation because due to their similar purpose they offer almost the same functionality. Selection of the suitable tool is determined by requirements that vary for different test automation projects. The main part of the thesis dealt with the design, implementation and deployment of automated regression tests of the web application Xythos document management server. Before implementation itself automation requirements were specified and a risk analysis for the whole automation project was conducted. After this preliminary analysis suitable automation tools were selected. The automated tests need to be flexible and easy to 67

75 6. CONCLUSION maintain. Selenium and TestNG frameworks were used successfully when the framework was developed using good coding practices, just as if it were a production application. Time was put into developing the libraries so that there was not a lot of reworking required or duplication in the code, and necessary changes could be made in one place. The framework was designed to simplify writing automated tests and to ensure their robustness. The robustness of created tests relies on various mechanisms of element localization on the tested web page. Changes in the internal structure of the web page or in elements can negatively affect tests. It is therefore necessary to design automated tests and test frameworks to deal with these changes. Linked to this is the development process itself. Use of unique identifiers for all elements of all web pages is required. Compliance with this rule facilitates greater test robustness because element locators used in tests do not have to be strongly connected to the inner structure of the web page. Making the tests and framework easy to maintain increases the return on investment for automation projects. The implemented framework and automated tests were just one part of the project. The last part of the project was to deploy the created tests. This was done by using Hudson as a server for continuous integration. This tool enables all members of the team to control execution of the test suites and to analyze results after tests are performed. Deployment of the automated test suites did not bring a significant increase in new issues detected during regression testing. This fact could be assumed on the basis of the information presented in Section Most of the bugs found by automated testing were detected in the process of creating the tests themselves. My work has contributed to the clarification of issues related to test automation and to those of the tools used to create and execute automated tests. Another asset is that this work will serve as a documentation of the entire project that has been missing until now. This work also serves for presentation purposes. Automated tests implemented in the context of this work will be used for automated testing of the Xythos document management server and they will be continuously developed. Further continuation of the project is the implementation of remaining test suites. However, the increasing amount of deployed tests raises other problems related to user interface test automation. These problems are mainly the long run-time of automated tests for the user interface, and problems associated with the maintenance of the automation framework. A solution to these problems was proposed in Section 5.8 and can be implemented as a continuation of this work. 68

76 Bibliography [1] BEUST, C.: TestNG - Welcome. 2004, [online], [cit ]. Available at: [2] CRISPIN, L.; GREGORY, J.: Agile Testing. A practical guide for testers and agile teams. Boston: Addison Wesley, 2009, ISBN , 533 s. [3] DIJKSTRA, E. W.: The humble programmer. 1972, [online], [cit ]. Available at: [4] GALIN, D.: Software Quality Assurance: from theory to implementation. London: Pearson Education Limited, 2004, ISBN , 590 s. [5] HOFFMAN, D.: Test Automation Architectures: Planning for Test Automation. 1999, [online], [cit ]. Available at: [6] KANER, C.; BACH, J.; PETTICHORD, B.: Lessons Learned in Software Testing: a context driven approach. New York: Willey, 2002, ISBN , 286 s. [7] IEEE Standard Glossary of Software Engineering Terminology. 1990, [online], [cit ]. Available at: [8] ODVARKO, J.: Firebug - Web Development Evolved. 2013, [online], [cit ]. Available at: [9] PATTON, R.: Testování Softwaru. Praha: Computer Press, 2002, ISBN , 313 s. [10] POSNER, K.; APPLEGARTH, M.: Project management pocketbook. Alresford: Management Pocketbooks Ltd, 1998, ISBN , 107 s. [11] Selenium Projects. [online], [cit ]. Available at: 69

77 6. CONCLUSION [12] STOBIE, K.; BERGMAN, M.: How to Automate Testing: The Big Picture. March 1992, [online], [cit ]. Available at: [13] THOLENCE, P.: FirePath , [online], [cit ]. Available at: 70

78 A Application under test description and test-case example A.1 User Web-based Interface Analysis User Web-based interface of the Xythos application consists of three main parts. Administration user interface (AdminUI) - used by system administrators to configure and manage the Xythos Server Web user interface (WebUI) - the main interface used by system users to access, manage and share content managed by the Xythos Server Mobile user interface (MobileUI) - the simplified interface which allows users to access Xythos server from mobile devices ( e.g. smartphones or tablets) Each part is developed by using different technologies. Admin UI uses only plain HTML but Web UI and Mobile UI use Ajax to dynamically create and update the content of web pages. When choosing specific automation solutions we must therefore be careful to select tools that can cooperate with all the required technologies. A.1.1 Admin UI Admin UI is the interface that allows system administrators to configure the server and to perform wide variety of tasks. The web application is divided into five basic parts described in the next table. This interface is written in plain HTML and contains only simple tables without any dynamically updated content. 71

79 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE AdminUI section File System User Management Server Administration Optional Features Tools Features create and manage directories, import files or manage file system create/manage user and groups, create roles server run-time information, logs, manage all kind of server settings manage workflows and classification logging tools Table A.1: AdminUI section description Figure A.1: Administration user interface 72

80 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE A.1.2 WebUI This interface is used by vast majority of users and enables them to perform tasks needed to access, manage and share different kinds of electronic documents. Besides basic functionality users can create wikis and dropboxes, manage workflows and workflow tasks, review documents, make comments etc. This interface is more complex comparing to AdminUI and uses Ajax technologies so lot of content is dynamically updated while the user is working on the page. Web user interface is divided into sections : 73

81 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE Main Menu - located at the top of the screen provides a link to your Home folder, Bookmarks and saved Searches, account options, and help. Main Tabs (Secondary Menu) - provides navigation links to various account and file management functions, and is located beneath the Main Menu. Navigation Toolbar - contains the breadcrumb trail, and file navigation options, and is located beneath the Main Tabs. Toolbar Buttons - contains all buttons appropriate to the screen you are currently viewing, and those you have selected from within your My Settings screen, and is located beneath the Navigation Toolbar. Document Manager Grid - contains all of the contents of the current folder with information pertaining to each item, and is located beneath the Toolbar Buttons. Footer - contains file size information, number of files, storage quota information and page navigation. 74

82 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE Figure A.2: Web user interface 75

83 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE A.1.3 MobileUI Mobile user interface is newly created feature that was introduced in the latest version of the product. This simplified interface provides only basic tasks as access and list entries, create new folder, upload file or review details about entries. All other features was not implemented from this view. Whole interface is designed to enable users work with it using the touch screen on mobile devices. Figure A.3: Mobile user interface 76

84 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE A.2 Test case example Listing A.1: example of complete test with outline package com. xythos. q a t e s t. t e s t c a s e. webui ; import com. xythos. q a t e s t. common. adminui. Authentication ;... / < s c r i p t type =" t e x t / j a v a s c r i p t "><!... ></s c r i p t > < t a b l e c l a s s =" javadoctable2 "> <tr > <td bgcolor ="# dee3e9 "> <h3> T e s t Case Name: DCM_FS_Autocomplete</h3> </td > </tr > <tr > <td > <b>objective </b> </td > </tr > <tr > <td > This T e s t Case i s used as demo. </td > </tr > <tr > <td > <b>not Included i n This T e s t Case</b> </td > </tr > <tr > <td > This i s j u s t a demo. </td > </tr > <tr > <td > <b>dependencies </b> </td > </tr > <tr > <td > <ul > 77

85 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE No dependencies </ul > </td > </tr > <tr > <td bgcolor ="# dee3e9 "> <b>included i n T e s t Case</b> </td > </tr > </table > < t a b l e c l a s s =" javadoctable2 " id =" _xy_tdtable "> <tr > <th > Use Case </th > <th > Test d e f i n i t i o n </th > <th > Description </th > <th > Expected Result </th > </tr > </table > / //@Test ( groups = { " demo " } ) public c l a s s DemoTC extends BaseTestCase { public s t a t i c f i n a l P r i n c i p a l USERNAME1 = P r i n c i p a l. createuser ( "demouser1" ) ; public s t a t i c f i n a l S t r i n g FOLDER1 = "Folder1" ; public s t a t i c f i n a l S t r i n g FOLDER2 = "Folder2" ; public s t a t i c f i n a l S t r i n g WIKI1 = "wiki1" public void c l a s s S etup ( ) { super. c l a s s S e t u p ( ) ; Log. logtc (DemoTC. c l a s s. getname ( ) ) ; Log. logtd ( "Setup" ) ; m_auth = getdefaultmodulemanager ( ). getauthentication ( ) ; m_ browserutils. openadminpage ( ) ; m_usermanagement. deleteuser (USERNAME1) ; 78

86 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE... public void classcleanup ( ) {... } / <a name="#demo T e s t Case"></a> <table > <tr > <th > Use Case </th > <th > Test d e f i n i t i o n </th > <th > Description </th > <th > Expected Result </th > </tr > < t r c l a s s =" _xy_td "> <td > <a href ="#Demo Test Case</a> </td > <td > Test c r a t i o n of f o l d e r. </td > <td > This t e s t v e r i f y t h a t user can c r e a t e f o l d e r </td > <td > Login i s s u c c e s s f u l and user can c r e a t e f o l d e r. </td > </tr > </table > ( groups = { "demo" } ) public void c r e a t e F o l d e r ( ) { Log. logtd ( "createfolder" ) ; m_ browserutils. openwebuipage ( ) ; 79

87 A. APPLICATION UNDER TEST DESCRIPTION AND TEST-CASE EXAMPLE m_auth. logintogrid (USERNAME1) ; m_filemanager. createnewfolder ( FOLDER1) ; asserttrue ( m_filemanager. i s E n t r y P r e s e n t (FOLDER1), "Folder is NOT present!" ) ; m_filemanager. domanageactionforentry ( PopupMenuLink. MNG_DEF_LOG, FOLDER1) ; m_fmdetails. setdefaultlogging ( true ) ; m_navigation. navigatetodetailsmenu ( DetailsMenu. COMMENTS) ; m_fmdetails. createcomments (new S t r i n g [ ] { "Cmnt1", " Cmnt2" } ) ; m_navigation. navigatetodetailsmenu ( DetailsMenu. STOR_QUOTA) ; m_fmdetails. setstoragequota ( "20" ) ; m_fmdetails. exitfromdetails ( ) ; m_auth. logout ( ) ; ( dependsonmethods = "createfolder", groups = { " demo" } ) //@Test public void copyfolder ( ) {... } } p r i v a t e Authentication m_auth ;... 80

88 B Framework source code Source code, scripts and documentation B.1 Content of electronic appendixes There is an electronic attachment stored in the information system as an archive file. This attachment contains all source code and scripts developed as a part of the thesis. This attachment contains: test automation framework scripts for test execution server and browser configuration file developed test suites framework and test documentation examples of test results scripts used for automated installation of the Xythos server application libraries and drivers for used tools data used for testing The folder structure is: automation_framework/ src/ sctipts/ config/ DBscripts/ remotebrowsers/ wfsutils/ testsuitejavadoc/ frameworkjavadoc/ screenshots/ test-results/ libs/ testdata/ 81