RSTAR An Object-Driven Testing Framework

Transcription

1 RSTAR An Object-Driven Testing Framework Mosaic, Inc. 505 N Lake Shore Dr., Suite 222 Chicago, Illinois Mosaic, Inc. All rights reserved

2 Introduction Executives everywhere are under pressure to reduce costs. Since testing is a large component of software development costs, it will inevitably come under scrutiny. Moreover, despite the need to reduce costs, the need to regression test large mission-critical systems quickly and thoroughly is a fact of life in business today. Organizations that either take too long to test or cut corners testing will eventually incur costly delays or production problems that will impact the bottom line. How can we reconcile the need to reduce costs with the need to regression test? Fortunately, there is a way. Current testing practices, both manual and automated, are inefficient. There are ample opportunities to reduce costs. The purpose of this white paper is to describe Mosaic s patent-pending RSTAR testing framework. The architecture makes it possible to significantly reduce testing costs and successfully automate regression testing. The key difference between the RSTAR framework and other frameworks is the increased level of reuse. The easiest way to reduce testing costs is simply to be able to do the same or more testing with fewer testing artifacts. The higher level of reuse enabled by the RSTAR framework accomplishes this and has many additional benefits. Test Automation There is a lot of confusion in the industry regarding how best to automate, what parts of testing can be automated, and what benefits are reasonable to expect from test automation. This confusion leads to unrealistic expectations from management, raises testing costs, and is another obstacle keeping organizations from realizing the substantial benefits that are available from automation. The execution of tests and the verification of their results are the only areas of testing that can be practically automated. In contrast, test planning as well as the design of test cases can not be automated. While these activities can not be automated, they can be made much more efficient and lay the foundation for much lower cost automation. The primary benefit of test automation is the ability to quickly regression test a system to see if code fixes or changes have impacted the quality and stability of a system. For large, mission-critical systems, the savings in reduced regression testing costs, lower defect rates and reduced cycle time make successfully implementing a test automation strategy a mission-critical activity. Page Mosaic, Inc.All rights reserved

3 Barriers to Automation Automation has been the holy grail of software managers and testers for many years. While automation tools have improved significantly over the years, most organizations have had trouble realizing the benefits of automation. There are a number of reasons for this: Many organizations do not have a well-defined testing process. Automation works best when there is an effective manual process already established to provide the foundation for the automation. Tool expertise is expensive and in short supply. Developing and maintaining automated scripts can be expensive. While the tools are very powerful, they do not enforce a robust, maintainable automation architecture. It is all too easy to automate in ways that are difficult and expensive to maintain. Data-Driven and Keyword Approaches Fundamentally testing involves exercising a system with a known initial state and predefined input. The actual results of the system are compared with predefined expected results. Thorough testing requires exercising a system with many variations in both the sequence of actions performed and test data (i.e., predefined input or expected results). Software testing, especially manual testing, can be both tedious and labor intensive. Moreover, the nature of software development is that a system will need to be re-tested every time the system is modified in order to verify that modifications work as intended and do not introduce unforeseen defects. Maintaining test cases can be expensive much like maintaining a software system. Automating test execution to reduce test execution time and reusing testing components to cut test planning and maintenance costs are two of the most common strategies to reduce testing costs. Data-driven testing and key word testing are the most common industry frameworks for accomplishing this. Data-driven testing involves using a table (usually a spreadsheet) where each row contains the input and expected results test data. The test exercise the software with the input data in the table and compares the actual results to the expected results in the table. The advantages of data-driven testing are the ability to easily add new conditions to the table and the ability to separate the test Page 2

4 from the test data. Data-driven testing enables a limited form of reuse. While the test can be reused with the different data conditions in the table, the table itself is bound to the test and therefore the test can not be easily reused with a different table of test data. Moreover, the test data must be customized to the test and therefore test data (i.e., individual rows) are not easily reused. This limits the ability to reuse the same test data with different tests. In contrast, keyword-driven testing uses a keyword to represent a specific action (e.g., login, enter, check, etc.). A driver is written to read the keywords and execute the scripts associated with the keyword. Data is usually embedded in the test or in an external table as in data-driven testing. Keyword-driven testing provides a high level of reuse for the keywords/scripts, but, like data-driven testing, the data is bound to the test, so the test is not easily used with different data and the data is not easily used with other tests. While both data-driven and keyword testing achieve some level of reuse, neither approach is designed to reuse the test data itself. Tools designed to be a repository of test cases, when they don t ignore data altogether, either imbed test data in the test case or in external datasheets. These approaches force data to be designed around a test, binding the data to the test limiting the reusability of the test and making the reusability of the test data impractical. Another issue with organizing test data around individual tests is that it is difficult to know if a given test condition is, in fact, tested. When test data is spread over numerous spreadsheets and imbedded as parameters in tests, there is no practical way to understand if a given condition is tested and, if so, where it is tested. This nonreusable test data is then difficult to maintain and works to lower the reusability of the associated tests and scripts. The RSTAR Object-Driven Test Framework Mosaic s RSTAR framework addresses the limitations of key-word and datadriven approaches and integrates manual and automated testing in one unified framework object-driven testing. The RSTAR framework differs from other frameworks in several respects: It achieves a much higher level of reuse because, unlike other frameworks, the test data (i.e., input and expected results) is fully reusable and integrated into the framework. It integrates the manual and automated tests into one unified framework. Manual tests are designed to be automated and drive the automation once automation scripts are written. Page 3

5 It is designed to work with commercial automation tools such as Hewlett Packard s QuickTest Professional (QTP), open source tools such as Selenium, or home grown solutions. The major reusable components of the framework include: Test Actions Test Scenarios Test data Automation scripts Each will be discussed below. Reusable Test Actions In the RSTAR framework, an action is the lowest level test component. It can be as small RSTAR Test Action as clicking on an OK button or as large as entering data into an Figure 1 entire window. While the framework itself does not enforce any restrictions on the size of an action, the size of the action is practically determined by the opportunities to reuse the action. Figure 1 illustrates an action in RSTAR. While it is important not to bind data to the action, the action definition includes the type of data (see data template discussion below) that the action needs. Figure 1 illustrates this. Individual test steps indicate the type of data (the data template form_data ) and the individual fields from the data template. Test step 1, for example, needs First_Name data. Specific instances of data are not bound to the action. An action may use multiple data templates. When automated, RSTAR will also retain the name of the script that automates the action. Actions Grouped Into Scenarios Confirm Scenario 2 End-to-End Scenario Pay Ship Meth Ship Addr. Login Scenario 1 Figure 2 Page 4

6 Reusable Scenarios Once defined, individual actions can be organized into larger and larger test scenarios as illustrated in Figure 2. The type of data needed by a scenario is simply inherited from its component actions. Scenarios can be nested. Both individual actions and scenarios can be reused in any number of tests. The ability to reuse tests is only lost when test data is bound to the test. Therefore, the RSTAR framework separates reusable test components from executable tests and delays binding data to the executable test until execution. Reusable Managed Test Data Reusable test data is the ingredient missing in most frameworks today. Typically test data (i.e., input and expected results) is put in EXCEL or similar data sheets. The data is organized around needs of a specific test, rather than the overall application needs, as depicted in Figure 3. This organization of data into many data sheets, each specific to a particular test, creates a number of problems including: Effort will be wasted creating the same data numerous times in different data sheets. Test Data Typical Approach Because data will be spread over numerous physical data sheets, changing the data is difficult. There is very limited visibility into where the data is used, or whether specific conditions even exist. Data Sheet 1: Customer data with WIS Address Data Sheet 2: Order data with WIS Address Data Sheet 3: Invoice data with WIS Address Test1: Add Customer Information Test 2: Enter Order Test 3: Create Invoice Contrast this approach with the approach Figure 3 depicted in Figure 4. In this case, the data is reusable, centrally stored, accessible, and easily maintained. Data is first organized around the needs of the application in logical tables called data templates. Specific instances of data, logical rows in the data template, called data profiles, can then be defined. If you want to update a given customer s information, it only has to be changed in one place. Moreover, the same data can be used in any number of tests both manual and automated. Page 5

7 The typical approach depicted in Figure 3 usually results in one data sheet per test. The results of one of our clients are representative. When they were able to reuse test data organized around the application, they went from 100 data sheets supporting 100 tests to 15 data templates supporting the same 100 tests an 85% reduction!! Test Data Reuse Meta data can also be included in the data profile that describe the purpose of the data profile, owner, or whatever data can make the data easier to use and maintain. When a test is automated the same data used by the manual test will be passed to the automation script. Additional data specific to the automation script can also be included. Executable Tests DSTAR Customer Template WIS Address Exxon Address General Motors Figure 4 Test1: Add Customer Information Test 2: Enter Order Test 3: Create Invoice Once the reusable actions, scenarios, data templates and data profiles are defined, executable tests can be developed using the reusable components. This is depicted in Figure 5. An executable test is simply a defined sequence of actions and scenarios with data assigned to the executable. The actions contain the type of data that must be assigned. Before the test can be executed, the tester must, as part of the executable definition, assign data profiles (individual rows in a data template) to the test. The assigned data will be bound to the executable at execution when the actions, scenarios, and data profiles are all assembled into the executable test. Figure 6 is an example of an RSTAR executable. Because the executable knows for each action the type of data and, if specified by the test planner, the individual fields used by each test step, the actual test data can be merged into the step descriptions. This makes it much easier for the manual tester, but it also enables user friendly automaton execution reports that show exactly what data was used by each action Actions With Managed Test Data Scenario 2 End-to-End Scenario Confir Data 3 m Pay. Pay Ship Ship Meth Ship Meth. Addr. Addr. Login User Scenario 1 Figure 5 Data Template n Data Template 1 Page 6

8 during an automation run. Automation Framework The RSTAR framework integrates the manual tests and the automation as shown in Figure 7. The manual tests should contain all the information needed to automate. All that is needed for automation is to develop automation scripts for each of the manual test actions and have a master control program (MCP) to control the execution sequence of the scripts and pass the required test data to the scripts. This is depicted in Figure 7. The tight linkage between the manual and automation tests has a number of benefits: The manual tests serve as specifications for the automation scripts. Automators do not need to also be subject matter experts. This can be especially beneficial if testing work is being outsourced. Figure 6 Test Executable There is significant flexibility regarding what to automate and when to automate it. If there is a problem with the automation, the manual tests will be available. Tests will be easier to review and update. Automated tests can be created by subject matter experts without writing new code. Existing actions, scenarios, and test data can be used to create new executables. A new executable can be executed manually, or if all of the action scripts needed by the new executable have been written, the executable will be automatically automated. Master Control Program The MCP is an automation driver. It gets the information required to execute the test script sequence, assigned data, etc. from the RSTAR database (see Page 7

9 Figure 8) and then calls the individual action scripts in the appropriate order and passes the assigned test data. The MCP also logs results back to the RSTAR database. Note that the executable definition (see Figure 6) includes the location of the automation driver intended to execute the test. The MCP and automation scripts are written in the selected automation tool s scripting language. Different executables can use different automation tools with different MCPs. Mosaic has developed MCPs for a variety of tools including commercial tools such as Hewlett Packard s QTP and open Scenario 2 End-to-End Scenario Manual Test Planning Drives Automation Confir Data 3 m Pay. Pay Ship Ship Meth Ship Meth. Addr. Addr. Login User Scenario 1 Control Program Scripts Figure 7 Data Template n Data Template 1 RSTAR Automation Framework Test Exec. Results source tools such as Selenium. It should also be noted that, while tools like QTP and Selenium are designed to automate systems with graphical user interfaces (GUI), the framework applies just as well to automating non GUI applications, though a different tool, may be required. Action Scripts There should be a 1-1 relationship between automation scripts and actions. Because of this architecture, the action scripts tend to be relatively simple and less automation expertise is required to develop them. Note that in Figure 1, part of the actions definition was a script name. The RSTAR framework is designed to directly automate the manual tests. Moreover, the action sequence performed by each test as well as the assigned test data are all taken directly from the manual test. Scripts are written in the automation tool s scripting language to execute each action. These scripts operate directly on the application under test, and perform basic functions or actions such as making Figure 8 menu selections, clicking on buttons and entering test data into text boxes. Because actions are designed to be reusable, each script will tend to perform a simple test action. Limiting each script to perform one action greatly increases the script s reusability since the script can be used to perform its action within many different testing scenarios and many Page 8

10 data variations. Data variations can be application oriented (e.g., customer information), or other types of variations (e.g., browser to use, url to bring up in the browser, time out parameter, etc.) Modularizing test scripts to focus on discrete actions not only assures a high level of reusability, it also allows the test actions specified by the manual testers to be easily translated into their associated scripts for automation. The modularization also minimizes maintenance since application changes usually only require updates or re-sequencing to the specific modular script(s) that processes the affected applications rather than all of the tests that use those broader functional areas. Our experience on projects indicates that over 90% of actions will remain unchanged from one release to the next. Execution History Every time an executable is run, an instance of the executable is created in the database. Future changes to an action, scenario, or test data will not affect this instance. If the executable is run manually, status information can be updated by the tester e.g., timings, pass/fail, screen shots, defect information, etc. If the executable run is automated, then the MCP and action scripts can update the status information. RSTAR will retain a record of all executions. Management Information The RSTAR framework makes the testing process transparent. Tests, test data, execution results are all in one central repository. A key benefit of RSTAR is the ability to provide not only tactical information to the tester but also progress and status information to testing management. A few examples include: Reports can be generated showing where each data profile is being used and, perhaps more importantly, which data profiles are not being used. The later is a form of data condition coverage that can be very powerful. Reports can be generated showing the automation status of actions, and executables. Reports can be generated showing the status of executables e.g., when they last ran, whether they passed or failed, etc. Page 9

11 Conclusion The pressures to reduce testing costs, automate regression testing, and provide more visibility into testing progress and status are a fact of life in business today. Mosaic s RSTAR testing framework directly addresses these realities; It achieves the highest level of reuse in the industry. When there are fewer artifacts to develop, costs are reduced. Manual testing is fully integrated with automated testing. The manual tests actions are the specifications for the automation scripts. This simplifies the development of automation scripts. It also enables subject matter experts to create new automated tests without writing new code. It works with commercial, open source, and home grown automation tools. It provides the progress and status information needed to make informed decisions. Page 10