Managing Data in Test Automation Frameworks. Vladimir Belorusets, PhD Xerox Corp

Transcription

1 Managing Data in Test Automation Frameworks Vladimir Belorusets, PhD Xerox Corp

2 About Author Dr. Vladimir Belorusets is an SQA Manager at Xerox Corp, Palo Alto, CA responsible for quality of enterprise content management applications. He has more than two decades of extensive experience in software development, test automation, test management, and software engineering methodology. Vladimir is a Certified Tester by American Software Testing Qualifications Board. Vladimir earned his Ph.D. in Control Systems from Moscow Institute for Systems Analysis, Russian Academy of Sciences. He taught numerous courses on functional and performance testing in various Bay Area computer schools. He has multiple publications in ST&QA magazine and StickyMinds.com. Dr. Belorusets is a member of the Strategic Advisory Board and the Conference Program Board at Software Test Professionals. 2

3 What is a Framework? A framework is a semi-complete application. A framework provides a reusable, common structure that can be shared between applications From JUnit in Action by Vincent Massol Examples of software frameworks are: Eclipse HP Quality Center STAF SAFS 3

4 Test Automation Framework A test automation framework (TAF) is a structure into which you plug-in scripts and data from independent test automation tools TAF is an integration point for management and control of tools automating QA processes 4

5 Data in Test Automation Framework Whether you select a commercial TAF or develop it in house, the ease of test input manipulation and maintenance is one of the key aspects to your success in test automation! This presentation provides a flexible and proven approach to test data management derived from a small number of simple design principles 5

6 Global & Local Test Data We classify test data into two categories: Global data are shared and visible to all scripts within a test set They usually represent the configuration parameters, such as server name, starting web page URL, etc Local data are visible only within the originating script 6

7 Test Scripts, Test Sets, and Data 7

8 Questions to the Architect TAF extracts copies of the scripts (along with the associated data files) from the central script repository and runs them on distributed hosts How to implement global data? How to change the script s local data in the multiple test sets? How to preserve the original test data? Do we need to keep multiple instances of the data files for each script? 8

9 Vladimir s Design Rules (1/6) I have developed a list of 6 general design rules that have proven to be efficient for organizing test data in the TAF 1. Test data must be separated from the test scripts Advantages: The same script can be reused, without changes, for testing a different functionality in a product just by modifying the test data Apply the same scripts for internationalized software Significantly reduce the total number of scripts that must be supported 9

10 Vladimir s Design Rules (2/6) 2. Test data should be presented within the tables Advantages: Data-driven tests can be designed easily Data-driven test is a technique that allows one automation script to implement many test cases by iteratively reading through a data spreadsheet one row at a time, using the values in that row to drive the test. 10

11 Vladimir s Design Rules (3/6) 3. Data tables should be provided in external files and be easily available for review and modification to the test case consumers Advantages: A subject matter expert should be able to generate test data for his tests without having any programming skills. If it takes considerable effort to find the data file navigating through the tens of subdirectories, then the TAF is unusable 11

12 Vladimir s Design Rules (4/6) 4. Global test set data must be separated from the local script data Advantages: If script data files contain both global and local data, it takes much time to modify the same data in all data files. If we have a central repository for global data, we need to do the modification only once per test set. 12

13 Vladimir s Design Rules (5/6) 5. Local test data should be uniquely associated with both the test script and the test set that contains the script Advantages: An association with the test set is necessary to run the same script in the multiple test sets but with different data. I assign two dimensions to all local data: one index for the script, and the other one for the test set 13

14 Vladimir s Design Rules (6/6) 6. Local data for each test set should be separated but coexist within the same data file Advantages: If we have only one data file with coexisting local data for every test set, it simplifies data inventory and search. Data coexistence also protects against data conflicts and overwriting. 14

15 Architectural Solution (1/4) Each script is associated with only one data file stored as an MS Excel spreadsheet (Rule 1 and 2) Easy access to data files (Rule 3) can be provided by creating a script/data catalog using MS Excel Group and Outline feature. 15

16 Architectural Solution (2/4) Each data file has multiple worksheets with one mandatory, named Default. That worksheet contains the original test data provided by the script developer All worksheets have the same structure: the first row contains headers (parameter names), and all other rows contain test data values. Multiple rows indicate a data-driven test 16

17 Architectural Solution (3/4) To modify the original local test data, you should create a worksheet with the test set name and enter the new test data there. This way, local data for different test sets are associated with the test set names (Rule 5) and coexist in one data file (Rule 6) 17

18 Architectural Solution (4/4) The algorithm to access local test data is: Each script reads the associated data from the worksheet that has the test set name the script belongs to If there is no such worksheet, the original data are automatically read from the Default worksheet If the subject matter experts want to modify the original test data, all they need to do is to create a new worksheet with the test set name 18

19 Another Problem to Solve You want to run a test set of 100 scripts developed by others that use ServerName as an input parameter for each script. The server s name in your environment differs from one in the data files Do you need to modify that variable by assigning the same value in all 100 scripts? The answer is fortunately, No. The solution is global data for the test set 19

20 Implementation of Global Data (1/3) In our approach, global test data are implemented through the User Environment Variables that one can see in the Control Panel Global variables are specified once at the beginning of each test set They are accessible to all scripts belonging to that test set 20

21 Implementation of Global Data (2/3) Every test set starts with a Setup script that creates the environment variables for all global data by reading an Excel data file, like all other scripts in TAF Unlike other script data, the Setup data file has only one worksheet with two mandatory columns: Order and Test Set. The other column headers are named after the global variables 21

22 Implementation of Global Data (3/3) Each row in a worksheet presents global variables/values for one test set The Order column describes the sequence in which the test sets will be executed by TAF 22

23 Test Set Structure Every test set contains the following scripts: Setup script sets the global variables Start script starts the application under test Automated test cases Finish script closes the application Reset script removes global variables (Optional) Close Tool script releases the automation tool license once the test cases are completed 23

24 Q & A 24