Effective Test Automation a la Carte

Transcription

1 Europe s Premier Software Testing Event Stockholmsmässan, Sweden Testing For Real, Testing For Now Effective Test Automation a la Carte Martin Gijsen, Consultant, The Netherlands

2 Effective Test Automation à la Carte Martin Gijsen Test automation architect Independent consultant 2009

3 Overview Effective test automation Domain Specific Test Languages Three example solutions (free tools only) Where to get the automator Conclusion Q & A

4 Test automation scenarios About 2/3 of all test automation projects fail sooner or later Brian LeSuer (testing experience magazine, December 2008 issue, page 47) total cost / effort 3 time 1. manual testing 2. effective automated testing 3. less effective automated testing

5 Effective test automation Record & Playback only works in case of (very) little manual maintenance to the testware An automated test is a program Test automation requires software engineering For continuity of benefits, focus on: 3 R's: Tests must be easy to write, review and revise Maintainability and maintenance sensitivity

6 Domain Specific Test Languages Used by testers to express automated tests Instructions defined by (or with) testers Only the essence of a test case should remain: Natural, high abstraction level No irrelevant interfacing details No irrelevant tooling details The result is low maintenance No programming Clear test report

7 A good DSTL based test case account first name last name address open account John Doe 11 Test street, Testburg open account Jane Doe 27 Testability lane, Testville account amount deposit John ,00 amount from account to account transfer 1234 John Jane account amount check balance John ,00 check balance Jane 1.234,00

8 Example instruction documentation Name and aliases Description 'begin test case' and 'begin testcase' Indicates the beginning of a (new) test case. Parameters 1 The identification of the test case, preferably unique within the test. Optional. 2 The test case description. Optional. Pre-condition Post-condition None. If the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one. Error condition Example... If in a procedure definition.

9 How does it work? Tester domain: Instructions Focus is on WHAT to test Requires analysis skills Test Test report Developer domain: Scripting Source code Instruction definitions Focus is on HOW to test it Requires software engineering skills Test automation solution System Under Test

10 Three examples Real systems and test solutions: 1. Web services based billing system 2. Web application for batch payments 3. Generic solution for payment systems Different systems, interfaces & tooling

11 Web services based billing system Interface: web services (SOAP / XML) Interface type: synchronous messaging Tooling: ETA Framework, Axis, Java Multiple implementations of instructions for phased test execution

12 A sample test case url select end point name select wholesale customer Joy inc. cust nr first name last name... create end customer John Doe... cust nr service add subscription X item nr cust nr check billing item X

13 The design Test Test report WSDL Axis Test engine Instruction logic Web serv. client Billing system

14 Web app for payment batches Interface: web application (J2EE) Interface type: Graphical user interface Tooling: ETA Framework, WebDriver, Java

15 A sample test case name set screen item name order letter account accountnr set screen item name order letter amount amount set screen item name order letter date desireddate user name log on JohnDoe account amount date... enter order letter ,67 today...

16 The design Test Test report Test engine Instruction logic WebDriver Web application

17 Payment systems Interfaces: Iso15022 / SWIFT MT messages Iso20022 / SWIFT MX / UNIFI messages Interface type: asynchronous messaging Tooling: ETA Framework, Java Low abstraction level is raised using procedures (not in example)

18 A sample (low level) test case define structure name msgout name value set msgout.field1 value1 set msgout.field2 value2 connection type name send message tosut mt103 msgout connection name period receive message fromsut msgin within 10 seconds value 1 value 2 compare?msgin.field1 value1 compare?msgin.field2 value2

19 The design Test Test report Test engine Instruction logic UNIFI protocol SWIFT protocol Incoming msgs Connection logic Payment system

20 Bringing it all together Test engine WS logic Web app logic Payment logic WS client WebDriver Protocol X Protocol Y Incoming msgs Connection logic System Under Test

21 The test automation menu 1. DSTL Define your own 2. Test engines ETA Framework Fit / FitNesse STAF xunit Interfacing components Web services Web app Messaging Java Swing Vendor lock-in (optional)

22 Where to get the automator Main criteria: Skills (programming software engineering) Availability Cost Train someone from the test team Borrow from development Hire or reserve developer for test team Hire external consultant

23 Conclusions Test automation is software engineering - apply the right skills to succeed A DSTL enables testers and reduces maintenance Automated testing with free software is possible for many systems and interfaces (even combined) While every system and DSTL is different, the basic design of test solutions can be the same Non-programming testers and non-testing developers can be effective at automated testing

24 Q & A?

25 See also Exploring no man's land with keyword driven testing - presentation at Free Test 2009 Effective automated testing with a DSTL - white paper for the ETA Framework and its documentation