MO. 27. Feb 2006, 17:00 UHR

Transcription

1 061 MO. 27. Feb 2006, 17:00 UHR AUTOMATED FUNCTIONAL TESTING QUALITÄTSSICHERUNG IN DER SOFTWARE- ENTWICKLUNG: NUR ROUTINIERTES TESTING FÜHRT ZUM ZIEL WIRD PRÄSENTIERT VON Slide 0

2 Automated Functional Testing Markus Pilz Managing Director

3 Greenliff Offering: Professional Software Testing Incorporated February 2004 Today 9 Test Consultants Located at Technopark Zürich Network of Partners in Switzerland, Germany, USA, Rumania and India Slide 2

4 Our Believes Testing activity is widely underestimated both in its importance and in its complexity Testing is an exciting discipline that requires specific know-how Good software testing requires broad software engineering experience If you say quality, you also need to say quality control (= testing) Slide 3

5 Automated Functional Testing

6 Functional Testing Functional testing Also called black box, specification-based or behavioral testing Treats the system as a black-box Selection of test scenarios without regard to source code or data structures Test development based on functional requirements Structural testing Also called white-box or code-based testing Know-how of system internals used for test design Difference mainly relevant during test design Slide 5

7 The V Model Requirements System Specification System Design Module Specification System Test Integration Test Unit Test Acceptance Tests Implementation Slide 6

8 Automated Unit Testing Junit as well known example for Java Available for all scripting and programming languages Done by the developers themselves Showed many developers the need and the benefit of testing Further develops into nightly test and build and continuous integration Slide 7

9 Nightly Test and Build R/D Developer Checkins QAT Build 00:30 Smoke Test 01:20 Size Measuring 01:25 (a) Distribution 01:30 Web Pages Reporting 24x7 Code Mgmt System Build Per Change 24x7 Checkout 00:00 Benchmark 20:30 File Dir Sructure (a) includes distribution tests (b) includes historical and statistic data Result Parsing 24x7 TCK 02:00 Report Data Base (b) Slide 8

10 Automated System Tests? Unit testing verifies isolated subsystems and modules Integration testing verifies the interaction of subsystems Need for simulators and mock objects Unit and integration testing can not substitute system tests End to end testing with all components in place For the users, system tests are the real tests on the real system Can it be automated? Slide 9

11 What does it mean? We want to execute our manual tests automatically We need Test driver (program or script) Test cases (scripts) Test data Building up an automated test infrastructure and automating test cases is a normal software development project Maintenance needed You must find / you can find better qualified engineers Slide 10

12 Prerequisites Manual testing process exists Functional specifications and design documents available Detailed test case descriptions including expected results A stand-alone test environment Dedicated hardware Accurately replicates the production environment Tests can be repeated Data base can be restored to a known state for every test Remote control of network components Some software development know how Slide 11

13 Manual versus Automatic Write test plan Specify tests cases Test case documentation Manual Automatic Plan tests. Set-up test environment Study requirements. Specify test cases. Should correlate with the test specification May be generated Effort Similar effort Similar effort Similar effort Test case management Document management Management of test cases, test scripts and test data Similar effort Test case development Write detailed test procedures Develop test tools, test scripts, test data Much higher effort for automated testing Test execution Slow. High costs for personal and HW. Much faster. Tedious task is no problem Much less effort for automated testing Test report Manual updating Is generated. Much faster, higher accuracy Much less effort for automated testing Test case maintenance Update test procedures Update test scripts, test cases, test data Higher effort for automated testing Slide 12

14 Return On Investment Higher costs on Initial set-up New releases costs automatic manual Lower costs on Test run ROI: do not forget consequential costs Number of test runs is crucial Regression testing is the real goal test run Slide 13

15 Regression Testing is the Goal Run your system tests after every change After every minor release After every bug fix After every developer check in Automation is the only viable way to regression testing Speed Reliability Cost efficiency tedious work problem Slide 14

16 Manual Testers Loose Their Work Automated testing can not replace manual testing It is an addition to your testing process It frees manual tester from tedious regression testing Automated system testing is expensive You can not downsize your test department Avoid record/playback as a test method Manual testing is still needed Only automate tests that are repeated Only automate tests that are straightforward to automate Adapt a data-driven test methodology Slide 15

17 Data Driven Automated Testing Write a generic test script per use case Configured by test data from Excel or.xml file Specific use case are defined by the test data Test data Can directly be extracted from your manual test specification ;-) Can be maintained by testers without programming skills Matches manual test process very well Slide 16

18 GUI Testing GUI tests are not testing the GUI, but testing the system through the GUI You need tool support Simple record / replay does not work Recordings contain test data (=not generic) Only working programs can be recorded Every application change requires new recording Recordings are not reliable Pop-up windows, unpredictable events Maintenance effort is skyrocketing Slide 17

19 Conclusions Ad-hoc testing can not be automated Automated system testing is no replacement for manual testing Test new features manually Take over some tests to the automatic regression testing Regression testing is the real goal for test automation Design for testability could solve a lot of problems Testers and developers need to collaborated more Slide 18