Agenda. The Need 4 Speed. Promises of test automation. Quality of the test case. Quality of the test case

Transcription

1 Agenda Software Test Automation part I: Techniques for automating test execution BłaŜej Pietrzak Test activities Capture & playback Scripting Script processing Comparisons The Need 4 Speed Software testing needs to be: effective (at finding defects) efficient performing tests as quickly and cheaply as possible Testing is ~30% - 40% (for critical systems ~80%) of total effort Tested programs contain faults. Promises of test automation Reduce effort required for adequate testing Increase testing which can be done in limited time (tests run in minutes) Savings up to 80% of manual testing effort No savings but better quality software produced quicker Ability to find faults 1

2 Cost of test case modification Ability to verify more than one feature in the same test case Manual test Cost of test case execution, analysis and debugging Manual test Automated after many runs Manual test Ekonomia Ekonomia Automated first run Automated first run 2

3 Tester Developer Test automator Overnight testing Advantages Regression tesing (run tests against new version of a program) Run more tests more often(not same tests faster) Perform tests that are difficult to do manually (load, events not producing immediate output) Better use of resources (automating boredom; testers should spend time designing better test cases) Consistency and repeatability of tests (different configurations; checking standards feature implemented the same way in all aplications) Reuse of tests Earlier time to market (tests repeated quickly) Increased confidence (less stress) Limitations Automated tests found only 15% of the defects; manual testing found 85% Bach, 1997 Test automation does not improve test effectiveness Automated chaos just gives faster chaos Test automation may limit software development (economic reasons) Amount of effort to automate single test case takes from 2 to 10 times effort required to run the test manually (occasionally up to 30 times) Automated test have no imagination Limitations Don t replace manual testing when tests are run rarely when software is very volatile when results are easy to verify by human and hard to automate (color scheme, audio) when tests involve physical interaction (swiping a card through a card reader) Problems Can be hard to pick tests for overnight testing (eg. all tests need up to 2 weeks to run) If no thought is given to maintenance when automating, updating entire tests suites can cost as much (or more) than performing all tests manually Problems Unrealistic expectations (tools don t solve all problems; people are overoptimistic) Poor testing practise (automating chaos just gives faster chaos) Expectation that automated tests will find a lot of new defects (most likely the first run) False sense of security (incomplete tests, incorrect expected outcomes; automation preserves such defective results indefinitely) Technical problems (bugs in tools, building testable software) Organizational problems (choosing tools, training, management, coding standards) 3

4 Tool support for lifecycle testing Test design tools (to derive test inputs or test data for eg. from specification (logical tools), from existing data (random records from database physical tools)) Test management tools Static analysis tools (analyses code without execution, gives metrics) Coverage tools Debugging tools Dynamic analysis tools (when software is running, memory leaks, etc.) Simulators (meltdown procedures for a nuclear power plant) Capacity testing tools (performance, load) Test execution and comparison tools Test activities Identify test conditions (what can be tested) Design test cases (how the what will be tested) Build the test cases preparing test scripts (usu. held in a file, implements one or more test cases, navigation, set-up, clear-up procedures, verification) test inputs test data expected outcomes Execute test cases Compare test outcomes to expected outcomes Automating test case design Usually test input data generators Generate large number of tests cannot distinguish which are important Cannot identify missing aspects or requirements Code based test input generation if (a > 0) else Generates test input data Cannot generate expected outcomes Cannot identify missing code Software works as coded Generated test input data for a: 0, 1, 1000, -2 Interface based test generation Check help function for every control Try to edit data in all display only fields Check all web links Generates test input data Based on well defined interface (GUI, web) Expected outcome partially generated (general sense and only negatively missing page) Specification based test generation /** arg2!= 0 */ void show(int arg1, double arg2); Specification must be in a form that can be analyzed by a program (business rules, states, transitions) Generates test input data May sometimes generate expected output Depends on quality of specification 4

5 Ad hoc testing Vague manual scripts Run up Scribble Open file with a list Add some items Delete an item Detailed manual script (detailed input: Enter Sweden ; detailed expected result: Item added as number 3 in the list) screen shots can be expected results expensive, boring duplication Test inputs 1. Move mouse to Scribble icon 2. Double click 3. Move mouse to Open option 4. Click 5. Move mouse to countries.dcm in the file list 6. Double click 7. Move mouse to List menu 8. Click 13. Type Sweden 14. Move mouse to OK button 15. Click Sub Main Dim Result As Integer 'Initially Recorded: :21:03 'Script Name: atest StartBrowser " "WindowTag=WEBBrowser1" Window SetContext, "WindowTag=WEBBrowser1", "" Browser SetFrame,"Type=HTMLFrame;HTMLId=LEFT_FRAME","" Browser NewPage,"HTMLTitle=WebTest Publishing House","" EditBox Click, "Type=EditBox;Name=EDITBO1", "Coords=59,10" InputKeys "mgabor@@z.pl" ListBox Click, "Type=ListBox;Name=LISTBO1", "Text=Networks" ComboBox Click, "Type=ComboBox;Name=COMBO1", "" ComboListBox Click, "Type=ComboListBox;Name=COMBO1", "Text=Introductory." PushButton Click, "Type=PushButton;Name=SUBMIT1" End Sub Looks impressive Repeatable, Reusable (?) Comments needed Log - audit trail (from a tool) Gives documentation of what was tested Playtime for invited users, system crashes but you see what was typed in Can be tedious (adding 100 new clients) Manual verification Not v. readible Needs documenting (purpose!!) Tied v. tightly to specifics of what was recorded Often it s cheaper to record again then change existing script Values, inputs hard coded Additional work to add verification How much to compare: whole screen? (current time displayed) absolute minimum? (adding value to a sorted list) something between these two extremes 5

6 Tests failing the second time This file already exists alter script to check for the extra prompt and respond it alter script to check for a file and delete it Collect set up in one place Scripting techniques Build maintainable scripts. No spaghetti test scripts! Very similar to programming Recording gives linear scripts one script for every test case Scripting languages offer more power and flexibility that would ever be used by recording (control structures, reading data from different sources; Robot uses SQABasic) Scripting techniques Good set of test scripts: Fewer scripts (less than one script for each test case) Small scripts Each script has a clear, single purpose Specific documentation for users and maintainers, clear, up-to-date Many scripts reused by different test cases Structured, well organized, following good programming practices Easy to maintain Annotated, easy to read Scripting techniques Two fundamental approaches to test case implementation: prescriptive descriptive Techniques: linear scripts structured scripting shared scripts data driven scripts keyword driven scripts Linear scripts advantages Effect of test recording Contains: keystrokes, mouse moves, etc. One script for each test case Comparisons may be included You can quickly start automating Gives audit trail of what was done Doesn t require programming skills Good for demonstrations! Useful for conversions (eg. testing for Y2K conversion) Linear scripts disadvantages The process is labor intensive Everything tends to be done from scratch every time Inputs and comparisons hard-wired into the scripts No script reuse, no script sharing Expensive to change If anything happens when the script is replayed and did not happen when it was recorded, the script can easily become out of step with SUT 6

7 Structured scripting Control structures Calling structures Modularity Does require programming skills Test data still hard-wired into scripts More robust than linear If Message = Replace existing file? LeftMouseClick Yes EndIf Shared scripts Scripts are used by more than one test case Call ScribbleOpen( countries ) Call ScribbleSaveAs( countries2 ) Passing data to scripts For actions that are done often or are susceptible to change Less effort to implement similar tests Lower maintenance costs than for linear scripts Eliminates obvious repetitions More scripts to manage Test data still hard-wired into scripts Data driven scripts Tests inputs stored in a separate file The same script can be used to run different tests OpenFile ScribbleData For each record in ScribbleData Read INPUTFILE Read NAME1 Read NAME2 Read OUTPUTFILE Call ScribbleOpen(INPUTFILE) Type NAME1 Type NAME2 Call ScribbleSaveAs(OUTPUTFILE) EndFor countries, Sweden, USA, counries2 countries, France, Germany, test2 Data driven scripts INPUTFILE ADDNAME MOVE FROM MOVE TO countries Sweden USA 4 1 Norway countries2 DEL POS 2 7 Test cases made possible by data driven approach Data driven scripts Automated regression test cases.. Software under test Keyword driven testing Data file becomes a description of a test case using a set of keywords to indicate tasks to be performed Descriptive approach script states what the test case does not how it does it Control script For each TEST_ID OpenFile TEST_ID For each record in test file Read keyword Call keyword EndFor CloseFile TEST_ID EndFor Supporting scripts ScribbleOpen AddToList DoodleOpen AddCircle SaveAs Test tool side 7

8 Keyword driven testing ScribbleTest1 ScribbleOpen countries AddToList Sweden USA SaveAs countries2 DoodleTest1 DoodleOpen smallpad AddCircle SaveAs test1 Test cases side Reduces script maintenance costs Speeds up implementation Non programming testers can implement tests Tool and platform independent Script pre-processing Script manipulation for easier writing and maintaining scripts Beautifier checks layout of scripts; edits scripts to make them conforms some standards Static analysis (Type F1 or Type {F1} ) General substitutions (constant for a start page s URL) Test specific substitution (for e.g. to change test data in a script) Automated comparisons Reference testing Predict test outcomes in advance rather than verify the first set of actual outcomes Automated executions generates a lot of outcomes What should be compared? Automated comparison can hide a defect (when there s a defect in expected outcomes) Dynamic comparisons performed while a test case is executing should we continue running script when differences are revealed? (eg. big volume of data created or printed) script programming can bring errors tools support increased complexity, higher maintenance costs (instructions in the scripts) Sub Main Result = HTMLVP (CompareData, "Type=HTML;HTMLId=PARAGRAPH1", "VP=Object Data") End Sub Expected results saved in separate files; single file for each verification point; low readability Post execution comparison comparison done after running a script comparators given with execution tool or some utility programs reasonable approach for positive testing ( Sweden added) not appropriate for testing invalid inputs (deleting item at invalid position) to compare outputs like files, content of databases, etc. commercial tools better support dynamic comparison Post execution comparison Passive approach Active approach (we save some results during a test case in order to compare them afterwards) more detailed outcome of test cases will be retained comparison can be done offline different comparators can be used requires programming in to the test script 8

9 Complex comparison Simple masking enables to compare actual and expected outcomes with known differences between them ignoring dates and times (also formats) unique identity code or generated numbers output in different order different values within an acceptable range different text formats Faktura nr F1234/2003 Data: 20-CZE Lp. Symbol Nazwa Cena 1 P11 Pióro K32 Kreda 3.20 Razem: data excluded from comparison ignoring complete record or specifying fields that are to be ignored IGNORE (1, 16, 25) IGNORE (3, 7, 17) IGNORE (12, 12, 22) but have to specify different instructions for different invoices Płatne do: 30-CZE-2003 Search technique for masking IGNORE_AFTER ( Faktura nr, 10) IGNORE_AFTER ( Data:, 11) IGNORE_AFTER ( Płatne do:, 11) Search technique using regexp eg. checking invoice number format, date format etc. F[0-9]{4}\/20[0-9]{2} [0-9]{2}\-[A-Z]{3}\-20[0-9]{2} [0-9]{2}\-[A-Z]{3}\-20[0-9]{2} Different types of outcome Text files Databases and binary files convert actual and expected outcomes to textual format Screen based outcomes bitmap comparisons (what if generated bitmaps?) get GUI attributes and report in a text form problems with non-standard implementations Multimedia application can give output information (text format) when a sound was played (file name, file size may seem enough) Communicating applications Specialized tools needed Simple comparison Expected output = actual output Invoice no.f1234/2005 Date: 10-Sep No. Symbol Name Price 1 P11 Pióro Total: Payment: 30-Sep-2005 Invoice no.f1234/2003 Date: 20-Jun No. Symbol Name Price 1 P11 Pióro K32 Kreda 3.20 Total: Payment: 30-Jun

10 Expected Actual Comparison filters Filter Filter Edited expected Edited actual Compare Real differences An editing or translating step that is performed on both an expected outcome file and the corresponding actual outcome file Comparison filters Many filters can be performed in a series Filters remove differences from both outcomes simple comparison can be used to highlight unexpected differences Tasks of filters: substitution data extractions sorting Filter example while (<>) { s/j[0-9]{5}/<job No.>/g; s/[0-9]{2}-[a-z][a-z]{2}-[0-9]{2}/<date>/g; s/([01]?[1-9] 2[0-3]):[0-5][0-9]/<Time>/g; print; } Job numer: J12345 Job numer: <Job No.> Run date: 20-Jan-97 Run date: <Date> Time started: 12:45 Time started: <Time> Job numer: J15469 Run date: 14-Aug-97 Time started: 13:36 Filter Job numer: <Job No.> Run date: <Date> Time started: <Time> Compare No differences Comparison filters advantages Availability of text manipulation tools and languages (sed, awk, grep, perl, tcl, etc.) Reuse More stringent comparison criteria (you work only on individual parts of outcomes) Easier implementation (complex comparison is devided into many small steps) Simple comparison tools can perform complex comparisons Comparison filters disadvantages Programming skills required Filter maintenance required (when format of output changes) Should be well documented and managed and easy to find Summary Automated tests found only 15% of the defects; manual testing found 85% Bach, 1997 Automated chaos just gives faster chaos Automating test case execution and comparison Simple comparisons + comparison filters 10

11 Literature and Links Software Test Automation use of test execution tools; Mark Fewster and Dorothy Graham Rational Robot documentation Quality Assessment Thank You for your attention What is your general impression (1-6) Was it too slow or too fast? What important did you learn during the lecture? What to improve and how? 11