Seminar on Model-Based Testing as a Service Application Modelling Author: Antti Heimola Date: Dec-4, 2008 Time: Thu Dec- 4, 2008 Location: Tampere: TUT/Department of Software Systems, Tietotalo, class room TC131 1 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Content Model based Testing Case: KENDO - Testing modelling/automation needs - Time it takes 2 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Model based Testing James Bach s Blog: The Unbearable Lightness of Model-Based Testing Question: How do you suggest that you apply this technique with a complex application, something like MS Word for instance. I m not sure what technique you are referring to. If you are talking about using state models to describe a system, then it s interesting that you ask that, because I asked Harry Robinson that same question (it was even about applying it to Word) after seeing an otherwise fascinating talk about statebased testing that he gave, years ago. As I recall, he wasn t ready to answer that question. But I am. My answer is: ask Harry. Seriously, he has actually worked with model-based testing tools at Microsoft, in the years since I challenged him on this point. 3 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Model based Testing Wolfgang Grieskamp 4 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Application Modelling: Case KENDO GUI modelling tool flow design tool Specification tool Test case generation tool NOT linked to UML or other SW design schemes 5 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
6 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
System Success Factor of Model based Testing - technical z : System Modelling Other tools SSF = 3x+2y+z * Test Execution API y : Test Execution Control Other tools Test Service API Communication PC Successful model based testing requires support from test execution control tool and SUT test services testability built in x : Bluetooth, USB, WLAN, IP Communication System Under Test Test Support (testability) Target * Not any real formula, just a notion Test Services 7 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Test Service example Model, i.e. Specification of a system event: Left soft key is Options Options menu contains following items: Sort, Instructions,, Exit Specification converted to test scripts: vw_verifytext vw_verifymenu Options Sort/Instructions/ /Exit Test scripts driving SUT using provided Test Services: Question of test automation needs. Many test tool vendors are happy with press button support Test Service must support Capture screen minimum while Test Execution Control takes care of text verification Capture screen support is NOT enough for verifying the whole menu content more sophisticated Test Support required 8 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Levels enabling cross Platform Modelling & Testing System Modelling + Test Mode CPP NFT: Performance, Stress, Endurance.. Probability Priority (test data) Test Execution API Test Execution Control Advanced Features Memory Usage CPU Linux S60 PC Etc. Test Service API Communication Bluetooth, USB, WLAN, IP Communication System Under Test Test Support (testability) Test Services Test Service API Communication Bluetooth, USB, WLAN, IP Communication System Under Test Test Support (testability) Test Services Test Service API Communication Bluetooth, USB, WLAN, IP Communication System Under Test Test Support (testability) Test Services In case Application flow (functionality) is the same, no modifications needed, the model can be executed on any platform supported by Test Execution Control and Test Services 9 HR2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
History 1. Ross Collard: Test Design: Developing Test Cases from Use Cases, 1999 2. KeyWord based test automation system 2001 Idea of integrating Use Cases & KWs 3. Mikko Nikkanen: KENDO I, 2005: Use Case based (test) modelling 4. Mikko Mäkinen: KENDO II, 2007: Application modelling Requests for productizing the approach 5. Productized (internal) version 2008 10 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
History 1999 * 2008 2001 ** *) Ross Collard: Developing test cases from Use Cases, Software Testing&Quality Engineering, July/August 1999 **) Mikko Mäkinen: Model based approach to Software testing 2007, page 28 11 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Some Principles UI level Application Modelling UI flow Design Target: Model all paths, not only some parts of them In case a model is too complex, most likely the GUI is too complex to use User Roles: Model Designer, Test Designer, Tester Clearly separated events: User Actions, System Responses Use of KeyWord based approach was logical: Press a button, Tap, Open Menu Managing complex models System Model, Application Models, Sub-Models, Use/Test Cases 12 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
KENDO User Roles & Key features Model Simulation. Model Designer, Test Designer, Tester Specification generation 13 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
14 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Concept Reference: Implementing Lean Software Development Mary and Tom Poppendieck, 2007 - Write executable specifications instead of requirements - Myth: The job of testing is to find defects : The job of tests is to PREVENT defects, not to find them 15 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
KENDO time it takes S60 and Maemo Modeling time Minutes Minutes Minutes Minutes Minutes Hours Hours Model Designer Test Designer (ASTE) Execution Total * Total ** Application Specification Testing Phonebook 85 330 55 65 21 9,3 7,5 Calendar 120 420 125 110 15 13,2 11,2 Notepad 90 310 115 55 12 9,7 8,2 Messaging 80 345 50 130 41 10,8 8,6 Gallery 120 360 60 60 32 10,5 8,5 Average 99 353 81 84 24 10,7 8,8 Median 90 345 60 65 21 10,5 8,5 * Excluding * before specification test execution type time support, (column ** after F) the specification type support ** Estimated with new auto-complete feature (75% of column C) Maemo (810) application modeling times with Kendo Application Model Designer Specification Test Designer Testing Execution Total [min] Total [h] Notes 120 180 60 60 120 540 9,0 Clock 100 120 45 60 120 445 7,4 Bookmarks 90 150 60 75 90 465 7,8 Contacts 120 180 90 60 120 570 9,5 Average [h] 1,8 2,6 1,1 1,1 1,9 8,4 8,4 Median [h] 1,8 2,8 1,0 1,0 2,0 8,4 8,4 16 HR 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008
Application Modelling or Model based Testing? Thank You antti.heimola@nokia.com P.S. SW testers are still needed, e.g. exploratory testing 17 2008 Nokia AppModelling.ppt / Antti Heimola Seminar on Model-Based Testing as a Service Tampere Dec-4, 2008