Model Based Testing Practices

Transcription

1 Model Based Testing Practices Idea, Approach & Solution Bhaswati Sadhukhan Global Business Services (GBS) IBM(I) Pvt. Ltd. Kolkata, India Abstract--This paper on Model Based Testing Practices Idea, Approach & Solution contains the brief overview of Model Based Testing, its basic features; describes why this has been gaining acceptance over conventional testing. This also includes different approaches of Model Based Testing, the high level concept of steps to implement this kind of testing in a project like how to build a model, process of generation of expected input & expected output, how to run a test, it shows comparison of actual outputs with expected one in details. The paper also includes short description of few popular tools in this scenario, provides an idea how Model Based Testing can be used with agile methodology supported projects etc. Also furnished author s experience in a project that induced MBT technique in generalized view with benefits and constraints. Index Terms MBT, MDA, Model, PIM, PSM, SUT, Online & Offline MBT. T INTRODUCTION his paper on Model Based Testing contains the overview of Model Based Testing, its features; describes it s advantages over conventional testing. This also includes different approaches of Model Based Testing, the high level concept of steps to implement this kind of testing in a project, it includes short description of few popular tools in this scenario, provides an idea how Model Based Testing can be used with Agile methodology supported projects etc. Furnished author s experience in a project that used MBT technique in generalized view, and it s benefits and challenges. OVERVIEW OF MBT In order to perform any kind of verification or Testing, everyone has to use own model. That model may be very much person dependent, remain very short period of time in our brain. It can help us to know how the system under test (SUT) should act under a specified situation. In model-based testing, the model of the system s behavior can be made explicit and this model acts as the basis for the complete automation of the testing (As shown in the figure 1). Model based testing (MBT) is gaining importance for medium and large sized complex projects for adequate Test coverage and for its capability to accommodate faster changes in requirement. The idea is based on the concept of a model which should be platform independent, non executable & must be in technology agnostic format. A. Why it is gaining much importance in the market It is because when testing is done by generating and executing test cases (TC) using any popular automated testing tool that usually injects some constraint. Basically, the availability of the environment and other few factors would decide the number of test actually can run. Here the power of the strong model used in MBT comes in the picture. If the model describes the major functionality and the test cases can be generated automatically from it, it is very easy to maintain the test by tuning the test model any time to address the changes. The rest of the work goes under the automation. The maintenance of Test suite is much easier in MBT as well as the quality of the test. Model Driven Architecture [2] concept has two models - Platform independent model (PIM) which gets transformed into PSM (Platform specific Model) by the tool. The PSM can then be transformed into code via the code generation. The PIM can be reused to any platform with any technology whenever required. This PIM provides integrated system s functional behavior based on UML modeling standards. MBT is a kind of testing where the whole or sometimes the parts of the test cases are getting generated from a model which usually describes the functionality of the system under test. Industry known synonyms for MBT is Model driven testing, Test generation, Hardware in the loop, etc. A. Main Approaches There are two main approaches Online MBT, Offline MBT. In offline MBT, the Test Suites generate and can be executed ISQT Copyright

2 later on. Online MBT goes for the dynamic generation and execution of the Test cases. Online MBT - In the case of Online MBT, test generation as well as the execution happen on-the-fly. It gives out the test case in advance and not from the model. This kind of testing is quite random and the test strategy is to maximum coverage. Here after getting the output, next step can be designed and performed. Using Online MBT strategy we can test non deterministic systems. Offline MBT - In offline MBT, the Test Suites generate and can be executed later on. This can be helped to create a chain of tool. This also supports Automatic Test Case generation. MBT IMPLEMENTATION STEPS Name of The Technique Finite State Machines[1] Markov Chain Model (Markov process) Decision Tables State Charts Definition with Example It ensures that the generated test cases cover the model. It describes the finite no. of states & transitions between those states with action associated. In this method, the test cases get generated by the random process, it makes coverage criteria quite difficult to ensure. The mathematics of Markov chains provides formulas to determine expected values. The table shows the possible sets of conditions & actions as a result A state chart describes the states of a system & shows the events those results from a certain change of the system from one state to another. There are some specific steps for implementation of MBT. 1. Building of a model 2. Generation of expected input & expected output 3. Test Run 4. Comparison of actual outputs with expected one Comparison will lead to the further action to be decided for measuring quality Test Input Generation Building of a model Conversion of TC TC Execution Comparison Measure Quality Figure 1 Steps of MBT TYPE OF MODELS Generation of expected Result There are different kinds of models. One can be constructed from source code directly or sometimes requirements get reflected in the Test model. Design model is another approach where the intended behavior of the system is specified. Below is a tabular representation of different kind of techniques to construct a model of SUT. Building of a model IMPLEMENTATION STEPS DETAILS Sequential steps of building of any state based model: Listing of all possible inputs. Listing of all scenarios where each of the inputs can be applied or cannot be applied. Listing of all situations where depending of the context, the input causes different output. Model independence: While creating the test model it should be decided first to make the model isolated from the developer s model. The test model should put stress on negative testing scenario much more than the developer s one will restrict to the positive aspects of the SUT. The developer model should cover the complete set of user requirements where as the tester s model should concentrate on the critical business logic in detail apart from the requirement coverage. Sometimes, developer and tester share the same model and that reduces the benefits in greater extent. In that case usually, testers can use hybrid model. In this model, first developers model is prepared and that is being reviewed & enhanced with some extra input from testing point of view and can be used in the next development stage. a. Generation of expected input & expected output Now, tester can make use of the model created to generate the test cases. These are basically specifying the inputs and the expected output. For finite state machines, an algorithm is implemented which can traverse the state transition diagram at random. With the modeling tool and automation it is easy to create test with sequence of input ISQT Copyright

3 & the generated path. There is a mechanism in MBT called Test Oracle or some kind of test vision which can be designed in such a way that it can find the correctness of the output generated for a complex system. This should have the flexibility enough to address the changes. Creating the Oracle is a challenge. In short, that can be done by comparing the output with some pre calculated output. b. Test Run or Test Execution This phase is little bit different from the traditional one. Instead of generating complete test suite, test cases are generated and executed, the data that failed being recorded only and so on. If crashes it automatically reset the environment for further proceedings. The test generation tool comes with the MBT environment and the test scripts are generated from the test data within the tool. c. Comparison of actual outputs with expected one With the help of the automation process, MBT helps to compare the output with expected one and detects the failure. Its success depends on how good in quality & up to what extent completeness of the Test Oracle developed. d. Comparison will lead to the further action to be decided for measuring quality From verifying the output one can measure the test coverage as well as the reliability of the SUT to decide whether more test suite needs to be generated, sometimes to redesign or enhance the model used and when to stop testing. TOOLS USED IN MBT The tools used in MBT are not popular in traditional development or testing. They are specific in the sense that they have the dependency on the input model and they are associated with the model notation used. Leirios Test Generator (LTD) It is one of the popular commercial Offline MBT tools and supports the notation i.e. UML state diagram. This tool has been provided by LEIRIOS. Main features are as follows- The OS supported is WINDOWS or LINUX Automated test case and executable script generation Import UML models from widely used tools, Coverage reports Traceability from requirements to test cases Test scripts generation to any execution environment The integration between the system modeling tools, LTD & Test management tools is much limited, thus challenging. Another one is Conformiq Qtronic (CQ) which automatically generates tests from models of the system under test, it executes the tests and also analyses the results. It has the following features. It can be used as both online & offline MBT tool OS supported is WINDOWS or LINUX Supports any UML 2.0 models and variant of LISP as input It has some extra features like it can support nondeterministic systems testing Also provides some stopping criteria for the testing team Biggest disadvantage is debugging of the model for finding bugs, it is very time consuming. MBT IN AGILE PROJECTS MBT can also be made suitable for the projects where agile methodology is being followed. Agile methodology is highly accepted in those projects where things can change a lot. The independent testing team never can cope up with the latest changes made. Moreover, developers trend always to create only the necessary documents at that point of time thus affects the test coverage for the testing team at huge extent. Here MBT comes up to reduce the information gap problem. In Agile, after every iteration the application needs to be tested; the testing should not demand for much effort to match with the agile methodology. As MBT is a Test system developing task not the test case generation as usual followed in conventional testing so it is perfect choice to introduce MBT in agile projects where features are divided into tasks. Testing development using MBT techniques would be treated as task and the corresponding effort is estimated and handled just like the development activities is handled & monitored in agile process. CASE STUDY BENEFITS AND CHALLENGES Here to be mentioned one real life exposure in a project where MBT has been used its benefits and the constraints faced by us. It was a domain specific approach for a banking project, where all the general activities like Maintain Customer Information, Loan information, Credit Rating for the Customers, Tailoring were maintained as separate applications. Communication made between those applications was a sequential, Trigger based Communication with Oracle form in the front end. In our case, most of the modules were developed in Java but there was one small module dealing with customer information where.net was chosen because the bank already ISQT Copyright

4 was using content management internally using Share Point. The PIM was suitable for this scenario. The system model had been built with Enterprise Architect by SPARXS SYSTEMS. The test process started with analyzing the requirements and modeling the system. The same model had been used by the testing team with some enhancement to design the test cases. The model was validated for correctness and completeness. The system requirements have been fed from the UML models to Qtronic. Conformiq Qtronic is capable of generating tests automatically from the system model. The tests are independent of the source code and depend on the model and the interfaces of the system under test. User requirement was to introduce high business agility in the technical platform since in the banking sector rules, regulations, customer preferences, choices, partner integration are getting changed day by day. SOA oriented approach is the solution for this requirement. Our project leveraged Model Driven Architecture in this scenario. Now the Qtronic evaluates the requirement and that are traced to the model. The gap found is corrected. The validated models are then used for the test generation using tool. The test has been executed against the SUT. The failed cases have been analyzed; we had to correct the SUT or the model accordingly. The test coverage has been increased, cost & time saved in measurable amount. It was a project for FP estimated with 7 modules to be developed; it went to production, considering change requests came in one year finally size came up to FP calculated as a part of project closure. Normally this kind of combined effort takes at least 20,000 FP, due to the introduction of Model Driven architecture, the effort reduced in the long run though initial effort was high apparently. Still it has some dependency mainly on the skilled resources. CONCLUSION Introduction of MBT in the above mentioned scenario had provided the following benefits: The test model has been created at the same time when the development model created. That helped to reveal and rectify the gap in the development model beforehand. The development time had not wasted for rework on the wrong implementation. The requirement change & the technology change didn t affect the testers effort much because of the automation based on the model. Redesign of the test cases was of little effort when requirement changes. The requirement coverage was better, non-repetitive useful tests had generated because of a concrete model. Regression testing for each & every iteration was a great success as the test generated automatically from the models. Schedules can be made shorter, cost lowers and quality can be made better. It really focuses defects in early phases like in Requirements specification and design. Reusability of the model in the long run is the most useful benefit to grab. Some real challenges that had been faced by the team: The total success depends on the skilled resources that have the knowledge of modeling. It is hard to manage a very concrete model. Initial effort to build a non executable model and make it functional is a big challenge. In a complex situation where more than one model exists, it is very difficult to manage all the models with version control, also maintaining the chain from requirements up to generation of test. In short, MBT has been used successfully on various projects, in order to get flavor out of this, one has to undergo huge up-front cost to create a test model initially but in the long run, and that can be definitely made up by the lower maintenance costs. Since the end users concentrate on high quality product with reduced test maintenance cost, thus MBT is quite promising to grow. It needs commitment from management and testers group, necessary set of unique skills & availability of supporting tools. Overall, MBT can be considered as more efficient & capable of increasing the final product quality in greater extent. ACKNOWLEDGMENT The author gratefully acknowledges the contributions of Vijaya Seshadri, IBM (I) Pvt. Ltd. for her valuable comments & guidance. REFERENCES [1] I.K. El-Far and J.A. Whittaker. Model based [2] Model Driven Architecture (MDA). Software testing. In John J.Marciniak, editor, see Encyclopedia of Software Engineering, Volume 1, pages Wiley-Inter Science, BIOGRAPHY Author Bhaswati Sadhukhan is experienced in Software Testing on various platforms ranging Mainframe, DWH, Web application, etc. She is ISTQB & OCA certified professional. ISQT Copyright

5 ISQT Copyright