After 14 Years of Manual Interoperability Testing, Finally the Process has been Automated

Transcription

1 White Paper After 14 Years of Manual Interoperability Testing, Finally the Process has been Automated Teraquant Corporation 370 Stirrup Trail Monument CO Phone: Fax:

2 Automation of the Interoperability Test Process Page 2/12 In partnership with Multivendor Interworking and the Interoperability Process To build network based services such as Unified Communications and Web Conferencing, best-in-class network devices are connected together in a multivendor environment. Much effort is expended in Interoperability Testing to ensure devices from different vendors interwork and provide seamless services. The testing workflow process can be categorized as follows: 1. Development Tests (Unit, Module, Component, ) 2. Standardized Functional Testing (Testing against a standard) 3. Interoperability Tests (Implementation against each other, n x m testing) 4. Performance Tests (High, long-lasting load) Classical Conformance Testing, i.e. testing for conformance of protocol implementation against an industry standard or specification, is losing ground due to the extensive effort required to check all protocol message exchange scenarios. Conformance Testing uses a piece of test equipment which generates all the test scenarios and checks for received message syntax, timing etc. Once the test suite has been implemented in the testing, process is automated and repeatable. However, the end goal is interoperability with other vendor' s devices in order to build a service. Interoperability testing, pair wise testing, is accepted as a solution to improve interoperability and the standards bodies and industry forums arrange regular events, where vendors may test interoperability with fellow industry participants. Examples of such interoperability events include: ETSI - Test Specifications, PlugTestss OMA Test Specifications, Test Fest SIP Forum SIPit, SIMPLEt IMS forum - Plugfests Enter Interoperability Testing into a search enginee and you will receive Over 700,0000 hits on Google, more than 1.3 million hits with Yahoo.

3 Automation of the Interoperability Test Process Page 3/12 Interoperability Testing involves inter-connecting two devices. Stimulation of these two devices to implement the test plan is manual and automation and repeatability is difficult to achieve. In addition, a more time consuming and intense phase is analyzing the behavior of the devices afterwards i.e. which implementation between the two devices is correct, according to the standard or RFC and which implementation is not correct, therefore preventing interoperability. Some InterOp events analyze each test or interaction between the two interoperating devices to understand in protocol terms for correct implementation or which party did not follow the recommendation. Other InterOp events just determine if the intended test or session was successful e.g. the voice call went through or the message arrive error free and analyze the details only in case of failures. Where Costs Are Eaten Up In Interoperability Testing Traces captured with tools like Wireshark are key to understanding why devices from different vendors do not interoperate. They are used at most Interoperability Events worldwide such as InterOp, Testfests & PlugFests. The problem is analyzing hundreds of traces by visual inspection is manual, time-consuming, error-prone and requires an expert. < 50% Test Design Test Specification Test Verification Test Efforts > 50% Test Implementation Figure 1 The current test effort problem

4 Automation of the Interoperability Test Process Page 4/12 At a recent ETSI IMS InterOp Plugtest event, [ the following analysis was conducted of the time and effort expended during the test event: The setting 43 InterOp test cases defined by ETSI 6 vendors participating at the Plugtest 30 test sessions Approx. 900 test runs In 3 days The effort 150 hours of testing and (40%) 225 hours of validation of traces to analyze behavior (60%) Total: 375 hours! (100%) 48 ManDays Test design and test specification is the innovative phase that cannot be automated. The test runs and verifying the results and looking at problems of unsuccessful test are the phases occupying by far the most time and should be automated. The challenge is how to achieve this time-saving, so we can concentrate on what should and should not be tested, not how to test it. Reducing Costs in Interoperability Testing The key to efficiency and reduced time and costs is to increase reusability. Reusability across different types of testing, reusability of information between System Architects, Developers, Testers and Managers, reusability of test implementations and reusability of knowhow. This can all be captured for the purposes of computer automation in a common test language across the different phases i.e. Development Tests, Standardized Functional Testing, Interoperability Tests and Performance Tests. Such a test language developed by the Standards Institutes is TTCN (Test and Testing Control Notation). Having being used by the telecommunications and

5 Automation of the Interoperability Test Process Page 5/12 networking industry for 24 years, TTCN is now TTCN-3. What is TTCN-3? TTCN-3 is a testing technology Telecommunication, Software, Automotive, Distribution Industry & other industries to achieve repeatability. in its third release, widely used by the Energy, Electrical test automation and The test language has been specifically developed for testing purposes, full-featured and is open. i.e. allows for test scripts to be used on a variety of multivendor test platforms supporting a TTCN-3 compiler. TTCN-3 possesses a textual and graphical test scripting language look- test scripts as a scripting language. Don t reinvent the wheel! Use the test definitions produced in development for Automation in your QA and-feel and is used to describe test purposes and also to implement department. In addition, complete test suites useful here are available for free download for example from the ETSI web site. For more on TTCN-3, visit the ETSI web site /Benefits.htm Figure 2 - Reusability between different types of testing

6 Automation of the Interoperability Test Process Page 6/12 So to optimize efficiency, automate the part of Interoperability Testing that takes most of the time Verification. We need to reuse existing test frameworks and Profit from investments already done. Extensive Interoperability Test cases already exist for many SIP, IMS, HSS, IPv6, SS7, SIGTRAN & WiMax scenarios, as mentioned above. And these Interoperability Test scenarios exist in an industrial grade test automation framework and mature, well accepted processes, workflows and tools. That framework is TTCN-3. Figure 3 - The full InterOp test run at a glance How Those Costs Savings were Proven So how was this framework applied to the ETSI IMS Interop Plugtest event in Slovenia in November, 2008? This solution addresses the heart of the problem.

7 Automation of the Interoperability Test Process Page 7/12 Figure 4 - Configuration of the Trace Analyzer At this last particular IMS Workshop [ Theofanis Vassiliou-Gioles, CEO at Testing Technologies, talked about "How to increase efficiency in interoperability testing by using test automation frameworks", presenting methods to significantly reduce the time spent in verifying traces. The techniques were put to the test at the event in Slovenia, successfully reducing the time to evaluate each Interoperability Test up to 50% with 100% accuracy! After twelve hours of trace reading, even the most versatile expert encounters his limitations. This manual offline analysis of traces requires meticulous concentration, is error-prone and depends on experts who are able to understand and validate content and

8 Automation of the Interoperability Test Process Page 8/12 sequence of data captured in each interoperability session. By using Automated Trace Analysis, not only is the time spent to evaluate the tests reduced to minutes butthe validation accuracy is always 100% compared to manual analysis. Figure 5 - Pinpointing to the problem with a single click The configuration of the 30 test sessions were captured in 30 configuration files. Each configuration file provided the necessary parameters for the 43 test scenarios. The participating Devices under Test [DUT s] were interconnected and these test scenarios initiated. Wireshark software monitors the message flow between the DUT s. These traces are imported into Testing Technologies TTworkbench which applies Automatic Intelligent Trace Verification based on the same 43 TTCN-3 based test cases used for the Plugtest event.

9 Automation of the Interoperability Test Process Page 9/12 The Testing Technologies solution, built on top of the TTworkbench product, not only automatically validates traces, but also visualizes the interoperability scenarios graphically and produces human readable test reports. - All this, at a touch of a button. The panel discussion at the ETSI IMS Workshop identified the lack of thorough Conformance Testing as one of the main issues when evaluating interoperability test results. However, it has also been accepted that classical Conformance Testing, as performed in the days of ISDN, ATM, and GSM, is far beyond any economic reasoning. Testing Technologies' goal is to bring two worlds together - the Conformance Testing world with its typically high degree of test automation driven by a Tester Simulator and the Interoperability Testing world, where devices are simply plugged together and usage scenarios are used as test criteria. Results of the Trial The TTworkbench Automated Validation Solution has been successfully applied at the 2nd IMS Plugtest hosted by ETSI in Bled, Slovenia. Stephan Schulz, team member of ETSI Plugtest, summarizes Testing Technologies' tool support as follows: "All in all, we have been very satisfied with your support and performance of the tool. We have been able to significantly speed up conformance analysis. Results from the first test sessions included 100% matches of tool verdicts and manual analysis, and that is a strong statement." What did we Gain in Quantitative Terms? The total time to run tests was not affected because that is still manual. However, validation of the traces after the test session was drastically short circuited through the Intelligent Auto Trace Analysis done by TTworkbench. The first time this was done, time was allowed to tailor the Auto Trace Analysis. So total time to complete test execution and validation benefitted by a time reduction of 25%. The second time running these IMS test scenarios were used, time to execute and validate was reduced by 53% as the Auto Trace Analysis tool can be reused as-it-is.

10 Automation of the Interoperability Test Process Page 10/ Total time for validation Total time for execution 0 Manual Validation Automated Validation Check out the website of the IMS Workshop to obtain presentations and meeting minutes of workshops. [ Total time for preparation Total time for validation Total time for execution 50 0 Manual Validation Automated Validation w Prep

11 Automation of the Interoperability Test Process Page 11/12 How can this be used in a QA Laboratory? As mentioned above, TTCN-3 Test Suites covering many of your QA test purposes are available for free download from the ETSI web site. Ref These test cases can be run on any TTCN-3 compiler (such as TTworkbench) to provide a stimulus to a Device under Test (DUT) in a QA or Test lab, outside of an InterOp event, without the involvement of the other DUT to be interoperated with. These Test Suites preparing a device for the interoperability events, ensuring it has minimum conformance capabilities, and thus increasing the likelihood to interoperate with other devices. form the basis for test automation for Regression and Soak Testing processes.

12 Automation of the Interoperability Test Process Page 12/12 Further Information TTworkbench is distributed in the USA & Canada by Teraquant Corporation, a Test solutions and Test Systems Integrator based in Colorado. Teraquant provides highly focused Test Solutions, integrating Best-in-Class test products from industry leaders to meet specific test objectives and test plans. Teraquant works with many of the leading Telecom Vendors and Service Providers, providing test automation and Network Monitoring Systems for SS7, ISDN, wireless & VoIP. Teraquant can be reached at admin@teraquant.com located at: 370 Stirrup Trail Monument CO ph: fx: Editor Notes: Please admin@teraquant.com for further information or any questions.