Agile Test Methodology for B2C/B2B Interoperability

Transcription

1 Agile Test Methodology for B2C/B2B Interoperability A Dissertation by Jungyub Woo Department of Industrial and Management Engineering Pohang University of Science & Technology Chair of Committee: Committee Members: Dr. Hyunbo Cho Dr. Boonserm Kulvatunyou Dr. Sooyoung Kim Dr. Kwangsoo Kim Dr. Jaekoo Joo June 2007

2 ABSTRACT In the B2B/B2C environment, many applications and documents requires to be tested for conformance and interoperability. However because there are a lot of standards, solutions, specific environments, and test needs, it is difficult to develop all testing tools (i.e., testbeds) to cope with these test diversity. The testbed should be developed more rapidly and easier. In addition, the implemented testbed should be accommodated for various testing specifics. Thus we have proposed the agile test framework (ATF), which consists of test suite design and its execution model. The test suite is designed with a double layer: abstract and executable test suites. The execution model is also invented considering as reusability and plug-and-play concepts. This approach makes test engineer rapidly generate test suite and rapidly implement testbed. It also enables the implemented testbed to test various applications and documents based on diverse standard specifications. In other words, ATF is a test framework that has high reusability, high extensibility and high efficiency for the test. In conclusion, ATF makes B2B/B2C systems interoperable rapidly Key words: Agile test framework (ATF), business-to-business (B2B) integration, conformance test, interoperability test, standard, test automation, and test suite generation i

3 TABLE OF CONTENTS I. Introduction... 1 I.1. Rational for the Research... 1 I.2. Research Objectives... 2 I.3. Organization of Dissertation... 3 II. Literature Review... 5 II.1. Chapter Overview... 5 II.2. B2B Interoperability and Testing... 5 II.3. Requirements for B2B/B2C Testing... 6 II.4. Existing Test Approaches... 7 II.4.1 ebxml Implementation, Interoperability and Conformance (IIC) Test Framework7 II.4.2 RosettaNet Self-Test-Kit II.4.3 WS-I Test Tools II.4.4 Testing Test Control Notation (TTCN-3) II.4.5 Comparison Matrix among the Existing Test Frameworks II.5. Chapter Summary III. Test Suite Design for Agile Testing III.1. Chapter Overview III.2. Design Factors of Agile Test Suite III.3. Overview of Agile Test Suite Design III.3.1 Double-layers of Test Suite III.4. Abstract Test Suite Design III.4.1 Test Assertion III.4.2 Test Procedure III.4.3 Test Environment III.5. Executable Test Suite Design III.5.1 Scripting Primitives III.6. Chapter Summary IV. Execution Model for Agile Testing IV.1. Chapter Overview IV.2. Considerations of Agile Execution Model Design IV.3. Design of Agile Execution Model IV.3.1 Conceptual Architecture of Agile Execution Model IV.3.2 Definition of Test Components and Test Interfaces IV.4. A Reusable and Reconfigurable Testbed Development using Agile Execution Model36 IV.4.1 Developing Test Components IV.4.2 Realization of Test Interface IV.5. Case Study IV.5.1 RAMP Conformance Test IV.5.2 Quality of Schema Design IV.6. Chapter Summary ii

4 V. Test Harness and Test Execution using Agile Test Framework V.1. Chapter Overview V.2. Conformance Testing vs. Interoperability Testing V.2.1 Conformance Testing Harness V.2.2 Interoperability Testing Harness V.2.3 Test Harness Problems V.3. ATF-based Conformance Test Harness V.4. ATF-based Interoperability Test Harness V.4.1 Test Harness Design V.4.2 Development Test procedure V.5. Case Study V.6. Chapter Summary VI. Case Study Inventory Visibility and Interoperability VI.1. Chapter Overview VI.2. Inventory Visibility and Interoperability (IV&I) Project VI.2.1 AIAG Works VI.2.2 E-Kanban Business Scenario of IV&I Phase II VI.2.3 Testing Requirements VI.3. Test Approach VI.3.1 Information Mapping Test VI.3.2 RAMP Messaging Test VI.4. ATF Testbed Implementation VI.4.1 Test Case Development VI.4.2 Testing Tool Implementation VI.5. Chapter Summary VII. Concluding Remarks iii

5 LIST OF FIGURES Figure II-1 IIC Conformance Test Framework... 8 Figure II-2 IIC Interoperability Test Framework... 9 Figure II-3 High-level system diagram of RNSTK Figure II-4 Test flow diagram of RNSTK Figure II-5 High-level system diagram of WS-I Test Tools Figure II-6 Test procedure of the TTCN-3 test framework Figure III-1 Typical test scenario from a standard to testing Figure III-2 Agile Test Suites and their relationship Figure III-3 Design of Test Assertion Figure III-4 Example diagram of Test procedure Figure III-5 Design of Test Procedure Figure III-6 Design of Test Environment Figure III-7 General architecture and execution model of etsl Figure IV-1 Conceptual architecture of agile execution model Figure IV-2 Part of exemplary configuration information in an abstract test suite Figure IV-3 Sample code to retrieve a key value from XPATH validation properties Figure IV-4 Code to retrieve the key value of the WSDL URL Figure IV-5 Part of exemplary internal process script as WS-BPEL Figure IV-6 Part of a TVE WSDL service description Figure IV-7 (A) Sample RAMP executable test cases and (B) Internal Process Script Figure IV-8 RAMP Testbed Implementation Figure IV-9 (A) Sample QOD executable test cases and (B) Internal process script Figure IV-10 QOD Testbed Implementation Figure V-1 Conformance testing versus Inoperability testing Figure V-2 General conformance testing harness Figure V-3 Accordare s collaborative test harness Figure V-4 KorBIT interoperability test harness Figure V-5 Conformance testing harness for ATF Figure V-6 Test environment for conformance testing Figure V-7 Intelligent interoperability testing harness for ATF Figure V-8 Test Environment for Interoperability Testing Figure V-9 Test procedure for the connecting mode Figure V-10 Test procedure for the debugging mode Figure V-11 Exemplary SOAP message encryption and decryption Figure VI-1 AIAG E-Kanban Interaction Diagram Figure VI-2 Kanban authorization of IV&I Business Scenario Figure VI-3 Proposed semi-automated input testing procedure Figure VI-4 Proposed semi-automated output testing procedure Figure VI-5 Sample of test assertion for Information Mapping Test Figure VI-6 Sample of test assertion for R1010 requirement in the RAMP test iv

6 Figure VI-7 Test procedure for IV&I Customer Figure VI-8 Test procedure for IV&I Supplier Figure VI-9 Test Environment for IV&I Scenario Figure VI-10 A part of ETSL test case to verify supplier Figure VI-11 ATF-based IV&I testbed v

7 LIST OF TABLES Table II-1 B2B interoperability layers... 5 Table II-2 Comparison matrix for conventional and proposed test frameworks Table IV-1 Abstract definitions of Test Interfaces Table V-1 Example of WS-Security policy Table VI-1 Requirements of RAMP specification vi

8 I. INTRODUCTION I.1. Rational for the Research As the electronic collaboration through the Internet infrastructure, also called the business-to-business (B2B), has matured, the number of software solutions to support automated, secure, and reliable transactions has been growing. Because different businesses typically have different information requirements, processes, and protocols/interfaces to collaborate (or communicate) with others, the B2B/B2C requires several related standards as common references. There are several aspects of these standards such as those that specify regulation and conventions for common data format, communication protocols, and interfaces. There are a growing number of standards, potentially competing, addressing these aspects as requirements are discovered (Shaw et al. 2000). Even though B2B software solutions are implemented based on the same standard, they cannot avoid interoperability problems because 1) the standard may contain ambiguous requirements which could be differently interpreted, assumed, or understood by developers, 2) the standard may have defects or omissions, and 3) developers occasionally partially implement the standard, and 4) the standard is designed to be flexible (intentionally left ambiguous) allowing different specializations and implementations (Shaw et al. 2000) (Moseley et al. 2004). For these reasons a B2B/B2C solution must be tested in two different testing modes conformance and interoperability. Other testing modes may present as different requirements arise. B2B/B2C standards typically contain a large number of regulations and conventions. A large number of solutions must also be pair wise tested in the interoperability testing mode. This nature calls for automated testing facilities testbeds. Testbeds for different standards have different test components, different test materials, and different configurations because each of them focuses on different specific testing requirements even though they are similar and potentially related. This makes test case development expensive for domain experts because they have to learn different schemes when writing tests for different standards. Similarly, it makes test execution expensive for the solution provider, because they have to potentially develop different supporting test components and get familiar different artifacts when testing for different standards. In addition, because standard specific testbeds are ad hoc and short-sighted, they commonly have the characteristics below: - Low (re)usability: Testbeds are independently implemented without common scheme making their components useless in various standards and various environments. - Low extensibility: Testbed implementations cannot scale to address other standards than the one they are specifically designed for. - Low Efficiency: Most of automated testbeds provide uninformative test results. It leaves testing users with little clue when debugging their systems. In summary, this section describes the issues associated with the current practice of building a testing 1

9 facility for B2B standards. These issues, if resolved or alleviated, present potential cost and time savings in the B2B testing. This research proposes an agile test framework to address these issues and realize these potential savings. The proposed test framework will allow components within the testbeds developed in compliance with the framework to be reused for various B2B/B2C solutions, various testing modes, various standards, and various computing environments. In addition, time saving is realized through automatic configuration of these test components. The proposed test framework also aims at enhancing testing efficiency by enriching test materials and test result reports. I.2. Research Objectives The eventual objective of this research is to develop the Agile Test Framework (ATF) that is reusable, configurable, and efficient for the B2B solution testing. The detailed objectives are 1) to develop a specification necessary for designing and generating a modular test suite including multi-layer test cases, 2) to develop reusable and reconfigurable test components, 3) to develop guidelines for composing an ATF-based B2B test and for analyzing test results, and 4) to demonstrate the test framework s agility by showing actual use-case studies in various B2B tests. To achieve the first objective, following research issues are considered: - Modular test suite: The test suite consists of test cases and related test data. A test case includes information about test procedure and assertions. These two pieces of information are tightly coupled in conventional test cases consequently making them less reusable. They also have no configuration information of testbeds. It makes test cases used in limited testing tools. Therefore, the proposed test suites should be designed with modular concepts. It means each module should be easily separated and its relationship with the other module should be welldefined. The proposed test suites will also include information about test environment to represent test harness. - Double layer test case: The proposed test case consists of two layers, abstract and executable test case layers. The abstract test case allows test procedure to be captured in the domain/standard specific way. This enables a standard/domain expert to easily create test cases for its own tests because the expert does not need to learn the technical details about testbed execution, but to describe the test cases based on its own domain knowledge. The executable could be ready to be executed by testing tools. It should be generic for any B2B/B2C standard and environment. The second objective addresses following research issues: - Reusable test components: Test components are basic building blocks to form a testbed. We classify the test components into stationary ones and non-stationary ones. The stationary components are completely independent of the types of standards, solutions, testing modes, and test configurations of interest. Therefore, any testbed configurations re-use the stationary components as-is. On the other hand, even though the non-stationary components are re-used in any testbeds, they need to be flexibly re-defined or re-implemented according to specific 2

10 testing requirements. - Plug-n-play test components: Interfaces between test components and with systems under test (SUTs) are defined well. To support the interfaces, we define plug-and-playable test components. These components make testbeds built on ATF reconfigurable, versatile, and agile to various tests. Research issues related to the third objective are as follow: - Test suite development guideline: Test suites, especially an abstract test case, will be created by a domain expert who has developed and, therefore, best knows about the standard specification of interest. The proposed ATF will provide a guideline for developing the abstract test case. This will help the domain expert reflect his/her test requirements in the test case. The guideline will also include specifications of a translator that transforms the abstract test case to executable test cases. - Test execution guideline: The test case will be executed by a system developer (namely a testing user). The proposed test framework will provide a guideline for executing tests. This will help the testing user test target system. - Test reports analysis guideline: After the testing, the testing user will receive test reports from the testbed. The testing user will modify his/her system against interoperability problems reported. The test framework will provide a framework for generating informative test reports. The forth objective has research issues as follow: - ebms and Web Services messaging testbeds implementation: The messaging testbed for ebxml messaging and Web Services standards will implemented based on the proposed ATF. These testbeds will demonstrate ATF s agility. It will especially focus on the reusability of test components and test cases. - Information mapping testbed implementation: The information mapping testbed will be implemented based on the proposed ATF. The testbed will demonstrate the agility and reusability by showing that the testbed is formed with several test components used in the ebms and the Web Services testbeds. I.3. Organization of Dissertation The remainder of this dissertation is organized as follows. Chapter II provides literature survey regarding conventional test frameworks for B2B interoperability. A comparison of existing test frameworks is also presented to provide research rationales. Chapter III provides a design of test suite specification for test cases/materials. In order to consider a reusability of test case/materials, test suite specification considers the modular and layered concepts. Chapter IV proposes an execution model of the test framework for an agile testbed implementation. In this chapter, formal definitions of test components and distinctive features of the agile test framework such as reconfigurability and reusability are discussed. Chapter V addresses a guideline of usage of testbed on the basis of proposed test framework. In this chapter, conformance and interoperability test harnesses are introduced and it is 3

11 explained how to realize them. Chapter VI demonstrates an agility of the proposed test framework when a testbed is developed and used in order to resolve interoperability problems in the real world. Finally, summary of this dissertation and concluding remarks are presented in Chapter VII. 4

12 II. LITERATURE REVIEW II.1. Chapter Overview The objective of this Chapter is to review and discuss relevant research works that are underlying test frameworks for the B2B/B2C testing. Many test frameworks and testing tools have emerged for B2B/B2C standards. In particular, we give a comparison matrix for existing test frameworks and derive requirements for our test framework proposal from them. II.2. B2B Interoperability and Testing As the late 1990s saw the emergence of many standards for e-business, there are a number of papers discussing them. According to Hasselbring and Weigand (Hasselbring and Weigand 2001), the standardization for the exchange and processing of documents can be at the lexical level of character sets, at the syntactical level of document structures, and at a deeper semantic level of dictionary and integrity constraints. Respectively, Nurmilaakso and Kotinurmi (Nurmilaakso and Kotinurmi 2004) have argued that business partners have to know what information, when, and how to share before they can do e-business. Therefore, B2B standards prescribe business process, business information specification, and messaging infrastructure. Table II-1 shows B2B interoperability layers and typical ones of associated standards. Table II-1 B2B interoperability layers B2B layer Functionality Standards Business Information Business Process Business Messaging Information specification Information constrains Usage Process definition Process execution Transaction Choreography Orchestration Message packaging Reliable Security Addressing CC, UBL, OAGIS BODs, UNSPSC, VICS, PIDX, GISB, xcbl WSCI, BPML, ebxml BPSS, WS-BPEL, WS- Transaction, ebxml Registry, ebxml CPP/CPA, UDDI ebxml Messaging, AS1, AS2, WS- Authorization, SAML, WS-Federation, WS- Reliability, WS-Reliable Messaging, WS- Security, WS-Trust, WS-Policy, XML Signature, XML Encryption, XACML 5

13 Whittaker and Jorgensen (Whittaker and Jorgensen 1999) have argued that software often fails because of 1) improperly constrained input, 2) improperly constrained stored data, 3) improperly constrained computation, and 4) improperly constrained output. A standard-based B2B software solution requires testing because it prevents late detection of errors (possibly by dissatisfied users), which could require complex and costly repairs. Testing can detect errors, and evaluate and approve the solution s quality beforehand (Schieferdecker and Stepien 2003). An automated test approach helps in particular to efficiently repeat tests whenever needed for new system releases in order to assure the fulfillment of established system features in the new releases (Schieferdecker et al. 2001). II.3. Requirements for B2B/B2C Testing A test framework is defined as a set of utilities, style-sheets and documentation that describe and facilitate development, documentation and use of the tests (W3C 2002). Before the test development work, W3C recommends the following actions: - Understand the testing scope, - Assess the testing work already done for the referenced specifications and facilitate reuse of test materials where applicable, and - Assess the risks taken by using specification that has a lot of flaws such as contradictory statements or ambiguities. B2B environments continuously require new and emerging standard specifications and software solutions in pursuit of progresses in information technology and diverse market. It is expensive and time consuming to implement a new testbed for every emerging standard. A B2B testbed should be developed such that it can scale to facilitate testing of various standards and software solutions. Therefore agility of test framework is urgent. In the B2B environments, the testing agility could be considered for: - Testbed administrators: They implement a testbed based on a specific test framework for specific testing objectives. Therefore, they want the ability to quickly and easily construct a new testbed. - Domain/standard experts: On the implemented testbed, they develop the test cases and materials for test executions. They want the ability to quickly and easily create a test suite. - Testing users (software venders): By the testbed and test suites, they test their solutions. They want the ability to correctly resolve conformance and interoperability problems of their solutions Required agility must be satisfied by the proposed agile test framework. That is, proposed agile test framework will have characteristics as follows: - High reusability: Most of test materials (test cases and various test data) and test components can be reused. It minimizes the cost and efforts for developing a testbed. - High extensibility: The testbed, which is implemented based on the proposed agile test framework, can test various applications and documents and be used for diverse standard 6

14 specifications using the reconfiguration approach. - High Efficiency: The test suite document, test process, and test report should be user-friendly for various participants in the test. It assists them to catch interoperability problems of their applications/documents. As a result it minimizes the time and efforts necessary for debugging. II.4. Existing Test Approaches For the purpose of test automation, testing systems (testbeds) and test material/case (test suite) have been designed and developed. Most of test frameworks are proposed to meet the requirements of testing for a specific corresponding standard. Therefore they generally have narrow scope and provide insufficient testing services. In this chapter, we discuss four representative test frameworks: ebxml IIC, RosettaNet Self-Test-Kit, WS-I tools, and TTCN. II.4.1 ebxml Implementation, Interoperability and Conformance (IIC) Test Framework As many vendors implement ebxml (Electronic Business using Extensible Markup Language) compliant systems, interoperability among them has been a key for seamless e-business environments. In January 2000, the OASIS chartered the ebxml Implementation, Interoperability and Conformance (IIC 1 ) TC to provide a means for software vendors to create infrastructure and applications that adhere to the ebxml specifications and are interoperable. The OASIS IIC TC proposed a test framework, OASIS ebxml IIC Test Framework (IIC 2001), to verify interoperability among ebxml compliant e- business systems as well as their conformance to the standard. The initial scope of the IIC Test Framework covers both conformance and interoperability testing of ebxml Message Service Handlers (MSHs). II IIC Conformance Test Framework The IIC test framework provides a conformance test framework to verify whether a candidate system under test (SUT) conforms to the ebms specification. Figure II-1 depicts the conformance test harness in the IIC Test Framework. 1 ebxml Implementation, Interoperability and Conformance (IIC) Technical Committee, 7

15 SUT Testbed Conformance Test Cases Test Service ebms Message Transport Adapter Test Driver SUT Test Reports Figure II-1 IIC Conformance Test Framework The framework consists of a System under Test (SUT), Test Driver, and Test Server. Test Driver is a main test component. Test Service is an auxiliary test component. On the SUT side, the SUT and test service are tightly coupled. If the SUT gets a message from the testbed, test service notifies the next step to the SUT by predefined rules. On the other side (testbed side), the test driver is a component to drive the execution of each step of a test case; test cases are execution instructions that contain the procedures and assertion scripts; and test reports contain test results generated while performing test cases. The procedure to perform the conformance testing is as follows: first, the test driver generates a request message from a test case and sends it to the SUT. The message is sent directly to the SUT as a standard message (ebxml Messaging Services Specification Technical Committee 2001) through Hypertext Transfer Protocol (HTTP 2 ) or Simple Mail Transfer Protocol (SMTP 3 ). When the SUT receives the request message, the SUT notifies the test service of the received message. Second, based on the content in the received message, the test service requests that the SUT generates an ebxml message response back to the test driver. Third, the test driver verifies the received message by using the assertion scripts of the test case. These procedures continue until all test steps of the test case have been performed. Finally, the test driver generates test results. II IIC Interoperability Test Framework The IIC test framework also provides an interoperability test framework to verify whether two or more candidate SUTs can correctly communicate with each other. Figure II-2 depicts the interoperability test 2 HTTP Hypertext Transfer Protocol, 3 Send Mail Web Site, RFC 821 Simple Mail Transfer Protocol (SMTP), 8

16 harness in the IIC Test Framework. Driver Party Responder P arty Test Service Transport Adapter Test Driver Test Service SUT Interoperability Test Case Test Reports ebms Message SUT Figure II-2 IIC Interoperability Test Framework The framework consists of two SUTs, two test services, and test driver. Each SUT plays different roles: one party, the driver party, initiates the request of the test driver. The other party, the responder party, responds to the messages initiated by the driver party. The test driver is directly connected to the test service at the driver party. The procedure to perform the interoperability testing is as follows: first, the test driver generates a request message from a test case and sends it to test service of the driver party. The message is sent to the SUT of the responder party through HTTP or SMTP. The SUT of the responder party notifies the test service of the received message. Second, the test service requests that the responder SUT generates a response message and send it to the driver party. Third, the SUT in the driver party receives the response message and notifies the test service. Fourth, the test service notifies the test driver of the received message. Fifth, the test driver verifies the received message by using the assertion scripts of the interoperability test case. These procedures continue until all test steps of the test case have been performed. Finally, the test driver generates test results. II Weaknesses of IIC Test Framework for the B2B Standard testing Though the scope of the IIC Test Framework can be applied to all standard specifications (Durand et al. 2003) (Kulvatunyou et al. 2003), ebxml communities have primarily applied it to test ebms solutions (KorBIT 4, NIST 5 ) because this test framework has some weaknesses as described below: - Difficult to reuse the test case: The IIC TC already provides ebms conformance test suite. 4 Korea B2B Interoperability Testbed (KorBIT) Web Site, 5 NIST Information Technology Laboroty Web Site, ebxml Test Framework, 9

17 However, for another Simple Object Access Protocol (SOAP 6 ) conformance testing, none of the ebms test cases can be reused because 1) the test case is directly associated with the testing procedure which includes test steps and verifying assertions; and 2) each test step is directly associated with a specific message packaging method. The test case cannot be reused, for example for the Web Services SOAP test case, even though they have the same messaging steps and verifying rules as ebms test case. - Difficult to reuse the test component: The IIC test framework proposed the test driver and the test service as the test components. Except the ebms testing, the test components should be redesigned because the major parts of functionalities are dependents on that specific standard. Unfortunately, the IIC test framework did not distinguish dependent components and independent components. This makes test components have low reusability. This problem makes standard organizations hesitate to adapt IIC test framework as their common platform for testbed implementations. - Difficult to configure the test harness: The test harness is a deployment relationships between test components. The IIC test framework defines two test harnesses for the conformance and interoperability testings. However these test harnesses are fixed and test components and test case specification are designed upon these harnesses. It makes test components hard to adapt to a new a test harness. - Hard to make and understand the test case: The IIC test framework provides a generic executable test case specification, which is independent of B2B standards. In other words, the IIC test case uses a low level language in order to represent all B2B functionality. The needs to understand and specify test cases using this low-level language make it difficult for the domain (standard) experts to develop test cases. These test cases are also difficult to understand for the same reason. - Low testing efficiency: The IIC test framework provides a test report which only indicated whether each test step is passed or failed without any reason given. Even if there is a temporary network error, the test report outputs the same testing failure message. Therefore, testing user repeats the testing needlessly. Also testing uses do not get enough information from the testbed when they meet an interoperability problem during the test. This is one of the major reasons that a lot of software vendors disregard the B2B interoperability testing. II.4.2 RosettaNet Self-Test-Kit RosettaNet is a non-profit consortium aimed at establishing standard processes for the sharing of business information (B2B). RosettaNet is a consortium of major Computer and Consumer Electronics, Electronic Components, Semiconductor Manufacturing, Telecommunications and Logistics companies working to create and implement industry-wide, open e-business process standards. These standards form a common e-business language, aligning processes between supply chain partners on a global basis. The RosettaNet Self-Test-Kit (RNSTK) is a software package supplied by the RosettaNet 6 W3C Web Site, Simple Object Access Protocol, 10

18 consortium 7. The RNSTK is an application developed by RosettaNet; it is a test system for compliance with the RNIF specification. The purpose of the RNSTK is to provide the ability to validate RNIF interfaces. II Test framework of the RosettaNet Self-Test-Kit The RNSTK consists of 8 test components as listed below: - Controller: Overhead control of the test process. - Inbound: Handles incoming messages. - Outbound: Handles outgoing messages. - Repository: Storage for the Part Interchange Process (PIP) specifications and logs. - Results: Handles the result files and display pages. - Secure message: Handles security issues. - Standards: Holds message standards. - Validators: Handles validation against PIP and RNIF specification. Figure II-3 is a high-level diagram of the system where collaboration between functionality in the 8 test components is visualized. RNSTK Local Repository Result Engine Result Logs Status Engine Apache Web Server Config Engine SUT RosettaNet Test Controller RN Listener RNIF Message Figure II-3 High-level system diagram of RNSTK 7 RosettaNet Web Site, RosettaNet Official Homepage, 11

19 The RosettaNet solution that needs to be tested is in this system referred to as the system under test (SUT). The SUT sends business messages and action signals to the RNSTK and these messages and signals are validated by the RNSTK against a given XML-file containing the RNIF verification rules. The templates for the PIPs are stored in a repository on the servers local file system and every message sent to the RNSTK is validated against these templates. Figure II-4 describes how incoming and outgoing messages are handled by the RNSTK. The process is quite complex and requires a lot of logging and validation. The test begins with the user filling out configuration values and sends them to the server. Then the server starts a listener that waits for a message from the SUT. When the message is received it is processed for unpacking and MIME validation. After this the test system adds references and the message get validated against a RN DTD. When this is done the process is finished and results get saved and the status web page gets updated. The outbound processes are very similar but are in a reverse direction. Each PIP action has its own validation class in the repository that validates the incoming RN message from the SUT. The validation is conducted using the DTD file for the current PIP action and it is tested in a child-parent manner. All results from the validation are stored in a log that is later on presented to the testing user using the status engine (RosettaNet 2004). Inbound Process Configure Receive Incoming MIME Unpack & Validation Apply References (From Standard) Test Handler Update Status Validate Message Outbound Process Load Message Build Component Apply References (From Standard) Package Message (MIME) Update Status Send Message Figure II-4 Test flow diagram of RNSTK 12

20 II Weaknesses of RNSTK Test Framework for the B2B Standard testing Though the scope of the RNSTK Test Framework applies to broad test requirements of RosettaNet B2B interoperability testing (Kjellin 2005), it is difficult to scale it testing of other standards due to the following reasons: - Specific test suite: The test suite specification of a RNSTK is the RosettaNet specification itself (PIP and RNIF). That is, PIP describes the test procedure and RNIF defines the messages. Also all verification rules come from XML constraints documents of these standards. That is, RNSTK does not have a generic executable test script. - Low testing efficiency: This weakness is similar to that of the IIC test framework. That is the test report is limited to the passed and failed feedback without further information. II.4.3 WS-I Test Tools The Web Services Interoperability organization (WS-I 8 ) organization is dedicated to meeting the needs of Web Services developers, so as to enable them to develop and deploy interoperable web service on the computing platform and development language of their choosing. It is a primary work of WS-I to make profiles, which contains a list of named and versioned of Web Services specifications together with a set of implementation and interoperability guidelines. These profiles and guidelines recommend how the specifications should be used to develop interoperable web services. WS-I is also developing a set of testing resources (WS-I Test Tools 9 ). These testing tools and the configuration data used with them have helped Web Services developers ensure that their web services conform to the profiles and interoperability guidelines. The tools monitor the interactions with between web services, record those interactions, and analyze them to detect implementation errors. The web service itself is treated as a black box. The testing tools do not interact with the web service, nor do they have any view of the supporting code or infrastructure. II Test framework of WS-I Test Tools The WS-I test tools consists of 2 test components and a test assertion specification as listed below: - Monitor: A tool used to intercept and log interactions with a web service. This tool generates a log that is later processed by the analyzer to verify that the monitored interactions conform to a profile. - Analyzer: A tool used to process the logs generated by the Monitor to verify that the intercepted web service interactions conform to the given profile. - Test assertion: Test assertions are used by the Analyzer to verify that the observed behavior of a given web service conforms to the Profile. 8 WS-I Web Site, Web Service Interoperability Organization, 9 WS-I Web Site, Web Service Testing Tools, 13

21 Figure II-5 is a high-level diagram of the system where collaboration between functionality in the 2 test components is visualized. SUT (Provider) Capturing SUT (Requester) Message log Monitor Test results Test Assertion Repository Analyzer Figure II-5 High-level system diagram of WS-I Test Tools Two web service solutions (requester and provider) that need to be tested are in this system referred to as the system under test (SUT). The requester SUT sends to the provider SUT business message, which is captured by the Monitor. Then, this message is validated by the Analyzer against a given test assertion which is a set of verification rules for the target Web Services profile. Next, the repose message from provider is monitored and validated in the same procedure. II Weaknesses of WS-I Test Tools for the B2B Standard testing Though WS-I provides useful tools, the major weakness is associated with its difficulty to adapt as a test framework for automated B2B testing. The details of this and other weaknesses are summarized below (Smythe 2006): - No component driving the test: WS-I test tools consist of the Monitor and the Analyzer. In order to test a specific situation, testing users should directly operate their SUTs. - Limited test coverage: This weakness stems from the first one. Since the tool has no control over the SUT, it means that some negative conditions (i.e. error detection) cannot be tested. - Low testing efficiency: This weakness is similar to that of previous test frameworks. That is the test report is limited to the passed or failed feedback without further information. II.4.4 Testing Test Control Notation (TTCN-3) 14

22 II Test framework of TTCN-3 TTCN-3 is a language to define test procedures to be used for black-box testing of distributed systems (Grabowski et al. 2000). When stimuli are given to the system under test (SUT), its reactions are observed and compared with the expected ones. On the basis of this comparison, the subsequent test behavior is determined or the test verdict is assigned. If expected and observed responses differ, then a fault has been discovered which is indicated by a test verdict fail. A successful test is indicated by a test verdict pass. TTCN-3 allows an easy and efficient description of complex distributed test behaviors in terms of sequences, alternatives, loops and parallel stimuli and responses. Stimuli and responses are exchanged at the interfaces of the system under test, which are defined as a collection of ports. The test system can use a number of test components to perform test procedures in parallel. Likewise to the interfaces of the system under test, the interfaces of the test components are described as ports (Baker et al. 2001) (Schieferdecker and Grabowski 2002). TTCN-3 has a similar look and feel to a typical programming language. However, in addition to the typical programming constructs, it contains all the important features necessary to specify test procedures and campaigns for functional, conformance, interoperability, load and scalability tests like test verdicts, matching mechanisms to compare the reactions of the SUT with the expected range of values, timer handling, distributed test components, ability to specify encoding information, synchronous and asynchronous communication, and monitoring (Schieferdecker and Vassiliou-Gioles 2003). A TTCN-3 test specification consists of four main parts: - Type definitions for test data structures - Templates definitions for concrete test data - Function and test case definitions for test behavior - Control definitions for the test case execution The data type definitions are generated from the corresponding XML schema of the Web Services to be tested. The templates are based on the corresponding data types and the behaviors of the service being tested that consist of sequences of requests and responses. Figure II-6 is a test procedure of the TTCN-3 test frameworks. 15

23 Standard Specification 1) Generation of testdata structure Abstract Test Case 2) Generation of testdata 3) Generation of Test behavior 5) Adaptor according to the mapping rules 4) Compilation to executable tests SUT A D A P T O R Test System Test Compon Test ents Components Test Components Test Components Figure II-6 Test procedure of the TTCN-3 test framework An approach towards automated testing of system with TTCN-3 requires therefore the following steps. 1) The structure of the test data is derived from the XML definition. 2) Test data (i.e. the concrete values for test stimuli and observations) is created. 3) Test behavior (i.e. the sequences of test stimuli and observations) is created. 4) The resulting TTCN-3 module is compiled to executable code. 5) The tests are performed using a test adaptor, which follows the mapping rules for test data structure to encode and decode the standard messaging requests and replies. II Weaknesses of the TTCN-3 test framework Though the scope of the TTCN-3 can be applied to all software testing, it has rarely used for B2B testbed implementation because this test framework has the below weaknesses: - Hard to make and understand the test case: TTCN-3 provides abstract test case representation in which test behaviors are displayed in table and tree representation. However, every TTCN test case should contain all procedural information for testing realization which includes test components behavior, interface, and deployment. Even B2B testing shares a common test behavior, domain experts should define all information in every test case. It makes the process for creating each test case cumbersome and difficult. Also this problem makes TTCN-3 less reusability. - Program language of test component: TTCN-3 test component looks like a pseudo code rather than component. Defined test component cannot work by itself without real implementation. To execute the testing, tester should develop compiler for TTCN-3. There is no official 16

24 complier for TTCN-3. - No considering B2B/B2C natures: TTCN-3 has developed for general software testing. It means TTCN does not consider particular natures of B2B/B2C system. Especially, message components such as transport (messaging tool) and message store are absence. II.4.5 Comparison Matrix among the Existing Test Frameworks According to the characteristics above, the existing test frameworks are compared as shown in Table II-2. Table II-2 Comparison matrix for conventional and proposed test frameworks Characteristics IIC 1.0 IIC 1.1 RNSDK WS-I TTCN Proposed ATF Test component scalability Partially Partially Y Y Y Y Reusability Test component seamlessness Partially Partially Y Y N Y Test suite modularity N Partially Y N N Y Generic executable test Y Y N Y Y Y Extensibility suite Test component reconfigurability N N Y N N Y Power of expression High High Low Low High High Efficiency Testing diverse Partially Partially N N Y Y Intelligence of Test report N N N N N Y The IIC test framework has a rather low reusability because multiple functionalities are aggregated into a few test components and the test harness is not flexible. RNSDK shows high reusability, but its test scope is very narrow having specific test suites. WS-I testing tools have relatively higher reusability, while low test efficiency. The generic TTCN-3 does not address the reusability and test efficiency. II.5. Chapter Summary 17

25 This Chapter reviewed and discussed relevant research works, that is, test frameworks for the B2B/B2C testing. IIC, RNSDK, WS-I, and TTCN were surveyed, compared, and analyzed. Most of them are hard to develop test cases because their test suites and test components should be redevelopment rather than reused. The test suite of existing test frameworks is hard to understand, because it depends on a specific standard and its contents are mixed and less modular in a test case. In addition, the testbeds developed are quite solid. That means internal test components (modules) are fixed into the entire system, so that it is hard to be adapted for diverse test needs. After this survey, we have come to a conclusion that an agile test framework is urgent. 18

26 III. TEST SUITE DESIGN FOR AGILE TESTING III.1. Chapter Overview The objective of this Chapter is to show the test suite design for the agile testing. The test suite is a collection of test cases that are intended to be used as an input to a testing tool to verify that system under test has some specified set of behaviors and requirements (i.e., the behaviors and requirements listed in its specification). In particular, we have developed the test suite design as double-layer test case. III.2. Design Factors of Agile Test Suite Figure III-1 shows a typical test scenario: 1) a domain (standard) expert publishes a standard specification which includes requirements to be tested, 2) a software vender implements a system according to the specification, 3) simultaneously, a test engineer develops test cases to verify a system under test for these requirements, 4) the test engineer registers the test cases into the testing tool, and 5) the testing tool (testbed) executes the tests. However this scenario implies serious problems as following: - Understanding gap: the software vender and the test engineer could differently understand contexts in the standard specification because it is not written in a formal language (context free or sensitized), but in a natural language. In the case that a system fails a test, the software vender and the test engineer could argue in a circle. It makes test execution consume much time and test analysis difficult. - Test dependency: an existing test suite has been designed for a specific testing tool. The testing tool is also implemented for a specific standard specification. It means that once a test case is developed, it cannot be reused for any other testing tools and any other standard specifications. It makes test case development consume much time. - Tightly coupled test information: a test case should include two major contents: test procedure and assertion. However the conventional test suite has been designed with tightly coupled contents. Even though these test cases are working well in a testing tool, it is difficult to understand them for a testing user (generally software vender). It makes both test generation and analysis consume much time. 19

27 Figure III-1 Typical test scenario from a standard to testing In the conventional test suite designs, the problems above hamper an agility of the entire testing. To resolve these problems, we will take into consideration following factors to design a new test suite: - Double-layer design: as discussed, the understanding gap occurs between a software vendor and a test engineering, both of which have not directly participated in standardization of the specification of interest. This gap may be minimized only when the domain expert who writes the specification intervenes them by means of, for example, providing test suites in a formal language. However, the expert is probably in ignorance of the test suite design and syntax. Therefore, we consider two-layered design of test suites: an abstract layer and an executable layer. The abstract layer is for human users, especially for the domain expert, to argument test suites for his/her own specification (and requirements). The format and syntax for the abstract test suite may vary from specifications or users. On the other hand, the executable layer, after automated transformation from the abstract test suites, will be fed into a testbed. The neutral executable test suite is tightly coupled with (and interpreted by) the testbed only, but in other words independent of the specifications. In a short, this separation resolves the test dependency problem by making the testbed read only one format of (executable) test suites; as well as allows the domain expert to provide his/her own (abstract) test suites, thereby minimizing the understanding gap between the software vendor and the test engineering. - Modular design: the test suite mainly consists of three separated parts configuration data, ２０

28 test procedure, and test assertions. The modularity relaxes the test suite to be loosely coupled, thereby allowing the test engineer to independently update each part of a test suite. For example, the test engineer may replace a new test assertion script only for verifying a different test requirement. This design factor will enhance the reusability of existing test suites, as well as easiness to create new test suites. III.3. Overview of Agile Test Suite Design In the B2B/B2C testing, a test suite is a collection of test cases. A test suite also contains detailed instructions for each collection of test cases and information on the system configuration to be used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests. The proposed agile test suite consists of an executable test suite and an abstract test suite. The executable test suite is a test suite that is ready to be executed by the testbed. This means that there exists a test harness that is integrated with the suite and such that the test suite and the test harness together can work on a sufficiently detailed level to correctly communicate with the system under test (SUT). On the other hand, the abstract test suite is more readable and understandable by human operators who compose or execute it, so that it has familiar structures and terminologies with domain experts. For example, users who want to test a messaging application such as ebms or SOAP applications do not consider the Payload contents in the messages to be exchanged between messaging applications, but have interests only to represent the sequence of these messages. In this case, the test suite should focus to represent an order of the messages and header information of each message. In addition, it is better to omit content parts of the messages like using a default payload. III.3.1 Double-layers of Test Suite Figure II-1 illustrates the agile test suites and their relationships. Once an organization publishes a standard specification, many applications are developed according to it. Hereby there are three kinds of operators standard expert, testing user, and test suite developer. The standard expert has knowledge about detailed regulations and constraints of a standard. He/she should establish the test requirement document that declares objectives and purposes of tests. The testing user has an application/document to be tested. In order to consider his/her specifics, the testing user should compose the user requirement document that indicates limitations and constraints of the application/document under test. The user requirement could sometimes contain specific test requirements that are not mentioned in the standard specification, but required to invoke him/her own applicaton/document. The test and user requirements are commonly called as test metadata for the test suite. 21