Project number: RI Project acronym: SCI-BUS. Project full title: SCIentific gateway Based User Support

Transcription

1 Project number: RI Project acronym: SCI-BUS Project full title: SCIentific gateway Based User Support Research Infrastructures INFRA e-science environments D5.1 Integration and testing strategy Due date of deliverable: 31/12/2011 Start date of project: 01/10/2011 Lead beneficiary: 4D Actual submission date: 31/12/2011 Duration: 36 months WP5 Dissemination Level: PU Version: 1.0 SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI

2 1 Table of Contents 1 Table of Contents Status and Change History Glossary Introduction Role of SA2 work package Operation of 4D ETICS portal Quality assurance and testing Working procedures Support Test types Deployment testing API testing Service testing Description Tools Web testing Description Tools Load and stress testing Description Tools Agile Testing practices Test Driven Development Acceptance Test Driven Development Behaviour Driven Development Specification by example Gherkin syntax WP5:SA2 2

3 7.4 ATDD, BDD frameworks Continuous Integration Environment Quality attributes to be tested Metrics for measuring code quality Metrics provided by static analysers Metrics for measuring maintainability and reliability Release criteria guse delivery and testing processes Release procedure Bug tracking in guse Build and Test configurations for guse Risk Management Time scheduling Summary WP5:SA2 3

4 2 Status and Change History Status: Name: Date: Signature: Draft: Eva Takacs 15/12/2011 n.n. electronically Reviewed: Kyriacos Neocleous 21./12./201 1 n.n. electronically Approved: Peter Kacsuk 21/12/2011 n.n. electronically Version Date Pages Author Modification /12/2011 all ET creation /12/2011 all ET adding new content /12/ /12/2011 Zoltán Farkas József Kuti adding section on release procedure adding section on guse testing /12/2011 all KN review /12/2011 all ET compiling review comments WP5:SA2 4

5 3 Glossary ATDD BDD CI PM SVN TDD WP Acceptance Test Driven Development Behaviour Driven Development Continuous Integration Project Month Subversion, version control system Test Driven Development Work Package WP5:SA2 5

6 4 Introduction This document outlines the integration and testing strategy of SCI-BUS project. The document describes the integration and testing processes, tools and environment to be applied inside this project. The outlined integration and testing strategy takes into account the research and development nature of the project and also the specific needs of academic and industrial partners. The integration and testing strategy document intends to be a living document that has to be agreed by all involved parties and continuously verified against reallife activities throughout the duration of the project. The deliverable is divided into the following chapters, including this introduction: Chapters 1,2,3 Document Status chapters Chapter 4 Introduction Chapter 5 outlines the activity of the SA2 work package Chapter 6 describes the test types to execute in SCI-BUS and the related tools Chapter 7 gives a brief overview of agile testing practices Chapter 8 is a description on quality attributes to be tested Chapter 9 is a description of test metrics Chapter 10 is a short overview of the release criteria Chapter 11 outlines guse quality assurance and delivery processes Chapter 12 is the section on Risk Management Chapter 13 is time scheduling of the SA2 activity Chapter 14 is the summary of work WP5:SA2 6

7 5 Role of SA2 work package The role of SA2 work package is to govern testing and release procedures to make recommendations of quality assurance methodologies, procedures and to operate and support the necessary infrastructure. The overall quality control of the generic portal services (JRA1) and the portals using the services provided by JRA1 (application gateways implemented further by JRA2 activity) will be governed by a well defined quality assurance strategy. The goal of the quality assurance and release strategy plan is to define a common framework to ease development teamwork, verification and testing of software packages, and the release of quality software. Processes and tools are only as good as they help solving problems and increase productivity. As such, in SCI-BUS a lightweight development and release process is intended to be used, taking into account the research and development nature of the project, the homogeneity of the portals to be implemented, and also the homogeneity of partners (academic and industrial). Agile software development practices are highly recommended, however not enforced. As such, there are two major tasks of the SA2 work package: Operation of 4D ETICS portal Build, test and packaging infrastructure for the SCI-BUS continuous integration environment. Technical communication channel between project partners. Quality assurance and testing Test strategy, advanced test frameworks. Implementation of black-box tests based on use case descriptions. 5.1 Operation of 4D ETICS portal Agile software development concepts, such as early builds, tests and integration are foreseen. To realise these concepts, advanced continuous integration processes and supporting tools will be put in place. So the first task of SA2 is to operate an instance of the 4D ETICS portal (etics3.4dsoft.hu, internal to the project) to serve as a unified environment for building, testing, software packaging for the SCI-BUS partners needs. This portal will serve as a technical communication channel between project partners and will allow the project manager to follow the progress. As such, 4D ETICS portal has multiple roles in SCI-BUS: It is a software management and project technical collaboration tool WP5:SA2 7

8 It is one way for project manager to follow the technical progress It is a continuous integration and testing environment It is a multi-platform building, testing and packaging infrastructure for the SCI-BUS It has a distributed test execution environment for deployment, API, system tests and others It provides tools and resources to configure, manage and analyse build and test runs It provides a common interface for diverse projects to facilitate knowledge sharing and operations management It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions But, 4D ETICS is not It s not a replacement for integrated development environments (IDE) such as Eclipse, Netbeans, MS Visual Studio and others It s not a replacement for SourceForge ( - the open source software repository It s not a replacement for source code management systems like CVS or Subversion. 4D ETICS uses such systems and can be easily extended with support for more It s not a replacement for build tools like make, ant, Maven etc. 4D ETICS uses whatever native tool a specific project decides to use and doesn t force the usage of any particular tool (see guidelines/126-build-software-tools) It s not a replacement for QA tools like findbugs, checkstyle, junit, cppunit, coverage tools, web test tools etc. 4D ETICS provides a rich library of such tools that projects can activate as they wish when running builds and tests 5.2 Quality assurance and testing An implementation of different types of system tests is planned for SCI-BUS. Specifically, deployment, API, service, and web tests will be produced. If possible, regression tests based on these tests will be put in place. guse services implemented by JRA1 will be tested firstly and extensively (see section 11). System tests for JRA2 portals will be based on the test specifications provided by each partner. These test specifications will be the subject of DSA5.2 deliverable due in PM6. The test execution environment is the 4D ETICS portal. To sum up, the main quality assurance and testing tasks are the following: WP5:SA2 8

9 Implementation of black-box tests based on use cases or user stories Recommendation of test methodologies, software metrics (see sections 6, 7, and Error! No se encuentra el origen de la referencia.) Adding test tools as plugins into the 4D ETICS portal 5.3 Working procedures Portals to be implemented in SCI-BUS intend to serve a large number of user communities, therefore high quality software packages should be released. To achieve this goal, a certain number of working procedures will be applied during the project lifecycle. Firstly, in the time between major releases (one in the end of the first year and one in the end of the second year) frequent internal releases are recommended to be available for testing. Some kind of agility is intended to be introduced in the development lifecycle: frequent implementation, integration and testing phases will iterate, but this is of course up to the individual partners to decide. Regarding the testing workload, basic unit tests will be implemented by individual developers. High level system tests are to be implemented by the SA2 following the test specifications provided by each portal development team (JRA1 and JRA2). SA2 will assist this activity by providing test templates. When preparing test specifications, JRA2 portal developers should concentrate especially on newly implemented features based on guse services. In addition to guse features, all the individual requests coming from the JRA2 community will be taken into account by SA2. Another important note is that only automated tests will be run. SA2 will not execute manual tests. The test execution environment is the 4D ETICS portal. Each partner should add its project into the test execution environment. 4D ETICS portal is operated and fully supported by SA2. This portal will serve as a continuous integration and test running environment for all development. The only exception is when using Microsoft Visual Studio as the development environment. In this case, the individual requirements will be considered. As for release procedure, when a new software package is released for SA2, each partner should provide smoke tests basic tests which demonstrate that the package is in testable state. Moreover, at least one (preferred three) positive tests executing a real program path or functionality should be provided. If these tests do not run, the packages will be returned to the developers for further improvement. SA2 will start testing only if these basic tests run. Additionally, each development team should provide the test adequacy criterion, which is defined as the basic set of functionalities that must work in a specific release. At this point, no specific common project-level software metrics and coding rules are imposed. However, it is recommended for each development team to apply the most relevant software quality metrics for their special purposes. SA2 makes this option possible in the 4D ETICS portal, by providing several static analysers, metrics calculation utilities and source WP5:SA2 9

10 code rule checkers. Additionally, it is highly recommended to run static analysers inside build and test configurations. Static analysers are a powerful and cheap way to find out a large number of bugs without test case implementation. These plugins are available in 4D ETICS Test System. Regarding the work schedule, the guse services implemented by JRA1 will be tested first. Ideally, the testing of the portal developments will start after PM6, after producing deliverable DSA5.2. Finally, all the working procedures will be agreed by all involved parties and continuously verified against real-life activities throughout the duration of the project. 5.4 Support The 4D ETICS portal is fully supported by the SA2 team. The support is basically based. In some urgent situations, Skype and telephone calls can also be provided upon request. Detailed information can be found in the tutorial in SVN: - Kick- Off/ /GettingStartedTutorial_gUSEin4DEtics.docx and in the SCI-BUS wiki: ETICS. Mainly the following activities might require support: Authentication (SA2 support team) Authorisation adding new subsystems, users Setting up configurations, adding version control, build, test, packaging commands etc. Running builds, remote builds Access to packages, reports Plugin implementation Adding build servers Guidelines (SA2 support team could suggest build and test guidelines) There are some requirements to use the 4D ETICS portal. First of all, signed certificate from a trusted CA is required for using 4D ETICS resources. 4D ETICS uses x.509 certificates to authenticate its users, no password or username needed to initiate sessions. This certificate should be added into the browser prior to using the portal. Moreover, it is necessary to have knowledge of the module and the dependency structure, as well as of external libraries to be used. These external libraries can be uploaded into the 4D ETICS repository and can then be accessed by other users. WP5:SA2 10

11 Additionally, the build scripts (ant, maven, make and others) that will generate packages are required. As for guidelines and constraints, the 4D ETICS System is quite flexible in terms of usage policies. However, there are recommendations and constraints that need attention. The most important are: Organize the software in well manageable units (components) and functional sets (subsystems). This is not mandatory, projects are free to organize the software as they wish. Version the software when changes occur to be able to reproduce past configurations and generate reliable trend analysis data. Not mandatory, but there is a lot to lose in not doing it. Lock the software when it s ready to be released, so it cannot be modified anymore, and give it a new name/version/date. This is not mandatory, however it is strongly recommended and related to the next point. Only locked configurations can be published in the one and only public permanent Repository ( Locking and Registration). This is a hard constraint. However, users can create as many volatile repositories as they wish during the development and test phases. WP5:SA2 11

12 6 Test types This is a summary section for test types, tools and/or mechanisms which are likely to apply in SCI-BUS. 6.1 Deployment testing Deployment testing is a type of production testing which is performed after the software is deployed to verify its behaviour in a production environment. After the deployment is ready, basic testing (so called "smoke" testing) is required to prove the basic functionality or basic navigation is working appropriately. Particularly, a check could be made for the installation location of the supported files, or what happens if some files are removed, or local settings are modified. In many cases, software must execute on a variety of platforms and under more than one operating system environment. Deployment testing exercises the software in each environment in which the system is expected to operate. In addition, deployment testing examines all the installation procedures and specialized installation software (e.g. "installers") that will be used by customers, and all documentation that will be used to introduce the software to end users. Usually, in case of distributed systems, here are two kinds of deployments: Local test runs. In this case, the deployment items and test binaries are copied into the local deployment folder and run there. Remote test runs. In this case, the deployment is done in one or more remote computers. In the majority of cases, deployment tests require test script implementations. No out-of-the box solutions are available. Multi-node tests are distributed tests involving several interacting services deployed in several nodes (machines). The 4D ETICS system provides a CLI mechanism for multi-node tests and multi-node deployments tests. This mechanism provides the ability to define complex test scenarios as a deployment and testing model composed of nodes, containing the services and tests. - Additionally, the 4D ETICS system has integrated a Workflow Designer component extending the existing CLI-based distributed testing mechanism with advanced functionalities. The Workflow Designer supplies a high level front-end interface that is able to orchestrate distributed deployment and testing scenarios for applications. Multi-node tests could be designed and run through the 4D ETICS portal. To sum up, the objectives of the deployment testing are the following: To ensure that the software can be automatically deployed (including internal and external dependencies) To verify that once deployed, it starts and it is activated correctly WP5:SA2 12

13 To examine all the "installers" that will be used by customers on all the required platforms 6.2 API testing An API (Application Programming Interface) is a collection of software functions and procedures, called API calls, that can be executed by other software applications. The system possessing API could be system software, application software or libraries. API testing means that we use the system API to verify the system, effectively bypassing the use of its GUI. API testing is much faster and more stable than GUI testing. In this case we only test the business logic, which does not change so frequently. Often, regression tests are based on API tests. To be more efficient, API testing could be augmented with Acceptance Test Driven Development (ATDD) concepts, as well as with keyword driven and behaviour driven techniques. It is important to differentiate between API Testing and Unit Testing. API testing is not Unit testing. API Testing is black box testing with the aim of verifying the business logic. It normally covers the whole system. At the same time, unit tests are typically designed by the developers and there scope is limited to the basic functionality. So, the main characteristic of API testing is that GUI is not involved in API Testing at all. The steps of API testing are setting up the initial environment, invoking API methods with the required set of parameters, and finally analysing the results. Setting up the initial environment can be complex as the GUI cannot be used for this purpose. Some checks are required to be put in place in order to decide whether the system is ready for testing or not. Setting up the environment in case of API testing involves environment setup and application setup. One the one hand, database should be configured, application servers should be configured and started. On the other hand, operations such as instance creations before calling non static members of the class fall under application-specific setup. Bringing the application in a specific state also falls in this category. Initial conditions setup in API testing also involves creating the necessary conditions under which the API methods can be called. In certain cases, API methods can be called directly, while in other cases they can be called because of some event or in response to an exception. The result of API method call could be some data output, or some internal state change. As such, API tests produce different types of results WP5:SA2 13

14 modify (insert, update, delete) data bases (individual mechanisms required for assertions) trigger some events, interrupt processes (assertions on listeners) modify some resources, such as datasets, configuration files, registry (individual mechanism for assertions required) To sum up: API testing means that we are able to avoid using the GUI and use the system API to verify the system. API tests usually verify that part of the business logic which does not change frequently. API testing is much faster and more stable than GUI testing. Effective regression tests can be implemented. Acceptance Test Driven Development (ATDD) can be used input: user stories, SA2 output: test specification in Gherkin syntax (please see sections and for additional information on Gherkin syntax). 6.3 Service testing This section describes basic concepts and tools of service testing Description Service testing is functional and non-functional testing of REST and SOAP/WSDL-based web services and also different protocols. Particularly, the testing of services involves: invoking monitoring simulating/mocking functional security testing load testing compliance testing Basically, there are two approaches to web service testing: producer-server side and consumer-client side. The most basic scenario for a web service test is calling the endpoint and then verifying the response. But, usually, basic web service unit tests are not sufficient. Also, as part of web service testing, we have to verify object-xml mapping, endpoint WP5:SA2 14

15 resolution, error handling and configuration in general. Moreover, web services tend to be complex as web service requests or responses could be quite complicated XML structures. This fact makes the service endpoint difficult to test. Web service testing implies extensive work with huge XML structures. Sometimes we want to test the more complex scenarios working with several number of web services, configuration, interceptors, error handling, and so on. In that case, mocking techniques play an important role in service testing. Other aspects that come into play can be authentication, authorization, and encryption. Service testing relatively easy to automate. It could also be a powerful way to test as it is possible to implement robust tests because the service interfaces do not change changes frequently. Additionally, effective regression test suites could be implemented based on automated service tests Tools soapui - Functional testing tool for SOA and web service testing Homepage: Key features: supports multiple protocols such as SOAP, REST, HTTP, JMS, JDBC, SSL and so on enables mocking techniques enables creating Performance Tests runs Automated Functional Tests CLI for seamless integration into a continuous integration environment (4D ETICS) provides plugins for Maven, Eclipse, Netbeans, and IntelliJ soapuipro advanced reporting, coverage, support 6.4 Web testing This section outlines some of the web testing concepts and tools Description When testing web applications, several aspects should be taken into account. These are: TCP/IP communications, Internet connections, firewalls, applications that run in web pages (such as applets, javascript, plugins), applications that run on the server side (such as CGI scripts, database interfaces, loggers, dynamic page generators, ASP), WP5:SA2 15

16 interactions between HTML pages, interactions with internal or external web services, different browsers, particularities of mobile applications, security and performance issues. In case of web applications usually we apply functional testing interface testing browser compatibility testing mobile browser compatibility testing security testing load and stress testing Regarding the functional testing of web applications, the validity of the links in web pages, and on the input forms used for getting information from the user should be checked. Furthermore, data consistency is very important in web applications. Data integrity and errors during edit, delete, modify, insert and any DB-related functionality should be verified. Cookie testing is also an important consideration. Cookies - small files stored on user machine - are primarily used to maintain login sessions. The behaviour of the web application should be checked by enabling or disabling the cookies in browser options. Concerning interface testing in the case of web applications, mainly the communication between web server, application server, database server, external web services should be tested. One of the most important issues when testing web applications is browser compatibility. Web applications could be very dependent on browsers. Nowadays mobile browsing needs special attention - as GUI elements behave differently in smart phone environments. As more and more transactions take place on the web, appropriate security testing of web applications is necessary. Security testing is the process that determines whether confidential data is sufficiently protected (i.e. it is not exposed to unauthorized individuals or other entities) and users can perform only those tasks that they are authorized to perform. For example, a user should not be able to perform administrative tasks such as restricting other users from using the web site. A user should not be able to change the functionality of the web application in an unintended way etc.). Basically, security testing assures that individuals can only perform operations that they are authorised to perform, and that the confidential data is protected. Load and stress testing is discussed in section Tools Selenium web testing toolkit Homepage: WP5:SA2 16

17 Key features: runs on the majority of available browsers and operating systems can be scripted by many programming languages and testing frameworks. record-and-playback of interactions with the browser distributed environment test library for acceptance test (Robot Framework) is available 4D ETICS plugin for multi-node testing available 6.5 Load and stress testing This section outlines the load and stress testing concepts and tools Description In the testing literature, the term "load testing" is usually defined as the process of exercising the system under test by feeding it the largest tasks it can operate with. The maximum load can be the expected (peak) concurrent number of users on the application performing a specific number of transactions within a predefined duration. In the case of web applications, the load is defined in terms of concurrent users or HTTP connections. The load test is conducted to understand how the system behaves under specific circumstances. As a result of the load test the response times of all the important business critical transactions will be available. Moreover, load testing operates at a predefined load level, usually the highest load that the system can accept while still functioning as required. Additionally, load testing does not aim to break the system by overwhelming it, but to specify as exactly as possible the boundaries of the system and the circumstances under which it operates properly. This is done by running regression tests against the application at a specified maximum load. An another aspect of the load testing is to expose the defects in application related to buffer overflow, memory leaks and further memory management related problems. Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it. Under stress testing, various activities to overload the existing resources with excess jobs are carried out in an attempt to break the system down. For example, to double the number of active connections, to shut down the application server or network connection, take the database offline, are actions typically carried out when stress testing an application. The main purpose behind stress testing is to make sure that the system fails and recovers gracefully - this quality is known as recoverability. In that sense, stress testing is used to determine the application's robustness in terms of extreme load and assist system administrators to determine the necessary resources for the production environment. A common procedure employed after conducting stress testing is to analyse post-crash reports to determine the behaviour of the application after a failure, in order to make the necessary adjustments. The most notable challenge is to ensure that the WP5:SA2 17

18 system does well with the protection of sensitive data after the failure. In a successful stress testing, the system will come back to normal operation along with all its components and subsystems, and no sensitive data will have been exposed. Goals of load testing: expose bugs that do not surface in functional testing (memory management bugs, memory leaks, buffer overflows, etc). ensure that the application meets the expected load (for a Web application, this is the number of concurrent users or HTTP connections). Goals of stress testing: to break the system under test by overwhelming its resources or by taking resources away from it - to make sure that even if the system fails it recovers gracefully Tools Grinder - Java Load Testing framework for running tests on many load injector machines Homepage: ( Key features: Load testing of applications, services, protocols having Java API (web servers, SOAP and REST web services, and application servers) Distributed Framework Advanced HTTP support, proxy aware, SSL, cookies, and so on Record and replay of interactions between user and the web site Scripts in Jython Command line tools for seamless integration to CI environment of 4D ETICS JMeter - Load testing tool Homepage: Key Features: load testing tool for analysing and measuring the performance of a variety of services, web applications. Sort of unit test tool for JDBC database connections, FTP, LDAP, web services, JMS, HTTP and generic TCP connections. supports a variety of assertions, reports WP5:SA2 18

19 CLI for seamless integration with CI environment of 4D ETICS WP5:SA2 19

20 7 Agile Testing practices Agile has its roots in iterative development and emphasizes collaborative teams, frequent deliveries and ability to adapt to the changing business needs. 7.1 Test Driven Development The basic agile method is Test Driven Development - TDD. This is rather a development than a testing method. Also known as test-driven design, TDD works like this: for each small bit of functionality the programmers first write unit tests. Then they write the code that makes those unit tests pass. This forces the programmer to think about many aspects of the feature before coding it. Defining the tests with the requirements, rather than after, and using those tests to drive the development effort, focuses more on the business goal. In traditional environments, tests are derived from project artefacts such as requirements documents. The requirements and design come first, and the tests follow. Executing those tests happens (if at all) at the end of the project. This is a test-last approach. Therefore the key difference is that tests are created prior to the code in Test Driven Development approach. In that context in agile practice, testing is not really a phase of software development; rather it is integrated in it. Continuous testing (testing early and often) is an efficient practice to ensure continuous progress. Automated regression testing (unit and acceptance) is a technique to ensure continuous testing. Automated unit tests check the behaviour of individual functions/methods and object interactions. In the same time, automated acceptance tests usually check the behaviour of the whole system (Sometimes without the GUI, checking the underlying business logic.) 7.2 Acceptance Test Driven Development ATDD is similar to TDD, but contrary to TDD, it considers the functionality of the whole system from a business perspective (from the customer s point of view). There are three levels of ATDD: end-to-end API subsystem API testing means that we avoid the GUI and use the system API to verify the system. API testing is much faster and stable compared to GUI testing. In this case, we only test the business logic, which does not change frequently. We could apply mock objects when not possible to use real objects. WP5:SA2 20

21 Subsystem testing is necessary when the other two types of testing are not possible. In agile, where every iteration should result in some new achievement, some parts of the business logic are available. Mocking techniques also could apply for missing components. End-to-end acceptance testing means that the whole system is tested, i.e. the test involves the GUI. Testing the GUI is the most difficult part of testing because GUI testing is the slowest and the least robust. Despite any potential problems GUI testing should be automated. The required ratio of end-to-end testing considering all the test types to apply is about 20% or less. However, any deviance is possible, depending on the actual requirements. 7.3 Behaviour Driven Development TDD can also extend beyond the unit or developer facing test as we seen in section 7.1. Agile teams use customer facing or story tests to help drive implementation. These tests and examples (sometimes called specification by examples), written in a form understandable to both business and technical teams, illustrate requirements and business rules. Customer facing tests might include functional, system, end-to-end, performance, security, and acceptance tests. Programmers write code to make these tests pass, which shows the product owners and stakeholders that the delivered code meets their expectations. In that way, behaviour driven development - BDD is an extension of TDD (and/or ATDD) where the test classes and methods are natural language sentences enabling the stakeholders to understand the tests Specification by example While TDD is very useful, the key factor of agile development (and not just testing) is specification by example. In agile development features or user stories substitute elaborated specification, which is avoided because it is hard to change and usually it ends up being out of sync with the code, due to continuously changing end-user or other requirements. An abstract specification is difficult to understand and change. A much better method is to describe specification by examples. And if in addition, these examples are test cases that can be executed, then code, specification and tests are matching. Whenever the code is changing the tests can be executed to justify the modified code does not conflict with the specification. According to agile methodology, specification by example is determined by a team involving the project owners (possibly including a selection end users), the tester and the developer. It is very useful for the project owner to participate in this meeting, because the process of deducting examples (test cases) from the user stories may reveal elements that were not considered. At these meetings the project owners can answer promptly to the non-answered questions. The examples - test cases in SCI-BUS project might be natural English sentences. These sentences should contain the necessary input and output values. Using of Gherkin syntax, WP5:SA2 21

22 that is accepted by most of the BDD tools, is recommended to be used. An example applying Gherkin syntax is below. Feature Login User story: login somebody who already has an account. Scenario happy path Given a user with login name "Smiths" and a password "Shtims2011". When user is logging in as "Smiths" and enter password "Shtims2011", Then the message "Signed in" appears, And the user is logged in. Scenario wrong password Given a user with login name "Smiths" and a password "Smiths2011". When user is logging in as "Smiths" and enter password "Smiths2010", Then the message "Wrong login name or password" appears, And the user is not logged in. Scenario double login Given a user "Smiths logged in. When I'm logging in as "Smiths" Then an error message "Smiths has already been logged in" appears, And user "Smiths" remained logged in. The advantage is that the text file containing this syntax is both a test specification and also executable by the selected BDD tool. When the examples are ready, the tool generates some test template (called fixture code). The actual test code will be inserted in that template test file (fixture code). When this fixture code is ready, it can be executed by applying the selected BDD framework. The framework reads the specification by example, to obtain the input and output data and the name of the test functions to be executed. The fixture code WP5:SA2 22

23 sets the given state, calls the code under test and verifies the result contained by the specification Gherkin syntax The core elements of the Gherkin syntax are as follows: Features Every *.feature file conventionally consists of a single feature. A line starting with keyword Feature: followed by free indented text starts a feature. Each feature usually contains a list of scenarios. You can write whatever you want up until the first scenario, which starts with the word Scenario: on a new line. (see the example above) Scenarios The scenarios can be used to describe the feature, i.e. the specification. One scenario is an example of the feature, and also one test case that can be executed automatically. One scenario starts with the word Scenario:, has a title and one or more scenario steps. Steps Steps must start with one of the keywords Given, When, Then, But or And. Conventionally each step corresponds to a TDD/ATDD test, which has to be written by the team (ATDD) or the developers (TDD). These steps describe the preconditions, the actions and the verification steps of the acceptance tests. These steps are introduced with the Given, When and Then, respectively, but subsequent steps of the same kind can be also specified with the And keyword. There may be other alternative keywords for specifying the steps, like But. (see the example above) Scenario Outlines: In order to have data-driven tests (to run the same test with different parameters) scenario outlines can be defined. Detailed description on Gherkin syntax can be found at: ATDD, BDD frameworks As it can be seen in the above mentioned Gherkin syntax example, instead of writing verbose, step-by-step manual test scripts and comprehensive test documentation, lightweight documentation styles/tools are applied. Specifications by example are captured in a format supported by automated test frameworks like: WP5:SA2 23

24 FIT/Fitnesse ( Robot Framework ( Concordion ( SpecFlow ( The test could be executed manually, but more importantly that same test artefact becomes an automated test when the programmers fill the generated fixture code with the test code. As an example, Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It applies the keyword-driven testing approach. Practically, it means that a specific testing function is mapped to a keyword. Then these keywords could be used to design extensive ATDD or BDD style acceptance test specifications. Robot Framework testing capabilities can be extended by test libraries. For example, Selenium ( a powerful web testing toolkit could be added as testing library and extensive acceptance tests can be designed with it. An other example is Sikuli (a GUI testing tools based on image recognition - which was added by 4D Soft as a test library to that automation framework. The architecture of the system can be seen in Figure 1.. Figure 1.: Robot Framework having integrated Sikuli as a test library 7.5 Continuous Integration Environment Lastly, TDD, BDD or ATDD style tests are always run in a continuous integration environment. According to Martin Fowler, "Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly." WP5:SA2 24

25 In SCI-BUS, the 4D ETICS portal provides a continuous integration environment. See section 5.1 for more detail. WP5:SA2 25

26 8 Quality attributes to be tested The quality model presented in the first part of the standard, ISO/IEC 25010, classifies software quality in a structured set of characteristics and sub-characteristics as follows: Functionality - The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. [ISO 25010] Functional testing is highly recommended for guse and JRA2 portals. As there are several types of functional tests, in SCI-BUS high level system tests or acceptance tests are foreseen. These functional tests could be prepared in an ATDD manner or in traditional style (following a sort of waterfall model e.g. start to test when the implementation is ready). Though the name is acceptance test, actually this is test-first method where functionality is tested in a "specification by example" fashion. During functional testing we ensure that code is verified by executing the specification that is implemented as test cases, where each test case is a scenario. Functional security testing is also a type of functional testing, where it is tested whether a user with appropriate rights is able to use all the features for which he or she is authorized. On the one hand, we should test that nobody can use a feature without appropriate rights. The test is restricted to execute and use the software and to give any input attempting to access data without authorization. Risk driven testing - on the other hand - investigates illegal usage and modification of the available code. This is non-functional testing and requires special knowledge. Reliability - The ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations. [ISO 25010] This is an important attribute for the SCI-BUS project as we anticipate a large number of different communities. Reliability is usually measured by metrics such as the mean time between failure - MTBF. Usually, the MTBF metric could be calculated by processing data from bug tracking systems. This can be done applying an appropriate continuous integration tool such as 4D ETICS. SA2 could make the MTBF metric available through the Dashboard of 4D ETICS portal. Usability - The capability of the software to be understood, learned, used by and be attractive to the user when used under specified conditions. [ISO 25010] At this current phase, SCI-BUS project ignores planning usability test as automated tests are foreseen. If necessary, SA2 will offer usability testing at any later iteration of the project. However, we note that in some type of software, usability issues are as important as the features of the tool, and users will start using the software only if it is usable for them. WP5:SA2 26

27 Efficiency - The capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions. [ISO 25010] Efficiency testing in the SCI-BUS project will be done by load and stress testing made by SA2 in a later iteration. The detailed issues will be elaborated later in the project. Maintainability - The ease with which a software product can be modified to correct defects, modified to meet new requirements, modified to make future maintenance easier, or adapted to a changed environment. [ISO 25010] Maintainability could be tested - by metrics getting data from bug tracking systems. These metrics will be based on the number of newly introduced defects in a specific part (module) of the code. If modifications of a given module induce new defects in more cases, then this module is not maintainable enough and it is recommended to apply refactoring. Additionally, maintainability index is an other approach to follow code s maintainability. SA2 could provide these metrics through 4D ETICS s Dashboard. Portability - The ease with which the software product can be transferred from one hardware or software environment to another. [ISO 25010] In the case of SCI-BUS, portability testing means testing deployments in multiplatform environments, with different kind of grid middleware, and in the cloud. SA2 will implement the necessary deployment scripts and 4D ETICS will be the running environment. WP5:SA2 27

28 9 Metrics for measuring code quality This section outlines metrics recommended to be used during development. 4D ETICS Test System provides a large number of metrics for measuring software quality attributes. Some of them are: Lines of Code Cyclomatic Complexity Coverage Metrics (Line, Branch) Object Oriented Metrics A whole list of metrics together with definitions is maintained in 4D ETICS tutorial stored in the SCI-BUS official site, see - Kick- Off/ /GettingStartedTutorial_gUSEin4DEtics.docx) and the SCI-BUS wiki: ETICS. 9.1 Metrics provided by static analysers Static analysers are software engineering tools which analyse software code looking for possible bugs. So static analysis is performed without actually executing programs and this makes it very efficient. These types of tools produce a lot of possible bugs without writing test cases. 4D ETICS Test System has integrated as plugin most of the available open source static analysers. According to our experience, FindBugs proved to be the most usable. We strongly recommend using it inside builds. FindBugs ( is a static analyser to find bugs in Java programs. It looks for instances of "bug patterns code instances that are likely to be errors. (See Figure 2.). False positives are possible. WP5:SA2 28

29 Figure 2: FindBugs report 9.2 Metrics for measuring maintainability and reliability As mentioned in Section 8 metrics can be used to measure maintainability and reliability. These metrics are important for improving the testing process. Some of these metrics could be provided easily by getting data from bug tracking systems, and they could serve to give strong feedback on the quality of the code. Some metrics to measure maintainability and reliability are: Reliability growth model: A model that shows the growth in reliability over time during continuous testing of a component or system as a result of the removal of defects that result in reliability failures. Mean Time Between Failures: The arithmetic mean (average) time between failures of a system. The MTBF is typically part of a reliability growth model that assumes the failed system is immediately repaired, as a part of a defect fixing process. Mean Time To Recovery: The arithmetic mean (average) time a system will take to recover from any failure. MTTR typically includes testing to ensure that the defect has been resolved. Number of defects in 1000 lines of code (LOC) when the code is first released. This can be determined when the code is considered to be stable, i.e. no new defects are revealed. An acceptable value is below 2. We van compute this measure by: NumOfBugsRevealedAfterRelease * 1000 / total LOC Number of new bugs introduced in a given module. This is a good measure to test maintainability. Number of closed bugs in a given period. This is useful to measure how much bugs are fixed and validated in a period. WP5:SA2 29

30 Number of open bugs during the project. As the project evolves this number should decrease. This measure should be divided for critical, major, normal or minor defects. At the end of the project should be zero for critical and major defects. WP5:SA2 30

31 10 Release criteria Release criteria or test data adequacy criteria are some rules to decide when the testing should stop and the software product should be released. Test data adequacy criteria could be based on different approaches. The first one could be whether the specification has been tested or all the tests in a test plan have been executed. By applying TDD and ATDD, this should always be adequate, since we start from the specification (user stories). An other adequacy criterion is based on coverage that should be measured. Based on our own and others experiences, an acceptable branch coverage level is 80% for each method. This means that at least 80% of the program braches should be covered. 4D ETICS portal has built-in plugins for measuring coverage metrics for Java. In the case of SCI-BUS, taking into account the distributed nature of the portals and the underlying grid and cloud environments other aspects such as portability in certain environments, compatibility with different kind of browsers, and reaching the specified performance indicators, could serve as release criteria. JRA1 guse developers and also JRA2 portal developers will specify individually their appropriate test adequacy criteria. SA2 will consider them when implementing and running test cases. WP5:SA2 31

32 11 guse delivery and testing processes 11.1 Release procedure Currently guse is being developed in MTA SZTAKI's internal SVN server. The SVN server is set up following the 'tags', 'trunk', 'branches' hierarchy, where developers are using the: 'trunk' directory to store current developments, 'tags' directory to tag different releases, 'branches' directory to store developments related to different projects or releases. A release candidate of guse goes through continuous testing, thus development and testing phases overlap. Once developers think the current 'trunk' version is ready for testing, they deploy such a version of the portal, and hand it over to the testers. From this point on they carry on development, and fix bugs reported by the testers. The portal installation devoted for testing is updated occasionally so testers can check if the reported bugs are really fixed. Once testers report a release as ready (i.e. there are no annoying bugs ), developers tag the 'trunk' version, create an installation package of it (using one of their machines), and upload the files to the SourceForge SVN. Finally, the official portal installation operated by MTA SZTAKI is updated to the released version. It can be seen, that the above procedure requires a lot of human intervention, thus is really error-prone. Within the SCI-BUS project we aim to make use of the 4D ETICS testing and build system to automate the above process as much as possible. 4D ETICS offers a way to auto-build software packages based on an SVN repository. As of the next release of guse, MTA SZTAKI will move the active development from its internal SVN into SourceForge's SVN. Thus, 4D ETICS can be used to build guse packages based on the SourceForge repository. During the development phase the trunk version will be built occasionally by 4D ETICS, so developers will get detailed report about the status of the code. This method will help to eliminate for example the problem of missed commits (the developer has the file, it is used to build the package, but the final release won't have it, as it is not committed). Additionally, 4D ETICS will be used to perform some automated tests based on the packages built. This will ease the life of the testers as they will not have to run a number of basic tests. Once testers think a given 4D ETICS-based installation is ready to be released, developers will tag the trunk version, and 4D ETICS will be used to create an installation package of the tagged version. WP5:SA2 32

33 11.2 Bug tracking in guse guse makes use of the bug tracking and feature request handling features of SourceForge. The following tools can be used by externals to report bugs, ask for features and ask questions about any topic related to guse: Bug reporting: Feature requests: User forum: Build and Test configurations for guse Up to now for guse build, test and release purposes, the following ETICS configurations have been created. guse_r_3_3_2 for the release of the guse (building and packaging) This configuration was created for the kick-off meeting. guse_r_3_3_3 for the release of the guse (building and packaging of guse version see 2Figure 3.) 2Figure 3: guse release configuration guse-deploy auto-deployment configuration, containing auto deploy scripts WP5:SA2 33

34 This configuration is for auto deployment of guse services (see Figure 4.). Figure 4: guse configuration for auto deployment The automatically created (with 4D ETICS) guse deployment can be submitted on the local computer. The result is a deployed instance of the actual guse on the local computer. Currently no grid is connected. The deployment steps are as follows: 1. Building the guse packages from the latest source 2. Downloading Liferay-portal (bundled with tomcat) 3. Initializing the MySQL database with the default values MySQL is needed on the build server 4. Initializing the WAR files 5. Setting environment variables 6. Initializing tomcat and Liferay 7. Starting tomcat 8. Lastly initializing guse services, after tomcat has successfully started The result of the auto deployment can be seen in Figure 5. WP5:SA2 34

35 Figure 5: guse deployed on local machine guse-dev development version of guse These configurations have been created for the continuous integration purposes of daily development work (see Figure 6.). WP5:SA2 35