MODACLOUDS integration plan - Initial version. Francesco D Andria (ATOS), Elisabetta di Nitto (Polimi) Danilo Ardagna (Polimi), Alexander Gunka (BOC)

Transcription

1 Grant Agreement N FP Title: Authors: Editor: Reviewers: Identifier: Nature: MODACLOUDS integration plan - Initial version Francesco D Andria (ATOS), Elisabetta di Nitto (Polimi) Francesco D Andria (ATOS Spain S.A) Danilo Ardagna (Polimi), Alexander Gunka (BOC) Deliverable # D3.3.1 Report Version: 1 Date: 4/06/2013 Status: Diss. level: Final Public Executive Summary This deliverable D3.3.1 presents the first integration plan for the MODAClouds platform. It provides the strategy and the main steps that will be carried out by the consortium in order to integrate the MODAClouds software platform. It describes information about the common and collaborative development platform that will be provided by the consortium. The integrated MODAClouds software platform will be exploited as an open-source project. An update of this document will be provided at M16. It will adjust the current plan according to the available results. Copyright 2012 by the MODAClouds consortium All rights reserved. The research leading to these results has received funding from the European Community's Seventh Framework Programme [FP7/ ] under grant agreement n (MODAClouds).

2 Members of the MODAClouds consortium: Politecnico di Milano StiftelsenSintef InstitutulE-Austria Timisoara Imperial Collegeof Science, Technologyand Medicine SOFTEAM Siemens Program and System Engineering BOC Information Systems GMBH Flexiant Limited ATOS Spain S.A. CA Technologies Development Spain S.A. Italy Norway Romania United Kingdom France Romania Austria United Kingdom Spain Spain Published MODAClouds documents These documents are all available from the project website located at Public Final version 1.0, Dated June 4,

3 Contents 1. INTRODUCTION CONTEXT AND OBJECTIVES STRUCTURE OF THE DOCUMENT MODACLOUDS SPECIFICATION OVERVIEW MODACLOUDS LAYERS OR SYSTEMS INTERACTIONS ADOPTED DEVELOPMENT METHODOLOGY MODEL SELECTION MODACLOUDS SOFTWARE SOFTWARE PACKAGING MODACLOUDS SOFTWARE TESTING ACTIVITY Unit testing Integration testing activity System testing activity Site Tests activity INTEGRATION SCHEDULE DOW MILESTONES AND DEADLINES GANTT DIAGRAM FOR INTEGRATION DEVELOPMENT LIFECYCLE TOOLSET VERSION CONTROL SYSTEM Version Control System Selection Guidelines about how- to organize the MODAClouds of the code on a SVN repository BUILD AUTOMATION Build Automation Tool Selection CONTINUOUS INTEGRATION Continuous Integration Selection CONCLUSION REFERENCE ANNEX A: STATE OF THE ART ON VERSION CONTROL SYSTEMS CONCURRENT VERSIONING SYSTEM [7], [8] APACHE SUBVERSION [9] GIT [10] MERCURIAL [11] ANNEX B: STATE OF THE ART ON BUILD AUTOMATION TOOLS MAKE [13] ANT [12] MAVEN [14] GRADLE [15] ANNEX C: STATE OF THE ART ON CONTINUOUS INTEGRATION TOOLS CRUISE CONTROL [16] HUDSON / JENKINS [17], [18] TEAM CITY [19] Public Final version 1.0, Dated June 4,

4 1. Introduction 1.1. Context and objectives This deliverable D3.3.1 presents the first integration plan for the MODAClouds platform. It provides the strategy and the main steps that will be carried out by the consortium in order to integrate the MODAClouds software platform. It describes information about the common and collaborative development platform that will be provided by the consortium. The integrated MODAClouds software platform will be exploited as an open-source project. An update of this document will be provided at M16. It will adjust the current plan according to the available results. The MODAClouds integration strategy leverages on a holistic approach supported by best practices, tools and methodologies to ensure a smooth, timely, and good quality delivery of MODAClouds platform. In contrast to traditional enterprise application integration (EAI), MODAClouds leverages on Service-oriented integration (SOI). SOI defines integrating computing entities that use only service interactions in a serviceoriented architecture. Service-oriented integration addresses problems that allow integrating legacy as well as inflexible heterogeneous systems by enabling IT organizations to offer the functionality locked in existing applications as reusable services. The significant characteristics of services-oriented integration are: Well-defined, standardized interfaces Consumers are provided with easily-understood and consistent access to the underlying service. Opaqueness The technology and location of the application providing the functionality is hidden behind the service interface. In fact, there is no need for a fixed services provider. Flexibility Both the providers of services and consumers of services can change - the service description is the only constant. Provided both the provider and consumer continue to adhere to the service description, the applications will continue to work. To implement the above mentioned approach some tools will be used, like version control, bug-tracking, continuous integration along with collaboration tools like wikis, content management, integration groups, real-time communication, discussion forums as well as project management. Nevertheless, the software side of the MODAClouds project, as many other software projects, will be evolutionary, where the software is continuously updated with new releases that add incremental functionality (as it is stated in the DoW). However, it presents the challenge that it is necessary to keep a series of validation procedures to ensure from the first release a certain level of quality. From a functional point of view, the MODAClouds software will be evaluated in three different levels. Unit tests will be setup for each individual module of the architecture; these tests verify the internal isolation of each one of those modules. Integration tests will verify the interaction between two or more software components. Finally, the system test will verify the functionality of a specific release of the MODAClouds platform. Moreover, the integration activity also has to support the delivery of safe test beds infrastructure where the Case Studies will run. In such infrastructure the latest stable release of the MODAClouds software stack will be installed. Although in this infrastructure a collection of system tests will run on a daily basis to verify the health of the MODAClouds platform software, it is supposed to contain only stable and already tested and verified software Structure of the document This document, the MODAClouds D Integration plan (M8), is the first document of the D3.3.x and D3.4.x sagas. It highlights the MODAClouds integration strategy based on SOI. In details it defines best practices, tools and methodologies to ensure a smooth delivery of the MODAClouds platform, timely delivery and good quality, as well as the order in which the components and subsystems of the MODAClouds platform should be integrated, which builds to create when integrating the system, and how they are to be assessed. This document is organised as follow: Section 2 introduces the MODAClouds architecture from a high level point of view. Section 3 gives a brief overview of development methodologies that are commonly used and presented along with comparative analysis in order to justify the selection made by the consortium. Section 4 introduces the MODAClouds approach about the software packaging and the testing activities. Public Final version 1.0, Dated June 4,

5 Section 5 presents the integration steps according to the global work plan described in the description of work. Section 6 is dedicated in the analysis of three specific types of tools that have been used in order to organize and accelerate the development: a source code version control tool, a build automation tool and a continuous integration framework. The last section concludes the document. Public Final version 1.0, Dated June 4,

6 2. MODAClouds Specification This section provides information about the MODAClouds initial architecture, which is detailed in D3.2.1 MODACLOUDS architecture Initial version Overview Relevant monitoring data MODAClouds IDE Monitoring rules Self-adaptation constraints and strategies Deployment artifacts Runtime Platform Monitoring Platform Triggers & Data Self- Adaptation Platform Adaptation Actions Execution Platform Cloud App Cloud App Figure 1 MODAClouds high level architecture Figure 1 shows the four main components of the MODAClouds architecture. The MODAClouds IDE is in charge of supporting the design time activities performed by a cloud app developer, from the feasibility analysis to the code generation. The Execution Platform is in charge of offering all operations to deploy and control the execution of an application on a cloud. It offers an abstraction layer that allows the cloud app operator to abstract from the specificities of each cloud and to rely on homogeneous management operations that are then mapped into the cloud-specific ones. The Monitoring Platform is the one that collects monitoring information from the cloud app execution environments and filters and analyses them as needed. The Self-adaptation Platform is in charge of automating the execution of actions aiming at correcting a critical or potentially critical status or behaviour of the cloud app or at optimizing it based on the current knowledge of the execution environment. In order to identify the correction/optimization actions to be executed (e.g., create a new Virtual Machine, move the traffic from one app instance to another, ), the Self-adaptation Platform consumes and reasons on the information that are offered by the Monitoring Platform. Moreover, it relies on the Execution Platform for the actuation of the identified actions. The architecture has been designed keeping in mind the following aspects: As the development of the various parts of the MODAClouds architecture is supposed to proceed in parallel, the components should be loosely integrated. Public Final version 1.0, Dated June 4,

7 As we foresee three development/integration cycles, integration of functionalities can occur at different stages and as such each component and sub-component should be as independent as possible from the others. At design time MODAClouds aims at tackling many different aspects ranging from the analysis of the business opportunities and value of some cloud solutions over the others (risk and cost analysis) to the functional and non-functional design of cloud applications, to the analysis of different design alternatives, to the deployment of the final solution. Given the complexity of each of these problems when tackled in the new cloud-based context, we have chosen to address each problem separately from the others through the development of different tools that altogether constitute the MODAClouds IDE. We ensure integration and interoperability among these tools through the sharing of common meta-models. This integration approach offers also advantages from the exploitation point of view as it allows delivering, promoting and adopting each of the design-time tools belonging to the MODAClouds solution separately from the others. At runtime, the Execution Platform, the Monitoring Platform and the Self-adaptation Platform control the execution lifecycle of the cloud application and interact one with the other through point to point communication. The technology used for such communication is still under definition as it depends on the number of messages the various components will exchange in the reference time interval and on the synchronicity and delivery guarantees we expect to have. The interaction between the design-time and the runtime occurs through asynchronous exchange of relevant information from the design-time and the runtime. We also foresee the possibility for the runtime to provide data back to the design-time through what we call feedback loop. This aspect is not detailed in the current version of the architecture as it represents an advanced issue that we plan to tackle in the second half of the project MODAClouds layers or systems interactions Figure 2 shows a detailed view on the MODAClouds architecture. More specifically, the figure shows the internal structure of the MODAClouds IDE that is built out of the integration among the Risk & Cost Analysis Tool developed by WP2, the Functional Description Tool, the Data Mapping Component and the Deployer developed by WP4, and the Non Functional Modelling and Analysis Tool developed by WP5. These tools are integrated through a set of shared meta-models that allow them to have different viewpoints on the architectural description of a cloud-based application. These models incorporate functional and non-functional information about the application architecture. A subset of these models is fed into the runtime components for specific purposes. More specifically, Monitoring Rules are an input for the Monitoring platform. These define what to monitor in order to check if the QoS constraints expressed by the QoS Engineer are fulfilled. The Policies for Self-adaptation, the Initial Deployment Model and the QoS Model are input for the Selfadaptation platform. Finally, the Deployable Artefacts are the input of the execution platform. The figure also highlights the internals of the Self-adaptation Platform. It includes the models@runtime that are in charge of keeping alive, during execution, a model of the topological structure of the cloud application and of checking that any change to this structure is compatible with the constraints defined for the cloud application architecture. The second part of the Self-adaptation Platform is the Reasoner that, based on the information received from the Monitoring Platform and on the policies for self-adaptation that include a QoS model of the cloud app, is able to identify the best adaptation for the application. The figure also highlights the actors interacting with the MODAClouds solution. The extensive definition of the components, their interaction and of the shared metamodels is the subject of deliverable D As it is going to be explained in detail later on, knowing these interactions is important for a good design of integration and system tests. Public Final version 1.0, Dated June 4,

8 QoS modelling and analysis tool (WP5) Data Mapping Component (DMC) (WP4) MODAClouds IDE Shared Models MODACloudML Deployment and Provisioning Component(WP4) MODACloudML Functional Modelling Environment (WP4) Decision Making Toolkit (WP2) Desing Time Run Time Monitoring Rules Policies for selfadaptation Initial Deployment Model QoS Model Deployable Artefacts Runtime Platform Self Adaptation Reasoner Monitoring Platform Self-Adaptation Platform Current Model Target Model Execution Platform runtime Runtime GUI (WP4) Cloud App Cloud App Figure 2. Detailed view on the MODAClouds architecture Public Final version 1.0, Dated June 4,

9 3. Adopted Development Methodology Every software product or service requires respecting a specific software development process which imposes a specific execution paradigm for the various tasks or activities. Loosely speaking, four activities are defined: Requirement acquisition and analysis Design & Implementation Testing & documentation Deployment & maintenance The initial phase of any software development model begins with requirements definition and analysis. Customers typically have an abstract idea of what they want as an end result, but not what the software should do. Skilled and experienced software engineers recognize incomplete, ambiguous, or even contradictory requirements at this point. Requirements are typically placed into these categories: Architectural requirements describe what must be done by identifying the necessary system architecture of a system. Functional requirements describe the functionality that the system is to execute; for example, formatting some text or modulating a signal. They are sometimes known as capabilities. Non-functional requirements describe characteristics of the system that the user cannot affect or (immediately) perceive. Non-functional requirements are sometimes known as quality requirements. Constraint requirements impose limits upon the design alternatives or project/process operations. No matter how the problem is solved the constraint requirements must be adhered to. The requirements should be as complete as possible, ideally covering the whole set of desires with no missing information. On the other hand, feasibility should be respected so that requirements can be implemented within the constraints of the project. Consistency and non-ambiguity should also be involved in the set of requirement characteristics in order to eliminate potential contradictions. Finally, requirements must specify a level of importance and be verifiable through basic possible methods: inspection, demonstration, test (instrumented) or analysis (to include validated modelling & simulation). The first process took place during the first six (6) months of the MODAClouds project and preliminary concrete outputs has been documented as Deliverables D3.1.1 Requirements analysis strategy and D3.1.2 Requirements repository Initial Version. Once the general requirements are gathered from the client, an analysis of the scope of the development should be determined and clearly stated. Certain functionality may be out of scope of the project as a function of cost or as a result of unclear requirements at the start of development. If the development is done externally, this document can be considered a legal document so that ever there are disputes, any ambiguity regarding what was promised to the client can be clarified. Software design is the process of problem solving and planning for a software solution. After the purpose and specifications of software are determined, software developers will design or employ designers to develop a plan for a solution. It includes low-level component and algorithm implementation issues as well as the architectural view. Software design experience has resulted in the definition of a foundation in the form of a set of design concepts that provide the basis which more sophisticated methods can be devised. The first step in software design is software architecture, which is the set of structures needed to reason about the system, which comprise software elements, relations among them, and properties of both. The software architecture discipline is centred on the idea of reducing complexity through abstraction and separation of concerns. Software design can be conceptualised as a refinement of software architecture, decomposing macroscopic statements of functions in a stepwise fashion until programming language statements are reached. In each step, one or several instructions of a given program are decomposed into more detailed instructions. Software design also focuses in more system issues like performance, encompassing extensibility, fault-tolerance reliability, robustness, and security. A preliminary and from a very high level point of view an architecture is thoroughly described in D MODACLOUDS architecture Initial version. The deliverable that has been developed in parallel with the D3.3.1 The concretisation of software design is accomplished by the implementation phase, where the actual runnable computer program is realized. Ideally, the programming language should be completely decoupled from the software design outcome, but in many cases this rule is not followed, at least not to the full extent. Public Final version 1.0, Dated June 4,

10 Implementation still requires significant skills in order for the final software program or service to both conform to the design considerations and satisfy a set of quality requirements, such as reliability, portability, robustness and efficiency. Software testing is an essential and important phase of the software development process. This part of the process ensures that defects are recognized as soon as possible. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. It can be categorized based on its methodology, type, focal item and lifecycle. Regarding the methodology, we may have either white box testing, according to which the software components are tested in a non-transparent way including the internal data structures and algorithms, or the black box testing where components are only tested against their outcome based on defined input data sets. Testing comprise functional and non-functional types. The former include unit, system, integration, regression, acceptance, alpha and beta testing; the later consist of performance, usability, acceptance and security. All testing types focus on one or more software components. Finally, all tests follow a certain lifecycle which is heavily dependent on the development model, which will be thoroughly analysed below. Section 4 of this document details the MODAClouds strategy in terms software testing activity will be done during the life-cycle of the project. Documenting the internal design of software for the purpose of future maintenance and enhancement is done throughout development. This may also include the writing of an API, be it external or internal. The software engineering process chosen by the developing team will determine how much internal documentation (if any) is necessary. Documentation that in the MODAClouds context will be delivered as project deliverables, white papers and reports will go with the code can be categorized based on its focal point. So, there are requirements documentation, which presents the requirements and the corresponding analysis, technical documentation, which analyses the software architecture, design and implementation of the program or service, the end-user manuals that elaborate on the use of the platform and finally marketing documentation that analyses the potential market value and placement of the final product. All the aforementioned activities are organized based on a lifecycle which is dictated by the development methodology. Every software development methodology approach acts as a basis for applying specific frameworks to develop and maintain software. Figure 3: Development lifecycle in respect to the underlying activities Several software development approaches have been used since the origin of information technology. However, all of the proposed approaches fall into two main categories: Waterfall and Agile. Actually, there still have been several discussions regarding which approach to software development is better; however there isn t an absolute answer. Waterfall and Agile have a different set of features that make them depending on several internal or external influences related to the way to organize of the project as well as its resources: time to deliver, quantity of change in the specifications, typology of the requirements, skill of the resources involved in it etc. Public Final version 1.0, Dated June 4,

11 In literature there are a lot of works relative to such issue [1], [2], [3], [4], [5], [6] and many other. The following two tables just resume the advantages and disadvantages of each model: The following table summarizes the advantages and disadvantages of the Waterfall model. Detailed documentation. Pros Agreed and signed off requirements. Can be delivered using developers with a lower skill set. Reduced number of defects through thorough design planning. Defined start and end point for each phase, allowing progress to be easily measured. Slow start. Cons Fixed requirements difficult to change. No customer visibility of software until the development has been completed. Lack of flexibility making it difficult to change direction. Customers often unclear about their requirements initially. Table 1: Pros and Cons of the waterfall model The following table summarizes the advantages and disadvantages of the Agile model. Pros Quick start, incremental releases and regular customer reviews and feedback. Evolution of requirements over time. Ability to respond to change quickly. Less rework, achieved through continual testing and customer involvement. Real-time communication among the development team and customer. Cons Can be misinterpreted as unplanned or undisciplined. Needs a high-quality, customer facing development team. Needs a high-level of customer involvement. Lack of long-term detailed plans. Produces a lower level of documentation Model Selection Table 2: Pros and Cons of the Agile model According to [3] the method you use depends on several factors, some of which are: Type of work: is there a clearly defined outcome? Website development is creative and therefore lends itself to the agile method. Transactional system development, where there is a clearly defined outcome, is better suited to the Waterfall method. Type of customer: are they willing to work in an iterative way, do they have enough time to review and comment on regular iterations? View of IT within the organization: is IT seen as a valued partner or a necessary evil? If the latter then use the Waterfall method. Internal or external customer: is the customer internal or external? The Waterfall method works well with external customers where you can hold them to a signed contract, as opposed to internal customers who can force you into changes by gaining the backing of senior management. No matter if the project is considered an Agile or a Waterfall one, the main activities that are necessary to perform don t change. What changes are when they are performed, and how they are performed. In summary, there are six (6) main activities in software development: Public Final version 1.0, Dated June 4,

12 Table 3: Activities in respect to models (figure from [3]) In the context of MODAClouds, the following requirements needed to be satisfied: Best-as-possible final software product/service Fixed time schedule of production environment Significant requirement changes during the development Constant communication and exchange of ideas between end-users and developers Well defined documentation of every type (see above) Highly-capable but decentralized development team with constant remote communications but infrequent face-to-face meetings It is clear that neither approach fits perfectly on these requirements. The wrong thing to do in software development is to pick only Agile or Waterfall and completely discard the other one. Based on the aforementioned set of requirements it is incorrect to restrict ourselves into the selection of one of them and the only logical outcome is that a combination of the aforementioned models would produce the desired results in the optimal way, satisfying all aforementioned requirements. Therefore, in the context of MODAClouds project, the iterative waterfall model has been adopted, according to which a cyclic software development process is developed, beginning with an initial planning and ending with deployment with the cyclic interactions in between. Therefore the project will follow an incremental and iterative approach to the development of project results organized into three different phases. Section 5 of this document details the MODAClouds strategy in terms of the steps involved in the design, development and integration of the MODAClouds solution. Figure 4: Iterative model Public Final version 1.0, Dated June 4,

13 4. MODAClouds software Figure 2 (in the section 3) shows a comprehensive overview of the MODAClouds architecture. The MODAClouds software solution is composed of several distributed and heterogeneous building blocks (subsystems) coming from different partners belonging to the consortium and implemented using diverse programming language paradigms (even though most of the code will be implemented in Java). The approach the consortium agreed to follow to achieve an integrated software solution is to follow a model sharing approach for what concerns the design-time part and the SOA paradigm for what concerns the runtime. More specifically, the design-time MODAClouds IDE will be developed as a set of five tools that will be independent of each other but will agree on the format of the models they produce and use. Table 3.1 in Deliverable D3.2.1 offers a preliminary list of design models and their association as inputs and outputs to each tool belonging to the IDE. In the next months such a table will be revised and completed and will constitute the map for supporting the integration of the design-time part. Each of the three components of the runtime will offer a service-based interface through which it will be able to offer services to the other components. The execution and the monitoring platform will be the bottom-level components interacting directly with the target clouds and with the cloud apps. The self-adaptation platform will exploit the services offered by both of them to actuate actions on the execution platform based on the information received by the monitoring platform. Figure 2 (in the Section 3) as well as Section 3 of D3.2.1 MODAClouds architecture - Initial version outline how the design-time and run-time modules work together. Actually the interaction between the design-time and run-time is based on asynchronous exchange of relevant information as models conveniently stored in a common repository. In the following subsections of this chapter the most relevant MODAClouds integration topics are introduced Software Packaging In the context of the MODAClouds project, software packaging is a process that automatically determines how to integrate a diverse collection of computer programs based on the types of components involved and the capabilities of available translators and adapters in an environment. Software packaging provides a context that relates such mechanisms to software integration processes and reduces the cost of configuring applications whose components are distributed or implemented in different programming languages. The section 6 development lifecycle toolset provides an exhaustive overview of three specific types of tools that will be used in order to organize and accelerate the development: a source code version control tool, a build automation tool and a continuous integration framework MODAClouds software testing activity The main objective of software testing activity is to detect the failure of the software, so they may be corrected; taking into consideration that any software testing procedure can not verify that the software is fully functional under all possible conditions. Software testing can only verify that the software under test will function properly under specific conditions. The validation procedures of any software include the testing of the code under different environments, verifying that under certain conditions the software behaves as expected and required. This software validation can be divided in two different aspects: a Functional validation and a Non-Functional validation. The Functional validation aims to demonstrate if some specific functionality of the software works as expected. These functionalities should replicate the requirements or use case documentation. While the Functional Requirements specify a set of functions that a system or a system component must be able to perform, the Non-Functional validation evaluates the desired qualities of a problem solution other than those concerning its functionality, e.g.: its robustness, its efficiency, its security, its extensibility, etc. Public Final version 1.0, Dated June 4,

14 Unit testing The objective of the unit tests is to verify the functionality of each component in isolation, independently of other components or the final environment where the MODAClouds software is going to be deployed. Each one of the components represented in the figure 1 is independently developed by one of the partners of the project. It is the responsibility of the developers of the component to write unit tests for them. Typically, software testing tools offers the possibility to measure how much of the code is covered by unit tests. Those tools can help give to inspect how well a module is tested before starting the integration phase Integration testing activity Since the unit test verifies the functionality of the components in isolation, the integration tests verify that the interaction between components work as expected. Most components need to interact with others to offer their functionalities. A set of tests need to be written to make sure that this integration works as described, interacting with the other components of the same layer System testing activity A system test evaluates the behaviour of all the components of MODAClouds at the same time. It executes a scenario that will cover the majority of the functionality of MODAClouds. The scenario must also take into account the requirement analysis coming from Task Site Tests activity The site test evaluates the performance of all the components belonging to the MODAClouds system when they are installed in the test bed where the case studies will be installed. The work package 9 Case Study Implementation and Evaluation aims, together with the implementation of the case studies, at evaluating the test beds for the revision of the research activities of WPs 2-6. The evaluation performed in the beforehand mentioned work package will be also used as input to the task 3.5 belonging to the WP3 for analysing the results and drawing some general lessons learned. The work package 9 will deliver two series of deliverables will report about the evaluation of the MODAClouds test beds respectively 9.x.1 will be delivered on M21 and 9.x.2 delivered on M36. Public Final version 1.0, Dated June 4,

15 5. Integration Schedule The aim of this section is to present the steps involved in the integration of the MODAClouds layers (subsystems) coming from the technical work-packages (WPs 2, 4, 5 and 6) and to provide the corresponding platform to the Case Studies (WPs 7, 8, 9) for assessment. Each version or update of one of these subsystems may have interdependencies with specifics versions of other software components we should take into account. At the same time, while those software components are updated with new features, there is a stable environment already deployed and available for the case studies, that periodically is going to be updated. This makes necessary the design of an integration workflow, to deploy new versions of the MODAClouds software layer into the case studies infrastructure DoW milestones and deadlines According to the DOW the following milestones has been defined to support the MODAClouds development and integration approach. Milestone number / name MS2 / MODACloudML and Architecture Definition MS3 / First Integrated Solution Delivery MS5 / Final tools Delivery MS6 / Finalisation Work packages involved WP1-7, WP10-11 WP3-6, WP8, WP10 Expected date M12 M18 Validation criteria MODAClouds MODACloudML, architecture and integration plan are defined. Prototypes are developed in technical WPs. Case studies requirements are specified. Plans for standardization, collaboration, exploitation and dissemination are delivered. Final market analysis is performed First integrated solution is delivered. Case studies have defined first prototype design. Evaluation plans are defined. First reports on standardization, exploitation and dissemination are delivered. WP2-7 M30 Final releases of tools and methods are delivered. Final prototypes of case studies are delivered. MODAClouds integrated solution is updated. WP1, WP3, WP9-11 M36 Final evaluation plan, best practises and recommendations are delivered. Final reports are delivered. Table 4: List of milestones related to the MODAClouds development and integration 5.2. Gantt Diagram for Integration The following diagram (Figure 6) depicts the MODAClouds platform integration approach below detailed: The blue braces represent the integration activities belonging to the Task 3.3, while the three blue stars represents the MODAClouds integrated solution - Proof of concept (delivered due to M18); MODAClouds integrated solution - Initial version (delivered due to M30) and the MODAClouds integrated solution - Final version (delivered due to M36). The green braces represent the developments activities of the different tools belonging to the MODAClouds architecture while the three green triangles represents the code Public Final version 1.0, Dated June 4,

16 The red braces represent the developments activities of the four case studies will demonstrate the applicability of the MODAClouds solutions in four scenarios defined in different domains. The two red circles represent the prototypes of the four case studies. Figure 5: MODAClouds platform integration gantt Public Final version 1.0, Dated June 4,

17 6. Development Lifecycle Toolset As already mentioned an integration plan should provide concrete choices regarding the following questions: What is the core technology used? What is the Development methodology? What is the Source Code Version Control System? How is the Binary-Build process automated? How do we ensure that a new version of one module is compatible with the current version of the other modules Regarding core-technology and methodology we selected Spring Framework and waterfall approach respectively. The rest of the questions will be answered and justified in this chapter Version Control System A Version Control System (also known as a Revision Control System) is a repository of files, often the files for the source code of computer programs, with monitored access. Every change made to the source is tracked, along with who made the change, why they made it, and references to problems fixed, or enhancements introduced, by the change. Version control systems are essential for any form of distributed, collaborative development. Whether it is the history of a wiki page or large software development project, the ability to track each change as it was made, and to reverse changes when necessary can make all the difference between a well-managed and controlled process and an uncontrolled 'first come, first served' system. It can also serve as a mechanism for due diligence for software projects. Version control software, including the well-known SVN and Git, was designed from the ground up to allow teams of programmers to work on a project together without wasting man-hours on paperwork. Instead of manually scanning branches of code and associated notes, version control allows for a central repository that is organized, logical, and facilitates file updates, notation, and even merging. In the following paragraphs we present the four dominant revision control systems. The Section 9 Annex A: State of the art in Version Control System, provides a full analysis of state of the art in Version Control System Version Control System Selection There are a lot of opinions regarding which version control framework is the best, and can force programmers and project management teams into fierce debate. When choosing the right version control, one should consider that some of pros of one package you will come across are subjective, meaning the opinion of the programmer, and other factors, such as speed and IDE plug-in capabilities, overshadow the raw numbers. The main difference between version control systems is whether they are server based or peer-to-peer. Either they have a centralized repository where code is checked out and back in with changes, or a setup where the code is frequently updated from peer sources, a more decentralized network, to keep code current. Beyond that, there are several other factors to be considered like speed, functionality, and the learning curve associated with the system. In the context of MODAClouds, greater priority is assigned to speed and flexibility. Due to the nature of the project, branches should be created, managed and merged constantly. This happens whenever a new feature is introduced, a bug is fixed or an enhancement is applied. As such, branch management should be inexpensive in terms of time and space consumption. Developers are not centrally placed, but dispersed throughout many countries. As such, code decoupling to a central repository is vital and therefore no client-server solution can be followed. Therefore the consortium has selected Apache SVN as the primary VCS system. The SVN repository is located here: After the finalization of the project the consortium will create a Git repository which will contain all open source modules. Public Final version 1.0, Dated June 4,

18 Guidelines about how-to organize the MODAClouds of the code on a SVN repository Once the consortium agreed on having a common project repository common procedures were agreed to keep the repository "sane". Regarding the organization of the code there are some possible approaches to follow: 1. Completely free-style, where everyone puts anything anywhere, thus treating SVN just like FTP; 2. One big project with a single common trunk, branches and tags, where each sub-system is a subfolder of the big project; 3. Each sub-system (or project) with its independent sub-folder, each with its own `trunk`, `branches`, `tags` folder; The first approach is not recommended since this tends to become a major incomprehensible mess of files. The second approach is not recommended because each of the sub-systems has an independent development and release cycle that may be independent of the others. The third approach is definitely preferable. The consortium assumes the root of the repository is denoted as `/`; All projects fall directly under this root, with no additional hierarchy, like `/project-a`, `/project-b`; All projects are prefixed with the following: o o o o o o o o o `modaclouds-` for new software stemming out of MODAClouds; such as `/modacloudside`, `/modaclouds-load-balancer`, etc.; `mosaic-`, `cloud4soa-`, etc., for software that was imported and continued from those projects; in general software should be "grouped" through those suffixes by some meaningful criteria; in general acronyms should be avoided if the expanded version is not too long; (it helps self-describing the sub-system;) in general names should be lower-case separated by hyphens `-`, like seen above; "external" projects, such as those developed by other parties but customized for MODAClouds, like libraries, there will be an `/external` folder, and all these "imported" projects go inside there; all projects managed through SVN will have their own standard layout like `/modacloudside/trunk`, `/modaclouds-ide/branches/xxx`, `/modaclouds-ide/tags/vx.y.z`; all projects managed through other versioning systems (like Git or Mercurial) will be available as snapshots in a `.../snapshots/vx.y.z` folder like `/mosaic-nodecontroller/snapshots/v0.3.1`; in general version numbers (especially tags) should follow the Semantic Versioning, In order to facilitate the checkout of all the MODAClouds software for a release, there will be a `/releases/rx.y` folder wherein each sub-system is "tagged" like so `/releases/rx.y/modaclouds-ide` (instead of tagging in the `tags` folder we do it in this release folder). Additionally, if needed the consortium may impose ACL's such that only certain people can write to certain subsystems (but not deeper than this). For "large" or "binary" artifacts the consortium will provide a "repository" available either through HTTP or FTP, thus most of the above "artifacts" can, and should, be stored there Build Automation Build automation is the act of scripting or automating a wide variety of tasks that software developers do in their day-to-day activities including things like: compiling computer source code into binary code packaging binary code Public Final version 1.0, Dated June 4,

19 running tests deployment to production systems creating documentation and/or release notes Historically, developers used build automation to call compilers and linkers from inside a build script versus attempting to make the compiler calls from the command line. It is simple to use the command line to pass a single source module to a compiler and then to a linker to create the final deployable object. However, when attempting to compile and link many source code modules, in a particular order, using the command line process is not a reasonable solution. The make scripting language offered a better alternative. It allowed a build script to be written to call in a series, the needed compile and link steps to build a software application. GNU Make also offered additional features such as "makedepend", which allowed some source code dependency management as well as incremental build processing. This was the beginning of Build Automation. Its primary focus was on automating the calls to the compilers and linkers. As the build process grew more complex, developers began adding pre and post actions around the calls to the compilers such as a check-out from version control to the copying of deployable objects to a test location. The term "build automation" now includes managing the pre and post compile and link activities as well as the compile and link activities. In recent years, build management solutions have provided even more relief when it comes to automating the build process. Both commercial and open source solutions are available to perform more automated build and workflow processing. Some solutions focus on automating the pre and post steps around the calling of the build scripts, while others go beyond the pre and post build script processing and drive down into streamlining the actual compile and linker calls without much manual scripting. These tools are particularly useful for continuous integration builds where frequent calls to the compile process are required and incremental build processing is needed. The Section 10 Annex B: State of the art on Build Automation Tools, provides an exhaustive analysis of state of the art of the most used build automation tools Build Automation Tool Selection From the build automation perspective, MODAClouds bears some characteristics which will drive the selection of the suitable tool: Code from scratch. In the context of MODAClouds, vast majority of the coding is done from scratch, without porting an existing framework. Therefore, there is no need for conforming to the layout of existing source code during builds. Code uniformity, due to development force decentralization. Uniformity in code is considered a good practice, in the general case, and in the case of MODAClouds where developers do not work together in the same area, this feature becomes a necessity. Strong development background in Java, XML but not in scripting languages. This means that build paradigms that adopt scripts are difficult to be adopted since they require shallow learning curve (greater learning time). Portability. Developers work in different environments and therefore the build tool should be as portable as possible. Repeatable builds / Flexible Dependency Management. All it takes is a check out of your source and, if necessary, a Maven configuration file, and you can issue one command and all of the dependencies for your project will be downloaded and then the project will be built. No checking of binary jars into your source control, or manually copying things, or whatever. Also, it handles transitive dependencies. If you depend directly on one library, you don't have to list its dependencies as well. Maven automatically figures that out for you. Convention and configuration over scripting. Maven has set conventions for project layout. It constructs classpaths based upon your declared dependencies and their transitive dependencies. With ant, you have to explicitly write out XML "code" for the steps to compile, to construct jars, to set up your classpaths, etc. Maven is much more declarative. Public Final version 1.0, Dated June 4,

20 IDE support. A requirement of prime importance to all developers is that the selected build automation tool should be fully supported by the IDEs being used. Solutions that are not supported but instead require command line cannot be adopted in large, complex, project like MODAClouds. The solution that suits to all requirements is Maven. Maven is supported by all major IDEss, including Eclipse, NetBeans, IDEA, JBuilder and JDeveloper. It is suitable for enforcing common source code layout and it adopts an excellent dependency management subsystem. It is also very fast in repeatable, incremental builds of large, dispersed projects, like MODAClouds and it promotes ease-of-configuration without losing flexibility due to the adoption of the Convention-over-Configuration paradigm Continuous Integration Continuous Integration is a software development practice where the members of a team frequently integrate their work usually each contributor integrates his software code at least daily, leading to multiple integrations per day. Each integration cycle is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. A Continuous Integration server exposes the following set of abilities: Contact a source code management (SVN, CVS, VSS...) server and make updates of changes detected in a local directory. Launch one or more ant or maven script as needed for compilation and the packaging of archive files (jar, war and ear). Add plugins for code auditing, testing and measurement of test coverage unit such as FindBugs, checkstyle, PMD, emma, cobertura... Execution of scripts launch tests cactus. Launch application server and deployment of J2EE applications through the ant and maven script. Define inter-project dependencies. Perform tasks of publications such as sending mail notification about the progress of the build process, the result of the builds and test results. Send mail notification to people who have broken the code. Prepare summaries and metrics on the build process. The greatest and most wide ranging benefit of Continuous Integration is reduced risk. The trouble with deferred integration is that it's very hard to predict how long it will take to do, and worse it's very hard to see how far you are through the process. The result is that you are putting yourself into a complete blind spot right at one of tensest parts of a project - even if you're one of the rare cases where you aren't already late. Continuous Integration completely finesses this problem. There's no long integration, you completely eliminate the blind spot. At all times you know where you are, what works, what doesn't, the outstanding bugs you have in your system. Another benefit of Continuous Integrations is that it facilitates bug fixing. It doesn't get rid of bugs, but it does make them dramatically easier to find and remove. In this respect it's rather like self-testing code. If a bug is introduced and detected it quickly it's far easier to get rid of. It s also ease to diff debugging - comparing the current version of the system to an earlier one that didn't have the bug. Bugs are also cumulative. The more bugs you have, the harder it is to remove each one. This is partly because you get bug interactions, where failures show as the result of multiple faults - making each fault harder to find. It's also psychological - people have less energy to find and get rid of bugs when there are many of them - a phenomenon that the Pragmatic Programmers call the Broken Windows syndrome. As a result projects with Continuous Integration tend to have dramatically less bugs, both in production and in process. However, the degree of this benefit is directly tied to how good the test suite is. It's not too difficult to build a test suite that makes a noticeable difference. Usually, however, it takes a while before a team really gets to the low level of bugs that they have the potential to reach. Frequent deployment is valuable because it allows your users to get new features more rapidly, to give more rapid feedback on those features, and generally become more collaborative in the development cycle. This helps break down the barriers between customers and development - barriers are the biggest ones to successful software development. Public Final version 1.0, Dated June 4,