DEVELOPMENT OF A AUTONOMOUS BUILD AND DISTRIBUTION SYSTEM FOR SOFTWARE WITH HYBRID DEVELOPMENT MODELS

Transcription

1 Faculty of Computer Science Institute of Systems Architecture Diploma Thesis DEVELOPMENT OF A AUTONOMOUS BUILD AND DISTRIBUTION SYSTEM FOR SOFTWARE WITH HYBRID DEVELOPMENT MODELS Tino Breddin Mat.-Nr.: Supervised by: Prof. Dr. Alexander Schill, Dipl. Inf. Josef Spillner Submitted on August 10th, 2010

2 2

3 ABSTRACT The Open-Source Software business model has helped Open-Source Software (OSS) to gain acceptance within the software industry as viable software which can handle mission critical tasks. This model has also introduced a new way to develop Open-Source Software. Instead of being entirely developed by a open community of developers, a company supports and leads the development primarily. Moreover OSS projects in general benefit from the increasing interest, with more developers helping such projects. The distributed nature of OSS development teams requires new ways to manage projects in order to achieve good productivity. Concepts like Continuous Integration provide such methods, focusing on improving information sharing and instant feedback on changes within a development process. Continuous build systems are a central part of any infrastructure which implements Continuous Integration methods and as such holds an important role to improve software quality. Good software quality includes robustness in regard to the deployment environment of any given software. Better quality is perceived by users if they don t run into problems when setting up and using the software. Modern continuous build systems don t help development teams as much as desirable in such cases. Old concepts which build the foundation of such systems prohibit the enhancement of these to be of better use. Therefor the system which is developed in this thesis, named Swarm, is based on different concepts which are meant to support software development teams in a improved manner. Swarm s core is based upon the use of many so called production environments, as the environments in which software should be build and tested is called as part of Continuous Integration. To further integrate well with development processes based on Distributed Version Control Systems, the features of such systems are used to provide feedback for the various changes to a software project which are available through various communication channels. The prototypical implementation of the system presented in this thesis shows how software quality can be boosted further than existing continuous build systems can do. Moreover Swarm can be adapted to allow even more improvements which is reserved for future extensions. 3

4 4

5 Dedication For my great-grandma, Gertrud. She will always be remembered as the kind women she was. For my wife, Marie-Kristin, who always finds ways to make me smile. For my family, thanks for the unconditional support throughout my studies. 5

6 6

7 CONFIRMATION I confirm that I independently prepared the thesis and that I used only the references and auxiliary means indicated in the thesis. Dresden, August 10th,

8 8

9 CONTENTS 1 Introduction Motivation Idea Structure of the Thesis Summary Background Continuous Integration Concepts System: CruiseControl System: Hudson Summary Source Code Management History of Version Control Centralized Version Control Distributed Version Control Git Summary Collaborative Software Development

10 2.3.1 Sourceforge Github Summary Erlang/OTP Project Language Characteristics Open Telecom Platform Development Process Summary Summary Swarm A Distributed Build System Requirements Analysis Continuous Integration Distributed Version Control Systems System Integration Summary System Principles Single Project Support Deferred Authentication Git SCM Support Summary System Design Architecture Subsystem: Platform Management Subsystem: VCS Interface Subsystem: Data Storage Subsystem: Job Processor Subsystem: Operation and Management System Configuration Contents

11 3.3.8 Summary System Integration Event Hooks Hook Scripts Summary Workflows Latest Version Change Result Propagation New Change Summary Summary Evaluation Component-based Development Usage of Erlang/OTP Test Run Test Deployment Summary Conclusion Plugin Support Restful HTTP API Native VCS Support Inter-Instance Communication Summary List of Figures 72 List of Tables 74 Glossar 77 Acronyms 79 Bibliography 80 Contents 11

12 12 Contents

13 1 INTRODUCTION The topic of this thesis is introduced in this chapter by giving a detailed explanation of the initial motivation in section 1.1. The following section 1.2 concisely summarizes the idea which builds the foundation for this work. Lastly the structure of this thesis is presented in section 1.3 and the foundations for this thesis are summarized in section MOTIVATION In recent years OSS has emerged as a business opportunity which is now well known as the Open-Source Software business model. While some companies make their software available to the public to drive adoption others specialize on supporting already established OSS projects. While it becomes easier for all participants to use Open-Source Software, this is not necessarily the case when it comes to driving the development forward. Often companies have well established internal development procedures which they don t want to expose to the public. This makes it hard for external developers to contribute to a project. Furthermore in-house development driven by customer projects might not be integrated with the OSS version of the software. In parallel to the development of the OSS business model, the development models being used have change as well. This led to the adoption of continuous build systems to improve software quality and feedback about changes throughout the complete development cycle of a software project. This class of software already provides tremendous benefits for any software project. Still, these build systems have been designed with other requirements in mind than modern OSS projects are presented with. Software is expected to support many operating systems and integrate with an ever-changing and divers set of third-party software. If the gap between in-house development at companies and efforts of OSS developers could be bridged to connect the different workflows without ongoing manual effort, contributions would be much easier to integrate than it is the case at the moment. Such changes would enable the collaboration between OSS projects and corporations in new ways when it comes to software development. In addition the possibilities to more easily extend support for operating systems would add more benefits to the overall goal, to improve software quality which itself helps OSS projects to drive adoption. 13

14 1.2 IDEA The concept of continuous build systems needs to be reconsidered based on the requirements of modern OSS projects. The core functionality of build systems should be modified to provide the ability to build software on many different systems by default. Furthermore the outcome of builds should be easily sharable with other tools to enable the creation of sophisticated development processes. As a Proof of Concept (PoC) for these core functionalities a continuous build system should be developed which proves the aforementioned benefits while taking a simplistic approach in regard to other aspects of modern build systems. 1.3 STRUCTURE OF THE THESIS Chapter 1 opens up this thesis by introducing the underlying motivation and general idea to solve the described problems. Subsequently relevant topics, such as Continuous Integration (CI) and Source Code Management (SCM), are explored in detail in chapter 2 to provide the necessary background for further discussions. The system itself is developed in chapter 3. This includes the requirements analysis as well as the system design. The results are evaluated in chapter 4 in regard to the initial assumptions and gathered requirements. Finally this thesis concludes by discussing shortcomings and future improvements in chapter SUMMARY OSS development has led to rapid changes both in software development practices and software business models. Unfortunately the available software development tools don t allow OSS projects to fully exploit collaboration between all contributors and drive user adoption, as explained in further detail in section 1.1. Instead it is suggested in section 1.2 that a novel continuous build system should serve as a showcase of how new core functionalities can further enhance the software quality of OSS projects. Lastly section 1.3 describes the structure of this thesis. 14 Chapter 1 Introduction

15 2 BACKGROUND The system which is developed in this thesis is based on the achievements made in different areas of software development to address the issues presented in the section 1.1. As the thesis title implies the developed system is a continuous build system, thus the theory behind Continuous Integration will be explained in section 2.1 and highlight the need for such systems. The various emerging Distributed Version Control System (Distributed VCS), e.g. Git [git] and Mercurial [mercurial], have changed the way developers share code and contribute to OSS projects. Section 2.2 describes the history and basics of these systems briefly. The following section 2.3 gives examples for platforms which support collaborative software development within OSS projects. Subsequently the basics of Erlang, the programming language being used for the implementation of the system developed in this thesis, are explained in section 2.4. Section 2.5 summarizes the presented background while pointing out important aspects. 2.1 CONTINUOUS INTEGRATION The term Continuous Integration was first defined by [Beck2000] as part of a set of Extreme Programming (XP) best practices. Building and testing a software product every time a development task is completed was considered CI. [Beck2000] arguments that this regular cycle would instantly show whether new changes break the software and could be addressed much earlier than in traditional development models such as the Waterfall Model [Royce1987]. This simple concept has evolved from a suggested XP practice to a software development methodology [Duvall2007] which is based on several practices of which the most important are briefly explained in the following sections Concepts Build Automation Building the current version of a software should be possible by executing a single command. The assumption is that the more steps a build process involves the more unlikely it becomes that developers actually build the software during development. This might not hold true for small projects but the more complexity and components are added to a software the more difficult the build process becomes. Keeping the simplicity of building software during the development is a crucial part of Continuous Integration. 15

16 Build Fast During the development of software it is almost inevitable that the time which is needed to run a build increases. Building complex software can take several hours to finish which increases the time it takes until developer receives feedback by building. This leads to a state where nobody can tell whether recent changes have affected the system behavior or even broke the system entirely. To alleviate this situation one should focus on keeping the build process fast so that developers can actually build the software and test it after having applied changes. This can be achieved by creating components which are independent enough that it makes sense to build and test them individually after recent changes, while building and testing the whole system independently. Progress Must Be Visible Extreme Programming focuses on the interaction within a team instead of following strict processes. This creates a need for providing as much information as possible about the state of the development to any team member at any time. It should be visible how much work a team member was able to finish, which changes broke the system at which time by whom, how much attention has been put on individual components and whether a team member got stuck at a certain task. The goal is to keep all team members informed so that the most critical tasks can be tackled at any time. Each Commit triggers Builds A way to implement the principle described in the previous section is to build the complete system every time a developer commits to the code base. In a software development team one measure for progress are committed changes, thus it is feasible to automatically provide feedback for any commit which is visible to all team members. It is essential that developers are relieved from the burden to start this process manually since this should be done behind the scenes. Clone Production Environment While developers tend to have a development environment which is specifically geared towards their personal needs, this too often isn t the case for the build and test environments. To eliminate the risk of leaving issues undetected when running the developed system on the target platform, it is necessary to create a environment for the automatic builds and tests which resembles the target environment. This doesn t only affect the chosen operating system but also the complete hardware and software stack of the target system. The better the CI environment is modeled after the target environment the more confident can the development team be that they didn t miss critical bugs before deploying the software System: CruiseControl As one of the first successful OSS continuous build systems CruiseControl [cruisecontrol] has served as an example for many systems of that kind thereafter. Initially developed within ThoughtWorks the system has been published as OSS later and found wide adoption. This is underlined by the subsequent offerings of commercial products by ThoughtWorks based on 16 Chapter 2 Background

17 CruiseControl to meet corporate user s requirements. Technically CruiseControl is a traditional continuous build system which uses the server it is served from to run builds. Its flexible plugin architecture has helped to extend the base framework with various additional functionality which helped to spread adoption of the system even further. CruiseControl is still in active use and development with various spin-offs, written in other programming languages, being available System: Hudson The OSS continuous build system Hudson [hudson] has become a very popular alternative to CruiseControl due to its extensive SCM support and plugin architecture. Its development has been sponsored by Sun Microsystems thus attracting many users early on. Although it was initially being used as a continuous build system for Java projects, support for different projects has been added through the help of many additional plugins. The project keeps growing in popularity after receiving various OSS awards lately. It is nevertheless based upon the very same principles as CruiseControl like using the local server for running builds Summary Continuous Integration is comprised of various smaller concepts which are individually referred to as agile practices and as a whole help improving software development by sharing information. Notably the idea of making any progress visible to all members of a team as well as getting progress information as fast as possible support the idea of CI. Many OSS continuous build systems have picked up these concepts and help shaping the software build process to fit such agile practices. The OSS systems CruiseControl and Hudson stand out because of their wide user adoption which is influenced by the versatility of both tools. 2.2 SOURCE CODE MANAGEMENT When a team of software developers works on the same software it becomes increasingly important to be able to see who made which changes when. This need for organization led to the development of Version Control Systems (VCSs). Since version control is an integral part of any Continuous Integration environment it is necessary to highlight how version control started and how it has evolved until today. Section covers the roots of Version Control Systems and details the important milestones in the development of the state-of-the-art systems. The class of Centralized Version Control Systems is covered in section Furthermore section focuses on the recent developments in the area of Distributed Version Control Systems. Finally the takeaways of Source Code Management are presented in section History of Version Control The development of special purpose systems for controlling source code became first publicly noticed through [Rochkind1975] and [Glasser1978] who introduced the concept of using a SCM system and recent development in this area. [Tichy1982] showed the stages and culprits involved in working on such systems. The first widely used OSS SCM called RevisionControl System (RCS), introduced by [Tichy1985], was an important milestone because it nourished the development of many different approaches in the area of SCM as summarized by [Royce1987]. Further development has focused on Centralized Version Control Systems (Centralized VCSs) which are presented in detail in the following section A recent conceptual change has been introduced by Distributed Version Control Systems, covered in section Source Code Management 17

18 central repository developer developer developer Figure 2.1: A centralized version control workflow Centralized Version Control The first group of SCM systems which received wide adoption were Centralized Version Control Systems. These systems rely on a central location to store all versioned data as well as version history. These locations are referred to as central repository. Figure 2.1 depicts a typical Centralized VCS setup with one central repository and several developers. These users can only checkout a single version of the data in the central repository. No further versioning information is kept within a developer s copy of the data other than the version identifier itself. Thus all versioning knowledge is kept within the central repository. Furthermore developers are only allowed to push changes to the central repository. Those changes need to be based on the latest version of the data as well. This leads to a very inflexible process of working on the source code. However having this central authority can fit well into other parts of a software development process. Concurrent Versions System (CVS) [cvs] is a notable implementation of such systems as it gained popularity after its initial release in 1990 and is still in active development. Further improvements are provided by the OSS Centralized VCS Subversion as described by [Pilato2004] Distributed Version Control While Centralized VCSs employ a strict workflow based on a central repository the newer concept of Distributed Version Control Systems tries to support flexible workflows as much as possible by eliminating the need for central repositories. All copies of a repository are considered full-fledged repositories themselves because all versioning metadata is kept within a copy as well. Therefor developers are given the freedom to share data more easily as shown in figure 2.2. This unconstrained flexibility allows teams to create workflows which are specifically tailored towards their needs by creating roles and logical contracts for repositories. Figure 2.3 presents a simple workflow which contains a single shared repository, which can only be 18 Chapter 2 Background

19 shared repository developer developer developer Figure 2.2: A distributed version control setup. pushed to by a single maintainer. Developers can read from the shared repository and share data between each other. But ultimately final changes need to be send to the maintainer for inclusion in the team s shared repository. This example shows how versatile Distributed Version Control Systems are in terms of workflow adaptation. While there are many Distributed VCSs available, especially two systems have established themselves as the state-of-the-art. Mercurial [mercurial] is a system implemented in Python which focuses on providing a easy user interface, thus there is always only one way to do a certain task. Another approach is taken by Git [git] which is implemented in C and often provides the developer many options to accomplish a certain task. In general both systems are very similar regarding their features and performance Git Git has initially been developed by the Linux Kernel Development Team to support their complex requirements for SCM. Since then it has become very popular especially in the OSS community. Git provides many ways to organize and change the source code one is controlling with it. But the system developed in this thesis only needs a small subset of the features available thus the important features and their usefulness are discussed in this section. Branching Creating branches of the source code is a essential tool to separate different development paths with the same base source code. The concept is that one can work on a separate branch for a set of features and once all changes have been committed, these would be added to the main branch again. Creating a new branch used to be a time consuming operation in Centralized VCSs which often involves the locking of the main branch, thus blocking anybody from committing changes. 2.2 Source Code Management 19

20 maintainer shared repository developer developer developer Figure 2.3: A development workflow employing a maintainer to approve changes to the shared repository. Git performs branching operations very fast because it is able to operate locally instead of having to synchronize with a remote server. This allows for a more flexible usage of branches, such as having a separate branch for each known bug or feature which somebody is working on. Switching a branch is fast as well, so that one can change between so called Topic Branches easily if necessary, without mixing fixes for a bug or the implementation of a new feature. Developers are encouraged to use many branches instead of mostly working on the main branch. Merging When making extensive use of branches the actual merging of changes in a branch with another branch is very important. Just as with branching the merge operation itself is very fast which enables developers to merge often, to check that recent changes work well with the downstream of the project. Furthermore Git uses a multi-stage merging algorithm which is supposed to work very well without much manual interaction. Rebasing The rebase operation is used to align the changes in a branch to an upstream branch. It rolls back the local branch to the last commit it has in common with a upstream branch. Then it will apply the new changes from the upstream branch to the local branch. Finally the local changes will be re-applied on top of the upstream changes. If Git experiences merging problems the developer will have to manually merge the conflicts. The operation is supposed to be used to apply local changes on top of the upstream changes without actually merging them from the beginning. Therefor the upstream changes become the base for any local changes, thus one will rather edit the local changes than the ones from the upstream branch which are supposed to be working and well tested. 20 Chapter 2 Background

21 2.2.5 Summary Source Code Management has become an integral part of any software development process over the last decades. In this section an overview has been given over the History of Version Control beginning with RCS as the first widely used OSS Version Control System until the more recent systems, such as Mercurial, gained traction. The two dominant approaches for SCM, Centralized VCS and Distributed VCS, mostly differ in how flexible the workflows are which they enforce as described in detail in sections and Git, a popular Distributed VCS, puts an emphasis on the speed of any operation it performs and is used for the system developed in this thesis. 2.3 COLLABORATIVE SOFTWARE DEVELOPMENT Open-Source Software has led to the assembly of many teams which work in separate physical locations around the planet. Inevitably traditional software development tools which are used in corporations can t provide sufficient support for such flexible teams. This has led to the development of new tools and platforms which help developers to collaborate on projects productively while being physically separated. Two popular web platforms for collaborative development are presented in sections and Lastly section summarizes the benefit for development teams Sourceforge As one of the first project hosting platforms SourceForge became a hub for OSS projects. It started out in 1999 with source code hosting and added more advanced features ove time to become a full-fledged project hosting platform. These features include bug tracking, wikis, forums and mailing lists. The platform also expanded to offering commercial plans for closed projects, which essentially receive the same features Github Many project hosting platforms provide various options for the different features which are available, while some niche platforms focus on one option for each feature instead. Github is one such niche platform which started as a source code hosting platform for Git repositories. Later it has become a viable platform for complete projects though, while still focusing on giving developers only one option for any feature such as code reviewing or wikis. The platform has found wide adoption especially because of its early support for Git hosting. The common workflow for code sharing on Github is depicted in figure 2.4. Any developer can create a clone of an existing public repository, which is then tied to his own Github account. This repository can be used to share code changes with others since it is publicly available. Other developers can then incorporate these changes into their own copies of the original repository Summary Teams which are physically separated need not only a way to share information but also need to be able to setup processes which support their efforts. Because of the ubiquitous internet access especially web-based tools have been adopted for such purposes. Both SourceForge and Github, which were described in sections and 2.3.2, are popular among the large set of such platforms because auf their good feature set. 2.3 Collaborative Software Development 21

22 Github original repository cloned repository cloned repository developer developer developer Figure 2.4: A common workflow for repositories hosted on Github. 2.4 ERLANG/OTP PROJECT The Erlang/OTP project comprises the development of the programming language Erlang and a set of well supported applications referred to as Open Telecom Platform (OTP). The language itself was developed at the Ericsson CSLab from the 1980s until it was made available as OSS in 1998 [?]. The language and libraries are covered by the Erlang Public License [epl]. Since then Ericsson has continued driving the development of the language especially because it s still used for internal product development. Erlang/OTP is already widely used for telecom and messaging applications while the adoption for systems which require easy scalability and fault-tolerance is increasing steadily over the last years. This section introduces Erlang/OTP by giving a dense overview of the language in section itself and its libraries in section Section focuses on the current development process which is used by the Erlang community. Finally section summarizes the findings and shortly discussed the use of Erlang in this thesis Language Characteristics Erlang is often referred to as a concurrent functional programming language. It has several features which are meant to make programming concurrent applications easier. The most important of those are briefly described in this section in addition to some language basics. Virtual Machine Erlang source files (.erl) are compiled to byte-code (.beam). This byte-code is then interpreted at runtime by the Erlang Virtual Machine, called BEAM. This gives Erlang the advantage of good portability, since the compiled code is platform independent. The virtual machine itself is optimized for multi-core processors which improves its performance on modern hardware tremendously. 22 Chapter 2 Background

23 Lightweight Processes A fundamental language construct are processes. A process is some code which is executed sequentially. It has a very small memory footprint as well as fast creation and destruction performance. The Erlang Virtual Machine can manage millions of processes while only being memory bound. Processes can be executed in parallel making executing chunks of code in parallel very trivial. Listing 2.1 shows how a process can be started, which is called spawning in Erlang terminology. 1 spawn ( fun ( ) > e r l a n g : d i s p l a y ( hello_world ) end ). Listing 2.1: Spawning a process which prints a message to stdio. Message Passing Erlang processes don t share any memory. Thus messages are being used to send data between processes. Any process can send any type of data to any other process in the system. At a receiving process all incoming messages are being added to a receive queue from which the process can pull new messages. Therefor the language provides the receive keyword which makes a process block and pull the oldest message from the queue. Listing 2.2 shows how messages are send and received. 1 % sending message to myself ( the running process ) 2 s e l f ( )! { d i s p l a y, hello_world }. 3 4 % receive t h a t s p e c i f i c message 5 receive 6 { d i s p l a y, Msg} > e r l a n g : d i s p l a y (Msg) 7 end. Listing 2.2: Sending and receiving messages with one process. Hot Code Swapping The virtual machine provides the ability to swap Erlang byte-code at runtime, allowing systems to be upgraded without any downtime. Therefor two versions of any given module are kept in memory, the current version V and the older version V-1. If a new version of a module is loaded it becomes the current version V while the former current version becomes V-1. All new processes will be using the current version of the module, whereas already running processes will continue to use the older version. Yet the running processes can choose to upgrade to the current version at any time, making it possible to upgrade whole systems without having to restart running processes. Single Variable Assignment As mentioned earlier Erlang doesn t have any shared memory. This concept has been taken to the variable level by making variables immutable after being assigned a value. This allows for easier tracing of errors because the assignment of a value to a variable is explicit. However this concept can lead to messy code in practice if not used properly. Listing 2.3 shows how variables are assigned properly. 1 % proper assignment of a new v a r i a b l e 2 MyVar = % f a i l i n g assignment, because v a r i a b l e i s a l r e a d y assigned 5 MyVar = MyVar + 2. % r e t u r n s : exception e r r o r : no match of r i g h t hand side value % proper assignment 8 MyVar2 = MyVar + 2. Listing 2.3: Correct variable assignment and common failures. 2.4 Erlang/OTP Project 23

24 Distributed Erlang Message passing and lightweight processes give programmers a easy abstraction from multi-core processors allowing them to make use of all available resource without having to worry about multi-threading. Distributed Erlang provides the same level of abstraction on a system level. A single running Erlang virtual machines is called node. Erlang nodes can be connected in the sense that processes from each node can communicate through messages with each other. This makes it trivial to run any Erlang program on more than one Erlang node. The real advantage is that these nodes can reside on different physical machines, which allows programmers to run a distributed system on top of distributed Erlang Open Telecom Platform Erlang is shipped with a set of applications which act as the standard library for the language. These applications are referred to as OTP, named after their initial development as part of telephony development projects. These applications are known to be well tested and in production use for many critical systems, thus the Erlang/OTP team encourages users of Erlang to utilize the OTP as much as possible. The applications are constantly improved to ensure that they present a viable solution for common problems Development Process After Erlang/OTP has been made available to the public as Open-Source Software in 1998, the Erlang/OTP team at Ericsson continued to manage the development of the language. New releases would be published on a regular basis, but the development wasn t visible to the public. Furthermore the direction of development was largely based on customer s needs. The interaction with the Erlang community was mainly based on three mailing lists, one for bug reports, one for patches and one intended for general discussion. EEPs In order to be able to get involved in the development of the language itself one would have had to write an Erlang Enhancement Proposal (EEP) [eep]. Those proposals were meant as a base for discussing important language changes before any work is done. This process has worked well until 2008, when the Erlang community started to grow and it became apparent that Erlang/OTP would benefit from a better community involvement in the development process. In 2009 the Erlang/OTP team started to open up their development process by publishing the main development branch of Erlang/OTP on Github [erlangongithub]. Now developers are able to contribute changes to the development version by sending standardized patches to the patches mailing-list. Those patches will then be reviewed and potentially added to the development branch. Thus the open source community is now more involved into the development which has led to an increased interest in Erlang/OTP, judging from the number of patches submitted to the mailing list Summary Erlang is a mature programming language which is often used for programming highly concurrent applications. Together with its standard library OTP it allows developers to create scalable and fault-tolerant software with less effort than would be necessary with other programming languages such as C. Erlang is also a functional programming language which uses concepts such as pattern matching, no shared data and single-assignment variables to allow programmers to express difficult problems more easily. The language is Open-Source Software and available from its main source code repository on Github [erlangongithub]. 24 Chapter 2 Background

25 2.5 SUMMARY Continuous Integration is a software development methodology which emphasizes sharing of information within a project. The concepts behind CI and continuous build systems which actually implement these concepts were described in section 2.1. Section 2.2 presented the history and modern tools for Source Code Management which became increasingly important as software project became more ambitious and complex. It focused on Git as it is used within the system which is developed in this thesis. Subsequently the importance of project hosting platforms for collaborative software development were explained in section 2.3, giving Github as an example for such platforms. Lastly section 2.4 gives a short introduction to the programming language Erlang, which is used for the prototypical implementation of the system developed in this thesis. Its strength for concurrent programming are especially useful for the concurrent nature of the this system. 2.5 Summary 25

26 26 Chapter 2 Background

27 3 SWARM A DISTRIBUTED BUILD SYSTEM Continuous build systems have been gaining popularity ever since Continuous Integration has been adopted by more and more software development teams. As a result those systems have matured to a point where there are many available systems which provide very similar capabilities. The system which is proposed in this chapter takes a different approach in regard to some concepts which have been indicated in section 1.1. Section 3.1 creates the foundation for this proposal by specifying the high-level requirements for this system. Three important system principles are defined in section 3.2 before the system design is presented in elaborate detail in section 3.3. Subsequently the capabilities for integration with third-party systems is outlined in section 3.4 before some core workflows of the system are explained in section 3.5. Lastly this chapter is concluded by a summary of the proposed system in section REQUIREMENTS ANALYSIS This section presents the requirements for Swarm. Section outlines the requirements which are defined by existing continuous build systems and the concept of Continuous Integration. Furthermore the popularity of Distributed Version Control Systems requires some novel functionality as well which is explained in section In addition section describes ideas from general systems integration which relate to continuous build systems. To conclude all requirements are again summarized in section Continuous Integration Continuous Integration as a development methodology already defines a set of requirements which continuous build systems need to adhere to. These can also be identified by looking at some core features of existing continuous build systems. Project Repository Monitoring As part of a project which is setup within the continuous build system the respective source code repository should be monitored automatically. This allows the continuous build system to notice when the latest version within the repository changes. 27

28 Build Latest Version Furthermore the continuous build system should build and test the latest version of the source code, once it changed. The delay between the time when the change was published and the start of the build should be as small as possible. Make Results Publicly Available Continuous Integration stresses the fact that any information about the state of the source code should be shared within a team. Thus the results of builds of a project need to be presented in a publicly readable form, once available. That means even results from ongoing builds should be presented. Build in Production Environment The motivation of this thesis presented in section 1.1 states that builds need to be run on many systems to improve software quality rather than sticking to one particular environment. This requirement especially holds true for OSS projects. Therefor a continuous build system needs to be able to build the source code on as many platforms as possible, which are distinct from the one it is running on. This requirement also marks a conceptual change from existing continuous build systems, which focus on using the platform they are running on as the reference environment Distributed Version Control Systems The paradigm change within Source Code Management systems has allowed development teams to customize their development workflow towards their needs. In addition developers are presented with many now effective methods to control their source code, such as branching. Continuous build systems need to adapt to this change to be able to keep up with the changing pace of development. Build Topic Branches Globally shared repositories are not the first destination of changes within custom development workflows anymore. Teams might decide to only submit verified changes to their central repository, while changes which still need to be verified are kept in the developers repositories, some other shared repository or within a code review system. Either way, part of the verification should be the building of changes as soon as developers have finished working on these. A continuous build system should build such changes whether they live in a repository as topic branches, are kept as patches in a code review system or send as an to a mailing list. The same benefits of building the latest version of some source code apply to building potential changes to the latest version, which is the next step of Continuous Integration. This requirement marks another conceptual change from existing continuous build systems, since these only take the latest version in a repository into account when looking for changes System Integration A continuous build system is usually part of a larger combination of systems which support the development process of a team. Within this process each system has its dedicated purposes and brings distinct advantages. Just like other systems a continuous build system needs to be properly integrated into such an environment to fully exploit the advantages of Continuous Integration. 28 Chapter 3 Swarm A Distributed Build System

29 Share Results An important part of the integration is that results of builds for any change are shared with other systems within the build environment. This allows the setup of sophisticated processes which propagate the data to the correct communication channels. Thus a continuous build system should provide the foundation for sharing results with other communication channels. Share Results with other System Instances OSS projects are usually developed by many participants which can be both individuals and corporations as observed in the motivation of this thesis in section 1.1. Thus the development might not be centralized and based on a single defined process. Instead the different parties might have separate processes and development environments, but are still working on the same project. In such cases receiving information about changes which have been made within another environment can be as important as changes from within the own environments. Thus a continuous build system should be able to share both changes and their results with other instances of the system. This requirement defines another conceptual change from existing continuous build systems, because these have been regarded as stand-alone systems ever since. This sharing of information would truly federalize information between development environments Summary In this section the following set of core functionalities has been defined which should be provided by a next generation continuous build system. Project Repository Monitoring Build Latest Version Make Results Publicly Available Build in Production Environment Build Topic Branches Share Results Share Results with other System Instances They are influenced by the concepts behind Continuous Integration, Distributed Version Control Systems and system integration. Each of these requirements will be further discussed as part of the system design in section SYSTEM PRINCIPLES This section explains the principles which the bespoke system follows in addition to implementing the requirements gathered in section 3.1. These principles allow the system design, which is covered in detail in section 3.3, to be very precise without focusing on extensibility where it isn t needed. 3.2 System Principles 29

30 The following principles will be presented in detail in the following sections. Single project support Deferred authentication Git-only SCM support A summary of the applied system principles is given in section Single Project Support Popular continuous build systems often provide support for multiple projects at a time. This feature is required due to the nature of the usage of such systems and the groups of users. The monitored software tends to be project-related thus running for a specified amount of time during which the feedback from Continuous Integration is most important. After the project has finished the software might still be monitored but since no changes will be added the continuous build system won t add any benefit thereafter. In the case of such time-delimited projects the administrative overhead should be kept to a minimum. Having one instance of a continuous build system being able to monitor many projects eliminates the need to setup an instance per project. This saves the system administrators time as well as resources since no additional hardware is needed. Swarm is meant to be used by projects with different characteristics. Such projects should be ongoing rather than time-delimited. Many OSS projects don t have a pre-defined set of requirements which defines the scope of a project. Instead they are work-in-progress where new ideas and features are added continuously. Furthermore the targeted projects are generic platforms rather than specific instances of a software. Whereas in client projects the deployment environment is well known and teams can work towards that environment, OSS projects can t assume how the environment into which the software is deployed looks like in all cases. Thus OSS projects need to support as many combinations of hardware, operating system and software. This ensures that the software can be spread among as many user groups as possible. Based on these assumptions Swarm supports only one project per system instance. A administrator needs to setup an instance of Swarm per project which should be monitored. This design decision allows for a simple administration of an instance. Nevertheless The underlying architecture as explained in section is not completely tied to this concept. If necessary it can be extended to support the monitoring of multiple projects, which would also require the adaptation of other components such as the Operation, Administration and Maintenance (OAM) subsystem presented in section Chapter 3 Swarm A Distributed Build System

31 user request Web Server user data exchange LDAP Server any user only if user is SysAdmin web frontend serves configuration frontend serves SWARM instance Figure 3.1: A web server providing authentication through an LDAP service Deferred Authentication Swarm is a continuous build system which acts autonomously after a administrator has setup the project correctly. Thus it doesn t require a complex user management or flexible authentication support because only few changes need to be made thereafter. To keep the system simple and focused on its core functionality it won t attempt to provide any sort of authentication. Instead other means of security should be used to provide the right users access to the configuration frontend as well as giving all users access to the main web frontend. In this respect Swarm defers the authentication support to external systems which provide a mature and flexible security model already. This limitation, which is considered to be a feature, will also allow Swarm to be deployed in internal networks with custom security measures already in place. Since Swarm doesn t enforce any security model the employed security system simply needs to be made aware of a Swarm instance. All data available to the security system can be used as in any other case. A simple scenario for deferred authentication is the usage of web servers and Lightweight Directory Access Protocol (LDAP) to limit the access of users to certain websites as illustrated in figure 3.1. Since the Swarm configuration frontend is available via a unique Uniform Resource Locator (URL), the web server can be configured to provide only system administrators access to that particular URL and allow all users to view all other URLs served by Swarm. This kind of configuration is a routine operation and often used to secure certain URLs Git SCM Support As described in section various OSS Distributed Version Control Systems are available and often provide the same core set of functionality. To emphasis simplicity Swarm will only 3.2 System Principles 31