White Paper Challenges and Characteristics of Enterprise Continuous Integration
Introduction Challenges and Characteristics of Enterprise Continuous Integration Many software engineering organizations are implementing continuous integration as part of a move to agile development techniques. Organizations commonly start by using one of the open source continuous integration tools, such as CruiseControl, but quickly find that they require a lot of extension to meet the needs of a large enterprise. This whitepaper explores the challenges of continuous integration in the enterprise and looks at the characteristics of an Enterprise Continuous Integration (CI) tool. Many companies build their software at most nightly and frequently only when a release is imminent without realizing that they are putting themselves at risk of broken builds seriously impacting release dates and software quality. Thanks to books like Continuous Integration: Improving Software Quality and Reducing Risk development managers have begun taking action to make their departments more build efficient. Nightly or weekly builds are no longer good enough. Developers have thrown down the gauntlet forcing build engineers to produce reliable builds as soon as changes are made. Before describing the challenging environment in which build engineers work, we ll take a look at the foundations of the continuous integration movement. How did it start? Where did it come from? And what happens when companies start treating their builds as a precious commodity? 2
What s all the hubbub? It was common for teams to spend weeks, months, or even longer, verifying that their applications worked just prior to releasing them. That time spent integrating was frequently torturous. Developers would have to track down bugs introduced months in the past. An enormous amount of time was wasted trying to merge incompatible changes from different developers leading to integration storms of constantly broken builds and software. A good version control management system might shed some light on the nature of bugs encountered during integration, but frequently fixing these integration incompatibilities was a wild goose chase. The search was made more difficult because not everyone provided good commit comments. Often developers had simply forgotten what they were working on months before. The answer to this painful period in the software development lifecycle is continuous integration. Continuous Integration (CI) dictates that as soon as changes to an application are available, they are integrated with everyone else s changes and run through a rigorous build and test system. The types of changes being considered include modifications to code, environment reconfigurations, database schema migrations, updated versions of third party dependencies, operating system patches: just about anything that might impact a running application. Any changes to any part of a software product might mean that the application no longer works as expected, so it needs to be tested immediately. Forward thinking build engineers began downloading the various continuous integration implementations, such as CruiseControl, well before their managers were clued into the advantages. Typically, a covert installation on a spare machine tucked under a developer desk started making their lives easier. A grassroots push for continuous integration began to spring from one person knowing almost instantly when another team member had neglected to commit a new file. Once the enlightened build engineer revealed their secret CI machine to the rest of the team, it usually spread like wildfire. Now team members could monitor a simple web site to determine if they had made a mistake. Instead of test suites being pushed to the side and forgotten, their tests were being run continuously to verify the build. Developers started to feel good again about builds. As teams and applications grow over time, many enterprises find themselves with CI servers spread across the company in labs, under desks, etc. And often, conflicts arise over who owns what does IT own the servers or are they the sole domain of the build teams? Are the developers maintaining their own secret stash? 3 Challenges and Characteristics of Enterprise Continuous Integration
The Challenges At this point, the build engineer has ventured into true Enterprise Continuous Integration territory. The logical next step is to investigate whether and how all groups can be supported with a single system. A consolidated approach will help teams reuse procedures and tools, better utilize hardware, and free personnel from manual, low-value, babysitting tasks. The following table outlines some of the challenges in real-world, Enterprise CI. Challenge No centralized build-testdeploy infrastructure Impact Maturing organizations often wish to centralize the infrastructure for building, testing and releasing new products. With dispersed build environments, bringing together this infrastructure can be an overwhelming and time-consuming project. Difficulty scaling buildtest-deploy environment As the dispersed build environments grow in complexity and individual idiosyncrasies, so does the effort to scale these environments to meet the demands of growing teams, additional capabilities, wider geographical dependencies and more complicated build scenarios. Testing and resource administration increase in complexity Often, early projects have tightly integrated test teams and specific resource matrices for executing tests and evaluating results. As organizations mature and projects evolve in complexity, dedicated test teams need just-in-time access to the latest builds. Dispersed, individualized build environments place human barriers between developed products and the resources and human capital necessary to effectively test those products. Hardware utilization difficulties Larger, mature projects often require significant hardware resources for both building and testing. A decentralized build and test environment across multiple CI instances cannot effectively leverage managed virtual machines and/or pooled resources (both hardware and software). Barriers to sharing and reusing scripts Within an enterprise, reusable code components are shared between teams. Often, when these libraries are reflected on multiple continuous integration servers, the build scripts for those libraries begin to diverge. Dispersed artifacts lessen the likelihood that code, tests and scripts will be effectively shared by all development teams. Heterogeneous environments across teams When growing companies hire new developers or restructure existing teams, the ideal scenario includes enabling developers to move between projects effectively. Independently established continuous integration servers tend towards heterogeneous development environments and can greatly increase the associated costs of moving development resources between teams. Difficulty tracking builds and cross-project reporting Specific continuous integration server instances provide updates to a particular team, but aggregating status to a departmental or enterprise level remains a significant challenge. Correlating dependency and status across independent projects is often a complex manual process of merging build status data from multiple, independent CI sources. 4 Challenges and Characteristics of Enterprise Continuous Integration
STAR: Scalable, Turnkey, Automatic and Rapid The challenges of Enterprise CI require a CI solution that s scalable, turnkey, automatic and rapid. The solution to building a truly turnkey Enterprise CI system is the ability to break builds up into steps. Those steps (such as checking out from SCM or starting a test suite) can then be used as building blocks to create complex systems. It needs to scale to multiple teams and multiple projects. It needs to be turnkey so that anyone, anywhere can use the system. It needs to be fully automatic so that builds can be performed without specialized knowledge. It needs to be rapid because the code base, numbers of project and number of builds demand a fast turn-around. Scalable Scalability has many axes; an Enterprise CI system must scale by number of projects, size of an individual project, number of users of the system, number of machines in the build cluster and complexity of workflow. Enterprise CI is fundamentally about many projects and many users. Simple CI solutions don t step up to that challenge and are mostly aimed at controlling a small number of projects with a small team. Because teams and builds are large, the Enterprise CI system will inevitably have to manage a large cluster of build machines. The machines will likely be heterogeneous with a wide variety of hardware, operating systems and configurations. The system must be able to assign builds to appropriate hardware/os combinations. Because the system will be shared among users, projects and sites, built-in security in the form of authorization, authentication and permissions is necessary. The ability of users to start, stop, create, and delete must be under fine-grained control. Turnkey Enterprise CI has to be turnkey. The number of builds, the number of users (such as all developers) and the demands of 24/7 operation make babysitting a build impossible. Libraries of common steps can be provided by the Enterprise CI vendor and internally generated steps can be created in the system. Steps can then be strung together to create a complete build that is entirely controlled by the Enterprise CI system. Encapsulated build steps can then be shared across projects and across teams, reusing common steps and improving build development and maintenance. Turnkey implies that from the very beginning, the Enterprise CI system must be usable. It must be trivial to add an existing build script to the system without requiring a rewrite. Over time the build system can be broken up into shareable steps and refactored. Automatic The CI system must be automatic, builds must be launch-able with a single-click, but it must also be compatible with existing scripts. Enterprise CI needs to automate without forcing a rewrite. Three important forms of automatic are: on demand, on stimulus and on the clock. On demand builds should be a single click away. An Enterprise CI system will give anyone the ability to launch any build (if they have the right permissions) from a web-based interface with a single click. The entire build process needs to be encapsulated by the Enterprise CI system with no manual steps. On stimulus builds are typically caused by changes in the SCM system. The Enterprise CI system must monitor SCM (and perhaps individual branches or modules) for change. When change is detected the system can automatically start a build. 5 Challenges and Characteristics of Enterprise Continuous Integration
On the clock builds are the replacement for cron. Certain builds must be performed at certain times of the day (for example, a nightly build may be performed to give QA a fresh version in the morning). The Enterprise CI system must have flexible, time-based build launch facilities. Automation also helps ensure reliability in the process. Any build system that includes manual intervention is prone to error that can result in lost productivity for the entire development organization. individual job steps (such as a compile or link) to machines in a large cluster. Enterprise CI systems should be able to work in tandem with intra-step build parallelism software. Other Properties Three other properties are important to Enterprise CI: auditing, virtualization and reporting. Rapid As the system scales to multiple projects, or very complex individual projects, and as the number of builds increases because of the demands of continuous integration, builds have to be rapid. Hours long builds are a deal breaker. Speeding up software builds comes down to two sorts of parallelism: inter-step and intra-step. Continuous integration means that software builds are being performed continually on a cluster of build machines, and that developers are expecting results rapidly. It s not uncommon to see a requirement for a build and unit test cycle to take no longer than 15 minutes. Getting a build time down to minutes from hours can only, typically, be done using parallelism. The build can be manually broken up into large steps that can be performed in parallel: that s inter-step parallelism. For example, a build system might build the same software on multiple platforms simultaneously, or large components of the system might be built in parallel on different machines. The Enterprise CI system needs to be able to start and coordinate different job steps in parallel. Auditing Enterprise CI needs an auditing capability because organizations have to comply with internal audit and external regulations such as Sarbanes-Oxley. An Enterprise CI system needs to provide complete, end-to-end, traceability of the entire software production process. No single system, such as source code control, defect tracking, or test, provides the overview necessary to reliably record the information needed for the audit of a software production process. Enterprise CI systems need to have reliable storage of configuration information, build output and timestamping for every single build. That way, an audit can be passed easily by querying the CI system for all information related to a single build. Virtualization No organization is unaffected by virtualization. The demands of Enterprise CI are so great (in scalability, automation and repeatability) that virtualization is a natural fit. An Enterprise CI system needs to leverage and control virtual resources (managed by products such as VMware). Often, that s not enough to reach a build time measured in minutes. Doing that requires intra-step parallelism where software analyzes the structure of a build step. For example, software can analyze a single Make or ANT process to determine what fine-grained parallelism can be performed and then distribute For example, an Enterprise CI system should be able to provision virtual machines in real-time as needed by an individual build. This will maximize use of expensive build hardware by sharing builds across machines (using VMs), and will allow the wide variety of build configurations seen in an enterprise setting to be easily managed. 6 Challenges and Characteristics of Enterprise Continuous Integration
Virtualization also brings a time machine benefit: enterprises are often faced with the need to reproduce an old build. An Enterprise CI system will keep all information relevant to a historical build as part of an audit trail. If virtualization is used, then as hardware systems are upgraded to take advantage of the latest processor, memory or storage architectures, the historical build is still able to run. Reproducing an old build on new hardware is as simple as having the Enterprise CI system provision the same virtual machines it used 6, 12, or more months ago. About Stelligent Stelligent is the leading provider of Agile Automation and Transformation Services. Stelligent helps our global customers initiate, adapt and deliver software solutions faster than competitors. Stelligent is an award-winning consultancy and a destination employer for Agile thought leaders. Read more about Stelligent at www.stelligent.com and learn more about Agile engineering practices at the leading blog dedicated to early software quality and agile automation at www. testearly.com Reporting Because the number of builds running through an Enterprise CI is large, reporting becomes a necessity. And the ability to see the current state of the system is vital. Historical reporting can help to identify common problem areas (components of a build that frequently break, for example), trends (a build that is growing over time) and wear and tear (a build machine that frequently fails). Reporting also gives managers a heads-up about utilization of the build resources and when additional resources will be needed. About Electric Cloud Electric Cloud powers Continuous Delivery. We help mobile, embedded systems and enterprise web/it providers deliver better software faster by automating and accelerating build, test, and deployment processes at scale. Industry leaders like Qualcomm, SpaceX, Cisco, GE, Gap, and E*TRADE use Electric Cloud solutions and services to boost DevOps productivity and Agile throughput. For more information, visit electric-cloud.com. Viewing the current status shows actual build cluster utilization and recent failures. Drill down reporting lets developers examine a build failure by going from the Enterprise CI system down to a log file level with a few clicks. Overall, reporting provides the view into the most important indicator of software project health working code. Conclusion It s easy to get started with an open source CI system implemented by a small team. It s also easy to outgrow it. Enterprises should plan accordingly when choosing a CI system and verify that the CI system characteristics will meet the needs of enterprise software development, and not just the needs of one team s CI trial. 7 Challenges and Characteristics of Enterprise Continuous Integration
Corporate Headquarters Electric Cloud, Inc. 35 S. Market St, Ste 100, San Jose, CA 95113 T: 408.419.4300 F: 408.419.4399 info@electric-cloud.com www.electric-cloud.com Electric Cloud China New City Center Plaza, No.70, Room 908, Tong Chuan Road, Shanghai, 200333, China T: +86 13601825314 / +86 13761649476 china.info@electric-cloud.com Electric Cloud Europe 1650 Arlington Business Park Theale, Reading Berkshire RG7 4SA United Kingdom T: +44 (0) 0207.872.5500 europe.info@electric-cloud.com Electric Cloud Japan KK 22F Shibuya Mark City West 1-12-1 Dogenzaka, Shibuya-ku Tokyo 150-0043 Japan T: +81.3.4360.5375 japan-info@electric-cloud.com Electric Cloud, Inc. All rights reserved. Electric Cloud, ElectricAccelerator, and ElectricFlow are trademarks of Electric Cloud, Inc. All other names are used for identification purposes only and are trademarks of their respective companies.