November 2013 Daitan White Paper Software Continuous Integration & Delivery INCREASING YOUR SOFTWARE DEVELOPMENT PROCESS AGILITY Highly Reliable Software Development Services http://www.daitangroup.com Daitan Group 2013. All rights reserved. Other company and product names may be trademarks of their respective owners.
Software Continuous Integration and Delivery 1 ABSTRACT Once upon a time, people wrote code and then ran it to find how well it worked. Working individually, the developer was able to iterate quickly until the software was useful to solve a real problem. As software became more complex, it was broken into separate and independent sub-modules developed more or less in isolation by individual developers. Each sub-module was tested separately, the system was integrated back together, then tested again, staged, released, distributed and deployed. Software development cycles stretched over several months (or several years). The pendulum is swinging back. Software engineering is now evolving towards shortening the feedback loops and minimizing the time it takes from code being written to being in production. This time, we want to do it with more quality and predictability. The tools to achieve that are processes, automation, and collaboration. Moving across the levels depicted in the picture above is not simple. It involves not only adapting processes and automating them, but also changing organizational culture and structure. Continuous Delivery is not necessarily the right objective for all types of software, but generally most software development organizations are moving towards more Software Process Agility. Daitan Group 2013 1
2 AGILE DEVELOPMENT AND DEVOPS Agile Software Development means more iterations. Being Agile is recognizing that predictions of development complexity along with user or market behavior are imprecise. Rather than trying to define requirements and software work based on abstract assumptions, it is better to release software early and often and be prepared to measure and analyze data to adjust execution and move faster. DevOps is a contraction of two words: Developers and Operations, used to indicate the organizational need to integrate them into a collaborative environment to achieve the goals of Agile Development as defined in the previous paragraph. There is no magic recipe for doing Agile Development. There are guidelines, best practices, and process frameworks but each project should design the process to fit its needs. The process should also be reviewed and adapted using feedback from the team after each iteration. As an organization attempts to make software development more Agile and to integrate it into operations, the need for automation of processes, measurement of performance, and usage patterns becomes obvious very quickly. With Agile, developers and IT operations can meet user expectations of availability and continuity of services. Daitan Group 2013 2
3 SOURCE CODE CONTROL AND BUILD AUTOMATION A team of software developers working together must have a consolidated source code repository with versioning and a set of scripts that automate the process of compiling code and generating the executable software. This is the baseline in the Software Development Process. The repository should keep everything needed to build the application including source code, test scripts, properties files, database schema, install scripts, third-party libraries, etc. The team should create guidelines and rules for branching, committing and merging, minimizing the friction of code integration. The automated build system should compile, link and do everything else needed to get the application to run correctly for a given target environment. Virtually every organization developing software today uses source code control, ranging from clientserver tools such as CVS or Subversion (Open Source), ClearCase or Perforce (commercial) to distributed architectures like Git, Bitkeeper, Mercurial (Hg). Build automation tools range from make scripts to more sophisticated tools like Maven and Ant. Daitan Group 2013 3
4 TEST AUTOMATION Rather than testing software manually, automation employs tools to run test cases without manual intervention and compare the output to what was expected. In traditional waterfall software development, test cases were initially run manually and then automated through scripting. There was usually a lack of coverage and lag between coding and test automation, which caused automated quality assurance of software an expensive and often unsuccessful proposition. Agile developers have been researching and experimenting with new approaches to development and automated testing of software. Test-driven development (TDD), where unit test code is written to define the functionality before the actual code is written is one popular technique. Other related techniques are Behavior Driven Development (BDD) and Design Driven Development (DDD). A QA automation framework sets the rules of automation of a specific product. It integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort. Examples of unit test tools include TestNG, xunit, Cucumber (BDD, Ruby), Jasmine (JavaScript). Selenium (Open Source, Web Interfaces), FITnesse (Open Source, Integrated tool), WindMill are examples of QA automation end-to-end tools. Daitan Group 2013 4
5 CONTINUOUS INTEGRATION Continuous Integration (CI) is the practice of accelerating commits and testing of code by software developers. In the past developers would work for weeks before trying to integrate their work to the main tree, in a CI environment developers check-in their work several times a day. A CI server monitors code check-ins by developers and, as often as practical, it checks-out the new code, builds the system and runs unit and integration tests. If there is any failure in the process, the CI server alerts the developers. A broken build gets the attention and top priority from the team. Keeping developers working in isolation without frequent integration can appear to be more efficient, but in practice, integration of code is extremely laborious and the source of most of the software defects in traditional waterfall development. Mandating frequent integration keeps developers from going in different directions for too long and forces the automation of the testing and integration processes. Benefits of Continuous Integration: CI does not eliminate software bugs, but it accelerates their discovery and keeps people from going in the wrong direction for too long. Developers pay more attention to quality and tests since CI failures are very visible and usually block their work if there is a broken build. The system is always in a state that can be released to users. That provides more flexibility to the business and enables the automation of the release process and delivery to users. Jenkins (Open Source), Apache Continuum (Open Source), Travis CI (Open Source), Bamboo (Commercial) are some examples of tools for Continuous Integration. Semaphore and BuildHive are examples of a new generation cloud-based CI frameworks. Daitan Group 2013 5
6 RELEASE AUTOMATION AND CONTINUOUS DELIVERY Once you implement Continuous Integration, it becomes possible to consider automating the release process. By having your CI system run release scripts after a build finishes, you can deploy the application to a live test server that everyone can look at and possibly automatically run postdeployment service tests. As this process matures, you can consider extending the process and automate it all the way to the enduser. Continuous Delivery (CD) is the practice and ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead. In a Continuous Delivery environment, code is compiled and packaged by a build server every time a change is committed to the source control repository. It is automatically tested before it can be marked as releasable. Developers are aware that any code commit may be released to customers at any point. Code branching might still be necessary in a complex system, but only for limited periods of time. The automation system lets changes to the system be released on a granular basis. Releasing features to a subset of users and being able to seamlessly roll back to a previous version becomes possible. The result is that there is no delay between enhancements and bug fixes being implemented and available to users. Daitan Group 2013 6
ABOUT DAITAN GROUP Daitan Group is a software development service provider with focus in Telecom, Unified Communications and real-time Cloud/Web Solutions. We partner with technology vendors to help them develop their next software solution. Be it by providing expert software engineers to extend your development team or leveraging our expertise in automating engineering processes and implementing tools and frameworks, Daitan Group accelerate your software development. Implementation of Software QA automation, Test- and Behavior-Driven Development (TDD, BDD) and work with tools like Bamboo, Jenkins, Jira, Fitnesse, Hammer, Ant, Maven, Continuum, Perforce, Empirix Hammer, and TestNG are just a few examples of our organizational experience in the area of software process agility. To know more about what Daitan can do for you, please visit http://daitangroup.com Daitan Group 2013 7