1 Title: Continuous Delivery and Continuous Integration Conference: 13 th Annual Software Testing Conference 2013 Author: Tanvi Dharmarha Email: tbajajdh@adobe.com Organization Name: Adobe Systems Inc Address: I-1A, Sector 25A, Noida, UP -201301
2 I Abstract Conventional development process involves conceptualization of idea through extensive research, building algorithms, implementing the software features, testing and rolling out software in phased manner with releases of Alpha, Beta, Release Candidates and production. One of the biggest drawbacks of this development methodology is the time to market. It just takes too long to conceptualize an idea and get the fruits of an idea onto a customer s desk and more often than not by the time an idea reaches the customer either it is too obsolete or user preferences have changed and the benefits of the idea/feature are never realized. Every concept or idea has an expiry and it is extremely important that the idea is delivered quickly, the longer it takes to push an idea to the customer the less engaging the concept is. Smaller the turnaround cycle, more engaging is the experience for the end user. Customers love to see software evolve, not as a big bang but in small increments as they can experience more of the incoming changes rather than a whole suite of features, which is a bit too much for an individual to try out. The goal is to reduce time to market. Through this paper we will learn the processes and practices provided by Continuous Integration and Continuous Delivery which will considerably improve time to market and ensure constant delivery of valuable software to customers. II Status quo Inefficiency & increased time to market Most software organizations invest 4-6 months in customer survey and research trying to figure out what customers want. Around 20-30 days are spent in requirement elicitation and creating an SRS (Software Requirement Specification). The design and implementation can take anywhere between 3 weeks to 2 months depending on the complexity of the feature request. And finally comes the most time consuming part of deployment and testing. With applications getting more and more complex with multiple servers, components, templates, system tools, deployment and testing of the whole system has become a very tedious and time consuming process. Numerous setup and configuration instructions have to be applied to test software and these are commonly prone to human error. Thus isolating a misconfiguration in the whole package is another huge challenge. Repetitive deployments and retesting the whole software after minor fixes increases certification time considerably from a planned 20 days to 2 months. Certification also iterates in a phased manner, from QA, Alpha and Beta before the feature is actually released to the customer. The release rollout of the feature takes anywhere between 6-9 months since its conception. With technology pacing and competitive market conditions, we cannot afford to have such inefficiencies and increased time to market. Strategies need to be adopted to reduce these inefficiencies. III Goal Constantly deliver valuable software to customers. Achievable by delivering incremental software enhancements efficiently and reliably with small time to market and capability to roll back changes without affecting the subscriber base. One way to achieve this is to develop in an agile fashion, be ready to put experimental features in mature software and provide flexibility so that the feature can be exposed or hidden from a customer base (selective absorption of features). Reliability The roll out of experimental features brings us to another important aspect of delivery cycle that is reliability. We do not want to blow up a piece of software working for a majority of customers such that a select group of customers can try out an experimental feature. One of the ways to achieve this would be to deliver new features behind a feature toggle. Feature toggle can be seen as a configuration parameter which hides the feature from majority of customers and exposes the feature to a select group of customers. The software continues to behave as is for a majority of customers and only a select group of customers are exposed to the new feature via the feature toggle. This select group of users basically is the tier 1 customer that is passionate about software improvements, is loyal to product and is willing to accept minor flaws in new features. This select tier 1 group forms the early adoption
3 customer base that can provide valuable feedback on the usefulness of a feature. Things that improve the reliability of new feature roll out are - stable and representative test environments - good tests at various levels - fast feedback from tests within development setup, extreme control on environments on which the software gets deployed - software should be deployable / installable with very little configuration and with minimum non-invasive input from the person deploying the software thereby reducing the complexity of configurability of software - automation of deployment pipeline - roll out of software in tiers - ease of roll back by implementing a switch or a feature toggle which can disable a feature in run time with minimum down time so that customer experience for the early adopters is not jeopardized Roll back and willingness to drop features It happens very often that an idea which may sound valuable in principle may not have desired impact on the customer base. If we build up an ecosystem where we can put an idea onto customers desk in reasonable time and validate the usefulness we should also be ready to scrap ideas if they don t deliver on the expectations. This makes our software more manageable and lean with active features and not features which are present because of legacy reasons. This is a change in mindset for the developers where the affinity is not merely on the work that is done but also on the value that a work adds and an acceptability that things cannot turn out as expected. Controls should be setup which should be able to generate the matrices for the usefulness of a feature. Observing user behavior when interacting with a feature can give us an idea of usefulness and acceptance of a feature in general and can determine the features place in the software.
4 IV New Technology Offering Continuous Integration and Continuous Delivery For an organization to incorporate CICD into the software development life cycle, they must adhere to the following: Development branches & Feature toggles - A lot of companies have demonstrated that if all the developers in the company work on the same branch of software stack, the development process is synergized and homogenous - Working on the same branch of trunk could be a huge ask as we have to make sure that trunk is always stable and is constantly in releasable form. This means that every check-in that a developer makes does not compromise the stability of a release. - This also introduces a need to be able to hide a partial implementation from the customer since the partial implementations are also part of the regular release. This can be achieved by hiding the implementation behind a configuration parameter (feature toggle) and exposing the feature to every one or a group or a tier based on the comfort of the person or team developing the feature. Build Packaging - Builds must be packaged as SNAPSHOTs and uploaded in the snapshots repository. Each incoming build with fixes should replace the snapshot till a stable build arises. - Stable builds should be marked as Releases; should be versioned and stored in the Release Repository for distribution. Test & Test Environment - Tests should not be the sole responsibility of QA team; Team should be designed to be crossfunctional where a team is self sufficient on what it delivers along with the responsibility of the quality of tests. Tests, be it unit, component, integration or acceptance tests should be always green and should be a good representative of the desired behavior from a piece of functionality added to the software. - Goal should be to keep the tests green all the time
- Tests should have the code quality as much as possible as any production software and all the principle of development should be practiced when developing test framework or writing tests - Test framework should be developed with the view that writing tests should be a fun experience rather than being a burden or mundane task for the team developing a feature. - Tests should be written such that they should be able to answer questions about a feature and should be written with a critical view on the software. Tests should not be just validation of feature but should be written with a goal of exploring the usability of the software. - Unit tests should test the desired functionality of a class - Component tests should test the interaction of feature with other features which may interact with the new feature - Integration tests should be written with the goal of making sure that the feature in its entirety fits with the software and does not blow up the software - Acceptance tests should demonstrate that expected user experience can be achieved from the software with the addition of the new feature - Achieving a common framework from doing all these levels of testing has an additional advantage of writing tests with ease so the test framework should be designed in a way that it should be equally easy to write an acceptance test and a unit test and no additional skills are required to write one kind of test or the other - Tests should be fast, there are organizations which can run tens-of-thousands of tests in matter of seconds. This gives a very quick feedback in development and broken tests if identified early make it easy to locate the changes which caused a failure. - Tests can be treated as a source of documentation and tests should be able to explain how a feature is supposed to behave in various circumstances - A build management system should be carefully chosen which can run tests on each commit, a commit should not be made with failing tests - Test runner hosts should represent customer environment especially when running integration and acceptance tests, this makes sure that we do not have bug slip ups for identified scenarios because a particular flavor of OS or configuration was not tested - Minimize MANUAL tests, if we feel a need to do manual testing, write an automated test which replicates the manual steps it is achievable and saves us from regression bugs as a test once automated makes sure that we do not miss a scenario 5
6 Configuration - systems should have least amount of configuration required from the person deploying / installing the system - software should come packaged with all dependencies and on a supported platform it should be deployable with only a few simple command(s) and not many configurations - software should be tested on all supported platform in an automated fashion - Software should be continuously deployed and checked for configuration consistency alongside development and this process should not be a disjointed one. Deployment Pipeline - Gives visibility into the production readiness of the applications by giving feedback on every change to the system - Builds, Tools and other prerequisites software should be deployed in an automated fashion on the certification environments (dev/qa/stage). - Any reds (failures) in the deployment pipeline should raise alarms. Rollout tiers - Almost all the social networking companies and some product companies use the concept of tier rollouts where a feature is first exposed to a select group of customers and on successful acceptance by those customers the feature is made available to general public - The select group is dynamic and can be chosen based on the relevance of a feature. A particular feature may be more relevant for a teenager and thus a company which has information about the age group of its customers maybe inclined to first expose the feature to a particular age group and then asses the value that the feature created. - Added advantage of tier roll out is that if something goes wrong not every one is affected by the bug but only the people in that tier are affected. - To visualize tiers, consider we have a stable branch V which is rolled out every where, then a team decides to introduce a new feature make a release V+, the V+ could be rolled out to only a select group of deployments and thus deployments become tier 1 deployment. If successful, the V+ get released to another set of deployments called tier 2 and the features is promoted eventually to all the tiers. - The tiers have to be designed such that most sensitive of the customers to a particular area of the software are least affected.
7 Snapshot Repo Fetch Dependencies from Repo Publish libraries to Repo Software Trunk Acceptance Integration Component Unit Tests Acceptance Integration Component Unit Tests Commit Commit Configuration Trunk Publish update libraries to Repo System Environment Trunk Fig 3: Snapshot Releases (Non Public)
8 Release Repo Publish libraries to Release Repo Publish libraries to Release Repo Fetch Dependencies from Snapshot Repo Acceptance Integration Component Unit Tests Acceptance Integration Component Unit Tests Snapshot Repo Trigger Release Trigger Release Fig 4: Public Releases
9 Tools - Subversion system (SVN/Git/CVS/Perforce/VSS) - Build tools (maven, ant) - Building Packaging (Nexus/Seli using Rest Apis) - Build management system (TeamCity / Hudson) - Build runner systems (agents) (customized based on platforms on which the software would run) - Configuration management systems: standardized environment setup on which the system can be deployed and managed by some kind of a configuration management system (Puppet, Salt) - Feature / bug tracking system (jira / Watson/Bugzilla) - Release management (Plutora/ URelease/Jira) - Test frameworks (JUnit / NUnit / Jmeter / QTP / Rational Robo, etc) - Knowledge base (api docs / man docs / technical docs) - Production machine monitoring systems (tools provided by hardware vendors) - Usage tracking of features: systems built to gather productions usage statistics V Conclusion The mantra behind Continuous Integration and Continuous Delivery is to build small, review regularly and deploy quality software to turn your release into a business advantage. VI Other desired advantages Time & Cost Saving: Apart from the increased time to market, incremental development helps in finding bugs easily and early in the cycle. Bug isolation and fixing is quicker as only a small change has resulted in that bug. Adopting of Continuous Integration and Continuous Delivery processes can reduce implementation time by 15% and deployment and testing time considerably by 55-60%. VII Acceptable with existing Business Models /Organizations Products and services environments where ideas have to be tried out soon and rolled out require CICD practices so that they can stay ahead in competition. Some examples of successful CICD implementation include 1. Social networking sites like Facebook deploys multiple times a day with bug fixes, new features, and site improvements 2. Image and video hosting websites like Flickr deploy 8-10 times a day with home redesigning, network traffic fixes and other improvements. 3. High Frequency Trading Organizations deploy every alternate day with various quantitative configurations for price predictions 4. Travel and Tourism sites like Booking.com, Yatra and MakeMyTrip deploy multiple times a day with new deals, promotions, bug fixes VIII References http://continuousdelivery.com/ http://www.thoughtworks.com/continuous-integration http://www.quora.com/how-does-facebook-site-release-deploy-process-work http://agile.dzone.com/books/continuous-delivery-free http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
IX Authors Biography The author, Tanvi Dharmarha, is working as a Lead Software Engineer with Adobe Systems Inc and is leading the quality team for Anti-Piracy initiatives at Adobe. She holds an Engineering Degree in Informational Technology. Prior to Adobe she has worked with an online Trading Company, International Marketmakers Combination, based out of Amsterdam. 10