Zero-Touch Drupal Deployment Whitepaper Date 25th October 2011 Document Number MIG5-WP-D-004 Revision 01 1
Table of Contents Preamble The concept Version control Consistency breeds abstraction Automation The Technology Git Jenkins Fabric The Process Branches Precautions Going live Notifications How the Zero-touch experience changes things Further reading 2
Preamble It s become a standard for agencies developing web applications for clients to adopt a development workflow to apply changes to that app over its lifecycle. These workflows typically transition a change starting from a development environment, through to Q/A, then staged for client approval, and finally deployed to the live or production environment. The purpose of such workflows is, of course, to maintain a sane audit trail of change, reduce risk and liability, and be able to roll back changes easily. For these reasons, it s no surprise many development teams use version control software to manage a revision history of the application. However, more often than not, the actual processes of applying those changes in the version control system to the various environments through this workflow, are performed manually, dragging everything back to the same pitfalls related to human error, loss of time, and low incentive to commit early and often. mig5 believes this problem has been accepted as the way of things by most agencies. mig5 also believes this should no longer be a problem. A sysadmin s mantra is to automate laborious, mundane, and human error prone tasks so that they no longer throw roadblocks and allow people to get on with what s important. This whitepaper describes the methodologies mig5 has produced and implemented for a wide range of existing clients to apply the sysadmin mantra to Drupal site development and deployment through the dev/stage/live process. This methodology is called Zero-touch deployment. 3
The concept There are several key components or concepts that are required to make Zerotouch deployment work successfully. Version control It is absolutely crucial to keep a codebase in a version control system 1. A VCS provides an audit trail showing who made a change to the code and why. Most VCS, by maintaining internal databases tracking those changes, also provide convenient methods for reverting or undoing changes, if something goes wrong. One of the VCS solutions that is gaining increasing popularity due to its enormous power and flexibility, is Git 2. Originally authored by Linus Torvalds 3, creator of the Linux kernel 4, Git is a type of VCS known as distributed - that is, it does not require a central server that holds all the data. With Git, it is possible to commit changes to an internal database on your local system, because every Git repository considers itself a fully-fledged database that can optionally be synced with other copies of the repository located remotely. Git is fast, optimised for certain common version control operations such as maintaining separate diverging branches of a codebase, advanced capabilities for merging changes from different remote copies of repositories and branches, and more. 1 http://en.wikipedia.org/wiki/revision_control 2 http://git-scm.com/ 3 http://en.wikipedia.org/wiki/linus_torvalds 4 http://en.wikipedia.org/wiki/linux_kernel 4
If you d like to learn how to use Git, how it differs from other version control systems, and the gritty technical details on how it works, the Pro Git book 5 is highly recommended. Consistency breeds abstraction Another important concept when attempting to achieve zero-touch deployment, is to be consistent with your approach across different projects. If you re a Drupal agency, it is highly unlikely that you re building just one application - you probably have a range of different clients and different websites to build. For this reason, it s only natural to achieve zero-touch for all your existing projects, and more importantly, all future projects! The faster the process can be made, the more time you have to take on more work and make more money. The zero-touch tools mig5 builds and implements, are designed to be as highly abstracted as possible, in order to be applied to whatever project / server you re working on. For this reason, consistency is key, so that the Zero-Touch tools can afford to make some assumptions about how you work. Consistency means everything from: using the same version of operating system versions and software in each environment (dev, stage, live) using the same naming standards for repositories and branches using the same workflow (from dev to Q/A, to stage, to live) other techniques to keep consistent (database naming conventions etc) Automation The ultimate aim of Zero-touch is to automate away the process of applying a change that a developer has committed to a repository in his or her development environment, to the respective remote environment (Q/A, stage or production). 5 http://progit.org/book/ 5
The Zero-touch tools eliminate the need for a developer or release manager to, for example, login to a remote staging server via SSH and run a git pull or a custom Bash script to apply the change, adjust HTTP virtual host configurations or manually perform database modifications. How often did you do such a thing and forget to make a backup first, or perform your git pull on the live codebase and find it either broke unexpectedly, or caused a conflict because your co-developer make an on-the-fly change? Zero-touch puts the past behind you and makes deployment feel safe again and even fun! The Technology We ve talked about the concepts that help achieve Zero-touch. Now let s look at the actual tools mig5 implements to do the job. Git We ve discussed Git already as the version control system of choice. It is possible, however, to adapt the Zero-touch tools to your VCS of choice: be that Subversion, Bazaar, Mercurial or whatever you prefer. Jenkins Jenkins 6 is a Continuous Integration server, most commonly used for running test suites against codebases or applications. However, more generally it is a reactive tool that can run any arbitrary number of tasks either locally or on remote systems via SSH or other means of communication. Jenkins ability to recognise success vs failure when running tasks makes it important for Zero-touch, because it can help identify a problem and safely back out of what might ve been a disastrous change - especially to a live environment. 6 http://jenkins-ci.org/ 6
In Zero-touch, Jenkins is the gatekeeper that detects changes that occur to a Git repository (triggered from a developer s commit and push), and sends the instructions to make change occur on the remote environments. Fabric Fabric 7 is a Python API for running arbitrary SSH commands and deployments on local and remote hosts. It is very similar to Capistrano 8, a Ruby tool that provides commands for users to deploy over SSH. In Zero-touch, a suite of tasks exist in a configuration file or Fabfile that fetch changes, backup existing systems, and apply changes in the various environments, in an important order or chain of events. It is the Fabfile tasks that Jenkins executes when a change has occurred in Git, activating those remote procedures. 7 http://fabfile.org/ 8 https://github.com/capistrano/capistrano 7
The Process Now that we ve looked at the concepts and the tools used, you re probably starting to wonder What actually *happens* during a deployment? Branches As already described, the procedure begins with a developer pushing a change to git. In Zero-touch methodology, several branches exist in the git repository, labeled so as to be recognised for the different environments to deploy to. master branch is the core branch that developers use to commit changes from 8
their development environment. Developers can maintain their own branches or use feature or hotfix branches to work on specific changes, but should ultimately merge those changes into the master branch. The Q/A or testing environment is tracked by Jenkins by watching the master branch, effectively collating changes from a possibly multi-developer team into a version of the site that reflects all those changes. This is a good opportunity for your Q/A engineers/testers to review what s happening and if the changes reflect what s been asked for / fixes what bugs have been reported. When the changes as seen in the Q/A environment are looking good, they should be merged into the stage branch in Git. This triggers a deployment to the staging site by Jenkins, whereby the client can be asked to sign-off on the change if they are happy with it. Once approved, that change is then carried on to the live or prod branch in git. Jenkins once again detects this change, and applies the change to the production environment, taking the change live. Precautions In each step or deployment, the site and critical components such as the database are backed up first before anything happens. The backup is labeled with a recognisable name such as $repo_prior_to_build_n where N is the build number or identifier in Jenkins. In the (unlikely) event of an unexpected big disaster requiring manual intervention to recover from, this allows an engineer to quickly identify exactly which backup should be restored from: if build 47 broke everything, then the prior_to_build_47 backup is probably what should be used to restore from. In Zero-touch, there is no pristine live copy of a site that gets routine git pull s or updates applied. Even with a version control system, this can be unpredictable and difficult to revert automatically. Instead, every time a deployment occurs, an entire copy of the codebase is checked out fresh, in a unique directory (again reflecting the build identifier), 9
separate from the current live or previous codebase. A symlink is used to point to whatever is the latest build. The HTTP vhost configuration considers this symlink to be the real live path or DocumentRoot 9 in Apache terminology. In this way, the site is always served up using whatever codebase the symlink currently points to. Going live If all tests pass and everything is looking good with a new deployment, that symlink is altered to point to the new build. This is what takes the change live, without having to alter the HTTP vhost configuration. If something unexpected occurred, it is very easy to roll back the change by simply setting the symlink to point back to the previous build. A Fabric task exists to retain a configurable number of old builds in case this is necessary. Notifications Zero-touch takes advantage of Jenkins existing Notification API, which can be configured to alert personnel to a failed build via a variety of methods (e-mail, IRC 10 bot channel announcement, iphone push notification using Notifo 11, XMPP etc). Typically a successful deployment will be announced quietly, such as via an IRC announcement, whilst noisier alerts such as phone notifications and e-mails will be used in the event of a failed build, to get attention quickly. Because Jenkins is tracking changes to a git repository, it is even capable of guessing, from the commit author metadata, the culprit that broke the build, and able to notify that specific person (based on the assumption that if that developer 9 http://httpd.apache.org/docs/2.0/mod/core.html#documentroot 10 http://en.wikipedia.org/wiki/internet_relay_chat 11 http://notifo.com/ 10
pushed a change, they have knowledge of what change was meant to occur, are probably still at the office watching it occur, and able to respond reasonably quickly to fix the issue). 11
How the Zero-touch experience changes things Experience implementing Zero-touch for various Drupal agencies has identified numerous positive effects that the methodology has had for those agencies and development teams. These are the top 10 examples we ve seen. 1. Deployments start to occur more frequently with smaller changesets, reducing the effect of errors and risks associated 2. Audit trails become easier to track what changes have occurred 3. Developers become more confident with pushing changes 4. Developers take more ownership of their changes 5. Less potential for data loss occurs when more frequent pushing to git is occurring, instead of developers hanging on to their local copies 6. It is faster to work on multiple projects if they are all using Zero-touch 7. Less fatigue from developers who can concentrate on development and forget about deployment 8. More attention is paid to fixing problems before they reach production, because failed builds can t help but be noticed 9. Communication improves during a project 10.Security can be improved/tightened (developers don t all need SSH access or sudo, just a Jenkins user for remote tasks to work) Is this sounding like a big sigh of relief to you? Zero-touch only brings benefits and usually is straightforward to setup and even integrate into existing environments. Contact mig5 12 today to see if it s something you can take advantage of too! 12 http://mig5.net/contact 12
Further reading mig5 and others (clients) have written about Zero-touch elsewhere too: Zero-touch Drupal deployment with Jenkins, Aegir, Git, Fabric and Drush http://mig5.net/node/342 How CodeEnigma does hosting http://www.codeenigma.com/blog/codeenigma-drupal-linux-hosting One-touch provisioning and auto-monitoring of new servers http://mig5.net/node/340 Test your Drupal distro: Building a continuous integration server with Aegir and Jenkins http://mig5.net/content/test-your-drupal-distro-building-continuous-integration-serveraegir-jenkins Continuous Integration (Presentation at DrupalCamp Toulouse with CodeEnigma, November 2011) http://toulouse2011.drupalcamp.fr/ 13