Better Automation Environments Matthew B. Doar March 2006 Peninsula Linux Users' Group (PenLUG)
What's a Toolsmith? A catch-all term for build guy, SCM engineer, and tools wizard It's very useful when your project has one... and it's what I enjoy doing 2 / 29
What is an Automation Environment? A tool that performs a variety of automated tasks Automated means no manual interaction Automation is the mark of people who are thinking about what they are doing 3 / 29
What should it do? Regularly check out the latest versions of a project's source files Make sure that the build works by building one or more of the products Run unit tests on the new build Create a build report and test report, and publish them somewhere convenient Notify the responsible people if there were problems with the build or tests 4 / 29
What else should it do? Generate the appropriate SCM label, and build labels or release labels Tag the source files with the SCM label Move the generated products to somewhere that other people can easily find them Create a list of bugs that may have been fixed in this build Add an entry about the release to the bug tracking system, and mark certain bugs as possibly fixed in this release Notify people that a new release is available What else? 5 / 29
When Should Builds Happen? On Change - whenever something happens, for example, a developer commits a change to the source code Scheduled - once an hour, or twice a day, or every night Continuous - whenever the previous build finishes The choice depends on how long a build takes, and how the project's developers like to commit their changes 6 / 29
When Should Builds Happen? Source: http://www.pragmaticautomation.com/ cgi-bin/pragauto.cgi/monitor/whatsyour- BuildFrequency.rdoc 7 / 29
When Do Builds Fail? Source: http://jroller.com/page/imeshev/ 20051015 8 / 29
Unhelpful Build and Test Reports Too little or too much information at once, usually because the intended audience of each part of the report is not clear Can't be conveniently archived or relocated, e.g. links break, formatting files have non-obvious locations. Color is used by itself to distinguish information, instead of as an enhancement, which can make printing out a test report useless Too hard to work out which source files were used for the builds and tests from the report 9 / 29
A Better Report Different people want different things from reports, so the report is broken up into different sections The product name and environment are clearly laid out, along with the date and time The failures are highlighted, and are made easy to refer to in other documents such as email between developers The people who are responsible for each build and test are identified, along with the names of people who have made changes since the last report Historical trends can be monitored The report uses relative links, so that the report's files can be relocated and still be useful 10 / 29
A Better Report (Continued) 11 / 29
A Better Report 12 / 29
A Better Report (Continued) 13 / 29
Choosing an Automation Environment Must support all the platforms and tools that your products are built with Must be easy to configure for simple tasks, and should be easy to extend for more complex tasks Should support parallel builds on multiple machines Should be able to handle a complex schedule to avoid backups, for example. Should have easy integration with other tools such as your SCM tool and bug tracking system Have to be able to store it in your SCM tool, just like the rest of your development environment Other factors? 14 / 29
Shell Scripts and Batch Files Simple to develop, but hard to use well and hard to maintain as they grow Need to detect when tasks are already running Debugging can be complex, especially if many other systems are involved Hard to coordinate cron jobs on multiple machines Generation of HTML reports using scripts is tedious 15 / 29
Tinderbox Open source Perl scripts from the Mozilla Foundation General purpose scripts on the build machines perform the work, and send build results and logs in formatted email to Tinderbox, which generates HTML reports The HTML reports have colors for each build state, and links to warnings and errors in the HTML versions of the build logs, and also to the project's commit history Not particularly easy to configure, or welldocumented, but it is robust 16 / 29
Tinderbox Source: http://tinderbox.mozilla.org/firefox 17 / 29
Tinderbox (Continued) 18 / 29
CruiseControl Popular open source continuous integration framework, Java-focused and integrated well with the Java build tools Ant and Maven Many different plugins available, e.g. one for each kind of SCM tool Notifications can be by email, an RSS feed, or even X10 Not as easy to run on multiple machines Used as the core tool in the Pragmatic Programming book Pragmatic Program Automation by Mike Clark 19 / 29
CruiseControl Source: http://cruisecontrol.sourceforge. net/reporting/jsp/buildresultsjspscreenshot.gif 20 / 29
CruiseControl (Continued) 21 / 29
Anthill Commercial build and release management tool from Urbancode ($2500), originally based around Ant but now supports make and shell scripts as well More web-based administration, and a bit less configuration than CruiseControl Multiple concurrent builds are supported 22 / 29
Anthill Source: http://www.urbancode.com/products/ anthillpro/proscreenshots.jsp 23 / 29
Parabuild From Viewtier, a commercial automated software build management server, around $350 Easy installation and configuration Many features, plus fine-grained control over users and groups Has an undocumented API for remote interaction? 24 / 29
Parabuild Source: http://www.viewtier.com/products/ parabuild/screenshots/ screen_shot_1_build_status_list.htm 25 / 29
Other Automation Environments Gump is Apache's continuous integration tool, written in Python, integrated with Ant and uses a particular model of what a project looks like. BuildForge - starts at $25K for a site license Xenofarm (http://xenofarm.tigris.org) is used to develop Python, as well as the newer BuildBot Andy Pols' beetlejuice (http://www.pols.co.uk/beetlejuice) A huge comparison of automation environments: http://damagecontrol.codehaus.org/continuous+int egration+server+feature+matrix 26 / 29
Some Drawbacks of Automation You may forget how it used to work manually Debugging a process that has been automated is a bit harder Need to monitor the effects of the automated tasks on other people, e.g. is the CVS server always overloaded just after the top of the hour? Easier to generate droppings that eventually fill up a disk somewhere 27 / 29
Summary If there are two steps to a process, someone will forget one of the steps, or perform them in the wrong order Careful automation of certain key tasks really will make your product better The only way to work with multiple platforms Public but impersonal messages can defuse tensions There are a few automation environments available, but none are perfect, so choose wisely 28 / 29
Book Plug Practical Development Environments Matthew B. Doar O'Reilly, September 2005 $39.95 US, ISBN: 0-596-00796-5 29 / 29