Are You Failing Fast Enough? Tips & tricks for a speedy build system

Transcription

1 Are You Failing Fast Enough? Tips & tricks for a speedy build system Sarah Goff-Dupont Atlassian Greetings! If you ve come to hear a few tips for making your build system run faster then you re in the right place! 1 Before we begin... this talk is designed for people who have a basic continuous integration system in place. If you and your team are already doing full continuous delivery, this talk might be too basic for you. So if you d like to go find another session to attend, I won t be offended. My name is Sarah Goff-Dupont, and I have a confession to make.

2 I work at Atlassian. And as you may have heard, we make a build server called Bamboo. Not only that, but I am the marketing manager for Bamboo. Which means my credibility with you just took a nose-dive. 2 But before that, I did spend about 4 years as a test automation engineer, working mostly in Java. So maybe that gives me a bit of my nerd-cred back. You didn't come here to listen to a sales pitch. So for the next 20 minutes we're going to take a look at things from the conceptual level. Most of the screenshots you'll see are indeed from Bamboo. And I ll also show examples from Hudson, Jenkins and TeamCity since those are also very popular tools. But the purpose of this talk is to share with you some tips and practices that we at Atlassian have found useful --and that you can implement as well, no matter if you use Bamboo, Hudson, Jenkins, or one of the other CI products out there.

3 Why Do We Care? W.I.P. = 3 Start IssueD W.I.P. = 3 W.I.P. = 2 W.I.P. = 2 Start IssueC Code C Push Get Build Results Code C Push Start IssueB Code B Code B Push Get Build Results Code B Push Get Build Results Code B Push Start IssueA Push Get Build Results Code A Push Get Build Results Code A Push Get Build Results Close IssueA 8am 12pm 4pm 8am 12pm 4pm 8am 12pm 4pm 8am 12pm 4pm 8am 12pm Day 1 Day 2 Day 3 Day 4 Day 5 First, let's talk about why you even care how fast your builds run. 3 IMO, it's all about getting feedback to the developers --like the results of automated tests or whether there were any compile errors. If it takes hours for a developer to find out whether their latest changes broke anything, they're going to make that commit, then move on to something else for a while, then find out there are more changes needed on the previous commit, then spend (waste) some time getting their head back into it... and repeat that cycle several times. By the time the original task is complete, a developer may have 2 or 3 additional tasks in motion. Multi-tasking is great for computers. But let's face it: we humans do our best, and most efficient work, when we are focused. So the goal of an efficient build system is to get feedback to developers fast enough that they don't have time to get wrapped up in a new task --and end up in a confusing mess like this.

4 Fast Feedback Promotes Efficiency What s in it for managers, product owners, and other stakeholders? Efficiency! Oh yeah, how? Limiting the amount of work in progress......allows developers to focus...decreases chances of locked files and time wasted on merges...increases chances of finding bugs while they re still cheap to fix Some of you are managers, product owners, etc. You might say "Well, I'm not a developer, so if I get my devs faster feedback, what's in it for me?" 4 In a word: efficiency. We write better code, and write it faster, when we're not switching back and forth between 3 or 4 different tasks. We don t drive safely while talking on the phone, and we don t code safely while working on 3 user stories simultaneously. Why? Because we re distracted. This leads to more bugs being introduced and more features implemented incorrectly --and then we have to waste time cleaning up the mess. Another consequence of juggling 3 or 4 tasks is that each task will extend out over many days. And depending on what your source control system looks like, you may have one developer locking up files that another developer needs to work on and creating a bottleneck. Even if files aren't locked down when you're working on them, if another developer needs to work in the same code, we'll have to spend additional time working through merging our changes. The third reason fast feedback increases efficiency is that code changes might introduce defects in the application. Those are easiest and cheapest to address when they are discovered right away. The work is still fresh in the developer's mind, and more code hasn't yet been layered on top of it. So as much as we were all inspired by the Cirque d Eclipse performers last night, leave the juggling to them.

5 Fast Feedback Promotes Efficiency W.I.P. = 2 Start IssueC Push Get Build Results Push Get Build IssueC ResultsClose Start IssueB Push Get Build Results Push Get Build Results Close IssueB Start IssueA Push Get Build Results Push Get Build Results Push Get Build Results Close IssueA 8am 12pm 4pm 8am 12pm 4pm 8am 12pm 4pm 8am 12pm 4pm 8am 12pm Day 1 Day 2 Day 3 Day 4 Day 5 5 If we can get our build systems to give feedback so quickly that devs have only 1 or 2 tasks in progress, we simply waste less time. Ultimately, the more we can streamline our development workflows, the more features we can build in an iteration. And the faster we can deliver value to our customers. Or to put it another way, when our workflows are IN-efficient, we loose time --and when we loose time, we loose customers.

6 If you re going to fail, fail fast! #ece2012 All of this is a really long way of saying that if your code is going to fail for any reason, we want to discover that failure as fast as possible so we can address it right away. 6

7 Terminate Hung Builds So now that you care, let's get on with it! 7 Probably the easiest and cheapest improvements you can make is to ensure hung builds don't sit there hanging for hours and hours, locking up build agents while your queue gets all clogged up.

8 Terminate Hung Builds Most of the popular build tools handle this in basically the same way. 8 You set a time limit for how long the build should be allowed to run, and if that threshold is exceeded, the build is stopped. With the commercial tools, this is a standard feature. WIth Jenkins and Hudson, you just need to install the Build Timeout plugin. If you take only one action based on this talk, set your system up to terminate hung builds. It doesn t cost anything extra, it's easy, it's effective.

9 Terminate Hung Builds Repo Give me free-flowing builds, or give me death! CI Server 9 So if we visualize a build system, it might look something like this.

10 Automate Deploys So that was the easiest thing you can do. Next, let s jump right to probably the hardest thing: automate your deploys. 10

11 Automate Deploys Impact from automating deploys goes beyond fast feedback Encourages standardization across environments and discourages server drift Ops engineers can focus on improvements and innovations Prioritize low-level environments that receive frequent deploys 11 Why? Because to get fast feedback you should build with every commit, and test each build so any problems are found right away when they re easiest to fix. Now, unit tests alone won t find all your bugs because most bugs live in the integration points between different parts of your application. So you need to run some integration, API, and possibly UI-level tests. This usually means deploying the build to an environment. But to deploy each build by hand would require at least one person dedicated to it full time. So you automate it. Automating deploys IS an up-front investment in staff time, but it pays off. See, automated deploys themselves aren't all that hard to accomplish. It's the prerequisites: standardizing your environments is the biggest thing. You can't do auto deploys effectively if your servers are like snowflakes --if each one is a little different. Or from the glass-half-full perspective, automating deploys sort of forces you to eat your vegetables and keep your environments from drifting. And the more deploys you automate, the more your Ops engineers have bandwidth to work on more interesting things like scalability projects, production monitoring, increasing uptime, etc. Now, in my opinion, you get the most impact by automating deploys to your lowest-level environment, probably your dev or QA environment. This is where you re going to deploy most frequently. So make that those the priority, then replicate it upwards through your environments.

12 Automate Deploys Repo CI Server So with automated deploys, now our build system is lookin something like this. 12

13 Pass The Artifact, Please! If and when you have deploys automated, the next step is to pass a pre-packaged build to that deploy step rather than re-building a second (or third) time. 13

14 Pass Artifacts What is an artifact? JARs, WARs, TARs, ZIPs javadocs, rdocs cobertura.ser, checkstyle.xml, etc. Artifact passing saves time artifact passing for the win! Compile only once Check out code only once Store artifacts in Archiva, NexusPro, Artifactory, temp file server, or in the CI tool So, when I refer to an artifact, that of course includes the JARs, WARs and TARs that we re accustomed to with compiled languages. 14 But an artifact is really anything that you need manipulate or operate on in your builds: a ZIP file containing your source code is an artifact - a Checkstyle or Cobertura report is an artifact... for purposes of this discussion, we can keep the definition very broad. Now, when we started passing artifacts at Atlassian instead of rebuilding for each environment, we cut about 40 minutes off the time it takes to run a full test cycle. Not only do you save time on the compile step, but you skip the additional checkout step too. That can be a huge win if your code base is large and/or your network is prone to slowness. You can upload artifacts to external repositories like the ones listed here, if you need to. For most use cases, your CI tool will just handle it for you.

15 Pass Artifacts 15 Here s how it works: In the configs of the job that produces the artifact, you tell the tool where on the server to find it, and the file name. Then in the consumer job, you declare the artifact as a dependency, and tell the tool where to place it so the deploy step can find it. There are slight differences from tool to tool, but they all work in basically this way. The tool s documentation is your friend.

16 Pass Artifacts What else is in it for me? Confidence! Oh yeah, how? Deploy the build you tested No un-tested code sneaks in No file corruption There are other benefits to artifact passing as well: confidence being the biggest one. 16 When you build an artifact once, then pass it up through all your environments, you know with certainty that the build you are about to release into the wild is the exact same build you worked so hard to vet in your pre-production environments. No worries about whether it was built from the right revision number, and no opportunities for a network hiccup to corrupt files as they're being checked out from the repo. Give your build one last check on Staging, hit the 'deploy to prod' button, and head out for a beer.

17 Pass Artifacts Repo CI Server Let s check in on our build system... lookin pretty good. 17 But I think we can do even more. So, what if you don't have automated deploys? Is killing hung builds the only thing you can do to tighten the feedback loop?

18 Run Tests In Parallel 18 Thankfully, no. Running your automated tests in parallel can make a huge difference. In fact, it s probably the most effective way to speed up your builds.

19 Parallelize Tests Prioritize parallelization of longer-running tests Break the test suite into batches, and run each as a separate job You are limited only by the number of build agents you have 19 First, let me say that this is most effective when applied to integration-, API-, and UI-level tests. Unit tests run really fast anyway, so your biggest return on investment is working with tests that involve calls across different modules of the application or across a network. Parallelizing tests is done by dividing your test suite into batches and running each batch as it's own job. All the major CI servers are capable of running jobs in parallel, so long as you have build agents available to do so.

20 Parallelize Tests I want to share my personal favorite way to split tests into batches: with groups. 20 Groups are a feature of several testing frameworks: TestNG (java & Ruby), Proboscis (python), and PHP Unit to name a few I ll use Java & TestNG as my example Inside annotation, you can specify one or more groups that the test or class belongs to. The groups are arbitrary, so you can slice n' dice your collection based on functional area of the application, level of the technology stack, or even whether or not the test is currently expected to pass.

21 Parallelize Tests So back in your CI tool, you set up jobs based on groups. 21 When it's time to run tests, you pass in the group name as a parameter, and only the tests belonging to that group will run in that job. Doesn't matter if the tests in that group are distributed across classes or packages. If you're executing tests via Maven, Gradle or Ant, this is pretty trivial.

22 Parallelize Tests You can take this idea of parallel test jobs about as far as you want. 22 As you can see here, some of our CI builds for JIRA have over 20 jobs running in parallel The next thing you ll want to do is aggregate the results from your testing batches so you have one nice easy number to look at. Bamboo, TeamCity, and most of the other commercial CI tools do it for you automatically. Jenkins and Hudson do not... get what you pay for?

23 Parallelize Tests Repo CI Server So back in our build system Hey, now we re getting somewhere!!

24 Create Multiple Feedback Loops Last on our journey today is the idea of dividing your build & testing cycles into tiers, or loops, that run at strategic intervals. 24 Now, what the heck are loops and what do I mean by "strategic intervals"?

25 Create Multiple Feedback Loops Let's say that your entire build & test phase takes 3 hours to run through. That's a long time for a developer to wait before getting any results back about their commit. 25 And let's say that you have 20 committers, and between them, your repository sees an average of 5 commits per hour. Unless you have an army of build agents, your queue is going to clog up FAST, making the wait time even longer.

26 Create Multiple Feedback Loops Repo CI Server So try this: 26 First, create one build/job/pipeline that gets triggered by each commit, and does just 4 things: source code checkout, unit tests, artifact packaging, and storing the artifact somewhere where downstream loops can get at it. That loop might take minutes to execute. Long enough to document the changes made and get a cup of coffee before you get the results but not long enough to get going on another user story. That becomes your inner loop. It's a quick sanity check for your developers.

27 Create Multiple Feedback Loops Repo CI Server Test Env. Then, create an outer-loop that deploys your build to a test environment and executes the longer-running integration and UI-level tests. 27 Ideally, you'll pass this job the artifact you packaged up in the inner-loop. Otherwise, you can recompile as your first step. Run that a few times each day, maybe 3 times during working hours. That keeps the overall load on your build system low and keeps your queue from getting backed up, and it's still frequent enough for there to be a small number of changes between each run. So when new failures pop up, you can still trace them to the offending commit pretty quickly.

28 Create Multiple Feedback Loops Repo CI Server Test Env. For extra credit, some projects will benefit from a nightly build that runs your entire test suite against multiple platforms. 28 Atlassian does this with most of it's products. We deploy builds of JIRA, for example, to a linux box that uses Oracle as the DB, a linux box that uses MySQL, a windows box using Postgress, etc, etc... there are about 8 combinations of OS, DB and browser that our products get tested against. Platform-specific bugs are fairly rare, and are almost never show-stoppers. So running this gauntlet once a day is probably enough. And it's strategic to run it at night when your devs aren't making commits and your build agents would otherwise be sitting idle. Now, none of these ideas are silver bullets by themselves. Each of you will cherry-pick the combination of concepts that are best suited for your projects, and apply them. And you won't get it perfect the first time. So it's important to measure and track your progress so you know exactly what's working and what not.

29 Know Where You re Starting From! 29 Fortunately, any CI tool can generate reports on build duration. Bamboo also has native support for reporting how long builds are waiting in the queue before they're picked up. Get to know these reports before you start making any changes so you know where your baseline is.

30 Know Where You re Starting From! There are also plugins that can help you out. 30 Jenkins offers the Global Build Stats plugin which lets you refine your reports to look at, say, average build time for only passing builds, or only failed builds. Jenkins also has the Wasted Minutes plugin which tallies the total amount of time wasted by all builds in the queue Similarly, TeamCity has the Queue Statistics Report, and Bamboo offers a queue report out of the box. Bamboo also has the Build Times plugin that graphs the duration of each step and identifies bottlenecks in your pipeline These are all useful plugins, and they don t cost anything extra. So take advantage.

31 Know Where You re Starting From! x3 x6 31 Because you might feel like it's good enough to have a gut feeling that your build times are improving. But when you present all your hard work to management and tell them you deserve a raise, they're gonna want to see some stats.

32 Q & A Sarah Goff-Dupont #ece At this point, I'm going to stop so we have time for Q&A. I've had a great time. I hope you've enjoyed this talk, and gotten a little bit of inspiration for improvements to try out in your build systems. (CLICK) Here is my contact information. If you d like a copy of the slide deck from today, I m happy to share it with you. And if you enjoyed this talk, feel free to send a tweet about it with the #ece2012 hash tag --that ll help me persuade Christian and the selection committee that I should come back next year ;o) And with that, let s open it up for questions.