Continuous Testing with ElectricCommander. Electric Cloud, Inc. 2009

Transcription

1 Continuous Testing with ElectricCommander Electric Cloud, Inc. 2009

2 Continuous Testing with ElectricCommander Drive quality earlier in the development process with continuous testing. Large development organizations have increasingly been adopting select agile practices to enable faster development, create more reliable software, and reduce costs. Chief among these practices is providing developers early and frequent feedback from users, from other developers, and especially from the programming, build, and test tools. In the latter category, continuous integration and continuous testing are emerging as valuable techniques for identifying errors early so they can be corrected immediately by the developer who made the code changes. Most organizations that want to implement these practices face the challenge of scalability how to implement continuous integration and testing for code bases that contain tens of millions of lines of code, involve many teams of developers, or run on numerous targets and platforms. Electric Cloud specializes in tools purpose-built for projects of true enterprise scale. This white paper discusses ElectricCommander and how it uniquely delivers enterprise-level continuous integration with full support for all forms of continuous testing without requiring modification to existing infrastructure, test tools, or established practices. The Benefits of Early Defect Detection It is a well-established axiom in software development that the later a defect is detected, the more expensive it is to fix. Many sources claim, and many managers confirm, that the cost of remediation jumps exponentially as the software moves from one phase to the next: from requirements and design to coding, testing, and production. Research suggests the costs of defect repair are 1x during requirements and design, more than 10x during coding and testing, and 100x after release to production. These numbers, 1 which are sometimes referred to as the rule, probably understate the cost of fixing errors postproduction, because they do not reflect the unseen costs of lost productivity nor the lost sales resulting from customer-visible defects. The high cost of defects is one of the main drivers of the agile development movement, which holds that development organizations should release frequently, solicit constant feedback, and easily accommodate changes initiated by bug discovery or modifications of customer requirements. One of the key principles underlying this agile approach is test early, test often. Continuous Integration A leading best practice in support of this test often dictum is one that has gained considerable acceptance during the last few years: continuous integration. In its most basic form, continuous integration, or CI, rebuilds the software and runs unit tests every time code is checked into the main repository. The goal is to identify, as early as possible, coding problems that will break the build; that is, prevent it from completing or running satisfactorily. Build-breaking problems can then be resolved quickly and development can continue apace. 2.

3 Good CI software can determine whose code broke the build. If the CI server can identify the problem quickly, then the corrective steps can be taken while the developer is in immediate touch with the code changes and the rationale behind them. If, instead, the defective code is found a day or a week later, the developer is likely to be working on something else, and re-entering the defective code to correct the error will involve a longer ramp-up. In addition, by not having all the details at his fingertips, the developer is more likely to make changes that break other code or that lead to unexpected repercussions. This cost is compounded by the likelihood that other code has been layered on top of the defect, and so fixing the defective code can result in expensive changes to other modules and tests. CI implementations can run into problems of scale, however. Sites that have large code bases, large teams, or multi platform environments can find it difficult to run the build cycle sufficiently fast enough for the CI server to validate builds multiple times during the workday. This scenario is particularly true at sites where developers check in code frequently. Several means of addressing this issue are available today. The first is to use a build engine that can run large builds efficiently. Products such as ElectricCommander from Electric Cloud, for example, are capable of running builds in parallel, so that they can extract the maximum benefit from server pools, clouds, and other build resources. ElectricCommander enables sites to create highly customized builds and farm out parts of the process to individual servers, maximizing the benefits of parallelization within a single build. A second way that sites with large code bases can accelerate the CI cycle is through a technique called preflight. In preflight, the developer s code is not immediately checked into the main code base an action that triggers a new CI cycle. Rather, the code is kept in a sandbox and a smaller-scale build is performed there to obtain some initial validation. Frequently, this build is a subset of the full build sufficient to generate running code that can be unit tested without rebuilding sections of the code base that are unlikely to be touched by the changes. In this way, developers can quickly determine any immediate errors or unexpected impacts on related code before the code is checked in for the fuller CI build. This preflight capability is one of the distinguishing traits of enterprise-level CI servers, such as ElectricCommander. To deliver preflight, a CI package must understand sandboxing and must support highly customizable build scripting, so that the right builds can be triggered during this stage. Ideally, CI servers that deliver this benefit have automated build capabilities that minimize the amount of special coding required for preflight. Using CI with parallel builds and running preflights are agile techniques that permit developers to work on large code bases without the plodding and heaviness that is characteristic of the traditional waterfall model. The use of these techniques with an enterprise CI server such as ElectricCommander enables software projects to progress in ways that simply were not possible a few years ago. For example, a major Wall Street financial services company today uses ElectricCommander as a centralized build and test service, supporting the development work of 500 separate project teams spread across the globe. CI and preflight are the only way they can advance development on such vast projects. At leading-edge sites, the concept of early notification of defects is being enhanced by an emerging best practice: continuous testing. Continuous Testing Agile testing today starts with the developer, rather than with the QA department. Most modern development organizations require developers to write unit tests to exercise their code before they check it into the SCM system or run preflight tasks. These unit tests are small, fast tests that validate the basic function of the code and exercise its behavior with boundary conditions and other error-prone scenarios. 3.

4 Customer Story Intuit Attains Truly Continuous Integration Silicon Valley based Intuit, Inc., is known for its series of highly intuitive financial software products aimed at consumers and small and medium-sized businesses (SMBs). The company s products include TurboTax, QuickBooks, Quicken, and numerous other titles. Each product had a large code base that previously could not be revved frequently because builds took more than three hours which meant a build could rarely be run more than once a day. After searching for an effective solution, Intuit chose Electric Cloud products to automate its builds. It set up a build cluster consisting of 25 inexpensive x86-based servers on which the Electric Cloud software could leverage its parallel capabilities effectively. For more information on this deployment, see: downloads/ec-cs_intuit.pdf. Today, according to John Burt, senior manager of SCM at Intuit, the company runs its build integration truly continuously: With Electric Cloud, we compile and link every 30 minutes. And because of the quick turnaround on fixes, the costs of broken builds are a thing of the past. The company s current practice is to use the last good build of the day as the clean, fresh build from which developers begin work each morning. Due to the fast turnaround Electric Cloud technology delivers, this build contains all nonbreaking check-ins and includes advisories to developers whose pending code generates errors. As coding progresses, the developer builds up an increasingly large set of unit tests. Best practice suggests that all these tests should be run, rather than just a few of them, so that any unexpected effects on existing code are flagged immediately. In most sites that use CI, developers run unit tests either when they check code in or when they run preflight integration. However, some IT organizations encourage developers to run tests more often, even continuously. The rationale is that the collected tests serve as early warning mechanisms of unexpected repercussions of a code change. They are sensors that will fail when a change in the code detrimentally affects the operation of another, perhaps distant routine. By using continuous testing (CT), the developer can be alerted to specific problems as early as possible. At times, a developer fully expects new changes to break some existing code. At other times, however, he might not realize that changing a routine has a significant effect on a module whose presence is unknown to him. For example, a developer at a financial services firm might be implementing a new way of calculating customer deposit requirements for certain kinds of trades. Because he has CT enabled, he discovers that his code is affecting a formula that is mandated by certain government contracts and that cannot be changed without prior permission. At that point, the developer can stop the work and alert his supervisors, who can determine whether to continue or whether to stop until approval is sought and granted. If the developer had not used CT and had waited until end of day to run tests, the discovery would come after a full day had been invested greatly compounding the likely costs. As the preceding example shows, CT is an excellent mechanism for identifying in near real time the scope of modifications without the need for code check-in. 4.

5 Customer Story Automating Large-Scale Build and Test A publicly owned vendor of data warehouse appliances has created a sophisticated build and test system using ElectricCommander. It provides multiplatform builds in parallel with integrated unit testing, smoke testing, and UAT that run in a continuous integration cycle. The Electric Cloud software correlates code change lists to the entries in the defect tracking system, automatically updates the defects, and, optionally, closes them. By also integrating the build with a static code checker and a unit test database, the company can additionally identify non-error improvements in both code and tests. Tying test results to triggers for specific builds and by running these tasks in parallel enables the company to complete more builds and run more tests automatically than it ever could using manual approaches. This integration has resulted in direct benefits to the company s bottom line: Defects are identified and closed sooner and confidence in the code base has increased dramatically. In CT, various triggers set off a designated group of unit tests that generally includes all new tests a developer has written. If any tests fail, the developer is alerted through a variety of mechanisms. Typically, instant message or delivers the warning and includes specific information about the breaking test. ElectricCommander and Continuous Testing Most CT tools today are stand-alone packages and must be integrated with the development and testing workflow at the site. However, ElectricCommander includes CT in its core feature set. This means sites that already rely on Electric Cloud technology can simply integrate automated CT with their existing CI practices. See Figure 1. Figure 1. Errors from compilations and unit tests displayed in a project dashboard. Notice that 1514 compilation steps were performed in parallel on this build and that the errors can be logged directly to the defect-tracking system (here, JIRA from Atlassian Corporation). ElectricCommander provides the full range of CT capabilities; it is not limited to unit tests. In fact, it runs all tests be they unit tests, integration tests, system tests, UA tests, or even specialty tests such as black box and fuzz testing. All these can be run in the background on a continuous basis by ElectricCommander. 5.

6 ElectricCommander handles large test suites, as shown in Figure 2. Notice that a total of 2919 unit tests were run (in slightly more than 15 minutes), while 25 integration tests (shown in the top two bars) were also run. This means that a developer working on code would know every 15 minutes or so whether modifications were disrupting code in unexpected places in the project. And by using Electric-Commander to run more than 3,000 tests in every cycle, this assurance can be provided with a very high degree of confidence. Figure 2. Output from ElectricCommander showing the results of auto-mated test cycles that include thousands of unit tests. ElectricCommander is a build-test-deploy solution tailored to enterprise software organizations and ISVs that have complex development needs. Its stock in trade is the automation of the build and test segments of the application development lifecycle by accelerating these activities with parallel technologies that scale from workgroup code bases to trans-enterprise projects. Customer Story BioWare: Many Platforms, One Build Automatically BioWare is a gaming company that is wholly owned by Electronic Arts. Its games are known for stunning visuals that can run on numerous gaming and PC platforms. These capabilities are crucially important in the intensely competitive gaming market. Because of this competitiveness, small changes in time to market can make the difference between a game s success and failure. As a result, BioWare needed to perform multiple complex builds under intense time pressure. Unfortunately, as games got closer to release, the number of check-ins would increase significantly and the company found its CI server could not keep up. As a result, progress slowed down significantly just before release. To remedy this problem, BioWare installed ElectricCommander. This solution gave it centralized control of the build environment, which was deployed on a dedicated build host configured with three test systems. Now, a procedure syncs the source at check-ins and runs a build on every platform in parallel. After each platform s build completes, ElectricCommander runs unit tests followed by an in-game system test. The result, according to BioWare Lead Technical QA Manager Sam Johnson, is that ElectricCommander s CI capabilities have vastly reduced the number of broken builds. By cutting build times from 60 minutes to 15, the team can run more CI cycles and obtain a nearly 90 percent reduction in broken builds. Says Johnson: Because there are so few broken builds, the developers are able to really focus on what happened in a particular instance and fix it so that the next time the build goes even more smoothly. Particularly remarkable is that BioWare garnered these benefits without modifying the company s build or test procedures and without changing its SCM, building, or testing tools. This seamless integration was due to ElectricCommander s support for all leading development tools and packages and the nearly turnkey operation of its build and CI servers. 6.

7 Integrated Solution One of ElectricCommander s core strengths is the full integration of its capabilities with existing development tools. For example, it interacts directly with all mainstream SCM packages, build systems, and testing tools. It leverages this integration to deliver advanced and unique benefits. For example, ElectricCommander can recognize and monitor comments placed in change lists to refer to an entry in a defect-tracking system, and ElectricCommander can close the defect automatically when the designated tests complete successfully. ElectricCommander can perform builds on multiple platforms simultaneously and run tests on those platforms, so that IT organizations can validate that unit and integration tests were not only run, but run on every platform the product supports. Because of this high level of intelligence and integration, Electric Cloud is viewed as the most complete CI platform available today. Not only can it handle the full complement of CI responsibilities on large, complex projects, but it also offers preflight and CT support in any configuration. Results from CT can be sent to developers via , or displayed in their development environment, as illustrated in Figure 3. Figure 3. ElectricCommander results displayed inside Eclipse. Results can optionally be posted to real-time dashboards, from which developers and managers can monitor development progress and project health. These ElectricCommander dashboards are highly configurable to reflect the needs and preferences of the IT organization. In addition, Electric Cloud products have all been optimized to run procedures on virtual machines. These capabilities extend to cloud computing as well, where ElectricCommander is ideally suited for managing and driving testing storms on the cloud. Current users of Electric Cloud products use this feature to run builds and tests in parallel on local systems, remote systems, virtual machines, and clouds simultaneously and with no special configuration or complex set-up procedures. To learn more about how ElectricCommander, CI, and CT can improve your organization s development process, request a demo today at or go to the Electric Cloud website at Copyright 2009 Electric Cloud, Inc. All rights reserved. Electric Cloud, ElectricInsight, ElectricAccelerator and Electric Make are trademarks of Electric Cloud. Other company and product names may be trademarks of their respective owners. 1 First studied by B. W. Boehm in Software Engineering Economics (Prentice Hall, 1981) and confirmed by W. S. Humphreys in A Discipline of Software Engineering (Addison-Wesley, 1995). 7.