Whitepaper: Implementing Agile in a Waterfall World introduction Are we caught in a Waterfall world? Statistics demonstrating the high failure rate of software development suggests we might be. Yet despite increasing evidence as to the limitations of Waterfall, many organisations still seem stuck in their Waterfall world, and furthermore software development the problems 1 80-90% of software does not meet performance goals find it difficult to move to a more Agile landscape. To understand why organisations are hesitant to change and sometimes struggle it is important to understand the principles of each methodology and their impact on the organisation. Swapping development methodologies is not simple and like any change requires the right approach by management. 80% of systems are late and over-budget 40% of developments fail 10-20% of systems meet their success criteria In this whitepaper we will explore the principles, problems, and promises of each methodology, as well as providing four important ways and a range of simple and pragmatic tips that can assist organisations in their transition to Agile. February 2009 1
a waterfall world: principles & problems Craig Larman s studies 2 demonstrate the inherent limitations of the Waterfall model, showing: Of 1,027 projects, only 13% were deemed successful - 82% of the projects cited the Waterfall style of scope management as the single largest contributing factor for failure. Of 6,700 projects - four out of fi ve key factors contributing to project failure were associated with, and aggravated by, the Waterfall model, including an inability to deal with changing requirements, and problems with late integration. Of 400 projects using the waterfall model 10% of the developed code was actually deployed, and of that, only 20% was actually used. The waterfall model is a sequential software development process which flows steadily downwards like a waterfall through its development phases, as shown here 3. But the problem with Waterfall lies in its expectations it assumes that a business can identify all its requirements upfront, at the start of a project. It relies on this to define a plan which is then followed throughout the project s lifecycle. 2
Changes to the scope of the project, or to the development design and direction, are then dealt with through change control processes, making adjustments difficult and lengthy. In the Waterfall system, the business owner is rarely involved during the development cycle because clear delivery expectations have already been documented, scoped and signed off. But as most businesses know, all requirements cannot be defined at the commencement of a project because in reality business conditions and requirements change, and generally the business is not particularly good at requirements definition. They only see the product for the first time during the verification phase the fourth phase in the development model and are then able to identify changes and new ideas for how the product should work, once they see what is achievable. Not only are these changes identified late in the game, but each needs to go through its own change control process, which spends valuable time reviewing changes and their impact, redeveloping and re-testing approved changes. This leads to a reduction in overall quality, and an unpredictable increase in project time and budget. embracing an agile world: principles and problems Agile is all about embracing change, and managing software development in smaller iterative processes that involve the business owner. It thus differs from Waterfall in its approach to implementation. Agile methodology grew from the real experiences of professional software developers who had experienced challenges with the Waterfall model. Their need to create software that met business needs and delivered value as early as possible, helped to create a clear value proposition for moving to the Agile model. Designed to involve business owners in iterative development and verification cycles, Agile helps to clarify direction and reprioritise work. The emphasis of the Agile model is on building a product that is aligned with both customer needs and company goals. It does this by developing production-ready pieces of functionality throughout the process. Each piece of functionality is verified by the business and continually improved. Further functionality is added throughout the life of the project, making Agile a step-by-step process to success, rather than Waterfall s lengthy, restricted implementation that frequently delivers an unsuccessful end result. February 2009 3
Sprint Activity Models The first model outlines the Sprint Activity; the second depicts multiple occurrences of Sprint Activity. However, the problem with the Agile model lies in its implementation: businesses which are accustomed to living in a Waterfall world find it extremely difficult to make the change smoothly and seamlessly, particularly in accepting a model that does not work to the traditional idea of inputting requirements and then receiving the results. Rather, Agile requires continuous testing, which whilst ensuring better results, requires greater input throughout the process. waterfall promises Waterfall promises a familiar, well-understood methodology that organisations are well-accustomed to. However, from the statistics provided, it also promises a substantial failure rate and a limited ability to deal with change. 4
agile promises As a newer model, the Agile world is clearly less explored, and so, does not have the familiarity or understanding to match the Waterfall model. An Agile approach, however, can offer business benefits including: Accelerated delivery of initial business value Maximised value throughout the development project Increase in the ability to adapt to change Alignment of software with business needs Increase of visibility into the progress of development Reduction of the overall risk associated with software development Allows requirements to emerge and evolve in an agile fashion Increase in users understanding of development capabilities, which serves to improve the end product Creation of a set of development best practices that allow for incremental delivery of high quality software Increase in levels of teamwork, self-organisation and accountability The following diagram 4 displays the key differences between the Agile and Waterfall development models: February 2009 5
What the evidence suggests is that Agile may offer organisations a more effective model; however, this effectiveness is counteracted by difficulties in implementation. Change is not simple, and so, Adaptra has compiled four key ways to assist organisations who may be considering moving to Agile, or who are in the process of replacing Waterfall. implementing agile successfully Over the last 2 years, Australian organisations have been increasingly utilising Agile for software development. However, the Agile model has proven to be a challenge for large corporations to successfully implement. Integrating new Agile processes into an organisation that has invested time and money in the Waterfall model can be a daunting task. These 4 valuable tips are designed to assist you in ensuring that your transition to Agile is a successful one. 1] prevent organisational resistance The primary challenge for the early adopters of Agile software development was organisational resistance. To be truly effective, the Agile approach needs to reach right across the business, not just the IT organisation. The business needs to move away from traditional Waterfall practices and change how it engages with the IT organisation. In order to achieve this: The business must assign a business owner who has commitment to the project and is available to make decisions and verify direction. The business must trust that the IT organisation will deliver as promised. The IT organisation must deliver what they promise. The business must actively participate and be accountable for setting requirements and validating that these requirements have been met. The business needs to be prepared to exploit the software deliveries in order to gain maximum business benefit. Dynamic communication processes must be implemented to ensure all parties are aware of progress, any issues, and decisions that need to be made. 6
hint 1.1 getting business buy-in If you are attempting to get buy-in from the business on moving to Agile, try utilising the following constructive arguments: By using Agile, the business retains control over the development, because the two organisations are more aligned and the business owner helps to reprioritise work. Under the direction of the business owner, who is involved in this reprioritisation, the features which provide the highest return of investment (ROI) will be completed fi rst. This ensures that the development process provides more benefi ts to the business in a shorter time period. The Agile approach s focus on features that will incur a higher ROI means that critical core functionality will be developed fi rst, so that as more value is added, testing can be performed to ensure the consistent quality of the solution. 2] develop user stories User Stories are a simple way of capturing user requirements throughout a project, and are a useful tool to meet business requirements. They are an effective alternative to writing lengthy requirement specifications at the beginning of the project. what is a user story? A User Story is a simple statement about what a user wants to do with a feature of the software. It is written from the user s perspective, and should not use technical jargon or state design goals. Instead, the Story should be written in business language that is understandable to all readers. A User Story should focus on the who, what and why of a feature. It should not focus on how. For example, as a Data Entry Offi cer, I would want to login to the application so I can access the data. The format would be: As a [user role], I want to [goal], so I can [reason]. In projects using the Waterfall approach, these stories take a more complicated and less user-friendly format. Often they are captured from a lengthy analysis process and situated in protracted, over-long documents. This is clearly not the ideal situation for an effective or timely project. February 2009 7
Experienced testers often build their test cases around these complex User Stories. Unfortunately, this too often results in big bang testing, where steps cannot be tested individually. The test then hinges on a substantial set of deliverables instead of incrementally evolving with the application, as the Agile approach requires. hint 2.1 capturing user stories In workshops, users will often recount failings of their current system or process, sometimes even telling stories about how they see things improving in the future. This is an ideal time to capture User Stories. Once User Stories are created, they can be tested both individually and incrementally. User Stories will change the role of testing from outcome-based to test cases that are based on the look and feel of the functionality. The business analyst will test the interface whilst traditional system testers will perform end-to-end and full lifecycle type testing, cross-system integrations and data conversions. 3] embrace change and increase business engagement In projects using the Waterfall approach, technical specifications are written up-front, outlining the expenses of any potential changes during the project s development. In attempts to avoid scope creep and the dangers of a neverending project, changes are resisted, and when unavoidable, are kept to the absolute bare minimum. However, in an Agile approach, development change is anticipated. In fact, it s expected. The timescale is fixed, with requirements emerging and evolving as the product is developed. hint 3.1 the role of the business owner in agile For the Agile approach to be eff ective, it is imperative that the business owner: Be actively involved Understand the concept Make the necessary trade-off decisions, exchanging existing scope for new The fi xed timetables and evolving requirements of Agile enable a fixed budget, whilst the scope of the product and its features remain variable. 8
The Agile approach creates far superior business engagement and customer satisfaction by actively involving the business owners, ensuring high visibility of the product and process with the flexibility to change when change is needed. These are vital benefits that will create far more positive and enduring working relationships. hint 3.2 communicating with the business owner Agile also increases the communication between the development team and the business owner. If the developers need clarifi cation on a requirement, the business owner will be contacted. This means that the business owner may initially receive more questions than they expect until the development team has cultivated a thorough grasp of what the business owner is trying to accomplish. This communication is critical, because the business owner understands the business and its customers more than the developers do. One way to involve the business owner is by inviting them to the daily development meetings this gives the owner important insights into the status of the project on a daily basis. Depending on the iteration length, with Agile there is an opportunity to reassess the project deliverables every two to four weeks. As value is added, priorities may also change. However, it is important to note that Agile is not a license to be indecisive and inconsistent. Change is welcome to the detail but alterations to fundamental design, direction and strategy are not. The strategy and overall direction must be set and championed by the business owner, who is also responsible for the validation and approval of change at the detail level. It is crucial that the business owner understands the flexibility to change details, and the resulting impact on existing detail. 4] avoid delays test as you go! When transitioning to Agile development, businesses often make the mistake of enacting consecutive Mini Waterfalls in their approach to testing. When organisations postpone testing to the end of the iteration, they in effect produce a Mini Waterfall. In this situation, once testing is finally implemented there is not time to repair the mistakes and defects discovered. With limited opportunity to move with agility, de-scope, re-scope or fix any bugs, the final product will most likely result in a number of unresolved defects and incomplete features. February 2009 9
This is an all-too-common fault which can be caused by the incomplete or ineffective integration of testing into the project. To fully transition to Agile development, the project needs to be test-driven. This includes acceptance and integration testing. Testers who move from Waterfall-based projects often struggle with Agile testing because they are used to being handed a complete system and fully-documented solution. Because of the iterative nature of the Agile approach, it is likely that the tester will be required to test unstable or incomplete parts of the system. To prevent the late discovery of problems in the system, Testing and Quality Assurance should be moved to the start of the iteration. They should also be considered as an ongoing activity over the course of the iteration, rather than a frantically-compressed activity completed at the end of the project. hint 4.1 test early and often Small incremental releases made visible to both the business owner and project team throughout the development process will help to identify any issues early on and allow an easier response to change. 10
The visibility provided by the Agile development process helps to ensure that any necessary decisions can be made at the earliest possible opportunity, whilst there is still time and manoeuvrability to make a material difference to the outcome. The Agile approach thus requires testers throughout the project, effectively increasing the cost of resources. However, this cost is offset by the reduction of very significant risks which cause many projects to fail when using the Waterfall model. For example, the unexpected financial costs of a long and unpredictable test phase, with the potential to exceed project timeframes and budgets due to unsatisfactory testing, could easily far exceed the costs of testing within the Agile approach. By automating the testing of various levels a business can introduce regression testing capability for a minimal cost. This allows you to reduce risks by maintaining high-quality testing with the potential of limiting testing resourcing needs and avoiding additional costs. It is vital that a company defines and regularly reviews its automated testing strategy. The strategy must encompass both positive and negative test scenarios of functionality, and should also recommend when to integrate new functionality into the automated tests. hint 4.2 automating testing At the lowest test level the unit test automated testing can be achieved by developers writing automated test scripts as part of the development iteration. As the number of iterations increase, these scripts can be replayed to ensure that no subsequent development has aff ected work done in previous iterations. Testers need to be clear on what parts of the system are considered production-ready and at a testing-stage, and what parts are not. The Agile approach requires strong communication on every piece of functionality between the developer and tester. The Waterfall approach where the testing team and development team interact only via a defect management system simply does not work on Agile development projects. This Agile approach to testing as opposed to the Waterfall approach enables the business owner to make early and regular adjustments, as well as giving the project team advanced warning of any quality issues. February 2009 11
conclusion Evidence suggests that organisations are looking for alternatives to the Waterfall approach to manage software development projects. Whilst the Agile approach offers an attractive alternative with its emphasis on building the right solution and ability to evolve in line with the business requirements transition from one to the other can prove troublesome for many organisations. With Agile, organisations may be on their way to developing better software faster, but without the proper use and understanding of this new approach, projects can suffer from confusion caused by requirements creep, integration shortcomings, and testing delays. It s also easy to believe that you are following the Agile approach correctly, whilst making classic mistakes. However, with the right implementation, the Agile approach has the potential to provide organisations with a superior method of software development, enabling better systems sooner into production. Agile thus offers a more effective and efficient software development process, as long as the organisation has the will and capacity to avoid the dangers of the transition period. For further details or support on implementing the Agile approach into your organisation or on how we can assist you with your project management needs, please email our solutions team at adaptra@adaptra.com.au. references 1. Lycett, M.; Macredie, R.D.; Patel, C.; Paul, R.J.; Migrating agile methods to standardized development practice, Computer, Volume: 36, Issue: 6, Jun 2003, p79 85. 2. Craig Larman, Agile and Iterative Development: A Manager s Guide (Agile Software Development Series), Cockburn, Alistair and Highsmith, Jim, (Series Editors), August 2003 3. Paul Hoadley, Waterfall model, Wikipedia commons, 2005 4. VersionOne, GA, USA 2006 5. Agile Software Development: A Survey of Early Adopters, L. Vijayasarathy and D. Turk, Colorado State University, Journal of Information Technology Management, Vol.XIX, No.2, 2008) 12