Measuring ROI of Agile Transformation Title of the Paper: Measuring Return on Investment (ROI) of Agile Transformation Theme: Strategic & Innovative Practices Portfolio, Programs & Project (PPP) Management for redefining India Keywords: Agile, Scrum, Transformation, Adoption, Metrics, ROI Abstract This paper presentsdifferent measurement models to measure outcome of Agile transformation in an organization. The paper covers various measurement models that can be used during different phases of Agile transformation. The paper also presents outcome of a survey conducted by the author on how different organizations are measuring effectiveness of their Agile implementation. The paper also presents a case study of Agile transformation where these different effectiveness measurement models were applied successfully. It covers various aspects like Business case definition of Agile implementation, Agile Transformation roadmap, Agile readiness assessment, Agile current state assessment, Agile effectiveness evaluation and ROI of Agile implementation. Learning objectives for the readers are to learn Agile Transformation roadmap, Learn Metrics for measuring effectiveness of Agile implementation and learn Models for measuring project progress in Agile environment. Table of Contents Introduction... 2 Building a Business Case... 3 Building Transformation Roadmap... 5 Agile Readiness Assessment Model... 6 Agile Pilot... 8 Agile Current State Assessment... 8 Metrics in Agile Environment... 9 Metrics - Lead Time / Cycle Time... 11
Metrics - Cost of Delay... 11 Metrics - Agile Maturity Index... 11 Scrum Implementation Assessment Model... 12 Presenting Progress & ROI... 13 Survey... 14 Case Study... 15 Conclusion... 15 Introduction John, VP of Engineering of a Product Development organization attended a business conference and came to know about something called Agile. The concept of Agile appeared very interesting to him and he decided to apply Agile in his engineering organization. He presented this idea to the executive s board and they also liked this idea. Why not? Who would not like to have a software development model that produces results Faster, Cheaper and Better! But when John asked for approval for this investment, people were taken aback! They were hoping this to be just a minor process change, with little or no investment need and something that can start producing results soon. The board realized that Agile Transformation is a much more involved affair, might take 6 months to a year to product results and might require the organization to change the way it works currently. The board naturally asked for ROI of this big exercise and asked for enough checks and balances to ensure that this investment does not go down the drain. It was now John s turn to justify this whole transformation exercise! You might think that the story mentioned above would be a fictional story, but it was a real scenario in one of the product development startup organization. The readers might have experienced that in many organizations Agile transformation starts with lot of inflated expectations and if those are not handled properly then it would result in major disappointment for the stakeholders involved. Hence this paper would to try to present different models which can be applied by Project, Program or Portfolio managers to measure returns and effectiveness of Agile adoption for their own organization. Agile adoption needs to be taken up with a structured approach and following different activities can be considered in this journey.
Building a Business Case It is extremely critical for any organization to be clear about the reasons why it is planning to adopt Agile. Without clear objectives, the Agile adoption might lose its focus and might result in pseudo-agile environment (also referred as Scrum-but ). Following parameters can be considered for developing a Business Case for adopting Agile: Customer Benefits ROI Parameters Business Case Business Benefits Investment Needs Project Benefits Figure 1: Business Case for Agile Transformation Identify the Customer benefits: Values and principles of Agile put lot of emphasize on delivering value to the customer/end-user. It is very clear in this sense that customer is primary beneficiary of Agile implementation of any service or product based organization. Customer would get frequent feature updates and new enhancements (or resolution of issues) sooner. Customer can also request for changes in the features planned earlier, due to change in his dynamic business scenario, instead of a formal & rigid change request process. Identify the Business benefits:
The organization needs to clearly identify the benefit it would have by implementing Agile model. Please note that the benefits and business case is going to change based on the type of the organization i.e. Product based, Services based or Startup. Benefits for Services based organizations: - Higher customer satisfaction: Since customer would be regularly engaged in product development lifecycle with the vendor (the organization developing the product), high customer satisfaction would be a natural result. This can be captured through existing metrics of the organization like Customer Satisfaction Index captured through Customer Satisfaction Survey. - Long term engagement: Agile core principles encourage all the parties involved in project/product development to work as partners working together towards a common goal. We have observed that some of the organizations have been able to build strong partnership with the customer through true implementation of Agile methodology instead of old Customer-Vendor model. Benefits for Product based organizations: - Faster Time to Market: It is important for any product based organization to roll out new features of product quickly in the market. It helps in responding to customer requests for new features quickly and also helps in responding to competitor s product releases. - Shorter revenue realization cycle: In many product based environments, revenue is directly linked with the features delivered or the product release made available to the customer. With Agile model, an organization can roll out new features in the product in periodic intervals and thus can get revenue periodically in chunks. Benefits for startup organizations: - Quick Validation of Business Idea: Some of the startup organizations would have a new product idea and it is important for them to validate this product idea quickly with the potential customers. This helps in ensuring that the product idea has a business potential and also helps in getting quick feedback from early adopters and ensure that the efforts are not spent on features that are not valuable to potential customers. - Inflow of Revenue: It is important for startup organizations to have steady inflow of revenue to maintain its existence. Hence shorter and frequent releases of product or services are definitely required.
Identify the Project benefits: Shorter cycles: Shorter cycles help in managing the project better, with limited scope, focused team and periodic integration and testing. Frequent customer feedback: Frequent demos of features developed per iteration would help in getting quick & frequent customer feedback. This will help in ensuring that the team is delivering the project as per customer s expectations and any gaps in expectations can be bridged well in time. Identify the Investment required: Agile does not come for free! The organization needs to invest in budget, resources, time & efforts to implement it properly. Following aspects needs to be considered while identifying the investment needs: - Infrastructure required [For Continuous Integration etc.] - Tools & licenses [Like Automation, Agile Task Management etc.] - Training & Coaching [For Team, Scrum Master, Product Owners etc.] - Reward & Recognition [For best performing Agile teams] Identify Return on Investment (ROI) parameters: It is important for any organization to define the parameters on which it would calculate and review its return on investment. Only with clear and precise parameters, ROI of Agile Transformation can be identified. The organization can also use collaboration tools like Innovation Games [1] to identify the current pains areas, which can be used while defining parameters. Following are some of the parameters that can be considered: - Change in project delivery timelines - Value delivered to the customer - Improvement of team efficiency - Reduction of waste - Payback period Building Transformation Roadmap
An organization needs to define a roadmap of rolling out Agile in the organization, various phases and checkpoints to ensure that things are moving in right direction. Agile Readiness Assessment Model Figure 2: Roadmap of Agile Transformation An organization needs to understand where it currently stands from Agile readiness perspective. This would help the organization to know its current state and how well it is prepared to go for Agile or what are gaps it needs to bridge before it can move to Agile. Following model provides a set of parameters that can be considered by an organization to identify Agile readiness of its different teams. The parameters can be tailored as per the organizational environment and needs. Question Guidelines for Points Points Team Size 5 Team size of 7 to 9 3 Team size less than 7 2 Team with 10 to 15 members 1 More than 15 members 2 Team Composition 5 - Completely cross functional; no dependency on external teams 3 - Cross functional with some dependency on external teams 0 - Not Cross functional; major dependencies outside the team 3
Business Value Stream Coverage Team's Agile expertise Candidate ScrumMaster within the team Experience of Agile Project Management Tool Test Driven Development (TDD) Experience QA Automation Tools Experience Product Owner (PO) / Business Analyst(BA) availability PO/BA's Agility PO/BA's Feedback Business Users' availability 5 - The team covers the entire business value stream 3 - The team covers more than 50% of the business value stream 1 1- The team covers less than 50% of the business value stream 0 - The team covers less than 10% of the business value stream 5 - Avg. 3 years + exp. in Agile/Scrum 3 - Avg. 1 to 3 years of exp. in Agile/Scrum 1 1 - Less than 1 years of exp. in Agile/Scrum 5 - Avg. 2 years + exp. As ScrumMaster 3 - Avg. 1 to 2 years of exp. As ScrumMaster 3 1 - Less than 1 years of exp. As ScrumMaster 5 - Used Agile Tool for at least 2 consecutive releases 3 - Used Agile Tool for at least 1 release 1 1 - Learning Tool on the job 0 - Never used any Agile Tool 5 - Avg. 2 years + exp. on TDD 3 - Avg. less than 2 years + exp. on TDD 1 1 - Learning on the project 0 - No TDD experience within the team 5 - Avg. 1 years + exp. on tools like Selenium 3 - Avg. less than 1 years + exp. on tools like Selenium 3 1 - Learning on the project 0 - No QA Automation experience 5 - Available &co-located with the Team 3 - Distributed but available on calls daily 1 - Distributed but available on calls weekly 3 0 - Prefers to communicate through Functional Specification Document (FSD) 5 - Willing to express functionality in Epics/User Stories 3 - Will split the Functional Specification 3 Document (FSD) in user stories 0 - May not be available to split FSD into stories 5 - Committed to and demands frequent Demos 3 - Intermittently available for Demos 3 0 - Don't have time for frequent Demos 5 - Actively involved in Demos and feedback 3 - Can attend Demos Intermittently 3 0 - Not accessible frequently Total 27 Table 1: Team Agile Readiness Assessment Model
We have successfully used this assessment model to identify the gaps required to be resolved for a smooth Agile implementation. It also helped in identifying the right teams to participate in Agile pilot. Agile Pilot One of the first steps for an organization in its Agile journey is to conduct a Pilot with small set of teams. One of the activities an organization will have to work on in the pilot is to identify the Agile methodology to implement i.e. Scrum, XP, Kanban, Lean, SAFe, DAD etc. The metrics and tracking mechanism would depend on the methodology identified; hence it is an important aspect of overall transformation. Figure 3: State of Agile Survey 2011, VersionOne[2] Agile Current State Assessment Once the teams start implementing Agile, one of the important assessments that they should conduct is Agile current state assessment. This assessment would help them understand how team is placed on different aspects like Product Ownership, Release Planning, Iteration Planning, Development Practices, Quality Assurance Practices, and Team Cohesiveness etc. This will help the team to identify the gaps in its current implementation of Agile and team can identify a plan to bridge these gaps. The teams should
conduct this assessment together (so that they have collective feedback and awareness about improvement aspects) every quarter. Figure 4: Agile Current State Assessment[3] It is important for an organization to ensure that its Agile adoption is moving in correct direction and team is not falling into trough of disillusionment. Project can fall into trough of disillusionment, if it ends up focusing on just the practices but not focusing Agile principals and thus resulting in fractured or pseudo Agile implementation. Metrics in Agile Environment Figure 5: Gartner Hype Cycle[4]
Metrics provides important insights on how the project is being executed against the defined parameters. However metrics in Agile environment should be carefully designed and used to ensure that they do not deviate from the core Agile methodology. Business Metrics Earned Business Value Return on Investment (ROI) Release Cycle Cost of Delay (COD) Lead Time Cycle Time Process Metrics Velocity Backlog Size User Stories Done per Iteration User Stories carried forward to next iteration Quality Metrics Defects count per story / Defect density Acceptance tests per story Defects per story Defect slippage Figure 6: Types of Metrics in Agile Agile environment emphasizes on applying lean and lightweight process models in project management instead of conventional (i.e. waterfall) methods which would need lot of data to be collected, processed, comprehended and presented. Some of the following charts can be used to track the progress of the project in Agile environment: Sprint Burndown Chart Release Burndown Chart Area Chart / Cumulative Flow Diagram (CFD) Velocity Chart Test Automation metrics Code Quality& code coverage metrics
Metrics - Lead Time / Cycle Time Lead Time and Cycle Time provides a good measure of time taken for completion of work. In waterfall process, typically cost of delay could be higher since there could be multiple Work in Progress (WIP) items. Agile recommends completion of one feature at a time and then move to the next feature, thus reducing overall WIP items. - Lead time: It is the time between the initiation and delivery of a User Story. It is the time a Story enters the Deployed stage minus the time the story has entered the Backlog stage. - Cycle time: It is the time between two successive deliveries; it is the time between two stories entering the last stage (i.e. Deployed stage). Reducing the cycle time is of extreme importance when optimizing processes. The team needs to ensure that they limit the work in progress items per iteration, which effectively means that more features are delivered periodically. Metrics - Cost of Delay Cost of Delay (COD) is an important parameter, which can provide us insights on what could be the cost if a high priority feature is delivered late. The delays in the system can also be identified through Value Stream Mapping exercise. The objective of Agile implementation is to reduce the delays (measured as COD) in the system and ensure continuous delivery of value. Following are the parameters for identifying Cost of Delay: Cost of Delay [5] = User-Business Value + Time Criticality + Risk Reduction or Opportunity Enablement The features that are having high Cost of Delay and are of shortest duration are selected first for implementation. Metrics - Agile Maturity Index It is important for any organization to measure effectiveness and efficiency of its Project being implemented in Agile environment. One of the models we developed to measure effectiveness of Agile implementation is Agile Maturity Index. An organization can define its own parameters, develop a
questionnaire covering those parameters and assess various project teams on these parameters as represented in Table 2. Process Group Rating Applicability Score Average Release Management Effectiveness 6 7 86% Sprint Management Effectiveness 7.5 9 83% Product Backlog Management 7 7 100% Frequent Product Delivery 7 8 88% Quality Assurance Effectiveness 5.5 6 92% Maturity Index 90% 1.79 88% 0.88 Collaboration & Self-organization 2.5 3 83% Measurement, Retrospection& Improvements 1 3 33% 33% 0.33 Customer focus 5 5 100% 100% 1.00 Agile Maturity Index 4.12 Table 2: Agile Maturity Index Scrum Implementation Assessment Model An organization should build rewards and recognition mechanism so that the team members feel appreciated and encouraged for effective implementation of Agile. One of the models we developed was Best Scrum Implementation Award. A panel of Agile coaches evaluated various teams on different parameters related with Agile/Scrum and provided their ranking. Based on this feedback, team implementing Agile/Scrum in best possible way was identified and appreciated. In similar manner, a team member playing most effective role as Scrum Master across teams can also be identified and awarded. This would create a healthy competition amongst teams to implement Agile in proper way and also would build an environment of appreciation and recognition.
Presenting Progress & ROI Figure 7: Best Scrum Implementation Assessment Model An Agile project should have a defined governance model and periodic governance meetings with the required stakeholders so that they are aware of the progress being achieved after Agile adoption has started. The person responsible for project updates (Project Manager / Program Manager or Scrum Master) should ensure that Project Dashboard is presented to all the involved stakeholders periodically. The dashboard can contain release predictability, progress made (and benefits realized) so far and the next plan. Project Information Project Overview Reporting Period Prepared By Benefits Realized During the Reporting Period Quantitative Benefits Benefit Description Date Realized (e.g. Delivery cycle reduction, Revenue realization)
Qualitative Benefits Benefit Description Date Realized (like Value delivered to customer, Team efficiency, Customer/Team satisfaction) Benefits Not Realized During the Reporting Period Benefit Description Planned Corrective Action New Benefits Benefit Description Assigned To Table 3: Agile Transformation Report Survey I conducted a survey to analyze what parameters are used by different organizations, management team and coaches to evaluate the outcome of Agile transformation. Following were some of the key attributes used by different individuals and organizations: - Value delivered to the end user / customer - Achievement of Business Goals - Reduction in Cycle Time - Reduction of costs, defects, waste, delays in the organization - "Break-even point" i.e. the point where the value generated covers the costs incurred on Agile transformation From the third party professional surveys, the state of Scrum report [6] presents that delivering value to the customers has been a primary criteria for organizations for moving to Agile. State of Agile survey [7] identified that the common focus areas for Agile transformation seems to accelerate product delivery,
handle changing priorities and increase overall team productivity. The Agile implementation seems to be successful in managing changesin customer priorities, increasing team productivity and improving project visibility to its stakeholders etc. Case Study A product development organization applied Agile in its entire product development cycle and following were the benefits achieved through this transformation: - Release timeframe changed from 1.5 years to 3 months. Due to Agile environment, the release cycle changed to every quarter, which was a significant benefit for the organization to recognize revenue (due to new features releases) much earlier. - Shorter delivery cycles significantly helped in having frequent integrations of different modules and resolution of integration issues much earlier in the entire cycle. - Frequent demos helped in building client confidence and opened new business opportunities. It helped the organization to convert a short-term customer contract to a long-term partnership. - Customer started receiving early visibility for new features through the integrated environment and could provide their feedback regularly. - Automation of testing activities resulted in significant effort reduction: o Manual regression testing efforts: 5 days o Automated regression testing efforts: 4 hours o 90% effort reduction Conclusion Through various independent professional surveys, the survey conducted by the author and based on practical experiences and case studies, it has been observed that planned and well implemented Agile environment can provide significant ROI to any organization. Agile project management does provide value to the end users / stakeholders as well as it provides multiple benefits to the organization delivering the project, if applied correctly. It is important for an organization to define its objectives for adopting Agile very clearly and should measure progress onthese objectives periodically. The metrics and various models mentioned in this article can be applied to capture the qualitative and quantitative information, which can provide clear details on what the returns or benefits achieved by adopting Agile project management methodology. Implementation of Agile methodology with focus on Agile values and principals would definitely provide the desired results and can result in greater satisfaction for customer as well as other stakeholders.
References [1]Innovation Games - http://www.innovationgames.com/ [2]State of Agile Survey 2011, VersionOne - http://www.versionone.com/pdf/2011_state_of_agile_development_survey_results.pdf [3] Agile Assessment by Rally - www.rallydev.com [4] Hype Cycle -https://en.wikipedia.org/wiki/hype_cycle [5] Cost of Delay - http://scaledagileframework.com/wsjf/ [6] Stateofscrumreport- https://www.scrumalliance.org/why-scrum/state-of-scrum-report [7] State of Agile survey- http://www.versionone.com/pdf/state-of-agile-development-survey-ninth.pdf