Leveraging Agile and CMMI for better Business Benefits Presented at HYDSPIN Mid-year Conference 2014 28-Jun-2014
Outline 2 Context Key Business Imperatives Agile Adoption and CMMI Roadmap CMMI+Agile Best Practices & Key Benefits Overall Performance Results Cautions/Key Risks Concluding Remarks
Context 3 Sonata s portfolio of services include: Product Engineering Services to ISVs, ADM, Managed Services, Testing, BI & Analytics, IMS and SMAC Sonata has a strong quality management system evolved through more than two decades by aligning & adopting to various international standards and models such as ISO9001, CMMI, ISO27001 and ISO2000-1 After the CMMI appraisal in Apr 2011, there were various triggers driving the process improvement initiatives Voice of customer Significant increase in the number of customers looking for faster and high quality delivery at an optimum cost Looking for lean processes which can rapidly respond to the changing business needs without compromising the process discipline
Context 4 Voice of the Process Process Performance Baseline Report May 12 Process Compliance & Effectiveness Analysis Portion of non-compliances due to QMS not aligned to Agile Delivery Methods Voice of the Business Adopting CMMI DEV v1.3 and getting assessed at Maturity Level 5 Enhance the QMS aligning to CMMI DEV v1.3 and Agile Delivery Methods The Voices of Customer, Business and Process were analyzed, considered during Senior Management Review and decision was taken to adopt Agile methods as a part of business execution strategy & CMMI DEV v1.3 reappraisal roadmap.
Key Business imperatives 5 Faster time to market Swift responsiveness to changing business needs Better visibility, control and collaboration Improved Product Quality & Resilience Intolerance to waste and overheads Ownership and Accountability Improvement & sustenance of Team s morale
Agile Adoption with CMMI DEV v1.3 Roadmap 6 People Enablement Agile Training to Scrum Masters & Team Members Agile orientation to customer s teams wherever required Establishing processes, guidelines, templates to make Agile projects compatible to CMMI 1.3 Tools evaluation and recommendation Process Awareness and Orientation Sessions for Agile Establish PPBs at each ODC level based on past data Develop PPM for predicting performance at Sprint & Release level Evaluating the progress periodically and fine-tune the processes Collate Best Practices and Learnings in the Knowledge repository (VAULT)
QMS enhanced to include Agile Delivery Lifecycle 7
Agile Delivery Model tailored for an ODC 8
Project Planning 9 Epic level User Story level Task level E0 Based on one line requirement E1 (Gate 1) Single page business need Project Brief Discussions E2 (Gate 2) High level requirements Solution blueprints E3 (in Agile Sprints) Detailed Solution design UX and application Wireframes Based on one liner Greater than +/- 50% variations estimated from actuals Discussions between business, architecture, IT solutions (Customer Technical Team and Sonata) High level estimates with assumptions and dependencies +/- 50% variations estimated from actuals Estimates at individual requirements / user story level Based on solution with assumptions on detailed integration between components +/- 30% variations estimated from actuals Estimates based on technical tasks to complete a user story Sized in story points Productivity based on velocity of the team normalised across projects +/- 10% variations estimated from actuals Reviewed and validated by Customer Technical Team
Two Week Sprint 10 Sonata Software Ltd., Sonata Internal 10
CMMI+Agile Best Practices (1 of 3) 11 CMMI+Agile Best Practices Release Planning Use PPB and PPM for arriving at Release Schedule for a Minimum Viable Product(MVP) Use PPB and PPM for deciding the Team Resource Mix and Capacity Time Boxed Sprints Sprint Planning User Story Walkthrough by Product Owner Solution Blueprint Walkthrough, if applicable Use PPM for Effort Prediction and decide how much user stories to commit Use PPB & PPM for optimum assignment of user stories based on complexity Key Benefits Realistic Schedule Forecast Improved Capacity and Team Planning Requirement Clarity and Completeness Improved Sprint Planning CMMI Practice Agile Practice
12 CMMI+Agile Best Practices (2 of 3) CMMI+Agile Best Practices Sprint Execution Requirement Traceabilty through tool Use of Decision Analysis techniques for better Design and Project Management Use of Code Quality tools such as Sonar, Resharper etc. Sub-process monitoring for controllable metrics such as Coding Delivery Rate, Code Complexity, Test Coverage etc. Refactoring Automated Unit Testing using Junit, Nunit etc. Continuous Integration Show and Tell Demos Automated Build and Deployment Key Benefits Reduced Process Overhead Improved Product Quality CMMI Practice Agile Practice
13 CMMI+Agile Best Practices (3 of 3) CMMI+Agile Best Practices Release and Sprint Retrospective Use of Causal Analysis and Resolution techniques Organizational improvement beyond the project team Organizational Process Areas such as Organizational Training, Process Focus, Definition, Process Performance, Performance Management Innovations implemented through OPM process resulting in significant improvement in delivery Organization Knowledge and Process Asset Repositories System s View and Product Integration Use of Product Integration approach and practices Key Benefits Effective Learning & process improvements Holistic approach to Agile/Lead Practices Adoption Risk reduction through process standardization, people capability development and innovations Robust Product Integration CMMI Practice Agile Practice
Framework for Quantitative Project Management 14 ODC1 PPB Control Critical Sub process (Control Chart) ODC Business Objectives (GM & CSAT) Project QPPOs (Productivity and Defect Density) PPM Composition: List of independent variables: 1)Delivery rates at sub-process level a) Product Backlog Analysis Delivery rate-distribution b) Design Delivery ratec) Coding Delivery rate- Developer and Designer Skill-Expert/Medium/Novice Distribution d) Code Review Delivery Rate-Distribution e) Unit Test Scripting and Testing Delivery Rate-Distribution f) Test Case Design, Test case Review Delivery Rates g) Manual QA Testing Del Rate h)automation Scripting, Automation Script Review, Automation Testing Del Rates i) Testing Defect Density 2)Size of user stories (Sprint) Story Point 3) Available Capacity or Revised Capacity Identifying Risks in completing the Iteration within available capacity Identify critical sub process (sensitivity analysis)- Coding Delivery rate, Testing Defect Density Effort Prediction(Big Y): a) Developer Skill (X1) b) Designer skill (X2) Defect prediction 1. Junit test coverage 2. Unit testing delivery rate 3. Class complexity level Do CAR (if any deviations) Change /Improve process ODC1 PPM 14
Purpose of Prediction Model in an ODC 15 Purpose To assess the certainty of completing the committed product backlogs in an iteration within the available capacity. The prediction is done based on the historical performance from past iterations on each sub-process delivery rates from PPB. Simulation Model predicts the total effort based on the following inputs Size of Product Backlog Items In-scope of sub-process tasks for Sonata offshore team Skill of Developer and Designer PPB Values for sub-process delivery rates Other inputs that will influence the certainty of completing the committed backlog items in an iteration include Initial Capacity (based on Team Size, Ideal hours per day, Number of working days and exclusions due to leaves and holidays) Revised Capacity (based on Revised Team Size (addition of shadow or backup resources), Revised Ideal hours per day, Revised number of working days and exclusions due to leaves and holidays) 15
Effort prediction Model Usage during Sprint 8 Planning. 16 Sprint 8 has 72 story points in scope with 145 days of available capacity. Simulation model was used to predict the certainty of completing 72 story points in 145 days. The result of initial run predicted 87% certainty as this was a good certainty there were no What-If scenarios established. But it was decided to rerun the model mid sprint, after 7 days wherein the certainty was seen to be 100%. 16
Effort prediction Model Usage during Sprint 8 Planning 17 From the output of the prediction model, it is inferred that it is 87% certain that all the planned backlog items(72 story points) in Sprint 8 can be completed with 145 man days of effort, which is the team capacity. As the certainty level is more than 75%, what-if analysis was not done. Sensitive sub processes and metrics to be monitored under Control chart 1. Coding delivery rate 17
Effort PPM Usage during Sprint Planning 18 Parameter being predicted Project Objective Input parameters used for prediction Type of prediction model used Prediction Range/Value Certainty/ Confidence Level of prediction Risk assessment based on prediction results and mitigation plan/s implemented Effort for Iteration Completion Story Delivery Rate * Size of user Story * In-scope of Sub-process tasks for Sonata offshore team * Skill of Developer Simulation Model 145 man days 87% Minimal risk 18
PPM Usage during mid Sprint 19 The prediction model was rerun after 7 days into Sprint by providing actual effort spent so far for each of the user stories. It was observed that there is 99% certainty after 7 days. This confirmed that the scope of the sprint will be met with the available capacity and velocity of the team was also good which provided 99% certainty within 7 days into sprint. 19
Effort prediction Model Usage during Sprint 4 Planning 20 During mid sprint with 7 days effort we get 99% certainty. Sensitive sub processes and metrics to be monitored under Control chart 1. Coding delivery rate 20
Effort PPM Usage during mid Sprint 21 Parameter being predicted Project Objective Input parameters used for prediction Type of prediction model used Prediction Range/Value Certainty/ Confidence Level of prediction Risk assessment based on prediction results and mitigation plan/s implemented Effort for Sprint Completion Story Delivery Rate Actual Effort of tasks completed after 5 days into sprint. Simulation Model 145 99% No risk 21
Defect prediction using PPM 22 Defect prediction using PPM is done through simulation model. The inputs that control defect prediction are as below Size of the sprint in story points Resource level skill assigned to each user story. 22
Defect prediction PPM during planning 23 For a target SIT defect density of 0.3 and current assignment of user stories predicted a certainty of 100% From the model it has been identified that skill level of resource assigned to each user story has major impact on defect density. Varying the skill level of the resources between user stories is an option that could be used to control defects. Apart from this the sonar parameters are regularly monitored for class level complexity and unit test coverage. 23
Agile Adoption Statistics @ Sonata 24 Detail FY12 FY14 Percentage of Agile projects 20% 60% Application Development and Maintenance 1 customer 4 customers Product Development and Engineering services 1 customer 3 customers Testing and Migration 1 customer Number of Agile engagements initiated during the year 25% 75% Distribution of current Agile projects based on location Co-located 37% Distributed 100% 63% Distribution of Agile resources Total Agile trained delivery personnel (actively working on Agile engagements) 55 522 Total Agile trained personnel (including those not working on Agile engagements) 220 850 Sonata trained Scrum Champions 1 35 Certified Scrum Masters (external certification) 1 5
Overall Performance Results - Shift in Performance Baseline 25 Productivity increased by 30% Defect Detection Efficiency increased to 100% and sustained 95 % of customers rated as Very Good and Excellent compared to 70% before Delivered Defect Density reduced by 75% Total Testing Delivery Rate(Person-hour/Test Unit) reduced by 6% Test Scripting Delivery Rate(Person-hour/Test Unit) reduced by 10%
Qualitative Benefits from CMMI+Agile Practices 26 Better Scope and Change management Higher Productivity Lower Turnaround Time(TAT) Higher Customer Satisfaction Lower Project Management Overhead Frequent Monitoring and Corrective actions Continuous Integration
Cautions/Key Risks 27 Product Owner non-availability or too many owners Everyone in the team does not participate in Sprint Planning meeting Allowing scope change within a Sprint Not enabling Test driven development & Continuous Integration through automation Contract is not aligned to Agile way of planning and delivery Not having a defined set of processes & metrics for Agile projects Team members and Scrum Masters not trained and oriented with Agile practices and tools Not empowering the team Compromising on Definition of Done (DoD) Not inspecting and adapting for future sprints
Concluding Remarks 28
29 Confidential.
Thank You