AGILE Burndown Chart deviation - Predictive Analysis to Improve Iteration Planning A. Mr. Dhruba Jyoti Chaudhuri 1, B. Ms. Aditi Chaudhuri 2 1 Process Excellence Group, Tata Consultancy Services (TCS) Ltd., India 2 Process Excellence Group, Tata Consultancy Services (TCS) Ltd., India Abstract - While AGILE development aims mainly at incremental development and delivering product (solution) in a time-boxed fashion; measurement framework is still not matured to assess and benchmark performance at organization and/or industry level. Burn down chart is one such key metric that tracks adherence to scope, effort and indirectly to schedule and is mandatory for daily Stand-up meetings. Though ideal target would be ZERO deviation; in real life we observe both +ve/-ve slippage(s). This paper is aimed at providing pointers, possible roadmap to analyze Burn down chart deviation to setup a predictable band or operating limit that would help improving Iteration planning to include suitable risk contingency reserve (+ve slippage: means possible push back some scope to Product Backlog, while ve slippage: means probable provisioning of more scope into Iteration). This would ensure, greater assurance of completing planned Iteration Backlog within specific Iteration; the critical success factor of Agile planning/ execution. Keywords: AGILE, Burndown, Iteration, Planning, Statistical Predictive 1 Introduction AGILE development primarily aims at developing and delivering product and/or solution in a time-boxed fashion, focusing on iterative and incremental development towards delivering tangible outcome (set of functionality/ system behavior). Each time-box is essentially an Iteration (also called as Sprint or Scrum ), where a subset (called Iteration or Sprint Backlog) of total work scope (called as Product Backlog) is selected. This selected scope could ideally be completed in a specific Iteration and be ready to go live after Iteration is complete or with selected Release or through additionally planned System Test and/or Acceptance Test Cycle. During Iteration, a Task plan is prepared (for decomposed scope, as selected in Iteration backlog), by the team with Ideal (effort) hours is assigned to decomposed tasks/ activities. As completion of selected scope is extremely important, an Iteration Burn down Chart is prepared to show how total planned/ allocated effort would be consumed (Total to Zero) from start to end of Iteration. This planned Effort Burn Down is prepared keeping in mind, how much equivalent scope would be completed/ remained on daily basis. During Iteration execution, a revised effort estimate is put to account for remaining Iteration scope at the end of each day. Though ideal (expected) Burn down performance would be a ZERO deviation; in real life it is often observed to be deviating ( actual VS planned ) both in +ve and ve direction. As Iterations are time-boxed, to understand this behavior is of extreme importance; as +ve slippage indicates growing possibility of some amount of scope may have to be pushed back to Product or Release Backlog from Iteration backlog (due to probable unfinished scope), while ve slippage means; possible under-utilized resource and more scope could have been provisioned into this iteration. It also guides to evaluate process effectiveness and trigger improvement cycle in regard to Scope management, Planning and Estimation, Risk Management, Issue (Impediments) Management and Resource Utilization etc. This paper provides a framework and roadmap towards how, at an organization level, an expected and predictable band could be established and benchmarked, by analyzing a number of AGILE projects and a good number of Burndown chart behaviors. This requires periodic refinement and calibration based on future Organization data and Industry Benchmark, if available. This could be initiated at project level, then at organization level etc. 2 Burn down chart what is it? A Burndown chart is a simple but powerful tool to measure AGILE Project progress and manage deviation. Iteration Burndown represent, daily, the remaining work (basis iteration backlog) over specific iteration lifetime. It could be at Iteration, Release and/or Project level. It s a great management tool as it provides both project team and all other stakeholders with a common view of iteration and/or project progress. Sample Burndown charts are shown below with possible interpretation and opportunities for improvement, towards better planning, monitoring and control of AGILE projects.
Figure 1: Sample Burndown chart #1 Possible interpretation: Iteration Planning and Estimation Scope/ Task was under estimated. Figure 4: Sample Burndown chart #4 Possible interpretation: Typical scenario or special cause for half of the iteration timeline, iteration Burndown didn t happen; then scope may have cut down (though scope adjustment is not allowed during iteration, and unfinished scope would automatically be returned to Product or Release backlog) to match rest of the iteration timeline. Or, the estimation was grossly on the higher side. 3 Improvement Opportunity, Objective and Business Case Figure 2: Sample Burndown chart #2 Possible interpretation: Normal expected performance and variation. 3.1 Opportunity for Improvement No Industry and/or Organization level guideline and Benchmark available to i) quantitatively understand and analyze Burndown Chart behavior, as a metric and ii) build predictability and enable better Iteration planning & execution. 3.2 Objective Establish Organization, Unit, Account, Project Level Metric for i) Internal Benchmarking and ii) Drive process improvement and maturity in AGILE execution. Figure 3: Sample Burndown chart #3 Possible interpretation: i) 1 st part - Iteration Planning and Estimation Scope/ Task may have overestimated and ii) 2 nd part - Some tasks may have underestimated or some special cause/ impediments (issues) may have caused this. 3.3 Business Case Better planning of Iteration Backlog and Ideal Estimation to i) include risk reserve (both +/- deviations), ii) understand impediments and causes for these deviations and integrate with process improvement to attain next level maturity. iii) Improved Iteration Task Planning and Estimation.
4 Burn down chart analysis proposed benefits Table 1 Benefit Articulation of Burn down chart analysis Measurement Project Planning Trigger for Causal; enabling improvement and optimization Sprint Retrospective (Learning and Feedback loop) Baseline and Benchmark Effective Risk Management Time to Market Helps understand current performance and ability to deliver Helps in Iteration Planning; deciding on Iteration Backlog Scope better keeping in mind the operating/ predictable limit of deviation: planning adequate reserve for possible +ve slippage and having a backup scope or other tasks that could be additionally completed, in case of ve slippage To understand reasons for +ve and/or ve slippage and correct process controls; for example; i) Story point or Value scoring guideline ii) Ideal Task estimation iii) Decomposition of User Story into Tasks iv) Coverage of various tasks like SDLC, Review/ Testing, Project Management etc v) Competence and Productivity of Team v) Planned VS expected Velocity vi) Dependency, Issues/ Impediments causing delay and resolution cycle time etc Iteration Retrospection to analyze these deviation, identify improvement opportunities and adopt continuous improvement cycle Baseline performance in order to improve own performance and benchmark with other (different relationship, Organization Unit, Organization, Industry etc) performance and if better, adopt Best Practice(s) and/or learn from failure reasons Looking for key triggers and effective planning of Risk mitigation and contingency reserve Greater assurance of completion of Iteration and Release Backlog on Time-boxed fashion 5 Burn down chart performance predictive analysis approach Data collected for three different project execution, for same account and customer For each project, Burndown chart data captured, for various iterations Burndown chart behavior analyzed o To understand daily deviations (planned vs. actual) Extreme outliers (special cause) eliminated Statistical Data Analysis done on data o Normality & Descriptive Statistics, Box Plot, Dot Plot, Time Series, CNTL Charts etc Predictable Band (sample) selected based on outputs, it s interpretations and finally team s decision Same is carried out at Account/ customer level to understand overall predictability Need periodic calibration based on o Causal analysis, elimination of Special cause o Influence of Common cause o Corrective action taken to improve Iteration Planning and monitoring Burn down chart monitoring Data collection mechanism Normally, day wise Ideal (planned/ expected) hours and actual (revised) hours to complete remaining scope of work, is captured; from which % variation is derived, as experienced and recorded in different days in iteration. Similarly, data is captured for other iterations, as well. Day # Ideal % Deviation 1 28 285 1.79 2 245 2.4 3 21 2-4.7 4 18 15-8.33 5 145 149 2.7 12 12. 7 8 7-12.5 8 5-1.7 9 2. Figure 5: Sample Burndown chart and Data Collection
7 Burndown Chart Analysis: Determination of Predictive Band 7.1 Project level Burndown chart analysis, for three sample projects (for same customer) were conducted using various standard techniques like Box Plot, Time Series, Dot Plot, Descriptive Statistics etc., and the same has been depicted below for reference. Three sample projects are represented as Case1, Case2 and Case3. Case 1 Case 2 Case2 Smoothing Plot for Case2 5 95.% PI Alpha.2 MAPE 14.87 MAD 1.54 MSD 41.324-5 4 8 12 1 2 24 28 32 3 4 Figure 9: Time Series (Smoothing Plot) for Case 1 Case1 1 5 Smoothing Plot for Case1 95.% PI Alpha.2 MAPE 37.177 MAD 23.53 MSD 7.74-3 Summary for Case2 3 Anderson-Darling Normality Test A -Squared.97 P-V alue.13 M ean 1.72 StDev 21.11 V ariance 445.234 Skewness.88833 Kurtosis 1.127 N 39 M inimum -35.294 1st Q uartile. M edian 5.793 3rd Quartile 17.978 M aximum 7.939 95% Confidence Interval for -5 3.232 1.912 95% Confidence Interv al for 3.482 13.8-1 1 12 18 24 3 3 42 48 54 4 8 1 12 14 1 95% C onfidence Interval for StDev 17.244 27.194 Figure : Time Series (Smoothing Plot) for Case 1 Figure 1: Normality & Descriptive Plot for Case 1 Summary for Case1 Anderson-Darling Normality Test A-Squared 2.39 P-V alue <.5 7.595 StDev 32.5 Variance 14.45 Skewness -.27887 Kurtosis 2.3535 N 57 Dotplot of Case2 Minimum -85.714 1st Quartile -1.958-5 5 1 4.82 3rd Q uartile 23. Maximum 1. 95% C onfidence Interval for -.94 1.153 95% C onfidence Interval for -28-14 14 Case2 28 42 5 7 2.41 1. 95% Confidence Interv al for StDev 27.232 39.59. 2.5 5. 7.5 1. 12.5 15. Figure 7: Normality & Descriptive Plot for Case 1 Case 3 Figure 11: Dot Plot for Case 1 Dotplot of Case1 Smoothing Plot for Case3 1 5 95.% PI Alpha.2 Case3 MAPE 353.18 MAD 2.2 MSD 114.94 - -5 Case1 5 1-5 -1 3 9 12 15 18 21 24 27 3 33 Figure 8: Dot Plot for Case 1 Figure 12: Time Series (Smoothing Plot) for Case 1
Summary for Case3 Anderson-Darling Normality Test A -Squared.45 P -V alue.3 M ean 7.2 S tdev 32.794 V ariance 1.488 S kew ness -.14898 Kurtosis 1.519 N 33 M inimum -82.451 1st Quartile -7.39 M edian 2.8 3rd Q uartile 27.114-5 5 1 M aximum 92.98 95% Confidence Interval for -4.42 18.831 95% Confidence Interv al for -.28 22.324 2.3731 43.3772 Figure 7: Normality & Descriptive Plot for Case 1-5 5 1 15 2 Figure 13: Normality & Descriptive Plot for Case 1 Dotplot of Case3 95% C onfidence Interval for StDev 4. Time Series predicted value lies between Box Plot 95% CI for (except for Case 2, where variation is minimum) 5. Two prediction bands have been chosen as; i) Planning band: This deviation could be considered as normal (most probable) scenario during Iteration planning, while ii) Risk band: Risk/ Contingency reserve (Scope, Effort etc) may be required during Iteration planning.. For the Planning band; Box Plot 95% CI for has been chosen and for Risk band 1 st Quartile and 3 rd Quartile range has been picked up. Only exception considered for Case 2, as Time series prediction was out of both 95% CI for and 1 st Q & 3 rd Q range. - -5 Case3 5 1 Box Plot - 95% CI for Time Series Descriptive Stats Instance LCL UCL Predicted LCL UCL 1st Q 3rd Q Case 1 2.4 1. 4.8.24-5.23 2.72-1.95 23.7 Case 2 3.48 13.7 5.79 24. -15.78 5.29 17.97 Case 3 -.2 22.3 2.82 9.28-4.3 58.92-7.39 27.1 Figure 14: Dot Plot for Case 1 Predicted Band - Planning Predicted Band - Risk Instance LCL UCL LCL UCL Case 1 2.4 1. -1.95 23.7 Case 2 3.48 13.7-15.78 24. Case 3 22.3-7.39 27.1 Boxplot of Case1, Case2, Case3 Figure 1: Sample selection of predictive bands 1 Data 5-5 -1 Case1 Case2 Case3 Note: In this case, the Planning Band has been considered as Voice of Process and is expected to be factored into normal course of Iteration Planning exercise, while for Risk Band, it would be advisable to plan for additional (contingency) reserve; as it is assumed that process is not matured enough and may still have wide variations in these range(s). Figure 15: Box Plot for Case 1, Case 2 and Case 3 7.2 Account or Customer level High level observations: 1. Case 1 and Case 2: Distribution is not Normal. Guidance is taken from other statistical analysis also. 2. Time Series (Smoothing): LCL and UCL limits are too wide, because of variations in both positive and negative directions. 3. Box Plots provide a good starting point to understand behavioral pattern and setup initial baseline (operating limits). Here, all three (3) projects data was analyzed together and consolidated, for a specific customer; similar process applied on sample outputs (Box Plot, Time Series prediction, Dot Plots and Descriptive Statistics) to come up with desired prediction band. Probability distribution analyzed together with all deviation points (positive and negative) and also all positive and all negative separately to understand distribution patterns. As distribution found not normal, guidance is taken from other statistical analysis.
Summary for Case - 1 DIST A nderson-darling Normality Test Case1 -Burndown Chart Deviation A -Squared 1.32 P -V alue <.5 1 M ean.954-3 -15 15 3 45 S tdev 18.3437 V ariance 33.4929 S kewness -.88774 Kurtosis.495918 N 115 M inimum -38.85 1st Quartile -.792 M edian 4.215 3rd Q uartile 18.392 M aximum 5. 95% Confidence Interv al for 3.578 1.3451 95% Confidence Interval for 2.43 8.4484 95% Confidence Interval for StDev 1.244 21.779 Case1 -Burndown Chart Deviation 5-5 2 4 8 1-1 Overall Dist ST DEV.95 4.21 18.34 95% CI LCL UCL 1st Q 3rd Q 4.21 2.4 8.44 -.7 18.4 Figure 17: Overall distribution (basic statistics) 95% CI LCL UCL 4.54 2.74 1.17 Figure 2: Box Plot statistics 2 Summary for Case1 -Burndown A nderson-darling Normality Test A-Squared 4.8 P-V alue <.5 2.72 StDev 21.53 Variance 42.371 Skew ness 1.59931 Kurtosis 2.5318 N 9 Minimum.33 1st Quartile 4.1 12.917 3rd Q uartile 28.383 4 8 1 Maximum 1. 95% Confidence Interv al for 1.9.2 95% Confidence Interval for Case1 -Burndown 1 5-5 Smoothing Plot for Case1 -Burndown 95.% PI Alpha.2 MA PE 317.981 MA D 21.312 MSD 7.248 1.4 2.9 1. 12.5 15. 17.5 2. 22.5. 95% Confidence Interval for StDev 18.5.21-1 1 13 2 39 52 5 78 91 14 117 Positive Deviations ST DEV 2.7 12.92 21.5 95% CI LCL UCL 12.92 1.4 2. Figure 18: Distribution of +VE deviations (basic statistics) 95% CI Predicted LCL UCL 9.31-42.81 1.52 Figure 21: Time Series (Smoothing) Plot Dotplot of Case1 -Burndown Summary for Case1 -Burndown A nderson-darling Normality Test A-Squared 2.5 P-V alue <.5-22.38 StDev 24. Variance 588.374 Skewness -1.4223 Kurtosis 1.9588 N 3-8 - -4-2 Minimum -85.714 1st Quartile -35.43-12.479 3rd Q uartile -4.315 Maximum -.332 95% Confidence Interv al for -3.5-14.11 - -5 Case1 -Burndown 5 1 95% Confidence Interval for -18.39-9.24 95% Confidence Interval for StDev 19.74 31.41-3 -2-15 -1 Figure 22: Dot Plot (Overall distribution) Negative Deviations ST DEV -22.37-12.48 24.2 95% CI LCL UCL -12.48-9. -18.39 Figure 19: Distribution of -VE deviations (basic statistics) Prediction Band LCL UCL Optimistic Band -9. 1.17 Probable Band - Planning -12.48 12.92 Possible Band - Risk -18.39 2. Figure 23: Probable selection of three possible bands
3 2 1-1 -2-3 Optimistic Band Probable Band - Planning Possible Band - Risk Figure 24: Graphical representation of possible bands Notes: i) Prediction bands, shown here, is indicative only and act as guidance, however, it may vary from project to project, customer to customer and team composition and maturity. ii) Cells are highlighted with different colors to denote sample selection of possible prediction bands. 8 Conclusions This paper focuses on importance of analyzing Burndown chart behavior to come up with possible Prediction Band towards improving better planning and management of Iterations. Each deviation analysis helps us to identify improvement opportunities in scope/ task planning, estimation, competence, impediments, issue resolution, effort distribution, defect prevention, planning risk reserve (additional scope during iteration planning, if early finish or team sits idle etc). 9 References [1] In-house tutorial on Statistical Techniques, Tata Consultancy Services Ltd., India [2] Minitab Tool iii) Box plot is shown to be a good starting point taking guidance from Dot Plot to identify maximum density band. iv) CNTL charts didn t provide useful prediction due to wide moving range variation, as deviation observed both in positive and negative direction, often. v) Here, single exponential smoothing technique is used to observe meaningful prediction (if any), however, other variations of Time Series could also be used and accuracy measures (MAD, MAPE, MSE) could be observ vi) Other predictive analysis techniques also could be explored for meaningful and best-fit outcome observed and found applicable. vii) As we execute more Iteration(s), more Agile projects, calibration is required, as we gather more experience and data points.