Quantitative Software Management Using Metrics to Develop a Software Project Strategy Donald M. Beckett QSM, Inc. 2000 Corporate Ridge, Suite 900 McLean, VA 22102 (360) 697-2640, fax: (703) 749-3795 e-mail: don_beckett@qsm.com http://www.qsm.com Quantitative Software Management, Inc. All rights reserved. Slide #1
Outline Overview Measurement, Expense or Investment State of the Industry: Project Estimation Staffing and Schedule Understanding Trade-offs Conclusion Questions? Quantitative Software Management, Inc. #2
Overview Does this sound familiar? Quantitative Software Management, Inc. #3
Measurement: Expense or Investment Software measurement (and process improvement) are viewed as expenses: Overhead Lean, agile organizations want to reduce overhead But, how do organizations become lean & agile? Measurement activities not directly related to revenue generation Part of cost of doing business 3 5% on average Project management averages 14% Quantitative Software Management, Inc. #4
Measurement: Expense or Investment What does software measurement provide? 1. Knowledge of an organization s s capabilities 2. Identifies patterns and trends (Strengths to leverage and weaknesses to correct) 3. Insight into projects in time to make effective mid-stream corrections 4. Ability to benchmark against competition or the industry in quality, productivity, and time to market 5. Quantitative basis for evaluating project and organizational performance Improves ability to meet commitments, avoid pitfalls, and evaluate trade-offs Quantitative Software Management, Inc. #5
State of the Industry: Project Estimation Software estimates are not project plans Estimates contain uncertainty about two key components: Scope of the requirements (project size) Team productivity Estimates define a range of possible outcomes Trade-offs can be examined Impossible solutions identified Quantitative Software Management, Inc. #6
The Cone of Uncertainty Project Definition Not enough information is available early in the development lifecycle to make accurate estimates Precision is not accuracy Low level requirements Quantitative Software Management, Inc. #7
Actual vs. Estimated Effort Effort Growth 60 50 40 Percent 30 20 10 0 Smaller At Size Larger Actual vs. Estimated Effort Quantitative Software Management, Inc. #8
Actual vs. Estimated Schedule Schedule Growth 60 50 40 Percent 30 20 Percent 10 0 Smaller At Size Larger Actual vs. Estimated Schedule Quantitative Software Management, Inc. #9
Actual vs. Estimated Size Size Growth 100 90 80 70 Percent 60 50 40 30 20 10 0 Smaller At Size Larger Actual vs. Estimated Size Quantitative Software Management, Inc. #10
Change Happens Software development is more of a discovery process than a manufacturing one Software projects tend to grow as the implications of requirements are better defined Change must be Managed (Change control process) Anticipated Its implications quantified Quantitative Software Management, Inc. #11
In Summary Average schedule growth is 8% Average cost/effort growth is 16% Average size growth is 15% So how can we use this information to create more accurate estimates? Quantitative Software Management, Inc. #12
Modeling Increased Size Create best project estimate based on proposed size Use historically based productivity Account for project constraints (staff, effort, schedule) Create estimate based on 15% size growth Does this account for projected schedule & effort growth? Quantitative Software Management, Inc. #13
500 FP Project Staffing & Probability Analysis 1 2 Avg Staff Life Cycle (people) <500 FP project> 3 4 5 6 7 8 7 6 M ilestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R 10-99.9R 5 4 3 2 1 Avg Staff Life Cycle (people) Oct '07 1 2 3 4 5 6 7 8 9 10 11 Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep '08 0 Duration Effort Cost Peak Staff MTTD Start Date SOLUTION PANEL - <500 FP project> C&T Life Cycle 6.7 9.4 29 37 493.4 643.6 5.7 5.7 1.823 1.823 12/23/2007 10/1/2007 PI=16.5 M BI=3.6 Eff FP=500 Months PM $ (K) people Days 9.4 months duration 37 person months effort 50% probability Quantitative Software Management, Inc. #14
500 FP Project Evaluate Probability of Current Estimate Life Duration (Months) Risk Profile <500 FP project> 13 Life Effort (PM ) Ris k Profile <500 FP project> 100 12 80 11 10 9 Life Duration (Months) 60 40 Life Effort (PM) 8 20 7 0 10 20 30 40 50 60 70 80 90 100 Assurance Level (%) Life Duration (Months) Risk Profile - Probability demo <500 FP project> 0 0 10 20 30 40 50 60 70 80 90 100 Assurance Level (%) Likely outcomes 10.2 months schedule, 43 effort months Life Effort (PM) Ris k Profile - Probability demo <500 FP project> Assurance Level (% ) 1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 Life Duration (Months ) 7.26 7.90 8.25 8.48 8.66 8.82 8.96 9.09 9.21 9.33 9.45 9.57 9.69 9.81 9.94 10.08 10.24 10.42 Assurance Level (% ) 1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 Life Effort (PM) 0.00 5.11 12.20 16.99 20.79 24.05 26.98 29.69 32.27 34.75 37.20 39.65 42.13 44.71 47.42 50.35 53.61 57.41 Project: Probability demo Quantitative Software Management, Inc. #15
15% Growth (575 FP) Staffing & Probability Analysis 1 2 Avg Staff Life Cycle (people) <15% size growth> 3 4 5 6 7 8 8 Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R 10-99.9R 6 4 2 Avg Staff Life Cycle (people) Oct '07 1 2 3 4 5 6 7 8 9 10 11 Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep '08 0 SOLUTION PANEL - <15% size growth> Duration Effort Cost Peak Staff MTTD C&T 7.3 35 603.7 6.5 1.681 Life Cycle 10.2 46 787.5 6.5 1.681 Months PM $ (K) people Days Start D a te 12/28/2007 10/1/2007 PI=16.5 MBI=3.4 Eff FP=575 10.2 months duration 46 person months effort Quantitative Software Management, Inc. #16
15% Growth (575 FP) Evaluate Probability of Current Estimate Life Duration (M onths) Risk Profile <15% size grow th> 14 L ife Effo r t (PM ) Ris k Pr o file <15% size grow th> 120 13 100 12 11 10 Life Duration (Months) 80 60 40 Life Effort (PM) 9 8 20 7 0 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 A ssurance Level (% ) A ssurance Level (% ) Life Duration (Months) Risk Profile - Probability demo <15% size growth> Life Effort (PM) Ris k P rofile - Probability demo <15% size growth> Assurance Level (% ) 1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 Life Duration (Months ) 7.83 8.51 8.88 9.12 9.32 9.48 9.63 9.77 9.91 10.03 10.16 10.29 10.41 10.55 10.69 10.84 11.00 11.20 Assurance Level (% ) 1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 Life Effort (P M) 0.00 8.61 16.76 22.27 26.64 30.39 33.76 36.89 39.85 42.71 45.52 48.33 51.19 54.15 57.28 60.65 64.40 68.77 Project: Probability demo Averages close to numbers predicted for effort and schedule growth (10.2 duration and 43 staff months of effort) Quantitative Software Management, Inc. #17
Staffing & Schedule Validate Estimate with History C&T Duration (Months) vs Effective FP 100 C&T Effort (PHR) vs Effective FP 1,000 Schedule varies by a factor of 3.5 from +1σ to -1σ Effort varies by 100a factor of 8 from -1σ to +1 σ 10 C&T Duration (Months) 10 C&T Effort (PHR) (thousands) 1 1 100 1,000 Effective FP 0.1 100 1,000 Effective FP Current Solution Historical Projects QSM 2005 Business FP Avg. Line Style 1 Sigma Line Style Project: Untitled What is normal variability? Quantitative Software Management, Inc. #18
How Should Project Effort Be Expended A Case Study 838 projects that had data reported for Analysis/Design as well as Construction and Test phases Average Effort applied to Analysis/Design = 20% 474 projects in the sample used <= 20% analysis/design effort Average Analysis/Design Effort = 11% 364 projects in the sample used > 20% analysis/design effort Average Analysis/Design Effort = 33% Size profiles of samples very similar Quantitative Software Management, Inc. #19
Observations Projects with <20% effort in Requirements and Design Took 12% longer to complete Averaged 5.6% more effort (median 24.4% greater) Had an average staff 14.6% higher But these projects did excel at one thing: Found 63.7% more defects in systems test Had 127% more defects in the first 12 months after delivery Quantitative Software Management, Inc. #20
Understanding Trade-offs a b Size = Effort Time Productivity 1 b 4 3 3 where a and = = Additional schedule has a much larger impact on a software project than increased effort Quantitative Software Management, Inc. #21
The Estimating Conundrum Schedule / Effort Tradeoff Impossible Zone Uncertainty about Size and Productivity creates uncertainty about the Duration-Effort curve Effort Feasible Solutions Impractical Zone Duration Quantitative Software Management, Inc. #22
Conclusions Measurement is an integral part of management Information required to make precise estimates is unavailable at project start-up Estimate uncertainty decreases rapidly with more information Project estimates historically understate effort, schedule, & size Estimating based on a larger size or at a higher assurance level can account for this The trade-off between schedule & cost/effort is non-linear Quantitative Software Management, Inc. #23
Conclusions Effort spent in Analysis & Design pays big dividends Reduces overall project effort (cost$$$$) Reduces overall project schedule Improves project quality Quantitative Software Management, Inc. #24
Questions? Quantitative Software Management, Inc. #25