A Fool with a Tool: Improving Software Cost and Schedule Estimation

Transcription

1 2006 International Software Measurement and Analysis Conference A Fool with a Tool: Improving Software Cost and Schedule Estimation Ian Brown, CFPS Booz Allen Hamilton

2 A fool with a tool is still a fool. 1

3 Agenda Why Estimate? The Challenges of Software Estimation Benefits of Using a Tool Pitfalls of Using a Tool Improving Software Estimation 2

4 Why Estimate? Why do we need to estimate? Because we need to know how much to budget or to bid Because resources (people) are scarce and we need to know how to allocate them in the best way Because we need to better predict what it will take the next time Staff Level 12 Project Staffing Plan Months Estimate: Hours; Months The purpose of this presentation is to discuss the benefits and pitfalls of software estimation tools as well as other key estimation best practices 3

5 Challenges of Software Estimation Software development projects are notorious for cost and schedule overruns 50% 40% 46% Challenged Projects by Cost Overruns Succeeded 34% Project Resolution by Type Challenged 51% 30% 20% 10% 0% 40% 27% 18% 7% 2% <20% 21-50% % % >200% Challenged Projects by Schedule Overruns Failed 15% Source: The Standish Group, CHAOS Chronicles Version 3.0, % 20% 10% 0% 30% 29% 26% 14% 1% <20% 21-50% % % >200% Some of this problem is due to inaccurate or optimistic cost and schedule estimates up front 4

6 Challenges of Software Estimation Why is software estimation so difficult? Dynamic environment Changing languages and technologies Changing requirements Often treated as a side show Lack of relevant historical data Would you want this guy doing your estimates? Expert judgment approach (and other manual estimation techniques) not usually reliable Complex activities not always factored Challenges of estimating software size 5

7 Benefits of Tools Automated COTS estimation tools leverage large historical data sets and flexible input parameters Examples Tools are parametric in nature, meaning calculations are based on complex statistical algorithms Outputs from model are based on input assumptions Size Personnel skills and experience Development environment Productivity factors Labor rates Estimates can be generated with as much or as little information as is available Tools typically estimate all development life cycle activities, including various levels of testing Tools can handle the complex factors that can impact cost and schedule on a software project 6

8 Pitfalls of Tools The same flexibility that makes estimation tools so useful also provides the opportunity for real problems Estimation tools can produce virtually any number you want them to produce Estimated activities from the tool may not match up with expected activities on the project The path to the Dark Side of estimation tools: Let s just assume a better productivity factor. I m sure we ll be able to reuse most of the software. That estimate does not fit the project budget. We need to change the parameter input assumptions. It s what the tool says, so it must be right. A fool with a tool is still a fool 7

9 Improving Software Estimation Use an estimation tool and understand the potential pitfalls The benefits of using a parametric estimation clearly outweigh the pitfalls If you are aware of the potential dangers, you can avoid them Software development is an inherently complex activity that can be influenced in many ways by many different factors Manual estimation methods simply cannot account for all the possible factors and impacts A tool provides the ability to consider these factors in the estimate and to conduct what if scenarios or sensitivity analysis fairly easily 8

10 Improving Software Estimation It s All About the Process In order to improve, you first must understand where you are In order to understand where you are, you need to set a baseline In order to baseline, you should have a repeatable, standard process The process should be documented and followed Develop work breakdown structure (WBS) Estimate software size Establish key project parameters Assess detailed project parameters If an estimation process is not defined and followed, it is impossible to tell where improvements are even necessary or possible Develop and Perform risk refine cost assessment and model sensitivity analysis Crosscheck and validate estimates 9

11 Improving Software Estimation Measure the Process Baseline the current estimation process: track estimates and collect actual data once project is complete Set improvement goals and establish measurement processes to ensure information is collected and analyzed Root cause analysis and detective work will identify weak areas in the estimation process and may suggest possible improvements Questions: How accurate are our schedule estimates? Schedule Variance Goal: Indicators: How accurate are our effort estimates? Effort Variance Improve Estimation Capability Measures: How accurate are our cost estimates? Cost Variance How accurate are our size estimates? Size Variance Estimated Schedule Actual Schedule Estimated Effort Actual Effort Estimated Cost Actual Cost Estimated Size Actual Size 10

12 Improving Software Estimation Size Matters Size is the most critical driver of cost and schedule on a software project. All other things equal, the larger the size, the greater the effort and cost and the longer the schedule Same design, same construction, but the first is twice the square feet which house will cost more to build? 11

13 Improving Software Estimation Size Matters (cont d) Having a consistent, well-defined, standard size measure as part of the estimation process is critical Use IFPUG function points! End user Application Being Assessed Inputs Outputs Inquiries Internal Data Outputs Inputs Inquiries External Interfaces Other Applications/Systems 12

14 Improving Software Estimation Size Matters (cont d) Recent pilot study in Booz Allen to evaluate function points versus objects as a size measure Track 1 Project Team Track 2 CFPS Estimate size based on object definition Estimate size based on function points Estimate cost based on past experience Estimate cost with parametric tool Analyze estimates versus actuals Average FP estimation variance of 2.83% demonstrates highly reliable estimation methodology Correlation between FP size and actual cost: Release CAERS 2.07 CAERS 3.0 It 1 CAERS 3.0 It 2 Function Points Estimated Cost $361,537 $175,534 $205,703 Actual Cost $359,500 $162,377 $204,800 Variance 0.56% 7.50% 0.43% This strong correlation indicates a strong predictive relationship between FPs and actual cost It also demonstrates the consistency of the FP sizing methodology over multiple releases 13

15 Improving Software Estimation Make sure that estimated activities from the tool match up with actual activities on the project Some activities on the project may not be included in the estimate produced by the tool Training Installation Data cleansing, etc. = Some activities in the estimate may not be part of your project Example: Project contract is for requirements through program test. System test and end user acceptance test are outside the contract scope. Be sure to back this cost and schedule out of the estimate produced by the tool 14

16 Improving Software Estimation Re-estimate throughout the project life cycle Project scope and requirements will change cost and schedule estimates should be reviewed and revised to reflect these changes System Experts Project Manager Function Point Analysis and Parametric Model Documentation Baseline Cost and Schedule Estimates Volatility and scope creep may be factored into original estimate, but the cost, schedule, and size estimates should still be reviewed as part of the configuration control process to ensure the project is still on track Revised Cost and Schedule Estimates Decide which changes to to include in in scope of of project REPEAT Change Requests New Requirements Redefined Requirements Assess Impact to to Cost and Schedule Client Project Team Client Project Manager Project Team 15

17 Improving Software Estimation Don t assume that the tool is necessarily right Review the estimates in the context of the project and other benchmarks. Answering these questions helps to increase the confidence in the estimates Are the estimates consistent with other projects or similar size and nature? Are the estimates consistent with previous organizational experience? How does the projected productivity compare to industry best practice standards? Does the projected staffing plan make sense? Develop a crosscheck estimate with another method or tool If the estimates are wildly different, review and reconcile 16

18 Improving Software Estimation Conduct post-project reviews (PPR) PPRs are critical to improving both the estimation process and the accuracy of future estimates Collect lessons learned, identify weak areas, and suggest potential improvements Compare actuals to estimates (both initial and final estimates) and analyze variance Check size for accuracy Review parameter assumptions Review project activities Calibrate model if appropriate 17

19 OK, I ve heard your little speech now what? Estimation does not have to be a guessing game Leverage the power of estimation tools, but understand the nuances and potential stumbling blocks of doing so Establish an estimation process document the steps and follow them Use function points to size your software as a component of the estimation process Measure - baseline the environment and track improvement over time Don t be just another fool with a tool! 18

20 Contact Information Ian Brown, Certified Function Point Specialist Senior Associate McLean, VA (703)