Quantitative Cost and Schedule Risk Assessment and Management for Large Infrastructure Projects RT-05. Bill Roberds, Sc.D. Travis McGrath, Ph.D., P.E.

Transcription

1 Quantitative Cost and Schedule Risk Assessment and Management for Large Infrastructure Projects RT-05 Bill Roberds, Sc.D. Travis McGrath, Ph.D., P.E. PMI is a registered trade and service mark of the Project Management institute, Inc. Golder Associates, Inc. Redmond, WA

2 Introduction Cost and Schedule overruns are common on large infrastructure projects. Historically 1 : 90% of estimates for transportation infrastructure projects have been low On average, cost estimates for road projects have been 20% short of final costs Estimates for some projects have been off by 100% or more (e.g., the Big Dig ) 1. Flyvbjerg et.al., Journal of the American Planning Association, Summer,

3 Project Cost and Schedule Uncertainty Poor cost and schedule estimates lead to: Cost and schedule over-runs or under-runs Poor decisions (i.e., among alternatives) Resource competition among projects Media attention Public mistrust (and therefore funding problems) 3

4 Probabilistic Cost and Schedule Risk Assessment and Risk Management (RA/RM) Mitigate these problems by: Quantifying project cost and schedule uncertainty Identifying and prioritizing critical risks and key opportunities Quantifying potential benefits of riskmanagement strategies Improving project understanding and communication Increasing confidence in estimates and budgets 4

5 Outline Key Concepts Process and Examples Performance to Date Challenges for Better Implementation 5

6 Key Concepts Comprehensive look at the project at appropriate level of detail Quantify uncertainty in key project assumptions Utilize independent perspective (for validation) Employ a collaborative, team approach Achieve consensus (when possible) Focus attention on the key issues Update as the project changes significantly 6

7 Key Concepts, cont d Quantify uncertainty in project cost and schedule, which is a function of: Sequence of all significant project activities Base activity costs and durations Risks, Opportunities, and other uncertainties Prioritize critical activities, risks, and opportunities Identify and evaluate strategies to improve project performance 7

8 Describing Uncertainty Relative Likelihood Mean (Expected) Value Range in values Standard Deviation Cumulative Probability 100% 75% 50% 25% 0% Percentiles 10th 90th 80% confidence interval Value Probability Density Function (PDF) Cumulative Distribution Function (CDF) 8

9 Base + Risk Approach versus Traditional Single-Valued Approach Aggregate Risk and Opportunity Relative Likelihood Base (with uncertainties) Base + Risk 0 Traditional Contingency below the line + = Cost Traditional Traditional Conservative Single-valued above the line (Deterministic) Estimate 9

10 Outline Key Concepts Process and Examples Performance to Date Challenges for Better Implementation 10

11 Steps in Quantitative Cost and Schedule Risk Assessment and Risk Management Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Risk Register Workshop setting Develop Base Factors Develop Cost and Schedule Uncertainty Model Assess Risk Factors Collaborative effort Quantify Uncertainty and Sensitivity in Cost, Schedule, and Other Measures Identify and Evaluate Risk- Management Strategies Update Periodically or at Key Milestones Project team implements mitigation strategy and monitors performance 11

12 Steps: Structuring the RA/RM Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Project team presents current plan Clarify project scope and delivery strategy List all significant project assumptions Develop project flow chart for each alternative: Sequence of major project activities (all precedence requirements) Based on team s strategy and schedule Reasonableness / accuracy check on schedule logic Appropriate level of detail Basis for integrated cost and schedule model 12

13 Example Project Flow Chart Pre-Construction Activities Programmed: 1/1/2005 Develop Alternatives Design 2 and ROW Funding Env Resource 1 Inventory 3 Design to 10% 4 S+50% Design to 30% + Design to 60% Design Approval 8 7 F-1 month Prepare Environmental Resource Doc + Approval Reports 6 5 Finalize Design and PS&E 9 S+50% Relocate Utilities 10 Acquire ROW 11 Permitting 12 Ad / Bid / Award / NTP Construction Activities Widen Existing Embankment 15 Construct New 14 Embankment 16 Paving 20 Open to Traffic 21 Demo Old Bridge 22 Programmed: Construction 1/1/2007 Funding 13 Remove Old Close-out Alignment Items Complete 25 Bridge Approaches 17 Remaining Bridge Construction 19 S + 1 month In-Water Bridge Construction 18 Critical path 13

14 Steps: Reviewing Project Estimates Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Collaborative effort Confirm that cost matches scope Confirm that schedule matches scope Check reasonableness of design and strategy Identify and remove contingency and conservatism Note: Estimate is for the ultimate cost and schedule 14

15 Steps: Defining the Base Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Base Factors The Base The planned scope and strategy, with no problems Unbiased, excluding contingency and/or float Map the base estimates to flow chart activities For each flow chart activity, develop: Base cost (current $) Base duration (calendar time) Cost-escalation rate Include significant uncertainties and correlations 15

16 Example Base Factors (excerpt) Flowchart Number Project Activity Base Cost 1 (2005 $M) Base Duration 1 (months) Average Escalation Rate 1 (%/yr) 0 Costs to Date Design and ROW Funding milestone - 2 Develop Alternatives Env. Resource Inventory Design to 10% Prepare Resource Reports Environmental Doc + Approval Design to 30% + Approval Ad / Bid / Award / NTP Widen Existing Embankment Construct New Embankment Bridge Approaches In-Water Bridge Construction Remaining Bridge Construction Close-out Items Complete milestone - Note 1. May be uncertain and described with probability distributions; only the expected values are shown here. 16

17 Steps: Developing the Risk Register Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Risk Register Develop Base Factors Identify all potential events - risks and opportunities - outside the base assumptions Comprehensive and non-overlapping Appropriate level of detail Include significant relationships among risks Consider possible causes and effects 17

18 Steps: Quantifying the Risks Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Risk Register Develop Base Factors Assess Risk Factors For each risk and opportunity, quantify: Consequences if event occurs Changes in costs and durations to affected activities Other measures? Likelihood of occurrence, consistent with consequences Include significant uncertainties and correlations Ensure consistent with base 18

19 Steps: Quantifying the Risks, cont d Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Risk Register Develop Base Factors Assess Risk Factors Risk-factor assessments: Based on objective analysis when data are available Based on subjective assessments (from the experience and judgment of SMEs) if data are limited When properly elicited, subjective assessments have been shown to be accurate, defensible, and the best we can do 19

20 Example Event Tree for Structuring Risks and Opportunities Structure Type A. Cut-and-Cover Probability $ ($M) Months P A 1 P 0 ( base ) 1 0 Vertical Concept & Location 1. Tunnel P 1 2. At-grade P A 1 B. Bored / Mined P B 1 = 1- P A 1 P B 1 P P P 2 3. Bridge / Aerial C. Concrete Box P C 3 P C 3 P P 3 = 1-P 1 -P 2 D. Steel Plate Girder P D 3 = 1 - P C 3 P D 3 P Note: Probability of alternative concepts may consider factors other than cost or duration 20

21 Example Risk Register and Risk Factors (excerpt) Item D8 P4 Risk or Opportunity Project alignment different than assumed Includes related design, ROW, and construction impacts Potential mutually-exclusive outcomes include: A. EIS alignment ( base ), which passes through town on existing roadway alignment B. Alternative alignment #1, which bypasses town to north on new alignment C. Alternative alignment #2, which passes through town on new alignment to east of EIS alignment Delays in required transportation agency approvals Several agencies have review and approval authority over the design. Delays in approval are possible from delays in review cycle. Affected Project Activities 60% Design and ROW Acquisition 60% Design Probability of Occurrence Discrete distribution: A. 40% (Base) B. 40% C. 20% Cost change perfectly correlated to duration change Discrete distribution: A. 10% (Base) B. 60% C. 30% Cost Change (2005 $M) A. 0 (Base) B. 1.0 to Design; -5 to ROW C. 1.0 to Design; Uniform (2, 10) to ROW Additional escalation costs are simulated as a function of duration change Duration Change (months) A. 0 (Base) B. 6 to Design; -3 to ROW C. 6 to Design; Uniform (3, 9) to ROW Discrete distribution: A. 0 (base) B. 3 C. 6 21

22 Steps: Developing the Integrated Cost and Schedule Model Probabilistic computer model Can include Shut-down windows Decisions and other triggered actions Funding or other resource constraints Uncertainty in critical path Delay-related costs Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Base Factors Develop Cost and Schedule Uncertainty Model Develop Risk Register Assess Risk Factors 22

23 Steps: Quantifying Uncertainty in Project Cost and Schedule Combine base and risk factors in the model Simulate results of interest Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Base Factors Develop Cost and Schedule Uncertainty Model Develop Risk Register Assess Risk Factors Quantify Uncertainty and Sensitivity in Cost, Schedule, and Other Measures 23

24 Integrated Cost and Schedule Model _ Event X Uncertainty in Total Escalated Project Cost and Duration, plus Sensitivities Start Activity A Activity C Activity B End $ time T escalation cash flow Activity Base Costs (and Uncertainties) Activity Base Durations (and Uncertainties) Risk Events May (likelihood 9, 2006 of occurrence, and uncertainty Event _ Y in activity cost and duration changes if the event occurs) 24

25 Example Monte Carlo Sample Set Showing Cost and Schedule Correlation 4th Bore Opening Date Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Total Project Cost ($M) unescalated escalated 25

26 Cumulative Probability (Percentile) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Example CDF for Project Cost Showing Contributing Factors 2,400 Base (with no risk or opportunity; in 2005 $) With Risk and Opportunity, but Without Escalation (2005 $) With Risk and Opportunity and Escalation (Year-of-Expenditure $) With Risk and Opportunity and $400M/year Funding Constraint (YOE $) Mean risk and opportunity 2,500 2,600 2,700 2,800 2,900 3,000 Mean cost escalation 3,100 3,200 3,300 3,400 3,500 3,600 3,700 3,800 3,900 4,000 4,100 Mean funding constraint 4,200 4,300 4,400 4,500 Total Project PMI is a registered Cost trade and (2005 service mark of $Million the Project Management or institute, YOE Inc. $Million) 26

27 Example CDF for Project Schedule Showing Contributing Factors Cumulative Probability (Percentile) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Jan 2015 Base (with no risk or opportunity) With Risk and Opportunity but Without Funding Constraint With Risk and Opportunity and $400M/year Funding Constraint Mean risk and opportunity Jan 2016 Jan 2017 Mean impact of funding constraint Jan 2018 Jan 2019 Jan 2020 Jan 2021 Jan 2022 Jan 2023 Jan 2024 Overall Project Completion Date 27

28 Example Probability of Meeting Specified Target Cumulative Probability (Percentile) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 2,400 Base (with no risk or opportunity; in 2005 $) With Risk and Opportunity, but Without Escalation (2005 $) With Risk and Opportunity and Escalation (Year-of-Expenditure $) With Risk and Opportunity and $400M/year Funding Constraint (YOE $) 2,500 2,600 2,700 Probability of meeting $3.7B budget = 85% Probability of meeting $3.7B budget = 37% 2,800 2,900 3,000 3,100 3,200 3,300 3,400 3,500 3,600 3,700 3,800 3,900 Alternatively, could identify required budget for desired confidence level (e.g., 80 th percentile) 4,000 4,100 4,200 4,300 4,400 4,500 Total Project PMI is a registered Cost trade and (2005 service mark of $Million the Project Management or institute, YOE Inc. $Million) 28

29 4,000 Example Uncertainty in Cash Flow of Expenditures (Unconstrained Funding Scenario) Project Cash Flow Without $400M/year Funding Constraint (YOE $M) 3,500 3,000 2,500 2,000 1,500 1, Cumulative - 90th Percentile Cumulative - Mean Cumulative - 10th Percentile Annual - 90th Percentile Annual - Mean Annual - 10th Percentile Year 29

30 Example Impact of Scenario Assumptions on Uncertainty in Cash Flow Annual Cashflow (YOE $M) Constraint reduces peak expenditure by $270M at the mean Constraint delays peak expenditure by 5 years Year No Funding Constraint - 90th Percentile No Funding Constraint - Mean No Funding Constraint - 10th Percentile Funding Constraint - 90th Percentile Funding Constraint - Mean Funding Constraint - 10th Percentile However, constraint increases total cost significantly (increased schedule and thus escalation) 30

31 Steps: Quantifying Sensitivity in Project Cost and Schedule Evaluate sensitivity of cost and schedule to the uncertain base and risk factors: Mean Standard Deviation Percentile (combination) Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Base Factors Develop Cost and Schedule Uncertainty Model Develop Risk Register Assess Risk Factors Prioritize critical activities and risks in costs and time Quantify Uncertainty and Sensitivity in Cost, Schedule, and Other Measures 31

32 Example Sensitivity of 80 th Percentile of Project Cost to Uncertain Inputs Uncertain Factors (base uncertainty or risk/opportunity) Contribution ($M) to 80th Percentile of Escalated Project Cost B.3. Limited Number of Qualified Bidders D.26. Extended Pre-Construction Admin Costs C.35. Extended Construction Admin Costs 16.3 Identified Minor Risks (aggregate) 11.8 Unidentified Risks (aggregate) 11.7 E.1. Challenge to ROD/FONSI due to public opposition 7.8 C.25. Unanticipated Mitigation of Construction T raffic 6.7 B.11. Non-Responsive Low Bid 6.5 D.7. Unanticipated Mitigation for Impacts Outside of... 6 C.14. Valid Differing Site Conditions (DSC) (e.g., C.7. Inexperienced Contractor (Separate from C D.5. More Fire Protection / Ventilation Required 3.9 D.14. Conversion from Metric to English Units 3.8 C.5. Contractor Unfamiliarity with CalT rans Admin W Tunnel 2.9 C.15. T unnel Face Instability / Excessive Inflow 2.4 E Tunnel 2 C.29. Unanticipated Work Stoppage due to Adverse E.5. Unexpected / Late Number of Public Comments B.6. Poor Material Prices and Availability 1.7 C.20. Slope Instability / Retaining Wall Failure 1.6 B.5. Opposition from Local Contractors / Business D.16. Problems in Design Coordination (Including D.17. Design Changes / Breakage (Separate from E13. Additional Impact Analysis 1.1 August 2005$ / non-row escalation rate 1.1 C.16. T unnel Collapse During Construction D.4. Change in Design Standards (e.g., Seismic)

33 Steps: Managing Risk Identify potential riskmanagement actions (e.g., prevention; mitigation) Quantify likely implementation costs Quantify likely benefits Simulate net effect Ties in to Value Engineering Structuring: Review Project Plan, Develop Flow Chart, and Document Assumptions Review Cost and Schedule Estimates Develop Base Factors Develop Cost and Schedule Uncertainty Model Quantify Uncertainty and Sensitivity in Cost, Schedule, and Other Measures Identify and Evaluate Risk- Management Strategies Develop Risk Register Assess Risk Factors Project team implements mitigation strategy and monitors performance 33

34 Risk Management Relative Likelihood Risks and Opportunities with Risk Management Base (with uncertainties; now including RM costs) Total with Risk Management Performance (e.g., cost) Note: Base costs include mitigation implementation costs 34

35 Example Risk Management: Mitigated Cost Uncertainty (if Mitigation Succeeds) Probability Mitigated (Workshop Plan) Unmitigated (July ) The mitigation plan would save $10M at the owner s desired confidence level (80 th percentile) Total Project Cost (Future $M) 35

36 Example Risk Management: Mitigated Schedule Uncertainty July 2004 Mitigated July 2004 Unmitigated Cumulative Probability (Percentile) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Jun 2008 Sep 2008 Dec 2008 Mar 2009 Jul 2009 Oct 2009 Jan 2010 May 2010 The mitigation plan would save 7-8 months at the owner s desired confidence level (80 th percentile) Revenue Service Date 36

37 Outline Key Concepts Process and Examples Performance to Date Challenges for Better Implementation 37

38 Performance Favorable reaction from public (i.e., public ok with ranges expressing uncertainty) Improved project team understanding Focused risk management Improved confidence in cost and schedule estimates, however Not many projects evaluated using this approach have been completed Can t fully evaluate performance (yet) As an interim measure, compared results from risk assessments to published costs for completed projects 38

39 Performance: Bias Comparisons Flyvbjerg et. al. (2002) calculated inaccuracy (bias) in cost estimates for completed projects as the percent difference between actual (ultimate) project cost and estimated cost at the time of decision to build As an interim measure, the authors calculated the apparent bias in project team estimates relative to the mean RA/RM results for projects still in design as follows: [mean unescalated cost from risk assessment team cost estimate (unescalated)] team cost estimate (unescalated) 39

40 Performance: Results from Risk Assessments vs. Published Results for Completed Projects Frequency From 62 Risk Assessments (2002 to 2005) on Northwestern US Road Projects After Flyvbjerg et. al. (2002) for 167 Completed Road Projects (Worldwide) Mean = 20% Standard deviation = 30% Mean = 14% Standard deviation = 24% Bias in Cost Estimate (%) 1 (numbers are upper bounds for bins) Similar partial validation of approach Note: Larger variability in completed projects likely reflects larger data set and variability among agencies 40

41 Performance: Apparent Bias in Cost Estimate versus Design Level Bias in Cost Estimate (%) From 62 Risk Assessments on Northwestern US Road Projects Mean plus one standard deviation Mean Mean minus one standard deviation Conceptual Preliminary Final Design Level at Time of Risk Assessment 41

42 Outline Key Concepts Process and Examples Performance to Date Challenges for Better Implementation 42

43 Challenges for Better Implementation If properly implemented and averaged over many projects, risk-based approaches appear to get closer to the ultimate project cost than does deterministic estimating, However, RA/RM is not perfect: Bids for some recent projects evaluated using probabilistic RA/RM (by the authors and by others) have fallen outside the RA/RM ranges Effort for RA/RM for some projects has been excessive This implies that probabilistic RA/RM still needs improvement 43

44 Challenges for Better Implementation 1. Convincing the owner that the RA should include all significant uncertainties Excluding major uncertainties makes the results: Conditional on key assumptions (often forgotten) Underestimate the true range of uncertainty, and perhaps inaccurate as well Examples of major uncertainties that are often excluded for political or other reasons: Uncertainty in the amount or timing of funding Potential delays due to process and management inefficiencies Uncertainty in key policies within stakeholder orgs Uncertainty in project scope (i.e., evaluating a particular scope instead of all possible project scopes) 44

45 Challenges for Better Implementation 2. Removing optimism and/or management bias (e.g., hidden agendas) from assessments Consensus is desirable, but should not be achieved at the expense of accuracy If necessary, risk assessors should infuse their experience from other projects to de-bias and/or expand ranges of critical assessments to ensure reasonable results 45

46 Challenges for Better Implementation 3. Making results of RA/RM understandable and useful to the owner Make sure the RA addresses the owner s specific needs Educate the owner on how to interpret and use the results Don t let the owner assign more certainty or precision to the results than is justifiable Qualify any limitations to avoid misunderstanding 46

47 Summary Probabilistic risk assessment and risk management for large infrastructure projects can costeffectively provide Better understanding and communication of project uncertainties More confidence in estimates Better project management and decision making However, careful (and better) implementation is required to ensure accurate and defensible results 47

48 Contact Information Dr. Travis McGrath, P.E. Golder Associates, Inc NE Union Hill Rd, Suite 200 Redmond, WA (425)