Assessment of Transport Projects: Risk Analysis and Decision Support Assistant Professor Kim Bang Salling Presentation at the RISK Conference 2011: December 1 st 2011 DGI Byen, Copenhagen
Outline Background Introduction Methodologies Cost Benefit Analysis (CBA) Quantitative Risk Analysis (QRA) Feasibility Risk Assessment (FRA) Accumulated Descending Graphs (ADG) The UNITE-DSS Decision Support Model Uncertainties in Transport Project Evaluation Conclusion Perspective 2
Background The Manual for socio-economic analysis in the transport sector (2003) Unique guidelines for evaluating transport infrastructure projects Lack of uncertainty handling Expected revision 2012-2013 Building decision support with a twist Rational decision making involves the assessment of both the benefits and the losses (costs) The need for making good decisions in transport planning and evaluation are vital 3
Transport Planning and Assessment Ongoing transport planning: Research: - Societal goals as, for example networks and mobility, sustainable development, etc. - Prognoses/ forecasts Transport infrastructure project proposal Traffic models Decision support Cost-benefit analysis (CBA) - Concepts as for example Feasibility Risk Assessment (FRA) and Accumulated Descending Graphs (ADG) - The CBA-DK model and @RISK software - Urban & regional planning Impact models Multi-criteria analysis (MCA) - Case examples related to different modes - Design standards, etc. - Findings and recommendations 4
Introduction CBA & MCA produce single point estimates Informativ decision support Feasibility Risk Assessment (FRA) Accumulated Descending Graphs (ADG) Normally, uncertainties are handled by sensitivity tests Historical overview of uncertainties Construction cost overrun Traffic forecast underrun (traffic modelling) 5
Suez Canal Sydney Opera House Concorde Supersonic Aeroplane Boston's Artery/Tunnel Project, USA Humber Bridge, UK Boston- Washington- New York Great Belt Rail Tunnel, DK A6 Motorway Chapel-en-le- Frith/Whaley Shinkansen Joetsu Rail line, Japan Washington metro, USA Channel Tunnel, UK & France Karlsruhe- Bretten light rail, Germany Øresund Access links, DK & Sweden Mexico city metro line, Mexico Paris-Auber- Nanterre rail line, France Cost Overruns (%) Cost Overrun (%) Construction Cost Overruns (fixed prices) Construction cost overruns Construction Cost Overruns 2000% 100% 1800% 90% 1600% 80% 1400% 70% 1200% 60% 1000% 50% 800% 40% 600% 400% 200% 0% 30% 20% 10% 0% Channel Tunnel, UK & France Øresund Access links, DK & Sweden Great belt link, DK Øresund coast-tocoast link, DK & Sweden 6
Cost-Benefit Analysis (CBA) Method for evaluating the goodness of investments A systematic approach in listing costs and benefits Selection of the best performing alternative(s) Inputs derived from a lot of external sources Traffic models and impact models Key figue catalogues Output based upon single point criteria Net present value (NPV) Benefit cost ratio (BCR) Transferred model uncertainties!?!? 7
Uncertainty in transport appraisal Unit price principles are assumed certain Two types of impacts stands out: Travel time savings -> Benefit Construction costs -> Cost Unit Pricing Principles Sources of Uncertainty Literature supports the latter impacts by the so-called: Optimism Bias Relies on the key figure catalogue in calibrating and determining unit price settings. Model Uncertainty Relies on the model build up of impact and traffic models that provide the input towards decision support models. Randomness of the system Lack of knowledge 8
Optimism Bias and Reference Class Forecasting The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004) Reference Class Forecasting: Optimism Bias Inside View Outside View Uniqueness of Project Reference Class Forecasting Forecasting of particular projects Forecasting from a group of projects The Planning Fallacy (1) Identification of relevant reference classes (2) Establishing probability distribution (3) Placing and comparing the project Current Situation Optimism Bias Uplifts 9
Optimism Bias and uplifts Deriving uplifts is highly dependet on large data-sets Flyvbjerg from (AAU) has since 2003 developed a large database Unfortunately, it looks upon mega-projects Uplift values were derived on basis of Reference Class Forecasting i.e. statistical measurements on various project pools Applying uplifts still produces single point rate of returns BUT, the data collected can be transformed and used in another way. Risk Analysis 10
Risk Control Infrastructure assessment TRANSPORT INFRASTRUCTURE PROJECT: General information (Technical, political, economical, etc.) SAFETY SOCIETAL GOALS: PHILOSOPHY: Definition of goals, fundaments for priorities and standards ACCEPTANCE CRITERIA: Societal acceptance, budgetary constraints etc. Appraising the information brought above RISK ANALYSIS (TRADITIONAL): RISK IDENTIFICATION: Definition of risk components - impacts RISK ASSESSMENT: Describe and quantify risk by evaluation RISK EVALUATION: Compare risk to acceptable standards 11
Monte Carlo Simulation 12
Input Distributions Distinction between non-parametric and parametric Non-parametric is used when experts have to make the judgments Parametric are used when data and/or theory underpins the judgments Non-Parametric distributions: Uniform Triangular/Trigen Parametric distributions: Normal Erlang (Gamma) > Construction Cost PERT (Beta) -> Travel time savings 13
Level of Knowledge (LoK) The LoK ranges from low to medium to high Distinction between Parametric and Non-Parametric distributions 14
PERT Distribution Based upon a beta distribution with the assumption that the mean can be derived from: Min Mode Max Min 4 Mode Max Mean Mean PERT Triang vs 6 3 This makes it ideal for modelling experts opinion Stands out compared to the Triangular distribution Triangular Beta-PERT 15
Data fit (Rail) Demand forecasts Demand forecasts (user benefits) are set against prior Reference classes derived from Flyvbjerg et al. (2003) 27 rail projects were compared where the inaccuracy on average were 39% lower than predicted I have fitted a PERT curve around the data from Flyvbjerg et al. (2003) 16
-150% -125% -100% -75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% Data fit (Road) Demand forecasts 183 road projects were compared where the inaccuracy on average were 9% lower than predicted 5,0% Fit Comparison for Inaccuracy in Traffic Forecasts RiskPERT(-78.5;9.6%;179.34%) -0,487 90,0% 1,057 5,0% Input Beta-PERT 17
Erlang Distribution Based upon a gamma distribution defined upon a shape and a scale parameter (k, ) The shape parameter, k, depicts the skewness of the distribution whereas the scale,, is based upon data 2 1.5 K=2 K=5 K=10 K=20 1 0.5 0 0 0.5 1 1.5 2 2.5 3 18
Data fit (Rail) Investment costs Flyvbjerg et al. Compared 58 rail projects Approximately 88% of the probability mass is above 0 which indicates that rail type projects are underestimated The fitted probability distribution contributes to the fact that an Erlang distribution is very well suited 19
-100% -75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250% Data fit (Road) Investment costs 167 road projects were compared where the inaccuracy on average were 20% lower than predicted, with k = 8 5,0% Fit Comparison for Cost Overrun for Road Projects RiskErlang(8;0.09) -> (-33.6%;20.2%;222.6%) -0,156 90,0% 0,569 5,0% Input Erlang 20
Recommendation High level of knowledge Risk analysis in decision support: Combination of data from Flyvbjerg, Successive Principle and Risk Analysis: Large-scale implementation in UNITE Definition of distributions Empirical data to feed the distributions Assigning probability distributions: Investment Cost Gamma (Erlang) distribution Travel Time Savings Beta (PERT) distribution Mode Impact Distribution Low High Rail Travel time savings PERT -90% 140% Rail Construction cost Erlang (k = 23) -40% 120% Road Travel time savings PERT -80% 180% Road Construction cost Erlang (k = 8) -30% 120% A negative sign for travel time savings means that benefits have been overestimated and a negative sign for construction costs means that costs have been underestimated 21
Uncertainties in Transport Project Evaluation (UNITE) Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages (5) Evaluation methodology WP5 project leader: Steen Leleur (DMG) (3) Uncertainty calculation of cost estimates WP3 project leader: Bo Friis Nielsen (DTU Informatics) (4) Uncertainty calculation in transport models WP4 project leader: Otto Anker Nielsen (TMG) (2) Organizational context of Modelling, an empirical study WP2 project leader: Petter Næss (AAU) (1) Systematic biases in transport models (recognized ignorance), an empirical study WP1 project leader: Bent Flyvbjerg (Oxford University) 22
The Case Study: HH-Connection Connecting Denmark with Sweden: Scandinavian link Currently, close to the capacity limit on Oresund HH-Connection (alternatives) Description (Alignment of connection) Cost (million DKK) Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700 Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500 Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500 Alternative 4 Bridge for road (2x2 lanes) 6,000 Note! 1 7.5 DKK 23
The UNITE DSS Modelling Framework The UNITE-DSS Decision Support Model for Risk Assessment Determinstic Calculation Stochastic Calculation I) Cost-benefit analysis III) Reference Class Forecasting IV) Reference Scenario Forecasting Results: Point estimates in terms of NPV, BCR, IRR Impact: Travel time savings Impact: Travel time savings II) Optimism Bias Uplifts Determination of Beta-PERT distribution Determination of scenarios and triple estimates Impact: Investment costs Determination of inputs to the Beta-PERT distribution Trtiple estimate parameters to the Beta-PERT distribution Results: Point estimates in terms of NPV, BCR, IRR Results: Certainty graphs and certainty values Results: Certainty graphs and values for scenarios 24
Deterministic Module Entry data 25
Results: Cost-Benefit Analysis HH-Connection (alternatives) Cost (million DKK) BCR NPV (million DKK) Alternative 1 7,700 1.50 5,530 Alternative 2 5,500 0.16-6,640 Alternative 3 11,500 2.71 28,240 Alternative 4 6,000 3.08 17,860 Construction costs by far the largest contributor of costs User Benefits by far the largest contributor of benefits Consists of Ticket revenue and time savings Relies on the prognosis of future number of passengers i.e. demand forecasts 26
Results : Optimism Bias Uplifts HH-Connection (alternatives) Cost (uplifted) (million DKK) BCR (orig.) (from slide 8) BCR (uplifts): 80% uplift Alternative 1 12,090 1.50 0.97 Alternative 2 8,640 0.16 0.10 Alternative 3 15,180 2.71 1.75 Alternative 4 7,920 3.08 1.98 The BCR are lower, however, still point estimates towards DM Moreover an advanced form of sensitivity analysis Imply to introduce risk analysis and Monte Carlo simulation 27
Stochastic module - @RISK The UNITE-DSS model is assigned an add-on software model named @RISK A range of distribution functions are shown Two non-parametric distrbutions have been tested/applied (green) Three parametric distributions have been tested/applied (orange) 28
Input in UNITE-DSS Construction cost Shape parameter k = 8 for road projects and k = 23 for rail projects (including air) The mean ( ) and standard (std) deviation is calculated k The scale parameter ( ) is calculated on basis of the succesive principle 29
Results (RCF): Monte Carlo simulation 30
Reference Scenario Forecasting Accomodates scenario analysis and RCF Vertical regime: Economic development due to link Horizontal regime: Integration between borders 31
Results from RSF 32
The coupling of methodologies in achieving feasibility risk assessment 33
Conclusions The UNITE-DSS model has been developed and functions as a flexible assessment tool applicable for wider risk oriented assessment for transport projects across different modes. The developed type of accumulated descending graph is found to be useful to inform about uncertainty relating to assessment of transport projects. Dependent on the information available parameter-based or parameter-free input probability distributions should be applied. It is possible to accommodate the recent results stemming from Optimism Bias and Reference Class Forecasting to produce relevant input to the PDFs for travel time savings and construction costs. 34
Perspectives Investigation of introducing non-monetary aspects to the modelling framework as discussed in some of the papers is highly relevant Correlations between impacts are under review as to whether a general implementation is possible/needed The distinguishing between lack of knowledge (uncertainty) and inherent randomness of the system (variability) uncertainty should be investigated further Finally, the combinations of Optimism Bias and Risk Analysis needs further implementation especially, the need for reference classes are obvious 35
www.transport.dtu.dk/unite Large-scale investigation of uncertainties New up-to-date database information with regard to demand forecasts (and transport models) Involvement of researcher from Princeton and Oxford Universities Cross-disciplinarian research with practical applicability 36
Thank you for listening! 37
Extra slides for presentation if needed
Integration level (Index 100 in 2024) Scenario Trend Development Scenario Trend Development Economic Growth and Level of Integration 160 150 140 130 120 High Middle Low 110 100 90 2024 2029 2034 2039 2044 2049 2054 2059 2064 2069 2074 Years of evaluation 39
Separation of Uncertainty Nature of Uncertainty Uncertainty (Epistemic): Due to lack of Knowledge Variability Uncertainty (Ontological): Due to inherent variability within the system Traditional aspects of modelling and policy analysis: - Limited and inaccurate data - Measurement error - Incomplete knowledge - Limited uncerstanding - Imperfect Models - Subjective judgments - Ambiguities - etc. Behavioural variability (Micro) Societal variability (Meso & Macro) Natural randomness 40
Cost-Benefit Analysis P 1 2 P P' Q P P' Q' Q P P' Q Q' B Existing Travellers Newly GeneratedTravellers 1 2 P A Price - P P E B Q Quantity - Q Q Q 41
Costs Large changes in the Demand Curve Demand curve: Cars (Øresund Fixed Link) 500 400 300 ADT shift from 865 before to 10.000 after 200 100 0 0 2000 4000 6000 8000 10000 12000 ADT Cost per car before 300 DKK cost after 100 DKK Q k P Q 2,31 4,45 P 10 8 42
Cost-Benefit Analysis Strengths: Transparency all aspects are included in the analysis Comparable Consistent, mostly due to the new manual Systematical data collection Weaknesses: False sense of transparency how to decide and undcover all aspects Practical measuring problem models and unit prices Generations equity same value today as last century Social equity (we are all a-like) Individual welfare Aggregation of individual welfare 43
Dispute of criteria NPV vs. IRR As shown before the IRR expression is a polynomial equation with several roots Gradient and discount rate determines the choice IRR is independent from r NPV is dependent on r NPV NPVA NPVB Hence, changing r to r* creates problems from the two projects suggested A and B. r r* IRR A A IRR B B IRR 44
Dispute about criteria NPV vs. BCR Given the system below with respectively costs and benefits for three system alternatives For a very short evaluation period of 1 year the NPV and B/Crate are calculated 45
Public vs. Private Tax distortion of 1.2 is introduced due to the financing of projects through taxes: E.g. Person A willing to perform a job for 100 DKK Person B is willing to pay to get the job done for 110 DKK 50% tax would endure that Person B would pay 55 DKK Society loses the actual surplus of 10 DKK Net Taxation factor is introduced of 1.17: Since we operate with market prices, a private company would endure duties, taxes etc. on commodities The State obviously does not have to pay that 17% has been found as an average 46
Research Outcomes 47
Full scale uplifts from COWI and Flyvbjerg 48
Beta Distribution Typically parameterized by two shape parameters [, ]: 49
Gamma Distribution Typically parameterized by a shape and scale parameters [k, ]: f ( x) k k x k 1! k 1 e kx, x 0, k 2,3,4... and f ( x) 0 for other x k 1 k while the variance var 1 k k 50
Succesive Calculation Post Beskrivelse Mængde Enhed a b C m s varians*10-6 1 Opstartsarbejde 1 stk. 37.500 187.500 450.000 210.000 82.500 6.806 Post Beskrivelse Mængde Enhed a b c m s varians*10-6 2 Boldbaner 50.000 m 2 30 75 120 3.750.000 900.000 810.000 1 Opstartsarbejde 1 stk. 37.500 187.500 450.000 210.000 82.500 6.806 3 Andre græsarealer 25.000 m 2 8 15 30 412.500 112.500 12.656 2 Boldbaner 50.000 m 2 3.006.000 5.234 2.1 Rydning og afretning 50.000 m 2 11,25 12,3 13,35 615.000 21.000 441 4 Parkanlæg 20.000 m 2 8 23 60 540.000 210.000 44.100 2.2 Dræn 50.000 m 2 14,7 16,5 18,75 829.000 40.500 1.640 5 Befæstede arealer 15.000 m 2 90 225 330 3.285.000 720.000 518.400 2.3 Vandingssystem 50.000 m 2 9,75 12,75 13,5 615.000 37.500 1.406 6 Afsluttende arbejde 1 stk. 37.500 150.000 375.000 172.500 67.500 4.556 2.4 Muld og planering 50.000 m 2 12 13,5 15,9 684.000 39.000 1.521 7 Generelle forhold 8.370.000 Sum -10 % 0 % 20 % 167.400 502.200 252.205 2.5 Såning 50.000 m 2 4,5 5,25 6 262.500 15.000 225 Kalkuleret middelværdi 8.537.400 1.648.724 3 Andre græsarealer 25.000 m 2 7,5 15 30 412.500 112.500 12.656 4 Parkanlæg 20.000 m 2 7,5 22,5 60 540.000 210.000 44.100 Tilhørende spredning, beregnet som kvadratroden af summen af variansen 1.284.026 5 Befæstede arealer 15.000 m 2 90 225 330 3.285.000 720.000 518.400 6 Afsluttende arbejde 1 stk. 37.500 150.000 375.000 172.500 67.500 4.556 7 Generelle forhold 7.626.000 sum -10 % 0 % 20 % 152.520 457.560 209.361 Kalkuleret middelværdi 7.778.520 801.114 Tilhørende spredning, beregnet som kvadratroden af summen af variansen 895.050 51
Data fitting The data fits are conducted by Maximum likelihood estimators: Estimates the distribution parameters Maximum likelihood parameter estimation is to determine the parameters that maximize the probability (likelihood) of the sample data The goodness of fits interpreted by using Chi-squared [ 2 ] statistics: The sum of differences between observed and expected outcomes 2 where O is an observed outcome and E is an expected frequency O E E 2 52
Background literature (international) 2002 2003 2004 53
Background literature (National) 2007 2007 2008 54
Back et al. (2000) Four bullet points for estimating construction costs with probability distributions have been proposed in: Upper and lower limits which ensures that the analyst is relatively certain values does not exceed. Consequently, a closed-ended distribution is desirable. The distribution must be continuous The distribution will be unimodal; presenting a most likely value The distribution must be able to have a greater freedom to be higher than lower with respect to the estimation skewness must be expected. 55
Composite Model for Assessment CBA B/C MCA A B C D Alt. 1 Alt. 2 Alt. 3........ SMART AHP 56
A Brief History 1950 s: Introduction of CBA in USA Highway s connecting East-West 1960 s: CBA Methodology reaches Europe New Motorway Schemes 1970 s: Traditional traffic impacts are introduced 1980 s: The methodology reaches Denmark together with widespread impacts within the Multi-Criteria methodology 1990 s: Full implementation in Denmark a general acceptance of CBA & MCA 2003: The Danish Ministry of Transport published in 2003 a guideline for making socio-economic analysis in the Danish Transport Sector 57