Assessment of Transport Projects: Risk Analysis and Decision Support

Transcription

1 Assessment of Transport Projects: Risk Analysis and Decision Support Assistant Professor Kim Bang Salling Presentation at the RISK Conference 2011: December 1 st 2011 DGI Byen, Copenhagen

2 Outline Background Introduction Methodologies Cost Benefit Analysis (CBA) Quantitative Risk Analysis (QRA) Feasibility Risk Assessment (FRA) Accumulated Descending Graphs (ADG) The UNITE-DSS Decision Support Model Uncertainties in Transport Project Evaluation Conclusion Perspective 2

3 Background The Manual for socio-economic analysis in the transport sector (2003) Unique guidelines for evaluating transport infrastructure projects Lack of uncertainty handling Expected revision Building decision support with a twist Rational decision making involves the assessment of both the benefits and the losses (costs) The need for making good decisions in transport planning and evaluation are vital 3

4 Transport Planning and Assessment Ongoing transport planning: Research: - Societal goals as, for example networks and mobility, sustainable development, etc. - Prognoses/ forecasts Transport infrastructure project proposal Traffic models Decision support Cost-benefit analysis (CBA) - Concepts as for example Feasibility Risk Assessment (FRA) and Accumulated Descending Graphs (ADG) - The CBA-DK model software - Urban & regional planning Impact models Multi-criteria analysis (MCA) - Case examples related to different modes - Design standards, etc. - Findings and recommendations 4

5 Introduction CBA & MCA produce single point estimates Informativ decision support Feasibility Risk Assessment (FRA) Accumulated Descending Graphs (ADG) Normally, uncertainties are handled by sensitivity tests Historical overview of uncertainties Construction cost overrun Traffic forecast underrun (traffic modelling) 5

6 Suez Canal Sydney Opera House Concorde Supersonic Aeroplane Boston's Artery/Tunnel Project, USA Humber Bridge, UK Boston- Washington- New York Great Belt Rail Tunnel, DK A6 Motorway Chapel-en-le- Frith/Whaley Shinkansen Joetsu Rail line, Japan Washington metro, USA Channel Tunnel, UK & France Karlsruhe- Bretten light rail, Germany Øresund Access links, DK & Sweden Mexico city metro line, Mexico Paris-Auber- Nanterre rail line, France Cost Overruns (%) Cost Overrun (%) Construction Cost Overruns (fixed prices) Construction cost overruns Construction Cost Overruns 2000% 100% 1800% 90% 1600% 80% 1400% 70% 1200% 60% 1000% 50% 800% 40% 600% 400% 200% 0% 30% 20% 10% 0% Channel Tunnel, UK & France Øresund Access links, DK & Sweden Great belt link, DK Øresund coast-tocoast link, DK & Sweden 6

7 Cost-Benefit Analysis (CBA) Method for evaluating the goodness of investments A systematic approach in listing costs and benefits Selection of the best performing alternative(s) Inputs derived from a lot of external sources Traffic models and impact models Key figue catalogues Output based upon single point criteria Net present value (NPV) Benefit cost ratio (BCR) Transferred model uncertainties!?!? 7

8 Uncertainty in transport appraisal Unit price principles are assumed certain Two types of impacts stands out: Travel time savings -> Benefit Construction costs -> Cost Unit Pricing Principles Sources of Uncertainty Literature supports the latter impacts by the so-called: Optimism Bias Relies on the key figure catalogue in calibrating and determining unit price settings. Model Uncertainty Relies on the model build up of impact and traffic models that provide the input towards decision support models. Randomness of the system Lack of knowledge 8

9 Optimism Bias and Reference Class Forecasting The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004) Reference Class Forecasting: Optimism Bias Inside View Outside View Uniqueness of Project Reference Class Forecasting Forecasting of particular projects Forecasting from a group of projects The Planning Fallacy (1) Identification of relevant reference classes (2) Establishing probability distribution (3) Placing and comparing the project Current Situation Optimism Bias Uplifts 9

10 Optimism Bias and uplifts Deriving uplifts is highly dependet on large data-sets Flyvbjerg from (AAU) has since 2003 developed a large database Unfortunately, it looks upon mega-projects Uplift values were derived on basis of Reference Class Forecasting i.e. statistical measurements on various project pools Applying uplifts still produces single point rate of returns BUT, the data collected can be transformed and used in another way. Risk Analysis 10

11 Risk Control Infrastructure assessment TRANSPORT INFRASTRUCTURE PROJECT: General information (Technical, political, economical, etc.) SAFETY SOCIETAL GOALS: PHILOSOPHY: Definition of goals, fundaments for priorities and standards ACCEPTANCE CRITERIA: Societal acceptance, budgetary constraints etc. Appraising the information brought above RISK ANALYSIS (TRADITIONAL): RISK IDENTIFICATION: Definition of risk components - impacts RISK ASSESSMENT: Describe and quantify risk by evaluation RISK EVALUATION: Compare risk to acceptable standards 11

12 Monte Carlo Simulation 12

13 Input Distributions Distinction between non-parametric and parametric Non-parametric is used when experts have to make the judgments Parametric are used when data and/or theory underpins the judgments Non-Parametric distributions: Uniform Triangular/Trigen Parametric distributions: Normal Erlang (Gamma) > Construction Cost PERT (Beta) -> Travel time savings 13

14 Level of Knowledge (LoK) The LoK ranges from low to medium to high Distinction between Parametric and Non-Parametric distributions 14

15 PERT Distribution Based upon a beta distribution with the assumption that the mean can be derived from: Min Mode Max Min 4 Mode Max Mean Mean PERT Triang vs 6 3 This makes it ideal for modelling experts opinion Stands out compared to the Triangular distribution Triangular Beta-PERT 15

16 Data fit (Rail) Demand forecasts Demand forecasts (user benefits) are set against prior Reference classes derived from Flyvbjerg et al. (2003) 27 rail projects were compared where the inaccuracy on average were 39% lower than predicted I have fitted a PERT curve around the data from Flyvbjerg et al. (2003) 16

17 -150% -125% -100% -75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% Data fit (Road) Demand forecasts 183 road projects were compared where the inaccuracy on average were 9% lower than predicted 5,0% Fit Comparison for Inaccuracy in Traffic Forecasts RiskPERT(-78.5;9.6%;179.34%) -0,487 90,0% 1,057 5,0% Input Beta-PERT 17

18 Erlang Distribution Based upon a gamma distribution defined upon a shape and a scale parameter (k, ) The shape parameter, k, depicts the skewness of the distribution whereas the scale,, is based upon data K=2 K=5 K=10 K=

19 Data fit (Rail) Investment costs Flyvbjerg et al. Compared 58 rail projects Approximately 88% of the probability mass is above 0 which indicates that rail type projects are underestimated The fitted probability distribution contributes to the fact that an Erlang distribution is very well suited 19

20 -100% -75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250% Data fit (Road) Investment costs 167 road projects were compared where the inaccuracy on average were 20% lower than predicted, with k = 8 5,0% Fit Comparison for Cost Overrun for Road Projects RiskErlang(8;0.09) -> (-33.6%;20.2%;222.6%) -0,156 90,0% 0,569 5,0% Input Erlang 20

21 Recommendation High level of knowledge Risk analysis in decision support: Combination of data from Flyvbjerg, Successive Principle and Risk Analysis: Large-scale implementation in UNITE Definition of distributions Empirical data to feed the distributions Assigning probability distributions: Investment Cost Gamma (Erlang) distribution Travel Time Savings Beta (PERT) distribution Mode Impact Distribution Low High Rail Travel time savings PERT -90% 140% Rail Construction cost Erlang (k = 23) -40% 120% Road Travel time savings PERT -80% 180% Road Construction cost Erlang (k = 8) -30% 120% A negative sign for travel time savings means that benefits have been overestimated and a negative sign for construction costs means that costs have been underestimated 21

22 Uncertainties in Transport Project Evaluation (UNITE) Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages (5) Evaluation methodology WP5 project leader: Steen Leleur (DMG) (3) Uncertainty calculation of cost estimates WP3 project leader: Bo Friis Nielsen (DTU Informatics) (4) Uncertainty calculation in transport models WP4 project leader: Otto Anker Nielsen (TMG) (2) Organizational context of Modelling, an empirical study WP2 project leader: Petter Næss (AAU) (1) Systematic biases in transport models (recognized ignorance), an empirical study WP1 project leader: Bent Flyvbjerg (Oxford University) 22

23 The Case Study: HH-Connection Connecting Denmark with Sweden: Scandinavian link Currently, close to the capacity limit on Oresund HH-Connection (alternatives) Description (Alignment of connection) Cost (million DKK) Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700 Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500 Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500 Alternative 4 Bridge for road (2x2 lanes) 6,000 Note! DKK 23

24 The UNITE DSS Modelling Framework The UNITE-DSS Decision Support Model for Risk Assessment Determinstic Calculation Stochastic Calculation I) Cost-benefit analysis III) Reference Class Forecasting IV) Reference Scenario Forecasting Results: Point estimates in terms of NPV, BCR, IRR Impact: Travel time savings Impact: Travel time savings II) Optimism Bias Uplifts Determination of Beta-PERT distribution Determination of scenarios and triple estimates Impact: Investment costs Determination of inputs to the Beta-PERT distribution Trtiple estimate parameters to the Beta-PERT distribution Results: Point estimates in terms of NPV, BCR, IRR Results: Certainty graphs and certainty values Results: Certainty graphs and values for scenarios 24

25 Deterministic Module Entry data 25

26 Results: Cost-Benefit Analysis HH-Connection (alternatives) Cost (million DKK) BCR NPV (million DKK) Alternative 1 7, ,530 Alternative 2 5, ,640 Alternative 3 11, ,240 Alternative 4 6, ,860 Construction costs by far the largest contributor of costs User Benefits by far the largest contributor of benefits Consists of Ticket revenue and time savings Relies on the prognosis of future number of passengers i.e. demand forecasts 26

27 Results : Optimism Bias Uplifts HH-Connection (alternatives) Cost (uplifted) (million DKK) BCR (orig.) (from slide 8) BCR (uplifts): 80% uplift Alternative 1 12, Alternative 2 8, Alternative 3 15, Alternative 4 7, The BCR are lower, however, still point estimates towards DM Moreover an advanced form of sensitivity analysis Imply to introduce risk analysis and Monte Carlo simulation 27

28 Stochastic module The UNITE-DSS model is assigned an add-on software model A range of distribution functions are shown Two non-parametric distrbutions have been tested/applied (green) Three parametric distributions have been tested/applied (orange) 28

29 Input in UNITE-DSS Construction cost Shape parameter k = 8 for road projects and k = 23 for rail projects (including air) The mean ( ) and standard (std) deviation is calculated k The scale parameter ( ) is calculated on basis of the succesive principle 29

30 Results (RCF): Monte Carlo simulation 30

31 Reference Scenario Forecasting Accomodates scenario analysis and RCF Vertical regime: Economic development due to link Horizontal regime: Integration between borders 31

32 Results from RSF 32

33 The coupling of methodologies in achieving feasibility risk assessment 33

34 Conclusions The UNITE-DSS model has been developed and functions as a flexible assessment tool applicable for wider risk oriented assessment for transport projects across different modes. The developed type of accumulated descending graph is found to be useful to inform about uncertainty relating to assessment of transport projects. Dependent on the information available parameter-based or parameter-free input probability distributions should be applied. It is possible to accommodate the recent results stemming from Optimism Bias and Reference Class Forecasting to produce relevant input to the PDFs for travel time savings and construction costs. 34

35 Perspectives Investigation of introducing non-monetary aspects to the modelling framework as discussed in some of the papers is highly relevant Correlations between impacts are under review as to whether a general implementation is possible/needed The distinguishing between lack of knowledge (uncertainty) and inherent randomness of the system (variability) uncertainty should be investigated further Finally, the combinations of Optimism Bias and Risk Analysis needs further implementation especially, the need for reference classes are obvious 35

36 Large-scale investigation of uncertainties New up-to-date database information with regard to demand forecasts (and transport models) Involvement of researcher from Princeton and Oxford Universities Cross-disciplinarian research with practical applicability 36

37 Thank you for listening! 37

38 Extra slides for presentation if needed

39 Integration level (Index 100 in 2024) Scenario Trend Development Scenario Trend Development Economic Growth and Level of Integration High Middle Low Years of evaluation 39

40 Separation of Uncertainty Nature of Uncertainty Uncertainty (Epistemic): Due to lack of Knowledge Variability Uncertainty (Ontological): Due to inherent variability within the system Traditional aspects of modelling and policy analysis: - Limited and inaccurate data - Measurement error - Incomplete knowledge - Limited uncerstanding - Imperfect Models - Subjective judgments - Ambiguities - etc. Behavioural variability (Micro) Societal variability (Meso & Macro) Natural randomness 40

41 Cost-Benefit Analysis P 1 2 P P' Q P P' Q' Q P P' Q Q' B Existing Travellers Newly GeneratedTravellers 1 2 P A Price - P P E B Q Quantity - Q Q Q 41

42 Costs Large changes in the Demand Curve Demand curve: Cars (Øresund Fixed Link) ADT shift from 865 before to after ADT Cost per car before 300 DKK cost after 100 DKK Q k P Q 2,31 4,45 P

43 Cost-Benefit Analysis Strengths: Transparency all aspects are included in the analysis Comparable Consistent, mostly due to the new manual Systematical data collection Weaknesses: False sense of transparency how to decide and undcover all aspects Practical measuring problem models and unit prices Generations equity same value today as last century Social equity (we are all a-like) Individual welfare Aggregation of individual welfare 43

44 Dispute of criteria NPV vs. IRR As shown before the IRR expression is a polynomial equation with several roots Gradient and discount rate determines the choice IRR is independent from r NPV is dependent on r NPV NPVA NPVB Hence, changing r to r* creates problems from the two projects suggested A and B. r r* IRR A A IRR B B IRR 44

45 Dispute about criteria NPV vs. BCR Given the system below with respectively costs and benefits for three system alternatives For a very short evaluation period of 1 year the NPV and B/Crate are calculated 45

46 Public vs. Private Tax distortion of 1.2 is introduced due to the financing of projects through taxes: E.g. Person A willing to perform a job for 100 DKK Person B is willing to pay to get the job done for 110 DKK 50% tax would endure that Person B would pay 55 DKK Society loses the actual surplus of 10 DKK Net Taxation factor is introduced of 1.17: Since we operate with market prices, a private company would endure duties, taxes etc. on commodities The State obviously does not have to pay that 17% has been found as an average 46

47 Research Outcomes 47

48 Full scale uplifts from COWI and Flyvbjerg 48

49 Beta Distribution Typically parameterized by two shape parameters [, ]: 49

50 Gamma Distribution Typically parameterized by a shape and scale parameters [k, ]: f ( x) k k x k 1! k 1 e kx, x 0, k 2,3,4... and f ( x) 0 for other x k 1 k while the variance var 1 k k 50

51 Succesive Calculation Post Beskrivelse Mængde Enhed a b C m s varians* Opstartsarbejde 1 stk Post Beskrivelse Mængde Enhed a b c m s varians* Boldbaner m Opstartsarbejde 1 stk Andre græsarealer m Boldbaner m Rydning og afretning m 2 11,25 12,3 13, Parkanlæg m Dræn m 2 14,7 16,5 18, Befæstede arealer m Vandingssystem m 2 9,75 12,75 13, Afsluttende arbejde 1 stk Muld og planering m ,5 15, Generelle forhold Sum -10 % 0 % 20 % Såning m 2 4,5 5, Kalkuleret middelværdi Andre græsarealer m 2 7, Parkanlæg m 2 7,5 22, Tilhørende spredning, beregnet som kvadratroden af summen af variansen Befæstede arealer m Afsluttende arbejde 1 stk Generelle forhold sum -10 % 0 % 20 % Kalkuleret middelværdi Tilhørende spredning, beregnet som kvadratroden af summen af variansen

52 Data fitting The data fits are conducted by Maximum likelihood estimators: Estimates the distribution parameters Maximum likelihood parameter estimation is to determine the parameters that maximize the probability (likelihood) of the sample data The goodness of fits interpreted by using Chi-squared [ 2 ] statistics: The sum of differences between observed and expected outcomes 2 where O is an observed outcome and E is an expected frequency O E E 2 52

53 Background literature (international)

54 Background literature (National)

55 Back et al. (2000) Four bullet points for estimating construction costs with probability distributions have been proposed in: Upper and lower limits which ensures that the analyst is relatively certain values does not exceed. Consequently, a closed-ended distribution is desirable. The distribution must be continuous The distribution will be unimodal; presenting a most likely value The distribution must be able to have a greater freedom to be higher than lower with respect to the estimation skewness must be expected. 55

56 Composite Model for Assessment CBA B/C MCA A B C D Alt. 1 Alt. 2 Alt SMART AHP 56

57 A Brief History 1950 s: Introduction of CBA in USA Highway s connecting East-West 1960 s: CBA Methodology reaches Europe New Motorway Schemes 1970 s: Traditional traffic impacts are introduced 1980 s: The methodology reaches Denmark together with widespread impacts within the Multi-Criteria methodology 1990 s: Full implementation in Denmark a general acceptance of CBA & MCA 2003: The Danish Ministry of Transport published in 2003 a guideline for making socio-economic analysis in the Danish Transport Sector 57