Ex-Post Evaluation of Projects and Activities Financed under the LIFE Programme

Transcription

1 Directorate General Environment, Unit E4 LIFE Ex-Post Evaluation of Projects and Activities Financed under the LIFE Programme Final Report Part 1: Methodology and description of the LIFE Programme and its policy context Julyl 2009

2 COWI A/S Parallelvej 2 DK-2800 Kongens Lyngby Denmark Tel Fax wwwcowicom Directorate General Environment, Unit E4 LIFE Ex-Post Evaluation of Projects and Activities Financed under the LIFE Programme Final Report Part 1: Methodology and description of the LIFE Programme and its policy context July 2009 Document no 7-1 Version 1 Date of issue July 2009 Prepared BIM, TVGU, STEH, ROKR Checked NEO, IL, TIH Approved BIM This report has been prepared as a result of an independent evaluation by COWI being contracted by the Directorate General Environment The views expressed are those of the Consultant and do not necessarily reflect those of the European Commission

3 1 Table of Contents Part 1: Methodology and description of the LIFE Programme and its policy context 2 1 Introduction 2 2 Methodology 3 21 Evaluation design 3 22 Approach to data collection and analysis 5 23 The evaluation process - activities, time and resources 12 3 The LIFE Programme and its policy context The anatomy of the LIFE Programme Overview of LIFE projects during the period 1996 to Table of Appendices Appendix 1 Appendix 2 Appendix 3 Appendix 4 Appendix 5 Appendix 6 Appendix 7 Appendix 8 Indicators for assessment of results and impacts Detailed questions, success criteria and indicators E-survey questionnaire E-survey responses and data validity Nature and Environment projects selected for in-depth study Third country projects selected as case projects List of people met Bibliography O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

4 2 Part 1: Methodology and description of the LIFE Programme and its policy context 1 Introduction This document constitutes Part 1 of the final report in connection with the Ex-post Evaluation of Projects and Activities under the LIFE Programme This first part of the report provides the background for the evaluation, explaining the methodology used (Chapter 2) and some key characteristics of the LIFE Programme and the projects co-financed under it (Chapter 3) O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

5 3 2 Methodology This chapter presents the evaluation methodology applied The methodology was developed during the inception phase of the evaluation based on the Terms of Reference (TOR) and the Consultant's proposal and following the EU Evaluation Guidelines 1, and the standard evaluation criteria as specified in these guidelines A comprehensive report on the development of the methodology was given in the Inception Report This chapter provides an overview of the evaluation design in section 21, the approach to data collection and analysis in section 22 and, finally, in section 23, an overview of the evaluation process is provided 21 Evaluation design 211 Evaluation criteria During the inception phase, the evaluation criteria suggested in the TOR were reviewed in light of the objectives listed in the TOR and the requirements of the EU Evaluation Guidelines This led to a revision of the criteria thereby narrowing and focusing the scope of the evaluation The following criteria were identified as focus criteria: Effectiveness, efficiency, sustainability and utility In addition the criteria of acceptability and consistency were identified as secondary criteria 212 Three components and nine evaluation questions The evaluation is structured into three main components Figure 21 illustrates how the evaluation criteria apply 1 Evaluating EU Activities, A Practical Guide For The Commission Services, July 2004 O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

6 4 Figure 21 Components and evaluation criteria Source: Developed by Consultant Results and impacts The first component, comprising the bulk of the evaluation, concerns the evaluation of the results and impacts generated by the LIFE Programme, ie the analysis of the actual LIFE intervention and its effects, taking point of departure in the LIFE projects The evaluation of results and impacts comprise the three LIFE components: NAT, ENV and TCY The analysis of LIFE NAT and ENV has been carried out in the form of country studies 2, which feed into thematic studies as illustrated by the left side of the diagram in Figure 21 Policy The second component relates to the LIFE Programme as a policy and considers its relations to other policies and how it is assessed by stakeholders Programme management Finally, the component concerning programme management comprises an evaluation of organisation and working methods of the LIFE Unit in managing the LIFE Programme, focusing on the effectiveness and efficiency of the Unit in evaluating and selecting projects and monitoring selected projects The nine main evaluation questions which guide this evaluation emerge from the above mentioned components 1 What are the results and impacts of the LIFE Programme? 2 In each Member State, except Bulgaria, where no LIFE projects were implemented in the evaluation period O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

7 5 Effectiveness: 2 To what extent are the results of the LIFE Programme in line with the objectives? 3 To what extent do the LIFE unit organisational setup, and the modalities for selection/monitoring of projects, contribute to the effectiveness of the programme/projects? Efficiency: 4 To what extent is the cost at which results have been achieved reasonable when considering the costs of comparative interventions? 5 To what extent do the LIFE unit organisational setup, and the modalities for selection/monitoring of projects, contribute to the efficiency of the programme/projects? 6 Utility: To what extent do the results and impacts generated by the LIFE Programme address the environmental issues facing the EU and the stakeholders involved? 7 Sustainability: To what extent are the positive results and impacts generated by the LIFE Programme likely to last after the termination of the projects? 8 Consistency/coherence: To what extent has the LIFE Programme been complementary/supportive/contradictory to other EU policies and programmes? 9 Acceptability: To what extent do stakeholders accept the LIFE Programme and LIFE projects? These nine evaluation questions are supported by a number of sub-questions and a detailed framework for assessing each question in the form of success criteria, indicators and data sources Appendix 2 provides an overview of these for each of the components in the evaluation The framework described in Appendix 2 is used as the reference guide for data collection and analysis as is further described in the section below 22 Approach to data collection and analysis Data collection and analysis was performed in accordance with the framework provided in Appendix 2 All data collected was used to validate (or negate) the success criteria, which fed into the analysis of the evaluation questions and the formulation of conclusions and recommendations The process varied for the evaluation's three components and is therefore described in separate sections in the following sections 221 The results and impacts component The evaluation of results and impacts implied two major steps, as illustrated in Figure 22 The first step was an assessment of the effects (results and impacts) On the basis of the evaluation on effects, an assessment on effectiveness, efficiency, utility and sustainability was made O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

8 6 Figure 22 Evaluation of results and impacts Source: Elaborated by Consultant Division into two steps emphasises the fact that the assessment on the effects (results and impacts) of the LIFE Programme is an output in itself and that an assessment is necessary in order to gauge effectiveness, efficiency sustainability and utility Step 1: Assess effects The assessment of effects requires the identification of indicators which can be used to describe the effects For the Nature and Environment components the indicators established are described separately in Appendix 1 For the Third Countries component, the indicators have been integrated in the framework described in Appendix 2 The Nature and Environment indicators are divided according to the following main effect categories: Environmental (improved state, reduced pressure, etc) Management (improved management, planning, administration) Awareness (increased awareness) Demonstration ("showcase effect") Innovation (development of new technology or improvement of processes) In the assessment of effects, we distinguished between results and impacts We defined results as the end-of-project situation, as reported in final reports and project summaries For on-going projects, results were assessed based on expected results as reported on in mid-term reports, etc Impacts were defined as the medium to long term effects in the period after finalisation of the project and assessed primarily on the basis of e-survey responses and in-depth project studies The completion dates of the projects obviously vary considerably and some projects are still "on-going" (in progress) In general, the assessment of impacts has greater validity for older projects Impacts for recent or on-going projects were assessed based on expected impacts, taking into account the results and relevant framework conditions We used three main sources of project level data, as illustrated in Figure 23 below In addition to amassing the project level data, we have also carried out interviews with national focal points (key/relevant country representatives), the monitoring team, DG Environment policy units and LIFE Unit staff The data collected through these interviews is thematic rather than project-specific and thus serve as a fourth data/information source on results and impacts of the LIFE Programme O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

9 7 Figure 23 Project level data used to assess effects of LIFE Environment and LIFE Nature Source: Developed by the Consultant The primary purpose of the e-survey 3 was to gather data on the results and impacts of as many projects as possible through targeting the project managers of LIFE projects The questions in the e-survey were organised according to the indicators for results and impacts We had hoped for a response rate of 25%, however, in total 275 projects managers of LIFE Nature or Environment projects completed the questionnaire, corresponding to 15% of all Nature and Environment projects implemented during the evaluation period In addition, 219 project managers partly completed the questionnaire We assessed that 50 of the non-completed responses can be used as data sources, which brought the response rate to 18% Originally, the methodology was built on the assumption of a higher response rate The implications in using the results from the e-survey were different for the Nature and Environment components due to the response rate, different characteristics of the portfolio of projects and the number of projects within the themes For the Nature component, with 771 projects in total and e-survey responses covering 158 projects the response rate was 20% Furthermore, the Nature projects were dealt with as one large theme and the projects within the theme were relatively homogeneous For this reason the e-survey data was assessed as representative of the total Nature project population and acceptable as a basis for forming conclusions at aggregate (EU) level 3 The e-survey was designed to provide input to all three components and consisted of two sections: One section aimed at project managers of LIFE Nature and Environment projects in the period while a general section aimed at all stakeholders concerned with the LIFE Programme The first section aimed to generate data on results and impacts of the projects as well as solicit the opinions and perceptions of the project managers of the management and administration of the LIFE Programme The general section of the e-questionnaire aimed at obtaining comments and opinions on the LIFE Programme in general from a broad base of stakeholders The e-survey questionnaire is included as Appendix 3 O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

10 8 LIFE Environment projects constitute a very heterogeneous group by comparison Even when considering the projects categorised under the same theme (eg air), there is much variance At the same time the response rate for Environment projects was also lower with 167 responses for 1076 projects in total (16%) The response rate was lower for some Environment themes and higher for others Considering the variance in projects, the e-survey results were not considered as sufficient basis to form conclusions The thematic studies therefore reviewed a number of project summaries within each theme in order to obtain an extra data source on project effects and ensure validity and authenticity Appendix 4 provides more background of the analysis of the representativeness of the e-survey data The 105 projects selected for in-depth study were selected according to a range of criteria in order to attain the highest possible level of representation compared to the overall portfolio of Nature and Environment projects The criteria did not incorporate country or thematic specific criteria In Appendix 5, the selection process, criteria and list of projects is described The in-depth studies were performed through review of final reports and monitoring files, interviews with project managers, and site visits where relevant A site visit has been made where this was logistically possible and allowed for actual observation of results and/or impacts of the project The in-depth project studies were performed as part of the country studies but fed into the thematic studies as well No projects were selected for the ENV themes on forests and chemicals as there are very few projects, which fall under these two themes These thematic studies relied to some extent on desk reviews of selected projects instead of in-depth project studies From the BUTLER database, we extracted descriptions of project results given in project summaries as well as the project rating designated by the LIFE Unit in the more recent projects In total 398 of the 1847 Environment and Nature projects have been scored The data collection and analysis related to LIFE Third Countries (TCY) was guided by specific evaluation questions, success criteria and indicators, as described in Appendix 2 This is because the LIFE TCY component was implemented with objectives that differ substantially from the LIFE Nature and LIFE Environment components 4 Consequently, the approach to data collection and analysis also differs from that used for LIFE Environment and LIFE Nature Data collection on LIFE TCY included a desk review, interviews and project case studies The desk review was based on the information available on the LIFE Website, BUTLER (progress reports, monitoring reports and evaluation reports) and information contained in the relevant reports/publications provided by the LIFE Unit Interviews were carried out with the key stakeholders in the LIFE Unit in DG Environment and in DG RELEX, and DG AIDCO In addition, 17 of the 179 projects co-financed by LIFE TCY in 16 countries, were selected as case studies, including document review, telephone interviews with the national country coordinators, the EU delegations and the beneficiaries The selection procedure and the projects selected are documented in Appendix 6 4 During the evaluation period Cyprus, Estonia, Latvia, Lithuania, Malta, Slovenia and Poland were TCY countries and became part of LIFE NAT and ENV upon their accession to the EU However, the accession countries had the possibility to apply for projects under the NAT and ENV components Most accession countries made use of this option and, consequently, only Cyprus and Malta have actually implemented TCY projects during the evaluation period The country studies in Malta and Cyprus will include TCY related questions and the data collected will feed into the TCY study Hence, the country reports (dealing with the implementation of LIFE Nature and Environment) on Malta and Cyprus will not contain a specific assessment on the TCY projects This assessment is made in the TCY report However, the country reports may make reference to TCY projects and their importance in relation to implementation of Nature and Environment projects, if any O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

11 9 Step 2: Assess evaluation criteria The next step was to use the assessment of results and impacts to assess the LIFE Programme on the criteria of effectiveness, efficiency, utility and sustainability This involves comparison on the effects with various "benchmarks", as illustrated in Figure 22 above: Effectiveness compares results with objectives Efficiency compares results with costs against a benchmark of costs of similar interventions Sustainability compares impacts with results and also relates this to overall/global objectives Utility compares effects with the needs, problems and issues to be addressed The terminology used in assessing effectiveness and efficiency in the country studies and thematic studies is shown in Table 21 Table 21 Terminology in assessing effectiveness and efficiency Very low Low Medium High Effectiveness - projects Effectiveness - policy Sustainability None of the projects effective A few of the projects effective The majority of the projects effective All projects 100% effective with a few exceptions Targets generally not met Targets met to a limited extent The majority of targets met Targets met with a few exceptions No projects are sustainable A few projects are sustainable The majority of the projects are sustainable All projects sustainable with a few exceptions Very high All projects 100% effective All targets met All projects sustainable This implies a need to specify which benchmarks to apply for this particular evaluation This is reflected in Appendix 2, which contains the list of evaluation questions together with associated success criteria, indicators and sources of data Some general observations with regard to the benchmarks highlighted in the list above are highlighted below Defining the benchmarks relating to objectives was a challenge for this evaluation as it covered several generations of LIFE with corresponding LIFE Regulations, which, in turn, contained similar but different objectives Based on the regulations (including the LIFE+ Regulation), we established one "generic" overview of objectives as illustrated in Table 22 O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

12 10 Table 22 LIFE Objectives Overall policy objective Intermediate policy objectives Specific objectives To contribute to sustainable development; To favour the integration of the environment into other EU policies; To lead to new solutions for the environmental problems To contribute to the implementation, development and enhancement of Community environmental policy and legislation LIFE-NAT: To contribute to the implementation of Council Directive 79/409/EEC (Birds Directive) and Council Directive 92/43/EEC (Habitats Directive) LIFE-ENV: To contribute to the development of innovative and integrated techniques and methods and to the further development of Community environment policy Objectives for each theme are indicated in Appendix 1 5 LIFE-TCY: To contribute to the establishment of capacities and administrative structures needed in the environmental sector and in the development of environmental policy and action programmes in third countries bordering the Mediterranean and the Baltic Seas 6 Source: Elaborated by the Consultant Note: None of the LIFE regulations distinguish between overall policy objective and intermediate policy objectives in the same as illustrated in the table This distinction has been made by the Consultant In relation to the benchmark for costs of similar interventions our intention was to identify standard unit prices for the various types of results under each theme based on information available in eg impact assessment for Directives, Directive specific financing plans, available sector specific studies, etc However, due to insufficient data this was generally not possible The assessment on efficiency therefore relied on expert assessments and the perceptions of key stakeholders The benchmark relating to needs, problems and issues was addressed with a point of departure in the 6th Environmental Action Plan and the associated documents, including thematic strategies For third countries, the point of departure was country documents, such as national environmental action plans 222 The programme management component The overall aim of the programme management component was to analyse the extent to which the organisational set-up of the LIFE Unit and the modalities for selection and monitoring of LIFE projects contribute to the effectiveness and efficiency of the programme The programme management was evaluated via a) our assessment and b) feedback from beneficiaries (e-survey and interviews) cf below The combination of COWI expert assessment and beneficiary assessment allowed for a solid database with which to evaluate programme management 5 Further specification and elaboration of thematic objectives is done in connection with thematic studies and reflected in thematic reports 6 Other than the countries of Central and Eastern Europe which have concluded Association Agreements with the European Community O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

13 11 Figure 24 Approach to programme management evaluation Source: Developed by Consultant COWI assessment of programme management COWI's expert assessment of programme management was based on desk reviews, interviews and comparisons to other programmes On top of that, a LEAN inspired process optimisation analysis of key processes monitoring processes, was undertaken The analysis was conducted through two workshops facilitated by COWI consultants, where LIFE Unit staff analysed work processes in detail and identified potentials for efficiency gains The aim of the LEAN analysis was to provide: A measure of how man-power resources are spent in selection and monitoring processes A description of 'the good flows' and 'the bad flows' in the programme management cycle An increased focus on learning and improvement internally in the LIFE Unit Beneficiary assessments Through the e-survey and interviews with project managers and national focal points, the beneficiaries' assessments of programme management were solicited These data sources are combined and triangulated in order to answer the evaluation questions related to programme management 223 The policy component This section deals with the policy oriented evaluation themes consistency (hereunder coherence) and acceptability These themes touch upon how the LIFE programme links and fits in with other policies and instruments and how they are accepted by the broader public, both locally and nationally Coherence and consistency When assessing consistency, the main issue is how the LIFE programme links to other EU programmes and policies The success criterion is for the programme to support and complement other instruments and policies rather than to be funding exact same objectives (double funding) or on the other hand contradictory objectives to other programmes and policies We have checked consistency between the relevant instruments, hereunder, most notably, the financial instruments under the Structural Funds, Cohesion Funds and Framework Programmes In relation to the LIFE Nature component, the Community Agricultural Policy (CAP) and the Community Fishery Policy (CFP) are policies, which potentially contradict with environmental policy in general and the LIFE Programme and this was also analysed O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

14 12 The data collection has partly been a desk study and partly an analysis of the interviews in relation with the country studies However, we have also relied on the programme management analysis of the project selection and monitoring process The assessment was based on the following: Examining how the project selection process in a consistent way supports the EU policies Observing project managers and LIFE unit staff perceptions of the level of consistency and acceptability Beneficiaries' assessment of programme policy consistency and acceptability National focal points' perception of consistency and acceptability Acceptability The evaluation term 'acceptability' describes different target groups' attitudes towards the LIFE Programme both in general and for specific LIFE projects The main issue is to assess the level of acceptability of the LIFE Programme among project managers via the e-survey Acceptability can, for obvious reasons, be difficult to asses in objective terms and to a large extent we have drawn on the project managers' perceptions of local and national stakeholders' acceptance of the individual projects 23 The evaluation process - activities, time and resources Figure 25 shows an overview of the main activities in the evaluation The evaluation was divided into three overall phases: The inception phase, the data collection and analysis phase and the reporting phase Due to the complexity and size of the evaluation there was some overlap between the phases 231 Inception phase The inception phase began on 24 July 2008, when the contract was signed, and ended on 6 September 2008 when the first version of the inception report was submitted to the Steering Committee The main activities during the inception phase encompassed: Kick-off meeting with the LIFE Unit and informal meetings with LIFE Unit staff Internal kick-off and inception workshops for the evaluation team Collection of material on the LIFE Programme and ensuring access to ESAP and BUTLER databases (on selection and monitoring of projects, respectively) Preparation of consolidated evaluation methodology including formulation of evaluation questions, success criteria and indicators for each of the LIFE components and themes to be evaluated Set-up of a project database to store information retrieved from BUTLER to enable overview of projects per country, theme, etc as well as statistical analysis of projects and finance data Preparation of inception report describing the methodology and approach The Steering Committee provided comments to the inception report which was revised and extended to cover the final version of the questionnaire to be used in the e-survey and reporting on activities in the data collection phase The final inception report was submitted 6 November 2008 O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

15 13 Figure 25 Main activities in the evaluation Source: Developed by Consultant 232 Data collection and analysis phase This phase of the evaluation formally began on 7 September 2008 and included activities, which crosscut all components of the evaluation and activities, specific to each component These are described below This phase, which overlaps with the reporting phase, ended in mid-march when the country and thematic studies were finalised Cross-cutting activities: Project database, e-survey, and selection of projects for in-depth study The establishment of a project database containing information on all LIFE projects during the period was initiated during the inception phase The access to the BUTLER database granted to the Consultant, unfortunately, did not allow the Consultant to make extractions from BUTLER The LIFE Unit was very helpful in supplying tables extracted from BUTLER and the database has been continuously updated with new data throughout the evaluation process A raw version of the database created is submitted to DG Environment along with the final report The e-survey was launched 27 October 2008 and closed on 2 December 2008 Selection of ENV and NAT projects for in-depth study: In the original time schedule for the evaluation it was foreseen that the projects for in-depth study would be selected by 1 October Due to data collection difficulties there was considerable delay On 19 December, the final list was sent to the LIFE Unit (see details on selection process, criteria and final list in Appendix 5) O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

16 14 Programme management component The activities under the programme management component were initiated immediately after the inception phase and included the following: Desk review of the relevant material (evaluation guide, standard administrative provisions/common provisions, contract with monitoring teams, Vademecum, description of selection and monitoring teams, and GHK-study on monitoring teams) Interviews with staff from the LIFE Unit and monitoring team Two LEAN workshops with LIFE Unit staff Beneficiaries' assessment based on input from e-survey and country studies Analysis and report drafting Results and impacts component This component included country studies (which encompass in-depth project studies), thematic studies and the TCY study The activities are described below Country studies Country study guidelines: Guidelines were developed in a final draft version to be used in the country study workshop on 1 October 2008 The guidelines comprise background on the evaluation; description of outputs and tasks; templates for country study reports, project sheets (reporting on in-depth project studies) and interview reports; and interview guides The guidelines have been developed continuously during the period and the latest version was issued 2 December 2008 Country study workshop: On 1 October 2008, a country study workshop was held for the consultants responsible for country studies During this workshop, the templates and guidelines were presented and explained and the approach to the country studies discussed Pilot country studies: In the time schedule developed in the inception phase, it was foreseen that pilot country studies on Denmark and Sweden would be carried out and submitted by mid-october Due to the delays described above and more resources spent on e-survey, project selection, etc than originally foreseen, the pilot country studies were delayed It was decided to focus on Denmark and the pilot country report was submitted to the LIFE Unit on 17 November 2008 Information package: Each country study consultant received an information package containing data to be reviewed as part of the country study In addition to this, the country study consultants reviewed the latest national environmental action plan or similar document The package included: - Reference to the LIFE web-site - Extractions from the database concerning numbers of projects and projects according to year, theme, etc - Project summaries for all projects in the country - CD with monitoring files - Project reports for projects selected for in-depth study - LIFE Publications concerning the country in question - Results from the e-survey relevant to the country in question - Names and contact details of relevant interview persons O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

17 15 The main country study activities in each study are listed below - initial desk review of project summaries and suggestions for selection of projects for in-depth study - a desk review of the above mentioned documents - interviews with national coordinators - interviews with LIFE Unit country desk officers and monitoring teams - in-depth project studies for selected projects, including interviews with project managers and site visits to projects with observable results and impacts - compiling and assessing data on results and impacts according to indicator framework - assessing the effectiveness, utility and sustainability of the LIFE projects in respect to the country's needs and issues - reporting, in the form of a country report (chapter in final evaluation report), and project sheets, reports from interviews, etc to be provided to thematic teams for further study Thematic studies For both NAT and ENV themes, a categorisation of all projects (771 NAT and 1076 ENV) into themes were made and documented to the LIFE Unit (see Part 4 for details) The thematic studies comprised the following for each theme: Studying relevant EU policy documents Interviews with relevant staff in the LIFE Unit and DG ENV Review of results of e-survey Review of results of country studies, notably in-depth project studies Review of project results as reported in BUTLER Analysis/aggregation of project level data on effects to the thematic level Assessing effectiveness, efficiency, utility, sustainability Reporting TCY Study The main activities included desk study, 17 project studies, interviews with DG Environment officers and EC Delegations, and analysis and report writing 233 Reporting phase The reporting phase was initiated with the progress report submitted 6 January 2009, which was structured in accordance with the structure for this draft final report From January and until submission of the draft final report 20 April 2009, we worked in a targeted manner within the report structure as revised by comments received at the Steering Committee meeting concerning the progress report The report was subsequently revised again based on comments received resulting in the final report as presented in this volume O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

18 16 3 The LIFE Programme and its policy context This chapter provides a brief overview of the main characteristics of the LIFE Programme and its policy context More detailed and targeted descriptions on various parts of the programme are provided where relevant in sections of this evaluation report The chapter is structured into two main sections: The first section provides an overview of the main features of the Programme and the second section describes the LIFE projects during the evaluation period 31 The anatomy of the LIFE Programme 311 Generations of LIFE The LIFE Programme has existed since 1992 Until 2006, it was implemented in four phases or generations each governed by a separate Council Regulation The current, fifth generation (LIFE+- not encompassed by this evaluation), runs from 2007 to 2013 The evaluation period of covers three generations of LIFE: LIFE II ( ) LIFE III ( ) LIFE III-extension ( ) 312 Why LIFE? A description of the objectives The scope and priorities of the LIFE Programme, as outlined in its governing regulations, have developed over the three generations However, the programme has maintained a dual objective of implementation and development of Community environmental policy and legislation Over the years, there has been a trend towards increasing focus on implementation rather than development of policy and legislation O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

19 17 Table 31 Key developments of the LIFE programme Phase Regulation Years General Objective of LIFE Key EU developments LIFE I Council Regulation (EEC)1973/ Contribution to the development and implementation of Community environmental policy and legislation by financing: 1995: Austria, Finland and Sweden new MS (a) priority environmental actions in the Community; (b) (i) technical assistance actions with third countries [ ] LIFE II Council Regulation (EC) 1404/ Contribution to the development and, if appropriate, implementation, of Community environment policy and legislation LIFE III Regulation (EC) 1655/ Contribution to the implementation, updating and development of Community environment policy and of environmental legislation, in particular as regards the integration of the environment into other policies, and to sustainable development in the Community 2001: Restructuring of DG ENV; and adoption of the EU SDS 2002: the 6th EAP adopted 2004: eastern enlargement with 10 new MS LIFE III extension Regulation (EC) 1682/ Unchanged 2006: renewed SDS adopted LIFE + Regulation (EC) 614/2007 Source: elaborated by consultant Contribution to the implementation, updating and development of Community environmental policy and legislation, including the integration of the environment into other policies, thereby contributing to sustainable development In particular, LIFE+ shall support the implementation of the 6th EAP, including the thematic strategies, and finance measures and projects with European added value in Member States 2007: Romania and Bulgaria new MS Programme priorities clearly linked to the 6th EAP 313 Components in the LIFE Programme The LIFE Programme was divided into three components during the evaluation period: LIFE Environment, LIFE Nature and LIFE third countries (TCY) LIFE Environment (ENV) The scope of LIFE Environment has been fairly similar during the three generations of LIFE and has focused on the development of innovative and integrated techniques, demonstration projects, and development of Community environmental policy The share of the LIFE Programme budget allocated for LIFE Environment was 46% under LIFE II and 47 % under LIFE III and LIFE III extension LIFE Nature (NAT) The specific objectives of LIFE NAT have remained fairly constant during the evaluation period and have focused on the implementation of the Birds and Habitats Directives, as exemplified by art 31 of O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

20 18 the LIFE III Regulation: "The specific objective of LIFE-nature shall be to contribute to the implementation of Council Directive 79/409/EEC of 2 April 1979 on the conservation of wild birds, Council Directive 92/43/EEC of 21 May 1992 on the conservation of natural habitats and of wild fauna and flora and, in particular, the Natura 2000 European network established by the latter Directive" Similarly to LIFE Environment, the LIFE Nature budget share was 46% and 47 % under LIFE II and III, respectively LIFE Third countries (TCY) The focus of LIFE Third Countries was on providing support to capacity building in the environmental sector and on the development of environmental policy and action programmes in third countries bordering on the Mediterranean and the Baltic Sea 7 LIFE TCY was allocated 5% of the budget under LIFE II and 6% under LIFE III and LIFE III extension Other Under LIFE II, a separate budget was set aside for "accompanying measures needed to monitor, evaluate or promote the actions undertaken during the first stage and under points 1 [environment, hereunder nature] and 2 [TCY] and dissemination of information on the experience and the results obtained of such actions" These actions were allocated 3% of the total budget under LIFE II 314 The LIFE cycle The LIFE Programme is basically organised in an annual cycle Each year project proponents can apply for LIFE co-financing according to criteria and guidelines announced by the LIFE Unit The project applications are submitted to the LIFE Unit by the Member States The project applications are then evaluated by the LIFE Unit and projects selected according to a set of criteria During implementation, the projects are monitored by the LIFE Unit This cycle is illustrated in Figure 31 below 7 Other than the countries of central and Eastern Europe which have concluded Association Agreements with the European Community and are referred to in Article 6(1) (LIFE III art 5) O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

21 19 Figure 31 The LIFE Cycle Source: Developed by Consultant 315 The input: Budget The LIFE regulations stipulate the total budget allocated to the LIFE programme for the phase to which the regulation applies The annual average budget of the programme has increased from phase to phase - from EUR 1125 million under LIFE II to EUR 1586 million under LIFE III extension O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

22 20 Table 32 The budget developments of the LIFE Programme LIFE programme Years Budget million EUR Annual Average million EUR LIFE I Council Regulation (EEC)1973/ LIFE II Council Regulation (EC) 1404/ LIFE III Regulation (EC) 1655/ LIFE III extension Regulation (EC) 1682/ LIFE + Regulation (EC) 614/ , Source: the respective LIFE regulations 32 Overview of LIFE projects during the period 1996 to 2006 During the period 1996 to 2006, 2033 projects were approved for LIFE co-financing These included 771 Nature projects, 1076 Environment projects and 179 Third Country projects Of these projects, 291 are still on-going 8 This section provides a brief overview of the projects More specific overviews per country and theme are provided in the relevant country and thematic reports The data presented is from the BUTLER database of the LIFE Unit It should be noted that there are several uncertainties: The data on commitments in the BUTLER database mostly stems from the original project budgets and hence, does not reflect the actual payments made (or foreseen) as budgets are often changed during the projects' lifetime It was not possible to supply data on payments which could be aligned with the BUTLER data as this data exists only in the accounting systems from where the necessary tables could not be created The BUTLER database also includes the projects closed down Through accounting data from , it was possible to identify some of these projects and delete them from the database, but interview data indicates that there are additional projects in BUTLER, which were, in fact, never financed Table 33 LIFE projects and commitments LIFE Generation LIFE Environment LIFE Nature LIFE TCY Total Projects Commitm ent (million EUR) Projects Average commitm ent (EUR) Projects Commitm ent (million EUR) Projects Commit ment(mi llion EUR) LIFE II LIFE III LIFE III extension Total Source: Data in BUTLER database 8 For the purpose of this evaluation on-going projects are defined as projects with end-year in 2009 or later as registered in the BUTLER database O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

23 21 The overall committed amount during this period amounted to EUR 1,315 million As shown in Table 33, the number of projects approved annually decreased during the evaluation period whereas the LIFE contribution to each project has increased Table 34 Annual average number of projects and commitment per project, LIFE Generation LIFE Environment LIFE Nature LIFE TCY Total Projects per year Commitm ent per project (million EUR) Projects per year Commit ment per project (million EUR) Projects per year Commit ment (million EUR) Projects per year Commit ment per project (million EUR) LIFE II LIFE III LIFE III extension Total Source: Data in BUTLER database 321 Themes For the purpose of this evaluation, we have divided Nature and Environment projects according to the themes listed in Table 35 These themes have been inspired by the LIFE+ Regulation and their categorisation is described in more detail in Part 4 on the thematic studies Table 35 shows that the Environment themes 'water', 'natural resources and waste' and 'strategic approaches' are the most prevalent themes in terms of number of projects Although less so this is also the case when looking at the LIFE contribution For the Nature projects, about three quarters are categorised as habitats project serving to implement the Habitats Directive and one fourth as bird project and thus serving to implement the Birds Directive The number of "other" projects 9 is very small 9 See Part 4 for further explanation O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

24 22 Table 35 Number of Environment and Nature projects and committed amounts according to themes Theme Environment Climate change Air Water Soil Forests Natural resources and waste Chemicals Strategic approaches Urban environment Nature Habitats Birds Other Number of projects Number of projects (%) Committed amount (million EUR) Committed amount (%) 1, % % 39 21% % 93 50% % % % 92 50% % 15 08% 61 05% % % 17 09% 95 08% % % 69 37% % % % % % % % 10 05% 29 02% Total 1, % % Source: BUTLER database Note that the total here is smaller than in Table 35 above as this table only covers NAT and ENV projects (not TCY) 322 Types of beneficiaries For the purpose of this evaluation, we have distinguished between three main types of beneficiaries: Public entities; NGOs and research institutions; and public and private enterprises Table 36 shows that public entities are the most typical beneficiary accounting for 44% of all projects However, public entities are particularly important in relation to Nature and Third Country projects and less so in relation to Environment projects, where the beneficiary is more often a public or private enterprise More detailed tables on the specific type of beneficiary can be found in Appendix 7 Table 36 Beneficiaries of LIFE Projects, Beneficiary type LIFE Environment LIFE Nature LIFE Third Countries Total Public entities 360 (33%) 448 (58%) 92 (51%) 900 (44%) Public and private enterprises NGOs and research organisations 429 (40%) 19 (2%) 14 (8%) 462 (23%) 194 (18%) 226 (29%) 58 (32%) 478 (24%) Not specified 93 (9%) 78 (10%) 15 (8%) 186 (9%) Total 1076 (100%) 771 (100%) 179 (100%) 2026 (100%) Source: BUTLER database O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

25 23 Appendix 1 Indicators for assessment of results and impacts During the inception phase, we have worked on identifying indicators to measure effects (results and impacts) at the thematic level, which can be applied to the projects analysed and which can be used as a basis for aggregation from the project level to the thematic level We have done this through: Reviewing the definition/objectives for the themes, as set out in the LIFE+ Regulation compared to the LIFE II and III regulations Reviewing a number of project summaries to trace type of result and impact for typical projects under the various themes Reviewing the 6th EAP, the related thematic strategies and EU regulatory documents Reviewing the GHK Study on the Establishment of Indicators to Assist Monitoring of Measures Financed by LIFE+ Reviewing indicators suggested by the European Environment Agency (Core Set of Indicators, ref Technical Report No 1, 2005 and Halting the Loss of Biodiversity, ref Technical Report No 11/2007) Our primary focus has been on developing indicators, which enable the measurement and subsequent aggregation of effects from project level to programmatic level Where relevant, this has been supplemented by other types of indicators focusing on a more indirect assessment of results and impacts Ie most indicators aim for a quantification of impact, but in the cases where this was deemed impossible, relative indicators using, for example, a scale from 1 to 5 are suggested The indicators developed are divided into two sets - one set pertaining to the results of the projects under each theme, and one set pertaining to the impacts of the projects By comparison the GHK study refers to three types of effect indicators: results, outcome and impact, where result indicators refer to the immediate effects, outcome indicators refer to intermediate effects and impact indicators refer to longer term effects 10 For the purpose of this evaluation we do not find it realistic to distinguish between intermediate and longer term impacts This may be carried out in an evaluation of individual projects but not in this programmatic evaluation given the large portfolio of projects analysed and the many different start and finish dates Rather, we have sought to develop both (short term) results indicators and (longer term) impact indicators for five different types of effects which address the types of changes mentioned under effects in the GHK study (ref figure 32, p 11) We have identified five different types of effects as listed below Environment: Improved state of or reduced pressure on environment, nature and/or human health Management: Improved management, planning, administration or regulation Awareness: Increased awareness among a broader range of stakeholders about the environmental issue in question Demonstration: "showcase effect" of demonstrating specific models, methods and/or technologies to potential users Innovation: Innovation/development of new technology or improvement of processes eg to save energy or other natural resources, cause less pollution or reduce the generation of waste 10 GHK Study, p 11 O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc

26 24 We have developed theme-specific indicators for environmental and management effects These are described below in a separate section on each theme We have also developed a number of general indicators which are NOT theme-specific for management, awareness, demonstration and innovation effects These are described in the final section at the end of this appendix As also stated in the GHK study, environmental effects are likely to occur mostly in the medium to long term and thus impact indicators seem more relevant than result indicators Nevertheless, in many cases, we suggest that the indicators for measuring both results and impacts be used, as some projects may have generated environmental effects even in the short term and these should be documented by the evaluation It will then also be possible to compare short term results with long term impacts and, hopefully, illustrate extended impact over time and the sustainability of the projects Below, we first present theme-specific indicators for the themes in the following order: LIFE NAT (one common set of indicators developed to cover the four types of conservation actions according to which the analysis of LIFE NAT will be structured) LIFE ENV - climate change - air - water - soil - forests - natural resources and waste - chemicals - urban environment - strategic approaches Finally, we present general indicators related to management, awareness, demonstration and innovation effects Indicators for results and impacts under LIFE NAT The NAT component of the LIFE Programme has existed in all the generations of LIFE being evaluated in this ex-post evaluation This component's focus has been relatively consistent throughout time period covering the implementation of the Habitat Directive and the Birds Directive Indicators have been developed accordingly As described in Chapter 3, the guiding principles for categorisation of NAT projects have not yet been settled It is assessed that results and impact indicators for the NAT projects, as a group, will remain applicable no matter which guiding principles for categorisation are applied O:\A000000\A001146\Final Report for PDF\Part 1_LIFE_finrep_1doc