Independent Evaluation of the Climate Investment Funds

Transcription

1 Independent Evaluation of the Climate Investment Funds Approach Paper Approved by the joint CTF/SCF Trust Fund Committee, September 6, 2012 Evaluation Oversight Committee African Development Bank Group, Operations Evaluation Department Asian Development Bank, Independent Evaluation Department European Bank for Reconstruction and Development, Evaluation Department Inter-American Development Bank, Office of Evaluation and Oversight World Bank Group, Independent Evaluation Group cifevaluation.org August 21, 2012

2 Acronyms and Abbreviations AfDB ADB CIF EBRD EvDs EOC FCPF FIP GEF GRPP IBRD IDA IDB IEG IFC GHG LDCF MDB NAMA NAPA OECD/DAC PPCR REOI RFP SCF SREP TOR UNFCCC UN-REDD African Development Bank Asian Development Bank Climate Investment Fund European Bank for Reconstruction and Development Evaluation Departments (of the five MDBs) Evaluation Oversight Committee Forest Carbon Partnership Facility Forest Investment Program Global Environment Facility Global and/or Regional Partnership Program International Bank for Reconstruction and Development International Development Association Inter-American Development Bank Independent Evaluation Group (World Bank Group) International Finance Corporation Greenhouse gas Least Developed Countries Fund Multilateral Development Bank Nationally Appropriate Mitigation Actions National Adaptation Program of Action Organisation for Economic Cooperation and Development / Development Assistance Committee Pilot Program for Climate Resilience Request for Expressions of Interest Request for Proposals Strategic Climate Fund Scaling Up Renewable Energy Program in Low Income Countries Terms of Reference United Nations Framework Convention for Climate Change The United Nations Collaborative Programme on Reducing Emissions from Deforestation and Forest Degradation in Developing Countries Fiscal Year of CIF Programs July 1 to June 30

3 i Contents Preface... iii Part I. The Substance of the Evaluation Purposes, Objectives, and Principles of the Climate Investment Funds Purpose and Scope of the Evaluation Evaluation Issues and Questions... 4 A. Development Effectiveness... 4 (1) Relevance... 4 (2) Efficacy... 5 (3) Efficiency, Financial Additionality, and Leverage... 5 (4) Sustainability... 5 B. Organizational Effectiveness... 5 (5) CIF Governance and Management... 5 (6) Administrative Efficiency... 6 (7) National Planning and Consultation Processes... 6 (8) Monitoring and Evaluation (M&E)... 6 (9) Safeguard Mechanisms Evaluation Design and Methodology... 7 Information Gathering and Analysis... 7 Major Evaluation Products... 8 Inception Report... 8 Interim Report and Progress Reports... 8 Final Reports... 9 Key Methodological Issues... 9 Selection of Countries for Case Study Visits... 9 In-Country Consultation Procedures... 9 Definition and Measurement of Key Concepts Presentation of Evidence Part II. The Evaluation Process Oversight and Management of the Evaluation International Reference Group Qualifications and Selection of the Evaluation Team Indicative Timetable and Schedule Dissemination... 16

4 9. Obligations of Key Participants in the Evaluation Obligations of the CIF Trust Fund Committees Obligations of the Evaluation Oversight Committee Obligations of the Evaluation Department Heads Obligations of the CIF Administrative Unit and MDB Committees Obligations of the Consultants Obligations of Other Parties (e.g., country offices of the involved MDBs) Annex A. Description of the Climate Investment Funds Annex B. Detailed Evaluation Questions Annex C. Indicative Work Packages Tables Table 1. Members of the Evaluation Oversight Committee... iii Table 2. Indicative Timetable Table A-1. Purpose and Objectives of the Two Climate Investment Funds Table A-2. Principles of the Two Climate Investment Funds Table A-3. CIF Participating Countries, by Region and Program Table A-4. Measures to Improve the Operations of the CIFs Table A-5. Donor Pledges and Contributions to the Climate Investment Funds, Fiscal Years (US$ millions) Table A-6. Trust Fund Commitments and Disbursements, by Program and MDB, Through June 30, 2012 (US$ millions)... 28

5 iii Preface The Clean Technology Fund (CTF) and the Strategic Climate Fund (SCF) are linked global partnership programs that were established in 2008 to address climate change mitigation and adaptation via concessional funding for projects executed by the Multilateral Development Banks (MDBs). The involved MDBs are the African Development Bank, the Asian Development Bank, the European Bank for Reconstruction and Development, the Inter- American Development Bank, the International Finance Corporation, and the World Bank. The governance frameworks of the two Climate Investment Funds (CIFs) provide that an independent evaluation of the operations of the CTF [and SCF] will be carried out jointly after three years of operations by the independent evaluation departments of the MDBs. Such evaluation will be based on the scope and reporting criteria agreed with the two Trust Fund Committees. In response to the formal request from the chairs of the CTF and SCF Trust Fund Committees, the five Evaluation Departments (EvDs) of the MDBs have agreed to oversee and manage this independent evaluation and have established an Evaluation Oversight Committee (EOC) for this purpose, the members of which are listed in Table 1. The May 2012 meeting of the joint CTF-SCF Trust Fund Committees approved a framework governing the independence of the evaluation. Table 1. Members of the Evaluation Oversight Committee Evaluation Department Principal Alternate Operations Evaluation Department, AfDB Detlev Puetz Madhu Mampuzhasseril Independent Evaluation Department, AsDB Kapil Thukral Kelly Hewitt Evaluation Department, EBRD Dennis Long Amelie Eulenburg Office of Evaluation and Oversight, IDB Monika Huppi Veronica Diez Gonzalez Independent Evaluation Group, Kenneth Chomitz (chair) World Bank Group (IFC and Chris Gerrard the World Bank) An initial consultation draft of this Approach Paper was circulated to CIF Committee and Sub-Committee members and observers prior to the May meeting. Based on consultation at that meeting, a revised draft was circulated, and posted on the evaluation website ( in order to receive feedback from CIF stakeholders and the public. The EOC has now further revised the Approach Paper based on the extensive comments received. The present document represents the final Approach Paper that the EOC is submitting, along with an evaluation budget, for approval by the CTF and SCF Trust Fund Committees. The first part of this Approach Paper (Chapters 1 4) addresses the substance of the evaluation and the second part (Chapters 5 9) covers the evaluation process.

6

7 1 Part I. The Substance of the Evaluation 1. Purposes, Objectives, and Principles of the Climate Investment Funds 1.1 The Clean Technology Fund (CTF) and the Strategic Climate Fund (SCF) are a pair of funds that were established in 2008 to help developing countries pursue low-emissions and climate-resilient development. They were established by the five Multilateral Development Banks (MDBs) to contribute to the goals of the Bali Action Plan adopted in December 2007 by the Conference of the Parties of the United Nations Framework Convention on Climate Change (UNFCCC). They represent interim measures designed for the MDBs to assist in filling immediate financing gaps, while the UNFCCC deliberates on the future of the climate change regime, including discussions on a future financial architecture and funding strategy for climate change (now called the Green Climate Fund) The independent evaluation of the Climate Investment Funds will be objectives-based and evidence-based in relation to their purposes, objectives, and principles. 1.3 The purpose of the CTF is to provide scaled-up financing to contribute to demonstration, deployment and transfer of low-carbon technologies with a significant potential for long-term greenhouse gas (GHG) emissions savings. That of the SCF is to provide financing to pilot new development approaches or scale-up activities aimed at a specific climate change challenge or sectoral response (Annex Table A-1). 1.4 The objectives of the programs are to provide new and additional financing (in the form of grants, concessional loans, and risk mitigation instruments) to complement existing bilateral and multilateral financing mechanisms in order to demonstrate and deploy transformational actions to mitigate and adapt to climate change. The programs also aim to promote international cooperation on climate change, to foster environmental and social cobenefits of sustainable development, and to learn lessons of experience. 1.5 The two programs have adopted a number of guiding principles (Annex Tables A-2 and A-4). Adaptation and mitigation programs to address climate action should be country-led and designed to support sustainable development and poverty reduction. Activities financed by the funds should be based on a country-led approach and should be integrated into countryowned development strategies, consistent with the Paris Declaration The MDBs, in collaboration with other development partners, should assist developing countries to build country-level knowledge, capacity and development project experience. Effective cooperation should be established between the CIF, the Global Environment Facility, and the United Nations, especially at the country level, to maximize synergies and avoid overlap. 1..See Annex A for a more detailed description of the Climate Investment Funds. See for more information about the Green Climate Fund. 2. The five principles of the 2005 Paris Declaration on Aid-Effectiveness are (a) ownership, (b) alignment, (c) harmonization, (d) managing for results, and (e) mutual accountability.

8 2 1.6 The SCF has established three subprograms: Pilot Program for Climate Resilience (PPCR), approved November 2008 to pilot and demonstrate ways in which climate risk and resilience may be integrated into core development planning and implementation. Forest Investment Program (FIP), approved July 2009 to support developing countries efforts to reduce emissions from deforestation and forest degradation by providing scaled-up bridge financing for readiness reforms and public and private investments. Scaling Up Renewable Energy Program in Low Income Countries (SREP), approved May 2009 to demonstrate the economic, social and environmental viability of low-carbon development pathways in the energy sector by creating new economic opportunities and increasing energy access through the use of renewable energy. 1.7 Collectively, the CTF and the three subprograms of the SCF have identified 48 participating countries, based on independent expert involvement (Annex Table A-3). The CIFs help finance country-specific investment projects to achieve nationally-defined objectives consistent with the objectives of the two programs. The funds are channeled through the five MDBs, which are responsible for working with national governments and other development partners to help prepare national investment plans, 3 individual projects, and associated financing packages to achieve mutually agreed objectives that support the national development agendas of the participating countries. 1.8 The share of CIF funding managed by each MDB is based on country requests, the quality of proposals, the comparative advantage of each MDB, and experience in a region/country. The MDBs rely on their own policies and procedures in developing and supervising activities financed by the CIFs, and report directly to the two Trust Fund Committees on operational matters. 1.9 The CIFs seek to catalyze the commercial private sector (including financial intermediaries), recognizing that many of the required investments will be privately financed, and that private entities have particular competencies that can uniquely contribute to adaptation and mitigation. Hence, the CIFs offer a mix of financial incentives, risk mitigation tools, technical assistance, and knowledge transfer that can help make adaptation and mitigation investments more attractive to the private sector. 3. Unless otherwise indicated, the term investment plan refers to CTF, FIP, and SREP investment plans, and to PPCR strategic programs for climate resilience.

9 3 2. Purpose and Scope of the Evaluation 2.1 The evaluation will be primarily a formative evaluation of how the programs are being implemented, since the two programs are less than four years old. 4 It will also contain summative assessments, where possible, of the programs more advanced activities, including CTF projects in implementation. For less-advanced investment plans and projects, the evaluation will rely on assessments of the underlying theory of change and logic models. 2.2 The evaluation will cover the two programs (CTF and SCF), the three targeted subprograms of the SCF, and the country-level activities that they are supporting. 2.3 The evaluation will cover the time period from initial CIF concept to the time at which the evaluation activities (such as country visits) are carried out. 2.4 The principal purposes of the evaluation are: (a) To assess the development effectiveness and the organizational effectiveness of the Climate Investment Funds to date. (b) To document experiences and lessons for the benefit of the Green Climate Fund. 2.5 Development effectiveness is understood in the broad sense to comprise the OECD/DAC evaluation criteria of relevance, efficacy, efficiency, and likely sustainability, 5 appropriately interpreted for global partnership programs like the CIFs. Organizational effectiveness refers to the proficiency of the programs governance and management structures, functions, and processes in facilitating the achievement of the programs purposes and objectives in an efficient and transparent manner. Thus, this evaluation, like other evaluations of global partnership programs, goes beyond the OECD/DAC principles to include an assessment of the governance and management of the partnership The evaluation will assess the development effectiveness and organizational effectiveness of the CIFs at different levels: the program and subprogram level, the country level, 7 and the project level. The organizational effectiveness of the partner environment at the country level in formulating national investment plans and individual projects is of 4. A formative evaluation is conducted during the implementation phase of a program to examine how the program is being implemented, the extent to which the program s actual operations correspond to its intended logic, and the immediate consequences of the program s implementation. A summative evaluation is conducted on a mature program, or after it has been implemented, to determine the extent to which the intended results were achieved. See Linda G. Morra Imas and Ray C. Rist, 2009, The Road to Results, pp See OECD/DAC 1991, Principles for Evaluation of Development Assistance, and OECD/DAC 2002, Glossary of Key Terms in Evaluation and Results Based Management. To avoid confusion, this Approach Paper uses the term efficacy in place of the narrow definition of the term effectiveness in the 2002 Glossary. 6. See IEG/DAC 2007, Sourcebook for Evaluating Global and Regional Partnership Programs, for the application of OECD/DAC evaluation criteria to evaluating GRPPs, and for approaches to assessing governance and management. This is available at 7. Including the CIFs multi-country programs in the Caribbean, the Middle East and North Africa, and the Pacific.

10 4 particular interest, because the projects financed are intended to be based on country-owned development strategies consistent with the objectives of the CIFs, and because the financing is intended to complement, not compete, with other sources of bilateral and multilateral financing. 2.7 The actions which the CIFs have taken to involve and target poor, vulnerable, and marginalized groups who are disproportionately impacted by the negative effects of climate change, including indigenous groups and women, is a significant cross-cutting issue for the evaluation. The evaluation will, among other things, pay attention to the involvement of such groups in the formulation of national investment plans, the extent to which the objectives and designs of plans and projects address their needs and concerns, and the extent to which monitoring systems have been put in place to assess the impacts upon them. 2.8 The extent to which the CIFs are generating awareness, knowledge, and lessons from experience is also a key issue. Effectively communicating and sharing lessons from CIFfunded investments aims to contribute to multiplier effects on GHG mitigation and reduced climate vulnerability at the national, regional and global levels. 3. Evaluation Issues and Questions 3.1 The evaluation will address a set of evaluation issues and questions in relation to the development effectiveness and the organizational effectiveness of the CIFs. Each of these will be explored at applicable levels (program, subprogram, country/regional, and project). 3.2 The following nine evaluation issues are illustrative of key formative and summative questions in relation to the purposes, objectives, and principles of the Climate Investment Funds. Annex B provides a more detailed elaboration of the issues and questions summarized here, drawing on input during consultations. The final set of evaluation questions under each of the nine issues, and the ways in which the questions will be addressed, will be outlined in the inception report prepared by the evaluation team. (See section 4 below.) An International Reference Group of independent experts will also be involved in reviewing the inception report. (See section 6 below.) A. Development Effectiveness (1) RELEVANCE The extent to which the CIF investment plans are consistent with the needs and priorities of beneficiary countries as articulated in the countries own development and climate-related strategies. The extent to which the programs are complementing or competing with other climate-related funds, and their consistency with the UN-led climate process. The extent to which the design, strategic approaches and investment priorities of the CIFs are appropriate for achieving their objectives. The extent to which the CIF actions or plans are plausibly designed to bring about sectoral, market, or behavioral transformation.

11 5 (2) EFFICACY The likelihood that the CIFs will achieve their stated objectives. The progress that the CIFs have made in terms of formulating and implementing national investment plans and projects, and the outputs and outcomes that have so far been achieved at the country level. The impact of the CIFs on the enabling environment and incentives for private sector investment in climate-resilient, low-carbon development. The extent to which CIF investment plans and projects are promoting and achieving economic, environmental, social, and gender equality co-benefits and avoiding negative impacts, with attention to impacts on women, marginalized groups and indigenous groups. How effectively, and on what issues, the CIFs are fostering and disseminating learning from the activities that they are supporting. (3) EFFICIENCY, FINANCIAL ADDITIONALITY, AND LEVERAGE The extent to which national investment plans and projects have been cost-effective in design (and where observed, in execution). The extent to which CIF funding represents the mobilization of additional funds (donor and otherwise, at the international or country levels) for low-emission and climate-resilient development consistent with the objectives of the CIFs. The extent to which MDB capabilities contributed to the mobilization and delivery of concessional climate financing on a significant scale. The extent to which CIF-supported investments have causally crowded in, or crowded out, additional public or private funding, including MDB cofinancing. (4) SUSTAINABILITY The extent to which CIF projects are likely to have long-lasting, transformational impacts. The extent to which the project designs have identified risks to the sustainability of the benefits and taken steps to mitigate these risks. The extent to which the benefits arising from the projects are likely to be sustained, taking into account the complementary activities of other development partners, and the institutional and human resource capacity of beneficiary countries. B. Organizational Effectiveness (5) CIF GOVERNANCE AND MANAGEMENT The extent to which have the programs governance arrangements have permitted and facilitated the effective participation and voice of different categories of stakeholders, taking into account their respective roles. The extent to which the programs governance and management arrangements have facilitated efficient decision making.

12 6 The extent to which the programs decision making and reporting have been transparently available to the public, subject to legitimate confidentiality requirements. The extent to which investment plans and projects comply with CIF investment criteria and guidelines. The extent to which real and perceived conflicts of interest have been identified and managed transparently at all levels (global, regional, and country). (6) ADMINISTRATIVE EFFICIENCY How effectively and efficiently the CIF Administrative Unit has carried out its administrative and management responsibilities, including managing CIF partnerships and external relations. How effectively and efficiently the MDB committees have managed coordination among the MDBs. How the preparation, approval time, and thoroughness of the review of national investment plans and projects has compared with ex-ante CIF expectations and with the experience of comparator organizations. (7) NATIONAL PLANNING AND CONSULTATION PROCESSES The effectiveness of country-level partnership arrangements in the formulation of national investment plans and the design and implementation of individual projects. The extent to which these processes have involved the participation and input of a wide range of stakeholder groups, including the commercial private sector, women, marginalized groups, and indigenous groups. The extent to which these processes have triggered institutional or procedural innovations at the country level, including the expanded involvement of the private sector, women, marginalized groups, and indigenous groups. The extent to which the resulting plans are country-led and integrated into countryowned development strategies consistent with the 2005 Paris Declaration principles, including harmonization with other sources of climate finance. (8) MONITORING AND EVALUATION (M&E) The quality of the design, implementation and utilization of M&E in assessing results, facilitating decision making, and learning at the project, country, and program levels. The extent to which the M&E systems are monitoring project outcomes for poor, vulnerable and disadvantaged groups such as indigenous groups and women. How effectively the MDBs are managing the inherent tensions that exist between building up country M&E systems and utilizing MDBs own systems in order to demonstrate accountability to taxpayers. (9) SAFEGUARD MECHANISMS The extent to which environmental and social risks are being efficiently identified and effectively mitigated.

13 7 The extent to which MDB safeguard procedures are consistent with each other and with CIF objectives, and the consequences of inconsistencies (if any). The extent to which the MDBs are providing detailed information on the application of their safeguards to each project. 4. Evaluation Design and Methodology 4.1 This comprises gathering, validating, cross-checking and analyzing information, and summarizing the evidence in a form that supports the findings. Annex C presents an illustrative work program based on the elements below. Information Gathering and Analysis 4.2 Primary and secondary data will be gathered, in the following ways, among others: Documentary review of the CIFs founding documents, minutes of meetings, annual reports, activities, etc. Portfolio review of national investment plans and projects. Review of CIF monitoring and evaluation data. Access to national and global databases on renewable energy, energy efficiency, transport, and forest investments, and on climate and development finance. Surveys and semi-structured interviews of Trust Fund Committee members and observers, MDB staff, CIF Administrative Unit staff, country-level stakeholders, informed observers, etc. Potential follow-up interviews, group discussions or focus groups for obtaining indepth feedback on issues arising from interviews and surveys. Missions to MDBs, and other key partners to interview staff involved in or knowledgeable about CIF operations. Missions to selected participating countries, comprising interviews with executing units of CIF projects, MDB resident offices, government authorities, involved civil society and private sector groups. (See paragraph 4.12 below.) 4.3 The primary information will be analyzed, among other methods, using the following methodologies: Synthesis of a coherent, time-ordered sequence of CIF actions at the program, country, and project levels. Analysis of the CIFs portfolios of national investment plans and projects. Country and regional partnership assessments based on missions to participating countries. Quantitative analysis of financial flows, and mitigation and adaptation investments. Qualitative analysis (category building, trend identification) of structured interviews and survey information. Synthesis, appraisal and triangulation of review findings, hypothesis building and testing. Feedback analysis: factual or logical errors, inclusion of new data.

14 8 Major Evaluation Products 4.4 The team of evaluation consultants that is contracted by the EOC henceforth called the evaluation team will be responsible for preparing the following evaluation products: An inception report An interim report Draft final report Final evaluation report All reports will comprise an executive summary, the main text, and annexes. Reports will be subject to length limits. The evaluation teams will deliver all reports solely to the EOC, who will share them with the International Reference Group, the CIF Administrative Unit, the Trust Fund Committees, and other CIF stakeholders. The draft final report will be made available for public comment and the final report will be approved by the EvDs and transmitted to the joint CTF-SCF Trust Fund Committee. INCEPTION REPORT 4.5 The evaluation team will prepare the inception report, approximately one month into the contract, after it has had the chance to meet with the EOC and the Administrative Unit, review background materials, and get a sense of the adequacy of the available data. This will provide details with respect to evaluation design and methodological issues such as methods for collecting new data, sampling, methods of analysis, quality assurance methods, and the countries to be visited (including the criteria and processes for selecting these countries). This is intended to ensure that the evaluation design and methodology uses the full expertise of the evaluation team as well as the full knowledge of the program. INTERIM REPORT AND PROGRESS REPORTS 4.6 The interim report will be based largely on desk reviews, meetings with the CIF Trust Fund Committees, the MDB Committees, and the CIF Administrative Unit, and (schedule permitting) with stakeholders at the Partnership Forum in Istanbul in November The report will provide preliminary answers to the evaluation questions relating to the global/program level, and the results of desk reviews of national investment plans and project portfolios. The EOC will provide the Trust Fund Committees and the CIF Administrative Unit with the opportunity to review this report for factual errors and to provide comments, but the evaluation team may proceed with the country visits before these comments are received. It is understood that the interim report is tentative and that findings are subject to revision as additional evidence is gathered, particularly from the country visits. 4.7 The evaluation team will also prepare monthly progress reports and back to office reports of the country visits to the EOC. Members of the EOC may participate in some of the country visits.

15 9 FINAL REPORTS 4.8 As explained in more detail in the second part of this Approach Paper, the EOC will provide the CIFs with the opportunity to review the draft final report for factual errors and to provide comments. There will also be an opportunity for public comment and consultation. The evaluation team will take these comments into account in finalizing the report, as appropriate. The final evaluation report will not be subject to modification or negotiation by the Trust Fund Committees, the CIF Administrative Unit, or other CIF stakeholders. Key Methodological Issues 4.9 Approaches to key methodological issues will be proposed by the evaluation team, reviewed by the International Reference Group (see section 6 below), and discussed and confirmed with the Evaluation Oversight Committee. The inception report will serve to finalize these issues and procedures and will be the entry point for the International Reference Group. Key methodological issues include, but are not restricted to the following. SELECTION OF COUNTRIES FOR CASE STUDY VISITS 4.10 The selection of countries will be cost-effective, unbiased, and representative of the 48 participating countries by region and program (Annex Table A-3). The stratification constraints below are necessary to permit comparisons among programs and MDBs, and to allow assessment of how procedures have changed as the CIFs have gained experience. Additional constraints may be desirable for analytic purposes. If the constraints do not fully determine the sample, the team will propose a random sampling method that satisfies the constraints The constraints include: 10 to 12 country visits Inclusion of at least two countries connected with each of the four programs or subprograms Inclusion of at least two countries associated with each MDB Inclusion of at least three countries with CTF projects under implementation Inclusion of both older and newer investment plans. IN-COUNTRY CONSULTATION PROCEDURES 4.12 The country case studies and project impact assessments should seek to costeffectively obtain the views and experience of as wide a range as possible of implementers, participants, stakeholders and observers. It is critical to include marginalized groups, indigenous groups and women. Others include government, MDB, and private sector staff directly involved in implementation; individuals and communities directly affected by plans and projects; other government personnel, including those with regulatory oversight and relevant monitoring and evaluation responsibilities, local leaders and representatives; academia and think tanks; members of the private sector including financiers.

16 The evaluation team may use innovative techniques including social media and mobile phone-based surveys to cost-effectively reach groups who are accessible by phone or internet while at the same time ensuring attention to those who do not have access. DEFINITION AND MEASUREMENT OF KEY CONCEPTS 4.14 Three crucial, related, and partially overlapping concepts are important to this evaluation: transformation, leverage, and additionality. These terms are subject to different interpretations. The evaluation team, with the advice of the International Reference Group and in consultation with the EOC, will specify definitions and protocols for quantifying or assessing these aspects of CIF impact. The following guidance is provided Transformational. The evaluation will elicit the understanding of transformational by the CIF framers and by the Trust Fund Committees. This will be compared with several widespread interpretations of the term: Sectoral or market transformation occurs when there is a large shift in the market share of a product or behavior for example, renewable power increases its share of power production from a negligible proportion to a majority. A plan or a project is transformational if it has a catalytic impact that is, if it triggers spontaneous replication, for instance by decreasing risk or by removing regulatory or other barriers. Transformation, in this sense, could contribute to transformation in first sense without being as immediately far-reaching. A plan or a project is transformational if it results in a step-change in development rather than an incremental change that is, if it shifts a country onto less carbonemitting and more climate-resilient development pathways In all cases, transformation is measured, conceptually, against a counterfactual of no CIF intervention. The evaluation will assess how transformation in these different senses apply, and if feasible will propose an approach to their quantitative measurement Additionality. Additionality of funding compares direct CIF-related funding to a without-cif counterfactual. This refers to public sector funding; private sector impacts are considered under leverage (below). At the international level, CIF additionality represents an increment to donor funding for development and climate. At the MDB level, CIF additionality represents an increment to available funds for loans, grants, and other investments. At the national level, CIF additionality represents an increment to public sector budgets and private sector resources. Given that funds may be fungible, analysis may include sectoral disaggregation of the use of additional funds It is recognized that defining a counterfactual is difficult. The evaluation will define a feasible approach to additionality measurement and will note its assumptions and limitations Leverage. Leverage is distinct from cofinancing, and refers to the extent to which a CIF investment is a causal factor in mobilizing or triggering the investment of additional funds which otherwise would not have been forthcoming for this purpose. Assessments of leverage should quantify the extent to which these crowded-in funds are private or public (the

17 11 latter including MDB funds). In principle, they should account for the extent to which the crowded-in funds represent diversions from other investments. They should allow for negative leverage, in which a CIF investment partially or totally crowds out an investment from other sources. They should also distinguish direct leverage (within CIF-supported projects) vs. indirect leverage (equivalent to a catalytic impact as defined above) It is recognized that quantification of leverage is difficult, and that it may not be possible within the scope of the evaluation to operationalize all the dimensions of leverage noted above. Presentation of Evidence 4.21 The evaluation will be evidence-based in relation to the purposes, objectives, and principles of the Climate Investment Funds. All findings and conclusions should be based on evidence which is presented in the evaluation report (including the annexes) so that these can be critically assessed, and on triangulation of different sources of evidence to verify and substantiate assessments. Such evidence may take the form of tabulations of data, compilation of survey results, analysis such as correlation or regression analysis, case study reports, testimonials, objective observations of measurable data, etc. In cases where the source of information is interviews, the method of selecting those to be interviewed should be presented in the evaluation report. In the case of surveys, the questionnaire, information on the population or samples, and the response rates should be presented in the report.

18 12 Part II. The Evaluation Process 5. Oversight and Management of the Evaluation 5.1 The EvDs have established an Evaluation Oversight Committee (EOC) to oversee the evaluation (see Table 1 for EOC composition). The Independent Evaluation Group (IEG) of the World Bank Group serves as the secretariat of the EOC. The Evaluation will be carried out by consultants under the supervision of the EOC. 5.2 The EOC is responsible for the following tasks: Preparing the Approach Paper for the evaluation, which sets out the detailed scope, structure, and purpose of the Evaluation, together with evaluation questions to be addressed. Determining a budget for the evaluation. Putting in place an arrangement through which the CIF Trust Fund Committees will pay for the costs of the evaluation. Issuing a Request for Expressions of Interest (REOI) from firms that are interested in carrying out the evaluation. Short-listing firms meeting desired criteria. Issuing a Request for Proposals (RFP) and the evaluation Terms of Reference (TOR), based on the Approach Paper, for the firms that are short-listed. Selecting and contracting the firm of consultants to undertake the work. Closely supervising the consultants work for quality assurance and timely completion. Reviewing the draft reports (inception report, interim report, draft final and final reports). 5.3 The final evaluation report that is submitted by the five EvDs will become the property of the CIFs. 5.4 The integrity of the evaluation requires the strict maintenance of independence, along the four dimensions of organizational independence, behavioral independence, protection from outside interference, and avoidance of conflicts of interest, as follows Organizational independence. The evaluation team of consultants that is commissioned to conduct the evaluation will report to the EOC, and the EOC will report to the heads of the five EvDs. The EOC is independent of the CIFs and the EvDs are independent of MDB management. The evaluation team s work is subject to the EOC s oversight, quality control, and editorial review. The final responsibility for content and form of the report lies with the EvDs. The heads of the five EvDs will sign off on the Approach Paper and the budget for the evaluation before these are submitted to the Trust Fund 8. These are the four standard dimensions for independence of evaluations. See Evaluation Cooperation Group of the Multilateral Development Banks, Template for Assessing the Independence of Evaluation Organizations,

19 13 Committees for approval, and on the final evaluation report before it is delivered to the Trust Fund Committees. The approved budget for the evaluation will be transferred to IEG, serving as the secretariat for the EOC. 5.6 Behavioral independence. The evaluation team is free to complete a high quality and uncompromising report prepared in accordance with the highest professional and ethical standards. Both the EOC and the evaluation team will respect and adhere to the relevant confidentiality and disclosure rules of the CIF and the respective MDBs, as well as existing client confidentiality agreements signed between the MDBs and their clients (both public and private sector). The final evaluation report will be disclosed promptly and publicly on the CIF website along with a formal response to the evaluation from the two Trust Fund Committees. 5.7 Protection from outside interference: After the Approach Paper and budget have been approved by the CIF Trust Fund Committees, the evaluation Terms of Reference, the Requests for Proposals, the selection of the evaluation team, the evaluation design, and the final evaluation report will not be subject to modification or negotiation by CIF stakeholders. The CIF Committees, Administrative Unit, and MDB Committees will assist the work of the evaluation team, among other things, by agreeing to be interviewed, and by helping to facilitate country and project-related visits and meetings. The EOC will provide the CTF and SCF Trust Fund Committees and the CIF Administrative Unit with the opportunity to review the draft evaluation report for factual errors and to provide comments. There will also be an opportunity for public comment and consultation. The evaluation team will take these comments into account in finalizing the report. The final evaluation report will not be subject to modification or negotiation by the CTF and SCF Trust Fund Committees, the CIF Administrative Unit, or other CIF stakeholders. 5.8 Avoidance of conflicts of interest. The EOC and the evaluation team will make every effort to avoid professional, personal or financial conflicts of interest that might limit the extent of an inquiry, limit disclosure, or weaken or slant the findings. None of the members of the EOC have had prior involvement with the CIF, other than in an evaluative capacity. All candidates to undertake the evaluation will be required to disclose any prior involvement or association with the CIFs. Given that the pool of candidates from which to draw evaluators with the required skills, knowledge, and experience may be limited, some potential conflicts of interest may have to be managed transparently rather than avoided completely. Any such occurrences will be reported in the final evaluation report.

20 14 6. International Reference Group 6.1 An International Reference Group (IRG) will be constituted. Its purpose is to enhance the evaluation s quality and credibility by providing review from a diverse and respected set of experts. The main tasks of the IRG are to: Review and comment on the inception, interim, and draft final reports. Provide the EOC and the evaluation team at these stages with technical expert review and comments on evaluation methodologies in terms of rigor, relevance, appropriateness of data interpretation, and success or failure to substantiate judgments. Assist to make links with other relevant ongoing studies, expert groups and a range of stakeholders and views. Provide an independent view and statement on the final evaluation report in terms of its quality and credibility. 6.2 The IRG will be self-governing, designating its own chair who will be responsible for consolidating and conveying IRG comments and statements. Members schedules permitting, the IRG will convene twice in person at the inception report stage and draft final report stage. The EOC Secretariat will assist with logistical arrangements if requested. 6.3 While all comments and suggestions by the IRG will be carefully considered, final responsibility for the evaluation rests with the EOC and the EvD Directors. 6.4 The qualifications of individual members of the reference group include: International recognition in their fields of expertise Reputation for integrity Expertise in climate issues and/or evaluation of global programs as applied to developing countries Absence of conflicts of interest. 6.5 Criteria for the reference group as a whole include: Four or six members Equal representation of developing country and developed country members Expertise or experience covering public and private sector and civil society Expertise covering climate mitigation, climate adaptation, and forests Diversity on multiple relevant dimensions to insure inclusiveness of views. 6.6 To form the reference group, the EOC will solicit nominations from multiple groups and individuals including CIF constituencies and others such as EvD directors; IPCC lead authors on relevant topics; organizations concerned with climate issues and climate finance; and recognized climate and evaluation experts. The EOC will choose a balanced group from among the nominees. The joint CTF and SCF Trust Fund Committee will approve the complete membership (but not individual members) of the International Reference Group.

21 15 7. Qualifications and Selection of the Evaluation Team 7.1 The Evaluation Team will be selected by competitive bidding. Since IEG acts as the Secretariat of the EOC, the bidding process will be undertaken by IEG following World Bank procurement rules and using World Bank systems. This entails advertising a request for expression of interest (REOI), selecting a shortlist from respondents to the REOI, and issuing requests for technical and financial proposals in response to the Terms of Reference derived from this Approach Paper. The technical proposals are rated according to prespecified criteria. The winning proposal is selected according to a prespecified weighted sum of technical and financial scores. 7.2 Candidates technical, professional, financial and economic capacity to carry out the assignment will be appraised in the selection procedure. Candidates will be required to have the following skills: Demonstrated and relevant expertise in evaluation (particularly of global partnership programs) In-depth knowledge of international development and climate issues Technical expertise in clean energy, transport, climate adaptation, and forestry in developing countries. Ability to draw on deep local or regional expertise with respect to its country-level studies. Diversity of the proposed team that enhances the quality of its work. 7.3 Evaluators must be able to undertake objective, unbiased evaluation. Therefore, bidders must divulge any prior involvement with the CIFs so that potential conflicts of interest may be assessed and ways to mitigate these devised. If the pool of potential experts without prior experience with the programs is limited, some potential conflicts of interest may have to be managed transparently rather than avoided completely. The TOR will define what would constitute a conflict of interest and indicate how any potential conflicts would be addressed. All such occurrences will be reported in the final evaluation report. 8. Indicative Timetable and Schedule 8.1 The Evaluation Oversight Committee envisages the following indicative and ambitious timeline for the evaluation. The actual timetable will depend on a variety of factors including the chosen scope and depth of the evaluation, the extent of consultations, the approval process for this document, and the contracting process for consultants. The Terms of Reference for the evaluation will specify the conditions under which the schedule can be amended once the evaluation has started.

22 16 Table 2. Indicative Timetable June 2012 August 2012 August September 2012 September October 2012 October 30 to November 7, 2012 December 2012 May 2013 November 2013 Draft Approach Paper posted for comment by CIF stakeholders and public. Request for Expressions of Interest issued. CIF approval of Approach Paper, budget, and International Reference Group. Request for Proposals issued. Evaluation team selected. Evaluation team joins the CIF Partnership Forum in Istanbul, Turkey. Inception report delivered, reviewed, and approved. Interim report discussed at the May meeting of the CIFs. Final evaluation report presented to CTF and SCF Trust Fund meeting. Dissemination 8.2 The EOC will release the final report on behalf of the heads of the five Evaluation Departments of the MDBs. The final report will be published on the CIF website and the CIF Evaluation website, along with a formal CIF response, and will be made freely available. The final report will become the property of the CIFs, but may not be modified. 8.3 The final report will be presented initially in English, and subsequently translated into French and Spanish per CIF practice. It will consist of a main text of no more than 50 pages, and annexes of no more than 100 pages. Additional information may be made available online. 9. Obligations of Key Participants in the Evaluation 9.1 To ensure adequate support of the evaluators, and the ability to draw independent conclusions, it is useful to detail the obligations of each party. Obligations of the CIF Trust Fund Committees Approve final Approach Paper and budget. Approve membership of the International Reference Group Handle any issues referred to it by the EOC. Review final report and provide a timely written response, which is made available to the public. Obligations of the Evaluation Oversight Committee Oversee and ensure the overall independence and quality of the evaluation process, while reporting to the heads of the five Evaluation Departments of the MDBs. Prepare the Approach Paper, the Request for Expressions of Interest, the evaluation Terms of Reference, and the Requests for Proposals.

23 17 Review proposals and select evaluators. Resolve issues that arise on contracts, conflicts of interest, or access to information 9 between the consultants and the program manager and staff. Review inception report, interim report, draft final, and final reports before submission to the CIF Trust Fund Committees. Obligations of the Evaluation Department Heads Sign off on the Approach Paper and the budget for the evaluation before these are submitted to the Trust Fund Committees for approval. Sign off on the final evaluation report before this is submitted to the Trust Fund Committees. Obligations of the CIF Administrative Unit and MDB Committees Facilitate the work of the evaluation, including country consultations and site visits and providing access to staff. Assign staff members who will (a) provide key documents, (b) facilitate contacts with program constituents and members of governing body. Obligations of the Consultants Inform the EOC, the CIF Administrative Unit, and MDB Committees in a timely fashion of contacts made with program constituents. Treat documents in a confidential manner, as appropriate. Not to publish evaluation results or outputs without permission from the program. Return all program documents used in the evaluation. Report on a timely basis any possible conflicts of interest. Obligations of Other Parties (e.g., country offices of the involved MDBs) Provide information on all aid-financed projects in the country in that sector. Assist in arranging appointments with country-level stakeholders. Arrange contracts for translation of surveys. 9. While it is the responsibility of the evaluation team to obtain documents and schedule meetings, the EOC will facilitate this as needed and to the extent possible.