Country and Regional Strategy Evaluation

Transcription

1 Federal Department of Foreign Affairs FDFA Swiss Agency for Development and Cooperation SDC Staff of the Directorate Evaluation and Corporate Controlling Division Country and Regional Strategy Evaluation Concept Bern, 12 January 2016

2 Country Strategy Evaluation Concept Contents 1. Introduction Definition and Aim Evaluation Areas, Evaluation Questions and Sources of Information Main Evaluation Steps, Responsibilities and Milestones Actors Involved Deliverables Dissemination Challenges... 8 Abbreviations CS EA FCD FDFA HQ HSD ICS MDG MERV PRSP SCO SDC SECO ToR Cooperation Strategy Evaluation Area Federal Council Dispatch Federal Department of Foreign Affairs Headquarters Human Security Division Internal Control System Millennium Development Goal Monitoring entwicklungsrelevanter Veränderungen (Monitoring of development related changes) Poverty Reduction Strategy Paper Swiss Cooperation Office Swiss Agency for Development and Cooperation State Secretariat for Economic Affairs Terms of Reference

3 1. Introduction The concept of the Country and Regional Strategy Evaluation is based largely on the experiences of four pilot evaluations 1 conducted between 2012 and These pilot evaluations have marked the basis for further developing, clarifying and simplifying the concept. The concept provides the methodological guidance for conducting future country and regional strategy evaluations. After an introduction, which aims at clarifying the definition and the objectives, the concept includes an introduction of the four evaluation areas and the standard evaluation questions. Furthermore, the concept defines the responsibilities of the evaluation, the process using milestones and the roles of different actors. Finally, it provides information concerning the expected deliverables, the dissemination of the evaluation results and challenges. Until 2007, SDC s Cooperation Strategies (CS) were assessed by independent evaluation teams. In September 2010, the Board of Directors mandated the Evaluation and Corporate Controlling Division to introduce a new approach with a stronger focus on learning and the application of a relatively high degree of standardization. In fulfilling the strengthened learning objective, assessments of CS s are realised by means of hybrid evaluations, which are conducted by a mixed team consisting of one external consultant and two internal resource persons of SDC ( peers ). Country and regional strategy evaluations constitute central instruments of the new SDC Evaluation Policy (2013). They are mandated and coordinated by SDC s Evaluation and Corporate Controlling Division. At the same time, country and regional strategy evaluations refer to OECD-DAC evaluation standards (OECD DAC). 2. Definition and Aim Country and regional strategy evaluations analyse the Cooperation Strategies which define the Swiss engagement in a particular country or region. Country and regional strategy evaluations apply to the following SDC Departments: South Cooperation, Cooperation with Eastern Europe and Humanitarian Aid. In addition, they appraise collaborations with Global Programs and their thematic links with the CS project/program portfolio. A joint engagement in the CS evaluations is to be envisaged, in case the CS embrace operations of other Swiss agencies (e.g. SDC, State Secretariat for Economic Affairs (SECO) or Directorate of Political Affairs / Human Security Division). The goal of country and regional strategy evaluations is to assess the relevance and coherence of the Swiss development cooperation in regard to national development priorities and the Message on Switzerland s International Cooperation. They assess the results achievement of the CS portfolio at the level of domains of intervention. In doing so, these evaluations help SDC s management in their strategic and operational steering and in improving aid effectiveness. 1 Evaluation of the Cooperation Strategies of Bangladesh (2012), Serbia (2013), Tanzania (2013/2014) and of the Regional Program for Iraq, Jordan, Lebanon and Syria (2013/2014). 1

4 The objectives of country and regional strategy evaluations include the following: To assess through a mutual learning process whether SDC and its partners reach the strategic objectives in the country or region, as defined in the CS, and to appraise the efficiency of the strategic and operational steering mechanisms of the Cooperation Office; To timely build the foundation for the definition of key elements for the new CS; To assess the significance of the Swiss contribution to national (and regional) development results, and to identify key factors, which enhance or hinder aid performance and results achievements; To identify good practices and innovative approaches as well as share experiences in managing CS. The purposes of these evaluations are: Steering and strategic management: Swiss Cooperation Office (SCO) and operational divisions benefit from their own involvement in the evaluation, particularly for the definition of a new cooperation strategy. The added value of the evaluation is twofold: First, it responds to thematic issues. Second, it contributes to management issues of the CS, by means of the assessed results framework (results chains) of the cooperation strategy and its monitoring system. Learning: Learning experiences at an institutional level are being promoted through an evaluation with peers. Peers become key persons in disseminating lessons learned and experiences outside their own sphere of responsibilities. Additionally, they may transfer such experience to their own units and (thematic) networks. Accountability: Country and regional strategy evaluations allow an account of Swiss achievements in different countries or regions, which is relevant for internal and external accountability. Country and regional strategy evaluations are part of the cycle management of Cooperation Strategies and are therefore linked to their planning, monitoring and reporting processes. They are timely well positioned with regard to the elaboration process of the new CS s 2 and/or in order to take informed decisions regarding the Swiss engagement in a country or region. In doing so, they may replace mid-term or end-term reviews of CS. 3. Evaluation Areas, Evaluation Questions and Sources of Information The country and regional strategy evaluations shall concern the following four Evaluation Areas (EA): EA 1: Context analysis (referring to both the partner country and the Swiss context) EA 2: Relevance and appropriateness of the projects/program portfolio with regard to the domains of intervention of the CS EA 3: Implementation of the CS and its portfolio EA 4: Results of the CS in relation to the results at country level. 2 Refer to chapters 6.5 and 6.6 of the SDC Guidelines for Elaborating Cooperation Strategies. 2

5 Besides the project/program portfolio, the evaluation includes also an assessment of the transversal themes (e.g. gender) defined in the CS. Following the availability and quality of CS monitoring data, the analysis of results achievements (EA 4) may be focused on a reduced number of thematic domains of intervention. The evaluation approach requires a preliminary document analysis. This analysis is based on existing key documents. The preparation phase is complemented by interviews at the headquarters (HQ) and with partners in Switzerland. This is followed by a project and program portfolio analysis, which details the characteristics of the CS. The findings are then summarized in an Inception Report. This report includes the time schedule of the field trip and provides additional methodological information (e.g. agenda of workshops). The Inception Report is an important reference document for the evaluation team and the respective Swiss Cooperation Office (SCO). It greatly facilitates the evaluation process during the field mission and enables a timely elaboration of the evaluation report. For each of the four Evaluation Areas, standard key questions are suggested. In addition, the SDC Operational Division, the SCOs and other concerned Swiss Agencies (SECO, Directorate of Political Affairs / Human Security Division) each submit up to six strategic evaluation questions. The definite evaluation questions are agreed upon by the Evaluation and Corporate Controlling Division and the evaluation team. This process shall ensure that the relevant criteria for the definition of the new CS are addressed by the evaluation. Evaluation Area 1: Context analysis (referring to the partner country context, the region and to the Swiss context) 1.1 Positioning and adaptation of CS with respect to country and regional context as well as Swiss policies 111. How well does the CS (strategic orientation, overall goal, domains of intervention and transversal themes, global challenges) reflect the development priorities, set by the partner country/countries and the policies of the Federal Council Dispatch (FCD)? 112. Which changes in the context (national and regional) were the most important and what effects may they have caused on the CS? Which adaptations have been taken? 1.2 Quality of context analysis 121. To what extent is the context analysis realistic and relevant? To what extent is the broad political context taken into account in the CS and ARs? Does the analysis include current issues (e.g. social and economic inequality, global challenges, power relations, regional disparities) and relevant stakeholders (e.g. private sector, state apparatus and political parties, institutions and powers)? Evaluation Area 2: Relevance and appropriateness of the projects/program portfolio with regard to the domains of intervention of the CS 2.1 Relevance of the projects/program portfolios 211. To what extent are the projects/program portfolios relevant, coherent and appropriate for achieving the results of the CS regarding its domains of intervention? 212. To what extent are the approaches being applied appropriately in the domains? Which innovative approaches produce added value? 3

6 Evaluation Area 3: Implementation of the CS and its portfolio 3.1 Management performance 311. How efficient is the portfolio management of the SCO (regarding transversal themes, collaboration with the global programs, financial and human resources and aid diplomacy)? What are its contributions to an optimal achievement of results? 3.2 Quality of the CS monitoring system 321. To what extent is the process management of the CS monitoring system relevant and efficient, in order to provide evidence-based data/information for accounting for results (reporting) and CS steering? 3.3 Coordination and aid effectiveness in the country set up 331. Which role do SCOs play within the network of different Swiss agencies in charge of development cooperation (SDC South Cooperation, Humanitarian Aid, Global Cooperation and Cooperation with Eastern Europe, SECO, Directorate of Political Affairs / Human Security Division), national governments and the donor community? Which added values result due to Switzerland s support in the respective countries? Evaluation Area 4: Results of the CS in relation to the results at country level 4.1 Domain results, effectiveness and contribution to country results 411. Which contributions of the Swiss Cooperation portfolio become visible at the output and outcome level, particularly regarding the achievement of the development results in the partner country? Which internal and external factors enhance or hinder aid performance and results achievements? 4.2 Sustainability and scaling up 421. Which innovations generated by field experience have been scaled up through policy dialogue, alliances, networking and dissemination? 422. Which actions have been taken at country level to enhance the sustainability of the Swiss investments? The evaluation approach is based on: a preliminary document analysis, interviews at the headquarters and a 10-day field mission. Whenever feasible, a project visit is included in the field mission program. The document Toolkit. Country and Regional Country Evaluation. June 2016 is a working aid for the external consultant, as it compiles standard documents (e.g. road map, terms of reference, structure of evaluation reports) and details on the evaluation process (e.g. methodological tools). Annual reports act as entry points for the document analysis during the preparation phase. The document analysis is complemented by information from recent quality evaluations and reviews, impact assessments as well as from end of phase reports of projects and programs. The following, non-exhaustive list provides an overview of those reference documents, which form the basis for country and regional strategy evaluations. The documents are listed according to the priority for the evaluation: SDC country or regional level: SCO annual reports*, mid-term reviews*, CS monitoring system and data*, MERV, Office Management Reports, Internal Control System (ICS) Report, quality assurance report commenting annual reports, audits. 4

7 SDC project level: executive summaries of evaluation and mid-term reviews*, end of phase reports*, impact studies, project/program documents (if needed). SDC / Federal Department of Foreign Affairs (FDFA): Federal Council Dispatch*. Recipient Country: national development strategy* including respective monitoring reports*, sector strategies (where relevant and available for CS domains). Other donors and sources: joint donor evaluations, statistics of World Bank and UN (MDG monitoring) as well as other secondary sources. * most important information sources 4. Main Evaluation Steps, Responsibilities and Milestones The CS are selected according to the following criteria: The country and regional strategy evaluation should be pursued in the penultimate year of CS implementation, in order to add value to future steering decisions and to enhance learning with regard to results achieved. Willingness to integrate country and regional strategy evaluation results into the decision and elaboration process of the new CS. Institutional balance among South Cooperation, Cooperation with Eastern Europe and Humanitarian Aid. Geographic and thematic balance for the period of two framework credits. A maximum of two to three country or regional strategy evaluations per year. Following these criteria, the Evaluation and Corporate Controlling Division, in consultation with the concerned Operational Department(s), selects the CS and seeks the approval of the SDC Board of Directors. The following table provides an overview of the evaluation process, which is harmonized according to the planning process of the new CS. Evaluation Steps Tasks Responsibility Time Frame Selection of CS Selection of the CS to be evaluated in the following year. SDC Evaluation and Corporate Controlling Division September before the year of evaluation Recruitment of Peers and Consultants Recruitment of 1 consultant and 2 peers (plus 1 peer from SECO/HSD, if relevant) working in relevant domains in other SCOs or headquarters. ToR are established for the consultant. SDC Evaluation and Corporate Controlling Division 10 months before field mission Terms of Reference Establishment of ToR with 11 standard and 6 prioritized evaluation questions suggested by concerned Operational Division/SCO. Elaboration of ToRs for the peers SDC Evaluation and Corporate Controlling Division, Operational Division/SCO 4 months before field mission 5

8 Data and Information Gathering Provision of requested information to the evaluator (consultant) Document analysis and meta-analysis by the evaluator Operational Division / SCO Consultant 3 months before field mission Interviews with responsible staff at SDC HQ and, if relevant, SECO/HSD and partners in Switzerland Consultant and peer(s) based in Berne Inception Report Elaboration of an Inception Report (20 pages), including an analysis summary of relevant documents for the four evaluation areas. The Inception Report represents the key reference document for the peers. Consultant 2 months before field mission Field Mission day field visit (including project visit, if feasible and relevant) Debriefing meeting, presentation of key findings and recommendations (at SCO and if possible HQ) Consultant, Peers Reporting Elaboration of Evaluation Report (25-30 pages) by consultant with contribution of peers Consultant, Peers 1 month after field mission and 1 month before planning workshop for the new CS Management Response Elaboration of management response Operational Division (with SCO), approved by concerned head(s) of department(s) 1-3 months after the approval of the report Dissemination Dissemination of the Evaluation Report and feeding lessons learnt into own units and thematic networks SDC Evaluation and Corporate Controlling Division, Peers Once the report is available 5. Actors Involved The ultimate responsibility for the management and supervision of the evaluation rests with the Evaluation and Corporate Controlling Division. The division selects the external evaluator (consultant) and the peers for each country and regional strategy evaluation. It takes charge of the Terms of Reference and suggests an appropriate time schedule for the evaluation process in cooperation with the evaluation team as well as with the involved Operational Division and SCO. The Evaluation and Corporate Controlling Division is also the focal point for the provision of information and methodological expertise. 6

9 The evaluation team consists of one external consultant and a maximum of two internal peers of the SDC (and of the SECO/HSD, if relevant). If needed, a national consultant may complete the evaluation team. The external consultant is the team leader and responsible for the overall evaluation process. Thus, he/she coordinates the process and provides appropriate methodological elements. He/she facilitates exchange between the evaluation team members in order to reinforce a mutual learning. He/she is responsible for the synthesis of information at the stage of writing the Evaluation Report. Peers are selected in accordance to clear criteria and a list of interested candidates. Peers may be coordinators/country directors, deputy coordinators or program officers of focal points within SDC. They contribute to thematic and process knowledge and facilitate learning experience. As pre-requisite, they do not and have not worked previously for the concerned country or region for at least 10 years. In addition, they do not work for the same Operational Division. However, they should be working on issues, similar to the ones to be evaluated. Peers should be very familiar with the institution as a whole and with the Swiss political context which determines the actions of SDC. Depending on the CS, a national consultant may be added to the evaluation team, in particular for the analysis of specific issues which are not covered by the other team members. The concerned Operational Division(s) and the SCO establish a Management Response, which is to be approved and signed by the head(s) of the concerned Department(s). The Management Response forms an integral part of the report. Subsequently, the report will be published by the Evaluation and Corporate Controlling Division. The Operational Division and the SCO are involved throughout the process of this evaluation, in particular they shall: supply all necessary documentation for the evaluation; suggest additional strategic evaluation questions; provide support to the evaluation team in organising the field visits and identifying key actors for interviews; participate in interviews and workshops; critically comment and countercheck the report; take a stand on the report and the recommendations (Management Response); use the evaluation results for defining the new CS. Other Swiss agencies in charge of development cooperation (SECO, Directorate of Political Affairs / Human Security Division) shall be consulted prior to the evaluation, as they might also be involved in the evaluation process. 6. Deliverables The final evaluation product is a report which analyses and synthesises the elements resulting from the four evaluation areas and additionally the strategic questions suggested by the concerned Operational Division and SCO. The consultant will be responsible for the final Evaluation Report. The report shall be based on inputs written by 7

10 each member of the evaluation team. The Evaluation Report should include an executive summary and should not exceed 25 pages. 7. Dissemination The Country and Regional Strategy Evaluations including the Management Responses will be made available to the public through SDC s website. Furthermore, the Evaluation and Corporate Controlling Division will disseminate reports pro-actively to those directly involved in CS, to SDC Networks and Units working on the concerned thematic domains of intervention as well as to strategic partners which were assessed and involved in the evaluation. Peer experiences will be presented through SDC intranet in a systematic way. 8. Challenges The quality of Country and Regional Strategy Evaluations is closely linked to: The availability and quality of documents for the meta-analysis. The availability of a quality CS monitoring system for analysing results achievements. The availability of qualified peers for the evaluation team. The availability of a qualified consultant to lead the evaluation team. For further information on the concept: Swiss Agency for Development and Cooperation - SDC Evaluation and Corporate Controlling Division Freiburgstrasse 130, 3003 Bern sektion.evaluation-controlling@eda.admin.ch 8