Review of relevant Finnish policies (incl. Development Policy Programme, Development

Transcription

1 ANNEX 3 EVALUATION MATRIX NON-EDITED Evaluation criteria and questions Supplementary questions / indicators Source of evidence Rating Relevance concerns whether the results, purpose and overall objectives of the intervention are in line with the needs and aspirations of the beneficiaries, and with the policy environment of the intervention. Is the intervention consistent with the needs and priorities of its target group and the policies of the partner country and donor agencies? Has the situation changed since the approval of the intervention (e.g. programme/project) document? To what extent is the AfT Action Plan appropriate in guiding AfT interventions (given the current global trade environment, AfT policy environment, etc)? Are the AfT Action Plan objectives, purpose and intended results consistent with the needs and priorities of the intended beneficiaries and with the policy environment? Are the AfT Action Plan focus areas (e.g. sectors and themes) and activities consistent with its intended effects? Are AfT interventions consistent with the needs and priorities of the intended beneficiaries? Finnish policies guiding AfT are feasible AfT thinking and objectives mainstreamed into planning, design and implementation of MFA-supported development cooperation % of MFA-funded projects that make reference to AfT Action Plan (in project document or other key documentation) or is clearly based on its priorities Have needs assessments been carried out at some stage of the planning? Review of relevant Finnish policies (incl. Development Policy Programme, Development Policy Guidelines on Agriculture & Food Security / Environment / Forest Sector, National Climate & Energy Strategy, International Strategy for Finland s Water Sector, Trade Policy Programme, Rural Development Strategy for International Development); MFA AfT Action Plan monitoring report MFA budget allocations to AfT (global, regional and national; MFA and OECD data) Statistics on the Finnish development cooperation (2010) Evaluation reports on Finland s cooperation with partner countries (e.g. Ethiopia, Western Kenya, Central Asia & South Caucasus, Namibia) 138 Aid for Trade

2 Are there explicit conceptual linkages between AfT on poverty reduction and inclusive growth? Is advisory support on incorporating inclusive growth and poverty reduction into AfT programmes adequate? To what extent have the cross-cutting issues been integrated in AfT activities? AfT Action Plan and relevant sectoral policies make clear conceptual linkages between trade & trade-related assistance and poverty reduction & inclusive growth Adequate advisory capacity on inclusive growth and poverty reduction available and utilised for AfT programme formulation and implementation Appraisal procedures in place and used to ensure that inclusive growth and poverty reduction are adequately taken into consideration in AfT programme formulation Extent to which the Finnish AfT interventions take into consideration the promotion of the rights and status of women and girls, promotion of gender and social equality (% of project documents that makes reference to these issues) Review of trade-related project documents Review of relevant Finnish policies (incl. Development Policy Programme, Development Policy Guidelines on Agriculture & Food Security / Environment / Forest Sector, National Climate & Energy Strategy, International Strategy for Finland s Water Sector, Trade Policy Programme, Rural Development Strategy for International Development) Review of the Quality Assurance procedures & capacities at MFA Assessment of MFA staff capacities related to poverty reduction & inclusive growth Assessment of external expertise used to provide advisory services to MFA on poverty reduction & inclusive growth Review of AfT project documents Evaluation report: The Cross-cutting Themes in the Finnish Development Cooperation ok Aid for Trade 139

3 Extent to which the Finnish AfT interventions take into consideration the rights of groups that are easily excluded and discriminated, particularly children, persons with disabilities, indigenous peoples and ethnic minorities, and promotion of equal opportunities for participation (% of project documents that makes reference to these issues) Extent to which the Finnish AfT interventions take into consideration combating HIV/AIDS, HIV/AIDS as a health problem and as a social problem (% of project documents that makes reference to these issues) Complementarity/Coherence measures if and how the goals that are central to Finland s development cooperation policy or the partner country s development policy have been taken into account in the planning and implementation. How have they affected the intervention? Coherence (internal) Are the objectives of the AfT Action Plan consistent and compliant with Finnish development policies and other policies (e.g. trade, environment)? Degree to which AfT Action Plan objectives are consistent and compliant with Finland s development policy programme and relevant sectoral and thematic policies Funding to AfT has increased in line with the overall growth in ODA Review of AfT Action Plan and relevant Finnish sectoral and thematic policies and strategies (incl. Development Policy Programme, Development Policy Guidelines on Agriculture & Food Security / Environment / Forest Sector, National Climate & Energy Strategy, International Strategy for Finland s Water Sector, Trade Policy Programme, Rural Development Strategy for International Development) a to d (a=7, a/b=6, b=5, b/c=4, c= 3, c/d=2 and d=1) 140 Aid for Trade

4 To what extent different policies and goals related to trade and private sector development are in coherence? Degree of coherence between the policies and goals related to trade and private sector development between Finland and key stakeholders To what extent has the AfT Action Plan guided the planning, design and implementation of Finland s development cooperation (sectoral/thematic)? Number of AfT-related projects in 2010 (as compared to a baseline value from year 2000) To what extent the AfT thinking has been mainstreamed into activities in other sectors in the Finnish development cooperation? Degree of consistency between AfT Action Plan objectives and AfT-related objectives of sectoral policies/strategies of MFA Links between agriculture, forestry and energy and the AfT Action Plan have been strengthened Number of AfT projects and programmes with trade and/or private sector development as significant objectives Coherence/complementarity linked to alignment below (country level) Is AfT a permanent element in the PRSPs or equivalent development plans of the developing countries? AfT is a permanent element in PRSPs or equivalent development plans in Finland s partner countries Review of trade & private sector development policies of key stakeholders (partner countries, main donor partners) AfT evaluation report by IDLO (Ethiopia, Zambia) Statistics on the Finnish Development cooperation (2010) MFA AfT Action Plan monitoring report Review of relevant Finnish policies (incl. Development Policy Programme, Development Policy Guidelines on Agriculture & Food Security / Environment / Forest Sector, National Climate & Energy Strategy, International Strategy for Finland s Water Sector, Trade Policy Programme, Rural Development Strategy for International Development) AfT project and programme documents Review of partner country PRSPs and/or national development plans; country stakeholders Aid for Trade 141

5 How are the aid interventions aligned with the policies, priorities and needs of the partner countries? AfT-related aid interventions are in line with the policies, priorities and needs of the partner countries Support is based on partner country s/ region s AfT strategy or similar Country-level needs assessments have been completed which show how changes in trade can contribute to the country s development goals A Diagnostic Trade Integration Study has been completed % of MFA-supported AfT projects/ interventions that are based on a needs assessment How do the Finnish AfT activities complement other donors activities in the sector? AfT activities have been planned in close coordination with other donors Finland s AfT projects and programmes complement other AfT work by Finland, the EU and multilateral agencies Project and programme documents include evidence of mapping of other donors activities in the sector Coherence (compliance) with international principles and priorities To which extent the Finnish AfT respects the common global principles for AfT? Degree of alignment of the Finnish AfT with common principles of AfT (e.g. OECD, WTO, EU) Review of key national policies & strategies of partner countries (incl. PRSP and/or national development plans, relevant sectoral policies & strategies) Interviews key stakeholders in partner countries Review of needs assessments conducted in partner countries Review of relevant project documents and/ or interview with relevant staff knowledgeable of the planning process MFA Development Cooperation Plans with key partner countries / regions (e.g. Africa in Finnish Development Policy) Donor Assistance Database (of partner countries) Interviews with donor partners AfT evaluation report by IDLO (Ethiopia, Zambia) Project and programme documentation Review the Finnish AfT Action Plan against the OECD/WTO/EU/UN principles 142 Aid for Trade

6 To what extent is AfT support compliant with MDG goals? Finnish AfT projects and programmes are in line with international norms and policy guidelines (Paris Declaration, UN resolutions on climate change, sustainable development, human rights, gender, the ILO conventions on decent work, WTO trade agreements) Finland increased trade-related assistance to 2 % of its ODA (or at least 15 million Euros per year) by 2010 AfT interventions have been designed to contribute towards achieving MDGs (one or more) Finland s AfT work contributes to poverty reduction (MDG1) Finland s AfT work has helped partner countries gain greater access to the markets of developed countries (MDG 8) Finland s AfT work has contributed towards a more open, rule-based, predictable, non-discriminatory trading and financial system (MDG 8) In cooperation with the private sector, Finnish AfT has made available the benefits of new technologies, especially information and communications (MDG 8) Finland s AfT work supports environmental sustainability (MDG 7) Review of OECD/WTO AfT documents (incl. questionnaires) OECD CRS database, Statistic on the Finnish development cooperation Progress reports on Finnish ODA in partner countries (e.g. Vietnam report includes Some examples of Finnish contribution to MDG progress in Vietnam MDG progress reports of Finland s main partner countries MDG Monitor (mdgmonitor.org) Review of recent evaluations of Finland s development cooperation Aid for Trade 143

7 Coordination (linked to harmonisation below) concerns the interaction with relevant groups and other donors in a partner country, ideally so that synergies occur and conflicts or overlaps do not. How is the coordination among different actors and stakeholders functioning at the field level and between the capital and the field? Effective coordination mechanism in place among MFA, other actors and key stakeholders at the field level Effective coordination mechanism in place between MFA staff in the capital and in the field Finland s AfT projects and programmes do no overlap with the work of other organisations/ agencies country stakeholders, donor partners Recent evaluations of Finnish development cooperation. To what extent there is interaction between Finland and other donors to create more relevant, coherent, effective support to AfT? Regular discussions have taken place between MFA and other donors in order to ensure more coherent and effective/relevant support to private sector and trade development in developing countries Finland actively participates joint donor platforms, roundtables, etc. on AfT related issues country stakeholders, donor partners Meeting minutes, proceedings from donor consultations/roundtables, etc. To what extent have synergies between Finland s multilateral and bilateral support and different instruments have been strengthened? Strategic partnerships established with multi- and bilateral actors to maximize synergies based on a systematic review of the comparative advantages, potential synergies and effectiveness of various modalities Areas of collaboration with multilateral agencies which enhance synergies between Finland s multilateral and bilateral cooperation have been identified Review of MFA s partnership strategies at HQ, regional and country level Review of MFA s aid modalities 144 Aid for Trade

8 To what extent Finland s AfT policy and actions have influenced the interaction between stakeholders and interventions in different levels and sectors? The degree of influence that the Finnish AfT policy and actions have had on the interaction between stakeholders The degree of influence that the Finnish AfT policy and actions have had on the interventions at different levels (global, regional, national, local) and sectors (environment & climate change, rural development, private sector development, etc.) country stakeholders, donor partners Meeting minutes, proceedings from donor consultations/roundtables, etc. Efficiency concerns how well the various activities have transformed the available resources into the intended outputs (sometimes referred to as a result), in terms of quantity, quality and time. Can the costs of the intervention be justified by the results? To what extent the aid interventions have transformed the available resources into the intended outputs and results, in terms of quantity, quality and time? Can the costs of the AfT interventions be justified by the achievements (i.e. were activities cost-efficient)? Have efficiency gains been generated e.g. through the use of new or different modalities? Degree to which AfT interventions/ activities have transformed available resources into expected results (outputs, outcomes) (quality, quantity and time) MFA AfT Action Plan monitoring report Project/programme evaluations/assessments Relevant thematic EVA-11 evaluations (e.g. water, energy, sustainability dimensions in addressing poverty reduction, natural disasters & climate change, agriculture & rural development, cross-sectoral themes) and partner country evaluations AfT evaluation report by IDLO (Ethiopia, Zambia) Are the organizational arrangements and management systems for guiding and controlling the AfT adequate? Clear division of labour between MFA HQ (AfT Team/Trade & Development Team /desk officer) and the Embassy and/or local partners Interviews MFA, HQ & embassies MFA Annual Report on Finnish Development Cooperation MFA operational guidelines & organisational structure Aid for Trade 145

9 There is a smooth flow of information between the MFA and the embassies, ensuring coherence between AfT-related work at the MFA and the country level MFA AfT Team has adequate representation/expertise from key sectors and effectively advises staff in charge of the implementation of AfT-related interventions AfT training, toolkits and/or guidelines developed and improved Sufficient human and financial resources are allocated for effective implementation of AfT interventions A set of AfT indicators in use for systematic monitoring of the impact of AfT cooperation and regular reports produced accordingly The AfT Action Plan has been monitored by the cross-departmental trade and development team The trade impact of Finnish aid projects and programmes is regularly monitored Accountability for the AfT (incl. clear roles & responsibilities) is well understood both by those implementing, supporting implementation and overseeing implementation of AfT interventions The MFA and embassies have deepened their analysis of key private sector issues Progress reports of AfT projects MFA reports on country cooperation 146 Aid for Trade

10 Effectiveness describes how well the results achieved have furthered the attainment of the purpose of the intervention. Has the intervention achieved its objectives or will it do so in the future? To what extent the AfT Action Plan has been transformed into concrete actions? Degree to which the AfT Action Plan objectives have been transformed into concrete interventions MFA AfT monitoring report Interviews MFA, partner country stakeholders, donor partners Has the AfT Action Plan and interventions achieved their purpose or will they do so in the future? What were the major factors influencing the achievement or non-achievement of the objectives? Degree to which the AfT Action Plan, aid interventions and aid modalities have achieved the expected results Degree to which the expected results are to be achieved (during or beyond the intervention/project) MFA AfT monitoring report Progress reports of AfT projects Interviews MFA, partner country stakeholders, donor partners Impact concerns whether there has been a change towards the achievement of the overall objective(s) as a consequence of the attainment of the purpose of the intervention. Both intended and unintended impacts are reviewed. What are the overall effects of the intervention, intended and unintended, long term and short term, positive and negative? Are there any discernible features, which could indicate towards possible longer terms impact, positive or negative, intentional or unintentional? Evidence on any features that indicate towards longer-term impact generated by Finnish AfT support Recent evaluations of Finland s AfT Project and programme documents, midterm reviews, project completion reports, etc. To what extent Finland s AfT (alone or jointly/in coordination with other development actors) has prioritised inclusive growth and supports poverty reduction? Inclusive growth and poverty reduction have been considered as overarching goals in the project design of key AfT interventions AfT project and programme documents Are there any discernible effects of AfT on poverty reduction and inclusive growth? Evidence on any discernible effects of trade and trade-related assistance on poverty reduction and inclusive growth Progress reports, mid-term reviews, evaluations, etc. of AfT projects Aid for Trade 147

11 Has Finland had any influence on policies of EC, OECD, WTO or UN agencies? Evidence on the influence that Finland has had on AfT policies of the EC, OECD, WTO and UN agencies Finland has participated actively in AfTrelated discussions and preparations at all levels of EU cooperation Interviews of relevant staff at the EC, OECD, WTO and UN agencies (ILO, UNCTAD, UNDP, UNIDO) Minutes and speeches from high-level policy meetings (EC, OECD, WTO, UN agencies) Sustainability can be described as the degree to which the benefits produced by the intervention continue after the external support has come to an end. It is a central theme in all evaluation work and relates to all the elements of the logical framework of an intervention. Will the benefits produced by the intervention be maintained after the termination of external support? Will the possible benefits produced by the AfT be maintained after the termination of external support? Provisions have been made by the partner countries to commit (financial & human) resources to sustain the results beyond the assistance provided by MFA (and other partners) country stakeholders, donor partners Recent evaluations of Finland s development cooperation How is financial sustainability of AfT support services? Sustainability considerations of the results beyond the AfT project duration (or after the AfT intervention) have been included in the project document, or have been used to inform the planning of the AfT intervention Recent evaluations of Finland s development cooperation Finnish added value concerns specific knowledge, skills, approaches, priorities and processes that Finland has provided as a contribution to success What is the added value provided by the Finnish support (Finland-specific knowledge, skills, approaches, priorities, etc.)? Evidence on the Finnish added value generated through AfT interventions (broken down by sector/theme) Finland s participation in AfT-related projects and programmes follows Finnish comparative advantage (e.g. focusing on ICTs, forestry or energy) Recent evaluations of Finland s development cooperation Interviews with key stakeholders, donor partners, international organizations, MFA staff responsible for implementing AfT projects 148 Aid for Trade

12 To what extent the cluster approach adopted for integration of the Finnish stakeholders has been beneficial for enhancing the Finnish added value? Has Finland a special way of promoting, planning and implementing of the AfT? Would any achievement of AfT be unaccomplished without Finland s AfT support? Evidence on how and to what extent the cluster approach (for integration of the Finnish stakeholders) has been beneficial for enhancing Finnish added value Finnpartnership has effectively promoted business partnerships between enterprises in Finland and those in developing countries Evidence on any special Finnish features in promoting, planning and implementing AfT Evidence on the critical contributions generated through Finnish AfT support that made it possible to accomplish the expected results Interviews MFA HQ, embassies, Finnfund, Finnpartnerships Finnfund and Finnpartnership annual reports Recent evaluations of Finland s development cooperation Review of good practices (e.g. Dar es Salaam Institute for Sustainable Development; Energy & Environment Partnership; Finnpartnership) country stakeholders country stakeholders Aid for Trade 149