UNITED NATIONS INDUSTRIAL DEVELOPMENT ORGANIZATION

Transcription

1 May 2009 UNITED NATIONS INDUSTRIAL DEVELOPMENT ORGANIZATION TERMS OF REFERENCE Independent Mid-term Evaluation of UNIDO project USGHA06005 Trade Capacity Building for Ghana I. Background This project is part of a joint effort of UNIDO and the World Trade Organization to assist a number of developing countries in overcoming global trade obstacles in three areas: i) improving supply-side capacity and competitiveness; ii) proving conformity with market requirements; and iii) enhancing connectivity with world markets. Ghana was selected because of its exports suffering from constraints in all the three areas, in spite of the country having considerable comparative advantages. The Swiss State Secretariat for Economic Affairs (SECO) decided to fund this Trade Capacity Building for Ghana project and an agreement with UNIDO was reached in December Project objectives: The overall development objective of the project is to enhance the export performance of Ghana. This should be achieved by creating conditions for strengthening supply capacity in selected agricultural and industrial sectors, establishing a credible conformity assessment infrastructure and fostering integration into the multilateral trading system, while at the same time strengthening local consumer protection. The project has five immediate objectives: 1) Capabilities of Ghana Standard Board (GSB) in standard-setting: Strengthening the capabilities of Ghana Standard Board (GSB) to enable them to develop and publish adequate standards for selected export products and to provide training to Ghanaian enterprises and farmer-based organizations on priority public and private standards, while at the same time use these standard-related activities to enhance and further institutionalise consumer protections; 2) Traceability system pilot application for horticulture: Establishment of a national traceability system for horticultural products with the Ghana Export Promotion Council; 3) GSB systems certification capabilities: Reviewing and developing the management certification system capability of GSB, including ISO 22000; 1

2 4) Laboratories upgrading: Upgrading and accreditation against ISO of calibration and testing laboratories, the latter in the area of chemical, microbiological, pesticide residues and textile testing; 5) Upgrading of Plant Protection and Regulatory Services Directorate (PPRSD) as EU Competent Authority for horticulture: Upgrading of PPRSD within the Ministry of Food and Agriculture to become the nominated EU Competent Authority in the field of horticulture. Project budget: The project is funded by SECO, with the total budget of $2.42 million, excluding UNIDO support costs. So far nearly 39% of this amount has been committed and/or spent. Project No. Allotment US$ Expenditure US$ % Implemented USGHA ,417, , % Source: UNIDO INFOBASE as of 12 May 2009 Table 1 presents the allocation of the budget to the five objectives. More than 80% of the project budget is dedicated to objectives 1 and 4 on strengthening GSB s standard-setting capabilities and laboratory upgrading. Table 1: Budget allocation among project objectives # Immediate objective Budget (USD) 1 Capabilities of Ghana Standard Board (GSB) in standardsetting % of total budget 803, Traceability system pilot application for horticulture 148, GSB systems certification capabilities 180, Laboratories upgrading 1,175, Upgrading of PPRSD as EU Competent Authority for horticulture 111, TOTAL 2,417, Source: Project document Project duration. The implementation of the project started in March 2007 for an expected duration of three years. However, due to a number of reasons to be elucidated by the evaluation, the implementation experienced delays. It has been requested by the Government of Ghana at the last steering committee to extend the duration. Considering the delay experienced and the pending activities SECO and UNIDO have in principle agreed to extend the project duration until December

3 II. Purpose of the evaluation This independent mid-term evaluation will be conducted as a response to an explicit request from the donor. It will feed into the ongoing thematic evaluation of UNIDO projects in the area of Standardization, Metrology, Testing and Quality (SMTQ) that is being conducted throughout 2009 by the UNIDO Evaluation Group with financial support from SECO. This thematic evaluation aims at deriving Key Success Factors (KSF) of SMTQ interventions from the analysis of more than 20 UNIDO projects in the SMTQ area. The mid-term evaluation of the Ghana Trade Capacity Building for Ghana project is a timely opportunity to test and validate some of the KSF emerging from the thematic evaluation. A preliminary draft report of the thematic evaluation found that a thorough intervention logic and logical framework are particularly important KSFs that predetermine the success of SMTQ projects already during the design stage. Together with the donor it has therefore been decided that the mid-term evaluation should be taken as an opportunity to elaborate on this KSF and to come up with a preliminary standard logframe for SMTQ projects that would be based on inputs from the thematic evaluation and the mid-term evaluation. The purpose of the mid-term evaluation is three-fold: Determine the extent to which the objectives, outcomes and outputs defined in the project document have been met and assess the likelihood of achieving them upon project completion; Elucidate the reasons of implementation delays and identify strengths and weaknesses of the project implementation and management so far, including project monitoring and self-evaluation (M&E) mechanisms; identify potential options for improvement, which could include modifications of the project planning (logical framework) and of the project steering, implementation and management mechanisms (steering committee; responsibilities of UNIDO and project staff, scheduling, etc) and could lead to a revised project document; Feed evaluation results into the ongoing thematic evaluation of UNIDO SMTQ projects and prepare a tentative standard logframe for SMTQ projects. III. Evaluation criteria and management Evaluation criteria The evaluation will apply the standard DAC evaluation criteria and examine, more specifically, the following aspects: Relevance How relevant is the project to the: o target groups needs o development priorities of the Government of Ghana and the institutional context o UNIDO organizational strategy o SECO s policies and priorities Are appropriate target groups being assisted by the project? 3

4 Is the project design adequate to address the problem(s) at hand? What internal and external factors have influenced the ability of beneficiary groups and UNIDO to meet projected targets? Does the project remain relevant considering possible changes in context? Is there a need to reformulate project design and the logframe given changes in the country and operational context? Efficiency The extent to which: UNIDO and Government/counterpart inputs have been provided as planned and were adequate to meet requirements The quality of UNIDO inputs and services was as planned The implementation is timely and reasons for delays The least costly resources and processes were used in order to achieve the objectives There was coordination with other projects and possible synergy effects Has the project reached the expected number of beneficiaries (institutions, targeted trainees etc.) within the expected time frame? Are the project s activities in line with the schedule of activities as defined by the project team and annual work plans? Are the disbursements and project expenditures in line with expected budgetary plans? Effectiveness To what extent have the expected outputs and outcomes been achieved or are likely to be achieved? How is their quality? How do the stakeholders perceive them? Were the targeted beneficiary groups actually reached? What outputs and outcomes has the project achieved so far (both qualitative and quantitative results)? Has the project generated any results that could lead to changes of the assisted institutions operation? Does the current performance indicate probability in achieving the project objectives? Have there been any unplanned effects? Sustainability To what extend are the benefits from the project likely to continue after the project completion in terms of financial, institutional, technical and environmental sustainability and local ownership? Does the project have an exit strategy? Is it accurate and realistic? Impact What is the intervention theory of the project that maps out how inputs and activities are supposed to outputs, outcomes and impact at company level and beyond? Identification of the assumptions on external factors and impact drivers that are considered necessary to achieve impact; Assessment of the plausibility of the intervention theory; 4

5 Project coordination and management The extent to which: The national management and overall coordination mechanisms have been efficient and effective. Do partners have clear roles and responsibilities? Do partners fulfill their roles and responsibilities (e.g. providing strategic support, monitoring and reviewing performance, allocating funds, providing technical support, following up agreed/corrective actions )? The UNIDO HQ based management, coordination, quality control and technical inputs have been efficient and effective Monitoring and self-evaluation were carried out effectively, based on indicators for outputs and objectives Mechanisms are in place to assure synergy with other UNIDO projects and activities of other donors Relationship with the thematic evaluation. The mid-term evaluation will apply the Key Success Factors (KSF) developed under the thematic evaluation, validate the practicality of the KSF, determine their scope of applicability and feed the evaluation results into the thematic evaluation. Examples of KSFs emerging from the thematic evaluation include: the wider context and enabling environment; cooperation modalities; institutional context of the beneficiary organization(s); practical implementation issues; project design and management issues; technical aspects of National Quality Infrastructure; sustainability issues and some further factors. Evaluation management The evaluation will be managed by the UNIDO Evaluation Group in consultation with SECO. It will encompass the following steps: Desk review The evaluation team will review and analyze documents related to the project (e.g. design, progress and terminal reports; technical reports from consultants/subcontractors; methodological documents, tools and training guideline; etc). Relevant documents of the Government of Ghana and other development organizations will also be consulted. Field visits and interviews Interviews with the UNIDO project manager/s in Vienna will be conducted prior to the field visit to Ghana Interviews with a sample of consultants and/or technical assistance providers who were hired by UNIDO to support the project in the country Field visits in Ghana to carry out in-depth interviews and analysis of the project interventions Interviews with representatives of all stakeholder groups (Government counterparts, donor, supported institutions and a sample of their clients, private sector representatives; etc) in Ghana; and For each type of these interviews, the consultant/team should first develop their ideas for the content and format of the survey/interview forms that will be applied to capture the information required, as well as the method to be used in administering them and 5

6 tabulating the results. Field interviews can take place either in the form of focusgroup discussions or one-to-one consultations. Reporting The evaluation team will present its preliminary findings to the local stakeholders at the end of the field mission and take into account their feed-back in drafting the evaluation report. The length of the report should be around pages with a 3-page executive summary in English. The draft report will be shared with UNIDO and project staff, the Government of Ghana and SECO for factual validation and comments. On the basis of this feedback, the evaluation team will prepare the final report of the mid-term evaluation and integrate the findings into the thematic evaluation report. Quality Assurance of the Evaluation Report. The report will be subject to quality assurance by SECO and the UNIDO Evaluation Group. The quality of the evaluation report will be assessed and rated against the criteria set forth in the Checklist on evaluation report quality (Annex 1). IV. Evaluation team and timing Evaluation Expert. The evaluation will be carried out by an International Evaluation Consultant with extensive experience in and knowledge of SMTQ and evaluation issues to be selected jointly by UNIDO and SECO who will be contracted by SECO. The tasks of the evaluator are specified in the job description attached to these terms of reference. The evaluator must not have been involved in the design and/or implementation, supervision and coordination of and/or have benefited from the project under evaluation. This principle is underlined in the UNIDO Evaluation Policy: For independent evaluations, the members of an evaluation team must not have been directly responsible for the policy-setting, design or overall management of the subject of evaluation (nor expect to be so in the near future). The evaluator will be requested to sign a declaration that none of the above situations exists and that he/she will not work nor seek concurrent employment with the UNIDO manager/s responsible for the project that is subject to this evaluation for a period of six months after the expiration of his/her contract with this evaluation. Timing. The evaluation is tentatively scheduled for July The final report will be prepared within six weeks after completion of the field mission and will be submitted to SECO, UNIDO and the Government of Ghana. 6

7 Annex 1. Checklist on evaluation report quality Rating system for quality of evaluation reports Report quality criteria UNIDO Evaluation Group Assessment notes Rating (a) Did the report present an assessment of relevant outcomes and achievement of project me objectives? (b) Were the report consistent and the evidence complete and convincing? (c) Did the report present a sound assessment of sustainability of outcomes or did it explain why this is not (yet) possible? (d) Did the evidence presented support the lessons and recommendations? (e) Did the report include the actual project me costs (total and per component or project)? (f) Quality of the lessons: Were lessons readily applicable in other contexts? Did they suggest prescriptive action? (g) Quality of the recommendations: Did recommendations specify the actions necessary to correct existing conditions or improve operations ( who? what? where? when?). Can they be implemented? (h) Was the report well written? (Clear language and correct grammar) (i) Were all evaluation aspects specified in the TOR adequately addressed? (j) Was the report delivered in a timely manner? A number rating 1-6 is used for each criterion: Highly Satisfactory = 6, Satisfactory = 5, Moderately Satisfactory = 4, Moderately Unsatisfactory = 3, Unsatisfactory = 2, Highly Unsatisfactory = 1, and unable to assess = 0. 7