TERMS OF REFERENCE Final external evaluation of an extra budgetary project Improvement of teacher qualification and setting up of system for the management of learning outcomes in Guinea Bissau TITLE Final external evaluation of the extra budgetary project on Improvement of teacher qualification and setting up of system for the management of learning outcomes in Guinea Bissau. BACKGROUND The large number of unqualified teachers in the basic education system in Guinea Bissau results from the interruption of the teacher training system during the period of political tension, which led to the integration of many teachers without adequate initial training or without any training in the system. To tackle this problem, UNESCO Office in Dakar initiated a project in 2009 to assist the Ministry of Education in the reconstruction of the education system in Guinea Bissau that aimed at developing effective in service and pre service teacher training systems to sustain a body of highly qualified teachers capable of delivering quality education. This 1.203.450 $ project was funded by the Italian Funds in Trust (IFIT) and because of a smooth running of the operations, the donor allocated additional 400.000 $ for the period between 2011 and December 2012, allowing not only to continue the work already initiated, but also to support the Ministry in elaborating a long term teaching policy (TTISSA). See the Phase II project document. PURPOSE AND USE OF THE EVALUATION The main objectives of this final evaluation are: to assess the effectiveness, relevance and sustainability of the project; to measure the degree of implementation of the project and the quality of the process, products and achievements in relation to what was originally intended or subsequently formally amended; to develop a better understanding of how and to what extent this project is contributing to facilitating teachers in achieving better qualification and the Ministry of Education in improving in service and pre service teacher training systems and policies; and 1 P age
to formulate recommendations that will be useful for designing and implementing future similar interventions. The expected uses of the evaluation by key stakeholders are: - For accountability purposes: generating evidence of key achievements and challenges will inform decision making processes of the donor, Ministry of Education, UNESCO Dakar as executing agency, UNESCO s Senior Management, and the Executive Board, as well as programming staff. - For learning purposes: o informing Guinea Bissau Ministry of Education (MoE), beneficiaries, UNESCO Dakar as executing agency and other relevant stakeholders about the results so that they can consider whether and to what extent their engagements have been useful and complementary with the interventions of key partners, as well as what further efforts need to be made in the future to build on and sustain the achievements made so far o generating a clear understanding of teachers issue in the country and how and to what extent this project has delivered the adequate mechanisms to improve the status quo in a progressive and sustainable matter, as well as identifying main challenges, obstacles and how these were overcome, and lessons learned will inform the development of future similar interventions and implementation mechanisms by UNESCO and other key stakeholders. This evaluation was planned according to the standard requirements for extrabudgetary project implementation. It constitutes the final evaluation, which builds on monthly (internal) and annual reports (submitted to the donor), and the first phase of the project and its mid term evaluation. SCOPE This evaluation covers mainly the phase two of the project (from 2012 present), while it builds on the evaluation for the phase one that took place in 2011, as an overall assessment regarding the extent to which the intended objectives had been achieved and the level of sustainability of the results. The evaluation will be based on 4 following standard evaluation criteria according to the definition by OECD DAC (2002): relevance, efficiency, effectiveness and sustainability. However, relevance, effectiveness and sustainability criteria should be given more importance in the evaluation. The following are indicative evaluation questions that shall be refined during the inception phase of the evaluation, based on the theory of change that is expected be developed in the initial phase of the evaluation. 2 P age
1) Relevance: The extent to which the project is aligned to the priorities and policies of UNESCO, target group(s), donor(s) and other stakeholders. Indicative questions to consider while conducting the evaluation: 1. Are the 3 components on pre service and in service training and pilot bilingual education consistent with the political orientations in Guinea Bissau? 2. Is the project consistent with the C/5 and C/4 1 (including the relevant expected results, Strategic Objectives and UNESCO s Global Priorities), the Millennium Development Goals and other international development goals? 3. Are the project interventions adapted to the local context, including the model used for the training of teachers on bilingual education? 4. To what extent are the components of this intervention aligned with complementary to/or overlapping with those of other education actors, implementing agencies or development partners? 5. To what extent is the pre service teacher training programme proposals relevant to the pressing demand of increasing teacher s body, taking into account the high number of unqualified in service teachers in the country while considering their competence level 2) Effectiveness: The extent to which a project attains its objectives. It has to do with the immediate effects of project outputs on target groups (and beyond). Indicative questions to consider while conducting the evaluation: 1. To what extent were the in service and pre service teacher trainings (grades 1 to 6) improved at policy level, at implementing level, and at the teachers level? 2. To what extent were the national capacities reinforced through the implementation of this project? 3. Does the model used for the training of teachers on bilingual education prove to be useful to improve learning and is it scalable? 4. What were the major factors influencing the achievement or non achievement of the objectives of the project? 5. Did the project have effective monitoring mechanisms in place? 1 37C/5: UNESCO s Programme and Budget 2014-2017 http://www.unesco.org/new/en/bureau-of-strategic-planning/resources/programme-and-budget-c5/ 37C/4 : UNESCO s Medium-Term Strategy 2014-2021 http://www.unesco.org/new/en/bureau-of-strategic-planning/resources/medium-term-strategy-c4/ 3 P age
6. Has the project been cost effective, i.e. could the outcomes have been achieved at lower cost through adopting a different approach and/or using alternative delivery mechanisms? 7. To what extent have the project interventions been taking gender equality aspects into consideration? 3) Sustainability: This concerns the issue of whether the benefits of a project are likely to continue after (UNESCO) support/funding has been withdrawn. Indicative questions to consider while conducting the evaluation: 1. To what extent did UNESCO involve targeted beneficiaries (teachers, MoE, INDE, etc.) in the design, delivery, and monitoring of the project? 2. To what extent will national appropriation of the project results be ensured and more specifically by the Ministry of Education? In particular: a. How the 5 developed training modules and guides for in service teachers grades 1 to 6 will be used as reference at national level to deliver quality teaching in alignment with the curricular reform and the competencebased approach adopted by the Ministry? b. How the pilot in service teacher training on the 5 revised modules and guides can be extended nationally to reinforce the capacities of all the inservice teachers grade 1 to 6? c. How the proposal of pre service teacher training programmes will be used as a reference standard by the Ministry of Education? 3. To what extent will the benefits of national capacity reinforcement continue after UNESCO s support ceased? 4. Are any potential donors/partners interested in maintaining/further building upon the project outputs of the components on pre and in service trainings and on bilingual education? 4) Efficiency: This concerns the relation between the outputs (qualitative and quantitative) and the inputs. An efficient project uses the least costly resources to generate its planned outputs. Indicative questions to consider while conducting the evaluation: a. Have UNESCO s organizational structure, managerial support and coordination mechanisms adequately performed in function of timely delivery? b. Could more activities and outputs have been delivered with the same resources? c. Has UNESCO been focusing on the areas where it is best equipped to intervene, while aligning/coordinating with other developing partners? 4 P age
METHODOLOGY The different steps of the evaluation will be the following: Analysis of available documentation provided electronically by the project team Analysis and refinement of the logic framework through a series of criticalthinking questions (theory of change) to confirm and to assess the causal relationships between the different project milestones and the underlying critical assumptions and to assess Submission of an inception report at the initial stage of the evaluation for developing and agreeing upon the detailed methodological approach and time schedules In person and skype/telephone interviews and as necessary, focus groups and (electronic) surveys Field mission to Dakar to meet with the project coordination team and partners (such as AfDB) Field mission to Bissau to meet with the Project Officer, the implementing institution (INDE) national experts, and General Director, the Ministry of Education, UNESCO National Commission, national consultant providing technical support, UNICEF, and other relevant education stakeholders Submission of the draft report Debriefing mission in Dakar Submission of the final evaluation report Estimated total duration of the consultancy: one month (tentative dates: 24 August to 25, 2015) Tentative dates (to be confirmed) CONSULTANT 28 August Inception report Deliverables 4 17 18 25 Field work in Guinea Bissau and Dakar Draft evaluation report Final Report 3 23 UNESCO Feedback/suggestions on the Inception report Feedback/suggestions on draft evaluation report 5 P age
ROLES AND RESPONSIBILITIES The evaluation is expected to be conducted by a single evaluator. The evaluator will be responsible for data collection, analysis and report writing. S/he will also be responsible for logistics including office space, travel arrangements, printing of documentation, communications, laptop. The project team will provide the available documentation to the evaluator and facilitate contact with relevant stakeholders to be consulted during the evaluation. FEES AND TRAVEL ARRANGEMENTS The evaluator will be paid a fees lump sum of 7,500 USD. An additional amount not exceeding $5.000 will be included in the evaluator s contract to cover travel costs 2 according to UNESCO s Administrative Manual. S/he is expected to make his/her own travel arrangements. QUALIFICATIONS OF EVALUATOR Mandatory qualifications: Master degree or higher in Education and/or on a relevant field with at least 10 years of experience in the Education sector Extensive knowledge of, and experience in applying, qualitative and quantitative evaluation methods; Strong record in designing, planning, conducting/leading evaluations, in particular of education projects and programs; Ability to work under tight schedules with minimal supervision, excellent oral and writing skills in Portuguese (native or fluent) and in English (native or fluent), as well as working level knowledge in French (to be able to read and understand documents produced in French). Strong analytical and report writing skills Desirable qualifications: Expertise in basic education and more specifically on teacher training as well as on bilingual education. Detailed knowledge and understanding of the role of the UN and its programming Experience and knowledge of the UN gender equality and human rights mandate Previous experience in Guinea Bissau or PALOP. 2 Travel costs include economy-class air ticket and lump sum to cover all travel expenses. The consultant will be covered by UNESCO s insurance for service incurred accidents, illnesses and death, within the terms of and provisions of UNESCO s insurance policy. 6 P age
The candidate for this assignment shall submit a short technical and financial proposal (not exceeding $12,500, see FEES AND TRAVEL ARRANGEMENT) for the assignment, (no more than 5 7 pages), accompanied by a CV, as well as two examples of recently completed evaluation reports to Akemi Yonemura (a.yonemura@unesco.org; cc to c.collin@unesco.org and me.sagna@unesco.org) no later than July 24, 2015. The candidate should equally send a minimum of two relevant professional references. DELIVERABLES AND SCHEDULE Three main deliverables are expected: 1) An inception report which contains the (refined) intervention logic (theory of change) of the project (based on desk study), an evaluation plan and detailed methodology and a list of reviewed documents. The evaluation plan should contain the proposed data collection methods and data sources to be used for addressing each evaluation question/dimension. An evaluation matrix will connect questions to data collection methods/sources. 2) A draft evaluation report including preliminary evaluation findings and conclusions 3) A final evaluation report of not more than 20 pages, single space, excluding annexes and that shall be structured as follows: o Title Page o Executive Summary (not more than 3 pages in Portuguese and in English) o Table of Contents o Lists of Tables, Figures (if included) o Introduction: Background of the evaluation Purpose and scope of the evaluation Methodology of evaluation, including its limitations o Project description and Theory of Change o Findings and Conclusions o Lessons learned o Recommendations o Annexes (including further details on key questions, source of data collected, meetings held, timeline, the list of stakeholders consulted, key documents reviewed, ToRs, survey forms and aggregate findings, budgetary analysis, etc. that provide further evidence for the evaluation findings) The executive summary will be produced in Portuguese and in English while the other chapters will be elaborated in English. 7 P age
REFERENCES/MAIN DOCUMENTS TO REVIEW Campos Bartolo; Furtado Alexandre (2009). Política Docente na Guiné Bissau. Banco Mundial. Barreto, Antónia Belchior Ferreira, Alzira Rascão Saraiva e José Brites Ferreira (2010). Volume I. Referenciais de Formação Inicial de Professores do Ensino Básico. Barreto, Antónia Belchior Ferreira, Alzira Rascão Saraiva e José Brites Ferreira (2010). Volume II. Formação inicial de Professores do Ensino Básico Documentos orientadores de desenvolvimento curricular. Harmonized modules Bilingual education materials Ministry of Education. Development program for the education sector (2009 2020) Pole de Dakar & World Bank (2013). Rapport d état du système éducatif (RESEN): Diagnosis of the Guinea Bissau Education System (CSR). Available at: https://www.iipe poledakar.org/sites/default/files/fields/publication_files/resen guinee_bissau 2013_0.pdf United Nations Development Assistance Framework (UNDAF) 2013 2017 and relevant & revised documents (obtain from Resident Coordinator or UNICEF) Monthly activity reports, progress reports Evaluation reports : Rapport d évaluation à mi parcours du projet (Juillet 2010) et Rapport d évaluation du projet (Octobre 2011) 8 P age