Formative evaluation of UNICEF s Monitoring Results for Equity System (MoRES) UNICEF management response

Formative evaluation of UNICEF s Monitoring Results for Equity System (MoRES) UNICEF management response Evaluation title: FORMATIVE EVALUATION OF UNICEF S MONITORING RESULTS FOR EQUITY SYSTEM (MoRES) Year: 2014 Office and person in charge of management response: Programme Division, Ted Chaiban, Director Overall response to the evaluation: The formative evaluation of MoRES used an approach based on case studies and theory. The evaluation affirms that the concepts underpinning MoRES present UNICEF with a number of opportunities for achieving better results for the most disadvantaged children. It also provides important insights into the conceptual, technical and operational issues that need to be addressed in order to strengthen the approach as it is being applied in wider and more varied contexts. In addition, the evaluation provides a helpful analysis of the overall governance of MoRES and management of institutional change, noting both positive aspects and shortcomings. This analysis will be helpful for guiding the further mainstreaming of MoRES and informing major corporate initiatives in the future. Overall, the findings and recommendations of the formative evaluation together with knowledge and insights gained from ongoing field experiences have been very valuable in informing adjustments and adaptations of MoRES application by partners at country level to ensure that the approach fulfils its primary purpose - to accelerate results for the most disadvantaged children. Planned use of the evaluation: The formative evaluation is already informing the ongoing application of MoRES at the country level as well as the overall MoRES mainstreaming strategy for the organization (including by identifying areas on which headquarters and regional offices need to focus). The evaluation has been widely disseminated and discussed internally at global and regional technical network meetings as well as with management. Furthermore, it has been publicized through the internal communication website, ICON. The findings and recommendations were also discussed at the Global Management Team (GMT) meeting in February. The findings of the evaluation have been used to both (a) validate the work that continued during 2014, while the evaluation was taking place, to make sure the focus and priorities of the 2014 mainstreaming workplan were on track and to make any necessary adjustments; and (b) inform the next phase of MoRES: extending the application of the approach fully to all UNICEF country programmes of cooperation. This next phase involves adapting MoRES to various contexts and supporting Governments and partners to fine-tune existing national/subnational plans and programmes so that they become more focused on equity and are able to deliver even better results. The management response has been developed in consultation with regional and country offices, enabling a wider internalization of the findings across UNICEF. The insights and knowledge gained from this evaluation are valuable to the organization and contribute to the strategic efforts of UNICEF to learn from experience in order to achieve better results for children. Recommendation 1. Maintain the focus on equity: UNICEF should emphasize that the primary purpose of MoRES is to support UNICEF s refocus on equity through a central commitment to generating robust evidence. Overall, more will be achieved if MoRES is further mainstreamed as a

commitment to encouraging equity-focused decision-making based on robust evidence, rather than as an integrated UNICEF system. MoRES should be understood as not an one size fits all system but as an approach that (a) begins with an intent to accelerate progress towards achieving equity for children, (b) recognizes local context particularly the political and institutional landscape as its starting point, before (c) determining what MoRES as an evidence-based approach can add, and (d) establishing what the ideal partnership strategy is to achieve this. Management Response (Agree, Partially Agree, Disagree): Partially Agree UNICEF does not agree with the statement that the primary purpose of MoRES, in support of UNICEF s refocus on equity, is to generate robust evidence. As the recommendation further notes, a key objective of MoRES is to catalyse the use of this robust evidence to take management decisions and make course corrections quickly to more rapidly improve programmes/policies in order to achieve better results for disadvantaged children this use of data in a more strategic and timely way is the primary purpose. It is agreed that the focus on equity should be maintained, that MoRES should be communicated as an approach rather than as a UNICEF system, and that should be adaptable to context. These points have been clarified in recent briefing notes and guidance, and disseminated to field offices. Actions planned 1. Updates to main briefing note on MoRES to emphasize key points in this recommendation, including adaptation to context, while preserving the core of the approach and working with a range of partners to apply the approach, building on existing systems and approaches in countries. 2. Technical guidance on MoRES to reflect key points in this recommendation. Responsible office/person Group / Programme Division (PD) Group / Programme Division Expected completion date January August 2014 Implementation stage: Cancelled Actions taken The use of MoRES to improve equity-focused and evidence-based programming, as well as the need to adapt the approach to country contexts and to develop a clear partnership strategy at the outset, has been emphasized in all recent guidance notes and briefings. These points were clearly highlighted in the updated Briefing Note (January ), the widely disseminated compendium of 15 Supporting documents Accelerating Results for the Most Disadvantaged Children: Monitoring Results for Equity System (MoRES) Briefing Note, January The Determinant Analysis for Equity Programming, August 2014

3. Case studies documenting the application of MoRES, highlight MoRES as an approach to support the equity refocus and to achieve results for the most disadvantaged children Group Feb country case studies on MoRES, guidance on determinants analysis, document on links between existing technical guidance and MoRES, and the February Global Management Team session on MoRES. Pursuing Equity in Practice A compendium of country case studies on the application of the Monitoring Results for Equity System (MoRES), February Key Initiatives and Tools that Support the Application of MoRES, May 2014 4. Strategy for wider application of MoRES in in support of the equity strategy agreed and tracked at seniormanagement level of the organization. Deputy Executive Director (Programmes) February Strategy agreed at the GMT meeting and side meeting, 10-11 February. Key Messages and Decisions on MoRES Mainstreaming from the GMT meeting, and side meeting, 10-11 Feb Recommendation 2. Develop a mainstreaming strategy based on the enhanced conceptual clarity and findings produced by the formative evaluation report. The mainstreaming strategy should (a) recognize the importance of generating robust data-based evidence for equity gaps at national and decentralized levels, assessing the financial and capacity implications of more frequent monitoring; (b) communicate MoRES as an approach rather than a system; (c) include enhanced guidance and systems for knowledge management that support the application of the elements of MoRES; (d) devolve responsibility for the adaptation of the MoRES elements to fit the context and sector requirements to the regional and country levels; (e) introduce a resourcing and capacity plan for partners, country offices and regional offices; and (f) ensure that appropriate accountability mechanisms are in place at various levels within the UNICEF management structure [headquarters (HQ), regional offices (ROs) and country offices (COs)], with clearly defined roles and responsibilities. Management Response: Partially Agree

While the overall recommendation on having a clear mainstreaming strategy is noted and agreed, we partially agree with this recommendation because at the time of the evaluation, a mainstreaming strategy had already been developed for 2014 (which was not explicitly noted in the evaluation report) and a stocktaking exercise was conducted at the end of 2014. Some of the points noted in this recommendation were already included in the 2014 mainstreaming strategy even before the evaluation findings were released (e.g., communicating MoRES as an approach (point b), introducing a capacitystrengthening plan (part of point e), updating and developing guidance as needed, and enhancing systems for knowledge management (point c). However, for -2016, UNICEF is updating and improving the existing mainstreaming strategy, taking into account the evaluation findings, with the actions outlined below having been identified and prioritized based on a consultative process across the organization. (It should be noted that the observation on having appropriate systems for knowledge management is a broader agenda for the organization investments in platforms that would serve to improve overall knowledge management and exchange in UNICEF will also greatly improve knowledge management and exchange in applying MoRES.) Actions planned 1. Fully integrate the MoRES approach and its methodologies and tools into the Programme Policy and Procedure Manual (PPPM) and training materials. Through this, clearly explain link of MoRES with results-based management (including the Theory of Change) and the human rights-based approach to programming. Responsible office/person Field Results Group (FRG)with Group Expected completion date September Implementation stage: Cancelled Actions taken MoRES has been incorporated into the PPPM to some extent, but needs to be fully integrated, including in the sections covering the situation analysis, programme design, and monitoring and evaluation. Supporting documents UNICEF Programme Policy and Procedure Electronic Manual (UNICEF Intranet) Guidance on Conducting a Situation Analysis of Children s and Women s Rights: Taking a rightsbased, equityfocused approach to Situation Analysis, March 2012 2. Integrate the MoRES framework into insight/ram and related guidance to ensure that results statements and FRG/PD/Division of Data, Research and Policy (DRP) December 2014 Procedure on Using the Results Assessment Module (RAM 2.0) of VISION/inSight

indicators are linked to the determinant framework and are tracked on a frequent basis as part of programme monitoring. 3. Articulate the relationship between Harmonized Approach to Cash Transfers (HACT) processes and MoRES, particularly regarding field monitoring done through HACT and how this can be used to support frequent monitoring of bottlenecks, as stipulated in MoRES. 4. Update country plans for the application and documentation of MoRES experiences with identified support needs from regional offices and headquarters. Plans should be based on, or include, context analysis to determine how to adapt the approach, as well as an assessment of data availability, quality and critical gaps at country level and resource requirements. FRG with Group Deputy Regional Directors with regional MoRES focal points and PD sector leads May Draft field monitoring tools, incorporating data collection on bottlenecks, are available and are being reviewed by a field reference group. May Regional offices have been coordinating this exercise, and several have systems to track progress on the application of MoRES in various countries. Some sectors (e.g., Education) and regions [e.g., the Middle East and North Africa (MENA), and West and Central Africa (WCAR)] have undertaken a learning / capacity needs assessment to better tailor support to (FRG/PROCEDURE/20 14/003), November 2014 Field Monitoring Checklist, Draft, April Rationale for Field Monitoring Checklist (PowerPoint), April Key Messages and Decisions on MoRES Mainstreaming from the GMT meeting, and side meeting, 10-11 Feb MoRES Education Implementation and Needs Assessment Survey, February MoRES in WCARO: Update, February

country offices for the application of MoRES. MoRES in MENA Taking stock and moving forward: Briefing for PD, January MoRES - Learning Needs Assessment Survey, MENA, 2014 5. Capacity development of counterparts and partners is clearly prioritized and carried out as an integral part of the country application of MoRES in various sectors. Country offices supported by regional offices, and technical assistance from Group Continuous In countries where the application of MoRES has advanced, counterparts and partners have been oriented on the MoRES approach and have contributed to its adaptation based on context and requirements. Pursuing Equity in Practice A Compendium of Country Case Studies on the application of the Monitoring Results for Equity System (MoRES), February 6. Update global mainstreaming plan with inputs from country/regional levels and headquarters divisions, and institute a process for tracking progress; the global plan will consolidate key elements of regional-level plans as well as plans at global level to integrate PD Director with MoRES Working Group June Global mainstreaming plan is being updated.

the MoRES approach into sector-wide approaches and tools. 7. Update quality assurance tools and mechanisms for country programme documents and sectoral programmes to include core elements of MoRES, and define roles/responsibilities of HQ, ROs and COs 8. Develop and implement a knowledge exchange system, including relevant communities of practice that foster in-country and country-to-country learning on equity-focused programming among partners 9. Work with key UN partners to integrate the MoRES approach into wider UN guidance on monitoring equity Group (HQ and ROs) with FRG DRP PD/Public Partnerships Division (PPD) June 31 October Development of knowledge exchange platform has commenced. Public Financing for Children (PF4C) Community of Practice already created and in good use. Feb The draft inter-agency guide on frequent monitoring for equity has been prepared and is ready for field testing. Concept Note, Knowledge Exchange Platform for Equity- Focused Programming Guidance on UN Joint Approach to Frequent Monitoring for Equitable Results (Draft), January 10. Field test and finalize joint guidance DRP with support from PD/PPD Dec Field testing of the guidance is part of the United Nations Development Group (UNDG) workplan for. UNDG Work Plan: Results and Key Deliverables (Draft)

Recommendation 3: Resolve the technical challenges associated with MoRES processes and tools. There are some elements to MoRES which add significant even transformational value when they fill a gap in existing systems, particularly bottleneck analysis. However, across all contexts, there remain technical challenges to resolve, which signal a need for stronger guidance and continuous review. Management Response: (Agree, Partially Agree, Disagree) Agree If recommendation is rejected or partially accepted, report reasons: Actions planned 1. Sectors will review existing technical guidance and prepare any additional guidance required by the field, including addressing specific technical issues as needed. The updates to guidance will highlight opportunities for crosssectoral work and engaging multiple sectors in applying MoRES to improve outcomes for children. Further clarification will also be provided on adaptation of MoRES in humanitarian contexts. 2. Short Q & A addressing common issues from the field will be prepared and disseminated through the MoRES website and technical networks Responsible office/person Group/ Programme Division MoRES Technical Team (HQ and ROs) Expected completion date 30 September (pieces that are deemed urgent and critical by the field can be completed earlier) Implementation stage: Cancelled Actions taken Several Programme sections have updated plans for mainstreaming MoRES these plans include production of any additional technical guidance that may be required (these are being consolidated and will inform the updated - 2016 global mainstreaming strategy). May Q&A document is being drafted. Supporting documents Draft Programme Section Work Plans for MoRES Mainstreaming selected sectors

3. Strategic review of staffing profiles will be undertaken to identify critical human resource gaps and/or make adjustments to specific job descriptions to support application of MoRES at country level Regional Offices, PD and DRP with the Division of Human Resources August Recommendation 4: Develop a policy advocacy strategy at national level for stronger links (a feedback loop) between locally identified bottlenecks and access to the resources required to remove them. Sometimes the participation of the government in resolving bottlenecks is limited by capacity and resources. Furthermore, the complex/fragmented government planning and finance cycles in many countries and highly centralized budgets create further barriers, since resources are not easily or readily reallocated. Management Response: (Agree, Partially Agree, Disagree): Agree MoRES can be applied and adapted in various decentralized settings. However, for the feedback loop to work effectively there needs to be a clear and strong link to decision-making processes and accountability frameworks at different levels. There are opportunities to strengthen the work on linking the identification of bottlenecks and monitoring data on bottlenecks to the variety of decision-making, costing, budgeting and financing systems in different countries. If recommendation is rejected or partially accepted, report reasons: Actions planned 1. Develop and roll out a simple advocacy planning tool and guidance for advocacy to policymakers. The tool will support advocacy on linking equity analysis, bottleneck analysis and bottleneck monitoring to budgets at different levels of governance in different settings (e.g., low-income Responsible office/person DRP and PD Expected completion date September Implementation stage: Cancelled Actions taken Advocacy tool is being drafted. Supporting documents

and middle-income countries). 2. PPPM will include a specific focus on stakeholder engagement and advocacy with governments to support use of data for decisionmaking 3. Document adaptation of MoRES in various settings (related to level of decentralization, planning, budgeting and monitoring processes) FRG and PD Group September February The recent compendium of 15 country case studies on MoRES and the synthesis of findings also highlight how MoRES has been adapted to country contexts, including through taking into account different levels of decentralization. Pursuing Equity in Practice A compendium of country case studies on the application of the Monitoring Results for Equity System (MoRES), February 4. As part of the Toolkit on MoRES, develop and roll out operational guidance on how to use data from MoRES to remove identified bottlenecks related to budgets, Public Financial Management and local governance practices. PD (Social Inclusion and Policy Section) October Technical notes on relevant key topics are in draft form and will be repurposed to show clear links to MoRES. Recommendation 5: Evaluate MORES as an approach which supports the development and implementation of the Strategic Plan and Country Programme, rather than conduct a stand-alone evaluation of MoRES. Focus on the results to be achieved for the most disadvantaged children and the accelerated reduction in equity gaps.

Management Response: (Agree, Partially Agree, Disagree) Agree If recommendation is rejected or partially accepted, report reasons: Actions planned 1. UNICEF will evaluate the application the MoRES approach within individual programmes, paying attention to the contribution of MoRES to achieving results for disadvantaged children and in reducing equity gaps. Guidance will be prepared on how to formulate Terms of Reference for evaluations intended to include assessment of the role and contribution of MoRES to programme results. Sample Terms of Reference and related guidance will be included in the PPPM and disseminated to staff and management. 2. The 2016 midterm review of the Strategic Plan will review (among other issues) the application of Responsible office/person FRG / Evaluation Office DRP Expected completion date June September 2016 Implementation stage: Cancelled Actions taken Supporting documents

MoRES as an approach in the Plan s roll-out. Recommendation 6: UNICEF management should use the experience of MoRES roll-out (and evidence from other initiatives) to inform the management and implementation of future major organizational change initiatives within UNICEF. The experience of implementing MoRES within UNICEF has generated some very important lessons on how major change processes involving corporate priorities can be implemented successfully. Management Response: (Agree, Partially Agree, Disagree) Agree If recommendation is rejected or partially accepted, report reasons: Actions planned 1. Share lessons learned from the roll-out of MoRES particularly related to communication, field engagement, formalization of management structures, staff empowerment and generating short-term wins with senior management for application to other major organizational change initiatives within UNICEF Responsible office/person Deputy Executive Director (Programmes) Expected completion date Recommendation is continuous by its nature Implementation stage: Cancelled the sharing of lessons learned - Application of lessons learned Actions taken Lessons from evaluation were discussed at the GMT meeting. Supporting documents Key Messages and Decisions on MoRES Mainstreaming from the GMT meeting, and side meeting, 10-11 Feb