A National Strategy for the Development of Education Statistics (NSDES) Guidelines for country-level design of the NSDES

Transcription

1 A National Strategy for the Development of Education Statistics (NSDES) Guidelines for country-level design of the NSDES 0

2 CONTENTS 1. Introduction 1.1. Context, purpose and users of the guidelines 1.2. What is the NSDES? Essential features of a results-oriented NSDES A Results-Based Management (RBM) approach for the NSDES 2. Preparation of a NSDES 2.1. Getting Started Reaching consensus on the NDSES goal Using Data Quality Assessments (DQA) to prepare the ground 2.2. Designing the Results Framework Step 1. Defining results Step 2. Clarifying partners' roles and responsibilities Step 3. Identifying assumptions and risks Step 4. Identifying resources Step 5. Developing a monitoring and evaluation (M&E) matrix 2.3. Moving toward implementation Developing operational plans Using the M&E matrix during programme implementation Finalizing and mainstreaming the strategy ANNEX 'Tool kit': Tool 1. Prioritizing recommendations and setting strategic objectives Tool 2. Results matrix template Tool 3. Risk matrix template Tool 4. M&E matrix template Tool 5. Costed program plan template / annual work plan (AWP) template 1

3 1. INTRODUCTION 1.1. CONTEXT, PURPOSE AND USERS OF THE GUIDELINES In May 2015, participants from 160 countries gathered at the World Education Forum (WEF) hosted by the Republic of Korea, to agree on and adopt the Incheon Declaration for Education A key feature of the new education agenda is a resolve to develop "comprehensive national monitoring and evaluation systems in order to generate sound evidence for policy formulation and the management of education systems as well as to ensure accountability" (Incheon Declaration, 2015). 1 In the interests of better education data for strengthened education management and accountability, participants in the WEF of 2015 specifically requested co-conveners and partners to "support capacity development in data collection, analysis and reporting at the country level", enabling countries to "improve the quality, levels of disaggregation and timeliness of reporting to the UNESCO Institute for Statistics" (Incheon Declaration, 2015). The Capacity Development for Education (CapEd) programme is one of UNESCO's responses to the challenges raised by the "holistic, ambitious and transformative vision" of the Education 2030 development agenda. The programme supports countries in integrating the Sustainable Goal for Education (SDG 4) into education plans and monitoring systems. Component 2 of the CapEd programme specifically answers the call made by member states at Incheon, designed by UIS to assist countries in aligning and strengthening national data and education management information systems to monitor progress towards SDG 4 targets. Component 2 promotes a coherent mediumterm (3-5 years) approach for sustainably building capacities of national education statistical systems (NESS) by supporting: The setting up of an education data platform - comprising a Steering Committee made up of multiple stakeholders of the SDG 4 and a national technical team (NTT) - to develop and implement a sectorwide vision of education statistics; The design, implementation and monitoring of a country-level national strategy for the development of education statistics (NSDES). The purpose of these Guidelines is to accompany country-level stakeholders through a process of preparing their NSDES and medium-term capacity development program plan for education statistics, ensuring that such a strategic plan is results oriented. The process follows a 'learning by doing' approach, using a set of simple, flexible tools which can be adapted to suit specific contexts. The users of the Guidelines are members of the NSDES education data platform, specifically the National Technical Team guided by the Steering Committee. Where requested, this team work may be supported by technical assistance provided either by UNESCO or by externally sourced expertise, or both. Following a preparation phase during which the education data platform will be initiated, design of the NSDES will take place in the following phases: 1 Over the last 15 years, results-based management (RBM) has emerged as a key instrument for development effectiveness. The Incheon Declaration is a recent landmark in a series of international agreements that have all underscored the importance of increased accountability of governments and development partners towards the achievement of results. Among these international agreed principles are the Monterrey Consensus (2002), the Rome Declaration on Harmonization (2003), the Paris Declaration (2005), the Hanoi Conference on Managing for Development Results (2007), the Accra Agenda for Action (2008), and the Busan Partnership for Effective Development Cooperation (2011). 2

4 Mapping relevant data sources and identifying data gaps against the contextualised SDG 4 indicators framework; Conducting a situation analysis through a series of Data Quality Assessment (DQA) for various required data sources and producing the DQA Report, including recommendations; Developing and validating the NSDES. The Guide will take you through the final phase, that is, the process of designing a NSDES and strategic program plan. The process is illustrated by Figure 1 below, which also indicates the relevant tools corresponding to the stages in the 'learning by doing' approach. Figure 1. The process of designing an NSDES Mapping data sources and identifying data gaps in relation to SDG 4 indicators Conducting a situation analysis using Data Quality Assessment instruments Tool 1. Prioritizing recommendations and setting strategic objectives 1. GETTING STARTED Using the DQAs to design the NSDES Learning by doing Tool 2 Results matrix template and guidance Tool 3. Risk matrix template and guidance Tool 4. M&E matrix template and guidance Tool 5. Costed program plan template / annual work plan (AWP) template 2. DESIGNING THE RESULTS FRAMEWORK Step 1. Defining results Step 2. Clarifying partners' roles Step 3. Identifying assumptions and risk Step 4. Identifying resources Step 5. Developing a ME&L plan 3. MOVING TOWARD IMPLEMENTATION Developing an operational program plan Finalizing the strategy Monitoring, evaluation and learning (ME&L) 3

5 1.2. WHAT IS THE NSDES? A National Strategy for the Development of Education Statistics (NSDES) is a policy instrument designed by government and its partners, namely the education data platform. It provides a medium-term vision for a strengthened education data system and data management platform in a country: the National Education Statistics System (NESS). The NSDES should be results-oriented - outlining a coherent set of intended changes in the production, reporting and use of education statistics - and based on a sound analysis of the current education statistics situation, i.e. a series of DQAs (one per relevant data sources). It should be integrated into the national Education Sector Plan (ESP) and the overall multi-sector National Strategy for Developing Statistics (NSDS) and should include a monitoring and evaluation and (M&E) framework to guide implementation ESSENTIAL FEATURES OF A RESULTS-ORIENTED NSDES The essential characteristics of a results-oriented NSDES are as follows: i. It contributes to the country's development vision. The NSDES aligns with the government s international commitments and overall national development framework, the mission statement of its ESP, and the overall NSDS goal. As such, it is a living-document, responding to the country's evolving education development context ii. iii. iv. It is sector-wide. The NSDES covers all subsectors (early childhood education, primary, secondary, TVET, and higher education) and should also include non-formal education, as well as adult literacy in line with SDG4 targets. It recognizes the need for a sector-wide approach and reflects an awareness of equitable, quality lifelong learning It is strategic. By definition, the NSDES prioritizes intended results, including human, technical, and financial capacities, in order to achieve the development vision: the NSDES results framework shows a focus on specific, objectively measurable results and a clear intervention logic, ensuring planned results are realistic. It is achievable. Responding to actual financial, technical, and political constraints in the area of education statistics, the NSDES provides a framework for budget and management decisions; the extent to which the strategy is owned by key stakeholders largely determines NSDES feasibility. v. It is evidence-based and pays attention to a country's specific data-related challenges. The process of developing the NSDES begins with an education data quality assessment (DQA), providing an information base to guide design of the NSDES. In addition to the DQA, the NSDES is informed by various surveys and consultations undertaken to assess the readiness of country's to engage with the SDG 4 agenda (Box 1). vi. It is sensitive to the country context. The NSDES should be in line with other ESP strategies to strengthen the education system at all level; these are typically based on an analysis of the vulnerabilities specific to a country (e.g. conflicts, disasters, and economic crises) and address preparedness, prevention, and risk mitigation. 4

6 The process of preparing the NSDES includes the following features: It is a country-led process. The NSDES is the responsibility of the national government, which has to make the final decisions in terms of committing resources, as well as for its implementation. The strategy must be grounded in a government's commitment to the preparation process, and this commitment should deepen through the process. It is an inclusive process. The strategic planning process should be accompanied by a participatory policy dialogue among multiple stakeholders that builds consensus on, and commitment to, the development of strategy; existing mechanisms for such policy dialogue between the government and its development partners include the Sector Working Group (sometimes called the Local Education Group). It is a well-organized process. Clarity is required on the roles and responsibilities of the multiple stakeholders, especially those who lead and coordinate; NSDES structures may include; a steering committee to oversee and guide the process; a National Technical Team (NTT) to coordinate the technical work, bringing together all ministry directorates and departments; and selected working groups to focus on specific themes or subsectors. It is a process of growing mutual accountability for education statistics. This entails the respective accountability of multiple-stakeholders working together toward shared outcomes, where each stakeholder is accountable for its own contribution. It is a capacity-development process. Plan preparation is itself a form of 'learning by doing' capacity development, making the process of NSDES preparation as important as the strategy itself. It is a process that reflects the comparative advantage of UNESCO and its partners in addressing the systemic constraints and capacity gaps related to education statistics A RESULTS-BASED MANAGEMENT (RBM) APPROACH FOR THE NSDES RBM is an approach for strategic planning. Using this approach, all stakeholders ensure that their processes, products and services contribute to the achievement of intended results (outputs, outcomes and higher level goals or impact). The stakeholders in turn use information and evidence on actual results to inform decision making for programming accountability and reporting. Two key concepts used in RBM programming are outlined in this section and are elaborated through the step-by-step NSDES preparation process: the RESULTS CHAIN and the RESULTS FRAMEWORK. The NSDES Results Chain A result is change that can be described and measured. Results can be intended or unintended changes, positive or negative. It is expected that a results-oriented strategy and/or programme will lead to positive and intended change. But this is not always the case. As change can sometimes lead to unintended or negative consequences, it is important that the intervention is carefully monitored and adjusted in line with the programme's monitoring data. Typically, RBM strategic planning and programming entails three main levels of result, as follows. Result level 1. Impact: this is the long-term change in social or economic conditions. Result level 2. Outcome: this is a medium-term change in development conditions, measured in terms of institutional performance and/or changes in the behavior of a specific group of people or organization. 5

7 An outcome is the intended benefit to a specific community, group of people or organization that is measurable and specific. Intermediate outcomes are sometimes used to specify results proximate to an intended final outcome, but more measurable and achievable in the lifetime of a programme than the intended final outcome. Result level 3. Outputs are the supply-side deliverables, including the events, products, or services that result from a development intervention in the short-medium term. The key distinction between an output (a specific good or service) and an outcome is that the output typically is a change in the supply of goods and services (supply side), whereas the outcome reflects changes in the utilization of goods and services (demand side). Impact-outcome-output level changes are derived from a cause-and-effect relationship, which is set in motion by a development intervention or programme. The cause-and-effect sequence is the Results Chain, sometimes referred to as the results logic, or intervention logic, or logic model. The causality between results extends further to the attribution between inputs-activities and the results that are generated in the form of outputs, outcomes and impact. A results chain should also clearly represent the change achieved through the causeand-effect relationship between inputs-activities and the results. Generally, results-oriented planning requires a shift in the direction of thinking about how to achieve results. Conventional planning approaches often start with the interventions and required resources: "if X activities are implemented, using Y resources, then Z result will be achieved"; the focus here is on implementation. Using RBM entails planning "in reverse"; that is, we begin with the impact-level result (goal) and move to the objective-level results (outcomes) before detailing the interventions. Table 1. Corresponding elements in a generic logic model and the NSDES results chain. GENERIC RESULTS CHAIN Higher-order result/impact. The long-term effects near or at the top of the results chain in terms of improved social or economic conditions. Generally, a stand-alone intervention alone will not achieve the higher-order result; but a programme should identify the country development goal it seeks to influence. Program development objective The intended benefits to a specific community, group of people or organization that are to be realized through the programme. Program outcome (final and intermediate) The change in institutional performance or behavior change among users of outputs that demonstrates the uptake, adoption, or use of the program outputs by the targeted stakeholders. An intermediate outcome specifies a result that leads to the intended final outcome, but is more measurable and achievable in the lifetime of a project than the intended final outcome. Outputs The supply-side deliverables, including the events, products, or services that result from the intervention. NSDES RESULTS CHAIN Development goal The high-level country development goal or sectoral development goal; e.g. Sustainable Development Goal 4, the national poverty-reduction goal, and/or the NSDS goal. It may be achieved through a combination of development interventions, including the NSDES. NSDES Vision statement/goal This seeks to influence the above development goal or sectoral development goal. The vision/goal should clearly indicate the target group of the NSDES capacity-building intervention and what they will be doing better or differently as a result of the programme Strategic objective(s) of capacity development These are the final outcomes of the NSDES, intended to be achieved by the end of the program's first phase. They are informed by the DQA recommendations and describe the benefits to the targeted stakeholders (e.g. education data platform) in relation to one or more of four targeted NSDES components. Intermediate outcomes specify the results leading to a specific final outcome. They are intended to be achieved in 1 to 3 years of an overall five-year programme life-cycle. NSDES program outputs Knowledge, products and services 6

8 It is important to note that the terminology and level of detail used to describe the elements of a results chain can differ from programme to programme, depending on the scope of the intervention. In the case of the NSDES, the results chain needs to take into account the capacity-building function of a strategy to develop education statistics in a given country; the NSDES results chain also needs to anticipate the eventual integration of the NSDES into broader strategic planning frameworks such as the country's ESP and the umbrella multisector NSDS. Table 1 above explains the corresponding elements in the NSDES results chain and the generic logic model. Figure 2 below illustrates the NSDES results chain. Figure 2. The NSDES Results Chain Development Goal NSDES Vision statement/goal STRATEGIC OBJECTIVES/OUTCOMES Final intended outcomes of the NSDES. These are changes in institutional performance or behavior among individuals INTERMEDIATE OUTCOMES Capacity-related changes in skills or abilities that are achieved with the resources provided within a specific time frame OUTPUTS New products and services that are achieved with the resources provided within a specific time frame ACTIVITIES Actions taken to produce specific outputs INPUTS Financial resources and human resources, including technical assistance A results chain will always be embedded in a given context that reflects the overall situation, needs, issues, priorities and aspirations of key rights-holders. A range of factors economic, political, social, environmental or cultural will affect the achievement of results. The general rule is that one size does not fit all and results chains will vary from country to country. In this sense, the NSDES results chain outlined above is itself a generic logic model, and open to modification. For example, in some country contexts the intermediate outcome results-level may not be required, while in other more complex situations it is. 7

9 The NSDES Results Framework Generally, a results framework is an explicit articulation of the results chain, showing the different levels of results expected from a particular project, program, or development strategy. 2 This may be a graphic display or in matrix form The NSDES results framework comprises a results matrix and a complementary M&E matrix. As a conceptual presentation of a results chain, the results matrix captures the essential elements of the logical and expected cause-effect relationships among inputs, outputs, intermediate results or outcomes, and impact. But it also includes the following elements: Identified responsibilities of the different partners - whether they are government or a specific ministry, United Nations agency, NGO or any other implementing agency - who will achieve a given output and outcome; Identified underlying critical assumptions that must be in place for the intervention achieve the targeted outcomes and impact or high-level development goal and the risk of potential future events - which may be fully or partially beyond control - that may negatively affect the achievement of results; Estimated resources (financial, human and technical) required to achieve the agreed results. The preferred format and level of detail for results matrix vary by organization and by the scope and scale of the intervention, but all include the same basic components to guide implementers in achieving, and evaluators in assessing, results. Table 2. below presents a basic outline of a country-level NSDES results matrix. Table 2. Sample NSDES results matrix Intervention Logic Key Action Areas Implementing Partners Strategic Objective/Outcome 1 Intermediate Outcome 1.1. Output Strategic Objective/Outcome 2 Intermediate Outcome 2.1. Output Assumptions and Risks Indicative Costs (Total/Source/Gap) As mentioned above, the NSDES results matrix is accompanied by a detailed M&E matrix. In this matrix, results are further defined through indicators, which are quantifiable and measurable, or observable. With reference to baselines and target values, these are used to track progress toward achieving the results at different intervals of time In sum, the NSDES is underpinned by the results framework, including both results and M&E matrices. Besides providing a systematic approach for Education Data Platform to plan their strategies and select interventions that are most likely to address targeted problems, the results framework serves as a tool for: Consensus, coordination, and ownership, providing the opportunity for the Education Data Platform to coordinate the implementation approach and agree on the expected results; Managing implementation, using performance data to make corrective adjustments to activities, reallocate resources, and reevaluate targeted objectives or underlying assumptions; 2 Similar conceptual tools, also designed to organize information regarding intended outcomes and results, are used across different agencies: log-frames, logic models, theories of change, results chains, and outcome mapping. 8

10 Communication and reporting on the resources, activities, and outcomes to program staff, development partners, and other stakeholders; Harmonizing the efforts of diverse partners, clarifying efficient divisions of responsibilities for achieving specific results; Evaluating, clearly identifying how progress toward the targeted objective(s) will be measured and serving as the basic accountability tool for developing an evaluation approach to the intervention; Learning from experience, allowing the Education Data Platform to assess over time what approaches or interventions contribute most effectively to achieving specific development objectives, a process that helps identify good practices for replication. Table 3 provides an overview of the NSDES preparation process, the focus of each stage in the process, relevant tools to support preparation of the NSDES, and a suggested time-frame. Table 3. Stages and steps in preparing an NSDES Using the Data Quality Assessments to prepare the ground STAGE 1. Getting Started Reach consensus on the NSDES goal in line with the national education sector plan (ESP) strategic priorities Prioritize and use DQA recommendations to set strategic objectives Tool 1. Prioritizing recommendatio ns and setting strategic objectives STAGE 2. Designing the Results Framework Step 1. Defining results Formulate outcomes Tool 2. Results matrix template Formulate outputs and guidance Step 2. Clarifying Identify NSDES partners partners' roles and responsibilities Review and refine the NTT, if necessary Step 3. Identifying assumptions and risks Step 4. Identifying resources Step 5. Developing a monitoring and evaluation matrix Developing an operational program plan Monitoring and evaluating the NSDES Finalizing the strategy Integrate assumptions and risks into the results matrix Estimate the costs of the NSDES Assign the baseline, indicators and targets for outcomes and outputs Identify the means of verification Prepare to use the M&E data for learning and adjusting, during the program life-cycle STAGE 3. Toward implementation Use the results matrix to prepare a program plan / annual work plan Understanding how to use the M&E matrix for programme management and planning evaluation Prepare the NSDES document using the template provided Tool 3. Risk matrix template Tool 4. M&E matrix template and guidance Tool 5. Costed program plan template/awp template Time-frame Week day Consultative Meeting OR working session integrated into the Design Workshop Week 2/ day Design Workshop See Annex 2: Proposed Design workshop agenda Week 4/ day working session or workshop Week 6 Week 7-8 9

11 2. PREPARATION OF THE NSDES 2.1. GETTING STARTED In this section we will reach consensus on the NSDES goal, or vision statement, in line with (a) the national education policy frameworks; (b) strategic priorities of the National Statistical System (NSS); and (c) the Education 2030-SDG4 development agenda. Referring to the results of the data quality assessments (DQA), we will review the recommendations and reflect on these in light of the NSDES vision statement. Finally, we will prioritize the DQA recommendations and use these to set strategic objectives for the NSDES. The focus here is on establishing country-ownership of an evidence-based NSDES; and on reaching agreement on a strategic vision that is not in parallel with management structures and processes for existing policy frameworks and which contributes to the country's overall development priorities. In all the above sessions, the tasks must be undertaken collaboratively by stakeholders, specifically key members of the education data platform REACHING CONSENSUS ON THE NSDES GOAL At this stage, before the process of preparing the NSDES begins, the education data platform - including all stakeholder government authorities and development partners - is in place. This platform draws on those stakeholders of the overall National Strategy for Developing Statistics (NSDS), where it is in place, that are specifically engaged with the education sector, as well as other stakeholders; in the Tanzanian context, for example, these are the education sector partners who contribute to the Tanzanian Statistical Master Plan (TSMP). Aligning the NSDES with national policy frameworks You should begin by identifying the relevant policy frameworks for a country NSDES. For example, in the case of Tanzania, these include: The Tanzania Strategic Master Plan (TSMP)/NSDS. The development of this strategy includes putting in place a legal and institutional framework that will guide implementation to achieve the vision of TSMP, which is to have a well-integrated and efficient National Statistical System (NSS) that produces quality statistics for decision making in an objective, timely and cost effective manner. The Five Year Development Plan, 2016/ /21 (FYDP-II). The goal of the second implementation phase of the Tanzania Development Vision 2025, merging the FYDP and the National Strategy for Growth and Reduction of Poverty (NSGRP), is: Nurturing industrialisation for economic transformation and human development by The goal of the recent Education Sector Development plan, 2016/ /21 is: "Improved knowledge and skills for better livelihoods by 2020". If the country's commitment to the Education 2030 agenda and the SDG 4 targets have not already been integrated into the national sector plan, you may need to also reflect on SDG 4: Ensure inclusive and equitable quality education and promote life-long learning opportunities for all (see CapED training materials related to the SDG 4 target indicators metadata). 10

12 Drafting the NSDES vision statement / goal You now have a list of relevant policy statements. To reach consensus on the NSDES goal, you can follow this process and complete Table 4 below (examples are in italics). 1. Reflecting on the overall policy context, draft an initial version of the NSDES vision statement/goal. For example: By 2020 develop sector-wide capacities of National Education Statistical System (NESS) stakeholders to produce statistics and measure progress towards national, regional and global education targets. 2. Identify the major stakeholders who are relevant to the goal. 3. Identify why the goal is relevant; what does each major stakeholder expect to achieve from the goal? 4. Identify what role each stakeholder plays in achieving the goal. 5. Reflecting on the results of the discussion, redraft the NSDES vision statement/goal. In Table 4 below, the final vision statement/goal reflect changes, marked in bold, that reflect stakeholders' inputs. For example, the Ministry of Health and Social Welfare is particularly concerned with 'school readiness'; the President's Office for Regional Administration and Local government is concerned with 'inclusive basic education', and the Ministries of Community Development and Labour are particularly concerned with "skills development"; and all stakeholders are invested in "lifelong learning opportunities for all". By Year 5 develop sector-wide capacities of National Education Statistical System (NESS) stakeholders to produce quality statistics from multiple data sources and track life-long learning opportunities for all, particularly in terms of school readiness, inclusive basic education, and equitable skills development for better livelihoods. Table 4. Sample discussion results and consensus-based goal DRAFT NSDES vision statement/goal: By Year 5 develop sector-wide capacities of National Education Statistical System (NESS) stakeholders to produce statistics and measure progress towards national, regional and global education targets. WHO are the major stakeholders relevant to the goal? WHY is the goal relevant to each stakeholder? WHAT role does each stakeholder play in advancing the goal? National Bureau of Statistics (NBS) NESS is integral to the NSS Cross-sector coordination Ministry of Education, Science and Training NESS is a pre-requisite for ESDP implementation Sector-wide coordination, data management, sector-wide reporting Share/export relevant data with education sector management President's Office-RALG NESS is a pre-requisite for basic education services Ministry of Health and Social Linked sector data-sets, particularly Welfare related to ECD/ECCE indicators Ministry of Community Linked data-sets, particularly Development related to skills training Ministry of Labour, Employment Linked data-sets, particularly Integrate labour market and Youth Development related to skills training information (LMI) into the NESS FINAL NSDES vision statement/goal: By Year 5 develop sector-wide capacities of National Education Statistical System (NESS) stakeholders to produce quality statistics from multiple data sources and track life-long learning opportunities for all, particularly in terms of school readiness, inclusive basic education, and equitable skills development for better livelihoods USING DATA QUALITY ASSESSMENTS (DQA) TO PREPARE THE GROUND 11

13 Prior to the preparation of the NSDES, Data Quality Assessment (DQA) will have been conducted at country level, using instruments developed by the UIS (see CapED training materials). These include assessments of administrative routine data systems, household surveys, and learning assessment studies. The DQA(s) will generate a set of recommendations. These can be categorized in terms of the following capacity components of the NSDES. Component 1. Institutional and policy environment Component 2. Organizational/statistical processes Component 3. Technical capacity (data system and dissemination) Component 4. Human resource capacity Each component can be broken down into specific elements. These elements are a reference for you to group recommendations under each component. For example, the elements of Component 1 will guide the grouping of DQA recommendations related to the institutional and policy environment, as follows Component 1. Institutional and policy environment Recommendations related to: Institutional re-structuring and coordination Clarity of mandates, roles and responsibilities, collaboration and accountability among stakeholders Legal/policy reform for credible official statistics (transparency, confidentiality and public trust) Adequacy of financial resources Statistical quality awareness and relevance to users OTHER As recommendations may emerge from additional sources besides the DQA, the table includes 'OTHER', in order that further elements may be incorporated into the categorization framework. It is important to note that the above categorization is indicative only, as the grouping of recommendations may differ from one country context to another. To group DQA recommendations you can follow the process outlined below, using Tool 1 'Prioritizing DQA recommendations and setting objectives' in the Annex, 'Tool-kit', which includes examples in italics. Table 5 below presents the contents of Tool 1. It shows the following: components for DQA recommendations and elements of each component; DQA recommendations by component; the relative importance of each recommendation; and a strategic objective related to the overall component. 1. Review the recommendations provided in the DQA report; are they clear? Familiarize yourselves with each component and its elements provided in Column A; do you need to include additional elements, or will the category 'OTHER' suffice? 2. Map the DQA recommendations against each component. Ideally, this should be in relation to the elements of the component; or against the category 'OTHER'. Enter the information in Column B. 3. Score the relative importance of each recommendation on a simple scale of low to high (1-3), in Column C. 4. Reflect on the high-scoring recommendations. Brainstorm on an intended change in relation to a NSDES component. Decide on the strategic objective that describes the change and enter the information in Column C. What is a 'strategic objective'? A strategic objective is a higher-order outcome representing measurable capacity-related change in institutions, systems, organizations and population groups. Together, the achievement of all strategic objectives will result in the ultimate NSDES vision/goal. A strategic objective relates to one of the 4 capacity components of a NSDES, as outlined above and elaborated in Table 4 below. 12

14 It is critically important that you do not formulate a strategic objective in relation to a single recommendation. Rather, you should refer to the recommendations to guide your brainstorming on the strategic objective. A single recommendation will most likely form the basis of an Intermediate Outcome, which is discussed in 2.2, Step 1, below. TO DO examples 13

15 Table 5. Indicative categorization of DQA recommendations (Examples are found in the Annex, Tool 1) COLUMN A Component and elements of each component COLUMN B Recommendations Relative importance Component 1. Institutional and policy environment Institutional re-structuring and coordination Clarity of mandates, roles and responsibilities, collaboration and accountability among stakeholders Legal/policy reform for credible official statistics (transparency, confidentiality and public trust) Adequacy of financial resources Statistical quality awareness and relevance to users OTHER: Component 2. Organizational/Statistical processes Statistical methodologies (concepts, definitions and classifications in line with international standards) Reliable and accurate data sources to sufficiently portray reality Routine data collection processes External data sources to meet gaps in administrative data; harmonization of multiple data sources OTHER: Component 3. Technical capacity (data system and dissemination) Database software Basic services and connectivity Periodicity, timeliness and consistency of statistical outputs Data analysis and reporting methods and mechanisms Dissemination and user-feedback mechanisms OTHER: Component 4. Human resource capacity assessment and staff recruitment Training of staff at all levels in statistical skills and knowledge Incentive schemes to attract and retain competent staff Data management skills training and professional development for NESS staff OTHER: COLUMN C Strategic Objective 14

16 2.2. DESIGNING THE RESULTS FRAMEWORK In this section we will prepare a results matrix and accompanying monitoring and evaluation (M&E) matrix. Together, these constitute the results framework which sets the strategic direction and expected results of the NSDES in the country. The results matrix contains the following information: 1. Strategic objectives/outcomes (no more than 4) 2. Intermediate Outcomes 3. Outputs 4. Implementing Partners 5. Assumptions and risks 6. Indicative resources. A Results Matrix template (Tool 2) is provided in the Annex, 'Tool-kit' and includes examples in italics. STEP 1. DEFINING RESULTS By the end of the first stage of preparing a NSDES (Getting Started), you will have a the higher-order results of the strategy: the vision statement/goal and a set of strategic objectives. The task now is to define the following results in the NSDES results framework: outcomes, intermediate outcomes and outputs. Results are about change. To define results you will need to use 'change language' and not 'action language'. Action language expresses would-be results from the providers perspective and usually starts with by doing this or that. This type of language can be interpreted in many ways because it is not specific or measurable (e.g. improve statistical outputs). It tends to focus only on the completion of activities (e.g. establish EMIS units in 15 regional offices). Change language describes changes in the conditions of institutions and people. It sets precise criteria for success. It focuses on results and not on the methods to achieve them; expressions such as through this and that or by doing this and that should be avoided (Figure 3). The following are some examples of results using change language: The Education Platform develop, cost and implement an EMIS policy; The National Statistical Office (NSO) and line ministries follow a schedule of regular meetings; An adequate budget, including annual recurrent costs, is available for EMIS development and management; The Statistical Act includes measures for treating individual data with confidentiality; The National Technical Team puts in place appropriate mechanisms to prevent unauthorized access and use of individual data; A User Satisfaction Survey is conducted among users. Figure 3. Key elements of 'change language' Emphasize the 'new' condition Be specific: 'who' and 'what'? Don't include information on 'how'. Use active not passive language 15

17 Formulating strategic objectives as final outcomes You may find that the strategic objectives you have identified are expressed in 'action language'. If this is the case, you will need to formulate the objectives using 'change language'. In this way, the strategic objectives will represent the (final) outcomes of the NSDES. The following checklist may help you brainstorm and formulate an outcome, based on your strategic objectives. Checklist 1. Outcomes Checklist The outcome describes change (the results of actions) and not the actions themselves The outcome conveys high level institutional and/or behavioural changes The outcome is specific, it does not combine two or more different types of results and is not so general that it could cover just about anything The outcome includes a subject of the intended change (a who ) The outcome specifies a realistic time-fame (e.g. a five-year period) The outcome is clearly stated and the wording is unambiguous. Test the wording. The objective should not be interpreted differently by different constituents. Formulating intermediate outcomes While a final outcome is a change in institutional performance or behaviour change among the users of outputs, an intermediate outcome specifies an interim result that leads to the final outcome but is more measurable and achievable in the lifetime of the program. In a nutshell, an intermediate outcome is an improvement in the ability of stakeholders to take action. These can be: Positive changes that occur in an individual or a group of individuals, such as improvements in knowledge and skills, or changes in motivation and attitude with respect to a particular issue; and Changes that occur in the interactions among individuals and groups - in the broader organizational or social environment - which are embodied in improved processes or in new products and services. We can classify intermediate outcomes in terms of three types: 1. Enhanced skills and implementation know-how. Increased ability to act through: Acquisition of new knowledge or skills Application of new knowledge or skills 2. Raised awareness. Increased ability to act through improved: Understanding Attitude Motivation 3. Improved consensus and teamwork. Increased ability to act through improved: Coordination Processes for collaboration Communication and networking It is important to note that the above listing is not a sequence. Rather, you should draw on the typology when formulating one or more intermediate capacity outcomes in relation to a specific (final) outcome. 16

18 Before moving on to next element in the results framework, check the intermediate outcome: Does it describe change (the results of actions) and not the actions themselves? Does it include a subject of the intended change (a who )? Does is specify a realistic time-fame (e.g. a one to three-year period)? Is it clearly stated? Formulating outputs Outputs are commitments. They are results that a programme or project must achieve with the resources provided and within the time-period specified, usually less than 5 years. This makes outputs very different from outcomes or impacts, which are longer term changes that usually require the energy and resources of multiple partners. Outputs are the type of results over which programme managers have a high degree of influence and failure to deliver outputs may be viewed as a failure of the programme. In general, if the result is mostly beyond the control or influence of the programme or project, it cannot be an output. Outputs are deliverables that relate to operational change: the availability of new products and services. A single output will almost never be sufficient to achieve the institutional or behavioural changes implicit at the outcome level. On the other hand, if a programme or project has too many outputs, it can lose focus and be difficult to manage. A key feature of outputs is that they should be tangible. When defining outputs, be cautious about using words like "endorsed" or "empowered". Such words may be more appropriate at the outcome level, because they depend critically on the actions of others, outside the control of a programme or project. You may be tempted to list things like workshops and seminars as outputs. After all, they are deliverable and some workshops can be strategic if they gather decision takers in one room to build consensus. But, in most cases, workshops and seminars are activities rather than outputs. And outputs are not simply completed activities; they are the tangible changes in products and services, new skills or abilities that result from the completion of several activities. Checklist 2. The following checklist may help you brainstorm the formulation of outputs. Checklist The output is a new deliverable that can be developed by the NSDES programme The output is linked to one (and only one) outcome. Each output is stated using change language, and in the form of a deliverable, that will be completed in less than 5 years (see Figure 4 below). The output does not repeat the outcome statement above it. There is obvious or intuitive causality between the output and the outcome to which it contributes (If then). The sum of the outputs is sufficient to achieve the higher level outcome. The output is not a completed activity (eg. Training conducted; Workshop completed; Survey implemented these are activities). The output is specific; it does not combine two or more different types of results, and is not so general that it could cover just about anything. 17

19 Figure 4 Using change language to formulate an output A B C D Use results language to emphasize the future situation e.g. The capacity of statisticians to do X by undertaking Y is strengthened Be specific; are there particularly weak or under-resourced stakeholder groups? e.g. The capacity of statisticians in the 3 targeted MDAs to do X is strengthened by undertaking Y Take out information that relates to either strategy or activities e.g. The capacity of statisticians in the 3 targeted MDAs to do X is strengthened Emphasize change; shift from passive to active language e.g. Statisticians in 3 targeted MDAs have the capacity to do X STEP 2. CLARIFYING PARTNERS' ROLES A critically important element in the NSDES results matrix who is responsible for achieving specific results - i.e. outputs and intermediate outcome - and the type of role they play (Figure 5). Broadly, there are two types of partners: The lead partner: the organization responsible for initiating and managing the process mapped out through the NSDES results chain; this organization will most likely be drawn from the education data platform; Implementing partners: the organization(s) or group(s) - there may be more than one - responsible for implementing key action areas, which result in an output To specify partners in the results matrix, you should enter not only the organization(s) involved, but also the particular group(s) or department(s) within the organization and, if necessary, the particular individuals within the group(s). While identifying partners and their roles in relation to intended results, it is important to bear in mind four simple points: 1. A 'learning by doing' capacity development approach implemented through the NSDES will mostly like target partners themselves; i.e. members of the Education Data Platform. In Figure 5 below, these are 'Lead Partners'. 2. Other external individuals or groups who are not part of the education data platform or NTT may also be the users of particular outputs, or beneficiaries of specific intermediate capacity outcomes; for example Head teachers, school management boards, or consultancy groups providing targeted long-term technical support. In Figure 5 below, these are 'Implementing Partners'. 3. It may not be possible to identify in advance all NSDES partners responsible for managing, implementing and monitoring the strategic program plan; as the program evolves over time, so will the partnerships underlying the NSDES. In Figure 5 below, these are 'New Partners'. 18

20 4. Risks to the change process should be clearly identified, with suitable provision for monitoring and adaptation. A key question, which is addressed in the following section is this: How will the NSDES design team ensure that the work environment of identified implementation partners is conducive for using newly acquired skills? Figure 5. Partners and their roles Implementing partners (e.g. CSOs, external TA) New partners (emerging as the program evolves) Lead partners (NTT) Targeted beneficiaries (e.g. head teachers) STEP 3. IDENTIFYING ASSUMPTIONS AND RISKS Assumptions describe the conditions which must exist for the 'if-then' relationship between the different levels of results to behave as expected. As such, they are the linkages that bind together elements in the results chain. Assumptions may be internal or external to the development intervention itself. Risks on the other hand are future external negative events which can potentially alter the achievement of desired results. These may include strategic, environmental, financial, operational, organizational, political and regulatory risks. As such, risks are important in that they provide an opportunity to anticipate and specify reasons why a programme may not work out as intended. A country's NSDES will inevitably rely on some assumptions about factors that are beyond the control of the planners and implementers. Generally, results frameworks should not be based on critical assumptions that are perceived to have a low probability of holding true over the implementation period. If the risks are high, the entire strategy needs to be reconsidered. As Figure 6 illustrates, at the output-level assumptions about the transformation of activities into outputs are usually highly probable; that is, they can be expected to come true in most cases. Remember, outputs are commitments, so NSDES partners should have a high level of control over their achievement. But assumptions at the higher levels - for example, relating to the transformation of outputs into outcomes - can be expected to be less probable. New skills and abilities, or new services and products can be developed, but the NSDES partners have little control over whether these operational changes lead to changes in institutional performance and behaviour. Because NSDES partners can influence these actions, but not control them, assumptions are most important at: the interface between National goals and NSDES outcomes; and the interface between final outcomes and intermediate outcomes. 19

21 Figure 6. Assumptions and probability Decreasing programme management control Outputs to intermediate outcomes Probable assumptions Intermediate outcomes to final outcomes Less probable assumptions Outcomes to impact Often unpredictable assumptions It is important that assumptions are formulated after the results chain (impact-outcomes-outputs) - and the clarification of partners' roles - and before the indicators. This is because identification of assumptions is crucial and can lead to a redefinition of the results chain, or the identification of additional partners. To identify critical assumptions and risks, you may follow this process. 1. For each outcome and output considered critical in the results chain, identify conditions that are essential in order that the respective result is achieved. The expectation is that if the outputs have been delivered and the assumptions in the programme document still hold true, then the outcome will be achieved. Assumptions should be stated in positive language. Examples of assumptions for the NSDES include: There is political, economic, and social stability in the country; Sustained support and leadership of the government is well-established; Political will exists to implement the proposed strategy; Planned budget allocations are actually made by financial partners. Using the risk matrix (Table 6 below), enter those outcomes that carry risks. You will find a Risk Matrix template (Tool 3) in the Annex, 'Tool-kit', which includes examples in italics. 2. Identify risks for outcomes in the Results Matrix. Prioritize the identified risks, ranking them according to their likelihood of happening (low, medium or high); 3. Define a risk mitigation strategy for each medium-to-high risk. The following are a range of risk mitigation strategies that may be considered: Prevention: Prevent the risk from materializing or prevent it from having an impact on objectives; Reduction: Reduce the likelihood of the risk developing or limiting the impact in case it materializes; Contingency plan: Prepare actions to implement should the risk occur; Acceptance: Based on a cost/benefit analysis, accept the possibility that the risk may occur and go ahead without further measures to address the risk. During the operational programming stage (Section 2.3 of the Guidelines) it is good practice to incorporate these risk-mitigation strategies into the NSDES. Risk mitigation strategies may be included as key action areas. 20