National Level Results Based Monitoring and Evaluation System. Chris Bishop Medical Epidemiologist AIPHSS

Transcription

1 National Level Results Based Monitoring and Evaluation System Chris Bishop Medical Epidemiologist AIPHSS

2 Objective Identify and understand the 10 steps needed to develop a National Level Results Based Monitoring and Evaluation System

3 Steps to develop a National Level Monitoring and Evaluation System Step 1: Readiness Assessment Step 2: Agree on Outcomes Step 3: Select Key Indicators to monitor Outcomes Step 4: Baseline Data Step 5: Planning for Improvement: Selecting Results and Targets

4 Steps to develop a National Level Monitoring and Evaluation System Step 6: Monitoring for Results (Data Analysis) Step 7: Role of Evaluations Step 8: Reporting Findings Step 9: Using Findings Step 10: Sustaining the M&E System

5 Step 1: Readiness Assessment Need thorough understanding of institutional and organizational roles, responsibilities and capabilities, incentives (political, institutional or personal) and demands, what is needed to sustain the system Readiness Assessment 3 main parts: A. Incentives and Demands B. Roles and Responsibilities for assessing government performance C. Capacity Building Requirements

6 Step 1: Readiness Assessment 8 Key Questions 1. What potential pressures are encouraging the need for the M&E system within the Public Sector and Why? 2. Who is the advocate for the M&E System? 3. What is motivating the champions? 4. Who will own the system? Who will benefit? How much information is really needed?

7 Step 1: Readiness Assessment 5. How will the system directly support better resource allocation and the achievement of goals? 6. How will the institution, organization, champions and staff react to negative information? 7. Where is the capacity to support a results based M&E system? 8. How will the M&E system link project, program, sector and national goals?

8 Assessment Tools Health Metric Network (HMN): HMN has a framework and standards for establishing and assessing health information systems. Measure Evaluate: ; Capacity Building Guides: Processes that improve the ability of a person, group, organization or system to meet objective or to perform better. Data Demand and Use Tools: Core tools to stimulate data demand and capacity building and enhance evidence-based decision making. Data Quality Assurance: Methods for assessing M&E plans and systems that collect and report data for program management and reporting. PRISM: Performance of Routine Information System Management: A framework that ensure relevant data and guarantees that outcomes will be usable for decision making. M&E Skills Assessment

9 Enabling Factors Strong human, institutional and management capacity in the public sector Public-service known for integrity, honesty and professionalism A tradition of accountability and transparency Credible, legitimate political leaders (whole of government approach with performance linked to public expenditure Evaluation institutionalized: evaluation of policy, programs and projects Public involvement and satisfaction important Information demanded Strong inter-ministerial communication and co-operation

10 Indonesian Strengths and Challenges Over to you??

11 Step 2: Agree on Outcomes This section: Importance of Outcomes Issues to consider Participatory and consultative approach with main stakeholders Process for setting and agreeing on outcomes

12 Step 2: Agree on Outcomes Importance of Outcomes: Goals: Long term Outcomes: intermediate timeframe 5 to 10 years: Defines what success will look like Demonstrates whether success has been achieved Shows which road to take Target: 1 to 3 year

13 Step 2: Agree on Outcomes Issues to consider Strategic Priorities Desired Outcomes Outcomes should be linked to international issues, national development goals and plans, socio-economic and political benchmarks, political promises What are the priorities for the Indonesian MOH?

14 Step 2: Agree on Outcomes Participatory and consultative approach with main stakeholders Setting goals in isolation leads to lack of ownership Approach starts with development goals and continues with setting outcomes and building an indicator system Who are the MOH key stakeholders?

15 Step 2: Agree on Outcomes Process for setting and agreeing on outcomes Where are you going? Why are you going there? How will you know when you get there? 1. Identify Specific Stakeholder Representatives: How are they categorized? (MOH (MNCH, Nutrition, Curative etc.) NGO, donor, government) 2. Identify Major Concerns of Stakeholder Groups: FGD, Brainstorming, Surveys and Interviews, Strategic Plans

16 Step 2: Agree on Outcomes 3. Translate Problems in Statements of Possible Outcome Improvements: Outcome statement identifies road and destination and framed positively. 4. Disaggregate to Capture Key Desired Outcome: One improvement area in each outcome statement. (Increase vaccination rates among 1 year olds in the rural sector by 20% over the next 5 years)

17 Step 2: Agree on Outcomes 5. Develop a plan to assess how a government of institution or organization will achieve these outcomes: A list of tasks or activities may not mean that the desired outcomes have resulted. The actions and resources needed to manage also important. What is the resource envelope, do we have the skills, is it reasonable? Being busy not the same thing as achieving results How well has the MOH considered the resource envelope in deciding outcomes? Have skills gaps been identified?

18 Step 3: Selecting Key Indicators to Monitor Outcomes Outcome indicators are not the same as outcomes. Indicators are the quantitative or qualitative variables that provide a simple and reliable means to measure achievement, to reflect the changes connected to an intervention against the stated outcome. How will we know success or achievement when we see it? Are we moving toward achieving our desired outcomes?

19 Step 3: Selecting Key Indicators to Monitor Outcomes Need to consider: 1. Indicators needed at all levels of the M&E system 2. Translating Outcomes into Outcome indicators 3. CREAM 4. Use of proxy indicators 5. Pros and cons of using predesigned indicators 6. Constructing indicators and tracking performance information 7. Setting indicators using experience

20 Step 3: Selecting Key Indicators to Monitor Outcomes 1. Indicators are needed for all levels of results-based M&E systems (input, output, process, outcome and goals) with feedback at regular periods against expected targets. 2. Translating Outcomes into Outcome indicators: Complicated process; Need minimum number of indicators that answers the question Has the outcome been achieved; Take into account multiple stakeholder needs; several direct indicators might be needed to measure outcomes; disaggregate indicators (gender, location, profession etc.);

21 Step 3: Selecting Key Indicators to Monitor Outcomes 3. CREAM: (Qualitative and Quantitative) Clear: Precise Relevant: Appropriate Economic: Reasonable cost Adequate: Provide sufficient basis to assess performance Monitorable: Can be independently validated

22 Step 3: Selecting Key Indicators to Monitor Outcomes Quantitative: number, mean, median or percentage. Measures actual progress. Qualitative: compliance with, quality of, extent of and level of..provide insights into changes in institutional processes, attitudes, beliefs, motives and behaviours. Used with caution as subjectively measures perception.

23 Step 3: Selecting Key Indicators to Monitor Outcomes 4. Use of proxy indicators: Better to be approximately correct than precisely wrong. Sometimes difficult to measure outcome indicators directly as too costly or not feasible to collect data at regular intervals. Assess contributing factors during analysis to understand attribution

24 Step 3: Selecting Key Indicators to Monitor Outcomes 5. Pros and Cons of using predesigned indicators: MDGs, UNDPs (HDI), GF etc. Pros: Aggregated across similar countries, projects, programs and policies Reduce cost of building multiple unique measurement systems Greater harmonization Cons: Often do not address country specific goals Imposed i.e. top down Do not promote stakeholder participation and ownership Can lead to adoption of multiple competing indicators Need same case definitions and data collection methods

25 Step 3: Selecting Key Indicators to Monitor Outcomes 6. Constructing Indicators: Can be difficult. Make sure you have the right people: Technical experts and Policy experts. Is the indicated tested Is the information useful as a management tool? 7. Setting Indicators Lead core indicators Quick data entry and validation Simple reporting Large sample Short questionnaire Easy data collection

26 Step 4: Baseline Data on Indicators What are the sources of data? What are the data collection methods? Who will collect the data? How often will the data be collected What is the cost and difficulty to collect the data? Who will analyse the data? Who will report the data? Who will use the data?

27 Step 5: Planning for Improvements Setting targets: What can be achieved in a specific time toward reaching the outcome Definition of target: A specified objective that indicates the number, timing and location of that which is to be realized. Factors to consider when selecting indicator targets: Need clear understanding of the baseline or starting point; an average of the last three years performance, last year s performance, average trend, data over the past six months etc. Expected funding and resource levels; existing capacity, budgets, personnel, facilities. Targets should be feasible taking into account resource considerations as well as organizational capacity to deliver activities and outputs Set targets within a three year period. Easier to consider influences and to be realistic Targets can be ranges Targets specify what is being tracked, the expected amount of change or improvement and a timeframe for achievement

28 Step 6: Monitoring Results Not the same as monitoring workplans and activities (inputs and outputs) Results based M&E systems focus on outcomes Results based M&E systems demonstrate whether results have been achieved Key Principles: Identify information needs of stakeholders Identify units of anlyssis Sampling proceedures Data collection instruments (valid, reliable and timely) Frequency of data collection Clear responsibilities for data collection and reporting Expected methods of analysis and interpretation Data assurance proceedures Dissemination procedure: results information must move horizontally and vertically in the organization Follow-up on findings

29 Analysing Performance Data A. Are the data accurate? Do the data measure what they are supposed to measure? Were the calculations made correctly? B. Are the data reliable? Are you confident that you could come up with the same numbers if you were to review the source registers and documents again? C. Do any of the numbers cause you immediate concern or alarm even before doing any additional calculations? Sometimes you might find a figure that seems way out of line with all of the others! Did someone enter an extra 0 a common mistake that inflates numbers? D. How do the numbers compare to previous time periods? Is there a trend towards improvement even if the numbers are not as good as you had hoped? Drawing a simple graph to identify trends. Trends are gradual increases or decreases over time. Look carefully for trends, since they may occur so gradually that change is not observed. E. You might like to compare clinical indicators: clinical tuberculosis and number of cases detected, number of antenatal against deliveries, deliveries and post natal care, curative with outpatient cases, TT2 with ANC, number of CHW with number of referrals for ANC. Open or functional health facilities with maternal and infant mortality. Link indicator with other data sources e.g. stock out with drugs. Resources available may also explain your results. Low clinic attendance with stock out of drugs.

30 Analysing Performance Data F. If the reports identify specific problem areas, what might be the causes? Seasonal variations. Regular changes occurring according to the time of year. The incidence of measles, diarrhea, ARI and malaria, for example, varies seasonally. Increased accessibility of facilities due to seasonal weather changes can explain a suddenly increased number of patient visits. A sudden spike or drop may be due to seasonal variation. Line charts are particularly helpful for displaying and identifying seasonal variations. Variation in trends can occur for many reasons including: Political and Governance factors: including international ties to donors. International donors and agencies can direct domestic health priorities and strategies. The leadership and management of health programs including structures and delegations, the ability to strategically plan, prioritize and operationalize programs influences health outcomes. Think also about changes in community characteristics, such as an aging population or decreasing pool of potential new patients.

31 Analysing Performance Data External Factors: Geographical: affects access to health services and the availability of land and water supplies appropriate for cultivation of sustainable food crops. Rugged terrain prevents the rapid distribution of health and food supplies particularly during harsh winters and can significantly affect rural populations access to the most basic of health services. Adequate housing: Overcrowding can result in severe health and wellbeing problems as it places excessive demand on bathroom, kitchen and laundry facilities and can lead to the spread of infectious diseases Economic: The available economic resources to implement health programs to address national health priorities and reduce the incidence of disease affect health outcomes. Access to work and meaningful employment: Working conditions that do not provide stability of employment (such as highly casualised employment) are associated with adverse health outcomes. Conversely, learning environments can help people interact with others and develop networks which have positive impacts on health and wellbeing. Education: An individual s education status influences health seeking behavior Social environments: including the way in which households divide labor.

32 Analysing Performance Data External Factors: Income: Low incomes reduce people s access to the resources necessary to maintain and improve health Safety and justice: Perceptions of a lack of safety contribute to stress and levels of fear in individuals and reduce levels of trust and social connection. This can lead to limited access to the health and community services necessary to maintain health and wellbeing. Security can also influence migration patterns within and between countries. There is often in developing countries a high urban migration rate which prevents people from planting supplementary food crops in cities where land is limited. Ethnic/race based discrimination: Racism is a key determinant of health inequalities, with increased exposure to racism associated with worse mental health outcomes. Urban planning that promotes social mix and transport Gender: There can be a gender bias which influences a person s access to health services Culture: can influence child care practices and health seeking behavior Demographic patterns: settlement patterns, sanitation and animal husbandry practices

33 Analysing Performance Data Specific Health Related Factors: Clinic staffing: Lack of a female health worker, lack of female doctors, staff resignations or staff leaving Supply of Goods: Pharmacy not being delivered. No surgical supplies or contraceptives. Health services/community mobilization: Consumer engagement in the planning and delivering of health services such as a family action group or health shura may improve access and availability of health services. Accessible health and community service infrastructure: Access to health, social and community services builds capacity for individuals to maintain good health and avoid illness. Access to health services ensures more timely treatment and prevention

34 Analysing Performance Data G. Are there other sources of information that may assist in better understanding the causes of poor health comes. E.g. monitoring visits, supervisory visits, household surveys H. Discuss reports with colleagues when they are first prepared. Include discussions of reports (even preliminary findings) in regular staff meetings. Others can offer useful insights into why specific conditions are getting better or worse and have helpful suggestions about changes which could be made. I. Provide feedback to those who have sent reports from other units. Doing this can be as simple as acknowledging the receipt of the report and responding to any issues that require your action. Better yet, you can develop a routine mechanism to feedback some analysis of key indicators on reports you receive from all units.

35 Step 7: Role of Evaluations Evaluation: Assessment of a planned, ongoing or complete intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process. Evaluation plan which can assess policy programs, population demographics and behaviours

36 Step 8: Reporting Findings Reports address information needs of stakeholders What is to be reported, when it is reported and to whom Assess the current technical capacity of the organization to accumulate, assess and prepare analyses and reports Presentation of performance data needs to be clear and understandable reporting results in comparison to earlier data and to baseline. Only present the most important data Avoid jargon and acronyms Minimum background information to establish context Major points upfront Findings and recommendations should be organized around key outcomes and their indicators. Uses: What are some of the information needs at the MOH? How is information being used?

37 Step 9: Using Findings Uses: Respond to elected official and public demands Help formulate and justify budget requests Help make operations resource allocation decisions Trigger in-depth examinations of what performance problems exist and what corrections are needed Help motivate personnel to continue making improvements Provide data for in-depth evaluations Improve service provision Support strategic and other long-term planning efforts Communicate better with public and build trust Any others?? Establish committees for reporting results Feedback, knowledge and learning: Share and compare results findings with partners What else?? What are the obstacles??

38 Step 10: Sustaining the M&E System 7 critical components of sustaining results based M&E systems 1. Demand: Structured requirements for reporting results including legislation, regulations lead to sustained consistent demand for results 2. Clear Roles and Responsibilities: collecting, analysing and reporting for continuous system of data collection and analysis across levels of government 3. Trustworthy and Credible Information: both good and bad news, transparent information which can be independently validated. Need protection from political reprisals. 4. Accountability: CSO, NGOS, media, private sector. Problems acknowledged and addressed. 5. Capacity: Sound technical skills in data collection and analysis. 6. Incentives: encourage use of performance information 7. Evaluate the M&E System and Information:

39 Questions Terimah Kasih