Module 4: Formulating M&E Questions and Indicators

Transcription

1 Module 4: Formulating M&E Questions and Indicators

2 Four Steps to Developing an Establish the M&E planning team and: M&E Plan 1. Align projects and activities with program goals. 2. Identify information users and their needs (program stakeholders). 3. Develop program conceptual framework. 4. Outline key M&E questions and indicators and create M&E framework.

3 Learning Objectives Formulate process and outcome questions that M&E data can answer Create and define indicators linked to program logic model List key attributes of good indicators

4 Developing M&E Questions What questions are funders or HQ staff asking? What questions are more useful in assisting in program improvement and success? What questions are easier or more feasible to answer? What questions cost less to answer?

5 What Questions Do You Need to Answer With Program M&E Data?

6 Areas of Inquiry Quality of services Access to services Cost and cost-effectiveness Performance Effect of public health activities Effect of policy changes Disease surveillance

7 Process M&E Questions Process monitoring What is being done? By whom? Target population? When? How much? How often? Additional outputs? Resources used? (Staff, funds, materials, etc.) Process evaluation Is the content of the intervention or the activity being delivered as planned? Does the content of the intervention or the activity reflect the requisite standards?

8 Outcome M&E Questions Outcome monitoring Which outcome objectives are being met? By how much? Outcome evaluation Are project activities resulting in a significant level of desired change? If so, what is the level of change?

9 Answering the Question: Data and Indicators

10 What is Data?

11 Data Data are raw numbers or other findings which, by themselves, are of limited value to decision makers. Example: the number of PLWHA on ARVs means little to decision makers until this is analyzed and compared with the total number of PLWHA in Lesotho or compared with the number of PLWHA on ARVs during the previous year.

12 What is an Indicator? a variable (its value varies) that measures (objective calculation of value) one aspect of a program/project (zeroes in on a key dimension)

13 Indicators An indicator is a unit of information, measured over time, that documents change. An indicator provides evidence of how much has been or has not been achieved Indicators are usually quantitative (numberrelated) measures but may also be qualitative (narrative-related) observations. Indicators enable a large amount of data to be reduced down to its simplest form.

14 Indicator Example: Representing Knowledge about HIV/AIDS among employees in our work place % of employees who both correctly identify ways of preventing the sexual transmission of HIV and reject major misconceptions about HIV transmission Example: Representing condom use among employees : % of sexually active employees reporting the consistent use of a condom during sexual intercourse with a non-regular partner

15 Why are indicators important? Indicators serve as tools to examine trends and highlight problems When compared with targets, indicators can signal the need for corrective management action, evaluate the effectiveness of various management actions, and help determine if objectives are being achieved. It is important to note that there is no requirement to develop an indicator for every possible issue. Even the most ambitious program is likely to include a limited number of indicators.

16 What indicators are not They are NOT anything that can be measured. They should reduce a large amount of data down to its simplest form. They are NOT Objectives, targets, or results. They do not specify a particular level of achievement. The words improved, increased, gained, etc. do not belong in an indicator

17 Criteria for assessing an indicator Valid: Measures in fact what it intends to measure conceptually; measures only the specific phenomena it is intended to measure; will capture changes in the state of the phenomenon under study Reliable: Produces the same results when used more than once to measure the same phenomenon Precise: Is operationally defined in clear terms Timely: Provides a measurement at time intervals relevant and appropriate in terms of program goals and activities Feasible: is practical to collect and calculate Useful to management: Information provided by the measure is critical to decision making

18 Selecting Program Indicators Here are some aspects that could lead you to select an indicator: If it represents a priority local problem If it measures the objectives or goals of the program If it measures important activities or services If it can be used locally If it is action-oriented If it is based on epidemiologically determined most important threats to a population If it measures cost-effectiveness

19 Indicator Types and Examples Process Indicators: # of qualified trainers/providers trained # condoms distributed % of VCT clients that accept to be tested after pretest counseling Outcome Indicators: % of young people aged using condoms at last intercourse % of people with accepting attitudes to PLWHA Impact Indicators: % of adult population infected with HIV (prevalence rate)

20 Constructing Indicators

21 Indicator Calculation Examples % of (denominator) that are in (numerator) % of participants that are female Numerator Denominator # of females in the workshop # of participants % of babies born to HIV+ mothers that received Numerator # of babies born to HIV+ mothers that received Nevirapine Nevirapine Denominator # of babies born to HIV+ mothers % of ANC clients that test positive for HIV, among those tested Numerator Denominator # of HIV+ ANC clients # of ANC clients tested for HIV

22 Group Activity on Indicator Calculation Numerator Denominator % of VCT clients that attend a group education session on VCT % of nurses trained in VCT % of deliveries that take place in hospitals % of HIV+ men who get post-test counseling

23 Rate An indicator is often expressed as a rate. A rate is a measure of an event (numerator) within a specific population who is at risk for an event (denominator) during a specific period of time (usually one year), multiplied by a constant (usually 1,000 or 100,000). Example: the adult HIV prevalence rate is the number of HIV+ adults (numerator) among the adult population (denominator) in a specific one-year period, multiplied by 100.

24 Ratio An indicator is sometimes expressed as a ratio. A ratio is Example: the number of condoms distributed per male of reproductive age. Ratio of current school attendance among orphans to that among non-orphans aged 10-14

25 Exercise 4.1: Creating Indicators from Your Program Logic Model

26 Example (1) Indicator Definition Unit of Measurement Data Source (data collection method) Data Collection Frequency of Report Generation Responsibility for collection of data Report Audience % of HIV positive employees provided with ARVs No. of HIV+ employees provided with ARVs Number HIV + employees employees ARV register quarterly Six monthly HIV/AIDS work place coordinator Manage ment committ ee /Board of the company

27 Example (2) Indicator Definition Unit of Measurement Data Source (data collection method) Data Collection Frequency of Report Generation Responsibility for collection of data Report Audience % of employees educated on HIV prevention No. of employees educated on HIV prevention Number of employees employees Peer education register quarterly Six monthly HIV/AIDS work place coordinator Manage ment committ ee /Board of the company

28 Discussion What did you find easy in this exercise? What was difficult? Did everyone agree on what makes one indicator more important than another? Did everyone agree on what makes an indicator more feasible to get and use than another?